2024-03-04T05:54:19.376 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T05:54:19.376 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} --owner scheduled_nmordech@teuthology --archive /home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638 --name nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi --verbose -- /home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638/orig.config.yaml 2024-03-04T05:54:19.415 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T05:54:19.495 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638 branch: wip-nitzan-crimson-shard-pg-mapping-get-or-create description: crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} email: null first_in_suite: false job_id: '7580638' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-nitzan-crimson-shard-pg-mapping-get-or-create ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: 633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd thrashosds: min_in: 4 workunit: branch: wip-nitzan-crimson-shard-pg-mapping-get-or-create env: CRIMSON_COMPAT: '1' sha1: 633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd owner: scheduled_nmordech@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 4836 sha1: 633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd sleep_before_teardown: 0 suite: crimson-rados:thrash suite_branch: wip-nitzan-crimson-shard-pg-mapping-get-or-create suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlsRiv6BZzoFANvoD9juHdrbxi3MXvw3zBu/1yDTuv1+gmkPLYJCr7XgsZ+YtJJj4DCcQADwtCwRXr5hwaIUMQ= smithi066.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMFZcv76CLd2CM3w5dPYjbX7QYeAj/jEaNR1nJSu76qsrrYM794w1HuRY73Z3wK0f7ezvi7iPqh582AZT/KuZOk= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_down: 0 chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 0 delete: 50 read: 100 rollback: 0 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_05:47:36 tube: smithi user: nmordech verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-04T05:54:19.495 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa; will attempt to use it 2024-03-04T05:54:19.495 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks 2024-03-04T05:54:19.496 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T05:54:19.497 INFO:teuthology.task.internal:Checking packages... 2024-03-04T05:54:19.518 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd' 2024-03-04T05:54:19.518 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T05:54:19.518 INFO:teuthology.packaging:ref: None 2024-03-04T05:54:19.518 INFO:teuthology.packaging:tag: None 2024-03-04T05:54:19.518 INFO:teuthology.packaging:branch: wip-nitzan-crimson-shard-pg-mapping-get-or-create 2024-03-04T05:54:19.518 INFO:teuthology.packaging:sha1: 633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd 2024-03-04T05:54:19.518 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-nitzan-crimson-shard-pg-mapping-get-or-create 2024-03-04T05:54:19.654 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1550.g633e69c9 2024-03-04T05:54:19.655 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T05:54:19.661 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T05:54:19.661 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T05:54:19.685 INFO:teuthology.task.internal:Saving configuration 2024-03-04T05:54:19.702 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T05:54:19.708 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T05:54:19.732 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 05:47:51.790222', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlsRiv6BZzoFANvoD9juHdrbxi3MXvw3zBu/1yDTuv1+gmkPLYJCr7XgsZ+YtJJj4DCcQADwtCwRXr5hwaIUMQ='} 2024-03-04T05:54:19.755 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 05:47:51.788217', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMFZcv76CLd2CM3w5dPYjbX7QYeAj/jEaNR1nJSu76qsrrYM794w1HuRY73Z3wK0f7ezvi7iPqh582AZT/KuZOk='} 2024-03-04T05:54:19.755 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T05:54:19.761 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-03-04T05:54:19.761 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-04T05:54:19.761 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T05:54:19.832 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb3354eae50>, signals=[15]) 2024-03-04T05:54:19.833 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T05:54:19.839 INFO:teuthology.task.internal:Opening connections... 2024-03-04T05:54:19.839 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-03-04T05:54:19.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T05:54:19.946 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi066.front.sepia.ceph.com 2024-03-04T05:54:19.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T05:54:20.046 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T05:54:20.053 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-03-04T05:54:20.071 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-03-04T05:54:20.071 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-03-04T05:54:20.129 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-03-04T05:54:20.130 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-03-04T05:54:20.130 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-03-04T05:54:20.130 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-03-04T05:54:20.130 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-03-04T05:54:20.130 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-03-04T05:54:20.130 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T05:54:20.130 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-03-04T05:54:20.130 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-03-04T05:54:20.130 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T05:54:20.130 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-03-04T05:54:20.130 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T05:54:20.131 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T05:54:20.131 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T05:54:20.131 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-03-04T05:54:20.155 DEBUG:teuthology.orchestra.run.smithi066:> uname -m 2024-03-04T05:54:20.173 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2024-03-04T05:54:20.173 DEBUG:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2024-03-04T05:54:20.229 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="CentOS Stream" 2024-03-04T05:54:20.230 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="9" 2024-03-04T05:54:20.230 INFO:teuthology.orchestra.run.smithi066.stdout:ID="centos" 2024-03-04T05:54:20.230 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="rhel fedora" 2024-03-04T05:54:20.230 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="9" 2024-03-04T05:54:20.230 INFO:teuthology.orchestra.run.smithi066.stdout:PLATFORM_ID="platform:el9" 2024-03-04T05:54:20.230 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T05:54:20.230 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31" 2024-03-04T05:54:20.230 INFO:teuthology.orchestra.run.smithi066.stdout:LOGO="fedora-logo-icon" 2024-03-04T05:54:20.230 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T05:54:20.230 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://centos.org/" 2024-03-04T05:54:20.230 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T05:54:20.231 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T05:54:20.231 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T05:54:20.231 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2024-03-04T05:54:20.254 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T05:54:20.263 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T05:54:20.269 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T05:54:20.269 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-03-04T05:54:20.272 DEBUG:teuthology.orchestra.run.smithi066:> test '!' -e /home/ubuntu/cephtest 2024-03-04T05:54:20.287 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T05:54:20.293 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T05:54:20.293 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-03-04T05:54:20.329 DEBUG:teuthology.orchestra.run.smithi066:> test -z $(ls -A /var/lib/ceph) 2024-03-04T05:54:20.358 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T05:54:20.410 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T05:54:20.423 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-04T05:54:20.423 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-04T05:54:20.424 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T05:54:20.424 DEBUG:teuthology.orchestra.run.smithi066:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T05:54:20.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T05:54:20.440 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-03-04T05:54:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T05:54:20.441 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-04T05:54:20.497 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-361.el9.x86_64 2024-03-04T05:54:20.497 INFO:teuthology.task.kernel:Running kernel on smithi066: 5.14.0-361.el9.x86_64 2024-03-04T05:54:20.497 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum install -y kernel 2024-03-04T05:54:20.499 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-361.el9.x86_64 2024-03-04T05:54:20.499 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-361.el9.x86_64 2024-03-04T05:54:20.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-03-04T05:54:22.533 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-04T05:54:22.553 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-04T05:54:23.941 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2024-03-04T05:54:24.021 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-04T05:54:25.253 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-03-04T05:54:25.265 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 8.4 MB/s | 8.0 MB 00:00 2024-03-04T05:54:27.000 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-04T05:54:27.471 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-04T05:54:31.159 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-04T05:54:32.013 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 6.1 MB 00:00 2024-03-04T05:54:32.876 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-03-04T05:54:33.781 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 15 kB 00:00 2024-03-04T05:54:34.186 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2024-03-04T05:54:35.457 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2024-03-04T05:54:39.469 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-04T05:54:40.574 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-04T05:54:41.776 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T05:54:41.777 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T05:54:41.863 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T05:54:41.875 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T05:54:41.875 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-04T05:54:41.875 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T05:54:41.876 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T05:54:41.876 INFO:teuthology.orchestra.run.smithi066.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T05:54:41.876 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-04T05:54:41.876 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T05:54:41.876 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T05:54:41.876 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T05:54:41.876 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T05:54:41.876 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T05:54:41.877 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T05:54:41.877 INFO:teuthology.orchestra.run.smithi066.stdout:Install 4 Packages 2024-03-04T05:54:41.877 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T05:54:41.878 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 100 M 2024-03-04T05:54:41.878 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 123 M 2024-03-04T05:54:41.878 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T05:54:42.588 INFO:teuthology.orchestra.run.smithi066.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 14 MB/s | 6.0 MB 00:00 2024-03-04T05:54:42.889 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T05:54:42.889 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T05:54:42.988 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T05:54:42.996 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T05:54:42.996 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-04T05:54:42.996 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T05:54:42.996 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T05:54:42.996 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T05:54:42.996 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-04T05:54:42.996 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T05:54:42.996 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T05:54:42.997 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T05:54:42.997 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T05:54:42.997 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T05:54:42.997 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T05:54:42.997 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-03-04T05:54:42.997 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T05:54:42.997 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 100 M 2024-03-04T05:54:42.997 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 123 M 2024-03-04T05:54:42.997 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T05:54:43.130 INFO:teuthology.orchestra.run.smithi066.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 21 MB/s | 21 MB 00:00 2024-03-04T05:54:43.663 INFO:teuthology.orchestra.run.smithi066.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 31 MB/s | 33 MB 00:01 2024-03-04T05:54:44.162 INFO:teuthology.orchestra.run.smithi066.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 20 MB/s | 39 MB 00:02 2024-03-04T05:54:44.166 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T05:54:44.166 INFO:teuthology.orchestra.run.smithi066.stdout:Total 43 MB/s | 100 MB 00:02 2024-03-04T05:54:44.975 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 3.5 MB/s | 6.0 MB 00:01 2024-03-04T05:54:45.094 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T05:54:45.517 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T05:54:45.517 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T05:54:45.984 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 7.7 MB/s | 21 MB 00:02 2024-03-04T05:54:46.929 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T05:54:46.930 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T05:54:48.259 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 10 MB/s | 33 MB 00:03 2024-03-04T05:54:49.026 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 6.8 MB/s | 39 MB 00:05 2024-03-04T05:54:49.026 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T05:54:49.026 INFO:teuthology.orchestra.run.smithi064.stdout:Total 17 MB/s | 100 MB 00:06 2024-03-04T05:54:49.790 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T05:54:49.948 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T05:54:50.379 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T05:54:50.379 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T05:54:50.452 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T05:54:50.588 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T05:54:51.754 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T05:54:51.754 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T05:54:53.542 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T05:54:53.731 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T05:54:54.615 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T05:54:55.300 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T05:54:55.425 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T05:54:58.341 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T05:54:58.526 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T05:54:58.831 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T05:54:58.960 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T05:55:03.747 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:04.086 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:08.875 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:34.674 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:34.674 INFO:teuthology.orchestra.run.smithi066.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T05:55:34.675 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T05:55:34.762 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:35.240 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:35.240 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T05:55:35.240 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T05:55:35.240 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T05:55:35.652 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:35.652 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T05:55:35.652 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T05:55:35.652 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T05:55:35.652 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T05:55:35.652 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T05:55:35.652 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T05:55:35.653 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T05:55:35.653 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T05:55:36.153 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T05:55:36.309 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T05:55:36.309 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T05:55:36.309 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T05:55:36.309 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T05:55:36.310 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T05:55:36.310 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-03-04T05:55:36.310 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi066.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T05:55:36.310 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T05:55:36.436 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T05:55:36.437 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T05:55:36.437 DEBUG:teuthology.orchestra.run.smithi066:> sudo rpm -qi grub2-tools 2024-03-04T05:55:36.483 INFO:teuthology.orchestra.run.smithi066.stdout:Name : grub2-tools 2024-03-04T05:55:36.483 INFO:teuthology.orchestra.run.smithi066.stdout:Epoch : 1 2024-03-04T05:55:36.483 INFO:teuthology.orchestra.run.smithi066.stdout:Version : 2.06 2024-03-04T05:55:36.483 INFO:teuthology.orchestra.run.smithi066.stdout:Release : 68.el9 2024-03-04T05:55:36.483 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: x86_64 2024-03-04T05:55:36.484 INFO:teuthology.orchestra.run.smithi066.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T05:55:36.484 INFO:teuthology.orchestra.run.smithi066.stdout:Group : Unspecified 2024-03-04T05:55:36.484 INFO:teuthology.orchestra.run.smithi066.stdout:Size : 8264254 2024-03-04T05:55:36.484 INFO:teuthology.orchestra.run.smithi066.stdout:License : GPLv3+ 2024-03-04T05:55:36.484 INFO:teuthology.orchestra.run.smithi066.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T05:55:36.484 INFO:teuthology.orchestra.run.smithi066.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T05:55:36.484 INFO:teuthology.orchestra.run.smithi066.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T05:55:36.484 INFO:teuthology.orchestra.run.smithi066.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T05:55:36.484 INFO:teuthology.orchestra.run.smithi066.stdout:Packager : builder@centos.org 2024-03-04T05:55:36.484 INFO:teuthology.orchestra.run.smithi066.stdout:Vendor : CentOS 2024-03-04T05:55:36.484 INFO:teuthology.orchestra.run.smithi066.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T05:55:36.485 INFO:teuthology.orchestra.run.smithi066.stdout:Summary : Support tools for GRUB. 2024-03-04T05:55:36.485 INFO:teuthology.orchestra.run.smithi066.stdout:Description : 2024-03-04T05:55:36.485 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T05:55:36.485 INFO:teuthology.orchestra.run.smithi066.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T05:55:36.485 INFO:teuthology.orchestra.run.smithi066.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T05:55:36.485 INFO:teuthology.orchestra.run.smithi066.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T05:55:36.485 INFO:teuthology.orchestra.run.smithi066.stdout:hardware devices. 2024-03-04T05:55:36.485 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T05:55:36.485 INFO:teuthology.orchestra.run.smithi066.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T05:55:36.486 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T05:55:36.486 INFO:teuthology.task.kernel:Updating grub on smithi066 to boot 5.14.0-427.el9.x86_64 2024-03-04T05:55:36.486 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T05:55:37.060 INFO:teuthology.orchestra.run.smithi066.stderr:Generating grub configuration file ... 2024-03-04T05:55:38.446 INFO:teuthology.orchestra.run.smithi066.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T05:55:38.486 INFO:teuthology.orchestra.run.smithi066.stderr:done 2024-03-04T05:55:38.488 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2024-03-04T05:55:38.504 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.AdcTLF0tbj 2024-03-04T05:55:38.504 DEBUG:teuthology.orchestra.run.smithi066:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AdcTLF0tbj 2024-03-04T05:55:38.576 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.AdcTLF0tbj 2024-03-04T05:55:38.719 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.AdcTLF0tbj is 6KB 2024-03-04T05:55:38.768 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.AdcTLF0tbj 2024-03-04T05:55:38.783 DEBUG:teuthology.orchestra.run.smithi066:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T05:55:38.851 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T05:55:38.851 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T05:55:38.851 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T05:55:38.851 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T05:55:38.852 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T05:55:38.997 DEBUG:teuthology.orchestra.run.smithi066:> sudo shutdown -r now 2024-03-04T05:55:39.601 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:39.601 INFO:teuthology.orchestra.run.smithi064.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T05:55:39.601 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T05:55:39.690 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:40.162 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:40.162 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T05:55:40.162 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T05:55:40.162 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T05:55:40.605 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T05:55:40.605 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T05:55:40.606 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T05:55:40.606 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T05:55:40.606 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T05:55:40.606 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T05:55:40.606 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T05:55:40.606 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T05:55:40.606 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T05:55:41.096 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T05:55:41.239 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T05:55:41.240 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T05:55:41.240 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T05:55:41.240 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T05:55:41.240 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T05:55:41.241 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-04T05:55:41.241 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T05:55:41.241 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T05:55:41.366 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T05:55:41.366 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T05:55:41.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-03-04T05:55:41.404 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-03-04T05:55:41.404 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-03-04T05:55:41.404 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-03-04T05:55:41.404 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 68.el9 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8264254 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-03-04T05:55:41.405 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-03-04T05:55:41.406 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T05:55:41.406 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T05:55:41.406 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T05:55:41.406 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T05:55:41.406 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-03-04T05:55:41.406 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T05:55:41.406 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T05:55:41.407 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T05:55:41.407 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-427.el9.x86_64 2024-03-04T05:55:41.407 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T05:55:42.036 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-03-04T05:55:43.435 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T05:55:43.456 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-03-04T05:55:43.458 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-04T05:55:43.475 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.CkqamYYydo 2024-03-04T05:55:43.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CkqamYYydo 2024-03-04T05:55:43.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.CkqamYYydo 2024-03-04T05:55:43.693 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.CkqamYYydo is 6KB 2024-03-04T05:55:43.745 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.CkqamYYydo 2024-03-04T05:55:43.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T05:55:43.828 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T05:55:43.828 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T05:55:43.829 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T05:55:43.829 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T05:55:43.830 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T05:55:43.975 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-03-04T05:56:09.025 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-03-04T05:56:09.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-04T05:56:09.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T05:56:13.983 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-04T05:56:13.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T05:56:13.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T05:56:27.580 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-04T05:56:32.444 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-04T05:56:36.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-04T05:56:36.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T05:56:39.644 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-04T05:56:41.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T05:56:41.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T05:56:44.512 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-04T05:56:51.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-04T05:56:51.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T05:56:54.716 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-04T05:56:56.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T05:56:56.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T05:57:09.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-04T05:57:09.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T05:57:12.796 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-04T05:57:28.301 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-04T05:57:28.796 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T05:57:28.796 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-04T05:57:28.796 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-04T05:57:28.817 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-427.el9.x86_64 2024-03-04T05:57:28.817 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T05:57:28.818 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T05:57:28.818 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T05:57:29.818 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-04T05:57:29.819 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T05:57:29.875 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-03-04T05:57:29.925 DEBUG:teuthology.parallel:result is None 2024-03-04T05:57:30.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-04T05:57:30.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T05:57:49.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-04T05:58:10.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-04T05:58:10.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T05:58:11.334 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-03-04T05:58:11.716 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-04T05:58:11.716 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-427.el9.x86_64"... 2024-03-04T05:58:11.717 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-03-04T05:58:11.732 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-427.el9.x86_64 2024-03-04T05:58:11.732 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T05:58:11.732 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T05:58:11.732 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T05:58:12.733 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-03-04T05:58:12.733 DEBUG:teuthology.orchestra.run.smithi066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T05:58:12.806 INFO:teuthology.orchestra.run.smithi066.stdout:ttyS1 2024-03-04T05:58:12.849 DEBUG:teuthology.parallel:result is None 2024-03-04T05:58:12.850 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T05:58:12.857 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T05:58:12.858 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T05:58:12.860 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T05:58:12.878 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T05:58:12.885 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T05:58:12.894 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T05:58:12.895 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T05:58:12.917 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T05:58:12.953 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T05:58:12.960 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T05:58:12.960 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T05:58:12.982 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T05:58:13.022 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T05:58:13.027 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T05:58:13.036 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T05:58:13.045 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T05:58:13.047 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T05:58:13.054 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T05:58:13.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T05:58:13.080 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T05:58:13.121 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T05:58:13.131 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T05:58:13.131 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T05:58:13.150 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T05:58:13.177 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T05:58:13.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T05:58:13.297 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T05:58:13.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T05:58:13.422 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T05:58:13.422 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T05:58:13.489 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T05:58:13.535 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T05:58:13.589 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T05:58:13.674 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T05:58:13.723 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T05:58:13.723 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T05:58:13.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-03-04T05:58:13.799 DEBUG:teuthology.orchestra.run.smithi066:> sudo service rsyslog restart 2024-03-04T05:58:13.848 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T05:58:13.885 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T05:58:14.255 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T05:58:14.263 INFO:teuthology.task.internal:Starting timer... 2024-03-04T05:58:14.263 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T05:58:14.283 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T05:58:14.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-03-04T05:58:14.338 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-03-04T05:58:14.339 DEBUG:teuthology.orchestra.run.smithi066:> sudo service auditd rotate 2024-03-04T05:58:14.395 INFO:teuthology.orchestra.run.smithi066.stdout:Rotating logs: 2024-03-04T05:58:14.397 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T05:58:14.398 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-03-04T05:58:14.425 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-03-04T05:58:14.426 DEBUG:teuthology.orchestra.run.smithi066:> /usr/sbin/getenforce 2024-03-04T05:58:14.443 INFO:teuthology.orchestra.run.smithi066.stdout:Permissive 2024-03-04T05:58:14.443 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi066.front.sepia.ceph.com': 'permissive'} 2024-03-04T05:58:14.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T05:58:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T05:58:14.494 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T05:58:14.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T05:58:14.527 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T05:58:14.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-03-04T05:58:14.569 DEBUG:teuthology.orchestra.run.smithi066:> sudo /usr/sbin/setenforce permissive 2024-03-04T05:58:14.604 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T05:58:14.615 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-04T05:58:14.621 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-04T05:58:14.725 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T05:58:14.741 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T05:58:14.742 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi066.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T06:02:30.673 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi066.front.sepia.ceph.com')] 2024-03-04T06:02:30.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T06:02:30.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T06:02:30.765 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-04T06:02:30.854 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T06:02:30.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-04T06:02:30.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T06:02:30.955 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-03-04T06:02:31.032 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-04T06:02:31.033 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T06:02:31.043 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T06:02:31.044 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T06:02:31.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T06:02:31.047 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T06:02:31.047 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T06:02:31.076 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T06:02:31.093 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T06:02:31.108 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T06:02:31.122 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-03-04T06:02:31.128 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T06:02:31.135 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-03-04T06:02:31.152 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T06:02:31.164 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: ntpd: command not found 2024-03-04T06:02:31.168 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T06:02:31.181 INFO:teuthology.orchestra.run.smithi066.stdout:506 Cannot talk to daemon 2024-03-04T06:02:31.201 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T06:02:31.219 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-04T06:02:31.220 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T06:02:31.223 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T06:02:31.223 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-04T06:02:31.223 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T06:02:31.223 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T06:02:31.223 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T06:02:31.223 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T06:02:31.276 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-03-04T06:02:31.280 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T06:02:31.280 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-03-04T06:02:31.280 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T06:02:31.281 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T06:02:31.281 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T06:02:31.281 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T06:02:31.281 INFO:teuthology.run_tasks:Running task install... 2024-03-04T06:02:31.289 DEBUG:teuthology.task.install:project ceph 2024-03-04T06:02:31.289 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd'}} 2024-03-04T06:02:31.289 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd'} 2024-03-04T06:02:31.290 INFO:teuthology.task.install:Using flavor: crimson 2024-03-04T06:02:31.295 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-04T06:02:31.295 INFO:teuthology.task.install:extra packages: [] 2024-03-04T06:02:31.295 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd', 'tag': None, 'wait_for_package': False} 2024-03-04T06:02:31.296 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd 2024-03-04T06:02:31.298 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd', 'tag': None, 'wait_for_package': False} 2024-03-04T06:02:31.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd 2024-03-04T06:02:31.440 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-nitzan-crimson-shard-pg-mapping-get-or-create/633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/centos/9/flavors/crimson/ 2024-03-04T06:02:31.440 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1550.g633e69c9 2024-03-04T06:02:31.511 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-nitzan-crimson-shard-pg-mapping-get-or-create/633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/centos/9/flavors/crimson/ 2024-03-04T06:02:31.511 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1550.g633e69c9 2024-03-04T06:02:31.583 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-nitzan-crimson-shard-pg-mapping-get-or-create/633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-nitzan-crimson-shard-pg-mapping-get-or-create/633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-nitzan-crimson-shard-pg-mapping-get-or-create/633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T06:02:31.583 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:02:31.583 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T06:02:31.617 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-04T06:02:31.617 DEBUG:teuthology.orchestra.run.smithi066:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T06:02:31.657 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-nitzan-crimson-shard-pg-mapping-get-or-create/633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-nitzan-crimson-shard-pg-mapping-get-or-create/633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-nitzan-crimson-shard-pg-mapping-get-or-create/633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T06:02:31.658 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:02:31.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T06:02:31.687 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-04T06:02:31.688 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T06:02:31.690 DEBUG:teuthology.orchestra.run.smithi066:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T06:02:31.756 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T06:02:31.778 DEBUG:teuthology.orchestra.run.smithi066:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T06:02:31.808 INFO:teuthology.orchestra.run.smithi066.stdout:check_obsoletes = 1 2024-03-04T06:02:31.810 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum clean all 2024-03-04T06:02:31.837 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T06:02:31.902 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-03-04T06:02:31.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-03-04T06:02:32.060 INFO:teuthology.orchestra.run.smithi066.stdout:74 files removed 2024-03-04T06:02:32.087 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-radosgw 2024-03-04T06:02:32.148 INFO:teuthology.orchestra.run.smithi064.stdout:74 files removed 2024-03-04T06:02:32.175 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-03-04T06:02:32.856 INFO:teuthology.orchestra.run.smithi066.stdout:ceph packages for x86_64 197 kB/s | 85 kB 00:00 2024-03-04T06:02:32.899 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 213 kB/s | 85 kB 00:00 2024-03-04T06:02:33.289 INFO:teuthology.orchestra.run.smithi066.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-04T06:02:33.408 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-04T06:02:33.757 INFO:teuthology.orchestra.run.smithi066.stdout:ceph source packages 4.6 kB/s | 2.0 kB 00:00 2024-03-04T06:02:34.077 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 3.2 kB/s | 2.0 kB 00:00 2024-03-04T06:02:34.304 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-04T06:02:34.519 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-04T06:02:35.759 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-04T06:02:35.961 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-04T06:02:36.794 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-04T06:02:39.785 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-04T06:02:45.153 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-03-04T06:02:50.245 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-03-04T06:02:51.796 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 15 kB 00:00 2024-03-04T06:02:58.983 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 933 kB/s | 19 MB 00:21 2024-03-04T06:03:03.142 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-04T06:03:05.034 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 15 kB 00:00 2024-03-04T06:03:11.682 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 20 MB 00:19 2024-03-04T06:03:16.829 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-03-04T06:03:19.512 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:03:19.517 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:03:19.517 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-04T06:03:19.517 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:03:19.517 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:03:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 9.1 M 2024-03-04T06:03:19.518 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-03-04T06:03:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 3.2 M 2024-03-04T06:03:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 3.1 M 2024-03-04T06:03:19.518 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-04T06:03:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options 2024-03-04T06:03:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T06:03:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 5.4 M 2024-03-04T06:03:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 18 M 2024-03-04T06:03:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 25 k 2024-03-04T06:03:19.518 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T06:03:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T06:03:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T06:03:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 752 k 2024-03-04T06:03:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T06:03:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T06:03:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T06:03:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 496 k 2024-03-04T06:03:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T06:03:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 4.8 M 2024-03-04T06:03:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T06:03:19.519 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T06:03:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T06:03:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse 2024-03-04T06:03:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 45 k 2024-03-04T06:03:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 131 k 2024-03-04T06:03:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 162 k 2024-03-04T06:03:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T06:03:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 321 k 2024-03-04T06:03:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 298 k 2024-03-04T06:03:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 99 k 2024-03-04T06:03:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T06:03:19.520 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T06:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T06:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T06:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-03-04T06:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T06:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout:Install 30 Packages 2024-03-04T06:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-03-04T06:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:03:19.522 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 55 M 2024-03-04T06:03:19.522 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:03:22.813 INFO:teuthology.orchestra.run.smithi064.stdout:(1/32): ceph-base-19.0.0-1550.g633e69c9.el9.x86 1.7 MB/s | 5.4 MB 00:03 2024-03-04T06:03:22.913 INFO:teuthology.orchestra.run.smithi064.stdout:(2/32): ceph-selinux-19.0.0-1550.g633e69c9.el9. 252 kB/s | 25 kB 00:00 2024-03-04T06:03:23.247 INFO:teuthology.orchestra.run.smithi064.stdout:(3/32): libcephfs2-19.0.0-1550.g633e69c9.el9.x8 2.2 MB/s | 752 kB 00:00 2024-03-04T06:03:23.397 INFO:teuthology.orchestra.run.smithi064.stdout:(4/32): libradosstriper1-19.0.0-1550.g633e69c9. 3.2 MB/s | 496 kB 00:00 2024-03-04T06:03:24.214 INFO:teuthology.orchestra.run.smithi064.stdout:(5/32): librgw2-19.0.0-1550.g633e69c9.el9.x86_6 5.9 MB/s | 4.8 MB 00:00 2024-03-04T06:03:24.298 INFO:teuthology.orchestra.run.smithi064.stdout:(6/32): python3-ceph-argparse-19.0.0-1550.g633e 541 kB/s | 45 kB 00:00 2024-03-04T06:03:24.383 INFO:teuthology.orchestra.run.smithi064.stdout:(7/32): python3-ceph-common-19.0.0-1550.g633e69 1.5 MB/s | 131 kB 00:00 2024-03-04T06:03:24.493 INFO:teuthology.orchestra.run.smithi064.stdout:(8/32): python3-cephfs-19.0.0-1550.g633e69c9.el 1.5 MB/s | 162 kB 00:00 2024-03-04T06:03:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:(9/32): python3-rados-19.0.0-1550.g633e69c9.el9 3.2 MB/s | 321 kB 00:00 2024-03-04T06:03:24.693 INFO:teuthology.orchestra.run.smithi064.stdout:(10/32): python3-rbd-19.0.0-1550.g633e69c9.el9. 2.9 MB/s | 298 kB 00:00 2024-03-04T06:03:24.785 INFO:teuthology.orchestra.run.smithi064.stdout:(11/32): python3-rgw-19.0.0-1550.g633e69c9.el9. 1.1 MB/s | 99 kB 00:00 2024-03-04T06:03:24.787 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 20 MB 00:19 2024-03-04T06:03:24.910 INFO:teuthology.orchestra.run.smithi064.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 848 kB/s | 106 kB 00:00 2024-03-04T06:03:24.960 INFO:teuthology.orchestra.run.smithi064.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-04T06:03:25.011 INFO:teuthology.orchestra.run.smithi064.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-04T06:03:25.111 INFO:teuthology.orchestra.run.smithi064.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-04T06:03:25.153 INFO:teuthology.orchestra.run.smithi064.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-04T06:03:25.178 INFO:teuthology.orchestra.run.smithi064.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-04T06:03:25.220 INFO:teuthology.orchestra.run.smithi064.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 5.1 MB/s | 215 kB 00:00 2024-03-04T06:03:25.253 INFO:teuthology.orchestra.run.smithi064.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 1.0 MB/s | 34 kB 00:00 2024-03-04T06:03:25.287 INFO:teuthology.orchestra.run.smithi064.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-04T06:03:25.737 INFO:teuthology.orchestra.run.smithi064.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 687 kB/s | 309 kB 00:00 2024-03-04T06:03:26.063 INFO:teuthology.orchestra.run.smithi064.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-04T06:03:26.129 INFO:teuthology.orchestra.run.smithi064.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 390 kB/s | 26 kB 00:00 2024-03-04T06:03:26.205 INFO:teuthology.orchestra.run.smithi064.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-04T06:03:26.272 INFO:teuthology.orchestra.run.smithi064.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-04T06:03:26.355 INFO:teuthology.orchestra.run.smithi064.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-04T06:03:26.431 INFO:teuthology.orchestra.run.smithi064.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-04T06:03:26.548 INFO:teuthology.orchestra.run.smithi064.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-04T06:03:26.848 INFO:teuthology.orchestra.run.smithi064.stdout:(29/32): ceph-common-19.0.0-1550.g633e69c9.el9. 2.5 MB/s | 18 MB 00:07 2024-03-04T06:03:27.840 INFO:teuthology.orchestra.run.smithi064.stdout:(30/32): librados2-19.0.0-1550.g633e69c9.el9.x8 2.4 MB/s | 3.2 MB 00:01 2024-03-04T06:03:27.890 INFO:teuthology.orchestra.run.smithi064.stdout:(31/32): librbd1-19.0.0-1550.g633e69c9.el9.x86_ 3.0 MB/s | 3.1 MB 00:01 2024-03-04T06:03:28.465 INFO:teuthology.orchestra.run.smithi064.stdout:(32/32): ceph-radosgw-19.0.0-1550.g633e69c9.el9 1.0 MB/s | 9.1 MB 00:08 2024-03-04T06:03:28.470 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:03:28.470 INFO:teuthology.orchestra.run.smithi064.stdout:Total 6.1 MB/s | 55 MB 00:08 2024-03-04T06:03:28.471 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:03:28.513 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:03:28.513 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:03:28.996 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:03:28.997 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:03:29.893 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-04T06:03:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:03:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-04T06:03:31.092 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.0.0-1550.g633e69c9.el9.x86_64 2/34 2024-03-04T06:03:31.185 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.0.0-1550.g633e69c9.el9.x86_64 2/34 2024-03-04T06:03:31.281 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.0.0-1550.g633e69c9.el9.x86_64 3/34 2024-03-04T06:03:31.311 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.0.0-1550.g633e69c9.el9.x86_64 4/34 2024-03-04T06:03:31.373 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.0.0-1550.g633e69c9.el9.x86_64 4/34 2024-03-04T06:03:31.406 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-03-04T06:03:31.431 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-03-04T06:03:31.463 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 7/34 2024-03-04T06:03:31.478 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 7/34 2024-03-04T06:03:31.680 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 8/34 2024-03-04T06:03:31.710 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 8/34 2024-03-04T06:03:31.739 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-03-04T06:03:31.783 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.0.0-1550.g633e69c9.el9. 10/34 2024-03-04T06:03:31.819 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.0.0-1550.g633e69c9.el9.x86_64 11/34 2024-03-04T06:03:31.889 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 12/34 2024-03-04T06:03:31.917 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.0.0-1550.g633e69c9.el9.x86_6 13/34 2024-03-04T06:03:32.007 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1550.g633e69c9.el9.x86_6 13/34 2024-03-04T06:03:32.041 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-03-04T06:03:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-04T06:03:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-04T06:03:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-04T06:03:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-04T06:03:32.303 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-04T06:03:32.335 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-04T06:03:32.593 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:03:32.601 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:03:32.601 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-04T06:03:32.601 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:03:32.601 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:03:32.601 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 9.1 M 2024-03-04T06:03:32.601 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrading: 2024-03-04T06:03:32.601 INFO:teuthology.orchestra.run.smithi066.stdout: librados2 x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 3.2 M 2024-03-04T06:03:32.602 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1 x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 3.1 M 2024-03-04T06:03:32.602 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-04T06:03:32.602 INFO:teuthology.orchestra.run.smithi066.stdout: boost-program-options 2024-03-04T06:03:32.602 INFO:teuthology.orchestra.run.smithi066.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T06:03:32.602 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 5.4 M 2024-03-04T06:03:32.602 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 18 M 2024-03-04T06:03:32.602 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 25 k 2024-03-04T06:03:32.602 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T06:03:32.602 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T06:03:32.603 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T06:03:32.603 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2 x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 752 k 2024-03-04T06:03:32.603 INFO:teuthology.orchestra.run.smithi066.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T06:03:32.603 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T06:03:32.603 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T06:03:32.603 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1 x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 496 k 2024-03-04T06:03:32.603 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T06:03:32.603 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2 x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 4.8 M 2024-03-04T06:03:32.603 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T06:03:32.603 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T06:03:32.604 INFO:teuthology.orchestra.run.smithi066.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T06:03:32.604 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse 2024-03-04T06:03:32.604 INFO:teuthology.orchestra.run.smithi066.stdout: x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 45 k 2024-03-04T06:03:32.604 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 131 k 2024-03-04T06:03:32.604 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 162 k 2024-03-04T06:03:32.604 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T06:03:32.604 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 321 k 2024-03-04T06:03:32.604 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 298 k 2024-03-04T06:03:32.604 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 99 k 2024-03-04T06:03:32.604 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T06:03:32.604 INFO:teuthology.orchestra.run.smithi066.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T06:03:32.605 INFO:teuthology.orchestra.run.smithi066.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T06:03:32.605 INFO:teuthology.orchestra.run.smithi066.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T06:03:32.605 INFO:teuthology.orchestra.run.smithi066.stdout:Installing weak dependencies: 2024-03-04T06:03:32.605 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T06:03:32.605 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:03:32.605 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:03:32.605 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:03:32.605 INFO:teuthology.orchestra.run.smithi066.stdout:Install 30 Packages 2024-03-04T06:03:32.605 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrade 2 Packages 2024-03-04T06:03:32.605 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:03:32.606 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 55 M 2024-03-04T06:03:32.606 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:03:32.673 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-04T06:03:32.698 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.0.0-1550.g633e69c9.el9.x86_64 22/34 2024-03-04T06:03:32.788 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.0.0-1550.g633e69c9.el9.x86_64 22/34 2024-03-04T06:03:32.819 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.0.0-1550.g633e69c9.el9.x86_64 23/34 2024-03-04T06:03:32.858 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-04T06:03:33.340 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-04T06:03:33.367 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.0.0-1550.g633e69c9.el9.x86_64 26/34 2024-03-04T06:03:33.438 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.0.0-1550.g633e69c9.el9.x86_64 26/34 2024-03-04T06:03:33.509 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.0.0-1550.g633e69c9.el9.x86_64 27/34 2024-03-04T06:03:33.544 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.0.0-1550.g633e69c9.el9.x8 28/34 2024-03-04T06:03:35.369 INFO:teuthology.orchestra.run.smithi066.stdout:(1/32): ceph-base-19.0.0-1550.g633e69c9.el9.x86 2.0 MB/s | 5.4 MB 00:02 2024-03-04T06:03:35.477 INFO:teuthology.orchestra.run.smithi066.stdout:(2/32): ceph-selinux-19.0.0-1550.g633e69c9.el9. 233 kB/s | 25 kB 00:00 2024-03-04T06:03:35.775 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 29/34 2024-03-04T06:03:35.807 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 29/34 2024-03-04T06:03:35.861 INFO:teuthology.orchestra.run.smithi066.stdout:(3/32): libcephfs2-19.0.0-1550.g633e69c9.el9.x8 1.9 MB/s | 752 kB 00:00 2024-03-04T06:03:36.136 INFO:teuthology.orchestra.run.smithi066.stdout:(4/32): libradosstriper1-19.0.0-1550.g633e69c9. 1.8 MB/s | 496 kB 00:00 2024-03-04T06:03:36.398 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 29/34 2024-03-04T06:03:36.446 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.0.0-1550.g633e69c9.el9.x86_64 30/34 2024-03-04T06:03:36.525 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.0.0-1550.g633e69c9.el9.x86_64 30/34 2024-03-04T06:03:36.525 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T06:03:36.525 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T06:03:36.525 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:03:36.553 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.0.0-1550.g633e69c9.el9.x86_64 31/34 2024-03-04T06:03:36.911 INFO:teuthology.orchestra.run.smithi066.stdout:(5/32): ceph-radosgw-19.0.0-1550.g633e69c9.el9. 2.1 MB/s | 9.1 MB 00:04 2024-03-04T06:03:37.011 INFO:teuthology.orchestra.run.smithi066.stdout:(6/32): python3-ceph-argparse-19.0.0-1550.g633e 452 kB/s | 45 kB 00:00 2024-03-04T06:03:37.620 INFO:teuthology.orchestra.run.smithi066.stdout:(7/32): python3-ceph-common-19.0.0-1550.g633e69 216 kB/s | 131 kB 00:00 2024-03-04T06:03:38.012 INFO:teuthology.orchestra.run.smithi066.stdout:(8/32): python3-cephfs-19.0.0-1550.g633e69c9.el 414 kB/s | 162 kB 00:00 2024-03-04T06:03:38.229 INFO:teuthology.orchestra.run.smithi066.stdout:(9/32): librgw2-19.0.0-1550.g633e69c9.el9.x86_6 2.3 MB/s | 4.8 MB 00:02 2024-03-04T06:03:38.446 INFO:teuthology.orchestra.run.smithi066.stdout:(10/32): python3-rbd-19.0.0-1550.g633e69c9.el9. 1.3 MB/s | 298 kB 00:00 2024-03-04T06:03:38.529 INFO:teuthology.orchestra.run.smithi066.stdout:(11/32): python3-rados-19.0.0-1550.g633e69c9.el 621 kB/s | 321 kB 00:00 2024-03-04T06:03:38.555 INFO:teuthology.orchestra.run.smithi066.stdout:(12/32): python3-rgw-19.0.0-1550.g633e69c9.el9. 913 kB/s | 99 kB 00:00 2024-03-04T06:03:38.722 INFO:teuthology.orchestra.run.smithi066.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 552 kB/s | 106 kB 00:00 2024-03-04T06:03:38.755 INFO:teuthology.orchestra.run.smithi066.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 796 kB/s | 159 kB 00:00 2024-03-04T06:03:38.780 INFO:teuthology.orchestra.run.smithi066.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 756 kB/s | 44 kB 00:00 2024-03-04T06:03:38.856 INFO:teuthology.orchestra.run.smithi066.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-04T06:03:38.881 INFO:teuthology.orchestra.run.smithi066.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-03-04T06:03:38.898 INFO:teuthology.orchestra.run.smithi066.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 992 kB/s | 41 kB 00:00 2024-03-04T06:03:38.923 INFO:teuthology.orchestra.run.smithi066.stdout:(19/32): qatlib-23.11.0-1.el9.x86_64.rpm 5.1 MB/s | 215 kB 00:00 2024-03-04T06:03:38.940 INFO:teuthology.orchestra.run.smithi066.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 817 kB/s | 34 kB 00:00 2024-03-04T06:03:38.957 INFO:teuthology.orchestra.run.smithi066.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-04T06:03:39.349 INFO:teuthology.orchestra.run.smithi066.stdout:(22/32): ceph-common-19.0.0-1550.g633e69c9.el9. 2.7 MB/s | 18 MB 00:06 2024-03-04T06:03:39.599 INFO:teuthology.orchestra.run.smithi066.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 104 kB/s | 26 kB 00:00 2024-03-04T06:03:39.716 INFO:teuthology.orchestra.run.smithi066.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 398 kB/s | 309 kB 00:00 2024-03-04T06:03:39.792 INFO:teuthology.orchestra.run.smithi066.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-03-04T06:03:39.883 INFO:teuthology.orchestra.run.smithi066.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 404 kB/s | 67 kB 00:00 2024-03-04T06:03:40.150 INFO:teuthology.orchestra.run.smithi066.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 719 kB/s | 192 kB 00:00 2024-03-04T06:03:40.175 INFO:teuthology.orchestra.run.smithi066.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.1 MB/s | 838 kB 00:00 2024-03-04T06:03:40.934 INFO:teuthology.orchestra.run.smithi066.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-03-04T06:03:41.401 INFO:teuthology.orchestra.run.smithi066.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 4.4 MB 00:02 2024-03-04T06:03:41.735 INFO:teuthology.orchestra.run.smithi066.stdout:(31/32): librados2-19.0.0-1550.g633e69c9.el9.x8 2.0 MB/s | 3.2 MB 00:01 2024-03-04T06:03:42.710 INFO:teuthology.orchestra.run.smithi066.stdout:(32/32): librbd1-19.0.0-1550.g633e69c9.el9.x86_ 1.7 MB/s | 3.1 MB 00:01 2024-03-04T06:03:42.714 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:03:42.715 INFO:teuthology.orchestra.run.smithi066.stdout:Total 5.4 MB/s | 55 MB 00:10 2024-03-04T06:03:42.715 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:03:42.758 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:03:42.758 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:03:43.238 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:03:43.238 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:03:44.818 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:03:45.231 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-04T06:03:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librados2-2:19.0.0-1550.g633e69c9.el9.x86_64 2/34 2024-03-04T06:03:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:19.0.0-1550.g633e69c9.el9.x86_64 2/34 2024-03-04T06:03:45.514 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rados-2:19.0.0-1550.g633e69c9.el9.x86_64 3/34 2024-03-04T06:03:45.538 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs2-2:19.0.0-1550.g633e69c9.el9.x86_64 4/34 2024-03-04T06:03:45.601 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs2-2:19.0.0-1550.g633e69c9.el9.x86_64 4/34 2024-03-04T06:03:45.626 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-03-04T06:03:45.651 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-03-04T06:03:45.674 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 7/34 2024-03-04T06:03:45.690 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 7/34 2024-03-04T06:03:45.850 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 8/34 2024-03-04T06:03:45.887 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 8/34 2024-03-04T06:03:45.916 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-03-04T06:03:45.961 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-argparse-2:19.0.0-1550.g633e69c9.el9. 10/34 2024-03-04T06:03:45.988 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cephfs-2:19.0.0-1550.g633e69c9.el9.x86_64 11/34 2024-03-04T06:03:46.057 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 12/34 2024-03-04T06:03:46.086 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libradosstriper1-2:19.0.0-1550.g633e69c9.el9.x86_6 13/34 2024-03-04T06:03:46.175 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1550.g633e69c9.el9.x86_6 13/34 2024-03-04T06:03:46.210 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-03-04T06:03:46.329 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-04T06:03:46.362 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-04T06:03:46.402 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-04T06:03:46.434 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-04T06:03:46.463 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-04T06:03:46.488 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-04T06:03:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1550.g633e69c9.el9.x86_64 31/34 2024-03-04T06:03:46.672 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-03-04T06:03:46.672 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-03-04T06:03:46.672 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-03-04T06:03:46.672 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-03-04T06:03:46.672 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-03-04T06:03:46.672 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-03-04T06:03:46.672 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-03-04T06:03:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:03:46.826 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-04T06:03:46.859 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librbd1-2:19.0.0-1550.g633e69c9.el9.x86_64 22/34 2024-03-04T06:03:46.949 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:19.0.0-1550.g633e69c9.el9.x86_64 22/34 2024-03-04T06:03:46.981 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rbd-2:19.0.0-1550.g633e69c9.el9.x86_64 23/34 2024-03-04T06:03:47.021 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-04T06:03:47.500 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-04T06:03:47.528 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librgw2-2:19.0.0-1550.g633e69c9.el9.x86_64 26/34 2024-03-04T06:03:47.599 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librgw2-2:19.0.0-1550.g633e69c9.el9.x86_64 26/34 2024-03-04T06:03:47.621 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.0.0-1550.g633e69c9.el9.x86_64 32/34 2024-03-04T06:03:47.645 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1550.g633e69c9.el9.x86_64 32/34 2024-03-04T06:03:47.645 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:03:47.645 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T06:03:47.645 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T06:03:47.645 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T06:03:47.645 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:03:47.647 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T06:03:47.671 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rgw-2:19.0.0-1550.g633e69c9.el9.x86_64 27/34 2024-03-04T06:03:47.699 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-common-2:19.0.0-1550.g633e69c9.el9.x8 28/34 2024-03-04T06:03:47.732 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T06:03:47.732 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T06:03:48.891 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T06:03:48.891 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.0.0-1550.g633e69c9.el9.x86_64 1/34 2024-03-04T06:03:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 2/34 2024-03-04T06:03:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.0.0-1550.g633e69c9.el9.x86_64 3/34 2024-03-04T06:03:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.0.0-1550.g633e69c9.el9.x86_64 4/34 2024-03-04T06:03:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.0.0-1550.g633e69c9.el9.x86_64 5/34 2024-03-04T06:03:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.0.0-1550.g633e69c9.el9.x86_6 6/34 2024-03-04T06:03:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.0.0-1550.g633e69c9.el9.x86_64 7/34 2024-03-04T06:03:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1550.g633e69c9.el9. 8/34 2024-03-04T06:03:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.0.0-1550.g633e69c9.el9.x8 9/34 2024-03-04T06:03:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.0.0-1550.g633e69c9.el9.x86_64 10/34 2024-03-04T06:03:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.0.0-1550.g633e69c9.el9.x86_64 11/34 2024-03-04T06:03:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.0.0-1550.g633e69c9.el9.x86_64 12/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.0.0-1550.g633e69c9.el9.x86_64 13/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-04T06:03:48.893 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-04T06:03:48.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-04T06:03:48.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-04T06:03:48.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-04T06:03:48.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-04T06:03:48.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-04T06:03:48.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.0.0-1550.g633e69c9.el9.x86_64 31/34 2024-03-04T06:03:48.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-04T06:03:48.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.0.0-1550.g633e69c9.el9.x86_64 33/34 2024-03-04T06:03:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T06:03:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:03:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T06:03:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T06:03:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T06:03:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T06:03:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T06:03:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:03:49.829 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:03:49.882 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 29/34 2024-03-04T06:03:49.920 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 29/34 2024-03-04T06:03:50.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-03-04T06:03:50.514 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 29/34 2024-03-04T06:03:50.549 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-base-2:19.0.0-1550.g633e69c9.el9.x86_64 30/34 2024-03-04T06:03:50.631 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-base-2:19.0.0-1550.g633e69c9.el9.x86_64 30/34 2024-03-04T06:03:50.631 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T06:03:50.631 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T06:03:50.631 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:03:50.658 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-selinux-2:19.0.0-1550.g633e69c9.el9.x86_64 31/34 2024-03-04T06:03:50.688 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:34 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:03:50.865 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:03:50.865 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:03:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-04T06:03:50.866 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:03:50.866 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:03:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 122 M 2024-03-04T06:03:50.866 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-04T06:03:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T06:03:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: libasan x86_64 11.4.1-3.el9 CentOS-AppStream 417 k 2024-03-04T06:03:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 158 k 2024-03-04T06:03:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: libubsan x86_64 11.4.1-3.el9 CentOS-AppStream 186 k 2024-03-04T06:03:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T06:03:50.866 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T06:03:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T06:03:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-03-04T06:03:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:03:50.867 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:03:50.867 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:03:50.867 INFO:teuthology.orchestra.run.smithi064.stdout:Install 9 Packages 2024-03-04T06:03:50.867 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:03:50.867 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 123 M 2024-03-04T06:03:50.868 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.1 G 2024-03-04T06:03:50.868 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:03:51.287 INFO:teuthology.orchestra.run.smithi064.stdout:(1/9): libasan-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 417 kB 00:00 2024-03-04T06:03:51.371 INFO:teuthology.orchestra.run.smithi064.stdout:(2/9): libubsan-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 186 kB 00:00 2024-03-04T06:03:51.471 INFO:teuthology.orchestra.run.smithi064.stdout:(3/9): socat-1.7.4.1-5.el9.x86_64.rpm 2.9 MB/s | 300 kB 00:00 2024-03-04T06:03:51.529 INFO:teuthology.orchestra.run.smithi064.stdout:(4/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-04T06:03:51.596 INFO:teuthology.orchestra.run.smithi064.stdout:(5/9): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 186 kB 00:00 2024-03-04T06:03:51.672 INFO:teuthology.orchestra.run.smithi064.stdout:(6/9): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-04T06:03:51.855 INFO:teuthology.orchestra.run.smithi064.stdout:(7/9): libcephsqlite-19.0.0-1550.g633e69c9.el9. 169 kB/s | 158 kB 00:00 2024-03-04T06:03:52.105 INFO:teuthology.orchestra.run.smithi064.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 281 kB/s | 122 kB 00:00 2024-03-04T06:04:00.575 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1550.g633e69c9.el9.x86_64 31/34 2024-03-04T06:04:00.575 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /sys 2024-03-04T06:04:00.575 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /proc 2024-03-04T06:04:00.575 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /mnt 2024-03-04T06:04:00.575 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /var/tmp 2024-03-04T06:04:00.576 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /home 2024-03-04T06:04:00.576 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /root 2024-03-04T06:04:00.576 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /tmp 2024-03-04T06:04:00.576 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:04:01.508 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-radosgw-2:19.0.0-1550.g633e69c9.el9.x86_64 32/34 2024-03-04T06:04:01.540 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1550.g633e69c9.el9.x86_64 32/34 2024-03-04T06:04:01.540 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:04:01.540 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T06:04:01.540 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T06:04:01.540 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T06:04:01.541 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:04:01.543 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T06:04:01.611 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T06:04:01.611 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T06:04:02.775 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T06:04:02.775 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-base-2:19.0.0-1550.g633e69c9.el9.x86_64 1/34 2024-03-04T06:04:02.775 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 2/34 2024-03-04T06:04:02.775 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-radosgw-2:19.0.0-1550.g633e69c9.el9.x86_64 3/34 2024-03-04T06:04:02.775 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-selinux-2:19.0.0-1550.g633e69c9.el9.x86_64 4/34 2024-03-04T06:04:02.775 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs2-2:19.0.0-1550.g633e69c9.el9.x86_64 5/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libradosstriper1-2:19.0.0-1550.g633e69c9.el9.x86_6 6/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librgw2-2:19.0.0-1550.g633e69c9.el9.x86_64 7/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1550.g633e69c9.el9. 8/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-common-2:19.0.0-1550.g633e69c9.el9.x8 9/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cephfs-2:19.0.0-1550.g633e69c9.el9.x86_64 10/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rados-2:19.0.0-1550.g633e69c9.el9.x86_64 11/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rbd-2:19.0.0-1550.g633e69c9.el9.x86_64 12/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rgw-2:19.0.0-1550.g633e69c9.el9.x86_64 13/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-04T06:04:02.776 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-04T06:04:02.777 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-04T06:04:02.777 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-04T06:04:02.777 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-04T06:04:02.777 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-04T06:04:02.777 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-04T06:04:02.777 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-04T06:04:02.777 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-04T06:04:02.777 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-04T06:04:02.777 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-04T06:04:02.777 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-04T06:04:02.778 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:19.0.0-1550.g633e69c9.el9.x86_64 31/34 2024-03-04T06:04:02.778 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-04T06:04:02.778 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:19.0.0-1550.g633e69c9.el9.x86_64 33/34 2024-03-04T06:04:03.911 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T06:04:03.912 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:04:03.912 INFO:teuthology.orchestra.run.smithi066.stdout:Upgraded: 2024-03-04T06:04:03.912 INFO:teuthology.orchestra.run.smithi066.stdout: librados2-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.912 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.912 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:04:03.912 INFO:teuthology.orchestra.run.smithi066.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T06:04:03.912 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.912 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.912 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.912 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.913 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:04:03.914 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:04:04.276 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-test 2024-03-04T06:04:04.865 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:35 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:04:05.040 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:04:05.041 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:04:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-04T06:04:05.042 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:04:05.042 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:04:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 122 M 2024-03-04T06:04:05.042 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-04T06:04:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T06:04:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: libasan x86_64 11.4.1-3.el9 CentOS-AppStream 417 k 2024-03-04T06:04:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 158 k 2024-03-04T06:04:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: libubsan x86_64 11.4.1-3.el9 CentOS-AppStream 186 k 2024-03-04T06:04:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T06:04:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T06:04:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T06:04:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-03-04T06:04:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:04:05.043 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:04:05.043 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:04:05.043 INFO:teuthology.orchestra.run.smithi066.stdout:Install 9 Packages 2024-03-04T06:04:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:04:05.044 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 123 M 2024-03-04T06:04:05.044 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 1.1 G 2024-03-04T06:04:05.044 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:04:05.406 INFO:teuthology.orchestra.run.smithi066.stdout:(1/9): libasan-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 417 kB 00:00 2024-03-04T06:04:05.448 INFO:teuthology.orchestra.run.smithi066.stdout:(2/9): libubsan-11.4.1-3.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-03-04T06:04:05.490 INFO:teuthology.orchestra.run.smithi066.stdout:(3/9): socat-1.7.4.1-5.el9.x86_64.rpm 7.2 MB/s | 300 kB 00:00 2024-03-04T06:04:05.532 INFO:teuthology.orchestra.run.smithi066.stdout:(4/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-04T06:04:05.574 INFO:teuthology.orchestra.run.smithi066.stdout:(5/9): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-03-04T06:04:05.616 INFO:teuthology.orchestra.run.smithi066.stdout:(6/9): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-04T06:04:05.758 INFO:teuthology.orchestra.run.smithi066.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 859 kB/s | 122 kB 00:00 2024-03-04T06:04:06.050 INFO:teuthology.orchestra.run.smithi066.stdout:(8/9): libcephsqlite-19.0.0-1550.g633e69c9.el9. 196 kB/s | 158 kB 00:00 2024-03-04T06:04:28.369 INFO:teuthology.orchestra.run.smithi064.stdout:(9/9): ceph-test-19.0.0-1550.g633e69c9.el9.x86_ 3.2 MB/s | 122 MB 00:37 2024-03-04T06:04:28.373 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:04:28.373 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.3 MB/s | 123 MB 00:37 2024-03-04T06:04:28.373 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:04:28.393 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:04:28.393 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:04:29.253 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:04:29.254 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:04:30.040 INFO:teuthology.orchestra.run.smithi066.stdout:(9/9): ceph-test-19.0.0-1550.g633e69c9.el9.x86_ 4.9 MB/s | 122 MB 00:24 2024-03-04T06:04:30.043 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:04:30.043 INFO:teuthology.orchestra.run.smithi066.stdout:Total 4.9 MB/s | 123 MB 00:24 2024-03-04T06:04:30.043 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:04:30.066 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:04:30.066 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:04:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:04:30.690 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-03-04T06:04:30.738 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 2/9 2024-03-04T06:04:30.769 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : jq-1.6-16.el9.x86_64 3/9 2024-03-04T06:04:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 4/9 2024-03-04T06:04:30.851 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 5/9 2024-03-04T06:04:30.872 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 6/9 2024-03-04T06:04:30.913 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libasan-11.4.1-3.el9.x86_64 7/9 2024-03-04T06:04:30.935 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:04:30.935 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:04:30.941 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.0.0-1550.g633e69c9.el9.x86_64 8/9 2024-03-04T06:04:32.158 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:04:32.230 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-03-04T06:04:32.278 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 2/9 2024-03-04T06:04:32.323 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : jq-1.6-16.el9.x86_64 3/9 2024-03-04T06:04:32.368 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 4/9 2024-03-04T06:04:32.405 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 5/9 2024-03-04T06:04:32.434 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 6/9 2024-03-04T06:04:32.475 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libasan-11.4.1-3.el9.x86_64 7/9 2024-03-04T06:04:32.502 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephsqlite-2:19.0.0-1550.g633e69c9.el9.x86_64 8/9 2024-03-04T06:04:48.707 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1550.g633e69c9.el9.x86_64 8/9 2024-03-04T06:04:48.783 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.0.0-1550.g633e69c9.el9.x86_64 9/9 2024-03-04T06:04:50.264 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1550.g633e69c9.el9.x86_64 8/9 2024-03-04T06:04:50.345 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-test-2:19.0.0-1550.g633e69c9.el9.x86_64 9/9 2024-03-04T06:04:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:19.0.0-1550.g633e69c9.el9.x86_64 9/9 2024-03-04T06:04:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.0.0-1550.g633e69c9.el9.x86_64 1/9 2024-03-04T06:04:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.0.0-1550.g633e69c9.el9.x86_64 2/9 2024-03-04T06:04:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libasan-11.4.1-3.el9.x86_64 3/9 2024-03-04T06:04:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 4/9 2024-03-04T06:04:56.767 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/9 2024-03-04T06:04:56.768 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/9 2024-03-04T06:04:56.768 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : jq-1.6-16.el9.x86_64 7/9 2024-03-04T06:04:56.768 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 8/9 2024-03-04T06:04:57.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-03-04T06:04:57.195 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:04:57.196 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:04:57.196 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:57.196 INFO:teuthology.orchestra.run.smithi064.stdout: jq-1.6-16.el9.x86_64 2024-03-04T06:04:57.196 INFO:teuthology.orchestra.run.smithi064.stdout: libasan-11.4.1-3.el9.x86_64 2024-03-04T06:04:57.196 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:57.196 INFO:teuthology.orchestra.run.smithi064.stdout: libubsan-11.4.1-3.el9.x86_64 2024-03-04T06:04:57.196 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T06:04:57.196 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T06:04:57.196 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T06:04:57.196 INFO:teuthology.orchestra.run.smithi064.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-03-04T06:04:57.197 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:04:57.197 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:04:57.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-03-04T06:04:58.035 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:42 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:04:58.208 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:04:58.211 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-04T06:04:58.211 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-04T06:04:58.211 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-04T06:04:58.211 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:04:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 6.6 k 2024-03-04T06:04:58.212 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-04T06:04:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 2.0 M 2024-03-04T06:04:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 1.5 M 2024-03-04T06:04:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 247 k 2024-03-04T06:04:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 4.9 M 2024-03-04T06:04:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 118 M 2024-03-04T06:04:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T06:04:58.212 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T06:04:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T06:04:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T06:04:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T06:04:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T06:04:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T06:04:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T06:04:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T06:04:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T06:04:58.213 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T06:04:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T06:04:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T06:04:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T06:04:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T06:04:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T06:04:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T06:04:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T06:04:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T06:04:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T06:04:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T06:04:58.214 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T06:04:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T06:04:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T06:04:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T06:04:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T06:04:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T06:04:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T06:04:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T06:04:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T06:04:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T06:04:58.215 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:04:58.216 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:04:58.216 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-04T06:04:58.216 INFO:teuthology.orchestra.run.smithi064.stdout:Install 37 Packages 2024-03-04T06:04:58.216 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:04:58.216 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 132 M 2024-03-04T06:04:58.217 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.4 G 2024-03-04T06:04:58.217 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:04:58.447 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-test-2:19.0.0-1550.g633e69c9.el9.x86_64 9/9 2024-03-04T06:04:58.447 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-test-2:19.0.0-1550.g633e69c9.el9.x86_64 1/9 2024-03-04T06:04:58.447 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephsqlite-2:19.0.0-1550.g633e69c9.el9.x86_64 2/9 2024-03-04T06:04:58.447 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libasan-11.4.1-3.el9.x86_64 3/9 2024-03-04T06:04:58.447 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 4/9 2024-03-04T06:04:58.447 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/9 2024-03-04T06:04:58.447 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/9 2024-03-04T06:04:58.448 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : jq-1.6-16.el9.x86_64 7/9 2024-03-04T06:04:58.448 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 8/9 2024-03-04T06:04:58.714 INFO:teuthology.orchestra.run.smithi064.stdout:(1/37): ceph-19.0.0-1550.g633e69c9.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-04T06:04:58.834 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-03-04T06:04:58.834 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:04:58.834 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:04:58.834 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:58.834 INFO:teuthology.orchestra.run.smithi066.stdout: jq-1.6-16.el9.x86_64 2024-03-04T06:04:58.834 INFO:teuthology.orchestra.run.smithi066.stdout: libasan-11.4.1-3.el9.x86_64 2024-03-04T06:04:58.835 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:04:58.835 INFO:teuthology.orchestra.run.smithi066.stdout: libubsan-11.4.1-3.el9.x86_64 2024-03-04T06:04:58.835 INFO:teuthology.orchestra.run.smithi066.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T06:04:58.835 INFO:teuthology.orchestra.run.smithi066.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T06:04:58.835 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T06:04:58.835 INFO:teuthology.orchestra.run.smithi066.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-03-04T06:04:58.835 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:04:58.835 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:04:59.128 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph 2024-03-04T06:04:59.718 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:30 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:04:59.895 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:04:59.897 INFO:teuthology.orchestra.run.smithi066.stdout:========================================================================================= 2024-03-04T06:04:59.897 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-04T06:04:59.897 INFO:teuthology.orchestra.run.smithi066.stdout:========================================================================================= 2024-03-04T06:04:59.898 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:04:59.898 INFO:teuthology.orchestra.run.smithi066.stdout: ceph x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 6.6 k 2024-03-04T06:04:59.898 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-04T06:04:59.898 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 2.0 M 2024-03-04T06:04:59.898 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 1.5 M 2024-03-04T06:04:59.898 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 247 k 2024-03-04T06:04:59.898 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 4.9 M 2024-03-04T06:04:59.898 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 118 M 2024-03-04T06:04:59.898 INFO:teuthology.orchestra.run.smithi066.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T06:04:59.898 INFO:teuthology.orchestra.run.smithi066.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T06:04:59.899 INFO:teuthology.orchestra.run.smithi066.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T06:04:59.899 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T06:04:59.899 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T06:04:59.899 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T06:04:59.899 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T06:04:59.899 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T06:04:59.899 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T06:04:59.899 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T06:04:59.899 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T06:04:59.899 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T06:04:59.899 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T06:04:59.900 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T06:04:59.900 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T06:04:59.900 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T06:04:59.900 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T06:04:59.900 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T06:04:59.900 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T06:04:59.900 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T06:04:59.900 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T06:04:59.900 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T06:04:59.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T06:04:59.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T06:04:59.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T06:04:59.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T06:04:59.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T06:04:59.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T06:04:59.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T06:04:59.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T06:04:59.901 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T06:04:59.901 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:04:59.901 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:04:59.902 INFO:teuthology.orchestra.run.smithi066.stdout:========================================================================================= 2024-03-04T06:04:59.902 INFO:teuthology.orchestra.run.smithi066.stdout:Install 37 Packages 2024-03-04T06:04:59.902 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:04:59.903 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 132 M 2024-03-04T06:04:59.903 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 1.4 G 2024-03-04T06:04:59.903 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:05:00.197 INFO:teuthology.orchestra.run.smithi064.stdout:(2/37): ceph-mds-19.0.0-1550.g633e69c9.el9.x86_ 1.1 MB/s | 2.0 MB 00:01 2024-03-04T06:05:00.397 INFO:teuthology.orchestra.run.smithi064.stdout:(3/37): ceph-mgr-19.0.0-1550.g633e69c9.el9.x86_ 724 kB/s | 1.5 MB 00:02 2024-03-04T06:05:00.434 INFO:teuthology.orchestra.run.smithi066.stdout:(1/37): ceph-19.0.0-1550.g633e69c9.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-04T06:05:00.931 INFO:teuthology.orchestra.run.smithi064.stdout:(4/37): ceph-mgr-modules-core-19.0.0-1550.g633e 464 kB/s | 247 kB 00:00 2024-03-04T06:05:01.048 INFO:teuthology.orchestra.run.smithi064.stdout:(5/37): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-04T06:05:01.115 INFO:teuthology.orchestra.run.smithi064.stdout:(6/37): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-04T06:05:01.149 INFO:teuthology.orchestra.run.smithi064.stdout:(7/37): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-04T06:05:01.182 INFO:teuthology.orchestra.run.smithi064.stdout:(8/37): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-03-04T06:05:01.216 INFO:teuthology.orchestra.run.smithi064.stdout:(9/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-04T06:05:01.249 INFO:teuthology.orchestra.run.smithi064.stdout:(10/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-04T06:05:01.291 INFO:teuthology.orchestra.run.smithi064.stdout:(11/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-04T06:05:01.350 INFO:teuthology.orchestra.run.smithi064.stdout:(12/37): python3-cryptography-36.0.1-4.el9.x86_ 20 MB/s | 1.1 MB 00:00 2024-03-04T06:05:01.384 INFO:teuthology.orchestra.run.smithi064.stdout:(13/37): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 103 kB 00:00 2024-03-04T06:05:01.417 INFO:teuthology.orchestra.run.smithi064.stdout:(14/37): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-04T06:05:01.451 INFO:teuthology.orchestra.run.smithi064.stdout:(15/37): python3-requests-2.25.1-8.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2024-03-04T06:05:01.485 INFO:teuthology.orchestra.run.smithi064.stdout:(16/37): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-04T06:05:01.576 INFO:teuthology.orchestra.run.smithi064.stdout:(17/37): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-04T06:05:01.608 INFO:teuthology.orchestra.run.smithi066.stdout:(2/37): ceph-mgr-19.0.0-1550.g633e69c9.el9.x86_ 937 kB/s | 1.5 MB 00:01 2024-03-04T06:05:01.760 INFO:teuthology.orchestra.run.smithi064.stdout:(18/37): luarocks-3.9.2-1.el9.noarch.rpm 824 kB/s | 151 kB 00:00 2024-03-04T06:05:01.802 INFO:teuthology.orchestra.run.smithi064.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-04T06:05:01.894 INFO:teuthology.orchestra.run.smithi064.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-04T06:05:01.917 INFO:teuthology.orchestra.run.smithi066.stdout:(3/37): ceph-mds-19.0.0-1550.g633e69c9.el9.x86_ 1.1 MB/s | 2.0 MB 00:01 2024-03-04T06:05:02.053 INFO:teuthology.orchestra.run.smithi064.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-03-04T06:05:02.095 INFO:teuthology.orchestra.run.smithi064.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-03-04T06:05:02.101 INFO:teuthology.orchestra.run.smithi066.stdout:(4/37): ceph-mgr-modules-core-19.0.0-1550.g633e 1.3 MB/s | 247 kB 00:00 2024-03-04T06:05:02.130 INFO:teuthology.orchestra.run.smithi064.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 500 kB/s | 18 kB 00:00 2024-03-04T06:05:02.172 INFO:teuthology.orchestra.run.smithi064.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 555 kB/s | 23 kB 00:00 2024-03-04T06:05:02.214 INFO:teuthology.orchestra.run.smithi064.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 466 kB/s | 19 kB 00:00 2024-03-04T06:05:02.218 INFO:teuthology.orchestra.run.smithi066.stdout:(5/37): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-04T06:05:02.264 INFO:teuthology.orchestra.run.smithi064.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-03-04T06:05:02.285 INFO:teuthology.orchestra.run.smithi066.stdout:(6/37): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-04T06:05:02.306 INFO:teuthology.orchestra.run.smithi064.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-04T06:05:02.318 INFO:teuthology.orchestra.run.smithi066.stdout:(7/37): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-04T06:05:02.352 INFO:teuthology.orchestra.run.smithi066.stdout:(8/37): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-03-04T06:05:02.385 INFO:teuthology.orchestra.run.smithi066.stdout:(9/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-04T06:05:02.407 INFO:teuthology.orchestra.run.smithi064.stdout:(28/37): python3-more-itertools-8.12.0-2.el9.no 788 kB/s | 79 kB 00:00 2024-03-04T06:05:02.419 INFO:teuthology.orchestra.run.smithi066.stdout:(10/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-04T06:05:02.461 INFO:teuthology.orchestra.run.smithi066.stdout:(11/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-04T06:05:02.520 INFO:teuthology.orchestra.run.smithi066.stdout:(12/37): python3-cryptography-36.0.1-4.el9.x86_ 20 MB/s | 1.1 MB 00:00 2024-03-04T06:05:02.553 INFO:teuthology.orchestra.run.smithi066.stdout:(13/37): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-04T06:05:02.587 INFO:teuthology.orchestra.run.smithi066.stdout:(14/37): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-04T06:05:02.620 INFO:teuthology.orchestra.run.smithi066.stdout:(15/37): python3-requests-2.25.1-8.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2024-03-04T06:05:02.640 INFO:teuthology.orchestra.run.smithi064.stdout:(29/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-03-04T06:05:02.654 INFO:teuthology.orchestra.run.smithi066.stdout:(16/37): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-04T06:05:02.699 INFO:teuthology.orchestra.run.smithi064.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-04T06:05:02.799 INFO:teuthology.orchestra.run.smithi064.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 899 kB/s | 90 kB 00:00 2024-03-04T06:05:02.858 INFO:teuthology.orchestra.run.smithi064.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 615 kB/s | 36 kB 00:00 2024-03-04T06:05:02.879 INFO:teuthology.orchestra.run.smithi066.stdout:(17/37): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-03-04T06:05:03.000 INFO:teuthology.orchestra.run.smithi064.stdout:(33/37): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-03-04T06:05:03.055 INFO:teuthology.orchestra.run.smithi066.stdout:(18/37): luarocks-3.9.2-1.el9.noarch.rpm 864 kB/s | 151 kB 00:00 2024-03-04T06:05:03.097 INFO:teuthology.orchestra.run.smithi066.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-04T06:05:03.163 INFO:teuthology.orchestra.run.smithi066.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-04T06:05:03.175 INFO:teuthology.orchestra.run.smithi064.stdout:(34/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-03-04T06:05:03.230 INFO:teuthology.orchestra.run.smithi066.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-04T06:05:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:(35/37): ceph-mon-19.0.0-1550.g633e69c9.el9.x86 1.1 MB/s | 4.9 MB 00:04 2024-03-04T06:05:03.264 INFO:teuthology.orchestra.run.smithi066.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 319 kB/s | 11 kB 00:00 2024-03-04T06:05:03.275 INFO:teuthology.orchestra.run.smithi064.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-04T06:05:03.298 INFO:teuthology.orchestra.run.smithi066.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 532 kB/s | 18 kB 00:00 2024-03-04T06:05:03.331 INFO:teuthology.orchestra.run.smithi066.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 697 kB/s | 23 kB 00:00 2024-03-04T06:05:03.365 INFO:teuthology.orchestra.run.smithi066.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 580 kB/s | 19 kB 00:00 2024-03-04T06:05:03.407 INFO:teuthology.orchestra.run.smithi066.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 473 kB/s | 20 kB 00:00 2024-03-04T06:05:03.449 INFO:teuthology.orchestra.run.smithi066.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-04T06:05:03.491 INFO:teuthology.orchestra.run.smithi066.stdout:(28/37): python3-more-itertools-8.12.0-2.el9.no 1.9 MB/s | 79 kB 00:00 2024-03-04T06:05:03.541 INFO:teuthology.orchestra.run.smithi066.stdout:(29/37): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-04T06:05:03.583 INFO:teuthology.orchestra.run.smithi066.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2024-03-04T06:05:03.625 INFO:teuthology.orchestra.run.smithi066.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-04T06:05:03.692 INFO:teuthology.orchestra.run.smithi066.stdout:(32/37): ceph-mon-19.0.0-1550.g633e69c9.el9.x86 1.5 MB/s | 4.9 MB 00:03 2024-03-04T06:05:03.717 INFO:teuthology.orchestra.run.smithi066.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 389 kB/s | 36 kB 00:00 2024-03-04T06:05:03.750 INFO:teuthology.orchestra.run.smithi066.stdout:(34/37): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-04T06:05:03.776 INFO:teuthology.orchestra.run.smithi066.stdout:(35/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.2 MB/s | 427 kB 00:00 2024-03-04T06:05:03.801 INFO:teuthology.orchestra.run.smithi066.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 404 kB/s | 20 kB 00:00 2024-03-04T06:05:21.150 INFO:teuthology.orchestra.run.smithi066.stdout:(37/37): ceph-osd-19.0.0-1550.g633e69c9.el9.x86 6.0 MB/s | 118 MB 00:19 2024-03-04T06:05:21.155 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:05:21.155 INFO:teuthology.orchestra.run.smithi066.stdout:Total 6.2 MB/s | 132 MB 00:21 2024-03-04T06:05:21.172 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:05:21.220 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:05:21.220 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:05:22.272 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:05:22.272 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:05:24.221 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:05:24.286 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-03-04T06:05:24.325 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-03-04T06:05:24.355 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-03-04T06:05:24.427 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-03-04T06:05:24.487 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/37 2024-03-04T06:05:24.520 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-03-04T06:05:24.615 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-03-04T06:05:24.692 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-03-04T06:05:24.740 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-03-04T06:05:24.814 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-03-04T06:05:24.874 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/37 2024-03-04T06:05:24.912 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/37 2024-03-04T06:05:24.970 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/37 2024-03-04T06:05:25.026 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/37 2024-03-04T06:05:25.225 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/37 2024-03-04T06:05:25.274 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/37 2024-03-04T06:05:25.312 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/37 2024-03-04T06:05:25.347 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/37 2024-03-04T06:05:25.377 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/37 2024-03-04T06:05:25.456 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/37 2024-03-04T06:05:25.487 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/37 2024-03-04T06:05:25.520 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/37 2024-03-04T06:05:25.561 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/37 2024-03-04T06:05:25.596 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/37 2024-03-04T06:05:25.630 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/37 2024-03-04T06:05:25.722 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/37 2024-03-04T06:05:25.770 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/37 2024-03-04T06:05:25.854 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/37 2024-03-04T06:05:26.002 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/37 2024-03-04T06:05:26.126 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/37 2024-03-04T06:05:26.295 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1550.g633e69c9.el9. 31/37 2024-03-04T06:05:26.323 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-2:19.0.0-1550.g633e69c9.el9.x86_64 32/37 2024-03-04T06:05:26.351 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1550.g633e69c9.el9.x86_64 32/37 2024-03-04T06:05:26.351 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:05:26.351 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T06:05:26.351 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T06:05:26.351 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T06:05:26.351 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:05:28.165 INFO:teuthology.orchestra.run.smithi064.stdout:(37/37): ceph-osd-19.0.0-1550.g633e69c9.el9.x86 4.2 MB/s | 118 MB 00:27 2024-03-04T06:05:28.170 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:05:28.170 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.4 MB/s | 132 MB 00:29 2024-03-04T06:05:28.189 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:05:28.237 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:05:28.237 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:05:29.292 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:05:29.292 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:05:31.231 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:05:31.292 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-03-04T06:05:31.333 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-03-04T06:05:31.361 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-03-04T06:05:31.434 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-03-04T06:05:31.469 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/37 2024-03-04T06:05:31.502 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-03-04T06:05:31.605 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-03-04T06:05:31.683 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-03-04T06:05:31.731 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-03-04T06:05:31.813 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-03-04T06:05:31.865 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/37 2024-03-04T06:05:31.901 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/37 2024-03-04T06:05:31.952 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/37 2024-03-04T06:05:32.016 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/37 2024-03-04T06:05:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/37 2024-03-04T06:05:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/37 2024-03-04T06:05:32.303 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/37 2024-03-04T06:05:32.338 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/37 2024-03-04T06:05:32.367 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/37 2024-03-04T06:05:32.447 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/37 2024-03-04T06:05:32.485 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/37 2024-03-04T06:05:32.519 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/37 2024-03-04T06:05:32.560 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/37 2024-03-04T06:05:32.603 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/37 2024-03-04T06:05:32.654 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/37 2024-03-04T06:05:32.748 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/37 2024-03-04T06:05:32.785 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/37 2024-03-04T06:05:32.861 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/37 2024-03-04T06:05:33.012 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/37 2024-03-04T06:05:33.151 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/37 2024-03-04T06:05:33.319 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1550.g633e69c9.el9. 31/37 2024-03-04T06:05:33.356 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.0.0-1550.g633e69c9.el9.x86_64 32/37 2024-03-04T06:05:33.381 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1550.g633e69c9.el9.x86_64 32/37 2024-03-04T06:05:33.381 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:05:33.381 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T06:05:33.381 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T06:05:33.382 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T06:05:33.382 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:05:46.203 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 33/37 2024-03-04T06:05:46.233 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-osd-2:19.0.0-1550.g633e69c9.el9.x86_64 34/37 2024-03-04T06:05:46.261 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-osd-2:19.0.0-1550.g633e69c9.el9.x86_64 34/37 2024-03-04T06:05:46.261 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:05:46.262 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T06:05:46.262 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T06:05:46.262 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T06:05:46.262 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:05:46.750 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mon-2:19.0.0-1550.g633e69c9.el9.x86_64 35/37 2024-03-04T06:05:46.774 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mon-2:19.0.0-1550.g633e69c9.el9.x86_64 35/37 2024-03-04T06:05:46.774 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:05:46.774 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T06:05:46.775 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T06:05:46.775 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T06:05:46.775 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:05:46.992 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mds-2:19.0.0-1550.g633e69c9.el9.x86_64 36/37 2024-03-04T06:05:47.016 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mds-2:19.0.0-1550.g633e69c9.el9.x86_64 36/37 2024-03-04T06:05:47.016 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:05:47.016 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T06:05:47.017 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T06:05:47.017 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T06:05:47.017 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:05:47.084 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-2:19.0.0-1550.g633e69c9.el9.x86_64 37/37 2024-03-04T06:05:53.152 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 33/37 2024-03-04T06:05:53.180 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.0.0-1550.g633e69c9.el9.x86_64 34/37 2024-03-04T06:05:53.205 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.0.0-1550.g633e69c9.el9.x86_64 34/37 2024-03-04T06:05:53.206 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:05:53.206 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T06:05:53.206 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T06:05:53.206 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T06:05:53.206 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:05:53.681 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.0.0-1550.g633e69c9.el9.x86_64 35/37 2024-03-04T06:05:53.703 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.0.0-1550.g633e69c9.el9.x86_64 35/37 2024-03-04T06:05:53.703 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:05:53.703 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T06:05:53.703 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T06:05:53.703 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T06:05:53.703 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:05:53.915 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.0.0-1550.g633e69c9.el9.x86_64 36/37 2024-03-04T06:05:53.937 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.0.0-1550.g633e69c9.el9.x86_64 36/37 2024-03-04T06:05:53.937 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:05:53.937 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T06:05:53.937 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T06:05:53.937 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T06:05:53.937 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:05:54.011 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.0.0-1550.g633e69c9.el9.x86_64 37/37 2024-03-04T06:05:56.945 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-2:19.0.0-1550.g633e69c9.el9.x86_64 37/37 2024-03-04T06:05:56.946 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-2:19.0.0-1550.g633e69c9.el9.x86_64 1/37 2024-03-04T06:05:56.946 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mds-2:19.0.0-1550.g633e69c9.el9.x86_64 2/37 2024-03-04T06:05:56.946 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-2:19.0.0-1550.g633e69c9.el9.x86_64 3/37 2024-03-04T06:05:56.946 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mon-2:19.0.0-1550.g633e69c9.el9.x86_64 4/37 2024-03-04T06:05:56.946 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-osd-2:19.0.0-1550.g633e69c9.el9.x86_64 5/37 2024-03-04T06:05:56.946 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1550.g633e69c9.el9. 6/37 2024-03-04T06:05:56.946 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/37 2024-03-04T06:05:56.946 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 8/37 2024-03-04T06:05:56.947 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 9/37 2024-03-04T06:05:56.947 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 10/37 2024-03-04T06:05:56.947 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 11/37 2024-03-04T06:05:56.947 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 12/37 2024-03-04T06:05:56.947 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 13/37 2024-03-04T06:05:56.947 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 14/37 2024-03-04T06:05:56.947 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ply-3.11-14.el9.noarch 15/37 2024-03-04T06:05:56.947 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 16/37 2024-03-04T06:05:56.947 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/37 2024-03-04T06:05:56.948 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 18/37 2024-03-04T06:05:56.948 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-03-04T06:05:56.948 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 20/37 2024-03-04T06:05:56.948 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-03-04T06:05:56.948 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-03-04T06:05:56.948 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-03-04T06:05:56.948 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-03-04T06:05:56.948 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-03-04T06:05:56.948 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-03-04T06:05:56.948 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-03-04T06:05:56.949 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-03-04T06:05:56.949 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-03-04T06:05:56.949 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-03-04T06:05:56.949 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-03-04T06:05:56.949 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-03-04T06:05:56.949 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-03-04T06:05:56.949 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-03-04T06:05:56.949 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-03-04T06:05:56.949 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-03-04T06:05:57.907 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-03-04T06:05:57.908 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:05:57.908 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:05:57.908 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:05:57.908 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:05:57.908 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:05:57.908 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:05:57.908 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:05:57.909 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:05:57.909 INFO:teuthology.orchestra.run.smithi066.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T06:05:57.909 INFO:teuthology.orchestra.run.smithi066.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T06:05:57.909 INFO:teuthology.orchestra.run.smithi066.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T06:05:57.909 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T06:05:57.909 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T06:05:57.909 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T06:05:57.909 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T06:05:57.909 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T06:05:57.910 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T06:05:57.910 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T06:05:57.910 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T06:05:57.910 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T06:05:57.910 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T06:05:57.910 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T06:05:57.910 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T06:05:57.910 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T06:05:57.910 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T06:05:57.910 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T06:05:57.911 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T06:05:57.911 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T06:05:57.911 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T06:05:57.911 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T06:05:57.911 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T06:05:57.911 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T06:05:57.911 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T06:05:57.911 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T06:05:57.911 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T06:05:57.911 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T06:05:57.912 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T06:05:57.912 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T06:05:57.912 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T06:05:57.912 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:05:57.912 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:05:58.156 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-base 2024-03-04T06:05:58.745 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:29 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:05:58.865 INFO:teuthology.orchestra.run.smithi066.stdout:Package ceph-base-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:05:58.919 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:05:58.921 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-04T06:05:58.921 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:05:58.984 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install cephadm 2024-03-04T06:05:59.569 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:30 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:05:59.743 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:05:59.744 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:05:59.744 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-04T06:05:59.745 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:05:59.745 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:05:59.745 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 748 k 2024-03-04T06:05:59.745 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:05:59.745 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:05:59.745 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:05:59.745 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-04T06:05:59.745 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:05:59.746 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 748 k 2024-03-04T06:05:59.746 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 754 k 2024-03-04T06:05:59.746 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:00.101 INFO:teuthology.orchestra.run.smithi066.stdout:cephadm-19.0.0-1550.g633e69c9.el9.noarch.rpm 2.1 MB/s | 748 kB 00:00 2024-03-04T06:06:00.101 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:00.101 INFO:teuthology.orchestra.run.smithi066.stdout:Total 2.1 MB/s | 748 kB 00:00 2024-03-04T06:06:00.101 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:00.107 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:00.107 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:00.113 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:00.113 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:00.192 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:00.753 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 1/1 2024-03-04T06:06:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 1/1 2024-03-04T06:06:01.252 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 1/1 2024-03-04T06:06:01.551 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 1/1 2024-03-04T06:06:01.551 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:01.551 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:01.551 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:01.551 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:01.551 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:01.773 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-immutable-object-cache 2024-03-04T06:06:02.368 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:33 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:02.542 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:02.543 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:02.544 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-04T06:06:02.544 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:02.544 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:02.544 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 147 k 2024-03-04T06:06:02.544 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:02.544 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:02.544 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:02.544 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-04T06:06:02.544 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:02.545 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 147 k 2024-03-04T06:06:02.545 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 481 k 2024-03-04T06:06:02.545 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:03.205 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-immutable-object-cache-19.0.0-1550.g633e69 223 kB/s | 147 kB 00:00 2024-03-04T06:06:03.206 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:03.206 INFO:teuthology.orchestra.run.smithi066.stdout:Total 223 kB/s | 147 kB 00:00 2024-03-04T06:06:03.207 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:03.215 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:03.215 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:03.271 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:03.272 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:03.473 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1550.g633e69c9. 1/1 2024-03-04T06:06:03.499 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1550.g633e69c9. 1/1 2024-03-04T06:06:03.499 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:06:03.499 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T06:06:03.499 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:03.753 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:19.0.0-1550.g633e69c9.el9.x86_64 37/37 2024-03-04T06:06:03.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.0.0-1550.g633e69c9.el9.x86_64 1/37 2024-03-04T06:06:03.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.0.0-1550.g633e69c9.el9.x86_64 2/37 2024-03-04T06:06:03.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.0.0-1550.g633e69c9.el9.x86_64 3/37 2024-03-04T06:06:03.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.0.0-1550.g633e69c9.el9.x86_64 4/37 2024-03-04T06:06:03.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.0.0-1550.g633e69c9.el9.x86_64 5/37 2024-03-04T06:06:03.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1550.g633e69c9.el9. 6/37 2024-03-04T06:06:03.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/37 2024-03-04T06:06:03.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 8/37 2024-03-04T06:06:03.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 9/37 2024-03-04T06:06:03.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 10/37 2024-03-04T06:06:03.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 11/37 2024-03-04T06:06:03.754 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 12/37 2024-03-04T06:06:03.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 13/37 2024-03-04T06:06:03.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 14/37 2024-03-04T06:06:03.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 15/37 2024-03-04T06:06:03.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 16/37 2024-03-04T06:06:03.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/37 2024-03-04T06:06:03.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 18/37 2024-03-04T06:06:03.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-03-04T06:06:03.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 20/37 2024-03-04T06:06:03.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-03-04T06:06:03.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-03-04T06:06:03.755 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-03-04T06:06:03.756 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-03-04T06:06:03.756 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-03-04T06:06:03.756 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-03-04T06:06:03.756 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-03-04T06:06:03.757 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-03-04T06:06:03.757 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-03-04T06:06:03.758 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-03-04T06:06:03.758 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-03-04T06:06:03.758 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-03-04T06:06:03.758 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-03-04T06:06:03.758 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-03-04T06:06:03.758 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-03-04T06:06:03.758 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-03-04T06:06:04.539 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1550.g633e69c9. 1/1 2024-03-04T06:06:04.539 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:04.539 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:04.539 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:04.539 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:04.539 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-03-04T06:06:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:04.723 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:06:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T06:06:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T06:06:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T06:06:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T06:06:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T06:06:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T06:06:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T06:06:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T06:06:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T06:06:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T06:06:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T06:06:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T06:06:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T06:06:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T06:06:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T06:06:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T06:06:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T06:06:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T06:06:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T06:06:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T06:06:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T06:06:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T06:06:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T06:06:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T06:06:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T06:06:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T06:06:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T06:06:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T06:06:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T06:06:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T06:06:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T06:06:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:04.727 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:04.787 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr 2024-03-04T06:06:04.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-03-04T06:06:05.375 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:36 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:05.497 INFO:teuthology.orchestra.run.smithi066.stdout:Package ceph-mgr-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:05.538 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:49 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:05.552 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:05.554 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-04T06:06:05.554 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:05.620 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-dashboard 2024-03-04T06:06:05.657 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:05.713 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:05.714 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-04T06:06:05.714 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:05.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-03-04T06:06:06.210 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:37 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:06.336 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:50 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:06.384 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:06.386 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-03-04T06:06:06.386 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-04T06:06:06.386 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-03-04T06:06:06.386 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:06.386 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 1.8 M 2024-03-04T06:06:06.386 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-04T06:06:06.386 INFO:teuthology.orchestra.run.smithi066.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T06:06:06.386 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 31 k 2024-03-04T06:06:06.386 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 15 k 2024-03-04T06:06:06.386 INFO:teuthology.orchestra.run.smithi066.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T06:06:06.387 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T06:06:06.387 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T06:06:06.387 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T06:06:06.387 INFO:teuthology.orchestra.run.smithi066.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T06:06:06.387 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T06:06:06.387 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T06:06:06.387 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:06.387 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:06.387 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-03-04T06:06:06.387 INFO:teuthology.orchestra.run.smithi066.stdout:Install 11 Packages 2024-03-04T06:06:06.387 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:06.388 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 5.8 M 2024-03-04T06:06:06.388 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 22 M 2024-03-04T06:06:06.388 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:06.510 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:06.511 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:06.511 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-04T06:06:06.511 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:06.512 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:06:06.512 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 748 k 2024-03-04T06:06:06.512 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:06.512 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:06:06.512 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:06.512 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-04T06:06:06.512 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:06.512 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 748 k 2024-03-04T06:06:06.512 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 754 k 2024-03-04T06:06:06.512 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:06:06.942 INFO:teuthology.orchestra.run.smithi066.stdout:(1/11): ceph-grafana-dashboards-19.0.0-1550.g63 124 kB/s | 31 kB 00:00 2024-03-04T06:06:06.956 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-19.0.0-1550.g633e69c9.el9.noarch.rpm 1.6 MB/s | 748 kB 00:00 2024-03-04T06:06:06.957 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:06.957 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.6 MB/s | 748 kB 00:00 2024-03-04T06:06:06.957 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:06:06.963 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:06:06.963 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:06:06.967 INFO:teuthology.orchestra.run.smithi066.stdout:(2/11): ceph-prometheus-alerts-19.0.0-1550.g633 55 kB/s | 15 kB 00:00 2024-03-04T06:06:06.969 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:06:06.969 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:06:07.047 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:06:07.076 INFO:teuthology.orchestra.run.smithi066.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-04T06:06:07.252 INFO:teuthology.orchestra.run.smithi066.stdout:(4/11): ceph-mgr-dashboard-19.0.0-1550.g633e69c 3.2 MB/s | 1.8 MB 00:00 2024-03-04T06:06:07.493 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 1/1 2024-03-04T06:06:07.606 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 1/1 2024-03-04T06:06:07.610 INFO:teuthology.orchestra.run.smithi066.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 54 kB/s | 19 kB 00:00 2024-03-04T06:06:07.744 INFO:teuthology.orchestra.run.smithi066.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 821 kB/s | 548 kB 00:00 2024-03-04T06:06:07.852 INFO:teuthology.orchestra.run.smithi066.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-04T06:06:07.961 INFO:teuthology.orchestra.run.smithi066.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-04T06:06:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 1/1 2024-03-04T06:06:08.078 INFO:teuthology.orchestra.run.smithi066.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-04T06:06:08.220 INFO:teuthology.orchestra.run.smithi066.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-03-04T06:06:08.369 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 1/1 2024-03-04T06:06:08.369 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:08.369 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:06:08.369 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:08.369 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:08.369 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:08.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-03-04T06:06:09.111 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:53 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:09.287 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:09.287 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-04T06:06:09.288 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:09.288 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:06:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 147 k 2024-03-04T06:06:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:09.288 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:06:09.288 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:09.288 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-04T06:06:09.288 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:09.288 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 147 k 2024-03-04T06:06:09.289 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 481 k 2024-03-04T06:06:09.289 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:06:09.605 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-19.0.0-1550.g633e69 467 kB/s | 147 kB 00:00 2024-03-04T06:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout:Total 464 kB/s | 147 kB 00:00 2024-03-04T06:06:09.606 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:06:09.614 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:06:09.669 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:06:09.669 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:06:09.823 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:06:09.880 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1550.g633e69c9. 1/1 2024-03-04T06:06:09.903 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1550.g633e69c9. 1/1 2024-03-04T06:06:09.903 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:06:09.904 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T06:06:09.904 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:09.970 INFO:teuthology.orchestra.run.smithi066.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 290 kB/s | 863 kB 00:02 2024-03-04T06:06:09.975 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:09.975 INFO:teuthology.orchestra.run.smithi066.stdout:Total 1.6 MB/s | 5.8 MB 00:03 2024-03-04T06:06:09.995 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:10.015 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:10.015 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:10.196 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:10.196 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:10.693 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:10.768 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-03-04T06:06:10.798 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-03-04T06:06:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1550.g633e69c9. 1/1 2024-03-04T06:06:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:10.880 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:06:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:10.880 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:10.894 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-03-04T06:06:11.028 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-03-04T06:06:11.085 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-03-04T06:06:11.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-03-04T06:06:11.171 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-03-04T06:06:11.220 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-03-04T06:06:11.257 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-03-04T06:06:11.289 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1550.g633e69c9.el9 9/11 2024-03-04T06:06:11.540 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1550.g633e69c9.el 10/11 2024-03-04T06:06:11.572 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1550.g633e69c9.el9.noa 11/11 2024-03-04T06:06:11.712 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:55 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:11.830 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:11.885 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:11.887 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-04T06:06:11.887 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:12.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-03-04T06:06:12.368 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1550.g633e69c9.el9.noa 11/11 2024-03-04T06:06:12.368 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1550.g633e69c9.el 1/11 2024-03-04T06:06:12.368 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1550.g633e69c9.el9.noa 2/11 2024-03-04T06:06:12.368 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1550.g633e69c9.el9 3/11 2024-03-04T06:06:12.368 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-03-04T06:06:12.369 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-03-04T06:06:12.369 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-03-04T06:06:12.369 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-03-04T06:06:12.369 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-03-04T06:06:12.369 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-03-04T06:06:12.369 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-03-04T06:06:12.645 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:56 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:12.727 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-03-04T06:06:12.728 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:12.728 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:12.728 INFO:teuthology.orchestra.run.smithi066.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T06:06:12.728 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:12.728 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:12.728 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:12.728 INFO:teuthology.orchestra.run.smithi066.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T06:06:12.728 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T06:06:12.728 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T06:06:12.729 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T06:06:12.729 INFO:teuthology.orchestra.run.smithi066.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T06:06:12.729 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T06:06:12.729 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T06:06:12.729 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:12.729 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:12.818 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:12.819 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-04T06:06:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-04T06:06:12.819 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-04T06:06:12.819 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:06:12.819 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 1.8 M 2024-03-04T06:06:12.820 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-04T06:06:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T06:06:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 31 k 2024-03-04T06:06:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 15 k 2024-03-04T06:06:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T06:06:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T06:06:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T06:06:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T06:06:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T06:06:12.820 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T06:06:12.821 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T06:06:12.821 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:12.821 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:06:12.821 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-04T06:06:12.821 INFO:teuthology.orchestra.run.smithi064.stdout:Install 11 Packages 2024-03-04T06:06:12.821 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:12.822 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 5.8 M 2024-03-04T06:06:12.822 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 22 M 2024-03-04T06:06:12.822 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:06:12.978 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-04T06:06:13.570 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:44 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:13.650 INFO:teuthology.orchestra.run.smithi064.stdout:(1/11): ceph-grafana-dashboards-19.0.0-1550.g63 156 kB/s | 31 kB 00:00 2024-03-04T06:06:13.675 INFO:teuthology.orchestra.run.smithi064.stdout:(2/11): ceph-prometheus-alerts-19.0.0-1550.g633 67 kB/s | 15 kB 00:00 2024-03-04T06:06:13.747 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:13.748 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================================ 2024-03-04T06:06:13.749 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-04T06:06:13.749 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================================ 2024-03-04T06:06:13.749 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:13.749 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 7.4 M 2024-03-04T06:06:13.749 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-04T06:06:13.749 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T06:06:13.749 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T06:06:13.749 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T06:06:13.750 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T06:06:13.750 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T06:06:13.750 INFO:teuthology.orchestra.run.smithi066.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T06:06:13.750 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T06:06:13.750 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T06:06:13.750 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T06:06:13.750 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T06:06:13.750 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T06:06:13.750 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:13.750 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:13.751 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================================ 2024-03-04T06:06:13.751 INFO:teuthology.orchestra.run.smithi066.stdout:Install 12 Packages 2024-03-04T06:06:13.751 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:13.751 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 37 M 2024-03-04T06:06:13.752 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 209 M 2024-03-04T06:06:13.752 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:13.792 INFO:teuthology.orchestra.run.smithi064.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-04T06:06:13.876 INFO:teuthology.orchestra.run.smithi064.stdout:(4/11): ceph-mgr-dashboard-19.0.0-1550.g633e69c 4.2 MB/s | 1.8 MB 00:00 2024-03-04T06:06:13.925 INFO:teuthology.orchestra.run.smithi066.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 237 kB/s | 32 kB 00:00 2024-03-04T06:06:13.968 INFO:teuthology.orchestra.run.smithi066.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 424 kB/s | 17 kB 00:00 2024-03-04T06:06:14.035 INFO:teuthology.orchestra.run.smithi066.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-04T06:06:14.060 INFO:teuthology.orchestra.run.smithi066.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 381 kB/s | 34 kB 00:00 2024-03-04T06:06:14.210 INFO:teuthology.orchestra.run.smithi066.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-04T06:06:14.277 INFO:teuthology.orchestra.run.smithi066.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2024-03-04T06:06:14.302 INFO:teuthology.orchestra.run.smithi066.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-04T06:06:14.376 INFO:teuthology.orchestra.run.smithi064.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 41 kB/s | 19 kB 00:00 2024-03-04T06:06:14.461 INFO:teuthology.orchestra.run.smithi066.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 802 kB 00:00 2024-03-04T06:06:14.620 INFO:teuthology.orchestra.run.smithi066.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-04T06:06:14.645 INFO:teuthology.orchestra.run.smithi066.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-03-04T06:06:14.812 INFO:teuthology.orchestra.run.smithi066.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-04T06:06:14.968 INFO:teuthology.orchestra.run.smithi064.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 466 kB/s | 548 kB 00:01 2024-03-04T06:06:15.285 INFO:teuthology.orchestra.run.smithi064.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 456 kB/s | 144 kB 00:00 2024-03-04T06:06:15.410 INFO:teuthology.orchestra.run.smithi064.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 246 kB/s | 31 kB 00:00 2024-03-04T06:06:15.437 INFO:teuthology.orchestra.run.smithi066.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.5 MB/s | 7.4 MB 00:01 2024-03-04T06:06:15.440 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:15.440 INFO:teuthology.orchestra.run.smithi066.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-04T06:06:15.454 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:15.482 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:15.482 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:15.677 INFO:teuthology.orchestra.run.smithi064.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 707 kB/s | 188 kB 00:00 2024-03-04T06:06:15.869 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:15.869 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:16.244 INFO:teuthology.orchestra.run.smithi064.stdout:(10/11): protobuf-compiler-3.14.0-13.el9.x86_64 336 kB/s | 863 kB 00:02 2024-03-04T06:06:16.587 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:16.652 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-04T06:06:16.700 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-04T06:06:16.799 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-04T06:06:16.837 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-04T06:06:17.021 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T06:06:17.052 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T06:06:17.139 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-04T06:06:17.161 INFO:teuthology.orchestra.run.smithi064.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 749 kB/s | 2.0 MB 00:02 2024-03-04T06:06:17.166 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:17.166 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.3 MB/s | 5.8 MB 00:04 2024-03-04T06:06:17.186 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:06:17.206 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:06:17.206 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:06:17.381 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:06:17.381 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:06:17.836 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-04T06:06:17.951 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-04T06:06:17.987 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:06:18.067 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-03-04T06:06:18.106 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-03-04T06:06:18.229 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-03-04T06:06:18.360 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-03-04T06:06:18.418 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-03-04T06:06:18.504 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-03-04T06:06:18.553 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-03-04T06:06:18.581 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-03-04T06:06:18.613 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1550.g633e69c9.el9 9/11 2024-03-04T06:06:18.863 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1550.g633e69c9.el 10/11 2024-03-04T06:06:18.904 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1550.g633e69c9.el9.noa 11/11 2024-03-04T06:06:19.247 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-04T06:06:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1550.g633e69c9.el9.noa 11/11 2024-03-04T06:06:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1550.g633e69c9.el 1/11 2024-03-04T06:06:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1550.g633e69c9.el9.noa 2/11 2024-03-04T06:06:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1550.g633e69c9.el9 3/11 2024-03-04T06:06:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-03-04T06:06:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-03-04T06:06:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-03-04T06:06:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-03-04T06:06:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-03-04T06:06:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-03-04T06:06:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-03-04T06:06:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-03-04T06:06:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:20.060 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:06:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T06:06:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T06:06:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T06:06:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T06:06:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T06:06:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T06:06:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T06:06:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T06:06:20.061 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:20.062 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:20.328 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-04T06:06:20.351 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1550.g633e6 12/12 2024-03-04T06:06:20.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-04T06:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:05 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:21.204 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:21.206 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-04T06:06:21.206 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-04T06:06:21.206 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-04T06:06:21.206 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:06:21.206 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 7.4 M 2024-03-04T06:06:21.206 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-04T06:06:21.206 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T06:06:21.206 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T06:06:21.207 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T06:06:21.207 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T06:06:21.207 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T06:06:21.207 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T06:06:21.207 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T06:06:21.207 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T06:06:21.207 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T06:06:21.207 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T06:06:21.207 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T06:06:21.207 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:21.208 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:06:21.208 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-04T06:06:21.208 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-03-04T06:06:21.208 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:21.209 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 37 M 2024-03-04T06:06:21.209 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 209 M 2024-03-04T06:06:21.209 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:06:21.468 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 243 kB/s | 32 kB 00:00 2024-03-04T06:06:21.492 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 732 kB/s | 17 kB 00:00 2024-03-04T06:06:21.517 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-03-04T06:06:21.576 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-04T06:06:21.743 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-04T06:06:21.818 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-04T06:06:21.844 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-04T06:06:21.944 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 8.0 MB/s | 802 kB 00:00 2024-03-04T06:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-15 9.7 MB/s | 7.4 MB 00:00 2024-03-04T06:06:22.253 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-04T06:06:22.278 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 586 kB/s | 194 kB 00:00 2024-03-04T06:06:22.810 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1550.g633e6 12/12 2024-03-04T06:06:22.811 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1550.g633e6 1/12 2024-03-04T06:06:22.811 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-04T06:06:22.811 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-04T06:06:22.811 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-04T06:06:22.811 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T06:06:22.811 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T06:06:22.811 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-04T06:06:22.811 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-04T06:06:22.812 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-04T06:06:22.812 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-04T06:06:22.812 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-04T06:06:23.261 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-04T06:06:23.261 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:23.261 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:23.261 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:23.261 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T06:06:23.261 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T06:06:23.261 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T06:06:23.261 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T06:06:23.262 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T06:06:23.262 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T06:06:23.262 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T06:06:23.262 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T06:06:23.262 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T06:06:23.262 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T06:06:23.262 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T06:06:23.262 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:23.262 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:23.689 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-rook 2024-03-04T06:06:24.283 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:55 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:24.461 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:24.462 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-04T06:06:24.462 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-04T06:06:24.462 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-04T06:06:24.462 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:24.462 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 52 k 2024-03-04T06:06:24.463 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-04T06:06:24.463 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T06:06:24.463 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T06:06:24.463 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T06:06:24.463 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T06:06:24.463 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T06:06:24.463 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T06:06:24.463 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T06:06:24.463 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T06:06:24.464 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T06:06:24.464 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T06:06:24.464 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T06:06:24.464 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T06:06:24.464 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:24.464 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:24.464 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-04T06:06:24.464 INFO:teuthology.orchestra.run.smithi066.stdout:Install 13 Packages 2024-03-04T06:06:24.464 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:24.465 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 2.0 M 2024-03-04T06:06:24.465 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 26 M 2024-03-04T06:06:24.465 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:25.000 INFO:teuthology.orchestra.run.smithi066.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-03-04T06:06:25.017 INFO:teuthology.orchestra.run.smithi066.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 143 kB/s | 24 kB 00:00 2024-03-04T06:06:25.142 INFO:teuthology.orchestra.run.smithi066.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 174 kB 00:00 2024-03-04T06:06:25.201 INFO:teuthology.orchestra.run.smithi066.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 719 kB/s | 132 kB 00:00 2024-03-04T06:06:25.243 INFO:teuthology.orchestra.run.smithi066.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.0 MB/s | 211 kB 00:00 2024-03-04T06:06:25.268 INFO:teuthology.orchestra.run.smithi066.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 643 kB/s | 43 kB 00:00 2024-03-04T06:06:25.294 INFO:teuthology.orchestra.run.smithi066.stdout:(7/13): ceph-mgr-rook-19.0.0-1550.g633e69c9.el9 117 kB/s | 52 kB 00:00 2024-03-04T06:06:25.528 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 75 kB/s | 245 kB 00:03 2024-03-04T06:06:25.531 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:25.532 INFO:teuthology.orchestra.run.smithi064.stdout:Total 8.6 MB/s | 37 MB 00:04 2024-03-04T06:06:25.546 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:06:25.574 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:06:25.574 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:06:25.694 INFO:teuthology.orchestra.run.smithi066.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 35 kB/s | 15 kB 00:00 2024-03-04T06:06:25.794 INFO:teuthology.orchestra.run.smithi066.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 58 kB/s | 32 kB 00:00 2024-03-04T06:06:25.949 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:06:25.949 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:06:26.086 INFO:teuthology.orchestra.run.smithi066.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 202 kB/s | 59 kB 00:00 2024-03-04T06:06:26.111 INFO:teuthology.orchestra.run.smithi066.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 248 kB/s | 203 kB 00:00 2024-03-04T06:06:26.311 INFO:teuthology.orchestra.run.smithi066.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 398 kB/s | 90 kB 00:00 2024-03-04T06:06:26.428 INFO:teuthology.orchestra.run.smithi066.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.4 MB/s | 1.0 MB 00:00 2024-03-04T06:06:26.434 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:26.434 INFO:teuthology.orchestra.run.smithi066.stdout:Total 1.0 MB/s | 2.0 MB 00:01 2024-03-04T06:06:26.434 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:26.450 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:26.450 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:26.571 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:26.574 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:26.779 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:06:26.843 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-04T06:06:26.874 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-04T06:06:26.973 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-04T06:06:27.019 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-04T06:06:27.136 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:27.204 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T06:06:27.235 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T06:06:27.306 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-04T06:06:27.321 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-04T06:06:27.365 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-04T06:06:27.420 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-04T06:06:27.456 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-04T06:06:27.498 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-04T06:06:27.596 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-04T06:06:27.711 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-04T06:06:27.766 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-04T06:06:27.999 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-04T06:06:28.118 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-04T06:06:28.522 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-04T06:06:28.570 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-04T06:06:28.604 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-04T06:06:28.652 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-04T06:06:28.673 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-rook-2:19.0.0-1550.g633e69c9.el9.noarch 13/13 2024-03-04T06:06:29.397 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-04T06:06:29.503 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1550.g633e69c9.el9.noarch 13/13 2024-03-04T06:06:29.503 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1550.g633e69c9.el9.noarch 1/13 2024-03-04T06:06:29.503 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-04T06:06:29.503 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-04T06:06:29.503 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-04T06:06:29.503 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-04T06:06:29.504 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-04T06:06:29.504 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-04T06:06:29.504 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-04T06:06:29.504 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-04T06:06:29.504 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-04T06:06:29.504 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-04T06:06:29.504 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-04T06:06:29.892 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-04T06:06:29.892 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T06:06:29.893 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T06:06:29.894 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T06:06:29.894 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:29.894 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:30.130 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-cephadm 2024-03-04T06:06:30.498 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-04T06:06:30.525 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1550.g633e6 12/12 2024-03-04T06:06:30.727 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:01 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:30.902 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:30.903 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 138 k 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:30.904 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-04T06:06:30.905 INFO:teuthology.orchestra.run.smithi066.stdout:Install 6 Packages 2024-03-04T06:06:30.905 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:30.905 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 6.8 M 2024-03-04T06:06:30.905 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 33 M 2024-03-04T06:06:30.905 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:31.155 INFO:teuthology.orchestra.run.smithi066.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 227 kB 00:00 2024-03-04T06:06:31.313 INFO:teuthology.orchestra.run.smithi066.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-04T06:06:31.413 INFO:teuthology.orchestra.run.smithi066.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1550.g633e69c9.e 325 kB/s | 138 kB 00:00 2024-03-04T06:06:31.605 INFO:teuthology.orchestra.run.smithi066.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 199 kB/s | 58 kB 00:00 2024-03-04T06:06:31.639 INFO:teuthology.orchestra.run.smithi066.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-04T06:06:31.697 INFO:teuthology.orchestra.run.smithi066.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 180 kB/s | 51 kB 00:00 2024-03-04T06:06:31.702 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:31.702 INFO:teuthology.orchestra.run.smithi066.stdout:Total 8.5 MB/s | 6.8 MB 00:00 2024-03-04T06:06:31.702 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:31.712 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:31.712 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:31.810 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:31.810 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:32.122 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:32.295 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-04T06:06:32.350 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-04T06:06:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1550.g633e6 12/12 2024-03-04T06:06:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1550.g633e6 1/12 2024-03-04T06:06:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-04T06:06:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-04T06:06:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-04T06:06:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T06:06:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T06:06:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-04T06:06:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-04T06:06:32.894 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-04T06:06:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-04T06:06:32.895 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-04T06:06:32.909 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-04T06:06:32.997 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-04T06:06:33.074 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-04T06:06:33.103 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 6/6 2024-03-04T06:06:33.487 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-04T06:06:33.487 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:33.487 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:06:33.487 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:33.487 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:33.488 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:33.864 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-03-04T06:06:33.895 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 6/6 2024-03-04T06:06:33.895 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 1/6 2024-03-04T06:06:33.895 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-04T06:06:33.895 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-04T06:06:33.895 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-04T06:06:33.895 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-04T06:06:34.225 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-04T06:06:34.225 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:34.225 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:34.225 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:34.225 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T06:06:34.226 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T06:06:34.226 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T06:06:34.226 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T06:06:34.226 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T06:06:34.226 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:34.226 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:34.444 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:18 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:34.507 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-fuse 2024-03-04T06:06:34.628 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:34.630 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-04T06:06:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-04T06:06:34.630 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-04T06:06:34.630 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:06:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 52 k 2024-03-04T06:06:34.630 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-04T06:06:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T06:06:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T06:06:34.630 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T06:06:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T06:06:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T06:06:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T06:06:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T06:06:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T06:06:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T06:06:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T06:06:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T06:06:34.631 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T06:06:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:34.632 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:06:34.632 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-04T06:06:34.632 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-03-04T06:06:34.632 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:34.633 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.0 M 2024-03-04T06:06:34.633 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-03-04T06:06:34.633 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:06:34.818 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 181 kB/s | 24 kB 00:00 2024-03-04T06:06:34.835 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-03-04T06:06:34.910 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-03-04T06:06:34.935 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-03-04T06:06:34.969 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-04T06:06:34.994 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 733 kB/s | 43 kB 00:00 2024-03-04T06:06:35.019 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): ceph-mgr-rook-19.0.0-1550.g633e69c9.el9 155 kB/s | 52 kB 00:00 2024-03-04T06:06:35.100 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:06 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:35.162 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-03-04T06:06:35.203 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 72 kB/s | 15 kB 00:00 2024-03-04T06:06:35.277 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:35.278 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:35.278 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-04T06:06:35.278 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:35.278 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:35.278 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 859 k 2024-03-04T06:06:35.279 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-04T06:06:35.279 INFO:teuthology.orchestra.run.smithi066.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T06:06:35.279 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:35.279 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:35.279 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:35.279 INFO:teuthology.orchestra.run.smithi066.stdout:Install 2 Packages 2024-03-04T06:06:35.279 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:35.279 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 937 k 2024-03-04T06:06:35.279 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 3.1 M 2024-03-04T06:06:35.279 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:35.295 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 645 kB/s | 59 kB 00:00 2024-03-04T06:06:35.337 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 755 kB/s | 203 kB 00:00 2024-03-04T06:06:35.371 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 4.9 MB/s | 1.0 MB 00:00 2024-03-04T06:06:35.396 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 899 kB/s | 90 kB 00:00 2024-03-04T06:06:35.401 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:35.401 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-03-04T06:06:35.401 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:06:35.415 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:06:35.415 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:06:35.449 INFO:teuthology.orchestra.run.smithi066.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 463 kB/s | 79 kB 00:00 2024-03-04T06:06:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:06:35.520 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:06:35.749 INFO:teuthology.orchestra.run.smithi066.stdout:(2/2): ceph-fuse-19.0.0-1550.g633e69c9.el9.x86_ 1.8 MB/s | 859 kB 00:00 2024-03-04T06:06:35.751 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:35.751 INFO:teuthology.orchestra.run.smithi066.stdout:Total 1.9 MB/s | 937 kB 00:00 2024-03-04T06:06:35.751 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:35.773 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:35.773 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:35.854 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:35.854 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:36.042 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:36.172 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-04T06:06:36.174 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:06:36.243 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 2/2 2024-03-04T06:06:36.349 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-04T06:06:36.398 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-04T06:06:36.454 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-04T06:06:36.490 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-04T06:06:36.541 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-04T06:06:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-04T06:06:36.829 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-04T06:06:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-04T06:06:37.040 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 2/2 2024-03-04T06:06:37.040 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 1/2 2024-03-04T06:06:37.408 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-04T06:06:37.408 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:37.408 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:37.408 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-04T06:06:37.408 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:37.408 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:37.633 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-04T06:06:37.689 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-04T06:06:37.698 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-volume 2024-03-04T06:06:37.731 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-04T06:06:37.788 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-04T06:06:37.817 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.0.0-1550.g633e69c9.el9.noarch 13/13 2024-03-04T06:06:38.296 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:09 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:38.473 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:38.474 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:38.474 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-04T06:06:38.475 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:38.475 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:38.475 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-volume noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 264 k 2024-03-04T06:06:38.475 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-04T06:06:38.475 INFO:teuthology.orchestra.run.smithi066.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T06:06:38.475 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:38.475 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:38.475 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:38.475 INFO:teuthology.orchestra.run.smithi066.stdout:Install 2 Packages 2024-03-04T06:06:38.475 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:38.476 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 332 k 2024-03-04T06:06:38.476 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 1.5 M 2024-03-04T06:06:38.476 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1550.g633e69c9.el9.noarch 13/13 2024-03-04T06:06:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1550.g633e69c9.el9.noarch 1/13 2024-03-04T06:06:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-04T06:06:38.524 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-04T06:06:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-04T06:06:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-04T06:06:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-04T06:06:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-04T06:06:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-04T06:06:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-04T06:06:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-04T06:06:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-04T06:06:38.525 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-04T06:06:38.604 INFO:teuthology.orchestra.run.smithi066.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 533 kB/s | 69 kB 00:00 2024-03-04T06:06:38.888 INFO:teuthology.orchestra.run.smithi066.stdout:(2/2): ceph-volume-19.0.0-1550.g633e69c9.el9.no 640 kB/s | 264 kB 00:00 2024-03-04T06:06:38.889 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:38.889 INFO:teuthology.orchestra.run.smithi066.stdout:Total 801 kB/s | 332 kB 00:00 2024-03-04T06:06:38.889 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:38.899 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:38.899 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-04T06:06:38.927 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T06:06:38.928 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T06:06:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T06:06:38.929 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:38.929 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:38.944 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:38.944 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:39.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-03-04T06:06:39.184 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:39.364 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-04T06:06:39.407 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-volume-2:19.0.0-1550.g633e69c9.el9.noarch 2/2 2024-03-04T06:06:39.423 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-volume-2:19.0.0-1550.g633e69c9.el9.noarch 2/2 2024-03-04T06:06:39.423 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:06:39.423 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T06:06:39.424 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:39.759 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:23 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:39.935 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:39.936 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-04T06:06:39.936 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-04T06:06:39.936 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-04T06:06:39.936 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:06:39.936 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 138 k 2024-03-04T06:06:39.937 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-04T06:06:39.937 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T06:06:39.937 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T06:06:39.937 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T06:06:39.937 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T06:06:39.937 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T06:06:39.937 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:39.937 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:06:39.937 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-04T06:06:39.937 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-03-04T06:06:39.937 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:39.938 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 6.8 M 2024-03-04T06:06:39.938 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-03-04T06:06:39.938 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:06:40.181 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 227 kB 00:00 2024-03-04T06:06:40.214 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-volume-2:19.0.0-1550.g633e69c9.el9.noarch 1/2 2024-03-04T06:06:40.332 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-04T06:06:40.460 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1550.g633e69c9.e 314 kB/s | 138 kB 00:00 2024-03-04T06:06:40.590 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-04T06:06:40.590 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:40.590 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:40.590 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-volume-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:40.590 INFO:teuthology.orchestra.run.smithi066.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T06:06:40.591 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:40.591 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:40.875 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 111 kB/s | 58 kB 00:00 2024-03-04T06:06:40.901 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados-devel 2024-03-04T06:06:40.949 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 110 kB/s | 51 kB 00:00 2024-03-04T06:06:40.999 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 671 kB/s | 548 kB 00:00 2024-03-04T06:06:41.003 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:41.004 INFO:teuthology.orchestra.run.smithi064.stdout:Total 6.4 MB/s | 6.8 MB 00:01 2024-03-04T06:06:41.004 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:06:41.013 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:06:41.013 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:06:41.106 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:06:41.107 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:06:41.495 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:12 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:41.532 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:06:41.671 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:41.672 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:41.672 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-04T06:06:41.672 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:41.672 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:41.672 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 127 k 2024-03-04T06:06:41.672 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:41.672 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:41.672 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:41.673 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-04T06:06:41.673 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:41.673 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 127 k 2024-03-04T06:06:41.673 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 457 k 2024-03-04T06:06:41.673 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:41.695 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-04T06:06:41.744 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-04T06:06:42.012 INFO:teuthology.orchestra.run.smithi066.stdout:librados-devel-19.0.0-1550.g633e69c9.el9.x86_64 374 kB/s | 127 kB 00:00 2024-03-04T06:06:42.013 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:42.013 INFO:teuthology.orchestra.run.smithi066.stdout:Total 373 kB/s | 127 kB 00:00 2024-03-04T06:06:42.013 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:42.018 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:42.019 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:42.078 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:42.078 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:42.234 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:42.290 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-04T06:06:42.371 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librados-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:42.372 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-04T06:06:42.433 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-04T06:06:42.463 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 6/6 2024-03-04T06:06:42.834 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:43.180 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:43.180 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:43.181 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:43.181 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:43.181 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:43.181 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:43.281 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 6/6 2024-03-04T06:06:43.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 1/6 2024-03-04T06:06:43.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-04T06:06:43.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-04T06:06:43.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-04T06:06:43.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-04T06:06:43.448 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install libcephfs2 2024-03-04T06:06:43.599 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-04T06:06:43.599 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:43.599 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:06:43.599 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:43.600 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T06:06:43.600 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T06:06:43.600 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T06:06:43.600 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T06:06:43.600 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T06:06:43.600 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:43.600 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:43.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-03-04T06:06:44.041 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:15 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:44.160 INFO:teuthology.orchestra.run.smithi066.stdout:Package libcephfs2-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:44.215 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:44.216 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-04T06:06:44.216 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:44.287 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install libcephfs-devel 2024-03-04T06:06:44.415 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:28 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:44.594 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:44.594 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-04T06:06:44.595 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:44.595 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:06:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 859 k 2024-03-04T06:06:44.595 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-04T06:06:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T06:06:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:44.595 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:06:44.595 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:44.595 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-03-04T06:06:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:44.596 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 937 k 2024-03-04T06:06:44.596 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 3.1 M 2024-03-04T06:06:44.596 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:06:44.726 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 605 kB/s | 79 kB 00:00 2024-03-04T06:06:44.877 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:15 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:44.984 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-19.0.0-1550.g633e69c9.el9.x86_ 2.2 MB/s | 859 kB 00:00 2024-03-04T06:06:44.986 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:44.986 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.3 MB/s | 937 kB 00:00 2024-03-04T06:06:44.987 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:06:45.007 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:06:45.007 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:06:45.052 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:45.053 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-04T06:06:45.053 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:45.053 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 31 k 2024-03-04T06:06:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:45.053 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:45.053 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:45.053 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-04T06:06:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:45.054 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 31 k 2024-03-04T06:06:45.054 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 140 k 2024-03-04T06:06:45.054 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:45.087 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:06:45.088 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:06:45.270 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:06:45.297 INFO:teuthology.orchestra.run.smithi066.stdout:libcephfs-devel-19.0.0-1550.g633e69c9.el9.x86_6 128 kB/s | 31 kB 00:00 2024-03-04T06:06:45.298 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:45.298 INFO:teuthology.orchestra.run.smithi066.stdout:Total 127 kB/s | 31 kB 00:00 2024-03-04T06:06:45.298 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:45.302 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:45.302 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:45.329 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:45.329 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:45.416 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-04T06:06:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:45.494 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 2/2 2024-03-04T06:06:45.544 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:45.917 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:46.194 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:46.194 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:46.195 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:46.195 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:46.195 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:46.195 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:46.195 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 2/2 2024-03-04T06:06:46.195 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 1/2 2024-03-04T06:06:46.405 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados2 2024-03-04T06:06:46.539 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-04T06:06:46.539 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:46.539 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:06:46.539 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-04T06:06:46.539 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:46.539 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:46.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-03-04T06:06:46.997 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:17 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:47.117 INFO:teuthology.orchestra.run.smithi066.stdout:Package librados2-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:47.173 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:47.174 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-04T06:06:47.174 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:47.249 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librbd1 2024-03-04T06:06:47.376 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:31 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:47.555 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:47.555 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:47.555 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-04T06:06:47.556 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:47.556 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:06:47.556 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.0.0-1550.g633e69c9.el9 ceph-noarch 264 k 2024-03-04T06:06:47.556 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-04T06:06:47.556 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T06:06:47.556 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:47.556 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:06:47.556 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:47.556 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-03-04T06:06:47.556 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:47.557 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 332 k 2024-03-04T06:06:47.557 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.5 M 2024-03-04T06:06:47.557 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:06:47.681 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 556 kB/s | 69 kB 00:00 2024-03-04T06:06:47.838 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:18 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:47.956 INFO:teuthology.orchestra.run.smithi066.stdout:Package librbd1-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:47.964 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-volume-19.0.0-1550.g633e69c9.el9.no 648 kB/s | 264 kB 00:00 2024-03-04T06:06:47.966 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:47.966 INFO:teuthology.orchestra.run.smithi064.stdout:Total 811 kB/s | 332 kB 00:00 2024-03-04T06:06:47.966 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:06:47.975 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:06:47.975 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:06:48.012 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:48.013 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-04T06:06:48.013 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:48.017 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:06:48.017 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:06:48.097 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rados 2024-03-04T06:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:06:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-04T06:06:48.459 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.0.0-1550.g633e69c9.el9.noarch 2/2 2024-03-04T06:06:48.475 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.0.0-1550.g633e69c9.el9.noarch 2/2 2024-03-04T06:06:48.475 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:06:48.475 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T06:06:48.475 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:48.692 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:19 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:48.811 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rados-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:48.868 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:48.868 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-04T06:06:48.869 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:48.939 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rgw 2024-03-04T06:06:49.249 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.0.0-1550.g633e69c9.el9.noarch 1/2 2024-03-04T06:06:49.531 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:20 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-04T06:06:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:49.582 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:06:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.0.0-1550.g633e69c9.el9.noarch 2024-03-04T06:06:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T06:06:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:49.582 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:49.651 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rgw-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:49.705 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:49.707 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-04T06:06:49.707 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:49.781 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-cephfs 2024-03-04T06:06:49.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-03-04T06:06:50.371 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:21 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:50.479 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:34 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:50.490 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-cephfs-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:50.545 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:50.547 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-04T06:06:50.547 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:50.631 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rbd 2024-03-04T06:06:50.652 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:50.652 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-04T06:06:50.652 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:50.653 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:06:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 127 k 2024-03-04T06:06:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:50.653 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:06:50.653 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:50.653 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-04T06:06:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:50.654 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 127 k 2024-03-04T06:06:50.654 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 457 k 2024-03-04T06:06:50.654 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:06:51.196 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-19.0.0-1550.g633e69c9.el9.x86_64 234 kB/s | 127 kB 00:00 2024-03-04T06:06:51.197 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:51.197 INFO:teuthology.orchestra.run.smithi064.stdout:Total 233 kB/s | 127 kB 00:00 2024-03-04T06:06:51.197 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:06:51.203 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:06:51.203 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:06:51.227 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:22 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:51.261 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:06:51.261 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:06:51.347 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rbd-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:51.402 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:51.403 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-04T06:06:51.403 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:51.412 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:06:51.470 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-fuse 2024-03-04T06:06:51.522 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:51.978 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:52.060 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:23 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:52.234 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:52.235 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:52.235 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-04T06:06:52.235 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:52.235 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:52.235 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 86 k 2024-03-04T06:06:52.235 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:52.235 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:52.235 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:52.235 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-04T06:06:52.236 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:52.236 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 86 k 2024-03-04T06:06:52.236 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 256 k 2024-03-04T06:06:52.236 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:52.311 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:52.312 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:06:52.312 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:52.312 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:52.312 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:52.443 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-fuse-19.0.0-1550.g633e69c9.el9.x86_64.rpm 416 kB/s | 86 kB 00:00 2024-03-04T06:06:52.443 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:52.443 INFO:teuthology.orchestra.run.smithi066.stdout:Total 413 kB/s | 86 kB 00:00 2024-03-04T06:06:52.443 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:52.451 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:52.451 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:52.509 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:52.509 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:52.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-03-04T06:06:52.662 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:52.752 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:53.155 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:53.175 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:37 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:53.293 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:53.349 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:53.350 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-04T06:06:53.350 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:53.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-03-04T06:06:53.497 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:53.497 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:53.497 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:53.497 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:53.497 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:53.497 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:53.753 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-mirror 2024-03-04T06:06:53.987 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:37 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:54.160 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:54.161 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:54.161 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-04T06:06:54.161 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:54.161 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:06:54.161 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 31 k 2024-03-04T06:06:54.161 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:54.161 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:06:54.161 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:06:54.162 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-04T06:06:54.162 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:54.162 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 31 k 2024-03-04T06:06:54.162 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 140 k 2024-03-04T06:06:54.162 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:06:54.346 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:25 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:54.409 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-19.0.0-1550.g633e69c9.el9.x86_6 126 kB/s | 31 kB 00:00 2024-03-04T06:06:54.410 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:54.410 INFO:teuthology.orchestra.run.smithi064.stdout:Total 125 kB/s | 31 kB 00:00 2024-03-04T06:06:54.410 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:06:54.414 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:06:54.414 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:06:54.442 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:06:54.442 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:06:54.522 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:54.523 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:54.523 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-04T06:06:54.523 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:54.523 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:54.523 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 3.0 M 2024-03-04T06:06:54.523 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:54.523 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:54.523 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:54.523 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-04T06:06:54.523 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:54.524 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 3.0 M 2024-03-04T06:06:54.524 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 14 M 2024-03-04T06:06:54.524 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:54.532 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:06:54.669 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:55.059 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:55.104 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-mirror-19.0.0-1550.g633e69c9.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-04T06:06:55.104 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:55.105 INFO:teuthology.orchestra.run.smithi066.stdout:Total 5.2 MB/s | 3.0 MB 00:00 2024-03-04T06:06:55.105 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:55.115 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:55.115 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:55.192 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:55.192 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:55.348 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:55.348 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:55.348 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:06:55.348 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:55.348 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:06:55.348 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:55.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-03-04T06:06:55.646 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:55.721 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-mirror-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:55.747 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:55.747 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:06:55.747 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T06:06:55.747 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T06:06:55.747 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T06:06:55.747 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:56.132 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:40 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:56.252 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:56.307 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:56.308 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-04T06:06:56.308 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:56.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-03-04T06:06:56.937 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:40 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:56.979 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-mirror-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:56.979 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:56.979 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:56.979 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:56.979 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:56.979 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:57.055 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:57.111 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:57.112 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-04T06:06:57.112 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:57.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-03-04T06:06:57.206 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-nbd 2024-03-04T06:06:57.746 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:41 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:57.800 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:28 ago on Mon 04 Mar 2024 06:03:29 AM UTC. 2024-03-04T06:06:57.864 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:57.921 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:57.922 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-04T06:06:57.922 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:57.974 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 169 k 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 169 k 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 540 k 2024-03-04T06:06:57.975 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-04T06:06:57.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-03-04T06:06:58.208 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-nbd-19.0.0-1550.g633e69c9.el9.x86_64.rpm 727 kB/s | 169 kB 00:00 2024-03-04T06:06:58.209 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:06:58.209 INFO:teuthology.orchestra.run.smithi066.stdout:Total 722 kB/s | 169 kB 00:00 2024-03-04T06:06:58.209 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-04T06:06:58.218 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-04T06:06:58.218 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-04T06:06:58.277 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-04T06:06:58.277 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-04T06:06:58.447 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-04T06:06:58.553 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-nbd-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:58.555 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:42 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:58.673 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:58.727 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:58.728 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-04T06:06:58.728 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:58.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-03-04T06:06:59.006 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-nbd-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:06:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:59.355 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-04T06:06:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:06:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:06:59.355 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-04T06:06:59.359 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:43 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:06:59.479 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:06:59.534 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:06:59.536 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-04T06:06:59.536 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:06:59.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-03-04T06:06:59.608 DEBUG:teuthology.parallel:result is None 2024-03-04T06:07:00.171 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:44 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:07:00.291 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:19.0.0-1550.g633e69c9.el9.x86_64 is already installed. 2024-03-04T06:07:00.347 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:07:00.348 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-04T06:07:00.348 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:07:00.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-03-04T06:07:00.979 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:44 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:07:01.155 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:07:01.156 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:07:01.156 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-04T06:07:01.156 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:07:01.156 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:07:01.156 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 86 k 2024-03-04T06:07:01.156 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:01.156 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:07:01.156 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:07:01.156 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-04T06:07:01.156 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:01.157 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 86 k 2024-03-04T06:07:01.157 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 256 k 2024-03-04T06:07:01.157 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:07:01.371 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-19.0.0-1550.g633e69c9.el9.x86_64.rpm 401 kB/s | 86 kB 00:00 2024-03-04T06:07:01.371 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:07:01.372 INFO:teuthology.orchestra.run.smithi064.stdout:Total 399 kB/s | 86 kB 00:00 2024-03-04T06:07:01.372 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:07:01.379 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:07:01.379 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:07:01.436 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:07:01.437 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:07:01.585 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:07:01.672 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:07:02.113 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:07:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:07:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:02.442 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:07:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:07:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:02.442 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:07:02.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-03-04T06:07:03.271 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:47 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:07:03.446 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:07:03.447 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:07:03.447 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-04T06:07:03.448 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:07:03.448 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:07:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 3.0 M 2024-03-04T06:07:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:03.448 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:07:03.448 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:07:03.448 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-04T06:07:03.448 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:03.448 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.0 M 2024-03-04T06:07:03.448 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 14 M 2024-03-04T06:07:03.448 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:07:04.024 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-19.0.0-1550.g633e69c9.el9.x86_64.rpm 5.3 MB/s | 3.0 MB 00:00 2024-03-04T06:07:04.024 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:07:04.025 INFO:teuthology.orchestra.run.smithi064.stdout:Total 5.3 MB/s | 3.0 MB 00:00 2024-03-04T06:07:04.025 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:07:04.035 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:07:04.035 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:07:04.110 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:07:04.110 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:07:04.563 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:07:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:07:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:07:04.658 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T06:07:04.658 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T06:07:04.658 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T06:07:04.658 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T06:07:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:05.788 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:07:05.788 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:05.788 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:07:05.788 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:07:05.788 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:05.789 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:07:06.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-03-04T06:07:06.629 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:50 ago on Mon 04 Mar 2024 06:03:16 AM UTC. 2024-03-04T06:07:06.805 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T06:07:06.806 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:07:06.806 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-04T06:07:06.806 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:07:06.806 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T06:07:06.806 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.0.0-1550.g633e69c9.el9 ceph 169 k 2024-03-04T06:07:06.806 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:06.806 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T06:07:06.807 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T06:07:06.807 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-04T06:07:06.807 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:06.807 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 169 k 2024-03-04T06:07:06.807 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 540 k 2024-03-04T06:07:06.807 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T06:07:07.161 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-19.0.0-1550.g633e69c9.el9.x86_64.rpm 478 kB/s | 169 kB 00:00 2024-03-04T06:07:07.162 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T06:07:07.162 INFO:teuthology.orchestra.run.smithi064.stdout:Total 475 kB/s | 169 kB 00:00 2024-03-04T06:07:07.162 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T06:07:07.171 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T06:07:07.171 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T06:07:07.230 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T06:07:07.230 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T06:07:07.387 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T06:07:07.488 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:07:07.953 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:07:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.0.0-1550.g633e69c9.el9.x86_64 1/1 2024-03-04T06:07:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:08.245 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T06:07:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.0.0-1550.g633e69c9.el9.x86_64 2024-03-04T06:07:08.245 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:08.245 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T06:07:08.491 DEBUG:teuthology.parallel:result is None 2024-03-04T06:07:08.491 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd 2024-03-04T06:07:08.712 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T06:07:08.733 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-1550.g633e69c9.el9 2024-03-04T06:07:08.734 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1550.g633e69c9.el9 2024-03-04T06:07:08.734 INFO:teuthology.task.install:The correct ceph version 19.0.0-1550.g633e69c9 is installed. 2024-03-04T06:07:08.735 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd 2024-03-04T06:07:08.874 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T06:07:08.900 INFO:teuthology.orchestra.run.smithi066.stdout:19.0.0-1550.g633e69c9.el9 2024-03-04T06:07:08.900 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1550.g633e69c9.el9 2024-03-04T06:07:08.900 INFO:teuthology.task.install:The correct ceph version 19.0.0-1550.g633e69c9 is installed. 2024-03-04T06:07:08.901 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-04T06:07:08.902 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:08.902 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T06:07:08.931 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:08.931 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T06:07:08.977 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-04T06:07:08.977 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:08.977 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T06:07:09.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T06:07:09.072 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:09.072 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T06:07:09.106 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T06:07:09.179 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-04T06:07:09.180 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:09.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T06:07:09.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T06:07:09.274 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:09.274 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T06:07:09.307 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T06:07:09.381 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-04T06:07:09.382 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:09.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T06:07:09.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T06:07:09.475 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:09.475 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T06:07:09.507 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T06:07:09.580 INFO:teuthology.run_tasks:Running task ceph... 2024-03-04T06:07:09.658 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-04T06:07:09.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-03-04T06:07:09.660 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 777 /var/log/ceph 2024-03-04T06:07:09.692 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-04T06:07:09.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-04T06:07:09.727 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-04T06:07:09.762 INFO:tasks.ceph:Creating extra log directories... 2024-03-04T06:07:09.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-04T06:07:09.794 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-04T06:07:09.842 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-04T06:07:09.842 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-04T06:07:09.843 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638', 'branch': 'wip-nitzan-crimson-shard-pg-mapping-get-or-create', 'description': 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects}', 'email': None, 'first_in_suite': False, 'job_id': '7580638', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-nitzan-crimson-shard-pg-mapping-get-or-create'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-nitzan-crimson-shard-pg-mapping-get-or-create', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd'}}, 'owner': 'scheduled_nmordech@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 4836, 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd', 'sleep_before_teardown': 0, 'suite': 'crimson-rados:thrash', 'suite_branch': 'wip-nitzan-crimson-shard-pg-mapping-get-or-create', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlsRiv6BZzoFANvoD9juHdrbxi3MXvw3zBu/1yDTuv1+gmkPLYJCr7XgsZ+YtJJj4DCcQADwtCwRXr5hwaIUMQ=', 'smithi066.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMFZcv76CLd2CM3w5dPYjbX7QYeAj/jEaNR1nJSu76qsrrYM794w1HuRY73Z3wK0f7ezvi7iPqh582AZT/KuZOk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-03-04_05:47:36', 'tube': 'smithi', 'user': 'nmordech', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-04T06:07:09.843 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-04T06:07:09.860 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-04T06:07:09.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T06:07:09.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T06:07:09.962 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:09.962 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T06:07:09.992 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T06:07:09.993 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-03-04T06:07:10.046 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T06:07:10.047 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T06:07:10.047 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-04T06:07:10.047 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T06:07:10.047 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T06:07:10.047 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 06:07:04.918639927 +0000 2024-03-04T06:07:10.047 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 06:02:05.879495449 +0000 2024-03-04T06:07:10.047 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 06:02:05.879495449 +0000 2024-03-04T06:07:10.047 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 06:02:05.879495449 +0000 2024-03-04T06:07:10.047 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T06:07:10.111 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T06:07:10.111 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T06:07:10.112 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000177945 s, 2.9 MB/s 2024-03-04T06:07:10.112 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T06:07:10.167 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-03-04T06:07:10.221 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T06:07:10.221 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T06:07:10.221 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-04T06:07:10.222 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T06:07:10.222 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T06:07:10.222 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 06:07:04.918639927 +0000 2024-03-04T06:07:10.222 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 06:02:06.225488287 +0000 2024-03-04T06:07:10.222 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 06:02:06.225488287 +0000 2024-03-04T06:07:10.222 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 06:02:06.225488287 +0000 2024-03-04T06:07:10.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T06:07:10.286 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T06:07:10.286 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T06:07:10.286 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000178531 s, 2.9 MB/s 2024-03-04T06:07:10.288 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T06:07:10.342 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-03-04T06:07:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T06:07:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T06:07:10.397 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-04T06:07:10.397 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T06:07:10.397 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T06:07:10.397 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 06:07:04.918639927 +0000 2024-03-04T06:07:10.397 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 06:02:06.564481269 +0000 2024-03-04T06:07:10.397 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 06:02:06.564481269 +0000 2024-03-04T06:07:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 06:02:06.564481269 +0000 2024-03-04T06:07:10.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T06:07:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T06:07:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T06:07:10.462 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000170459 s, 3.0 MB/s 2024-03-04T06:07:10.464 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T06:07:10.518 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-03-04T06:07:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T06:07:10.573 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T06:07:10.574 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-04T06:07:10.574 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T06:07:10.574 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T06:07:10.574 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 06:07:04.919639907 +0000 2024-03-04T06:07:10.574 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 06:02:07.005472140 +0000 2024-03-04T06:07:10.574 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 06:02:07.005472140 +0000 2024-03-04T06:07:10.574 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 06:02:07.005472140 +0000 2024-03-04T06:07:10.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T06:07:10.638 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T06:07:10.638 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T06:07:10.638 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000174881 s, 2.9 MB/s 2024-03-04T06:07:10.639 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T06:07:10.694 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-03-04T06:07:10.695 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:10.695 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T06:07:10.712 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T06:07:10.713 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_1 2024-03-04T06:07:10.769 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T06:07:10.769 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T06:07:10.769 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-04T06:07:10.769 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T06:07:10.769 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T06:07:10.769 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-04 06:06:56.028165745 +0000 2024-03-04T06:07:10.769 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-04 06:02:08.424009261 +0000 2024-03-04T06:07:10.769 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-04 06:02:08.424009261 +0000 2024-03-04T06:07:10.769 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-04 06:02:08.424009261 +0000 2024-03-04T06:07:10.770 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T06:07:10.841 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-04T06:07:10.841 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-04T06:07:10.841 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000279007 s, 1.8 MB/s 2024-03-04T06:07:10.842 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T06:07:10.899 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_2 2024-03-04T06:07:10.955 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T06:07:10.955 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T06:07:10.956 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-04T06:07:10.956 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T06:07:10.956 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T06:07:10.956 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-04 06:06:56.028165745 +0000 2024-03-04T06:07:10.956 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-04 06:02:08.786006388 +0000 2024-03-04T06:07:10.956 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-04 06:02:08.786006388 +0000 2024-03-04T06:07:10.956 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-04 06:02:08.786006388 +0000 2024-03-04T06:07:10.956 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T06:07:11.026 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-04T06:07:11.026 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-04T06:07:11.027 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000203545 s, 2.5 MB/s 2024-03-04T06:07:11.027 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T06:07:11.084 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_3 2024-03-04T06:07:11.139 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T06:07:11.140 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T06:07:11.140 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-04T06:07:11.140 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T06:07:11.140 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T06:07:11.140 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-04 06:06:56.028165745 +0000 2024-03-04T06:07:11.140 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-04 06:02:09.136003610 +0000 2024-03-04T06:07:11.140 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-04 06:02:09.136003610 +0000 2024-03-04T06:07:11.140 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-04 06:02:09.136003610 +0000 2024-03-04T06:07:11.140 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T06:07:11.209 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-04T06:07:11.209 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-04T06:07:11.209 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000181409 s, 2.8 MB/s 2024-03-04T06:07:11.210 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T06:07:11.267 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_4 2024-03-04T06:07:11.321 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T06:07:11.321 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T06:07:11.321 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-04T06:07:11.321 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T06:07:11.321 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T06:07:11.321 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-04 06:06:56.029165737 +0000 2024-03-04T06:07:11.322 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-04 06:02:09.476000912 +0000 2024-03-04T06:07:11.322 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-04 06:02:09.476000912 +0000 2024-03-04T06:07:11.322 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-04 06:02:09.476000912 +0000 2024-03-04T06:07:11.322 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T06:07:11.390 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-04T06:07:11.390 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-04T06:07:11.390 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000184337 s, 2.8 MB/s 2024-03-04T06:07:11.391 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T06:07:11.447 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-03-04T06:07:11.447 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-03-04T06:07:11.447 INFO:tasks.ceph:Generating config... 2024-03-04T06:07:11.448 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-04T06:07:11.448 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-04T06:07:11.448 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-04T06:07:11.448 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] err to stderr = True 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] log flush on exit = True 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] log to file = False 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] log to stderr = True 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-04T06:07:11.449 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-04T06:07:11.450 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-04T06:07:11.450 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-03-04T06:07:11.450 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-03-04T06:07:11.450 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-04T06:07:11.450 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-03-04T06:07:11.450 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-04T06:07:11.450 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-04T06:07:11.450 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-03-04T06:07:11.450 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-03-04T06:07:11.450 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-04T06:07:11.450 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-03-04T06:07:11.450 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-03-04T06:07:11.450 INFO:tasks.ceph:[global] ms client mode = crc 2024-03-04T06:07:11.450 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-03-04T06:07:11.450 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-03-04T06:07:11.450 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-03-04T06:07:11.451 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-03-04T06:07:11.451 INFO:tasks.ceph:[global] ms service mode = crc 2024-03-04T06:07:11.451 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-03-04T06:07:11.451 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-03-04T06:07:11.451 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-04T06:07:11.451 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-04T06:07:11.451 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-04T06:07:11.451 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-04T06:07:11.451 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-04T06:07:11.451 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-04T06:07:11.452 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-04T06:07:11.452 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-04T06:07:11.452 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-04T06:07:11.452 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-03-04T06:07:11.452 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-04T06:07:11.452 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-03-04T06:07:11.452 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-03-04T06:07:11.452 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-03-04T06:07:11.452 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-03-04T06:07:11.452 INFO:tasks.ceph:Setting up mon.a... 2024-03-04T06:07:11.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-04T06:07:11.514 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.keyring 2024-03-04T06:07:11.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-04T06:07:11.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-04T06:07:11.596 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.64')] 2024-03-04T06:07:11.596 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.64', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-03-04T06:07:11.597 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:11.597 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T06:07:11.650 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.64 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-04T06:07:11.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:11.734+0000 7f07f2a71e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:11.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:11.734+0000 7f07f2a71e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:11.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:11.734+0000 7f07f2a71e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:11.735 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-04T06:07:11.735 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid cb66367d-3032-46d0-abf8-73af48df9eb9 2024-03-04T06:07:11.736 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-03-04T06:07:11.736 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2024-03-04T06:07:11.736 INFO:teuthology.orchestra.run.smithi064.stdout:fsid cb66367d-3032-46d0-abf8-73af48df9eb9 2024-03-04T06:07:11.736 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2024-03-04T06:07:11.735788+0000 2024-03-04T06:07:11.736 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-03-04T06:07:11.735788+0000 2024-03-04T06:07:11.736 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 17 (quincy) 2024-03-04T06:07:11.736 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 1 2024-03-04T06:07:11.736 INFO:teuthology.orchestra.run.smithi064.stdout:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2024-03-04T06:07:11.736 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-03-04T06:07:11.737 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T06:07:11.751 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cb66367d-3032-46d0-abf8-73af48df9eb9... 2024-03-04T06:07:11.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T06:07:11.794 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.841 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-03-04T06:07:11.842 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = 172.21.15.64 2024-03-04T06:07:11.843 INFO:teuthology.orchestra.run.smithi064.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: ms client mode = crc 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: ms cluster mode = crc 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: ms mon client mode = crc 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: ms mon cluster mode = crc 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: ms mon service mode = crc 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: ms service mode = crc 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_min_size = 2 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_size = 3 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = cb66367d-3032-46d0-abf8-73af48df9eb9 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 120 2024-03-04T06:07:11.844 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.845 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression algorithm = snappy 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression mode = aggressive 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore rocksdb cf = False 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: crimson alien thread cpu cores = 6-7 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: crimson osd obc lru size = 10 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 20 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 20 2024-03-04T06:07:11.846 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: err to stderr = True 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: log flush on exit = True 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: log to file = False 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: log to stderr = True 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: osd blocked scrub grace period = 3600 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug reject backfill probability = 0.3 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: osd delete sleep = 1 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: osd max backfills = 3 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: osd max markdown count = 1000 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-03-04T06:07:11.847 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub min interval = 60 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: osd shutdown pgref assert = True 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: osd snap trim sleep = 2 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-03-04T06:07:11.848 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-03-04T06:07:11.849 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: mon min osdmap epochs = 50 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune interval = 2 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune min = 15 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune txsize = 2 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default crimson = True 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: paxos service trim min = 10 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-03-04T06:07:11.850 INFO:teuthology.orchestra.run.smithi064.stdout: crimson seastar cpu cores = 0-2 2024-03-04T06:07:11.851 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-03-04T06:07:11.851 INFO:teuthology.orchestra.run.smithi064.stdout: crimson seastar cpu cores = 3-5 2024-03-04T06:07:11.851 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-03-04T06:07:11.851 INFO:teuthology.orchestra.run.smithi064.stdout: crimson seastar cpu cores = 0-2 2024-03-04T06:07:11.851 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-03-04T06:07:11.851 INFO:teuthology.orchestra.run.smithi064.stdout: crimson seastar cpu cores = 3-5 2024-03-04T06:07:11.855 INFO:teuthology.orchestra.run.smithi066.stdout:[global] 2024-03-04T06:07:11.855 INFO:teuthology.orchestra.run.smithi066.stdout: chdir = "" 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: auth supported = cephx 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: filestore xattr use omap = true 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: mon clock drift allowed = 1.000 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: osd crush chooseleaf type = 0 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: auth debug = true 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on old message = true 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on bug = true 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: mon pg warn max object skew = 0 2024-03-04T06:07:11.856 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default size = 2 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow primary affinity = true 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow pg remap = true 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on legacy crush tunables = false 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on crush straw calc version zero = false 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on no sortbitwise = false 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on osd down out interval zero = false 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on too few osds = false 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: mon_allow_pool_size_one = true 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.857 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: osd default data pool replay window = 5 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: mon allow pool delete = true 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: mon cluster log file level = debug 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: debug asserts on shutdown = true 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: mon health detail to clog = false 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: mon host = 172.21.15.64 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: ms client mode = crc 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: ms cluster mode = crc 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: ms mon client mode = crc 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: ms mon cluster mode = crc 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: ms mon service mode = crc 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: ms service mode = crc 2024-03-04T06:07:11.858 INFO:teuthology.orchestra.run.smithi066.stdout: osd_pool_default_min_size = 2 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: osd_pool_default_size = 3 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: fsid = cb66367d-3032-46d0-abf8-73af48df9eb9 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout:[osd] 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: osd journal size = 100 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub load threshold = 5.0 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub max interval = 120 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: osd mclock profile = high_recovery_ops 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: osd recover clone overlap = true 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: osd recovery max chunk = 1048576 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug shutdown = true 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug op order = true 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug verify stray on activate = true 2024-03-04T06:07:11.859 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug trim objects = true 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: osd open classes on start = true 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug pg log writeout = true 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: osd deep scrub update digest min age = 30 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: osd map max advance = 10 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: journal zero on create = true 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: filestore ondisk finisher threads = 3 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: filestore apply finisher threads = 3 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: bdev debug aio = true 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug misdirected ops = true 2024-03-04T06:07:11.860 INFO:teuthology.orchestra.run.smithi066.stdout: debug monc = 20 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore block size = 96636764160 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore compression algorithm = snappy 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore compression mode = aggressive 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore fsck on mount = True 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore rocksdb cf = False 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: crimson alien thread cpu cores = 6-7 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: crimson osd obc lru size = 10 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: debug bluefs = 20 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: debug bluestore = 20 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: debug osd = 20 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: debug rocksdb = 10 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: err to stderr = True 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: log flush on exit = True 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: log to file = False 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: log to stderr = True 2024-03-04T06:07:11.861 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd full ratio = 0.9 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd nearfull ratio = 0.8 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: osd blocked scrub grace period = 3600 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug reject backfill probability = 0.3 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: osd delete sleep = 1 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: osd failsafe full ratio = 0.95 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: osd max backfills = 3 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: osd max markdown count = 1000 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: osd objectstore = bluestore 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub min interval = 60 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: osd shutdown pgref assert = True 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: osd snap trim sleep = 2 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout:[mgr] 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: debug mgr = 20 2024-03-04T06:07:11.862 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: mgr/telemetry/nag = false 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout:[mon] 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: debug paxos = 20 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: mon data avail warn = 5 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: mon mgr mkfs grace = 240 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd reporter subtree level = osd 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd prime pg temp = true 2024-03-04T06:07:11.863 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: # 1m isn't quite enough 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: mon_down_mkfs_grace = 2m 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_filestore_osds = false 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: mon min osdmap epochs = 50 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: mon osdmap full prune interval = 2 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: mon osdmap full prune min = 15 2024-03-04T06:07:11.864 INFO:teuthology.orchestra.run.smithi066.stdout: mon osdmap full prune txsize = 2 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default crimson = True 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: paxos service trim min = 10 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout:[client] 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: rgw cache enabled = true 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable ops log = true 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable usage log = true 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout:[mon.a] 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.0] 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: crimson seastar cpu cores = 0-2 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.1] 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: crimson seastar cpu cores = 3-5 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.2] 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: crimson seastar cpu cores = 0-2 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.3] 2024-03-04T06:07:11.865 INFO:teuthology.orchestra.run.smithi066.stdout: crimson seastar cpu cores = 3-5 2024-03-04T06:07:11.873 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-04T06:07:11.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-04T06:07:11.934 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-04T06:07:11.934 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:11.934 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-04T06:07:11.948 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:11.948 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-04T06:07:12.002 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-03-04T06:07:12.002 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:12.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-04T06:07:12.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-04T06:07:12.081 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:12.081 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-04T06:07:12.136 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi066.front.sepia.ceph.com 2024-03-04T06:07:12.136 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:12.136 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-04T06:07:12.136 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-04T06:07:12.183 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:12.183 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-04T06:07:12.237 INFO:tasks.ceph:Setting up mon nodes... 2024-03-04T06:07:12.237 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-04T06:07:12.238 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-04T06:07:12.352 INFO:teuthology.orchestra.run.smithi066.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-04T06:07:12.355 INFO:tasks.ceph:Setting up mds nodes... 2024-03-04T06:07:12.355 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-04T06:07:12.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-04T06:07:12.403 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-04T06:07:12.419 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-04T06:07:12.471 INFO:teuthology.orchestra.run.smithi066.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-04T06:07:12.490 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-04T06:07:12.490 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-03-04T06:07:12.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-04T06:07:12.517 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-03-04T06:07:12.517 INFO:tasks.ceph:role: osd.0 2024-03-04T06:07:12.517 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-04T06:07:12.517 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-04T06:07:12.591 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T06:07:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T06:07:12.591 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T06:07:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T06:07:12.592 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T06:07:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-04T06:07:12.592 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T06:07:12.592 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T06:07:12.592 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T06:07:12.592 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T06:07:13.103 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-04T06:07:13.105 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-04T06:07:13.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-04T06:07:13.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-04T06:07:13.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-04T06:07:13.360 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-03-04T06:07:13.360 INFO:tasks.ceph:role: osd.1 2024-03-04T06:07:13.360 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-04T06:07:13.360 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-04T06:07:13.436 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T06:07:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T06:07:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T06:07:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T06:07:13.436 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T06:07:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-04T06:07:13.436 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T06:07:13.436 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T06:07:13.436 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T06:07:13.437 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T06:07:13.883 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-04T06:07:13.885 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-04T06:07:13.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-04T06:07:13.979 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-04T06:07:14.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T06:07:14.554 INFO:teuthology.orchestra.run.smithi064.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-04T06:07:14.555 INFO:teuthology.orchestra.run.smithi064.stderr:INFO 2024-03-04 06:07:14,556 seastar - Reactor backend: linux-aio 2024-03-04T06:07:14.574 INFO:teuthology.orchestra.run.smithi064.stderr:==28599==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-04T06:07:14.580 INFO:teuthology.orchestra.run.smithi064.stderr:==28599==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff93d1d000; bottom 0x7fcfca1be000; size: 0x002fc9b5f000 (205247606784) 2024-03-04T06:07:14.581 INFO:teuthology.orchestra.run.smithi064.stderr:False positive error reports may follow 2024-03-04T06:07:14.581 INFO:teuthology.orchestra.run.smithi064.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-03-04T06:07:14.620 INFO:teuthology.orchestra.run.smithi064.stderr:INFO 2024-03-04 06:07:14,621 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-03-04T06:07:14.621 INFO:teuthology.orchestra.run.smithi064.stderr:INFO 2024-03-04 06:07:14,621 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-03-04T06:07:14.621 INFO:teuthology.orchestra.run.smithi064.stderr:DEBUG 2024-03-04 06:07:14,621 [shard 0:main] seastar - reactor::drain 2024-03-04T06:07:15.112 INFO:teuthology.orchestra.run.smithi064.stderr:----------------------------------------------------- 2024-03-04T06:07:15.113 INFO:teuthology.orchestra.run.smithi064.stderr:Suppressions used: 2024-03-04T06:07:15.113 INFO:teuthology.orchestra.run.smithi064.stderr: count bytes template 2024-03-04T06:07:15.113 INFO:teuthology.orchestra.run.smithi064.stderr: 1 8 MallocExtension::Register 2024-03-04T06:07:15.113 INFO:teuthology.orchestra.run.smithi064.stderr:----------------------------------------------------- 2024-03-04T06:07:15.113 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-04T06:07:15.163 INFO:teuthology.orchestra.run.smithi064.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-04T06:07:15.165 INFO:teuthology.orchestra.run.smithi064.stderr:INFO 2024-03-04 06:07:15,165 seastar - Reactor backend: linux-aio 2024-03-04T06:07:15.193 INFO:teuthology.orchestra.run.smithi064.stderr:==28597==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-04T06:07:15.199 INFO:teuthology.orchestra.run.smithi064.stderr:==28597==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff93d1d000; bottom 0x7fcfc7fa9000; size: 0x002fcbd74000 (205283344384) 2024-03-04T06:07:15.199 INFO:teuthology.orchestra.run.smithi064.stderr:False positive error reports may follow 2024-03-04T06:07:15.199 INFO:teuthology.orchestra.run.smithi064.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-03-04T06:07:19.235 INFO:teuthology.orchestra.run.smithi064.stderr:----------------------------------------------------- 2024-03-04T06:07:19.235 INFO:teuthology.orchestra.run.smithi064.stderr:Suppressions used: 2024-03-04T06:07:19.236 INFO:teuthology.orchestra.run.smithi064.stderr: count bytes template 2024-03-04T06:07:19.236 INFO:teuthology.orchestra.run.smithi064.stderr: 1 8 MallocExtension::Register 2024-03-04T06:07:19.236 INFO:teuthology.orchestra.run.smithi064.stderr:----------------------------------------------------- 2024-03-04T06:07:19.236 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-04T06:07:19.236 INFO:teuthology.orchestra.run.smithi064.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid cb66367d-3032-46d0-abf8-73af48df9eb9 2024-03-04T06:07:19.276 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-04T06:07:19.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T06:07:19.829 INFO:teuthology.orchestra.run.smithi064.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-04T06:07:19.829 INFO:teuthology.orchestra.run.smithi064.stderr:INFO 2024-03-04 06:07:19,829 seastar - Reactor backend: linux-aio 2024-03-04T06:07:19.847 INFO:teuthology.orchestra.run.smithi064.stderr:==28781==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-04T06:07:19.853 INFO:teuthology.orchestra.run.smithi064.stderr:==28781==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcfd26e000; bottom 0x7f5466af1000; size: 0x00a89677d000 (724078940160) 2024-03-04T06:07:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:False positive error reports may follow 2024-03-04T06:07:19.854 INFO:teuthology.orchestra.run.smithi064.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-03-04T06:07:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:INFO 2024-03-04 06:07:19,893 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-03-04T06:07:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:INFO 2024-03-04 06:07:19,894 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-03-04T06:07:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:DEBUG 2024-03-04 06:07:19,894 [shard 0:main] seastar - reactor::drain 2024-03-04T06:07:20.395 INFO:teuthology.orchestra.run.smithi064.stderr:----------------------------------------------------- 2024-03-04T06:07:20.395 INFO:teuthology.orchestra.run.smithi064.stderr:Suppressions used: 2024-03-04T06:07:20.395 INFO:teuthology.orchestra.run.smithi064.stderr: count bytes template 2024-03-04T06:07:20.395 INFO:teuthology.orchestra.run.smithi064.stderr: 1 8 MallocExtension::Register 2024-03-04T06:07:20.395 INFO:teuthology.orchestra.run.smithi064.stderr:----------------------------------------------------- 2024-03-04T06:07:20.395 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-04T06:07:20.444 INFO:teuthology.orchestra.run.smithi064.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-04T06:07:20.446 INFO:teuthology.orchestra.run.smithi064.stderr:INFO 2024-03-04 06:07:20,446 seastar - Reactor backend: linux-aio 2024-03-04T06:07:20.482 INFO:teuthology.orchestra.run.smithi064.stderr:==28779==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-04T06:07:20.488 INFO:teuthology.orchestra.run.smithi064.stderr:==28779==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcfd26e000; bottom 0x7f54648dc000; size: 0x00a898992000 (724114677760) 2024-03-04T06:07:20.488 INFO:teuthology.orchestra.run.smithi064.stderr:False positive error reports may follow 2024-03-04T06:07:20.488 INFO:teuthology.orchestra.run.smithi064.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-03-04T06:07:24.524 INFO:teuthology.orchestra.run.smithi064.stderr:----------------------------------------------------- 2024-03-04T06:07:24.524 INFO:teuthology.orchestra.run.smithi064.stderr:Suppressions used: 2024-03-04T06:07:24.524 INFO:teuthology.orchestra.run.smithi064.stderr: count bytes template 2024-03-04T06:07:24.524 INFO:teuthology.orchestra.run.smithi064.stderr: 1 8 MallocExtension::Register 2024-03-04T06:07:24.524 INFO:teuthology.orchestra.run.smithi064.stderr:----------------------------------------------------- 2024-03-04T06:07:24.524 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-04T06:07:24.525 INFO:teuthology.orchestra.run.smithi064.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid cb66367d-3032-46d0-abf8-73af48df9eb9 2024-03-04T06:07:24.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-04T06:07:24.591 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-04T06:07:24.628 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-03-04T06:07:24.628 INFO:tasks.ceph:role: osd.2 2024-03-04T06:07:24.628 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-04T06:07:24.628 DEBUG:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-04T06:07:24.711 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T06:07:24.711 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T06:07:24.711 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T06:07:24.711 INFO:teuthology.orchestra.run.smithi066.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T06:07:24.711 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T06:07:24.711 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2024-03-04T06:07:24.711 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T06:07:24.711 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T06:07:24.711 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T06:07:24.711 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T06:07:25.229 INFO:teuthology.orchestra.run.smithi066.stdout:Discarding blocks...Done. 2024-03-04T06:07:25.231 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2024-03-04T06:07:25.231 DEBUG:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-03-04T06:07:25.391 DEBUG:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-04T06:07:25.432 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-04T06:07:25.507 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-03-04T06:07:25.507 INFO:tasks.ceph:role: osd.3 2024-03-04T06:07:25.507 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-04T06:07:25.507 DEBUG:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-04T06:07:25.587 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T06:07:25.587 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T06:07:25.587 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T06:07:25.587 INFO:teuthology.orchestra.run.smithi066.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T06:07:25.588 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T06:07:25.588 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2024-03-04T06:07:25.588 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T06:07:25.588 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T06:07:25.588 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T06:07:25.588 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T06:07:26.040 INFO:teuthology.orchestra.run.smithi066.stdout:Discarding blocks...Done. 2024-03-04T06:07:26.043 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2024-03-04T06:07:26.043 DEBUG:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-03-04T06:07:26.148 DEBUG:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-04T06:07:26.186 DEBUG:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T06:07:26.773 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-04T06:07:26.774 INFO:teuthology.orchestra.run.smithi066.stderr:INFO 2024-03-04 06:07:26,774 seastar - Reactor backend: linux-aio 2024-03-04T06:07:26.792 INFO:teuthology.orchestra.run.smithi066.stderr:==28767==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-04T06:07:26.799 INFO:teuthology.orchestra.run.smithi066.stderr:==28767==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc2c118000; bottom 0x7fa61d112000; size: 0x00560f006000 (369618870272) 2024-03-04T06:07:26.799 INFO:teuthology.orchestra.run.smithi066.stderr:False positive error reports may follow 2024-03-04T06:07:26.799 INFO:teuthology.orchestra.run.smithi066.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-03-04T06:07:26.839 INFO:teuthology.orchestra.run.smithi066.stderr:INFO 2024-03-04 06:07:26,839 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-03-04T06:07:26.839 INFO:teuthology.orchestra.run.smithi066.stderr:INFO 2024-03-04 06:07:26,839 [shard 0:main] osd - get_early_config: set --cpuset 0-2 2024-03-04T06:07:26.839 INFO:teuthology.orchestra.run.smithi066.stderr:DEBUG 2024-03-04 06:07:26,840 [shard 0:main] seastar - reactor::drain 2024-03-04T06:07:27.350 INFO:teuthology.orchestra.run.smithi066.stderr:----------------------------------------------------- 2024-03-04T06:07:27.350 INFO:teuthology.orchestra.run.smithi066.stderr:Suppressions used: 2024-03-04T06:07:27.351 INFO:teuthology.orchestra.run.smithi066.stderr: count bytes template 2024-03-04T06:07:27.351 INFO:teuthology.orchestra.run.smithi066.stderr: 1 8 MallocExtension::Register 2024-03-04T06:07:27.351 INFO:teuthology.orchestra.run.smithi066.stderr:----------------------------------------------------- 2024-03-04T06:07:27.351 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-04T06:07:27.408 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-04T06:07:27.410 INFO:teuthology.orchestra.run.smithi066.stderr:INFO 2024-03-04 06:07:27,410 seastar - Reactor backend: linux-aio 2024-03-04T06:07:27.447 INFO:teuthology.orchestra.run.smithi066.stderr:==28765==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-04T06:07:27.453 INFO:teuthology.orchestra.run.smithi066.stderr:==28765==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc2c118000; bottom 0x7fa61aefd000; size: 0x00561121b000 (369654607872) 2024-03-04T06:07:27.453 INFO:teuthology.orchestra.run.smithi066.stderr:False positive error reports may follow 2024-03-04T06:07:27.453 INFO:teuthology.orchestra.run.smithi066.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-03-04T06:07:31.516 INFO:teuthology.orchestra.run.smithi066.stderr:----------------------------------------------------- 2024-03-04T06:07:31.517 INFO:teuthology.orchestra.run.smithi066.stderr:Suppressions used: 2024-03-04T06:07:31.517 INFO:teuthology.orchestra.run.smithi066.stderr: count bytes template 2024-03-04T06:07:31.517 INFO:teuthology.orchestra.run.smithi066.stderr: 1 8 MallocExtension::Register 2024-03-04T06:07:31.517 INFO:teuthology.orchestra.run.smithi066.stderr:----------------------------------------------------- 2024-03-04T06:07:31.517 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-04T06:07:31.517 INFO:teuthology.orchestra.run.smithi066.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid cb66367d-3032-46d0-abf8-73af48df9eb9 2024-03-04T06:07:31.558 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-04T06:07:31.589 DEBUG:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T06:07:32.170 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-04T06:07:32.171 INFO:teuthology.orchestra.run.smithi066.stderr:INFO 2024-03-04 06:07:32,171 seastar - Reactor backend: linux-aio 2024-03-04T06:07:32.192 INFO:teuthology.orchestra.run.smithi066.stderr:==28950==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-04T06:07:32.198 INFO:teuthology.orchestra.run.smithi066.stderr:==28950==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc8ad86000; bottom 0x7f0b893fd000; size: 0x00f101989000 (1035113893888) 2024-03-04T06:07:32.198 INFO:teuthology.orchestra.run.smithi066.stderr:False positive error reports may follow 2024-03-04T06:07:32.199 INFO:teuthology.orchestra.run.smithi066.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-03-04T06:07:32.239 INFO:teuthology.orchestra.run.smithi066.stderr:INFO 2024-03-04 06:07:32,239 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-03-04T06:07:32.240 INFO:teuthology.orchestra.run.smithi066.stderr:INFO 2024-03-04 06:07:32,239 [shard 0:main] osd - get_early_config: set --cpuset 3-5 2024-03-04T06:07:32.240 INFO:teuthology.orchestra.run.smithi066.stderr:DEBUG 2024-03-04 06:07:32,240 [shard 0:main] seastar - reactor::drain 2024-03-04T06:07:32.743 INFO:teuthology.orchestra.run.smithi066.stderr:----------------------------------------------------- 2024-03-04T06:07:32.744 INFO:teuthology.orchestra.run.smithi066.stderr:Suppressions used: 2024-03-04T06:07:32.744 INFO:teuthology.orchestra.run.smithi066.stderr: count bytes template 2024-03-04T06:07:32.744 INFO:teuthology.orchestra.run.smithi066.stderr: 1 8 MallocExtension::Register 2024-03-04T06:07:32.744 INFO:teuthology.orchestra.run.smithi066.stderr:----------------------------------------------------- 2024-03-04T06:07:32.744 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-04T06:07:32.810 INFO:teuthology.orchestra.run.smithi066.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-04T06:07:32.812 INFO:teuthology.orchestra.run.smithi066.stderr:INFO 2024-03-04 06:07:32,812 seastar - Reactor backend: linux-aio 2024-03-04T06:07:32.850 INFO:teuthology.orchestra.run.smithi066.stderr:==28947==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-04T06:07:32.856 INFO:teuthology.orchestra.run.smithi066.stderr:==28947==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc8ad86000; bottom 0x7f0b871e8000; size: 0x00f103b9e000 (1035149631488) 2024-03-04T06:07:32.856 INFO:teuthology.orchestra.run.smithi066.stderr:False positive error reports may follow 2024-03-04T06:07:32.856 INFO:teuthology.orchestra.run.smithi066.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-03-04T06:07:36.897 INFO:teuthology.orchestra.run.smithi066.stderr:----------------------------------------------------- 2024-03-04T06:07:36.897 INFO:teuthology.orchestra.run.smithi066.stderr:Suppressions used: 2024-03-04T06:07:36.898 INFO:teuthology.orchestra.run.smithi066.stderr: count bytes template 2024-03-04T06:07:36.898 INFO:teuthology.orchestra.run.smithi066.stderr: 1 8 MallocExtension::Register 2024-03-04T06:07:36.898 INFO:teuthology.orchestra.run.smithi066.stderr:----------------------------------------------------- 2024-03-04T06:07:36.898 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-04T06:07:36.898 INFO:teuthology.orchestra.run.smithi066.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid cb66367d-3032-46d0-abf8-73af48df9eb9 2024-03-04T06:07:36.939 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-04T06:07:36.970 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-04T06:07:36.971 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:36.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-04T06:07:37.003 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:37.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-04T06:07:37.073 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:37.073 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-04T06:07:37.105 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:37.105 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-04T06:07:37.175 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:37.175 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-04T06:07:37.246 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:37.246 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-04T06:07:37.265 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:37.265 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-04T06:07:37.302 INFO:tasks.ceph:Adding keys to all mons... 2024-03-04T06:07:37.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-04T06:07:37.334 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-03-04T06:07:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQATZeVlFMIQDxAA+dOMhd63iB5UPMtQt08iWQ== 2024-03-04T06:07:37.334 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-03-04T06:07:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAYZeVlr8ZOIBAAsFqSbcwiNNP55wJIzh0sPg== 2024-03-04T06:07:37.334 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-03-04T06:07:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAQZeVlonUJFRAA3oNls1OLbYA20V68553AZw== 2024-03-04T06:07:37.335 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-03-04T06:07:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAfZeVleWM3HhAAVCQx0CBiuGxi+EsWRWWJ+A== 2024-03-04T06:07:37.335 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-03-04T06:07:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAkZeVlKMtDNhAAbVVE03T+mprkteokieEubA== 2024-03-04T06:07:37.335 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-03-04T06:07:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAQZeVlTW8YGBAA6edE2Gz4zbDManSVeidd6Q== 2024-03-04T06:07:37.335 INFO:teuthology.orchestra.run.smithi064.stdout:[client.1] 2024-03-04T06:07:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAQZeVlmcwfHBAACWVl+leOtb/qJAbVFC6Pnw== 2024-03-04T06:07:37.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T06:07:37.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T06:07:37.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T06:07:37.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T06:07:37.618 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T06:07:37.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T06:07:37.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T06:07:37.797 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-04T06:07:37.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-04T06:07:37.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-04T06:07:37.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:37.937+0000 7f922b81b9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:37.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:37.937+0000 7f922b81b9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:37.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-04T06:07:37.978 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-04T06:07:38.033 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-04T06:07:38.033 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-04T06:07:38.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-04T06:07:38.076 INFO:tasks.ceph.mon.a:Started 2024-03-04T06:07:38.076 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-04T06:07:38.076 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-04T06:07:38.077 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-04T06:07:38.080 INFO:tasks.ceph.mgr.x:Started 2024-03-04T06:07:38.080 INFO:tasks.ceph:Setting crush tunables to default 2024-03-04T06:07:38.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd crush tunables default 2024-03-04T06:07:38.155 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-03-04T06:07:38.153+0000 7fa5b20b79c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:38.155 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-03-04T06:07:38.153+0000 7fa5b20b79c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:38.155 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-03-04T06:07:38.153+0000 7fa5b20b79c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:38.182 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:38.181+0000 7f2bcbf68f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:38.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:38.180+0000 7f144bd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:38.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:38.212+0000 7f144bd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:38.215 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:38.214+0000 7f2bcbf68f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:38.215 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:38.214+0000 7f2bcbf68f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:38.286 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-03-04T06:07:38.297 INFO:tasks.ceph:check_enable_crimson: True 2024-03-04T06:07:38.297 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-03-04T06:07:38.297 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-03-04T06:07:38.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:38.397+0000 7ff5fbe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:38.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:38.403+0000 7ff5fbe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:38.487 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-03-04T06:07:38.486+0000 7fa5a9ff1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-03-04T06:07:38.487 INFO:tasks.ceph.mon.a.smithi064.stderr:Please be aware that this feature is experimental, untested, 2024-03-04T06:07:38.487 INFO:tasks.ceph.mon.a.smithi064.stderr:unsupported, and may result in data corruption, data loss, 2024-03-04T06:07:38.488 INFO:tasks.ceph.mon.a.smithi064.stderr:and/or irreparable damage to your cluster. Do not use 2024-03-04T06:07:38.488 INFO:tasks.ceph.mon.a.smithi064.stderr:feature with important data. 2024-03-04T06:07:38.488 INFO:tasks.ceph.mon.a.smithi064.stderr: 2024-03-04T06:07:38.693 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:38.692+0000 7f2bcbf68f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T06:07:38.749 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:38.749+0000 7f2bcbf68f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T06:07:39.291 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:39.290+0000 7f2bcbf68f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T06:07:39.301 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-04T06:07:39.301 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:39.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-04T06:07:39.330 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T06:07:39.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-04T06:07:39.347 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:39.346+0000 7f2bcbf68f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T06:07:39.394 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:39.393+0000 7f2bcbf68f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T06:07:39.398 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:39.398 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-04T06:07:39.430 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-04T06:07:39.430 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-04T06:07:39.451 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:39.451+0000 7f2bcbf68f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T06:07:39.482 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --cluster ceph osd new 3a3a055b-c92d-46db-8419-c77269e3c8fe 0 2024-03-04T06:07:39.497 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:39.496+0000 7f2bcbf68f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T06:07:39.607 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-04T06:07:39.602+0000 7fce38f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:39.610 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-04T06:07:39.610+0000 7fce38f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:39.622 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:39.622+0000 7f2bcbf68f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T06:07:39.690 INFO:teuthology.orchestra.run.smithi066.stdout:0 2024-03-04T06:07:39.696 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:39.695+0000 7f2bcbf68f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T06:07:39.702 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --cluster ceph osd new a0c83c7b-d570-4f81-8e8f-4de39735fcf2 1 2024-03-04T06:07:39.746 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:39.745+0000 7f2bcbf68f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T06:07:39.803 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:39.803+0000 7f2bcbf68f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T06:07:39.813 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-04T06:07:39.808+0000 7f1a1a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:39.816 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-04T06:07:39.816+0000 7f1a1a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:39.853 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:39.853+0000 7f2bcbf68f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T06:07:39.896 INFO:teuthology.orchestra.run.smithi066.stdout:1 2024-03-04T06:07:39.907 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --cluster ceph osd new 20499aa6-c9a3-4a33-a9ed-0a58bf7b343f 2 2024-03-04T06:07:40.018 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-04T06:07:40.012+0000 7fbd0671d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:40.021 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-04T06:07:40.020+0000 7fbd0671d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:40.101 INFO:teuthology.orchestra.run.smithi066.stdout:2 2024-03-04T06:07:40.113 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --cluster ceph osd new 60c18871-1171-43cd-b598-6560f6021eeb 3 2024-03-04T06:07:40.224 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-04T06:07:40.219+0000 7f801675d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:40.227 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-04T06:07:40.227+0000 7f801675d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:40.283 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:40.282+0000 7f2bcbf68f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T06:07:40.329 INFO:teuthology.orchestra.run.smithi066.stdout:3 2024-03-04T06:07:40.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-03-04T06:07:40.378 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-04T06:07:40.378 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-03-04T06:07:40.417 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:40.416+0000 7f2bcbf68f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T06:07:40.421 INFO:tasks.ceph.osd.0:Started 2024-03-04T06:07:40.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-03-04T06:07:40.456 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-04T06:07:40.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-03-04T06:07:40.498 INFO:tasks.ceph.osd.1:Started 2024-03-04T06:07:40.498 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-03-04T06:07:40.534 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-04T06:07:40.534 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-03-04T06:07:40.541 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:40.540+0000 7f2bcbf68f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T06:07:40.542 INFO:tasks.ceph.osd.2:Started 2024-03-04T06:07:40.542 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-03-04T06:07:40.578 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-04T06:07:40.578 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-03-04T06:07:40.593 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:40.592+0000 7f2bcbf68f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T06:07:40.594 INFO:tasks.ceph.osd.3:Started 2024-03-04T06:07:40.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T06:07:40.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:40.714+0000 7fd39b51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:40.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:40.723+0000 7fd39b51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:40.724 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:40.723+0000 7f2bcbf68f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T06:07:40.805 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:40.805 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":7,"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","created":"2024-03-04T06:07:38.204256+0000","modified":"2024-03-04T06:07:40.328293+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T06:07:40.328293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3a3a055b-c92d-46db-8419-c77269e3c8fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a0c83c7b-d570-4f81-8e8f-4de39735fcf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"20499aa6-c9a3-4a33-a9ed-0a58bf7b343f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"60c18871-1171-43cd-b598-6560f6021eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T06:07:40.819 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-04T06:07:40.819 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-04T06:07:40.842 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:40.841+0000 7f2bcbf68f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T06:07:41.026 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:41.025+0000 7f2bcbf68f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T06:07:41.076 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:41.075+0000 7f2bcbf68f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T06:07:41.222 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-04T06:07:41.244 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:41.243+0000 7f2bcbf68f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T06:07:41.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-04T06:07:41.318+0000 7efd9ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:41.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-04T06:07:41.324+0000 7efd9ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:41.394 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-04T06:07:41.394 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":7,"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","created":"2024-03-04T06:07:38.204256+0000","modified":"2024-03-04T06:07:40.328293+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T06:07:40.328293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3a3a055b-c92d-46db-8419-c77269e3c8fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a0c83c7b-d570-4f81-8e8f-4de39735fcf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"20499aa6-c9a3-4a33-a9ed-0a58bf7b343f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"60c18871-1171-43cd-b598-6560f6021eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T06:07:41.405 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-03-04T06:07:42.137 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:42.136+0000 7f2bcbf68f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T06:07:42.446 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:42.445+0000 7f2bcbf68f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T06:07:42.527 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:42.526+0000 7f2bcbf68f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T06:07:42.684 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:42.683+0000 7f2bcbf68f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T06:07:43.707 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:07:43.706+0000 7f2b6b437640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-04T06:07:47.809 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-04T06:07:47.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-04T06:07:47.915+0000 7f2aa7354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:47.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-04T06:07:47.921+0000 7f2aa7354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:48.114 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-04T06:07:48.114 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","created":"2024-03-04T06:07:38.204256+0000","modified":"2024-03-04T06:07:47.297968+0000","last_up_change":"2024-03-04T06:07:45.292995+0000","last_in_change":"2024-03-04T06:07:40.328293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T06:07:46.747241+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a3a055b-c92d-46db-8419-c77269e3c8fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1128228680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1128228680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1128228680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":1128228680}]},"public_addr":"172.21.15.64:6805/1128228680","cluster_addr":"172.21.15.64:6804/1128228680","heartbeat_back_addr":"172.21.15.64:6806/1128228680","heartbeat_front_addr":"172.21.15.64:6803/1128228680","state":["exists","up"]},{"osd":1,"uuid":"a0c83c7b-d570-4f81-8e8f-4de39735fcf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":7123794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":7123794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":7123794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":7123794}]},"public_addr":"172.21.15.64:6802/7123794","cluster_addr":"172.21.15.64:6801/7123794","heartbeat_back_addr":"172.21.15.64:6812/7123794","heartbeat_front_addr":"172.21.15.64:6811/7123794","state":["exists","up"]},{"osd":2,"uuid":"20499aa6-c9a3-4a33-a9ed-0a58bf7b343f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":4001264085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6803","nonce":4001264085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6807","nonce":4001264085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":4001264085}]},"public_addr":"172.21.15.66:6802/4001264085","cluster_addr":"172.21.15.66:6803/4001264085","heartbeat_back_addr":"172.21.15.66:6807/4001264085","heartbeat_front_addr":"172.21.15.66:6806/4001264085","state":["exists","up"]},{"osd":3,"uuid":"60c18871-1171-43cd-b598-6560f6021eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6805","nonce":2264908010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":2264908010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6809","nonce":2264908010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":2264908010}]},"public_addr":"172.21.15.66:6805/2264908010","cluster_addr":"172.21.15.66:6804/2264908010","heartbeat_back_addr":"172.21.15.66:6809/2264908010","heartbeat_front_addr":"172.21.15.66:6808/2264908010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T06:07:48.125 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-03-04T06:07:48.125 INFO:tasks.ceph:Creating RBD pool 2024-03-04T06:07:48.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-04T06:07:48.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:48.225+0000 7f5091a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:48.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:48.234+0000 7f5091a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:49.309 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'rbd' created 2024-03-04T06:07:49.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-04T06:07:49.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:49.428+0000 7f26fe19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:49.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:49.439+0000 7f26fe19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:50.310 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-04T06:07:50.329 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-04T06:07:50.329 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-04T06:07:50.330 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-04T06:07:50.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-04T06:07:50.330 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-04T06:07:50.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:50.452+0000 7f72e8cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:50.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:50.458+0000 7f72e8cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:50.709 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6800","nonce":986922839},{"type":"v1","addr":"172.21.15.66:6801","nonce":986922839}]},"active_addr":"172.21.15.66:6801/986922839","active_change":"2024-03-04T06:07:42.697840+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":3190391421}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":3607095261}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":1565430662}]}]} 2024-03-04T06:07:50.713 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-04T06:07:50.713 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-04T06:07:50.713 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T06:07:50.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:50.816+0000 7fa153c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:50.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:50.823+0000 7fa153c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.013 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:51.014 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":13,"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","created":"2024-03-04T06:07:38.204256+0000","modified":"2024-03-04T06:07:50.309704+0000","last_up_change":"2024-03-04T06:07:45.292995+0000","last_in_change":"2024-03-04T06:07:40.328293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T06:07:46.747241+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-04T06:07:48.452826+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a3a055b-c92d-46db-8419-c77269e3c8fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1128228680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1128228680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1128228680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":1128228680}]},"public_addr":"172.21.15.64:6805/1128228680","cluster_addr":"172.21.15.64:6804/1128228680","heartbeat_back_addr":"172.21.15.64:6806/1128228680","heartbeat_front_addr":"172.21.15.64:6803/1128228680","state":["exists","up"]},{"osd":1,"uuid":"a0c83c7b-d570-4f81-8e8f-4de39735fcf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":7123794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":7123794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":7123794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":7123794}]},"public_addr":"172.21.15.64:6802/7123794","cluster_addr":"172.21.15.64:6801/7123794","heartbeat_back_addr":"172.21.15.64:6812/7123794","heartbeat_front_addr":"172.21.15.64:6811/7123794","state":["exists","up"]},{"osd":2,"uuid":"20499aa6-c9a3-4a33-a9ed-0a58bf7b343f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":4001264085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6803","nonce":4001264085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6807","nonce":4001264085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":4001264085}]},"public_addr":"172.21.15.66:6802/4001264085","cluster_addr":"172.21.15.66:6803/4001264085","heartbeat_back_addr":"172.21.15.66:6807/4001264085","heartbeat_front_addr":"172.21.15.66:6806/4001264085","state":["exists","up"]},{"osd":3,"uuid":"60c18871-1171-43cd-b598-6560f6021eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6805","nonce":2264908010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":2264908010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6809","nonce":2264908010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":2264908010}]},"public_addr":"172.21.15.66:6805/2264908010","cluster_addr":"172.21.15.66:6804/2264908010","heartbeat_back_addr":"172.21.15.66:6809/2264908010","heartbeat_front_addr":"172.21.15.66:6808/2264908010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T06:07:51.027 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-04T06:07:51.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T06:07:51.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.130+0000 7f7b7846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.137+0000 7f7b7846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:51.328 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":14,"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","created":"2024-03-04T06:07:38.204256+0000","modified":"2024-03-04T06:07:51.314814+0000","last_up_change":"2024-03-04T06:07:45.292995+0000","last_in_change":"2024-03-04T06:07:40.328293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T06:07:46.747241+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-04T06:07:48.452826+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a3a055b-c92d-46db-8419-c77269e3c8fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1128228680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1128228680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1128228680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":1128228680}]},"public_addr":"172.21.15.64:6805/1128228680","cluster_addr":"172.21.15.64:6804/1128228680","heartbeat_back_addr":"172.21.15.64:6806/1128228680","heartbeat_front_addr":"172.21.15.64:6803/1128228680","state":["exists","up"]},{"osd":1,"uuid":"a0c83c7b-d570-4f81-8e8f-4de39735fcf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":7123794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":7123794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":7123794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":7123794}]},"public_addr":"172.21.15.64:6802/7123794","cluster_addr":"172.21.15.64:6801/7123794","heartbeat_back_addr":"172.21.15.64:6812/7123794","heartbeat_front_addr":"172.21.15.64:6811/7123794","state":["exists","up"]},{"osd":2,"uuid":"20499aa6-c9a3-4a33-a9ed-0a58bf7b343f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":4001264085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6803","nonce":4001264085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6807","nonce":4001264085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":4001264085}]},"public_addr":"172.21.15.66:6802/4001264085","cluster_addr":"172.21.15.66:6803/4001264085","heartbeat_back_addr":"172.21.15.66:6807/4001264085","heartbeat_front_addr":"172.21.15.66:6806/4001264085","state":["exists","up"]},{"osd":3,"uuid":"60c18871-1171-43cd-b598-6560f6021eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6805","nonce":2264908010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":2264908010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6809","nonce":2264908010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":2264908010}]},"public_addr":"172.21.15.66:6805/2264908010","cluster_addr":"172.21.15.66:6804/2264908010","heartbeat_back_addr":"172.21.15.66:6809/2264908010","heartbeat_front_addr":"172.21.15.66:6808/2264908010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T06:07:51.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-04T06:07:51.342 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-04T06:07:51.342 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-04T06:07:51.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-04T06:07:51.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.457+0000 7f00bcf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.457+0000 7fa9cbe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.457+0000 7f5d5826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.457+0000 7f6280c2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.465+0000 7fa9cbe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.465+0000 7f5d5826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.465+0000 7f00bcf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.466+0000 7f6280c2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.493 INFO:teuthology.orchestra.run.smithi064.stdout:38654705670 2024-03-04T06:07:51.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-04T06:07:51.496 INFO:teuthology.orchestra.run.smithi064.stdout:38654705670 2024-03-04T06:07:51.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-04T06:07:51.498 INFO:teuthology.orchestra.run.smithi064.stdout:38654705670 2024-03-04T06:07:51.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-04T06:07:51.499 INFO:teuthology.orchestra.run.smithi064.stdout:38654705670 2024-03-04T06:07:51.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-04T06:07:51.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.599+0000 7fae5911d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.607+0000 7fae5911d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.653+0000 7fc8df06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.653+0000 7f7ba6f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.661+0000 7fc8df06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.661+0000 7f7ba6f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.666+0000 7f6765b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:51.677+0000 7f6765b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:51.834 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:51.847 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-03-04T06:07:51.878 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:51.879 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:51.882 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:51.894 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2024-03-04T06:07:51.894 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-03-04T06:07:51.896 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2024-03-04T06:07:52.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-04T06:07:52.894 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-04T06:07:52.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-04T06:07:52.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-04T06:07:52.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:52.968+0000 7f49cd46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:52.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:52.979+0000 7f49cd46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:53.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:53.011+0000 7fedcdd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:53.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:53.012+0000 7fa6bd59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:53.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:53.019+0000 7fedcdd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:53.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:53.022+0000 7fa6bd59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:53.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:53.063+0000 7fd0ae6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:53.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:53.070+0000 7fd0ae6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:53.222 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:53.241 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-03-04T06:07:53.241 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:53.253 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-03-04T06:07:53.258 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:53.265 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:53.271 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2024-03-04T06:07:53.277 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2024-03-04T06:07:54.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-04T06:07:54.255 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-04T06:07:54.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-04T06:07:54.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-04T06:07:54.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:54.365+0000 7f6a06e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:54.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:54.378+0000 7f6a06e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:54.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:54.407+0000 7fe125514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:54.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:54.407+0000 7f5b1260f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:54.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:54.408+0000 7f7c88b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:54.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:54.415+0000 7fe125514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:54.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:54.416+0000 7f5b1260f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:54.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:54.417+0000 7f7c88b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:54.576 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:54.590 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-03-04T06:07:54.617 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:54.631 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2024-03-04T06:07:54.636 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:54.644 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:54.649 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2024-03-04T06:07:54.657 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-03-04T06:07:55.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-04T06:07:55.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-04T06:07:55.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-04T06:07:55.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-04T06:07:55.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:55.712+0000 7f2bcc46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:55.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:55.722+0000 7f2bcc46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:55.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:55.757+0000 7f0ab7b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:55.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:55.767+0000 7f0ab7b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:55.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:55.796+0000 7f276379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:55.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:55.796+0000 7f039526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:55.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:55.804+0000 7f039526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:55.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:55.804+0000 7f276379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:55.943 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:55.960 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-03-04T06:07:55.984 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:55.998 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2024-03-04T06:07:56.005 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:56.018 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-03-04T06:07:56.024 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-03-04T06:07:56.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2024-03-04T06:07:56.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-04T06:07:56.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-04T06:07:57.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-04T06:07:57.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-04T06:07:57.086 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.079+0000 7f03b0024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:57.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.090+0000 7f03b0024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:57.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.119+0000 7f94a826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:57.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.128+0000 7f94a826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:57.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.160+0000 7f57a3b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:57.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.168+0000 7f57a3b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:57.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.196+0000 7f78c6e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:57.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.204+0000 7f78c6e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout:38654705674 2024-03-04T06:07:57.341 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.2 2024-03-04T06:07:57.341 DEBUG:teuthology.parallel:result is None 2024-03-04T06:07:57.353 INFO:teuthology.orchestra.run.smithi064.stdout:38654705675 2024-03-04T06:07:57.364 INFO:teuthology.orchestra.run.smithi064.stdout:38654705674 2024-03-04T06:07:57.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705675 for osd.1 2024-03-04T06:07:57.365 DEBUG:teuthology.parallel:result is None 2024-03-04T06:07:57.377 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.0 2024-03-04T06:07:57.377 DEBUG:teuthology.parallel:result is None 2024-03-04T06:07:57.415 INFO:teuthology.orchestra.run.smithi064.stdout:38654705675 2024-03-04T06:07:57.427 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705675 for osd.3 2024-03-04T06:07:57.428 DEBUG:teuthology.parallel:result is None 2024-03-04T06:07:57.428 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-04T06:07:57.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:07:57.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.570+0000 7fee14edd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:57.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.577+0000 7fee14edd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:57.766 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:57.766 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:07:57.778 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-03-04T06:07:56.705133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107960,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379400,"statfs":{"total":386547056640,"available":386436505600,"internally_reserved":0,"allocated":643072,"data_stored":1886520,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.364879"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.310713+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:07:56.310713+0000","last_peered":"2024-03-04T06:07:56.310713+0000","last_clean":"2024-03-04T06:07:56.310713+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:07:56.310713+0000","last_undegraded":"2024-03-04T06:07:56.310713+0000","last_fullsized":"2024-03-04T06:07:56.310713+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.364332+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:07:50.364332+0000","last_peered":"2024-03-04T06:07:50.364332+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:07:50.364332+0000","last_undegraded":"2024-03-04T06:07:50.364332+0000","last_fullsized":"2024-03-04T06:07:50.364332+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:51.333321+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:07:51.333321+0000","last_peered":"2024-03-04T06:07:51.333321+0000","last_clean":"2024-03-04T06:07:51.333321+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:07:51.333321+0000","last_undegraded":"2024-03-04T06:07:51.333321+0000","last_fullsized":"2024-03-04T06:07:51.333321+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.398024+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:07:50.398024+0000","last_peered":"2024-03-04T06:07:50.398024+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:07:50.398024+0000","last_undegraded":"2024-03-04T06:07:50.398024+0000","last_fullsized":"2024-03-04T06:07:50.398024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.383748+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:07:50.383748+0000","last_peered":"2024-03-04T06:07:50.383748+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:07:50.383748+0000","last_undegraded":"2024-03-04T06:07:50.383748+0000","last_fullsized":"2024-03-04T06:07:50.383748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.308194+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:07:56.308194+0000","last_peered":"2024-03-04T06:07:56.308194+0000","last_clean":"2024-03-04T06:07:56.308194+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:07:56.308194+0000","last_undegraded":"2024-03-04T06:07:56.308194+0000","last_fullsized":"2024-03-04T06:07:56.308194+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.363941+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:07:50.363941+0000","last_peered":"2024-03-04T06:07:50.363941+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:07:50.363941+0000","last_undegraded":"2024-03-04T06:07:50.363941+0000","last_fullsized":"2024-03-04T06:07:50.363941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.311611+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:07:56.311611+0000","last_peered":"2024-03-04T06:07:56.311611+0000","last_clean":"2024-03-04T06:07:56.311611+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:07:56.311611+0000","last_undegraded":"2024-03-04T06:07:56.311611+0000","last_fullsized":"2024-03-04T06:07:56.311611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:51.329875+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:07:51.329875+0000","last_peered":"2024-03-04T06:07:51.329875+0000","last_clean":"2024-03-04T06:07:51.329875+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:07:51.329875+0000","last_undegraded":"2024-03-04T06:07:51.329875+0000","last_fullsized":"2024-03-04T06:07:51.329875+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":30459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:07:57.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:07:57.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.882+0000 7fa02f02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:57.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:57.888+0000 7fa02f02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:58.077 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:58.078 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:07:58.089 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-03-04T06:07:56.705133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107960,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379400,"statfs":{"total":386547056640,"available":386436505600,"internally_reserved":0,"allocated":643072,"data_stored":1886520,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.364879"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.310713+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:07:56.310713+0000","last_peered":"2024-03-04T06:07:56.310713+0000","last_clean":"2024-03-04T06:07:56.310713+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:07:56.310713+0000","last_undegraded":"2024-03-04T06:07:56.310713+0000","last_fullsized":"2024-03-04T06:07:56.310713+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.364332+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:07:50.364332+0000","last_peered":"2024-03-04T06:07:50.364332+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:07:50.364332+0000","last_undegraded":"2024-03-04T06:07:50.364332+0000","last_fullsized":"2024-03-04T06:07:50.364332+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:51.333321+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:07:51.333321+0000","last_peered":"2024-03-04T06:07:51.333321+0000","last_clean":"2024-03-04T06:07:51.333321+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:07:51.333321+0000","last_undegraded":"2024-03-04T06:07:51.333321+0000","last_fullsized":"2024-03-04T06:07:51.333321+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.398024+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:07:50.398024+0000","last_peered":"2024-03-04T06:07:50.398024+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:07:50.398024+0000","last_undegraded":"2024-03-04T06:07:50.398024+0000","last_fullsized":"2024-03-04T06:07:50.398024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.383748+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:07:50.383748+0000","last_peered":"2024-03-04T06:07:50.383748+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:07:50.383748+0000","last_undegraded":"2024-03-04T06:07:50.383748+0000","last_fullsized":"2024-03-04T06:07:50.383748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.308194+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:07:56.308194+0000","last_peered":"2024-03-04T06:07:56.308194+0000","last_clean":"2024-03-04T06:07:56.308194+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:07:56.308194+0000","last_undegraded":"2024-03-04T06:07:56.308194+0000","last_fullsized":"2024-03-04T06:07:56.308194+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.363941+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:07:50.363941+0000","last_peered":"2024-03-04T06:07:50.363941+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:07:50.363941+0000","last_undegraded":"2024-03-04T06:07:50.363941+0000","last_fullsized":"2024-03-04T06:07:50.363941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.311611+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:07:56.311611+0000","last_peered":"2024-03-04T06:07:56.311611+0000","last_clean":"2024-03-04T06:07:56.311611+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:07:56.311611+0000","last_undegraded":"2024-03-04T06:07:56.311611+0000","last_fullsized":"2024-03-04T06:07:56.311611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:51.329875+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:07:51.329875+0000","last_peered":"2024-03-04T06:07:51.329875+0000","last_clean":"2024-03-04T06:07:51.329875+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:07:51.329875+0000","last_undegraded":"2024-03-04T06:07:51.329875+0000","last_fullsized":"2024-03-04T06:07:51.329875+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":30459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:07:58.090 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-04T06:07:58.090 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-04T06:07:58.090 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-04T06:07:58.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-04T06:07:58.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:58.194+0000 7fd415cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:58.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:58.199+0000 7fd415cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:58.440 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:58.440 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T06:07:58.453 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-04T06:07:58.453 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-04T06:07:58.464 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-04T06:07:58.464 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-nitzan-crimson-shard-pg-mapping-get-or-create'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-nitzan-crimson-shard-pg-mapping-get-or-create', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd'}} 2024-03-04T06:07:58.465 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-04T06:07:58.465 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-04T06:07:58.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:58.568+0000 7f4f2e26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:58.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:58.577+0000 7f4f2e26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:58.787 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-04T06:07:58.787 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-04T06:07:58.787 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:07:58.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:58.890+0000 7f92a2f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:58.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:58.896+0000 7f92a2f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:59.083 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:59.084 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:07:59.096 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-03-04T06:07:58.705496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107960,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379400,"statfs":{"total":386547056640,"available":386436505600,"internally_reserved":0,"allocated":643072,"data_stored":1886520,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.365241"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.310713+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:07:56.310713+0000","last_peered":"2024-03-04T06:07:56.310713+0000","last_clean":"2024-03-04T06:07:56.310713+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:07:56.310713+0000","last_undegraded":"2024-03-04T06:07:56.310713+0000","last_fullsized":"2024-03-04T06:07:56.310713+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.364332+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:07:50.364332+0000","last_peered":"2024-03-04T06:07:50.364332+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:07:50.364332+0000","last_undegraded":"2024-03-04T06:07:50.364332+0000","last_fullsized":"2024-03-04T06:07:50.364332+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:51.333321+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:07:51.333321+0000","last_peered":"2024-03-04T06:07:51.333321+0000","last_clean":"2024-03-04T06:07:51.333321+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:07:51.333321+0000","last_undegraded":"2024-03-04T06:07:51.333321+0000","last_fullsized":"2024-03-04T06:07:51.333321+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.398024+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:07:50.398024+0000","last_peered":"2024-03-04T06:07:50.398024+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:07:50.398024+0000","last_undegraded":"2024-03-04T06:07:50.398024+0000","last_fullsized":"2024-03-04T06:07:50.398024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.383748+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:07:50.383748+0000","last_peered":"2024-03-04T06:07:50.383748+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:07:50.383748+0000","last_undegraded":"2024-03-04T06:07:50.383748+0000","last_fullsized":"2024-03-04T06:07:50.383748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.308194+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:07:56.308194+0000","last_peered":"2024-03-04T06:07:56.308194+0000","last_clean":"2024-03-04T06:07:56.308194+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:07:56.308194+0000","last_undegraded":"2024-03-04T06:07:56.308194+0000","last_fullsized":"2024-03-04T06:07:56.308194+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.363941+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:07:50.363941+0000","last_peered":"2024-03-04T06:07:50.363941+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:07:50.363941+0000","last_undegraded":"2024-03-04T06:07:50.363941+0000","last_fullsized":"2024-03-04T06:07:50.363941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.311611+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:07:56.311611+0000","last_peered":"2024-03-04T06:07:56.311611+0000","last_clean":"2024-03-04T06:07:56.311611+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:07:56.311611+0000","last_undegraded":"2024-03-04T06:07:56.311611+0000","last_fullsized":"2024-03-04T06:07:56.311611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:51.329875+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:07:51.329875+0000","last_peered":"2024-03-04T06:07:51.329875+0000","last_clean":"2024-03-04T06:07:51.329875+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:07:51.329875+0000","last_undegraded":"2024-03-04T06:07:51.329875+0000","last_fullsized":"2024-03-04T06:07:51.329875+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":30459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:07:59.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:07:59.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:59.200+0000 7f8848b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:59.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:59.207+0000 7f8848b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:59.396 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:07:59.397 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:07:59.409 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-03-04T06:07:58.705496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107960,"kb_used_data":628,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379400,"statfs":{"total":386547056640,"available":386436505600,"internally_reserved":0,"allocated":643072,"data_stored":1886520,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.365241"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.310713+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:07:56.310713+0000","last_peered":"2024-03-04T06:07:56.310713+0000","last_clean":"2024-03-04T06:07:56.310713+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:07:56.310713+0000","last_undegraded":"2024-03-04T06:07:56.310713+0000","last_fullsized":"2024-03-04T06:07:56.310713+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.364332+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:07:50.364332+0000","last_peered":"2024-03-04T06:07:50.364332+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:07:50.364332+0000","last_undegraded":"2024-03-04T06:07:50.364332+0000","last_fullsized":"2024-03-04T06:07:50.364332+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:51.333321+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:07:51.333321+0000","last_peered":"2024-03-04T06:07:51.333321+0000","last_clean":"2024-03-04T06:07:51.333321+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:07:51.333321+0000","last_undegraded":"2024-03-04T06:07:51.333321+0000","last_fullsized":"2024-03-04T06:07:51.333321+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.398024+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:07:50.398024+0000","last_peered":"2024-03-04T06:07:50.398024+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:07:50.398024+0000","last_undegraded":"2024-03-04T06:07:50.398024+0000","last_fullsized":"2024-03-04T06:07:50.398024+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.383748+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:07:50.383748+0000","last_peered":"2024-03-04T06:07:50.383748+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:07:50.383748+0000","last_undegraded":"2024-03-04T06:07:50.383748+0000","last_fullsized":"2024-03-04T06:07:50.383748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.308194+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:07:56.308194+0000","last_peered":"2024-03-04T06:07:56.308194+0000","last_clean":"2024-03-04T06:07:56.308194+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:07:56.308194+0000","last_undegraded":"2024-03-04T06:07:56.308194+0000","last_fullsized":"2024-03-04T06:07:56.308194+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-04T06:07:50.363941+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:07:50.363941+0000","last_peered":"2024-03-04T06:07:50.363941+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:07:50.363941+0000","last_undegraded":"2024-03-04T06:07:50.363941+0000","last_fullsized":"2024-03-04T06:07:50.363941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.311611+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:07:56.311611+0000","last_peered":"2024-03-04T06:07:56.311611+0000","last_clean":"2024-03-04T06:07:56.311611+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:07:56.311611+0000","last_undegraded":"2024-03-04T06:07:56.311611+0000","last_fullsized":"2024-03-04T06:07:56.311611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":145,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:51.329875+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:07:51.329875+0000","last_peered":"2024-03-04T06:07:51.329875+0000","last_clean":"2024-03-04T06:07:51.329875+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:07:51.329875+0000","last_undegraded":"2024-03-04T06:07:51.329875+0000","last_fullsized":"2024-03-04T06:07:51.329875+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":30459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":617617,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:07:59.410 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-04T06:07:59.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-03-04T06:07:59.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:59.513+0000 7f31f159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:59.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:07:59.519+0000 7f31f159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:07:59.711 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 14 2024-03-04T06:07:59.711 INFO:teuthology.orchestra.run.smithi064.stdout:fsid cb66367d-3032-46d0-abf8-73af48df9eb9 2024-03-04T06:07:59.711 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-03-04T06:07:38.204256+0000 2024-03-04T06:07:59.711 INFO:teuthology.orchestra.run.smithi064.stdout:modified 2024-03-04T06:07:51.314814+0000 2024-03-04T06:07:59.711 INFO:teuthology.orchestra.run.smithi064.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-04T06:07:59.712 INFO:teuthology.orchestra.run.smithi064.stdout:crush_version 3 2024-03-04T06:07:59.712 INFO:teuthology.orchestra.run.smithi064.stdout:full_ratio 0.95 2024-03-04T06:07:59.712 INFO:teuthology.orchestra.run.smithi064.stdout:backfillfull_ratio 0.9 2024-03-04T06:07:59.712 INFO:teuthology.orchestra.run.smithi064.stdout:nearfull_ratio 0.85 2024-03-04T06:07:59.712 INFO:teuthology.orchestra.run.smithi064.stdout:require_min_compat_client luminous 2024-03-04T06:07:59.712 INFO:teuthology.orchestra.run.smithi064.stdout:min_compat_client jewel 2024-03-04T06:07:59.712 INFO:teuthology.orchestra.run.smithi064.stdout:require_osd_release squid 2024-03-04T06:07:59.712 INFO:teuthology.orchestra.run.smithi064.stdout:stretch_mode_enabled false 2024-03-04T06:07:59.712 INFO:teuthology.orchestra.run.smithi064.stdout:allow_crimson=true 2024-03-04T06:07:59.713 INFO:teuthology.orchestra.run.smithi064.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-03-04T06:07:59.713 INFO:teuthology.orchestra.run.smithi064.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 14 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-03-04T06:07:59.713 INFO:teuthology.orchestra.run.smithi064.stdout:max_osd 4 2024-03-04T06:07:59.713 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6805/1128228680 v2:172.21.15.64:6804/1128228680 exists,up 3a3a055b-c92d-46db-8419-c77269e3c8fe 2024-03-04T06:07:59.713 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6802/7123794 v2:172.21.15.64:6801/7123794 exists,up a0c83c7b-d570-4f81-8e8f-4de39735fcf2 2024-03-04T06:07:59.713 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.66:6802/4001264085 v2:172.21.15.66:6803/4001264085 exists,up 20499aa6-c9a3-4a33-a9ed-0a58bf7b343f 2024-03-04T06:07:59.713 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.66:6805/2264908010 v2:172.21.15.66:6804/2264908010 exists,up 60c18871-1171-43cd-b598-6560f6021eeb 2024-03-04T06:07:59.723 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6805/1128228680 v2:172.21.15.64:6804/1128228680 exists,up 3a3a055b-c92d-46db-8419-c77269e3c8fe', 'osd.1 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.64:6802/7123794 v2:172.21.15.64:6801/7123794 exists,up a0c83c7b-d570-4f81-8e8f-4de39735fcf2', 'osd.2 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.66:6802/4001264085 v2:172.21.15.66:6803/4001264085 exists,up 20499aa6-c9a3-4a33-a9ed-0a58bf7b343f', 'osd.3 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.66:6805/2264908010 v2:172.21.15.66:6804/2264908010 exists,up 60c18871-1171-43cd-b598-6560f6021eeb'] 2024-03-04T06:07:59.724 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-03-04T06:07:59.832 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-04T06:07:59.833 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mon.a", 2024-03-04T06:07:59.833 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster": "ceph", 2024-03-04T06:07:59.833 INFO:teuthology.orchestra.run.smithi064.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-04T06:07:59.833 INFO:teuthology.orchestra.run.smithi064.stdout: "admin_socket_mode": "", 2024-03-04T06:07:59.833 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-03-04T06:07:59.833 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_client_required": "cephx, none", 2024-03-04T06:07:59.833 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_cluster_required": "cephx", 2024-03-04T06:07:59.833 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_debug": "true", 2024-03-04T06:07:59.833 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-04T06:07:59.834 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-04T06:07:59.834 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_service_required": "cephx", 2024-03-04T06:07:59.834 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-04T06:07:59.834 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_supported": "cephx", 2024-03-04T06:07:59.834 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio": "true", 2024-03-04T06:07:59.834 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-04T06:07:59.834 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_poll_ms": "250", 2024-03-04T06:07:59.834 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_reap_max": "16", 2024-03-04T06:07:59.834 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_async_discard": "false", 2024-03-04T06:07:59.834 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_block_size": "4096", 2024-03-04T06:07:59.835 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio": "false", 2024-03-04T06:07:59.835 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-04T06:07:59.835 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-04T06:07:59.835 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_inflight_ios": "false", 2024-03-04T06:07:59.835 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_enable_discard": "false", 2024-03-04T06:07:59.835 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_flock_retry": "3", 2024-03-04T06:07:59.835 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-04T06:07:59.835 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_inject_crash": "0", 2024-03-04T06:07:59.835 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-04T06:07:59.835 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring": "false", 2024-03-04T06:07:59.836 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring_hipri": "false", 2024-03-04T06:07:59.836 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-04T06:07:59.836 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-04T06:07:59.836 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-04T06:07:59.836 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-04T06:07:59.836 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_type": "", 2024-03-04T06:07:59.836 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_alloc_size": "1048576", 2024-03-04T06:07:59.836 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_allocator": "hybrid", 2024-03-04T06:07:59.836 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_buffered_io": "true", 2024-03-04T06:07:59.837 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_for_zeros": "false", 2024-03-04T06:07:59.837 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-04T06:07:59.837 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-04T06:07:59.837 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_compact_log_sync": "false", 2024-03-04T06:07:59.837 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-04T06:07:59.837 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-04T06:07:59.837 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-04T06:07:59.837 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-04T06:07:59.837 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_max_log_runway": "4194304", 2024-03-04T06:07:59.837 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_max_prefetch": "1048576", 2024-03-04T06:07:59.837 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_min_flush_size": "524288", 2024-03-04T06:07:59.838 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_min_log_runway": "1048576", 2024-03-04T06:07:59.838 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_replay_recovery": "false", 2024-03-04T06:07:59.838 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-04T06:07:59.838 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-04T06:07:59.838 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_sync_write": "false", 2024-03-04T06:07:59.838 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-04T06:07:59.838 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-04T06:07:59.838 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-04T06:07:59.838 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_allocation_from_file": "true", 2024-03-04T06:07:59.838 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_allocator": "hybrid", 2024-03-04T06:07:59.839 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-04T06:07:59.839 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-04T06:07:59.839 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-04T06:07:59.839 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-04T06:07:59.839 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-04T06:07:59.839 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-04T06:07:59.839 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-04T06:07:59.839 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_create": "true", 2024-03-04T06:07:59.839 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_create": "false", 2024-03-04T06:07:59.839 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_path": "", 2024-03-04T06:07:59.840 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_size": "0", 2024-03-04T06:07:59.840 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_path": "", 2024-03-04T06:07:59.840 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_preallocate_file": "false", 2024-03-04T06:07:59.840 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_size": "107374182400", 2024-03-04T06:07:59.840 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_create": "false", 2024-03-04T06:07:59.840 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_path": "", 2024-03-04T06:07:59.840 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_size": "100663296", 2024-03-04T06:07:59.840 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs": "true", 2024-03-04T06:07:59.840 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-04T06:07:59.840 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-04T06:07:59.841 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-04T06:07:59.841 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-04T06:07:59.841 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-04T06:07:59.841 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-04T06:07:59.841 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-04T06:07:59.841 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-04T06:07:59.841 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_autotune": "true", 2024-03-04T06:07:59.841 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-04T06:07:59.841 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-04T06:07:59.841 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-04T06:07:59.841 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-04T06:07:59.842 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size": "0", 2024-03-04T06:07:59.842 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-04T06:07:59.842 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-04T06:07:59.842 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-04T06:07:59.842 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-04T06:07:59.842 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_type": "2q", 2024-03-04T06:07:59.842 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-04T06:07:59.842 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_clone_cow": "true", 2024-03-04T06:07:59.842 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-04T06:07:59.842 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-04T06:07:59.842 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-04T06:07:59.843 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-04T06:07:59.843 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-04T06:07:59.843 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-04T06:07:59.843 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-04T06:07:59.843 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_mode": "none", 2024-03-04T06:07:59.843 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-04T06:07:59.843 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_csum_type": "crc32c", 2024-03-04T06:07:59.843 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-04T06:07:59.843 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_freelist": "false", 2024-03-04T06:07:59.843 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-04T06:07:59.843 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-04T06:07:59.844 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-04T06:07:59.844 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-04T06:07:59.844 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-04T06:07:59.844 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-04T06:07:59.844 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-04T06:07:59.844 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-04T06:07:59.844 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-04T06:07:59.844 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-04T06:07:59.844 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-04T06:07:59.844 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-04T06:07:59.844 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-04T06:07:59.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_small_allocations": "0", 2024-03-04T06:07:59.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-04T06:07:59.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_default_buffered_read": "true", 2024-03-04T06:07:59.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_default_buffered_write": "false", 2024-03-04T06:07:59.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-04T06:07:59.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-04T06:07:59.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-04T06:07:59.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-04T06:07:59.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-04T06:07:59.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-04T06:07:59.845 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-04T06:07:59.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-04T06:07:59.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-04T06:07:59.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fail_eio": "false", 2024-03-04T06:07:59.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-04T06:07:59.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-04T06:07:59.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-04T06:07:59.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-04T06:07:59.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-04T06:07:59.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-04T06:07:59.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mount": "false", 2024-03-04T06:07:59.846 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-04T06:07:59.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_umount": "false", 2024-03-04T06:07:59.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-04T06:07:59.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-04T06:07:59.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-04T06:07:59.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-04T06:07:59.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-04T06:07:59.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-04T06:07:59.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-04T06:07:59.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-04T06:07:59.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_ignore_data_csum": "false", 2024-03-04T06:07:59.847 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-04T06:07:59.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-04T06:07:59.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-04T06:07:59.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-04T06:07:59.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_op_age": "5.000000", 2024-03-04T06:07:59.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_alloc_size": "0", 2024-03-04T06:07:59.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size": "0", 2024-03-04T06:07:59.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-04T06:07:59.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-04T06:07:59.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-04T06:07:59.848 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_deferred_txc": "32", 2024-03-04T06:07:59.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size": "0", 2024-03-04T06:07:59.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-04T06:07:59.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-04T06:07:59.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_nid_prealloc": "1024", 2024-03-04T06:07:59.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-04T06:07:59.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-04T06:07:59.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-04T06:07:59.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-04T06:07:59.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_retry_disk_reads": "3", 2024-03-04T06:07:59.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_cf": "true", 2024-03-04T06:07:59.849 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-04T06:07:59.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-04T06:07:59.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-04T06:07:59.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-04T06:07:59.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-04T06:07:59.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-04T06:07:59.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_mem": "512", 2024-03-04T06:07:59.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-04T06:07:59.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-04T06:07:59.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-04T06:07:59.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-04T06:07:59.850 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-04T06:07:59.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-04T06:07:59.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-04T06:07:59.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_tracing": "false", 2024-03-04T06:07:59.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-04T06:07:59.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-04T06:07:59.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-04T06:07:59.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-04T06:07:59.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-04T06:07:59.851 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-04T06:07:59.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-04T06:07:59.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-04T06:07:59.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-04T06:07:59.852 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_zero_block_detection": "false", 2024-03-04T06:07:59.852 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-04T06:07:59.852 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-04T06:07:59.852 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-04T06:07:59.852 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-04T06:07:59.853 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-04T06:07:59.853 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-04T06:07:59.853 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-04T06:07:59.853 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-04T06:07:59.853 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-04T06:07:59.853 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-04T06:07:59.853 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-04T06:07:59.853 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-04T06:07:59.853 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-04T06:07:59.853 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_cluster_require_signatures": "false", 2024-03-04T06:07:59.853 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_cluster_require_version": "2", 2024-03-04T06:07:59.854 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_require_signatures": "false", 2024-03-04T06:07:59.854 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_require_version": "2", 2024-03-04T06:07:59.854 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_service_require_signatures": "false", 2024-03-04T06:07:59.854 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_service_require_version": "2", 2024-03-04T06:07:59.854 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_sign_messages": "true", 2024-03-04T06:07:59.854 INFO:teuthology.orchestra.run.smithi064.stdout: "chdir": "", 2024-03-04T06:07:59.854 INFO:teuthology.orchestra.run.smithi064.stdout: "client_acl_type": "", 2024-03-04T06:07:59.854 INFO:teuthology.orchestra.run.smithi064.stdout: "client_asio_thread_count": "2", 2024-03-04T06:07:59.854 INFO:teuthology.orchestra.run.smithi064.stdout: "client_cache_mid": "0.750000", 2024-03-04T06:07:59.854 INFO:teuthology.orchestra.run.smithi064.stdout: "client_cache_size": "16384", 2024-03-04T06:07:59.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_caps_release_delay": "5", 2024-03-04T06:07:59.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_check_pool_perm": "true", 2024-03-04T06:07:59.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-04T06:07:59.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_force_sync_read": "false", 2024-03-04T06:07:59.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_getattr_caps": "false", 2024-03-04T06:07:59.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_inject_tick_delay": "0", 2024-03-04T06:07:59.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-04T06:07:59.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_die_on_failed_remount": "false", 2024-03-04T06:07:59.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_dirsize_rbytes": "true", 2024-03-04T06:07:59.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_force_lazyio": "false", 2024-03-04T06:07:59.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_fs": "", 2024-03-04T06:07:59.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-04T06:07:59.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_inject_release_failure": "false", 2024-03-04T06:07:59.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_max_inline_size": "4096", 2024-03-04T06:07:59.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-04T06:07:59.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mds_namespace": "", 2024-03-04T06:07:59.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_metadata": "", 2024-03-04T06:07:59.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_gid": "-1", 2024-03-04T06:07:59.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_timeout": "300", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_uid": "-1", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mountpoint": "/", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_notify_timeout": "10", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc": "true", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_dirty": "104857600", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_objects": "1000", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_size": "209715200", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_target_dirty": "8388608", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_permissions": "true", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_quota": "true", 2024-03-04T06:07:59.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_quota_df": "true", 2024-03-04T06:07:59.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_max_bytes": "0", 2024-03-04T06:07:59.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_max_periods": "4", 2024-03-04T06:07:59.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_min": "131072", 2024-03-04T06:07:59.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_reconnect_stale": "false", 2024-03-04T06:07:59.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_shutdown_timeout": "30", 2024-03-04T06:07:59.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_snapdir": ".snap", 2024-03-04T06:07:59.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_tick_interval": "1", 2024-03-04T06:07:59.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_trace": "", 2024-03-04T06:07:59.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_try_dentry_invalidate": "false", 2024-03-04T06:07:59.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_use_faked_inos": "false", 2024-03-04T06:07:59.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_use_random_mds": "false", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog": "false", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog_port": "12201", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_monitors": "default=true", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog": "false", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog_level": "info", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "-", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_network": "", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_network_interface": "", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_isal": "false", 2024-03-04T06:07:59.859 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_level": "5", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_winsize": "-15", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zstd_level": "1", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_alien_op_num_threads": "6", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_seastar_cpu_cores": "", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location_hook": "", 2024-03-04T06:07:59.860 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location_hook_timeout": "10", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "daemonize": "false", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "daos_pool": "tank", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_alienstore": "0/5", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_allow_any_pool_priority": "false", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asok": "1/5", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asok_assert_abort": "false", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asserts_on_shutdown": "true", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_auth": "20/20", 2024-03-04T06:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bdev": "1/3", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bluefs": "1/5", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bluestore": "1/5", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_buffer": "0/1", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_ceph_exporter": "1/5", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cephfs_mirror": "0/5", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cephsqlite": "0/5", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_client": "0/5", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_compressor": "1/5", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_context": "0/1", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crimson_interrupt": "0/5", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crush": "1/1", 2024-03-04T06:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crypto": "1/5", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cyanstore": "0/5", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_deliberately_leak_memory": "false", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_disable_randomized_ping": "false", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_dpdk": "1/5", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_eventtrace": "1/5", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_filer": "0/1", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_filestore": "1/3", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_finisher": "1/1", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_fuse": "1/5", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_heartbeat_testing_span": "0", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_heartbeatmap": "1/5", 2024-03-04T06:07:59.863 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_javaclient": "1/5", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_journal": "1/3", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_journaler": "0/5", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_kstore": "1/5", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_lockdep": "0/1", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mclock": "1/5", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds": "1/5", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_balancer": "1/5", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_locker": "1/5", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_log": "1/5", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_log_expire": "1/5", 2024-03-04T06:07:59.864 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_migrator": "1/5", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_memstore": "1/5", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mgr": "2/5", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mgrc": "1/5", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mon": "20/20", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_monc": "0/10", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_ms": "1/1", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_none": "0/5", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objclass": "0/5", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objectcacher": "0/5", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objecter": "0/1", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_optracker": "0/5", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_osd": "1/5", 2024-03-04T06:07:59.865 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_paxos": "20/20", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_perfcounter": "1/5", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_prioritycache": "1/5", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rados": "0/5", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd": "0/5", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_mirror": "0/5", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_pwl": "0/5", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_replay": "0/5", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_refs": "0/0", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_reserver": "1/1", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw": "1/5", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_access": "1/5", 2024-03-04T06:07:59.866 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_datacache": "1/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_dbstore": "1/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_flight": "1/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_sync": "1/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rocksdb": "4/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore": "0/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_backref": "0/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_cache": "0/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_cleaner": "0/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_device": "0/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_epm": "0/5", 2024-03-04T06:07:59.867 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_journal": "0/5", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_lba": "0/5", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_odata": "0/5", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_omap": "0/5", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_onode": "0/5", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_t": "0/5", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_tm": "0/5", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_striper": "0/1", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_test": "0/5", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_throttle": "1/1", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_timer": "0/1", 2024-03-04T06:07:59.868 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_tp": "0/5", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_trace": "1/5", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "device_failure_prediction_mode": "none", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_graylog": "false", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_journald": "false", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_stderr": "true", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_syslog": "false", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event_tracing": "false", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_addr": "0.0.0.0", 2024-03-04T06:07:59.869 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_http_port": "9926", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_prio_limit": "5", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_sort_metrics": "true", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_stats_period": "5", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "fake_statfs_for_testing": "0", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "fatal_signal_handlers": "true", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "filer_max_purge_ops": "10", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "filer_max_truncate_ops": "128", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_apply_finisher_threads": "1", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_blackhole": "false", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_btrfs_clone_range": "true", 2024-03-04T06:07:59.870 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_btrfs_snap": "true", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_caller_concurrency": "10", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_collect_device_partition_information": "true", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_commit_timeout": "600.000000", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_inject_read_err": "false", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_omap_check": "false", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_verify_split": "false", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_dump_file": "", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fadvise": "true", 2024-03-04T06:07:59.871 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fail_eio": "true", 2024-03-04T06:07:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fd_cache_shards": "16", 2024-03-04T06:07:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fd_cache_size": "128", 2024-03-04T06:07:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fiemap": "false", 2024-03-04T06:07:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fiemap_threshold": "4096", 2024-03-04T06:07:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-04T06:07:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-04T06:07:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_inject_stall": "0", 2024-03-04T06:07:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_parallel": "false", 2024-03-04T06:07:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_trailing": "false", 2024-03-04T06:07:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_writeahead": "false", 2024-03-04T06:07:59.872 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_kill_at": "0", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs": "0", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size": "0", 2024-03-04T06:07:59.873 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_merge_threshold": "-10", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_odsync_write": "false", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_backend": "rocksdb", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_backend_path": "", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_thread_timeout": "60", 2024-03-04T06:07:59.874 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_threads": "2", 2024-03-04T06:07:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_punch_hole": "false", 2024-03-04T06:07:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-04T06:07:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-04T06:07:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-04T06:07:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-04T06:07:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-04T06:07:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-04T06:07:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-04T06:07:59.875 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-04T06:07:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-04T06:07:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_ops": "50", 2024-03-04T06:07:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-04T06:07:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_seek_data_hole": "false", 2024-03-04T06:07:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_sloppy_crc": "false", 2024-03-04T06:07:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-04T06:07:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_splice": "false", 2024-03-04T06:07:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_split_multiple": "2", 2024-03-04T06:07:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_split_rand_factor": "20", 2024-03-04T06:07:59.876 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_update_to": "1000", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_enable": "true", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-04T06:07:59.877 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-04T06:07:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-04T06:07:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_xfs_extsize": "false", 2024-03-04T06:07:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_zfs_snap": "false", 2024-03-04T06:07:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "fio_dir": "/tmp/fio", 2024-03-04T06:07:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "cb66367d-3032-46d0-abf8-73af48df9eb9", 2024-03-04T06:07:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_allow_other": "true", 2024-03-04T06:07:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_atomic_o_trunc": "true", 2024-03-04T06:07:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_big_writes": "true", 2024-03-04T06:07:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_debug": "false", 2024-03-04T06:07:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_default_permissions": "false", 2024-03-04T06:07:59.878 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_disable_pagecache": "false", 2024-03-04T06:07:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_max_write": "0", 2024-03-04T06:07:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_multithreaded": "true", 2024-03-04T06:07:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_require_active_mds": "true", 2024-03-04T06:07:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_set_user_groups": "true", 2024-03-04T06:07:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_move": "true", 2024-03-04T06:07:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_read": "true", 2024-03-04T06:07:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_write": "true", 2024-03-04T06:07:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-04T06:07:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_use_invalidate_cb": "true", 2024-03-04T06:07:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-04T06:07:59.879 INFO:teuthology.orchestra.run.smithi064.stdout: "gss_target_name": "ceph", 2024-03-04T06:07:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_file": "", 2024-03-04T06:07:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_inject_failure": "0", 2024-03-04T06:07:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_interval": "5", 2024-03-04T06:07:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "host": "smithi064", 2024-03-04T06:07:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-04T06:07:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-04T06:07:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-04T06:07:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-04T06:07:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-04T06:07:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-04T06:07:59.880 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-04T06:07:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-04T06:07:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-04T06:07:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-04T06:07:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-04T06:07:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-04T06:07:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-04T06:07:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "inject_early_sigterm": "false", 2024-03-04T06:07:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "jaeger_agent_port": "6799", 2024-03-04T06:07:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "jaeger_tracing_enable": "false", 2024-03-04T06:07:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_aio": "true", 2024-03-04T06:07:59.881 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_align_min_size": "65536", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_block_align": "true", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_block_size": "4096", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_dio": "true", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_discard": "false", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_force_aio": "false", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_ignore_corruption": "false", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_max_write_bytes": "10485760", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_max_write_entries": "100", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_replay_from": "0", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-04T06:07:59.882 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_write_header_frequency": "0", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_zero_on_create": "false", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_prefetch_periods": "10", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_prezero_periods": "5", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_write_head_interval": "15", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "key": "", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "keyfile": "", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_backend": "rocksdb", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_default_stripe_size": "65536", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_fsck_on_mount": "false", 2024-03-04T06:07:59.883 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-04T06:07:59.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_max_bytes": "67108864", 2024-03-04T06:07:59.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_max_ops": "512", 2024-03-04T06:07:59.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_nid_prealloc": "1024", 2024-03-04T06:07:59.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_onode_map_size": "1024", 2024-03-04T06:07:59.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-04T06:07:59.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_sync_submit_transaction": "false", 2024-03-04T06:07:59.884 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_sync_transaction": "false", 2024-03-04T06:07:59.884 INFO:teuthology.orchestra.run.smithi064.stdout: "librados_thread_count": "2", 2024-03-04T06:07:59.884 INFO:teuthology.orchestra.run.smithi064.stdout: "lockdep": "false", 2024-03-04T06:07:59.884 INFO:teuthology.orchestra.run.smithi064.stdout: "lockdep_force_backtrace": "false", 2024-03-04T06:07:59.884 INFO:teuthology.orchestra.run.smithi064.stdout: "log_coarse_timestamps": "true", 2024-03-04T06:07:59.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-04T06:07:59.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_flush_on_exit": "false", 2024-03-04T06:07:59.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_graylog_host": "127.0.0.1", 2024-03-04T06:07:59.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_graylog_port": "12201", 2024-03-04T06:07:59.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_max_new": "1000", 2024-03-04T06:07:59.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_max_recent": "10000", 2024-03-04T06:07:59.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_stderr_prefix": "", 2024-03-04T06:07:59.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_stop_at_utilization": "0.970000", 2024-03-04T06:07:59.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": "true", 2024-03-04T06:07:59.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_graylog": "false", 2024-03-04T06:07:59.885 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_journald": "false", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_stderr": "false", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_syslog": "false", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "max_rotating_auth_attempts": "10", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_action_on_write_error": "1", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_alternate_name_max": "8192", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_asio_thread_count": "2", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_export_pin": "true", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_dirs": "true", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_interval": "5", 2024-03-04T06:07:59.886 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_interval": "10", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_max": "-1", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_max_until": "-1", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_merge_size": "50", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_midchunk": "0.300000", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_min_start": "0.200000", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_minchunk": "0.001000", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_mode": "0", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_need_max": "1.200000", 2024-03-04T06:07:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_need_min": "0.800000", 2024-03-04T06:07:59.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_overload_epochs": "2", 2024-03-04T06:07:59.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-04T06:07:59.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-04T06:07:59.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_bits": "3", 2024-03-04T06:07:59.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-04T06:07:59.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_size": "10000", 2024-03-04T06:07:59.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-04T06:07:59.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_target_decay": "10.000000", 2024-03-04T06:07:59.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-04T06:07:59.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_grace": "15.000000", 2024-03-04T06:07:59.888 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_interval": "4.000000", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_mid": "0.700000", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_release_free_interval": "10", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_reservation": "0.050000", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_interval": "1", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_threshold": "262144", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-04T06:07:59.889 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_prealloc_inos": "1000", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_connect_bootstrapping": "false", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_damage_table_max_entries": "10000", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_auth_pins": "false", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_frag": "false", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_scatterstat": "false", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_subtrees": "false", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_decay_halflife": "5.000000", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_default_dir_hash": "2", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_defer_session_stale": "true", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_deny_all_reconnect": "false", 2024-03-04T06:07:59.890 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_keys_per_op": "16384", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_max_commit_size": "10", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_max_entries": "0", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_prefetch": "true", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_on_map": "false", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_early_reply": "true", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_enable_op_tracker": "true", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_enforce_unique_name": "true", 2024-03-04T06:07:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-04T06:07:59.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-04T06:07:59.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_random": "true", 2024-03-04T06:07:59.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-04T06:07:59.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-04T06:07:59.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-04T06:07:59.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-04T06:07:59.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-04T06:07:59.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-04T06:07:59.892 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_health_summarize_threshold": "10", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_health_dummy": "false", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_migrator_session_race": "false", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_join_fs": "", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_journal_format": "1", 2024-03-04T06:07:59.893 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_create_at": "0", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_export_at": "0", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_import_at": "0", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_at": "0", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_expire_at": "0", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_replay_at": "0", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_link_at": "0", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_mdstable_at": "0", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_openc_at": "0", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_rename_at": "0", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_shutdown_at": "0", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-04T06:07:59.894 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_event_large_threshold": "524288", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_events_per_segment": "1024", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_max_events": "-1", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_max_segments": "128", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_pause": "false", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_segment_size": "0", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_threshold": "128", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_warn_factor": "2.000000", 2024-03-04T06:07:59.895 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_caps_per_client": "1048576", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_completed_flushes": "100000", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_completed_requests": "100000", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_export_size": "20971520", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_file_recover": "32", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_files": "64", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_ops": "8192", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_snaps_per_dir": "100", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_metrics_update_interval": "2", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_min_caps_per_client": "100", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_min_caps_working_set": "10000", 2024-03-04T06:07:59.896 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_numa_node": "-1", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_complaint_time": "30.000000", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_duration": "600", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_size": "20", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_slow_op_size": "20", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_log_threshold": "5", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_ping_grace": "15", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_ping_interval": "5", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-04T06:07:59.897 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_caps": "30000", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_warning_threshold": "262144", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_replay_interval": "1.000000", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_root_ino_gid": "0", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_root_ino_uid": "0", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-04T06:07:59.898 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_shutdown_check": "0", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_skip_ino": "0", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_max_uid": "4294967294", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_min_uid": "0", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_rstat": "false", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_standby_replay_damaged": "false", 2024-03-04T06:07:59.899 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_symlink_recovery": "true", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_thrash_exports": "0", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_thrash_fragments": "0", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_tick_interval": "5.000000", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_valgrind_exit": "false", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_verify_backtrace": "1", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_verify_scatter": "false", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_wipe_sessions": "false", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool_debug": "false", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_device_bytes": "1073741824", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_page_set": "false", 2024-03-04T06:07:59.900 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_page_size": "65536", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_bytes": "134217728", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_messages": "512", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_disabled_modules": "", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_max_pg_num_change": "128", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mds_bytes": "134217728", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mds_messages": "128", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mon_bytes": "134217728", 2024-03-04T06:07:59.901 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mon_messages": "128", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_osd_bytes": "536870912", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_osd_messages": "8192", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_pool": "true", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_standby_modules": "true", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_stats_period": "5", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_stats_threshold": "5", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_tick_period": "2", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_allow_pool_delete": "true", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_allow_pool_size_one": "true", 2024-03-04T06:07:59.902 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_auth_validate_all_caps": "true", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_bytes": "104857600", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_directed_command_retry": "2", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_parallel": "3", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_log_interval": "1.000000", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_ping_interval": "10.000000", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-04T06:07:59.903 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_target_rank": "-1", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_file_level": "debug", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_file": "true", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_journald": "false", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-04T06:07:59.904 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_bootstrap": "false", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_start": "false", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_trim": "true", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cpu_threads": "4", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_crush_min_required_version": "hammer", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_daemon_bytes": "419430400", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_avail_crit": "5", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_avail_warn": "5", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_size_warn": "16106127360", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-04T06:07:59.905 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_json": "false", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_transactions": "false", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_extra_checks": "false", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_reef": "false", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_squid": "false", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_down_mkfs_grace": "120", 2024-03-04T06:07:59.906 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_election_default_strategy": "1", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_election_timeout": "5.000000", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ping_divisor": "2", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_enable_op_tracker": "true", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_fake_pool_delete": "false", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_force_quorum_join": "false", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_globalid_prealloc": "10000", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_detail_to_clog": "false", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_log_update_period": "5", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_max_detail": "50", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog": "true", 2024-03-04T06:07:59.907 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog_interval": "600", 2024-03-04T06:07:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-04T06:07:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_host": "172.21.15.64", 2024-03-04T06:07:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_host_override": "", 2024-03-04T06:07:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_initial_members": "", 2024-03-04T06:07:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-04T06:07:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-04T06:07:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-04T06:07:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-04T06:07:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-04T06:07:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease": "5.000000", 2024-03-04T06:07:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-04T06:07:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-04T06:07:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_full_interval": "50", 2024-03-04T06:07:59.909 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_max": "10000", 2024-03-04T06:07:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_max_summary": "50", 2024-03-04T06:07:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-04T06:07:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_log_epochs": "500", 2024-03-04T06:07:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-04T06:07:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-04T06:07:59.910 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_osd": "10000", 2024-03-04T06:07:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_pg_per_osd": "10000", 2024-03-04T06:07:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_pool_pg_num": "65536", 2024-03-04T06:07:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-04T06:07:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-04T06:07:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_force_trim_to": "0", 2024-03-04T06:07:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_skip_sanity": "false", 2024-03-04T06:07:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_memory_autotune": "true", 2024-03-04T06:07:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_memory_target": "2147483648", 2024-03-04T06:07:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_beacon_grace": "30", 2024-03-04T06:07:59.911 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_digest_period": "5", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_inactive_grace": "60", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_min_osdmap_epochs": "50", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_complaint_time": "30", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_duration": "600", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_size": "20", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_slow_op_size": "20", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_log_threshold": "5", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-04T06:07:59.912 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-04T06:07:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-04T06:07:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_in": "false", 2024-03-04T06:07:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-04T06:07:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-04T06:07:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-04T06:07:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_cache_size": "500", 2024-03-04T06:07:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-04T06:07:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-04T06:07:59.913 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_down_out_interval": "600", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_force_trim_to": "0", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-04T06:07:59.914 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_down_reporters": "2", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_report_timeout": "900", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-04T06:07:59.915 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-04T06:07:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-04T06:07:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-04T06:07:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_stuck_threshold": "60", 2024-03-04T06:07:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-04T06:07:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-04T06:07:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-04T06:07:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-04T06:07:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-04T06:07:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-04T06:07:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_probe_timeout": "2.000000", 2024-03-04T06:07:59.916 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_max_change": "0.050000", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_max_osds": "4", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_interval": "86400", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_max_keys": "100", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_timeout": "300", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_session_timeout": "300", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_smart_report_timeout": "5", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stat_smooth_intervals": "6", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-04T06:07:59.917 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_pool_min_size": "2", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_pool_size": "4", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_debug": "false", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_provider_kill_at": "0", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_requester_kill_at": "0", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_timeout": "60.000000", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_target_pg_per_osd": "100", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_tick_interval": "5", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_timecheck_interval": "300.000000", 2024-03-04T06:07:59.918 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_use_min_delay_socket": "false", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_older_version_delay": "604800", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_filestore_osds": "false", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_misplaced": "false", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-04T06:07:59.919 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_older_version": "true", 2024-03-04T06:07:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-04T06:07:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-04T06:07:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-03-04T06:07:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-04T06:07:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-04T06:07:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-04T06:07:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-04T06:07:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-04T06:07:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-04T06:07:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": "", 2024-03-04T06:07:59.920 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_tracing_enabled": "false", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_op_threads": "3", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_cm": "false", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_device_name": "", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_dscp": "96", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-04T06:07:59.921 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_local_gid": "", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_port_num": "1", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_sl": "3", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_support_srq": "true", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_type": "ib", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_reap_threshold": "5", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_before_connect": "false", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_ipv4": "true", 2024-03-04T06:07:59.922 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_ipv6": "false", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_msgr1": "true", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_msgr2": "true", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_port_max": "7568", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_port_min": "6800", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_retry_count": "3", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_retry_delay": "5", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_client": "false", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mds": "false", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mgr": "false", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mon": "false", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_osd": "false", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_client_mode": "crc", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_cluster_mode": "crc", 2024-03-04T06:07:59.923 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_cluster_type": "", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_compress_secure": "false", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_connection_idle_timeout": "900", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_connection_ready_timeout": "10", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_crc_data": "true", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_crc_header": "true", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_bad_msg": "false", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_bug": "true", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_old_message": "true", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_skipped_message": "false", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_coremask": "0xF", 2024-03-04T06:07:59.924 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_enable_tso": "true", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hugepages": "", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_lro": "true", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_memory_channel": "4", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_pmd": "", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_port_id": "0", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-04T06:07:59.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dump_on_send": "false", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_initial_backoff": "0.200000", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_max": "1.000000", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_type": "", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_network_congestion": "0", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_socket_failures": "0", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_learn_addr_from_peer": "true", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_max_accept_failures": "4", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_max_backoff": "15.000000", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_client_mode": "crc", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_cluster_mode": "crc", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_service_mode": "crc", 2024-03-04T06:07:59.926 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compress_min_size": "1024", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compress_mode": "none", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_pq_min_cost": "65536", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_public_type": "", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_service_mode": "crc", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_listen_backlog": "512", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_nodelay": "true", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_rcvbuf": "0", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_type": "async+posix", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "no_config_file": "false", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_completion_locks_per_session": "32", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-04T06:07:59.927 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inflight_ops": "1024", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_tick_interval": "5.000000", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_timeout": "10.000000", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "objectstore_blackhole": "false", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "openssl_engine_opts": "", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_delay_time": "5.000000", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_hist_halflife": "1000", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_max_low_ops": "2", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_max_ops": "4", 2024-03-04T06:07:59.928 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_slop": "0.020000", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_asio_thread_count": "2", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_async_recovery_min_cost": "100", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_scan_max": "512", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_scan_min": "64", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_degraded": "false", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_peering": "false", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_unfound": "true", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_beacon_report_interval": "300", 2024-03-04T06:07:59.929 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_duration": "30", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_max_block_size": "67108864", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_blkin_trace_all": "false", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_check_for_log_corruption": "false", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_update_on_start": "true", 2024-03-04T06:07:59.930 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_message_cap": "256", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_message_size_cap": "524288000", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_op_priority": "63", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_watch_timeout": "30", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_command_max_records": "256", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_compact_on_start": "false", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_chooseleaf_type": "0", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_update_on_start": "true", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_update_weight_set": "true", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-04T06:07:59.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_feed_pullee": "-1", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_misdirected_ops": "false", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_no_acting_change": "false", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_no_purge_strays": "false", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_op_order": "false", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-04T06:07:59.932 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_shutdown": "false", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_trim_objects": "false", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_snaps": "false", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_keys": "1024", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_stride": "524288", 2024-03-04T06:07:59.933 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_default_data_pool_replay_window": "5", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_default_notify_timeout": "30", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep": "0.000000", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_discard_disconnected_ops": "true", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_enable_op_tracker": "true", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_info": "true", 2024-03-04T06:07:59.934 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown": "true", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_function_tracing": "false", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_grace": "20", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_interval": "6", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_peers": "10", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_size": "2000", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_stale": "600", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_max_size": "100000", 2024-03-04T06:07:59.935 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_min_size": "1000", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal_size": "5120", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_kill_backfill_at": "0", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_cache_size": "50", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_dedup": "true", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_message_max": "40", 2024-03-04T06:07:59.936 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_share_max_epochs": "40", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_attr_name_len": "100", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_attr_size": "0", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_backfills": "1", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_markdown_count": "5", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_markdown_period": "600", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_name_len": "2048", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_namespace_len": "256", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_size": "134217728", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_blocked_by": "16", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_log_entries": "10000", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-04T06:07:59.937 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pgls": "1024", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_push_cost": "8388608", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_push_objects": "10", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_scrubs": "3", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_trimming_pgs": "2", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_write_op_reply_len": "64", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_write_size": "90", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-04T06:07:59.938 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_profile": "balanced", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_base": "805306368", 2024-03-04T06:07:59.939 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_cache_min": "134217728", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target": "4294967296", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target_autotune": "false", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_min_pg_log_entries": "250", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_min_recovery_priority": "0", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_report_interval": "5", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-04T06:07:59.940 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_num_cache_shards": "32", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_num_op_tracker_shard": "32", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_auto_affinity": "true", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_node": "-1", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_prefer_iface": "true", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objecter_finishers": "1", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore": "bluestore", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore_fuse": "false", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore_tracing": "false", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_complaint_time": "30.000000", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_duration": "600", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_size": "20", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_slow_op_size": "20", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_log_threshold": "5", 2024-03-04T06:07:59.941 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards": "0", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards_hdd": "5", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards_ssd": "8", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_pq_min_cost": "65536", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_queue_cut_off": "high", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_thread_timeout": "15", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_open_classes_on_start": "true", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_os_flags": "0", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_peering_op_priority": "255", 2024-03-04T06:07:59.942 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_delete_cost": "1048576", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_delete_priority": "5", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_trim_max": "10000", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_trim_min": "100", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-04T06:07:59.943 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_crimson": "true", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flags": "0", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_min_size": "2", 2024-03-04T06:07:59.944 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pg_num": "32", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pgp_num": "0", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_read_ratio": "70", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_size": "3", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_type": "replicated", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_push_per_object_cost": "1000", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recover_clone_overlap": "true", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_cost": "20971520", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active": "0", 2024-03-04T06:07:59.945 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_single_start": "1", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_op_priority": "3", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_priority": "5", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep": "0.000000", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_repair_during_recovery": "false", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_requested_scrub_priority": "120", 2024-03-04T06:07:59.946 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_auto_repair": "false", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_begin_hour": "0", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_begin_week_day": "0", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_chunk_max": "25", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_chunk_min": "5", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_cost": "52428800", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_during_recovery": "false", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_end_hour": "0", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_end_week_day": "0", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_event_cost": "4096", 2024-03-04T06:07:59.947 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_invalid_stats": "true", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_max_preemptions": "5", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_priority": "5", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_sleep": "0.000000", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-04T06:07:59.948 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_skip_data_digest": "false", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_smart_report_timeout": "5", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_cost": "1048576", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_priority": "5", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_target_transaction_size": "30", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-04T06:07:59.949 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tracing": "false", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_use_stale_snap": "false", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "osdc_blkin_trace_all": "false", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_kill_at": "0", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_max_join_drift": "10", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_min": "500", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_min_wait": "0.050000", 2024-03-04T06:07:59.950 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_propose_interval": "1.000000", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_max": "500", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_min": "10", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_stash_full_interval": "25", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_trim_max": "500", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_trim_min": "250", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "perf": "true", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "-", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrv": "", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "public_bind_addr": "-", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "public_network": "", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "public_network_interface": "", 2024-03-04T06:07:59.951 INFO:teuthology.orchestra.run.smithi064.stdout: "qat_compressor_enabled": "false", 2024-03-04T06:07:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "qat_compressor_session_max_number": "256", 2024-03-04T06:07:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_mon_op_timeout": "0", 2024-03-04T06:07:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_osd_op_timeout": "0", 2024-03-04T06:07:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_tracing": "false", 2024-03-04T06:07:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_atime_update_interval": "60", 2024-03-04T06:07:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-04T06:07:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_balance_parent_reads": "false", 2024-03-04T06:07:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_balance_snap_reads": "false", 2024-03-04T06:07:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blkin_trace_all": "false", 2024-03-04T06:07:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-04T06:07:59.952 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache": "true", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_policy": "writearound", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_size": "33554432", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_clone_copy_on_read": "false", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_compression_hint": "none", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_concurrent_management_ops": "10", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_clone_format": "auto", 2024-03-04T06:07:59.953 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_data_pool": "", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_features": "61", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_format": "2", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_map_options": "", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_order": "22", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_pool": "rbd", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_stripe_count": "0", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_stripe_unit": "0", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_enable_alloc_hint": "true", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_io_scheduler": "simple", 2024-03-04T06:07:59.954 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_order": "24", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_pool": "", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_splay_width": "4", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_localize_parent_reads": "false", 2024-03-04T06:07:59.955 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_localize_snap_reads": "false", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-04T06:07:59.956 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-04T06:07:59.957 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mtime_update_interval": "60", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_non_blocking_aio": "true", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_op_thread_timeout": "60", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_op_threads": "1", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_parent_cache_enabled": "false", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_plugins": "", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_burst": "0", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_limit": "0", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_exclude_ops": "0", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_burst": "0", 2024-03-04T06:07:59.958 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_limit": "0", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-04T06:07:59.959 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_read_from_replica_policy": "default", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_skip_partial_discard": "true", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_tracing": "false", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_validate_names": "true", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_validate_pool": "true", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "restapi_base_url": "", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "restapi_log_level": "", 2024-03-04T06:07:59.960 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_acl_grants_max_num": "100", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_admin_entry": "admin", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_backend_store": "rados", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_barbican_url": "", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_beast_enable_async": "true", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_counters_cache": "false", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-04T06:07:59.961 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_enabled": "true", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_expiry_interval": "900", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_lru_size": "10000", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_config_store": "rados", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_content_length_compat": "false", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_copy_obj_progress": "true", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cors_rules_max_num": "100", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cross_domain_policy": "", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_addr": "", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-04T06:07:59.962 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_password": "", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_username": "", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_require_ssl": "true", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-04T06:07:59.963 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_addr": "", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_auth": "token", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_namespace": "", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_prefix": "", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-04T06:07:59.964 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_token_file": "", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_buffersize": "524288", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_low_speed_time": "300", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-04T06:07:59.965 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_port": "6379", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_changes_size": "1000", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_num_shards": "128", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_window": "30", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-04T06:07:59.966 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-04T06:07:59.967 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dns_name": "", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dns_s3website_name": "", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dynamic_resharding": "true", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_gc_threads": "true", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_lc_threads": "true", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_ops_log": "false", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_quota_threads": "true", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_static_website": "false", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_usage_log": "false", 2024-03-04T06:07:59.968 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enforce_swift_acls": "true", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_exit_timeout_secs": "120", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_expose_bucket": "false", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_extended_http_attrs": "", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_filter": "none", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_frontends": "beast port=7480", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_deferred": "50", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_objs": "32", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_processor_period": "3600", 2024-03-04T06:07:59.969 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_init_timeout": "300", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_kafka_connection_idle": "30", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_domain": "", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_password": "", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_password_path": "", 2024-03-04T06:07:59.970 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_project": "", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_tenant": "", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_token": "", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_token_path": "", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_user": "", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_api_version": "2", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_domain": "", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_password": "", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_project": "", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_user": "", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-04T06:07:59.971 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_url": "", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_debug_interval": "-1", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_lock_max_time": "90", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_objs": "32", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_rules": "1000", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_worker": "3", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_thread_delay": "0", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_dnattr": "uid", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_searchfilter": "", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-04T06:07:59.972 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_http_headers": "", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_object_name_utc": "false", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attr_name_len": "0", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attr_size": "0", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_chunk_size": "4194304", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-04T06:07:59.973 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_listing_results": "1000", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_notify_retries": "10", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_put_param_size": "1048576", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_put_size": "5368709120", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_slo_entries": "1000", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_md_log_max_shards": "64", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_mp_lock_max_time": "600", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-04T06:07:59.974 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_max_gc": "300", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_num_async_rados_threads": "32", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_num_control_oids": "8", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_numa_node": "-1", 2024-03-04T06:07:59.975 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_chunk_size": "100", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_gc_interval": "600", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_thread_timeout": "600", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_tracing": "false", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_token": "", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_url": "", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_verify_ssl": "true", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-04T06:07:59.976 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_rados": "false", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_socket_path": "", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_push_interval": "2.000000", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_lanes": "3", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_partitions": "3", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_print_continue": "true", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-04T06:07:59.977 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_tracing": "false", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm": "", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm_id": "", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_region": "", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_request_uri": "", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_batch_size": "64", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-04T06:07:59.978 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_max_aio": "128", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_num_logs": "16", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_thread_interval": "600", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_resolve_cname": "false", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_run_sync_thread": "true", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_scheduler_type": "throttler", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_script_uri": "", 2024-03-04T06:07:59.979 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_service_provider_name": "", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_client_id": "", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_client_secret": "", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_entry": "sts", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_key": "sts", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_min_session_duration": "900", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_token_introspection_url": "", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_account_in_url": "false", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_auth_entry": "auth", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_auth_url": "", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_custom_header": "", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_need_stats": "true", 2024-03-04T06:07:59.980 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_tenant_name": "", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_token_expiration": "86400", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_url": "", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_url_prefix": "swift", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_lease_period": "120", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-04T06:07:59.981 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_thread_pool_size": "512", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_comment": "", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_createby": "", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_encoding": "", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_flag": "false", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_origin": "", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_tracker": "", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_trust_forwarded_https": "false", 2024-03-04T06:07:59.982 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_max_shards": "32", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_max_user_shards": "1", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_use_opa_authz": "false", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_counters_cache": "false", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_max_buckets": "1000", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_policies_max_num": "100", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-04T06:07:59.983 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_unique_email": "true", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_verify_ssl": "true", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone": "", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone_id": "", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup": "", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup_id": "", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgwlc_auto_session_clear": "true", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_block_size": "4096", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-04T06:07:59.984 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_size": "536870912", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_index_type": "binary_search", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_partition_filters": "false", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_perf": "false", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-04T06:07:59.985 INFO:teuthology.orchestra.run.smithi064.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "run_dir": "/var/run/ceph", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_block_create": "true", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_cache_lru_size": "67108864", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_default_max_object_size": "16777216", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_device_size": "53687091200", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_capacity": "16", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_main_device_type": "SSD", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-04T06:07:59.986 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_segment_size": "67108864", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "set_keepcaps": "false", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "setgroup": "", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "setuser": "", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "setuser_match_path": "", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "thp": "false", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "threadpool_default_timeout": "60", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "throttler_perf_counter": "true" 2024-03-04T06:07:59.987 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-04T06:07:59.989 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-04T06:08:00.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.092+0000 7f9219b07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:00.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.096+0000 7f9219b07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:00.110 INFO:teuthology.orchestra.run.smithi064.stdout:{} 2024-03-04T06:08:00.110 INFO:teuthology.orchestra.run.smithi064.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-04T06:08:00.120 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2024-03-04T06:08:00.134 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-04T06:08:00.134 DEBUG:teuthology.orchestra.run.smithi066:> type ceph-objectstore-tool 2024-03-04T06:08:00.153 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-04T06:08:00.153 INFO:teuthology.run_tasks:Running task rados... 2024-03-04T06:08:00.163 INFO:tasks.rados:Beginning rados... 2024-03-04T06:08:00.163 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-03-04T06:08:00.170 INFO:tasks.rados:joining rados 2024-03-04T06:08:00.171 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-04T06:08:00.171 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:08:00.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:08:00.171 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-04T06:08:00.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:00.172 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-03-04T06:08:00.172 INFO:tasks.rados:starting run 0 out of 1 2024-03-04T06:08:00.172 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-04T06:08:00.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-03-04T06:08:00.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.284+0000 7f718c79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:00.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.288+0000 7f718c79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:00.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.283+0000 7f5eaeedd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:00.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.283+0000 7f111da6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:00.293 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.293+0000 7f5eaeedd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:00.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.294+0000 7f111da6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:00.777 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:08:00.777 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_0' created 2024-03-04T06:08:00.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-04T06:08:00.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:08:00.951 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.945+0000 7fe06c39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.945+0000 7f2af8b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:00.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.953+0000 7fe06c39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:00.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:00.956+0000 7f2af8b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:01.780 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:08:01.781 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-04T06:08:01.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:08:01.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:01.940+0000 7f3978e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:01.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:01.948+0000 7f3978e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:02.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:02.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:02.470+0000 7f751f6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:02.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:02.475+0000 7f751f6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:02.783 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:08:02.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:08:02.805 DEBUG:teuthology.orchestra.run.smithi064:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0 2024-03-04T06:08:02.875 INFO:tasks.rados.rados.0.smithi064.stdout:adding op weight read -> 100 2024-03-04T06:08:02.875 INFO:tasks.rados.rados.0.smithi064.stdout:adding op weight write -> 50 2024-03-04T06:08:02.875 INFO:tasks.rados.rados.0.smithi064.stdout:adding op weight delete -> 50 2024-03-04T06:08:02.875 INFO:tasks.rados.rados.0.smithi064.stdout:adding op weight snap_create -> 50 2024-03-04T06:08:02.875 INFO:tasks.rados.rados.0.smithi064.stdout:adding op weight snap_remove -> 50 2024-03-04T06:08:02.875 INFO:tasks.rados.rados.0.smithi064.stdout:adding op weight write_excl -> 50 2024-03-04T06:08:02.875 INFO:tasks.rados.rados.0.smithi064.stdout:ceph version 19.0.0-1550-g633e69c9 (633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd) squid (dev) 2024-03-04T06:08:02.875 INFO:tasks.rados.rados.0.smithi064.stdout:Configuration: 2024-03-04T06:08:02.875 INFO:tasks.rados.rados.0.smithi064.stdout: Number of operations: 4000 2024-03-04T06:08:02.876 INFO:tasks.rados.rados.0.smithi064.stdout: Number of objects: 50 2024-03-04T06:08:02.876 INFO:tasks.rados.rados.0.smithi064.stdout: Max in flight operations: 16 2024-03-04T06:08:02.876 INFO:tasks.rados.rados.0.smithi064.stdout: Object size (in bytes): 4000000 2024-03-04T06:08:02.876 INFO:tasks.rados.rados.0.smithi064.stdout: Write stride min: 400000 2024-03-04T06:08:02.876 INFO:tasks.rados.rados.0.smithi064.stdout: Write stride max: 800000 2024-03-04T06:08:02.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:02.906+0000 7fd0c979d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:02.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:02.914+0000 7fd0c979d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:03.784 INFO:tasks.rados.rados.0.smithi064.stdout:1: write initial oid 1 2024-03-04T06:08:03.784 INFO:tasks.rados.rados.0.smithi064.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-03-04T06:08:03.784 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:08:03.790 INFO:tasks.rados.rados.0.smithi064.stdout:1: writing smithi06431743-1 from 416807 to 892056 tid 1 2024-03-04T06:08:03.799 INFO:tasks.rados.rados.0.smithi064.stdout:1: writing smithi06431743-1 from 1542129 to 2016807 tid 2 2024-03-04T06:08:03.799 INFO:tasks.rados.rados.0.smithi064.stdout:2: write initial oid 2 2024-03-04T06:08:03.799 INFO:tasks.rados.rados.0.smithi064.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-03-04T06:08:03.803 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:08:03.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:08:03.804 INFO:tasks.rados.rados.0.smithi064.stdout:2: writing smithi06431743-2 from 416807 to 892056 tid 1 2024-03-04T06:08:03.811 INFO:tasks.rados.rados.0.smithi064.stdout:2: writing smithi06431743-2 from 1542129 to 2016807 tid 2 2024-03-04T06:08:03.811 INFO:tasks.rados.rados.0.smithi064.stdout:3: write initial oid 3 2024-03-04T06:08:03.811 INFO:tasks.rados.rados.0.smithi064.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-03-04T06:08:03.817 INFO:tasks.rados.rados.0.smithi064.stdout:3: writing smithi06431743-3 from 433614 to 984112 tid 1 2024-03-04T06:08:03.824 INFO:tasks.rados.rados.0.smithi064.stdout:3: writing smithi06431743-3 from 1600611 to 2033614 tid 2 2024-03-04T06:08:03.824 INFO:tasks.rados.rados.0.smithi064.stdout:4: write initial oid 4 2024-03-04T06:08:03.824 INFO:tasks.rados.rados.0.smithi064.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-03-04T06:08:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:4: writing smithi06431743-4 from 450421 to 1076168 tid 1 2024-03-04T06:08:03.838 INFO:tasks.rados.rados.0.smithi064.stdout:4: writing smithi06431743-4 from 1659093 to 2050421 tid 2 2024-03-04T06:08:03.838 INFO:tasks.rados.rados.0.smithi064.stdout:5: write initial oid 5 2024-03-04T06:08:03.838 INFO:tasks.rados.rados.0.smithi064.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-03-04T06:08:03.846 INFO:tasks.rados.rados.0.smithi064.stdout:5: writing smithi06431743-5 from 467228 to 1168224 tid 1 2024-03-04T06:08:03.852 INFO:tasks.rados.rados.0.smithi064.stdout:5: writing smithi06431743-5 from 1717575 to 2067228 tid 2 2024-03-04T06:08:03.852 INFO:tasks.rados.rados.0.smithi064.stdout:6: write initial oid 6 2024-03-04T06:08:03.852 INFO:tasks.rados.rados.0.smithi064.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-03-04T06:08:03.860 INFO:tasks.rados.rados.0.smithi064.stdout:6: writing smithi06431743-6 from 484035 to 1260280 tid 1 2024-03-04T06:08:03.865 INFO:tasks.rados.rados.0.smithi064.stdout:6: writing smithi06431743-6 from 2059704 to 2084035 tid 2 2024-03-04T06:08:03.865 INFO:tasks.rados.rados.0.smithi064.stdout:7: write initial oid 7 2024-03-04T06:08:03.865 INFO:tasks.rados.rados.0.smithi064.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-03-04T06:08:03.871 INFO:tasks.rados.rados.0.smithi064.stdout:7: writing smithi06431743-7 from 500842 to 952336 tid 1 2024-03-04T06:08:03.878 INFO:tasks.rados.rados.0.smithi064.stdout:7: writing smithi06431743-7 from 1718186 to 2100842 tid 2 2024-03-04T06:08:03.878 INFO:tasks.rados.rados.0.smithi064.stdout:8: write initial oid 8 2024-03-04T06:08:03.878 INFO:tasks.rados.rados.0.smithi064.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-03-04T06:08:03.884 INFO:tasks.rados.rados.0.smithi064.stdout:8: writing smithi06431743-8 from 517649 to 1044392 tid 1 2024-03-04T06:08:03.891 INFO:tasks.rados.rados.0.smithi064.stdout:8: writing smithi06431743-8 from 1776668 to 2117649 tid 2 2024-03-04T06:08:03.891 INFO:tasks.rados.rados.0.smithi064.stdout:9: write initial oid 9 2024-03-04T06:08:03.891 INFO:tasks.rados.rados.0.smithi064.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-03-04T06:08:03.900 INFO:tasks.rados.rados.0.smithi064.stdout:9: writing smithi06431743-9 from 534456 to 1252801 tid 1 2024-03-04T06:08:03.905 INFO:tasks.rados.rados.0.smithi064.stdout:9: writing smithi06431743-9 from 1951503 to 2134456 tid 2 2024-03-04T06:08:03.905 INFO:tasks.rados.rados.0.smithi064.stdout:10: write initial oid 10 2024-03-04T06:08:03.905 INFO:tasks.rados.rados.0.smithi064.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-03-04T06:08:03.913 INFO:tasks.rados.rados.0.smithi064.stdout:10: writing smithi06431743-10 from 551263 to 1344857 tid 1 2024-03-04T06:08:03.919 INFO:tasks.rados.rados.0.smithi064.stdout:10: writing smithi06431743-10 from 1893632 to 2151263 tid 2 2024-03-04T06:08:03.919 INFO:tasks.rados.rados.0.smithi064.stdout:11: write initial oid 11 2024-03-04T06:08:03.919 INFO:tasks.rados.rados.0.smithi064.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-03-04T06:08:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:11: writing smithi06431743-11 from 568070 to 1036913 tid 1 2024-03-04T06:08:03.932 INFO:tasks.rados.rados.0.smithi064.stdout:11: writing smithi06431743-11 from 1552114 to 2168070 tid 2 2024-03-04T06:08:03.932 INFO:tasks.rados.rados.0.smithi064.stdout:12: write initial oid 12 2024-03-04T06:08:03.933 INFO:tasks.rados.rados.0.smithi064.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-03-04T06:08:03.940 INFO:tasks.rados.rados.0.smithi064.stdout:12: writing smithi06431743-12 from 584877 to 1128969 tid 1 2024-03-04T06:08:03.947 INFO:tasks.rados.rados.0.smithi064.stdout:12: writing smithi06431743-12 from 1610596 to 2172599 tid 2 2024-03-04T06:08:03.947 INFO:tasks.rados.rados.0.smithi064.stdout:13: write initial oid 13 2024-03-04T06:08:03.947 INFO:tasks.rados.rados.0.smithi064.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-03-04T06:08:03.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:03.942+0000 7f5624d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:03.954 INFO:tasks.rados.rados.0.smithi064.stdout:13: writing smithi06431743-13 from 601684 to 1221025 tid 1 2024-03-04T06:08:03.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:03.956+0000 7f5624d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:03.961 INFO:tasks.rados.rados.0.smithi064.stdout:13: writing smithi06431743-13 from 1669078 to 2201684 tid 2 2024-03-04T06:08:03.961 INFO:tasks.rados.rados.0.smithi064.stdout:14: write initial oid 14 2024-03-04T06:08:03.961 INFO:tasks.rados.rados.0.smithi064.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-03-04T06:08:03.970 INFO:tasks.rados.rados.0.smithi064.stdout:14: writing smithi06431743-14 from 618491 to 1313081 tid 1 2024-03-04T06:08:03.976 INFO:tasks.rados.rados.0.smithi064.stdout:14: writing smithi06431743-14 from 2011207 to 2218491 tid 2 2024-03-04T06:08:03.976 INFO:tasks.rados.rados.0.smithi064.stdout:15: write initial oid 15 2024-03-04T06:08:03.976 INFO:tasks.rados.rados.0.smithi064.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-03-04T06:08:03.984 INFO:tasks.rados.rados.0.smithi064.stdout:15: writing smithi06431743-15 from 635298 to 1405137 tid 1 2024-03-04T06:08:03.989 INFO:tasks.rados.rados.0.smithi064.stdout:15: writing smithi06431743-15 from 2069689 to 2235298 tid 2 2024-03-04T06:08:03.989 INFO:tasks.rados.rados.0.smithi064.stdout:16: write initial oid 16 2024-03-04T06:08:03.989 INFO:tasks.rados.rados.0.smithi064.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-03-04T06:08:03.995 INFO:tasks.rados.rados.0.smithi064.stdout:16: writing smithi06431743-16 from 652105 to 1097193 tid 1 2024-03-04T06:08:04.003 INFO:tasks.rados.rados.0.smithi064.stdout:16: writing smithi06431743-16 from 1728171 to 2181159 tid 2 2024-03-04T06:08:04.003 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 16 2024-03-04T06:08:04.003 INFO:tasks.rados.rados.0.smithi064.stdout:1: finishing write tid 1 to smithi06431743-1 2024-03-04T06:08:04.003 INFO:tasks.rados.rados.0.smithi064.stdout:3: finishing write tid 1 to smithi06431743-3 2024-03-04T06:08:04.003 INFO:tasks.rados.rados.0.smithi064.stdout:2: finishing write tid 1 to smithi06431743-2 2024-03-04T06:08:04.003 INFO:tasks.rados.rados.0.smithi064.stdout:1: finishing write tid 2 to smithi06431743-1 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:3: finishing write tid 2 to smithi06431743-3 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:7: finishing write tid 1 to smithi06431743-7 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:6: finishing write tid 1 to smithi06431743-6 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:2: finishing write tid 2 to smithi06431743-2 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:4: finishing write tid 1 to smithi06431743-4 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:1: finishing write tid 3 to smithi06431743-1 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:1: finishing write tid 4 to smithi06431743-1 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:1: oid 1 updating version 0 to 2 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:1: oid 1 version 2 is already newer than 1 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:1: oid 1 updating version 2 to 3 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:2: finishing write tid 3 to smithi06431743-2 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:8: finishing write tid 1 to smithi06431743-8 2024-03-04T06:08:04.004 INFO:tasks.rados.rados.0.smithi064.stdout:2: finishing write tid 4 to smithi06431743-2 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:2: oid 2 updating version 0 to 2 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:2: oid 2 version 2 is already newer than 1 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:2: oid 2 updating version 2 to 3 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:4: finishing write tid 2 to smithi06431743-4 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:3: finishing write tid 3 to smithi06431743-3 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:3: finishing write tid 4 to smithi06431743-3 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:3: oid 3 updating version 0 to 2 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:3: oid 3 version 2 is already newer than 1 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:3: oid 3 updating version 2 to 3 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:7: finishing write tid 2 to smithi06431743-7 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:6: finishing write tid 2 to smithi06431743-6 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:9: finishing write tid 1 to smithi06431743-9 2024-03-04T06:08:04.005 INFO:tasks.rados.rados.0.smithi064.stdout:12: finishing write tid 1 to smithi06431743-12 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:7: finishing write tid 3 to smithi06431743-7 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:10: finishing write tid 1 to smithi06431743-10 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:4: finishing write tid 3 to smithi06431743-4 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:8: finishing write tid 2 to smithi06431743-8 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:7: finishing write tid 4 to smithi06431743-7 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:7: oid 7 updating version 0 to 1 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:7: oid 7 updating version 1 to 3 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:7: oid 7 version 3 is already newer than 3 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:6: finishing write tid 3 to smithi06431743-6 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:4: finishing write tid 4 to smithi06431743-4 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:4: oid 4 updating version 0 to 1 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:4: oid 4 updating version 1 to 2 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:4: oid 4 updating version 2 to 3 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-04T06:08:04.006 INFO:tasks.rados.rados.0.smithi064.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:6: finishing write tid 4 to smithi06431743-6 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:6: oid 6 updating version 0 to 3 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:6: oid 6 version 3 is already newer than 1 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:6: oid 6 version 3 is already newer than 2 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:9: finishing write tid 2 to smithi06431743-9 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:11: finishing write tid 1 to smithi06431743-11 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:8: finishing write tid 3 to smithi06431743-8 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:8: finishing write tid 4 to smithi06431743-8 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:8: oid 8 updating version 0 to 1 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:8: oid 8 updating version 1 to 3 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:8: oid 8 version 3 is already newer than 2 2024-03-04T06:08:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-04T06:08:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-03-04T06:08:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:10: finishing write tid 2 to smithi06431743-10 2024-03-04T06:08:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:1: done (15 left) 2024-03-04T06:08:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:2: done (14 left) 2024-03-04T06:08:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:3: done (13 left) 2024-03-04T06:08:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:4: done (12 left) 2024-03-04T06:08:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:6: done (11 left) 2024-03-04T06:08:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:7: done (10 left) 2024-03-04T06:08:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:8: done (9 left) 2024-03-04T06:08:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:17: write initial oid 17 2024-03-04T06:08:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-03-04T06:08:04.011 INFO:tasks.rados.rados.0.smithi064.stdout:17: writing smithi06431743-17 from 668912 to 1305602 tid 1 2024-03-04T06:08:04.017 INFO:tasks.rados.rados.0.smithi064.stdout:17: writing smithi06431743-17 from 1903006 to 2268912 tid 2 2024-03-04T06:08:04.017 INFO:tasks.rados.rados.0.smithi064.stdout:18: write initial oid 18 2024-03-04T06:08:04.017 INFO:tasks.rados.rados.0.smithi064.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-03-04T06:08:04.025 INFO:tasks.rados.rados.0.smithi064.stdout:18: writing smithi06431743-18 from 685719 to 1397658 tid 1 2024-03-04T06:08:04.031 INFO:tasks.rados.rados.0.smithi064.stdout:18: writing smithi06431743-18 from 1845135 to 2285719 tid 2 2024-03-04T06:08:04.031 INFO:tasks.rados.rados.0.smithi064.stdout:19: write initial oid 19 2024-03-04T06:08:04.031 INFO:tasks.rados.rados.0.smithi064.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-03-04T06:08:04.041 INFO:tasks.rados.rados.0.smithi064.stdout:19: writing smithi06431743-19 from 702526 to 1489714 tid 1 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:19: writing smithi06431743-19 from 1903617 to 2302526 tid 2 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:12: finishing write tid 2 to smithi06431743-12 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:5: finishing write tid 1 to smithi06431743-5 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:9: finishing write tid 3 to smithi06431743-9 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:9: finishing write tid 4 to smithi06431743-9 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:9: oid 9 updating version 0 to 3 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:9: oid 9 version 3 is already newer than 1 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:9: oid 9 version 3 is already newer than 2 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:10: finishing write tid 3 to smithi06431743-10 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:10: finishing write tid 4 to smithi06431743-10 2024-03-04T06:08:04.047 INFO:tasks.rados.rados.0.smithi064.stdout:10: oid 10 updating version 0 to 3 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:10: oid 10 version 3 is already newer than 3 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:10: oid 10 version 3 is already newer than 1 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:16: finishing write tid 1 to smithi06431743-16 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:11: finishing write tid 2 to smithi06431743-11 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:15: finishing write tid 1 to smithi06431743-15 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:14: finishing write tid 1 to smithi06431743-14 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:12: finishing write tid 3 to smithi06431743-12 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:5: finishing write tid 2 to smithi06431743-5 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:5: finishing write tid 3 to smithi06431743-5 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:17: finishing write tid 1 to smithi06431743-17 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:12: finishing write tid 4 to smithi06431743-12 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:12: oid 12 updating version 0 to 4 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:12: oid 12 updating version 4 to 5 2024-03-04T06:08:04.048 INFO:tasks.rados.rados.0.smithi064.stdout:12: oid 12 updating version 5 to 6 2024-03-04T06:08:04.049 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-04T06:08:04.049 INFO:tasks.rados.rados.0.smithi064.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-03-04T06:08:04.049 INFO:tasks.rados.rados.0.smithi064.stdout:9: done (11 left) 2024-03-04T06:08:04.049 INFO:tasks.rados.rados.0.smithi064.stdout:10: done (10 left) 2024-03-04T06:08:04.049 INFO:tasks.rados.rados.0.smithi064.stdout:12: done (9 left) 2024-03-04T06:08:04.049 INFO:tasks.rados.rados.0.smithi064.stdout:20: write initial oid 20 2024-03-04T06:08:04.050 INFO:tasks.rados.rados.0.smithi064.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-03-04T06:08:04.052 INFO:tasks.rados.rados.0.smithi064.stdout:20: writing smithi06431743-20 from 719333 to 1181770 tid 1 2024-03-04T06:08:04.059 INFO:tasks.rados.rados.0.smithi064.stdout:20: writing smithi06431743-20 from 1962099 to 2319333 tid 2 2024-03-04T06:08:04.059 INFO:tasks.rados.rados.0.smithi064.stdout:21: write initial oid 21 2024-03-04T06:08:04.059 INFO:tasks.rados.rados.0.smithi064.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-03-04T06:08:04.065 INFO:tasks.rados.rados.0.smithi064.stdout:21: writing smithi06431743-21 from 736140 to 1273826 tid 1 2024-03-04T06:08:04.073 INFO:tasks.rados.rados.0.smithi064.stdout:21: writing smithi06431743-21 from 2020581 to 2336140 tid 2 2024-03-04T06:08:04.073 INFO:tasks.rados.rados.0.smithi064.stdout:22: write initial oid 22 2024-03-04T06:08:04.073 INFO:tasks.rados.rados.0.smithi064.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-03-04T06:08:04.082 INFO:tasks.rados.rados.0.smithi064.stdout:22: writing smithi06431743-22 from 752947 to 1365882 tid 1 2024-03-04T06:08:04.088 INFO:tasks.rados.rados.0.smithi064.stdout:22: writing smithi06431743-22 from 1962710 to 2352947 tid 2 2024-03-04T06:08:04.088 INFO:tasks.rados.rados.0.smithi064.stdout:23: write initial oid 23 2024-03-04T06:08:04.088 INFO:tasks.rados.rados.0.smithi064.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-03-04T06:08:04.097 INFO:tasks.rados.rados.0.smithi064.stdout:23: writing smithi06431743-23 from 769754 to 1457938 tid 1 2024-03-04T06:08:04.102 INFO:tasks.rados.rados.0.smithi064.stdout:23: writing smithi06431743-23 from 2021192 to 2369754 tid 2 2024-03-04T06:08:04.102 INFO:tasks.rados.rados.0.smithi064.stdout:24: write initial oid 24 2024-03-04T06:08:04.102 INFO:tasks.rados.rados.0.smithi064.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-03-04T06:08:04.110 INFO:tasks.rados.rados.0.smithi064.stdout:24: writing smithi06431743-24 from 786561 to 1266347 tid 1 2024-03-04T06:08:04.116 INFO:tasks.rados.rados.0.smithi064.stdout:24: writing smithi06431743-24 from 1796027 to 2263691 tid 2 2024-03-04T06:08:04.116 INFO:tasks.rados.rados.0.smithi064.stdout:25: write initial oid 25 2024-03-04T06:08:04.116 INFO:tasks.rados.rados.0.smithi064.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-03-04T06:08:04.123 INFO:tasks.rados.rados.0.smithi064.stdout:25: writing smithi06431743-25 from 403368 to 958403 tid 1 2024-03-04T06:08:04.133 INFO:tasks.rados.rados.0.smithi064.stdout:25: writing smithi06431743-25 from 1454509 to 2182184 tid 2 2024-03-04T06:08:04.134 INFO:tasks.rados.rados.0.smithi064.stdout:26: write initial oid 26 2024-03-04T06:08:04.134 INFO:tasks.rados.rados.0.smithi064.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-03-04T06:08:04.142 INFO:tasks.rados.rados.0.smithi064.stdout:26: writing smithi06431743-26 from 420175 to 1050459 tid 1 2024-03-04T06:08:04.149 INFO:tasks.rados.rados.0.smithi064.stdout:26: writing smithi06431743-26 from 1796638 to 2267971 tid 2 2024-03-04T06:08:04.149 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 16 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:5: finishing write tid 4 to smithi06431743-5 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:5: oid 5 updating version 0 to 6 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:5: oid 5 version 6 is already newer than 4 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:5: oid 5 version 6 is already newer than 6 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 6 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:14: finishing write tid 2 to smithi06431743-14 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:16: finishing write tid 2 to smithi06431743-16 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:11: finishing write tid 3 to smithi06431743-11 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:11: finishing write tid 4 to smithi06431743-11 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:11: oid 11 updating version 0 to 2 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:11: oid 11 updating version 2 to 3 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:11: oid 11 version 3 is already newer than 1 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-03-04T06:08:04.150 INFO:tasks.rados.rados.0.smithi064.stdout:15: finishing write tid 2 to smithi06431743-15 2024-03-04T06:08:04.151 INFO:tasks.rados.rados.0.smithi064.stdout:18: finishing write tid 1 to smithi06431743-18 2024-03-04T06:08:04.151 INFO:tasks.rados.rados.0.smithi064.stdout:16: finishing write tid 3 to smithi06431743-16 2024-03-04T06:08:04.151 INFO:tasks.rados.rados.0.smithi064.stdout:5: done (15 left) 2024-03-04T06:08:04.151 INFO:tasks.rados.rados.0.smithi064.stdout:11: done (14 left) 2024-03-04T06:08:04.151 INFO:tasks.rados.rados.0.smithi064.stdout:27: write initial oid 27 2024-03-04T06:08:04.151 INFO:tasks.rados.rados.0.smithi064.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-03-04T06:08:04.156 INFO:tasks.rados.rados.0.smithi064.stdout:27: writing smithi06431743-27 from 436982 to 1142515 tid 1 2024-03-04T06:08:04.164 INFO:tasks.rados.rados.0.smithi064.stdout:27: writing smithi06431743-27 from 1855120 to 2436982 tid 2 2024-03-04T06:08:04.164 INFO:tasks.rados.rados.0.smithi064.stdout:28: write initial oid 28 2024-03-04T06:08:04.164 INFO:tasks.rados.rados.0.smithi064.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-03-04T06:08:04.173 INFO:tasks.rados.rados.0.smithi064.stdout:28: writing smithi06431743-28 from 453789 to 1234571 tid 1 2024-03-04T06:08:04.180 INFO:tasks.rados.rados.0.smithi064.stdout:28: writing smithi06431743-28 from 1913602 to 2388604 tid 2 2024-03-04T06:08:04.180 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 16 2024-03-04T06:08:04.180 INFO:tasks.rados.rados.0.smithi064.stdout:17: finishing write tid 2 to smithi06431743-17 2024-03-04T06:08:04.180 INFO:tasks.rados.rados.0.smithi064.stdout:15: finishing write tid 3 to smithi06431743-15 2024-03-04T06:08:04.180 INFO:tasks.rados.rados.0.smithi064.stdout:15: finishing write tid 4 to smithi06431743-15 2024-03-04T06:08:04.180 INFO:tasks.rados.rados.0.smithi064.stdout:15: oid 15 updating version 0 to 4 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:15: oid 15 updating version 4 to 6 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:15: oid 15 version 6 is already newer than 6 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:16: finishing write tid 4 to smithi06431743-16 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:16: oid 16 updating version 0 to 6 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:16: oid 16 version 6 is already newer than 4 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:16: oid 16 version 6 is already newer than 6 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:13: finishing write tid 1 to smithi06431743-13 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:15: done (15 left) 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:16: done (14 left) 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:29: write initial oid 29 2024-03-04T06:08:04.181 INFO:tasks.rados.rados.0.smithi064.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-03-04T06:08:04.185 INFO:tasks.rados.rados.0.smithi064.stdout:29: writing smithi06431743-29 from 470596 to 926627 tid 1 2024-03-04T06:08:04.192 INFO:tasks.rados.rados.0.smithi064.stdout:29: writing smithi06431743-29 from 1572084 to 2190744 tid 2 2024-03-04T06:08:04.192 INFO:tasks.rados.rados.0.smithi064.stdout:30: write initial oid 30 2024-03-04T06:08:04.192 INFO:tasks.rados.rados.0.smithi064.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-03-04T06:08:04.198 INFO:tasks.rados.rados.0.smithi064.stdout:30: writing smithi06431743-30 from 487403 to 1018683 tid 1 2024-03-04T06:08:04.205 INFO:tasks.rados.rados.0.smithi064.stdout:30: writing smithi06431743-30 from 1514213 to 1992884 tid 2 2024-03-04T06:08:04.205 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 16 2024-03-04T06:08:04.205 INFO:tasks.rados.rados.0.smithi064.stdout:17: finishing write tid 3 to smithi06431743-17 2024-03-04T06:08:04.205 INFO:tasks.rados.rados.0.smithi064.stdout:17: finishing write tid 4 to smithi06431743-17 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:17: oid 17 updating version 0 to 1 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:17: oid 17 updating version 1 to 2 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:17: oid 17 updating version 2 to 3 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:14: finishing write tid 3 to smithi06431743-14 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:18: finishing write tid 2 to smithi06431743-18 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:18: finishing write tid 3 to smithi06431743-18 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:14: finishing write tid 4 to smithi06431743-14 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:14: oid 14 updating version 0 to 3 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:14: oid 14 version 3 is already newer than 2 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:14: oid 14 version 3 is already newer than 1 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-03-04T06:08:04.206 INFO:tasks.rados.rados.0.smithi064.stdout:18: finishing write tid 4 to smithi06431743-18 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:18: oid 18 updating version 0 to 1 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:18: oid 18 updating version 1 to 3 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:18: oid 18 version 3 is already newer than 3 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:21: finishing write tid 1 to smithi06431743-21 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:13: finishing write tid 2 to smithi06431743-13 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:13: finishing write tid 3 to smithi06431743-13 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:13: finishing write tid 4 to smithi06431743-13 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:13: oid 13 updating version 0 to 9 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:13: oid 13 version 9 is already newer than 9 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:13: oid 13 version 9 is already newer than 7 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 9 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:21: finishing write tid 2 to smithi06431743-21 2024-03-04T06:08:04.207 INFO:tasks.rados.rados.0.smithi064.stdout:22: finishing write tid 1 to smithi06431743-22 2024-03-04T06:08:04.208 INFO:tasks.rados.rados.0.smithi064.stdout:20: finishing write tid 1 to smithi06431743-20 2024-03-04T06:08:04.208 INFO:tasks.rados.rados.0.smithi064.stdout:13: done (15 left) 2024-03-04T06:08:04.208 INFO:tasks.rados.rados.0.smithi064.stdout:14: done (14 left) 2024-03-04T06:08:04.208 INFO:tasks.rados.rados.0.smithi064.stdout:17: done (13 left) 2024-03-04T06:08:04.208 INFO:tasks.rados.rados.0.smithi064.stdout:18: done (12 left) 2024-03-04T06:08:04.208 INFO:tasks.rados.rados.0.smithi064.stdout:31: write initial oid 31 2024-03-04T06:08:04.208 INFO:tasks.rados.rados.0.smithi064.stdout:22: finishing write tid 2 to smithi06431743-22 2024-03-04T06:08:04.208 INFO:tasks.rados.rados.0.smithi064.stdout:23: finishing write tid 1 to smithi06431743-23 2024-03-04T06:08:04.208 INFO:tasks.rados.rados.0.smithi064.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-03-04T06:08:04.212 INFO:tasks.rados.rados.0.smithi064.stdout:31: writing smithi06431743-31 from 504210 to 1110739 tid 1 2024-03-04T06:08:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:08:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:08:04.219 INFO:tasks.rados.rados.0.smithi064.stdout:31: writing smithi06431743-31 from 1572695 to 2195024 tid 2 2024-03-04T06:08:04.219 INFO:tasks.rados.rados.0.smithi064.stdout:32: write initial oid 32 2024-03-04T06:08:04.219 INFO:tasks.rados.rados.0.smithi064.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-03-04T06:08:04.226 INFO:tasks.rados.rados.0.smithi064.stdout:32: writing smithi06431743-32 from 521017 to 1319148 tid 1 2024-03-04T06:08:04.227 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-04T06:08:03.785493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107956,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379404,"statfs":{"total":386547056640,"available":386436509696,"internally_reserved":0,"allocated":675840,"data_stored":1900108,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.079996"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.829404+0000","last_change":"2024-03-04T06:08:00.829404+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.829404+0000","last_undegraded":"2024-03-04T06:08:00.829404+0000","last_fullsized":"2024-03-04T06:08:00.829404+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.837238+0000","last_change":"2024-03-04T06:08:00.837238+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.837238+0000","last_undegraded":"2024-03-04T06:08:00.837238+0000","last_fullsized":"2024-03-04T06:08:00.837238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.818819+0000","last_change":"2024-03-04T06:08:00.818819+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.818819+0000","last_undegraded":"2024-03-04T06:08:00.818819+0000","last_fullsized":"2024-03-04T06:08:00.818819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":148,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.789652+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:08:00.789652+0000","last_peered":"2024-03-04T06:08:00.789652+0000","last_clean":"2024-03-04T06:08:00.789652+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:08:00.789652+0000","last_undegraded":"2024-03-04T06:08:00.789652+0000","last_fullsized":"2024-03-04T06:08:00.789652+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.797748+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:08:00.797748+0000","last_peered":"2024-03-04T06:08:00.797748+0000","last_clean":"2024-03-04T06:08:00.797748+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:08:00.797748+0000","last_undegraded":"2024-03-04T06:08:00.797748+0000","last_fullsized":"2024-03-04T06:08:00.797748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.802057+0000","last_change":"2024-03-04T06:08:00.802057+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.802057+0000","last_undegraded":"2024-03-04T06:08:00.802057+0000","last_fullsized":"2024-03-04T06:08:00.802057+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.313279+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:07:56.313279+0000","last_peered":"2024-03-04T06:07:56.313279+0000","last_clean":"2024-03-04T06:07:56.313279+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:07:56.313279+0000","last_undegraded":"2024-03-04T06:07:56.313279+0000","last_fullsized":"2024-03-04T06:07:56.313279+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.808984+0000","last_change":"2024-03-04T06:08:00.808984+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.808984+0000","last_undegraded":"2024-03-04T06:08:00.808984+0000","last_fullsized":"2024-03-04T06:08:00.808984+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.792136+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:08:00.792136+0000","last_peered":"2024-03-04T06:08:00.792136+0000","last_clean":"2024-03-04T06:08:00.792136+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:08:00.792136+0000","last_undegraded":"2024-03-04T06:08:00.792136+0000","last_fullsized":"2024-03-04T06:08:00.792136+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.799251+0000","last_change":"2024-03-04T06:08:00.799251+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.799251+0000","last_undegraded":"2024-03-04T06:08:00.799251+0000","last_fullsized":"2024-03-04T06:08:00.799251+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.307816+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:07:56.307816+0000","last_peered":"2024-03-04T06:07:56.307816+0000","last_clean":"2024-03-04T06:07:56.307816+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:07:56.307816+0000","last_undegraded":"2024-03-04T06:07:56.307816+0000","last_fullsized":"2024-03-04T06:07:56.307816+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.810651+0000","last_change":"2024-03-04T06:08:00.810651+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.810651+0000","last_undegraded":"2024-03-04T06:08:00.810651+0000","last_fullsized":"2024-03-04T06:08:00.810651+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.314848+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:07:56.314848+0000","last_peered":"2024-03-04T06:07:56.314848+0000","last_clean":"2024-03-04T06:07:56.314848+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:07:56.314848+0000","last_undegraded":"2024-03-04T06:07:56.314848+0000","last_fullsized":"2024-03-04T06:07:56.314848+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.794156+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:08:00.794156+0000","last_peered":"2024-03-04T06:08:00.794156+0000","last_clean":"2024-03-04T06:08:00.794156+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:08:00.794156+0000","last_undegraded":"2024-03-04T06:08:00.794156+0000","last_fullsized":"2024-03-04T06:08:00.794156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.823755+0000","last_change":"2024-03-04T06:08:00.823755+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.823755+0000","last_undegraded":"2024-03-04T06:08:00.823755+0000","last_fullsized":"2024-03-04T06:08:00.823755+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.308428+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:07:56.308428+0000","last_peered":"2024-03-04T06:07:56.308428+0000","last_clean":"2024-03-04T06:07:56.308428+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:07:56.308428+0000","last_undegraded":"2024-03-04T06:07:56.308428+0000","last_fullsized":"2024-03-04T06:07:56.308428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.797136+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:08:00.797136+0000","last_peered":"2024-03-04T06:08:00.797136+0000","last_clean":"2024-03-04T06:08:00.797136+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:08:00.797136+0000","last_undegraded":"2024-03-04T06:08:00.797136+0000","last_fullsized":"2024-03-04T06:08:00.797136+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705680,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":188416,"data_stored":624411,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705680,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":126976,"data_stored":34043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705679,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705679,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:08:04.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:08:04.232 INFO:tasks.rados.rados.0.smithi064.stdout:32: writing smithi06431743-32 from 1747530 to 2229870 tid 2 2024-03-04T06:08:04.232 INFO:tasks.rados.rados.0.smithi064.stdout:33: write initial oid 33 2024-03-04T06:08:04.232 INFO:tasks.rados.rados.0.smithi064.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-03-04T06:08:04.237 INFO:tasks.rados.rados.0.smithi064.stdout:33: writing smithi06431743-33 from 537824 to 1011204 tid 1 2024-03-04T06:08:04.247 INFO:tasks.rados.rados.0.smithi064.stdout:33: writing smithi06431743-33 from 1806012 to 2432010 tid 2 2024-03-04T06:08:04.247 INFO:tasks.rados.rados.0.smithi064.stdout:34: write initial oid 34 2024-03-04T06:08:04.247 INFO:tasks.rados.rados.0.smithi064.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-03-04T06:08:04.252 INFO:tasks.rados.rados.0.smithi064.stdout:34: writing smithi06431743-34 from 554631 to 1103260 tid 1 2024-03-04T06:08:04.260 INFO:tasks.rados.rados.0.smithi064.stdout:34: writing smithi06431743-34 from 1748141 to 2234150 tid 2 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 16 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:21: finishing write tid 3 to smithi06431743-21 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:21: finishing write tid 4 to smithi06431743-21 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:21: oid 21 updating version 0 to 9 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:21: oid 21 version 9 is already newer than 8 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:21: oid 21 version 9 is already newer than 7 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:22: finishing write tid 3 to smithi06431743-22 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:22: finishing write tid 4 to smithi06431743-22 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:22: oid 22 updating version 0 to 6 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:22: oid 22 version 6 is already newer than 4 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:22: oid 22 version 6 is already newer than 6 2024-03-04T06:08:04.261 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:26: finishing write tid 1 to smithi06431743-26 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:19: finishing write tid 1 to smithi06431743-19 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:20: finishing write tid 2 to smithi06431743-20 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:19: finishing write tid 2 to smithi06431743-19 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:19: finishing write tid 3 to smithi06431743-19 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:19: finishing write tid 4 to smithi06431743-19 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:19: oid 19 updating version 0 to 10 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:19: oid 19 updating version 10 to 12 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:19: oid 19 version 12 is already newer than 12 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 12 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:25: finishing write tid 1 to smithi06431743-25 2024-03-04T06:08:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:29: finishing write tid 1 to smithi06431743-29 2024-03-04T06:08:04.263 INFO:tasks.rados.rados.0.smithi064.stdout:19: done (15 left) 2024-03-04T06:08:04.263 INFO:tasks.rados.rados.0.smithi064.stdout:21: done (14 left) 2024-03-04T06:08:04.263 INFO:tasks.rados.rados.0.smithi064.stdout:22: done (13 left) 2024-03-04T06:08:04.263 INFO:tasks.rados.rados.0.smithi064.stdout:35: write initial oid 35 2024-03-04T06:08:04.263 INFO:tasks.rados.rados.0.smithi064.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-03-04T06:08:04.266 INFO:tasks.rados.rados.0.smithi064.stdout:35: writing smithi06431743-35 from 571438 to 1195316 tid 1 2024-03-04T06:08:04.274 INFO:tasks.rados.rados.0.smithi064.stdout:35: writing smithi06431743-35 from 1806623 to 2436290 tid 2 2024-03-04T06:08:04.274 INFO:tasks.rados.rados.0.smithi064.stdout:36: write initial oid 36 2024-03-04T06:08:04.274 INFO:tasks.rados.rados.0.smithi064.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-03-04T06:08:04.281 INFO:tasks.rados.rados.0.smithi064.stdout:36: writing smithi06431743-36 from 588245 to 1287372 tid 1 2024-03-04T06:08:04.290 INFO:tasks.rados.rados.0.smithi064.stdout:36: writing smithi06431743-36 from 1865105 to 2354783 tid 2 2024-03-04T06:08:04.290 INFO:tasks.rados.rados.0.smithi064.stdout:37: write initial oid 37 2024-03-04T06:08:04.290 INFO:tasks.rados.rados.0.smithi064.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-03-04T06:08:04.300 INFO:tasks.rados.rados.0.smithi064.stdout:37: writing smithi06431743-37 from 605052 to 1379428 tid 1 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:37: writing smithi06431743-37 from 1923587 to 2556923 tid 2 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 16 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:23: finishing write tid 2 to smithi06431743-23 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:27: finishing write tid 1 to smithi06431743-27 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:26: finishing write tid 2 to smithi06431743-26 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:23: finishing write tid 3 to smithi06431743-23 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:23: finishing write tid 4 to smithi06431743-23 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:23: oid 23 updating version 0 to 4 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:23: oid 23 updating version 4 to 5 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:23: oid 23 updating version 5 to 6 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:26: finishing write tid 3 to smithi06431743-26 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:29: finishing write tid 2 to smithi06431743-29 2024-03-04T06:08:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:26: finishing write tid 4 to smithi06431743-26 2024-03-04T06:08:04.311 INFO:tasks.rados.rados.0.smithi064.stdout:26: oid 26 updating version 0 to 6 2024-03-04T06:08:04.311 INFO:tasks.rados.rados.0.smithi064.stdout:26: oid 26 version 6 is already newer than 4 2024-03-04T06:08:04.311 INFO:tasks.rados.rados.0.smithi064.stdout:26: oid 26 version 6 is already newer than 6 2024-03-04T06:08:04.311 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-04T06:08:04.311 INFO:tasks.rados.rados.0.smithi064.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-03-04T06:08:04.311 INFO:tasks.rados.rados.0.smithi064.stdout:20: finishing write tid 3 to smithi06431743-20 2024-03-04T06:08:04.311 INFO:tasks.rados.rados.0.smithi064.stdout:23: done (15 left) 2024-03-04T06:08:04.311 INFO:tasks.rados.rados.0.smithi064.stdout:26: done (14 left) 2024-03-04T06:08:04.311 INFO:tasks.rados.rados.0.smithi064.stdout:38: write initial oid 38 2024-03-04T06:08:04.311 INFO:tasks.rados.rados.0.smithi064.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-03-04T06:08:04.316 INFO:tasks.rados.rados.0.smithi064.stdout:38: writing smithi06431743-38 from 621859 to 1071484 tid 1 2024-03-04T06:08:04.326 INFO:tasks.rados.rados.0.smithi064.stdout:38: writing smithi06431743-38 from 1865716 to 2621859 tid 2 2024-03-04T06:08:04.326 INFO:tasks.rados.rados.0.smithi064.stdout:39: write initial oid 39 2024-03-04T06:08:04.326 INFO:tasks.rados.rados.0.smithi064.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-03-04T06:08:04.332 INFO:tasks.rados.rados.0.smithi064.stdout:39: writing smithi06431743-39 from 638666 to 1163540 tid 1 2024-03-04T06:08:04.341 INFO:tasks.rados.rados.0.smithi064.stdout:39: writing smithi06431743-39 from 1924198 to 2561203 tid 2 2024-03-04T06:08:04.341 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 16 2024-03-04T06:08:04.341 INFO:tasks.rados.rados.0.smithi064.stdout:25: finishing write tid 2 to smithi06431743-25 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:27: finishing write tid 2 to smithi06431743-27 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:20: finishing write tid 4 to smithi06431743-20 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:20: oid 20 updating version 0 to 5 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:20: oid 20 version 5 is already newer than 4 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:20: oid 20 updating version 5 to 6 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:28: finishing write tid 1 to smithi06431743-28 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:29: finishing write tid 3 to smithi06431743-29 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:31: finishing write tid 1 to smithi06431743-31 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:20: done (15 left) 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:40: write initial oid 40 2024-03-04T06:08:04.342 INFO:tasks.rados.rados.0.smithi064.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-03-04T06:08:04.349 INFO:tasks.rados.rados.0.smithi064.stdout:40: writing smithi06431743-40 from 655473 to 1371949 tid 1 2024-03-04T06:08:04.356 INFO:tasks.rados.rados.0.smithi064.stdout:40: writing smithi06431743-40 from 2099033 to 2655473 tid 2 2024-03-04T06:08:04.356 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 16 2024-03-04T06:08:04.356 INFO:tasks.rados.rados.0.smithi064.stdout:29: finishing write tid 4 to smithi06431743-29 2024-03-04T06:08:04.356 INFO:tasks.rados.rados.0.smithi064.stdout:29: oid 29 updating version 0 to 15 2024-03-04T06:08:04.356 INFO:tasks.rados.rados.0.smithi064.stdout:29: oid 29 version 15 is already newer than 14 2024-03-04T06:08:04.356 INFO:tasks.rados.rados.0.smithi064.stdout:29: oid 29 version 15 is already newer than 13 2024-03-04T06:08:04.356 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 15 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-04T06:08:04.356 INFO:tasks.rados.rados.0.smithi064.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-03-04T06:08:04.356 INFO:tasks.rados.rados.0.smithi064.stdout:29: done (15 left) 2024-03-04T06:08:04.356 INFO:tasks.rados.rados.0.smithi064.stdout:41: write initial oid 41 2024-03-04T06:08:04.357 INFO:tasks.rados.rados.0.smithi064.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-03-04T06:08:04.364 INFO:tasks.rados.rados.0.smithi064.stdout:41: writing smithi06431743-41 from 672280 to 1464005 tid 1 2024-03-04T06:08:04.371 INFO:tasks.rados.rados.0.smithi064.stdout:41: writing smithi06431743-41 from 2157515 to 2672280 tid 2 2024-03-04T06:08:04.371 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 16 2024-03-04T06:08:04.371 INFO:tasks.rados.rados.0.smithi064.stdout:25: finishing write tid 3 to smithi06431743-25 2024-03-04T06:08:04.371 INFO:tasks.rados.rados.0.smithi064.stdout:25: finishing write tid 4 to smithi06431743-25 2024-03-04T06:08:04.371 INFO:tasks.rados.rados.0.smithi064.stdout:25: oid 25 updating version 0 to 6 2024-03-04T06:08:04.372 INFO:tasks.rados.rados.0.smithi064.stdout:25: oid 25 version 6 is already newer than 4 2024-03-04T06:08:04.372 INFO:tasks.rados.rados.0.smithi064.stdout:25: oid 25 version 6 is already newer than 5 2024-03-04T06:08:04.372 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-04T06:08:04.372 INFO:tasks.rados.rados.0.smithi064.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-03-04T06:08:04.372 INFO:tasks.rados.rados.0.smithi064.stdout:27: finishing write tid 3 to smithi06431743-27 2024-03-04T06:08:04.372 INFO:tasks.rados.rados.0.smithi064.stdout:25: done (15 left) 2024-03-04T06:08:04.372 INFO:tasks.rados.rados.0.smithi064.stdout:42: write initial oid 42 2024-03-04T06:08:04.372 INFO:tasks.rados.rados.0.smithi064.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-03-04T06:08:04.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:04.370+0000 7fc2aa99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:04.378 INFO:tasks.rados.rados.0.smithi064.stdout:42: writing smithi06431743-42 from 689087 to 1156061 tid 1 2024-03-04T06:08:04.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:04.379+0000 7fc2aa99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:04.386 INFO:tasks.rados.rados.0.smithi064.stdout:42: writing smithi06431743-42 from 1699644 to 2483976 tid 2 2024-03-04T06:08:04.389 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 16 2024-03-04T06:08:04.389 INFO:tasks.rados.rados.0.smithi064.stdout:27: finishing write tid 4 to smithi06431743-27 2024-03-04T06:08:04.389 INFO:tasks.rados.rados.0.smithi064.stdout:27: oid 27 updating version 0 to 5 2024-03-04T06:08:04.389 INFO:tasks.rados.rados.0.smithi064.stdout:27: oid 27 updating version 5 to 6 2024-03-04T06:08:04.389 INFO:tasks.rados.rados.0.smithi064.stdout:27: oid 27 version 6 is already newer than 4 2024-03-04T06:08:04.389 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-04T06:08:04.389 INFO:tasks.rados.rados.0.smithi064.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-03-04T06:08:04.389 INFO:tasks.rados.rados.0.smithi064.stdout:24: finishing write tid 1 to smithi06431743-24 2024-03-04T06:08:04.389 INFO:tasks.rados.rados.0.smithi064.stdout:33: finishing write tid 1 to smithi06431743-33 2024-03-04T06:08:04.389 INFO:tasks.rados.rados.0.smithi064.stdout:31: finishing write tid 2 to smithi06431743-31 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:30: finishing write tid 1 to smithi06431743-30 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:28: finishing write tid 2 to smithi06431743-28 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:24: finishing write tid 2 to smithi06431743-24 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:24: finishing write tid 3 to smithi06431743-24 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:24: finishing write tid 4 to smithi06431743-24 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:24: oid 24 updating version 0 to 9 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:24: oid 24 version 9 is already newer than 9 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:24: oid 24 version 9 is already newer than 7 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:33: finishing write tid 2 to smithi06431743-33 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:31: finishing write tid 3 to smithi06431743-31 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:31: finishing write tid 4 to smithi06431743-31 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:31: oid 31 updating version 0 to 2 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:31: oid 31 updating version 2 to 3 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:31: oid 31 version 3 is already newer than 1 2024-03-04T06:08:04.390 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 3 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:36: finishing write tid 1 to smithi06431743-36 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:28: finishing write tid 3 to smithi06431743-28 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:30: finishing write tid 2 to smithi06431743-30 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:28: finishing write tid 4 to smithi06431743-28 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:28: oid 28 updating version 0 to 9 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:28: oid 28 version 9 is already newer than 8 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:28: oid 28 version 9 is already newer than 7 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:30: finishing write tid 3 to smithi06431743-30 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:30: finishing write tid 4 to smithi06431743-30 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:30: oid 30 updating version 0 to 18 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:30: oid 30 version 18 is already newer than 16 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:30: oid 30 version 18 is already newer than 18 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-04T06:08:04.391 INFO:tasks.rados.rados.0.smithi064.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:38: finishing write tid 1 to smithi06431743-38 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:35: finishing write tid 1 to smithi06431743-35 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:36: finishing write tid 2 to smithi06431743-36 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:37: finishing write tid 1 to smithi06431743-37 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:36: finishing write tid 3 to smithi06431743-36 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:33: finishing write tid 3 to smithi06431743-33 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:36: finishing write tid 4 to smithi06431743-36 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:36: oid 36 updating version 0 to 7 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:36: oid 36 updating version 7 to 9 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:36: oid 36 version 9 is already newer than 9 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:33: finishing write tid 4 to smithi06431743-33 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:33: oid 33 updating version 0 to 7 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:33: oid 33 updating version 7 to 8 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:33: oid 33 updating version 8 to 9 2024-03-04T06:08:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 9 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:39: finishing write tid 1 to smithi06431743-39 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:32: finishing write tid 1 to smithi06431743-32 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:24: done (15 left) 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:27: done (14 left) 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:28: done (13 left) 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:30: done (12 left) 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:31: done (11 left) 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:33: done (10 left) 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:36: done (9 left) 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:43: write initial oid 43 2024-03-04T06:08:04.393 INFO:tasks.rados.rados.0.smithi064.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-03-04T06:08:04.395 INFO:tasks.rados.rados.0.smithi064.stdout:43: writing smithi06431743-43 from 705894 to 1248117 tid 1 2024-03-04T06:08:04.402 INFO:tasks.rados.rados.0.smithi064.stdout:43: writing smithi06431743-43 from 1758126 to 2402469 tid 2 2024-03-04T06:08:04.402 INFO:tasks.rados.rados.0.smithi064.stdout:44: write initial oid 44 2024-03-04T06:08:04.402 INFO:tasks.rados.rados.0.smithi064.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-03-04T06:08:04.409 INFO:tasks.rados.rados.0.smithi064.stdout:44: writing smithi06431743-44 from 722701 to 1340173 tid 1 2024-03-04T06:08:04.417 INFO:tasks.rados.rados.0.smithi064.stdout:44: writing smithi06431743-44 from 1816608 to 2604609 tid 2 2024-03-04T06:08:04.418 INFO:tasks.rados.rados.0.smithi064.stdout:45: write initial oid 45 2024-03-04T06:08:04.418 INFO:tasks.rados.rados.0.smithi064.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-03-04T06:08:04.425 INFO:tasks.rados.rados.0.smithi064.stdout:45: writing smithi06431743-45 from 739508 to 1432229 tid 1 2024-03-04T06:08:04.433 INFO:tasks.rados.rados.0.smithi064.stdout:45: writing smithi06431743-45 from 1875090 to 2523102 tid 2 2024-03-04T06:08:04.433 INFO:tasks.rados.rados.0.smithi064.stdout:46: write initial oid 46 2024-03-04T06:08:04.433 INFO:tasks.rados.rados.0.smithi064.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-03-04T06:08:04.442 INFO:tasks.rados.rados.0.smithi064.stdout:46: writing smithi06431743-46 from 756315 to 1524285 tid 1 2024-03-04T06:08:04.449 INFO:tasks.rados.rados.0.smithi064.stdout:46: writing smithi06431743-46 from 1933572 to 2725242 tid 2 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:37: finishing write tid 2 to smithi06431743-37 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:32: finishing write tid 2 to smithi06431743-32 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:32: finishing write tid 3 to smithi06431743-32 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:37: finishing write tid 3 to smithi06431743-37 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:37: finishing write tid 4 to smithi06431743-37 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:37: oid 37 updating version 0 to 7 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:37: oid 37 updating version 7 to 9 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:37: oid 37 version 9 is already newer than 9 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 9 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:32: finishing write tid 4 to smithi06431743-32 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:32: oid 32 updating version 0 to 12 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:32: oid 32 version 12 is already newer than 12 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:32: oid 32 version 12 is already newer than 10 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 12 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-04T06:08:04.450 INFO:tasks.rados.rados.0.smithi064.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:40: finishing write tid 1 to smithi06431743-40 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:38: finishing write tid 2 to smithi06431743-38 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:39: finishing write tid 2 to smithi06431743-39 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:35: finishing write tid 2 to smithi06431743-35 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:41: finishing write tid 1 to smithi06431743-41 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:34: finishing write tid 1 to smithi06431743-34 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:40: finishing write tid 2 to smithi06431743-40 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:39: finishing write tid 3 to smithi06431743-39 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:32: done (12 left) 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:37: done (11 left) 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:47: write initial oid 47 2024-03-04T06:08:04.451 INFO:tasks.rados.rados.0.smithi064.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-03-04T06:08:04.456 INFO:tasks.rados.rados.0.smithi064.stdout:47: writing smithi06431743-47 from 773122 to 1332694 tid 1 2024-03-04T06:08:04.465 INFO:tasks.rados.rados.0.smithi064.stdout:47: writing smithi06431743-47 from 1992054 to 2643735 tid 2 2024-03-04T06:08:04.465 INFO:tasks.rados.rados.0.smithi064.stdout:48: write initial oid 48 2024-03-04T06:08:04.465 INFO:tasks.rados.rados.0.smithi064.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-03-04T06:08:04.473 INFO:tasks.rados.rados.0.smithi064.stdout:48: writing smithi06431743-48 from 789929 to 1424750 tid 1 2024-03-04T06:08:04.483 INFO:tasks.rados.rados.0.smithi064.stdout:48: writing smithi06431743-48 from 2050536 to 2789929 tid 2 2024-03-04T06:08:04.483 INFO:tasks.rados.rados.0.smithi064.stdout:49: write initial oid 49 2024-03-04T06:08:04.483 INFO:tasks.rados.rados.0.smithi064.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-03-04T06:08:04.489 INFO:tasks.rados.rados.0.smithi064.stdout:49: writing smithi06431743-49 from 406736 to 1116806 tid 1 2024-03-04T06:08:04.496 INFO:tasks.rados.rados.0.smithi064.stdout:49: writing smithi06431743-49 from 1709018 to 2364368 tid 2 2024-03-04T06:08:04.496 INFO:tasks.rados.rados.0.smithi064.stdout:50: write initial oid 50 2024-03-04T06:08:04.496 INFO:tasks.rados.rados.0.smithi064.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-03-04T06:08:04.502 INFO:tasks.rados.rados.0.smithi064.stdout:50: writing smithi06431743-50 from 423543 to 1208862 tid 1 2024-03-04T06:08:04.512 INFO:tasks.rados.rados.0.smithi064.stdout:50: writing smithi06431743-50 from 1767500 to 2566508 tid 2 2024-03-04T06:08:04.512 INFO:tasks.rados.rados.0.smithi064.stdout:51: write (excl) oid 2 current snap is 0 2024-03-04T06:08:04.512 INFO:tasks.rados.rados.0.smithi064.stdout:39: finishing write tid 4 to smithi06431743-39 2024-03-04T06:08:04.512 INFO:tasks.rados.rados.0.smithi064.stdout:39: oid 39 updating version 0 to 7 2024-03-04T06:08:04.512 INFO:tasks.rados.rados.0.smithi064.stdout:39: oid 39 updating version 7 to 9 2024-03-04T06:08:04.512 INFO:tasks.rados.rados.0.smithi064.stdout:39: oid 39 version 9 is already newer than 8 2024-03-04T06:08:04.512 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-04T06:08:04.512 INFO:tasks.rados.rados.0.smithi064.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-03-04T06:08:04.513 INFO:tasks.rados.rados.0.smithi064.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-03-04T06:08:04.517 INFO:tasks.rados.rados.0.smithi064.stdout:51: writing smithi06431743-2 from 440350 to 900918 tid 1 2024-03-04T06:08:04.523 INFO:tasks.rados.rados.0.smithi064.stdout:51: writing smithi06431743-2 from 1309629 to 1852295 tid 2 2024-03-04T06:08:04.529 INFO:tasks.rados.rados.0.smithi064.stdout:51: writing smithi06431743-2 from 2323973 to 2840350 tid 3 2024-03-04T06:08:04.529 INFO:tasks.rados.rados.0.smithi064.stdout:39: done (15 left) 2024-03-04T06:08:04.529 INFO:tasks.rados.rados.0.smithi064.stdout:52: write (excl) oid 14 current snap is 0 2024-03-04T06:08:04.529 INFO:tasks.rados.rados.0.smithi064.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2024-03-04T06:08:04.536 INFO:tasks.rados.rados.0.smithi064.stdout:52: writing smithi06431743-14 from 457157 to 992974 tid 1 2024-03-04T06:08:04.545 INFO:tasks.rados.rados.0.smithi064.stdout:52: writing smithi06431743-14 from 1768111 to 2170788 tid 2 2024-03-04T06:08:04.545 INFO:tasks.rados.rados.0.smithi064.stdout:35: finishing write tid 3 to smithi06431743-35 2024-03-04T06:08:04.545 INFO:tasks.rados.rados.0.smithi064.stdout:35: finishing write tid 4 to smithi06431743-35 2024-03-04T06:08:04.545 INFO:tasks.rados.rados.0.smithi064.stdout:35: oid 35 updating version 0 to 12 2024-03-04T06:08:04.545 INFO:tasks.rados.rados.0.smithi064.stdout:35: oid 35 version 12 is already newer than 12 2024-03-04T06:08:04.545 INFO:tasks.rados.rados.0.smithi064.stdout:35: oid 35 version 12 is already newer than 10 2024-03-04T06:08:04.545 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-04T06:08:04.545 INFO:tasks.rados.rados.0.smithi064.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-03-04T06:08:04.545 INFO:tasks.rados.rados.0.smithi064.stdout:38: finishing write tid 3 to smithi06431743-38 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:34: finishing write tid 2 to smithi06431743-34 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:41: finishing write tid 2 to smithi06431743-41 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:34: finishing write tid 3 to smithi06431743-34 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:34: finishing write tid 4 to smithi06431743-34 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:34: oid 34 updating version 0 to 13 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:34: oid 34 updating version 13 to 15 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:34: oid 34 version 15 is already newer than 15 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:38: finishing write tid 4 to smithi06431743-38 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:38: oid 38 updating version 0 to 5 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:38: oid 38 version 5 is already newer than 4 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:38: oid 38 updating version 5 to 6 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:40: finishing write tid 3 to smithi06431743-40 2024-03-04T06:08:04.546 INFO:tasks.rados.rados.0.smithi064.stdout:45: finishing write tid 1 to smithi06431743-45 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:40: finishing write tid 4 to smithi06431743-40 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:40: oid 40 updating version 0 to 1 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:40: oid 40 updating version 1 to 2 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:40: oid 40 updating version 2 to 3 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 3 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:46: finishing write tid 1 to smithi06431743-46 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:47: finishing write tid 1 to smithi06431743-47 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:44: finishing write tid 1 to smithi06431743-44 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:41: finishing write tid 3 to smithi06431743-41 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:41: finishing write tid 4 to smithi06431743-41 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:41: oid 41 updating version 0 to 9 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:41: oid 41 version 9 is already newer than 7 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:41: oid 41 version 9 is already newer than 8 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-03-04T06:08:04.547 INFO:tasks.rados.rados.0.smithi064.stdout:47: finishing write tid 2 to smithi06431743-47 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:42: finishing write tid 1 to smithi06431743-42 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:48: finishing write tid 1 to smithi06431743-48 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:47: finishing write tid 3 to smithi06431743-47 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:34: done (15 left) 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:35: done (14 left) 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:38: done (13 left) 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:40: done (12 left) 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:41: done (11 left) 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:53: read oid 10 snap -1 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:53: expect (ObjNum 9 snap 0 seq_num 9) 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:54: delete oid 20 current snap is 0 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:46: finishing write tid 2 to smithi06431743-46 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:47: finishing write tid 4 to smithi06431743-47 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:47: oid 47 updating version 0 to 8 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:47: oid 47 updating version 8 to 9 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:47: oid 47 version 9 is already newer than 7 2024-03-04T06:08:04.548 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-04T06:08:04.549 INFO:tasks.rados.rados.0.smithi064.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-03-04T06:08:04.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:04.549 INFO:tasks.rados.rados.0.smithi064.stdout:45: finishing write tid 2 to smithi06431743-45 2024-03-04T06:08:04.557 INFO:tasks.rados.rados.0.smithi064.stdout:44: finishing write tid 2 to smithi06431743-44 2024-03-04T06:08:04.562 INFO:tasks.rados.rados.0.smithi064.stdout:46: finishing write tid 3 to smithi06431743-46 2024-03-04T06:08:04.562 INFO:tasks.rados.rados.0.smithi064.stdout:50: finishing write tid 1 to smithi06431743-50 2024-03-04T06:08:04.567 INFO:tasks.rados.rados.0.smithi064.stdout:49: finishing write tid 1 to smithi06431743-49 2024-03-04T06:08:04.567 INFO:tasks.rados.rados.0.smithi064.stdout:46: finishing write tid 4 to smithi06431743-46 2024-03-04T06:08:04.567 INFO:tasks.rados.rados.0.smithi064.stdout:46: oid 46 updating version 0 to 18 2024-03-04T06:08:04.567 INFO:tasks.rados.rados.0.smithi064.stdout:46: oid 46 version 18 is already newer than 16 2024-03-04T06:08:04.568 INFO:tasks.rados.rados.0.smithi064.stdout:46: oid 46 version 18 is already newer than 18 2024-03-04T06:08:04.568 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 18 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-04T06:08:04.568 INFO:tasks.rados.rados.0.smithi064.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-03-04T06:08:04.570 INFO:tasks.rados.rados.0.smithi064.stdout:51: finishing write tid 1 to smithi06431743-2 2024-03-04T06:08:04.573 INFO:tasks.rados.rados.0.smithi064.stdout:44: finishing write tid 3 to smithi06431743-44 2024-03-04T06:08:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:44: finishing write tid 4 to smithi06431743-44 2024-03-04T06:08:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:44: oid 44 updating version 0 to 9 2024-03-04T06:08:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:44: oid 44 version 9 is already newer than 7 2024-03-04T06:08:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:44: oid 44 version 9 is already newer than 9 2024-03-04T06:08:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-04T06:08:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-03-04T06:08:04.578 INFO:tasks.rados.rados.0.smithi064.stdout:42: finishing write tid 2 to smithi06431743-42 2024-03-04T06:08:04.583 INFO:tasks.rados.rados.0.smithi064.stdout:48: finishing write tid 2 to smithi06431743-48 2024-03-04T06:08:04.586 INFO:tasks.rados.rados.0.smithi064.stdout:51: finishing write tid 2 to smithi06431743-2 2024-03-04T06:08:04.587 INFO:tasks.rados.rados.0.smithi064.stdout:45: finishing write tid 3 to smithi06431743-45 2024-03-04T06:08:04.587 INFO:tasks.rados.rados.0.smithi064.stdout:48: finishing write tid 3 to smithi06431743-48 2024-03-04T06:08:04.588 INFO:tasks.rados.rados.0.smithi064.stdout:48: finishing write tid 4 to smithi06431743-48 2024-03-04T06:08:04.588 INFO:tasks.rados.rados.0.smithi064.stdout:48: oid 48 updating version 0 to 6 2024-03-04T06:08:04.588 INFO:tasks.rados.rados.0.smithi064.stdout:48: oid 48 version 6 is already newer than 6 2024-03-04T06:08:04.589 INFO:tasks.rados.rados.0.smithi064.stdout:48: oid 48 version 6 is already newer than 4 2024-03-04T06:08:04.589 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 6 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-04T06:08:04.589 INFO:tasks.rados.rados.0.smithi064.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-03-04T06:08:04.592 INFO:tasks.rados.rados.0.smithi064.stdout:42: finishing write tid 3 to smithi06431743-42 2024-03-04T06:08:04.593 INFO:tasks.rados.rados.0.smithi064.stdout:42: finishing write tid 4 to smithi06431743-42 2024-03-04T06:08:04.593 INFO:tasks.rados.rados.0.smithi064.stdout:42: oid 42 updating version 0 to 6 2024-03-04T06:08:04.593 INFO:tasks.rados.rados.0.smithi064.stdout:42: oid 42 version 6 is already newer than 4 2024-03-04T06:08:04.593 INFO:tasks.rados.rados.0.smithi064.stdout:42: oid 42 version 6 is already newer than 6 2024-03-04T06:08:04.594 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-04T06:08:04.594 INFO:tasks.rados.rados.0.smithi064.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-03-04T06:08:04.594 INFO:tasks.rados.rados.0.smithi064.stdout:45: finishing write tid 4 to smithi06431743-45 2024-03-04T06:08:04.594 INFO:tasks.rados.rados.0.smithi064.stdout:45: oid 45 updating version 0 to 12 2024-03-04T06:08:04.595 INFO:tasks.rados.rados.0.smithi064.stdout:45: oid 45 version 12 is already newer than 10 2024-03-04T06:08:04.595 INFO:tasks.rados.rados.0.smithi064.stdout:45: oid 45 version 12 is already newer than 11 2024-03-04T06:08:04.595 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 12 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-04T06:08:04.595 INFO:tasks.rados.rados.0.smithi064.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-03-04T06:08:04.597 INFO:tasks.rados.rados.0.smithi064.stdout:49: finishing write tid 2 to smithi06431743-49 2024-03-04T06:08:04.598 INFO:tasks.rados.rados.0.smithi064.stdout:52: finishing write tid 1 to smithi06431743-14 2024-03-04T06:08:04.605 INFO:tasks.rados.rados.0.smithi064.stdout:53: expect (ObjNum 9 snap 0 seq_num 9) 2024-03-04T06:08:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:08:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:08:04.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:04.695+0000 7f4b4019d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:04.701 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:04.700+0000 7f4b4019d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:04.703 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-04T06:08:03.785493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107956,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379404,"statfs":{"total":386547056640,"available":386436509696,"internally_reserved":0,"allocated":675840,"data_stored":1900108,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.079996"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.829404+0000","last_change":"2024-03-04T06:08:00.829404+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.829404+0000","last_undegraded":"2024-03-04T06:08:00.829404+0000","last_fullsized":"2024-03-04T06:08:00.829404+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.837238+0000","last_change":"2024-03-04T06:08:00.837238+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.837238+0000","last_undegraded":"2024-03-04T06:08:00.837238+0000","last_fullsized":"2024-03-04T06:08:00.837238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.818819+0000","last_change":"2024-03-04T06:08:00.818819+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.818819+0000","last_undegraded":"2024-03-04T06:08:00.818819+0000","last_fullsized":"2024-03-04T06:08:00.818819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":148,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.789652+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:08:00.789652+0000","last_peered":"2024-03-04T06:08:00.789652+0000","last_clean":"2024-03-04T06:08:00.789652+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:08:00.789652+0000","last_undegraded":"2024-03-04T06:08:00.789652+0000","last_fullsized":"2024-03-04T06:08:00.789652+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.797748+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:08:00.797748+0000","last_peered":"2024-03-04T06:08:00.797748+0000","last_clean":"2024-03-04T06:08:00.797748+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:08:00.797748+0000","last_undegraded":"2024-03-04T06:08:00.797748+0000","last_fullsized":"2024-03-04T06:08:00.797748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.802057+0000","last_change":"2024-03-04T06:08:00.802057+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.802057+0000","last_undegraded":"2024-03-04T06:08:00.802057+0000","last_fullsized":"2024-03-04T06:08:00.802057+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.313279+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:07:56.313279+0000","last_peered":"2024-03-04T06:07:56.313279+0000","last_clean":"2024-03-04T06:07:56.313279+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:07:56.313279+0000","last_undegraded":"2024-03-04T06:07:56.313279+0000","last_fullsized":"2024-03-04T06:07:56.313279+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.808984+0000","last_change":"2024-03-04T06:08:00.808984+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.808984+0000","last_undegraded":"2024-03-04T06:08:00.808984+0000","last_fullsized":"2024-03-04T06:08:00.808984+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.792136+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:08:00.792136+0000","last_peered":"2024-03-04T06:08:00.792136+0000","last_clean":"2024-03-04T06:08:00.792136+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:08:00.792136+0000","last_undegraded":"2024-03-04T06:08:00.792136+0000","last_fullsized":"2024-03-04T06:08:00.792136+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.799251+0000","last_change":"2024-03-04T06:08:00.799251+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.799251+0000","last_undegraded":"2024-03-04T06:08:00.799251+0000","last_fullsized":"2024-03-04T06:08:00.799251+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.307816+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:07:56.307816+0000","last_peered":"2024-03-04T06:07:56.307816+0000","last_clean":"2024-03-04T06:07:56.307816+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:07:56.307816+0000","last_undegraded":"2024-03-04T06:07:56.307816+0000","last_fullsized":"2024-03-04T06:07:56.307816+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.810651+0000","last_change":"2024-03-04T06:08:00.810651+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.810651+0000","last_undegraded":"2024-03-04T06:08:00.810651+0000","last_fullsized":"2024-03-04T06:08:00.810651+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.314848+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:07:56.314848+0000","last_peered":"2024-03-04T06:07:56.314848+0000","last_clean":"2024-03-04T06:07:56.314848+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:07:56.314848+0000","last_undegraded":"2024-03-04T06:07:56.314848+0000","last_fullsized":"2024-03-04T06:07:56.314848+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.794156+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:08:00.794156+0000","last_peered":"2024-03-04T06:08:00.794156+0000","last_clean":"2024-03-04T06:08:00.794156+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:08:00.794156+0000","last_undegraded":"2024-03-04T06:08:00.794156+0000","last_fullsized":"2024-03-04T06:08:00.794156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.823755+0000","last_change":"2024-03-04T06:08:00.823755+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.823755+0000","last_undegraded":"2024-03-04T06:08:00.823755+0000","last_fullsized":"2024-03-04T06:08:00.823755+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.308428+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:07:56.308428+0000","last_peered":"2024-03-04T06:07:56.308428+0000","last_clean":"2024-03-04T06:07:56.308428+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:07:56.308428+0000","last_undegraded":"2024-03-04T06:07:56.308428+0000","last_fullsized":"2024-03-04T06:07:56.308428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.797136+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:08:00.797136+0000","last_peered":"2024-03-04T06:08:00.797136+0000","last_clean":"2024-03-04T06:08:00.797136+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:08:00.797136+0000","last_undegraded":"2024-03-04T06:08:00.797136+0000","last_fullsized":"2024-03-04T06:08:00.797136+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705680,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":188416,"data_stored":624411,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705680,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":126976,"data_stored":34043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705679,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705679,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:08:04.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:08:04.705 INFO:tasks.rados.rados.0.smithi064.stdout:42: done (12 left) 2024-03-04T06:08:04.705 INFO:tasks.rados.rados.0.smithi064.stdout:44: done (11 left) 2024-03-04T06:08:04.705 INFO:tasks.rados.rados.0.smithi064.stdout:45: done (10 left) 2024-03-04T06:08:04.705 INFO:tasks.rados.rados.0.smithi064.stdout:46: done (9 left) 2024-03-04T06:08:04.706 INFO:tasks.rados.rados.0.smithi064.stdout:47: done (8 left) 2024-03-04T06:08:04.706 INFO:tasks.rados.rados.0.smithi064.stdout:48: done (7 left) 2024-03-04T06:08:04.706 INFO:tasks.rados.rados.0.smithi064.stdout:53: done (6 left) 2024-03-04T06:08:04.706 INFO:tasks.rados.rados.0.smithi064.stdout:54: done (5 left) 2024-03-04T06:08:04.706 INFO:tasks.rados.rados.0.smithi064.stdout:55: write oid 32 current snap is 0 2024-03-04T06:08:04.706 INFO:tasks.rados.rados.0.smithi064.stdout:55: seq_num 53 ranges {490771=686315,1885075=406346} 2024-03-04T06:08:04.710 INFO:tasks.rados.rados.0.smithi064.stdout:55: writing smithi06431743-32 from 490771 to 1177086 tid 1 2024-03-04T06:08:04.716 INFO:tasks.rados.rados.0.smithi064.stdout:55: writing smithi06431743-32 from 1885075 to 2291421 tid 2 2024-03-04T06:08:04.716 INFO:tasks.rados.rados.0.smithi064.stdout:56: write oid 3 current snap is 0 2024-03-04T06:08:04.716 INFO:tasks.rados.rados.0.smithi064.stdout:50: finishing write tid 2 to smithi06431743-50 2024-03-04T06:08:04.716 INFO:tasks.rados.rados.0.smithi064.stdout:51: finishing write tid 3 to smithi06431743-2 2024-03-04T06:08:04.717 INFO:tasks.rados.rados.0.smithi064.stdout:56: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-03-04T06:08:04.723 INFO:tasks.rados.rados.0.smithi064.stdout:56: writing smithi06431743-3 from 507578 to 985495 tid 1 2024-03-04T06:08:04.729 INFO:tasks.rados.rados.0.smithi064.stdout:56: writing smithi06431743-3 from 1543557 to 2093561 tid 2 2024-03-04T06:08:04.734 INFO:tasks.rados.rados.0.smithi064.stdout:56: writing smithi06431743-3 from 2833665 to 2907578 tid 3 2024-03-04T06:08:04.734 INFO:tasks.rados.rados.0.smithi064.stdout:57: write (excl) oid 19 current snap is 0 2024-03-04T06:08:04.734 INFO:tasks.rados.rados.0.smithi064.stdout:57: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-03-04T06:08:04.743 INFO:tasks.rados.rados.0.smithi064.stdout:57: writing smithi06431743-19 from 524385 to 1077551 tid 1 2024-03-04T06:08:04.747 INFO:tasks.rados.rados.0.smithi064.stdout:57: writing smithi06431743-19 from 1602039 to 2012054 tid 2 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:57: writing smithi06431743-19 from 2577441 to 2924385 tid 3 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:58: read oid 28 snap -1 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:58: expect (ObjNum 27 snap 0 seq_num 27) 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:59: write (excl) oid 37 current snap is 0 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:49: finishing write tid 3 to smithi06431743-49 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:49: finishing write tid 4 to smithi06431743-49 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:49: oid 49 updating version 0 to 4 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:49: oid 49 updating version 4 to 6 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:49: oid 49 version 6 is already newer than 5 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 6 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:50: finishing write tid 3 to smithi06431743-50 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:50: finishing write tid 4 to smithi06431743-50 2024-03-04T06:08:04.754 INFO:tasks.rados.rados.0.smithi064.stdout:50: oid 50 updating version 0 to 12 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:50: oid 50 version 12 is already newer than 11 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:50: oid 50 version 12 is already newer than 10 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:51: finishing write tid 4 to smithi06431743-2 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:43: finishing write tid 1 to smithi06431743-43 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:51: finishing write tid 5 to smithi06431743-2 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:51: oid 2 updating version 0 to 11 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:51: oid 2 updating version 11 to 13 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:51: oid 2 version 13 is already newer than 10 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:51: oid 2 version 13 is already newer than 13 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 13 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:51: left oid 2 (ObjNum 50 snap 0 seq_num 50) 2024-03-04T06:08:04.755 INFO:tasks.rados.rados.0.smithi064.stdout:59: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-03-04T06:08:04.761 INFO:tasks.rados.rados.0.smithi064.stdout:59: writing smithi06431743-37 from 541192 to 1169607 tid 1 2024-03-04T06:08:04.767 INFO:tasks.rados.rados.0.smithi064.stdout:59: writing smithi06431743-37 from 1660521 to 2214194 tid 2 2024-03-04T06:08:04.772 INFO:tasks.rados.rados.0.smithi064.stdout:59: writing smithi06431743-37 from 2888511 to 2941192 tid 3 2024-03-04T06:08:04.772 INFO:tasks.rados.rados.0.smithi064.stdout:49: done (9 left) 2024-03-04T06:08:04.772 INFO:tasks.rados.rados.0.smithi064.stdout:50: done (8 left) 2024-03-04T06:08:04.772 INFO:tasks.rados.rados.0.smithi064.stdout:51: done (7 left) 2024-03-04T06:08:04.772 INFO:tasks.rados.rados.0.smithi064.stdout:60: delete oid 1 current snap is 0 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:52: finishing write tid 2 to smithi06431743-14 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:52: finishing write tid 3 to smithi06431743-14 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:52: finishing write tid 4 to smithi06431743-14 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:52: oid 14 updating version 0 to 13 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:52: oid 14 updating version 13 to 15 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:52: oid 14 version 15 is already newer than 14 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:52: left oid 14 (ObjNum 51 snap 0 seq_num 51) 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:43: finishing write tid 2 to smithi06431743-43 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:43: finishing write tid 3 to smithi06431743-43 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:43: finishing write tid 4 to smithi06431743-43 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:43: oid 43 updating version 0 to 7 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:43: oid 43 updating version 7 to 9 2024-03-04T06:08:04.773 INFO:tasks.rados.rados.0.smithi064.stdout:43: oid 43 version 9 is already newer than 9 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:55: finishing write tid 1 to smithi06431743-32 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:56: finishing write tid 1 to smithi06431743-3 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:55: finishing write tid 2 to smithi06431743-32 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:55: finishing write tid 3 to smithi06431743-32 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:55: finishing write tid 4 to smithi06431743-32 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:55: oid 32 updating version 0 to 21 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:55: oid 32 version 21 is already newer than 21 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:55: oid 32 version 21 is already newer than 19 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 21 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:55: left oid 32 (ObjNum 53 snap 0 seq_num 53) 2024-03-04T06:08:04.774 INFO:tasks.rados.rados.0.smithi064.stdout:57: finishing write tid 1 to smithi06431743-19 2024-03-04T06:08:04.782 INFO:tasks.rados.rados.0.smithi064.stdout:58: expect (ObjNum 27 snap 0 seq_num 27) 2024-03-04T06:08:04.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:04.852+0000 7f4ab059d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:04.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:04.858+0000 7f4ab059d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:04.892 INFO:tasks.rados.rados.0.smithi064.stdout:57: finishing write tid 2 to smithi06431743-19 2024-03-04T06:08:04.892 INFO:tasks.rados.rados.0.smithi064.stdout:56: finishing write tid 2 to smithi06431743-3 2024-03-04T06:08:04.892 INFO:tasks.rados.rados.0.smithi064.stdout:56: finishing write tid 3 to smithi06431743-3 2024-03-04T06:08:04.892 INFO:tasks.rados.rados.0.smithi064.stdout:57: finishing write tid 3 to smithi06431743-19 2024-03-04T06:08:04.892 INFO:tasks.rados.rados.0.smithi064.stdout:59: finishing write tid 1 to smithi06431743-37 2024-03-04T06:08:04.892 INFO:tasks.rados.rados.0.smithi064.stdout:57: finishing write tid 4 to smithi06431743-19 2024-03-04T06:08:04.892 INFO:tasks.rados.rados.0.smithi064.stdout:57: finishing write tid 5 to smithi06431743-19 2024-03-04T06:08:04.892 INFO:tasks.rados.rados.0.smithi064.stdout:57: oid 19 updating version 0 to 20 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:57: oid 19 version 20 is already newer than 19 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:57: oid 19 updating version 20 to 22 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:57: oid 19 version 22 is already newer than 22 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 22 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:57: left oid 19 (ObjNum 55 snap 0 seq_num 55) 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:59: finishing write tid 2 to smithi06431743-37 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:56: finishing write tid 4 to smithi06431743-3 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:43: done (7 left) 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:52: done (6 left) 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:55: done (5 left) 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:57: done (4 left) 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:58: done (3 left) 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:60: done (2 left) 2024-03-04T06:08:04.893 INFO:tasks.rados.rados.0.smithi064.stdout:61: delete oid 5 current snap is 0 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:56: finishing write tid 5 to smithi06431743-3 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:56: oid 3 updating version 0 to 12 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:56: oid 3 updating version 12 to 13 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:56: oid 3 version 13 is already newer than 12 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:56: oid 3 version 13 is already newer than 10 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 13 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:56: left oid 3 (ObjNum 54 snap 0 seq_num 54) 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:59: finishing write tid 3 to smithi06431743-37 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:59: finishing write tid 4 to smithi06431743-37 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:59: finishing write tid 5 to smithi06431743-37 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:59: oid 37 updating version 0 to 10 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:59: oid 37 updating version 10 to 12 2024-03-04T06:08:04.894 INFO:tasks.rados.rados.0.smithi064.stdout:59: oid 37 version 12 is already newer than 11 2024-03-04T06:08:04.895 INFO:tasks.rados.rados.0.smithi064.stdout:59: oid 37 updating version 12 to 13 2024-03-04T06:08:04.895 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 13 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-03-04T06:08:04.895 INFO:tasks.rados.rados.0.smithi064.stdout:59: left oid 37 (ObjNum 56 snap 0 seq_num 56) 2024-03-04T06:08:04.899 INFO:tasks.rados.rados.0.smithi064.stdout:56: done (2 left) 2024-03-04T06:08:04.899 INFO:tasks.rados.rados.0.smithi064.stdout:59: done (1 left) 2024-03-04T06:08:04.899 INFO:tasks.rados.rados.0.smithi064.stdout:61: done (0 left) 2024-03-04T06:08:04.899 INFO:tasks.rados.rados.0.smithi064.stdout:62: delete oid 27 current snap is 0 2024-03-04T06:08:04.907 INFO:tasks.rados.rados.0.smithi064.stdout:62: done (0 left) 2024-03-04T06:08:04.907 INFO:tasks.rados.rados.0.smithi064.stdout:63: read oid 15 snap -1 2024-03-04T06:08:04.908 INFO:tasks.rados.rados.0.smithi064.stdout:63: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-04T06:08:04.908 INFO:tasks.rados.rados.0.smithi064.stdout:64: snap_create 2024-03-04T06:08:04.922 INFO:tasks.rados.rados.0.smithi064.stdout:63: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-04T06:08:05.118 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:08:05.119 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":26,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":18,"num_osds":4,"num_up_osds":4,"osd_up_since":1709532465,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"unknown","count":8},{"state_name":"creating+peering","count":8}],"num_pgs":25,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":110546944,"bytes_avail":386436509696,"bytes_total":386547056640,"unknown_pgs_ratio":0.31999999284744263,"inactive_pgs_ratio":0.31999999284744263},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-03-04T06:07:38.205398+0000","services":{}},"progress_events":{}} 2024-03-04T06:08:05.131 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:08:05.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:08:05.279 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:05.274+0000 7f32e966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:05.281 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:05.281+0000 7f32e966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:05.471 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:08:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:08:05.483 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-04T06:08:04.707038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107956,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379404,"statfs":{"total":386547056640,"available":386436509696,"internally_reserved":0,"allocated":675840,"data_stored":1900108,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000965"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.829404+0000","last_change":"2024-03-04T06:08:00.829404+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.829404+0000","last_undegraded":"2024-03-04T06:08:00.829404+0000","last_fullsized":"2024-03-04T06:08:00.829404+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.837238+0000","last_change":"2024-03-04T06:08:00.837238+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.837238+0000","last_undegraded":"2024-03-04T06:08:00.837238+0000","last_fullsized":"2024-03-04T06:08:00.837238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.818819+0000","last_change":"2024-03-04T06:08:00.818819+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.818819+0000","last_undegraded":"2024-03-04T06:08:00.818819+0000","last_fullsized":"2024-03-04T06:08:00.818819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":148,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.789652+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:08:00.789652+0000","last_peered":"2024-03-04T06:08:00.789652+0000","last_clean":"2024-03-04T06:08:00.789652+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:08:00.789652+0000","last_undegraded":"2024-03-04T06:08:00.789652+0000","last_fullsized":"2024-03-04T06:08:00.789652+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.797748+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:08:00.797748+0000","last_peered":"2024-03-04T06:08:00.797748+0000","last_clean":"2024-03-04T06:08:00.797748+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:08:00.797748+0000","last_undegraded":"2024-03-04T06:08:00.797748+0000","last_fullsized":"2024-03-04T06:08:00.797748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.802057+0000","last_change":"2024-03-04T06:08:00.802057+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.802057+0000","last_undegraded":"2024-03-04T06:08:00.802057+0000","last_fullsized":"2024-03-04T06:08:00.802057+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.313279+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:07:56.313279+0000","last_peered":"2024-03-04T06:07:56.313279+0000","last_clean":"2024-03-04T06:07:56.313279+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:07:56.313279+0000","last_undegraded":"2024-03-04T06:07:56.313279+0000","last_fullsized":"2024-03-04T06:07:56.313279+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.808984+0000","last_change":"2024-03-04T06:08:00.808984+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.808984+0000","last_undegraded":"2024-03-04T06:08:00.808984+0000","last_fullsized":"2024-03-04T06:08:00.808984+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.792136+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:08:00.792136+0000","last_peered":"2024-03-04T06:08:00.792136+0000","last_clean":"2024-03-04T06:08:00.792136+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:08:00.792136+0000","last_undegraded":"2024-03-04T06:08:00.792136+0000","last_fullsized":"2024-03-04T06:08:00.792136+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.799251+0000","last_change":"2024-03-04T06:08:00.799251+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.799251+0000","last_undegraded":"2024-03-04T06:08:00.799251+0000","last_fullsized":"2024-03-04T06:08:00.799251+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.307816+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:07:56.307816+0000","last_peered":"2024-03-04T06:07:56.307816+0000","last_clean":"2024-03-04T06:07:56.307816+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:07:56.307816+0000","last_undegraded":"2024-03-04T06:07:56.307816+0000","last_fullsized":"2024-03-04T06:07:56.307816+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.810651+0000","last_change":"2024-03-04T06:08:00.810651+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.810651+0000","last_undegraded":"2024-03-04T06:08:00.810651+0000","last_fullsized":"2024-03-04T06:08:00.810651+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.314848+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:07:56.314848+0000","last_peered":"2024-03-04T06:07:56.314848+0000","last_clean":"2024-03-04T06:07:56.314848+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:07:56.314848+0000","last_undegraded":"2024-03-04T06:07:56.314848+0000","last_fullsized":"2024-03-04T06:07:56.314848+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.794156+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:08:00.794156+0000","last_peered":"2024-03-04T06:08:00.794156+0000","last_clean":"2024-03-04T06:08:00.794156+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:08:00.794156+0000","last_undegraded":"2024-03-04T06:08:00.794156+0000","last_fullsized":"2024-03-04T06:08:00.794156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1,"reported_epoch":15,"state":"creating+peering","last_fresh":"2024-03-04T06:08:00.823755+0000","last_change":"2024-03-04T06:08:00.823755+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.823755+0000","last_undegraded":"2024-03-04T06:08:00.823755+0000","last_fullsized":"2024-03-04T06:08:00.823755+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-04T06:07:56.308428+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:07:56.308428+0000","last_peered":"2024-03-04T06:07:56.308428+0000","last_clean":"2024-03-04T06:07:56.308428+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:07:56.308428+0000","last_undegraded":"2024-03-04T06:07:56.308428+0000","last_fullsized":"2024-03-04T06:07:56.308428+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-04T06:08:00.773669+0000","last_change":"2024-03-04T06:08:00.773669+0000","last_active":"2024-03-04T06:08:00.773669+0000","last_peered":"2024-03-04T06:08:00.773669+0000","last_clean":"2024-03-04T06:08:00.773669+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:08:00.773669+0000","last_undegraded":"2024-03-04T06:08:00.773669+0000","last_fullsized":"2024-03-04T06:08:00.773669+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-04T06:08:00.797136+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:08:00.797136+0000","last_peered":"2024-03-04T06:08:00.797136+0000","last_clean":"2024-03-04T06:08:00.797136+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:08:00.797136+0000","last_undegraded":"2024-03-04T06:08:00.797136+0000","last_fullsized":"2024-03-04T06:08:00.797136+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705680,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":188416,"data_stored":624411,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705680,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":124,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":126976,"data_stored":34043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705679,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705679,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:08:05.796 INFO:tasks.rados.rados.0.smithi064.stdout:63: done (1 left) 2024-03-04T06:08:05.797 INFO:tasks.rados.rados.0.smithi064.stdout:64: done (0 left) 2024-03-04T06:08:05.797 INFO:tasks.rados.rados.0.smithi064.stdout:65: write oid 40 current snap is 1 2024-03-04T06:08:05.797 INFO:tasks.rados.rados.0.smithi064.stdout:65: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-03-04T06:08:05.802 INFO:tasks.rados.rados.0.smithi064.stdout:65: writing smithi06431743-40 from 608420 to 1137831 tid 1 2024-03-04T06:08:05.810 INFO:tasks.rados.rados.0.smithi064.stdout:65: writing smithi06431743-40 from 1778096 to 2339107 tid 2 2024-03-04T06:08:05.815 INFO:tasks.rados.rados.0.smithi064.stdout:65: writing smithi06431743-40 from 2881850 to 3008420 tid 3 2024-03-04T06:08:05.815 INFO:tasks.rados.rados.0.smithi064.stdout:66: read oid 48 snap 0 2024-03-04T06:08:05.815 INFO:tasks.rados.rados.0.smithi064.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2024-03-04T06:08:05.815 INFO:tasks.rados.rados.0.smithi064.stdout:67: read oid 28 snap 0 2024-03-04T06:08:05.816 INFO:tasks.rados.rados.0.smithi064.stdout:67: expect (ObjNum 27 snap 0 seq_num 27) 2024-03-04T06:08:05.816 INFO:tasks.rados.rados.0.smithi064.stdout:68: read oid 13 snap -1 2024-03-04T06:08:05.816 INFO:tasks.rados.rados.0.smithi064.stdout:68: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-04T06:08:05.816 INFO:tasks.rados.rados.0.smithi064.stdout:69: read oid 43 snap -1 2024-03-04T06:08:05.816 INFO:tasks.rados.rados.0.smithi064.stdout:69: expect (ObjNum 42 snap 0 seq_num 42) 2024-03-04T06:08:05.816 INFO:tasks.rados.rados.0.smithi064.stdout:70: read oid 36 snap -1 2024-03-04T06:08:05.816 INFO:tasks.rados.rados.0.smithi064.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2024-03-04T06:08:05.816 INFO:tasks.rados.rados.0.smithi064.stdout:71: read oid 50 snap -1 2024-03-04T06:08:05.816 INFO:tasks.rados.rados.0.smithi064.stdout:71: expect (ObjNum 49 snap 0 seq_num 49) 2024-03-04T06:08:05.816 INFO:tasks.rados.rados.0.smithi064.stdout:72: read oid 8 snap 0 2024-03-04T06:08:05.817 INFO:tasks.rados.rados.0.smithi064.stdout:72: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-04T06:08:05.817 INFO:tasks.rados.rados.0.smithi064.stdout:73: read oid 27 snap 0 2024-03-04T06:08:05.817 INFO:tasks.rados.rados.0.smithi064.stdout:73: expect deleted 2024-03-04T06:08:05.817 INFO:tasks.rados.rados.0.smithi064.stdout:74: snap_create 2024-03-04T06:08:05.862 INFO:tasks.rados.rados.0.smithi064.stdout:65: finishing write tid 1 to smithi06431743-40 2024-03-04T06:08:05.863 INFO:tasks.rados.rados.0.smithi064.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2024-03-04T06:08:05.962 INFO:tasks.rados.rados.0.smithi064.stdout:67: expect (ObjNum 27 snap 0 seq_num 27) 2024-03-04T06:08:06.052 INFO:tasks.rados.rados.0.smithi064.stdout:65: finishing write tid 2 to smithi06431743-40 2024-03-04T06:08:06.052 INFO:tasks.rados.rados.0.smithi064.stdout:68: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-04T06:08:06.133 INFO:tasks.rados.rados.0.smithi064.stdout:72: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-04T06:08:06.212 INFO:tasks.rados.rados.0.smithi064.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2024-03-04T06:08:06.315 INFO:tasks.rados.rados.0.smithi064.stdout:71: expect (ObjNum 49 snap 0 seq_num 49) 2024-03-04T06:08:06.431 INFO:tasks.rados.rados.0.smithi064.stdout:65: finishing write tid 3 to smithi06431743-40 2024-03-04T06:08:06.431 INFO:tasks.rados.rados.0.smithi064.stdout:65: finishing write tid 4 to smithi06431743-40 2024-03-04T06:08:06.431 INFO:tasks.rados.rados.0.smithi064.stdout:65: finishing write tid 5 to smithi06431743-40 2024-03-04T06:08:06.431 INFO:tasks.rados.rados.0.smithi064.stdout:65: oid 40 updating version 0 to 12 2024-03-04T06:08:06.431 INFO:tasks.rados.rados.0.smithi064.stdout:65: oid 40 version 12 is already newer than 12 2024-03-04T06:08:06.431 INFO:tasks.rados.rados.0.smithi064.stdout:65: oid 40 updating version 12 to 14 2024-03-04T06:08:06.431 INFO:tasks.rados.rados.0.smithi064.stdout:65: oid 40 version 14 is already newer than 14 2024-03-04T06:08:06.431 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 14 (ObjNum 60 snap 1 seq_num 60) dirty exists 2024-03-04T06:08:06.432 INFO:tasks.rados.rados.0.smithi064.stdout:65: left oid 40 (ObjNum 60 snap 1 seq_num 60) 2024-03-04T06:08:06.432 INFO:tasks.rados.rados.0.smithi064.stdout:69: expect (ObjNum 42 snap 0 seq_num 42) 2024-03-04T06:08:06.806 INFO:tasks.rados.rados.0.smithi064.stdout:65: done (9 left) 2024-03-04T06:08:06.806 INFO:tasks.rados.rados.0.smithi064.stdout:66: done (8 left) 2024-03-04T06:08:06.806 INFO:tasks.rados.rados.0.smithi064.stdout:67: done (7 left) 2024-03-04T06:08:06.806 INFO:tasks.rados.rados.0.smithi064.stdout:68: done (6 left) 2024-03-04T06:08:06.806 INFO:tasks.rados.rados.0.smithi064.stdout:69: done (5 left) 2024-03-04T06:08:06.806 INFO:tasks.rados.rados.0.smithi064.stdout:70: done (4 left) 2024-03-04T06:08:06.806 INFO:tasks.rados.rados.0.smithi064.stdout:71: done (3 left) 2024-03-04T06:08:06.806 INFO:tasks.rados.rados.0.smithi064.stdout:72: done (2 left) 2024-03-04T06:08:06.806 INFO:tasks.rados.rados.0.smithi064.stdout:73: done (1 left) 2024-03-04T06:08:06.807 INFO:tasks.rados.rados.0.smithi064.stdout:74: done (0 left) 2024-03-04T06:08:06.807 INFO:tasks.rados.rados.0.smithi064.stdout:75: write oid 45 current snap is 2 2024-03-04T06:08:06.807 INFO:tasks.rados.rados.0.smithi064.stdout:75: seq_num 61 ranges {625227=721013,1952931=704669} 2024-03-04T06:08:06.812 INFO:tasks.rados.rados.0.smithi064.stdout:75: writing smithi06431743-45 from 625227 to 1346240 tid 1 2024-03-04T06:08:06.821 INFO:tasks.rados.rados.0.smithi064.stdout:75: writing smithi06431743-45 from 1952931 to 2657600 tid 2 2024-03-04T06:08:06.821 INFO:tasks.rados.rados.0.smithi064.stdout:76: read oid 16 snap -1 2024-03-04T06:08:06.821 INFO:tasks.rados.rados.0.smithi064.stdout:76: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-04T06:08:06.821 INFO:tasks.rados.rados.0.smithi064.stdout:77: write oid 46 current snap is 2 2024-03-04T06:08:06.821 INFO:tasks.rados.rados.0.smithi064.stdout:77: seq_num 62 ranges {642034=796262,1895060=564680} 2024-03-04T06:08:06.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:06.828 INFO:tasks.rados.rados.0.smithi064.stdout:77: writing smithi06431743-46 from 642034 to 1438296 tid 1 2024-03-04T06:08:06.835 INFO:tasks.rados.rados.0.smithi064.stdout:77: writing smithi06431743-46 from 1895060 to 2459740 tid 2 2024-03-04T06:08:06.835 INFO:tasks.rados.rados.0.smithi064.stdout:78: read oid 26 snap 0 2024-03-04T06:08:06.835 INFO:tasks.rados.rados.0.smithi064.stdout:78: expect (ObjNum 25 snap 0 seq_num 25) 2024-03-04T06:08:06.836 INFO:tasks.rados.rados.0.smithi064.stdout:79: read oid 7 snap -1 2024-03-04T06:08:06.836 INFO:tasks.rados.rados.0.smithi064.stdout:79: expect (ObjNum 6 snap 0 seq_num 6) 2024-03-04T06:08:06.836 INFO:tasks.rados.rados.0.smithi064.stdout:80: write oid 44 current snap is 2 2024-03-04T06:08:06.836 INFO:tasks.rados.rados.0.smithi064.stdout:80: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-03-04T06:08:06.841 INFO:tasks.rados.rados.0.smithi064.stdout:80: writing smithi06431743-44 from 658841 to 1130352 tid 1 2024-03-04T06:08:06.849 INFO:tasks.rados.rados.0.smithi064.stdout:80: writing smithi06431743-44 from 1553542 to 2261880 tid 2 2024-03-04T06:08:06.854 INFO:tasks.rados.rados.0.smithi064.stdout:80: writing smithi06431743-44 from 2964119 to 3058841 tid 3 2024-03-04T06:08:06.854 INFO:tasks.rados.rados.0.smithi064.stdout:81: read oid 30 snap -1 2024-03-04T06:08:06.854 INFO:tasks.rados.rados.0.smithi064.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-04T06:08:06.854 INFO:tasks.rados.rados.0.smithi064.stdout:82: snap_remove snap 1 2024-03-04T06:08:06.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:06.953+0000 7f6c5071d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:06.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:06.958+0000 7f6c5071d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:07.813 INFO:tasks.rados.rados.0.smithi064.stdout:82: done (7 left) 2024-03-04T06:08:07.814 INFO:tasks.rados.rados.0.smithi064.stdout:83: delete oid 49 current snap is 2 2024-03-04T06:08:07.814 INFO:tasks.rados.rados.0.smithi064.stdout:79: expect (ObjNum 6 snap 0 seq_num 6) 2024-03-04T06:08:07.898 INFO:tasks.rados.rados.0.smithi064.stdout:77: finishing write tid 1 to smithi06431743-46 2024-03-04T06:08:07.898 INFO:tasks.rados.rados.0.smithi064.stdout:81: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-04T06:08:07.991 INFO:tasks.rados.rados.0.smithi064.stdout:76: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-04T06:08:08.076 INFO:tasks.rados.rados.0.smithi064.stdout:75: finishing write tid 1 to smithi06431743-45 2024-03-04T06:08:08.076 INFO:tasks.rados.rados.0.smithi064.stdout:77: finishing write tid 2 to smithi06431743-46 2024-03-04T06:08:08.076 INFO:tasks.rados.rados.0.smithi064.stdout:77: finishing write tid 3 to smithi06431743-46 2024-03-04T06:08:08.076 INFO:tasks.rados.rados.0.smithi064.stdout:75: finishing write tid 2 to smithi06431743-45 2024-03-04T06:08:08.076 INFO:tasks.rados.rados.0.smithi064.stdout:77: finishing write tid 4 to smithi06431743-46 2024-03-04T06:08:08.077 INFO:tasks.rados.rados.0.smithi064.stdout:77: oid 46 updating version 0 to 26 2024-03-04T06:08:08.077 INFO:tasks.rados.rados.0.smithi064.stdout:77: oid 46 version 26 is already newer than 25 2024-03-04T06:08:08.077 INFO:tasks.rados.rados.0.smithi064.stdout:77: oid 46 version 26 is already newer than 24 2024-03-04T06:08:08.077 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 26 (ObjNum 62 snap 2 seq_num 62) dirty exists 2024-03-04T06:08:08.077 INFO:tasks.rados.rados.0.smithi064.stdout:77: left oid 46 (ObjNum 62 snap 2 seq_num 62) 2024-03-04T06:08:08.077 INFO:tasks.rados.rados.0.smithi064.stdout:78: expect (ObjNum 25 snap 0 seq_num 25) 2024-03-04T06:08:08.167 INFO:tasks.rados.rados.0.smithi064.stdout:80: finishing write tid 1 to smithi06431743-44 2024-03-04T06:08:08.167 INFO:tasks.rados.rados.0.smithi064.stdout:75: finishing write tid 3 to smithi06431743-45 2024-03-04T06:08:08.167 INFO:tasks.rados.rados.0.smithi064.stdout:75: finishing write tid 4 to smithi06431743-45 2024-03-04T06:08:08.167 INFO:tasks.rados.rados.0.smithi064.stdout:75: oid 45 updating version 0 to 14 2024-03-04T06:08:08.168 INFO:tasks.rados.rados.0.smithi064.stdout:75: oid 45 updating version 14 to 16 2024-03-04T06:08:08.168 INFO:tasks.rados.rados.0.smithi064.stdout:75: oid 45 version 16 is already newer than 16 2024-03-04T06:08:08.168 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 16 (ObjNum 61 snap 2 seq_num 61) dirty exists 2024-03-04T06:08:08.168 INFO:tasks.rados.rados.0.smithi064.stdout:75: left oid 45 (ObjNum 61 snap 2 seq_num 61) 2024-03-04T06:08:08.168 INFO:tasks.rados.rados.0.smithi064.stdout:80: finishing write tid 2 to smithi06431743-44 2024-03-04T06:08:08.168 INFO:tasks.rados.rados.0.smithi064.stdout:80: finishing write tid 3 to smithi06431743-44 2024-03-04T06:08:08.168 INFO:tasks.rados.rados.0.smithi064.stdout:80: finishing write tid 4 to smithi06431743-44 2024-03-04T06:08:08.168 INFO:tasks.rados.rados.0.smithi064.stdout:80: finishing write tid 5 to smithi06431743-44 2024-03-04T06:08:08.168 INFO:tasks.rados.rados.0.smithi064.stdout:80: oid 44 updating version 0 to 13 2024-03-04T06:08:08.168 INFO:tasks.rados.rados.0.smithi064.stdout:80: oid 44 version 13 is already newer than 11 2024-03-04T06:08:08.169 INFO:tasks.rados.rados.0.smithi064.stdout:80: oid 44 version 13 is already newer than 13 2024-03-04T06:08:08.169 INFO:tasks.rados.rados.0.smithi064.stdout:80: oid 44 updating version 13 to 14 2024-03-04T06:08:08.169 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 14 (ObjNum 63 snap 2 seq_num 63) dirty exists 2024-03-04T06:08:08.169 INFO:tasks.rados.rados.0.smithi064.stdout:80: left oid 44 (ObjNum 63 snap 2 seq_num 63) 2024-03-04T06:08:08.169 INFO:tasks.rados.rados.0.smithi064.stdout:75: done (7 left) 2024-03-04T06:08:08.169 INFO:tasks.rados.rados.0.smithi064.stdout:76: done (6 left) 2024-03-04T06:08:08.169 INFO:tasks.rados.rados.0.smithi064.stdout:77: done (5 left) 2024-03-04T06:08:08.169 INFO:tasks.rados.rados.0.smithi064.stdout:78: done (4 left) 2024-03-04T06:08:08.169 INFO:tasks.rados.rados.0.smithi064.stdout:79: done (3 left) 2024-03-04T06:08:08.169 INFO:tasks.rados.rados.0.smithi064.stdout:80: done (2 left) 2024-03-04T06:08:08.169 INFO:tasks.rados.rados.0.smithi064.stdout:81: done (1 left) 2024-03-04T06:08:08.170 INFO:tasks.rados.rados.0.smithi064.stdout:83: done (0 left) 2024-03-04T06:08:08.170 INFO:tasks.rados.rados.0.smithi064.stdout:84: read oid 23 snap -1 2024-03-04T06:08:08.170 INFO:tasks.rados.rados.0.smithi064.stdout:84: expect (ObjNum 22 snap 0 seq_num 22) 2024-03-04T06:08:08.170 INFO:tasks.rados.rados.0.smithi064.stdout:85: read oid 26 snap 0 2024-03-04T06:08:08.170 INFO:tasks.rados.rados.0.smithi064.stdout:85: expect (ObjNum 25 snap 0 seq_num 25) 2024-03-04T06:08:08.170 INFO:tasks.rados.rados.0.smithi064.stdout:86: write oid 2 current snap is 2 2024-03-04T06:08:08.170 INFO:tasks.rados.rados.0.smithi064.stdout:86: seq_num 65 ranges {692455=622009,2070506=712007} 2024-03-04T06:08:08.173 INFO:tasks.rados.rados.0.smithi064.stdout:86: writing smithi06431743-2 from 692455 to 1314464 tid 1 2024-03-04T06:08:08.185 INFO:tasks.rados.rados.0.smithi064.stdout:86: writing smithi06431743-2 from 2070506 to 2782513 tid 2 2024-03-04T06:08:08.185 INFO:tasks.rados.rados.0.smithi064.stdout:87: snap_create 2024-03-04T06:08:08.185 INFO:tasks.rados.rados.0.smithi064.stdout:84: expect (ObjNum 22 snap 0 seq_num 22) 2024-03-04T06:08:08.283 INFO:tasks.rados.rados.0.smithi064.stdout:85: expect (ObjNum 25 snap 0 seq_num 25) 2024-03-04T06:08:08.373 INFO:tasks.rados.rados.0.smithi064.stdout:86: finishing write tid 1 to smithi06431743-2 2024-03-04T06:08:08.373 INFO:tasks.rados.rados.0.smithi064.stdout:86: finishing write tid 2 to smithi06431743-2 2024-03-04T06:08:08.373 INFO:tasks.rados.rados.0.smithi064.stdout:86: finishing write tid 3 to smithi06431743-2 2024-03-04T06:08:08.373 INFO:tasks.rados.rados.0.smithi064.stdout:86: finishing write tid 4 to smithi06431743-2 2024-03-04T06:08:08.373 INFO:tasks.rados.rados.0.smithi064.stdout:86: oid 2 updating version 0 to 17 2024-03-04T06:08:08.374 INFO:tasks.rados.rados.0.smithi064.stdout:86: oid 2 version 17 is already newer than 16 2024-03-04T06:08:08.374 INFO:tasks.rados.rados.0.smithi064.stdout:86: oid 2 version 17 is already newer than 15 2024-03-04T06:08:08.374 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 17 (ObjNum 65 snap 2 seq_num 65) dirty exists 2024-03-04T06:08:08.374 INFO:tasks.rados.rados.0.smithi064.stdout:86: left oid 2 (ObjNum 65 snap 2 seq_num 65) 2024-03-04T06:08:08.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:08:08.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:08.588+0000 7fad90d07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:08.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:08.594+0000 7fad90d07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:08.784 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:08:08.784 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:08:08.796 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-04T06:08:08.707947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":116019326,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":245,"num_read_kb":25769,"num_write":369,"num_write_kb":63881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":247208,"kb_used_data":139912,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377240152,"statfs":{"total":386547056640,"available":386293915648,"internally_reserved":0,"allocated":143269888,"data_stored":144467150,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":115428958,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":25678,"num_write":232,"num_write_kb":61987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.922454"},"pg_stats":[{"pgid":"3.f","version":"18'9","reported_seq":25,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.627494+0000","last_change":"2024-03-04T06:08:01.882593+0000","last_active":"2024-03-04T06:08:04.627494+0000","last_peered":"2024-03-04T06:08:04.627494+0000","last_clean":"2024-03-04T06:08:04.627494+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:08:04.627494+0000","last_undegraded":"2024-03-04T06:08:04.627494+0000","last_fullsized":"2024-03-04T06:08:04.627494+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"18'13","reported_seq":31,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.826623+0000","last_change":"2024-03-04T06:08:01.884604+0000","last_active":"2024-03-04T06:08:04.826623+0000","last_peered":"2024-03-04T06:08:04.826623+0000","last_clean":"2024-03-04T06:08:04.826623+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:08:04.826623+0000","last_undegraded":"2024-03-04T06:08:04.826623+0000","last_fullsized":"2024-03-04T06:08:04.826623+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":18,"num_write_kb":4968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"18'12","reported_seq":31,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.547355+0000","last_change":"2024-03-04T06:08:01.888867+0000","last_active":"2024-03-04T06:08:04.547355+0000","last_peered":"2024-03-04T06:08:04.547355+0000","last_clean":"2024-03-04T06:08:04.547355+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:08:04.547355+0000","last_undegraded":"2024-03-04T06:08:04.547355+0000","last_fullsized":"2024-03-04T06:08:04.547355+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9949612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"18'13","reported_seq":30,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.591487+0000","last_change":"2024-03-04T06:08:01.871856+0000","last_active":"2024-03-04T06:08:04.591487+0000","last_peered":"2024-03-04T06:08:04.591487+0000","last_clean":"2024-03-04T06:08:04.591487+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:08:04.591487+0000","last_undegraded":"2024-03-04T06:08:04.591487+0000","last_fullsized":"2024-03-04T06:08:04.591487+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7798349,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":17,"num_write_kb":4679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"18'3","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.844549+0000","last_change":"2024-03-04T06:08:01.880739+0000","last_active":"2024-03-04T06:08:05.844549+0000","last_peered":"2024-03-04T06:08:05.844549+0000","last_clean":"2024-03-04T06:08:05.844549+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:08:05.844549+0000","last_undegraded":"2024-03-04T06:08:05.844549+0000","last_fullsized":"2024-03-04T06:08:05.844549+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504210,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"18'9","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.587389+0000","last_change":"2024-03-04T06:08:01.891545+0000","last_active":"2024-03-04T06:08:04.587389+0000","last_peered":"2024-03-04T06:08:04.587389+0000","last_clean":"2024-03-04T06:08:04.587389+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:08:04.587389+0000","last_undegraded":"2024-03-04T06:08:04.587389+0000","last_fullsized":"2024-03-04T06:08:04.587389+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7495823,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5232,"num_write":12,"num_write_kb":3579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.852740+0000","last_change":"2024-03-04T06:08:01.888970+0000","last_active":"2024-03-04T06:08:03.852740+0000","last_peered":"2024-03-04T06:08:03.852740+0000","last_clean":"2024-03-04T06:08:03.852740+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:08:03.852740+0000","last_undegraded":"2024-03-04T06:08:03.852740+0000","last_fullsized":"2024-03-04T06:08:03.852740+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"18'6","reported_seq":23,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.598228+0000","last_change":"2024-03-04T06:08:01.876040+0000","last_active":"2024-03-04T06:08:04.598228+0000","last_peered":"2024-03-04T06:08:04.598228+0000","last_clean":"2024-03-04T06:08:04.598228+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:08:04.598228+0000","last_undegraded":"2024-03-04T06:08:04.598228+0000","last_fullsized":"2024-03-04T06:08:04.598228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":158,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.836113+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:08:05.836113+0000","last_peered":"2024-03-04T06:08:05.836113+0000","last_clean":"2024-03-04T06:08:05.836113+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:08:05.836113+0000","last_undegraded":"2024-03-04T06:08:05.836113+0000","last_fullsized":"2024-03-04T06:08:05.836113+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.818988+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:08:03.818988+0000","last_peered":"2024-03-04T06:08:03.818988+0000","last_clean":"2024-03-04T06:08:03.818988+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:08:03.818988+0000","last_undegraded":"2024-03-04T06:08:03.818988+0000","last_fullsized":"2024-03-04T06:08:03.818988+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"18'13","reported_seq":32,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.792825+0000","last_change":"2024-03-04T06:08:01.884009+0000","last_active":"2024-03-04T06:08:04.792825+0000","last_peered":"2024-03-04T06:08:04.792825+0000","last_clean":"2024-03-04T06:08:04.792825+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:08:04.792825+0000","last_undegraded":"2024-03-04T06:08:04.792825+0000","last_fullsized":"2024-03-04T06:08:04.792825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7949612,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":17,"num_write_kb":4330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.855484+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:08:03.855484+0000","last_peered":"2024-03-04T06:08:03.855484+0000","last_clean":"2024-03-04T06:08:03.855484+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:08:03.855484+0000","last_undegraded":"2024-03-04T06:08:03.855484+0000","last_fullsized":"2024-03-04T06:08:03.855484+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"18'6","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.875465+0000","last_change":"2024-03-04T06:08:01.878597+0000","last_active":"2024-03-04T06:08:05.875465+0000","last_peered":"2024-03-04T06:08:05.875465+0000","last_clean":"2024-03-04T06:08:05.875465+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:08:05.875465+0000","last_undegraded":"2024-03-04T06:08:05.875465+0000","last_fullsized":"2024-03-04T06:08:05.875465+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4890771,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5413,"num_write":8,"num_write_kb":2158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.810817+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:08:03.810817+0000","last_peered":"2024-03-04T06:08:03.810817+0000","last_clean":"2024-03-04T06:08:03.810817+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:08:03.810817+0000","last_undegraded":"2024-03-04T06:08:03.810817+0000","last_fullsized":"2024-03-04T06:08:03.810817+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"18'22","reported_seq":51,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.864521+0000","last_change":"2024-03-04T06:08:01.890481+0000","last_active":"2024-03-04T06:08:05.864521+0000","last_peered":"2024-03-04T06:08:05.864521+0000","last_clean":"2024-03-04T06:08:05.864521+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:08:05.864521+0000","last_undegraded":"2024-03-04T06:08:05.864521+0000","last_fullsized":"2024-03-04T06:08:05.864521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13075681,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":9709,"num_write":29,"num_write_kb":8088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.830698+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:08:03.830698+0000","last_peered":"2024-03-04T06:08:03.830698+0000","last_clean":"2024-03-04T06:08:03.830698+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:08:03.830698+0000","last_undegraded":"2024-03-04T06:08:03.830698+0000","last_fullsized":"2024-03-04T06:08:03.830698+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"18'22","reported_seq":43,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.807481+0000","last_change":"2024-03-04T06:08:01.906823+0000","last_active":"2024-03-04T06:08:04.807481+0000","last_peered":"2024-03-04T06:08:04.807481+0000","last_clean":"2024-03-04T06:08:04.807481+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:08:04.807481+0000","last_undegraded":"2024-03-04T06:08:04.807481+0000","last_fullsized":"2024-03-04T06:08:04.807481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12134489,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":30,"num_write_kb":7626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"18'9","reported_seq":30,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.917850+0000","last_change":"2024-03-04T06:08:01.908948+0000","last_active":"2024-03-04T06:08:04.917850+0000","last_peered":"2024-03-04T06:08:04.917850+0000","last_clean":"2024-03-04T06:08:04.917850+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:08:04.917850+0000","last_undegraded":"2024-03-04T06:08:04.917850+0000","last_fullsized":"2024-03-04T06:08:04.917850+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6655473,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5283,"num_write":12,"num_write_kb":2532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.875772+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:08:03.875772+0000","last_peered":"2024-03-04T06:08:03.875772+0000","last_clean":"2024-03-04T06:08:03.875772+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:08:03.875772+0000","last_undegraded":"2024-03-04T06:08:03.875772+0000","last_fullsized":"2024-03-04T06:08:03.875772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"18'9","reported_seq":25,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.331862+0000","last_change":"2024-03-04T06:08:01.897227+0000","last_active":"2024-03-04T06:08:04.331862+0000","last_peered":"2024-03-04T06:08:04.331862+0000","last_clean":"2024-03-04T06:08:04.331862+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:08:04.331862+0000","last_undegraded":"2024-03-04T06:08:04.331862+0000","last_fullsized":"2024-03-04T06:08:04.331862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6957999,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":2888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.841301+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:08:05.841301+0000","last_peered":"2024-03-04T06:08:05.841301+0000","last_clean":"2024-03-04T06:08:05.841301+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:08:05.841301+0000","last_undegraded":"2024-03-04T06:08:05.841301+0000","last_fullsized":"2024-03-04T06:08:05.841301+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"18'16","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.832804+0000","last_change":"2024-03-04T06:08:01.896375+0000","last_active":"2024-03-04T06:08:05.832804+0000","last_peered":"2024-03-04T06:08:05.832804+0000","last_clean":"2024-03-04T06:08:05.832804+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:08:05.832804+0000","last_undegraded":"2024-03-04T06:08:05.832804+0000","last_fullsized":"2024-03-04T06:08:05.832804+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":21,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.865162+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:08:03.865162+0000","last_peered":"2024-03-04T06:08:03.865162+0000","last_clean":"2024-03-04T06:08:03.865162+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:08:03.865162+0000","last_undegraded":"2024-03-04T06:08:03.865162+0000","last_fullsized":"2024-03-04T06:08:03.865162+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"18'12","reported_seq":29,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.603753+0000","last_change":"2024-03-04T06:08:01.873075+0000","last_active":"2024-03-04T06:08:04.603753+0000","last_peered":"2024-03-04T06:08:04.603753+0000","last_clean":"2024-03-04T06:08:04.603753+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:08:04.603753+0000","last_undegraded":"2024-03-04T06:08:04.603753+0000","last_fullsized":"2024-03-04T06:08:04.603753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9848770,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.822537+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:08:03.822537+0000","last_peered":"2024-03-04T06:08:03.822537+0000","last_clean":"2024-03-04T06:08:03.822537+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:08:03.822537+0000","last_undegraded":"2024-03-04T06:08:03.822537+0000","last_fullsized":"2024-03-04T06:08:03.822537+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":115428958,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":25678,"num_write":232,"num_write_kb":61987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":174,"ondisk_log_size":174,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705685,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78500,"kb_used_data":51676,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94293340,"statfs":{"total":96636764160,"available":96556380160,"internally_reserved":0,"allocated":52916224,"data_stored":53506007,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705685,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67796,"kb_used_data":40972,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94304044,"statfs":{"total":96636764160,"available":96567341056,"internally_reserved":0,"allocated":41955328,"data_stored":42099627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":53024,"kb_used_data":26200,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94318816,"statfs":{"total":96636764160,"available":96582467584,"internally_reserved":0,"allocated":26828800,"data_stored":27043975,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":47888,"kb_used_data":21064,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94323952,"statfs":{"total":96636764160,"available":96587726848,"internally_reserved":0,"allocated":21569536,"data_stored":21817541,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:08:08.797 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:08:08.797 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:08:08.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:08:08.813 INFO:tasks.rados.rados.0.smithi064.stdout:84: done (3 left) 2024-03-04T06:08:08.813 INFO:tasks.rados.rados.0.smithi064.stdout:85: done (2 left) 2024-03-04T06:08:08.813 INFO:tasks.rados.rados.0.smithi064.stdout:86: done (1 left) 2024-03-04T06:08:08.814 INFO:tasks.rados.rados.0.smithi064.stdout:87: done (0 left) 2024-03-04T06:08:08.814 INFO:tasks.rados.rados.0.smithi064.stdout:88: write oid 6 current snap is 3 2024-03-04T06:08:08.814 INFO:tasks.rados.rados.0.smithi064.stdout:88: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-03-04T06:08:08.821 INFO:tasks.rados.rados.0.smithi064.stdout:88: writing smithi06431743-6 from 709262 to 1406520 tid 1 2024-03-04T06:08:08.827 INFO:tasks.rados.rados.0.smithi064.stdout:88: writing smithi06431743-6 from 2012635 to 2584653 tid 2 2024-03-04T06:08:08.831 INFO:tasks.rados.rados.0.smithi064.stdout:88: writing smithi06431743-6 from 3046388 to 3109262 tid 3 2024-03-04T06:08:08.831 INFO:tasks.rados.rados.0.smithi064.stdout:89: write oid 7 current snap is 3 2024-03-04T06:08:08.831 INFO:tasks.rados.rados.0.smithi064.stdout:89: seq_num 67 ranges {726069=772507,2071117=715676} 2024-03-04T06:08:08.839 INFO:tasks.rados.rados.0.smithi064.stdout:89: writing smithi06431743-7 from 726069 to 1498576 tid 1 2024-03-04T06:08:08.846 INFO:tasks.rados.rados.0.smithi064.stdout:89: writing smithi06431743-7 from 2071117 to 2786793 tid 2 2024-03-04T06:08:08.846 INFO:tasks.rados.rados.0.smithi064.stdout:90: write oid 22 current snap is 3 2024-03-04T06:08:08.846 INFO:tasks.rados.rados.0.smithi064.stdout:90: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-03-04T06:08:08.854 INFO:tasks.rados.rados.0.smithi064.stdout:90: writing smithi06431743-22 from 742876 to 1190632 tid 1 2024-03-04T06:08:08.860 INFO:tasks.rados.rados.0.smithi064.stdout:90: writing smithi06431743-22 from 1729599 to 2305286 tid 2 2024-03-04T06:08:08.865 INFO:tasks.rados.rados.0.smithi064.stdout:90: writing smithi06431743-22 from 3101234 to 3142876 tid 3 2024-03-04T06:08:08.865 INFO:tasks.rados.rados.0.smithi064.stdout:91: delete oid 14 current snap is 3 2024-03-04T06:08:08.890 INFO:tasks.rados.rados.0.smithi064.stdout:89: finishing write tid 1 to smithi06431743-7 2024-03-04T06:08:08.896 INFO:tasks.rados.rados.0.smithi064.stdout:91: done (3 left) 2024-03-04T06:08:08.896 INFO:tasks.rados.rados.0.smithi064.stdout:92: write (excl) oid 45 current snap is 3 2024-03-04T06:08:08.897 INFO:tasks.rados.rados.0.smithi064.stdout:92: seq_num 70 ranges {776490=714607,2246563=579356} 2024-03-04T06:08:08.906 INFO:tasks.rados.rados.0.smithi064.stdout:92: writing smithi06431743-45 from 776490 to 1491097 tid 1 2024-03-04T06:08:08.914 INFO:tasks.rados.rados.0.smithi064.stdout:92: writing smithi06431743-45 from 2246563 to 2825919 tid 2 2024-03-04T06:08:08.914 INFO:tasks.rados.rados.0.smithi064.stdout:93: write oid 27 current snap is 3 2024-03-04T06:08:08.914 INFO:tasks.rados.rados.0.smithi064.stdout:88: finishing write tid 1 to smithi06431743-6 2024-03-04T06:08:08.914 INFO:tasks.rados.rados.0.smithi064.stdout:93: seq_num 71 ranges {793297=789856,2305045=723014} 2024-03-04T06:08:08.923 INFO:tasks.rados.rados.0.smithi064.stdout:93: writing smithi06431743-27 from 793297 to 1583153 tid 1 2024-03-04T06:08:08.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:08.921+0000 7fdc7ead4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:08.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:08.928+0000 7fdc7ead4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:08.933 INFO:tasks.rados.rados.0.smithi064.stdout:93: writing smithi06431743-27 from 2305045 to 3028059 tid 2 2024-03-04T06:08:08.933 INFO:tasks.rados.rados.0.smithi064.stdout:94: read oid 14 snap -1 2024-03-04T06:08:08.933 INFO:tasks.rados.rados.0.smithi064.stdout:94: expect deleted 2024-03-04T06:08:08.933 INFO:tasks.rados.rados.0.smithi064.stdout:95: read oid 4 snap -1 2024-03-04T06:08:08.933 INFO:tasks.rados.rados.0.smithi064.stdout:95: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-04T06:08:08.933 INFO:tasks.rados.rados.0.smithi064.stdout:96: delete oid 49 current snap is 3 2024-03-04T06:08:08.933 INFO:tasks.rados.rados.0.smithi064.stdout:88: finishing write tid 2 to smithi06431743-6 2024-03-04T06:08:08.933 INFO:tasks.rados.rados.0.smithi064.stdout:90: finishing write tid 1 to smithi06431743-22 2024-03-04T06:08:08.935 INFO:tasks.rados.rados.0.smithi064.stdout:89: finishing write tid 2 to smithi06431743-7 2024-03-04T06:08:08.940 INFO:tasks.rados.rados.0.smithi064.stdout:88: finishing write tid 3 to smithi06431743-6 2024-03-04T06:08:08.957 INFO:tasks.rados.rados.0.smithi064.stdout:96: done (7 left) 2024-03-04T06:08:08.957 INFO:tasks.rados.rados.0.smithi064.stdout:97: snap_remove snap 0 2024-03-04T06:08:09.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:09.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:08:09.124 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:08:09.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:09.127+0000 7f98cd75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:09.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:09.132+0000 7f98cd75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:09.137 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-04T06:08:08.812186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":116019326,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":245,"num_read_kb":25769,"num_write":369,"num_write_kb":63881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":247208,"kb_used_data":139912,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377240152,"statfs":{"total":386547056640,"available":386293915648,"internally_reserved":0,"allocated":143269888,"data_stored":144467150,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":115428958,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":25678,"num_write":232,"num_write_kb":61987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.105148"},"pg_stats":[{"pgid":"3.f","version":"18'9","reported_seq":25,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.627494+0000","last_change":"2024-03-04T06:08:01.882593+0000","last_active":"2024-03-04T06:08:04.627494+0000","last_peered":"2024-03-04T06:08:04.627494+0000","last_clean":"2024-03-04T06:08:04.627494+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:08:04.627494+0000","last_undegraded":"2024-03-04T06:08:04.627494+0000","last_fullsized":"2024-03-04T06:08:04.627494+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"18'13","reported_seq":31,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.826623+0000","last_change":"2024-03-04T06:08:01.884604+0000","last_active":"2024-03-04T06:08:04.826623+0000","last_peered":"2024-03-04T06:08:04.826623+0000","last_clean":"2024-03-04T06:08:04.826623+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:08:04.826623+0000","last_undegraded":"2024-03-04T06:08:04.826623+0000","last_fullsized":"2024-03-04T06:08:04.826623+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":18,"num_write_kb":4968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"18'12","reported_seq":31,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.547355+0000","last_change":"2024-03-04T06:08:01.888867+0000","last_active":"2024-03-04T06:08:04.547355+0000","last_peered":"2024-03-04T06:08:04.547355+0000","last_clean":"2024-03-04T06:08:04.547355+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:08:04.547355+0000","last_undegraded":"2024-03-04T06:08:04.547355+0000","last_fullsized":"2024-03-04T06:08:04.547355+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9949612,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"18'13","reported_seq":30,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.591487+0000","last_change":"2024-03-04T06:08:01.871856+0000","last_active":"2024-03-04T06:08:04.591487+0000","last_peered":"2024-03-04T06:08:04.591487+0000","last_clean":"2024-03-04T06:08:04.591487+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:08:04.591487+0000","last_undegraded":"2024-03-04T06:08:04.591487+0000","last_fullsized":"2024-03-04T06:08:04.591487+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7798349,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":17,"num_write_kb":4679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"18'3","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.844549+0000","last_change":"2024-03-04T06:08:01.880739+0000","last_active":"2024-03-04T06:08:05.844549+0000","last_peered":"2024-03-04T06:08:05.844549+0000","last_clean":"2024-03-04T06:08:05.844549+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:08:05.844549+0000","last_undegraded":"2024-03-04T06:08:05.844549+0000","last_fullsized":"2024-03-04T06:08:05.844549+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504210,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"18'9","reported_seq":28,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.587389+0000","last_change":"2024-03-04T06:08:01.891545+0000","last_active":"2024-03-04T06:08:04.587389+0000","last_peered":"2024-03-04T06:08:04.587389+0000","last_clean":"2024-03-04T06:08:04.587389+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:08:04.587389+0000","last_undegraded":"2024-03-04T06:08:04.587389+0000","last_fullsized":"2024-03-04T06:08:04.587389+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7495823,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5232,"num_write":12,"num_write_kb":3579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.852740+0000","last_change":"2024-03-04T06:08:01.888970+0000","last_active":"2024-03-04T06:08:03.852740+0000","last_peered":"2024-03-04T06:08:03.852740+0000","last_clean":"2024-03-04T06:08:03.852740+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:08:03.852740+0000","last_undegraded":"2024-03-04T06:08:03.852740+0000","last_fullsized":"2024-03-04T06:08:03.852740+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"18'6","reported_seq":23,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.598228+0000","last_change":"2024-03-04T06:08:01.876040+0000","last_active":"2024-03-04T06:08:04.598228+0000","last_peered":"2024-03-04T06:08:04.598228+0000","last_clean":"2024-03-04T06:08:04.598228+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:08:04.598228+0000","last_undegraded":"2024-03-04T06:08:04.598228+0000","last_fullsized":"2024-03-04T06:08:04.598228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":158,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.836113+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:08:05.836113+0000","last_peered":"2024-03-04T06:08:05.836113+0000","last_clean":"2024-03-04T06:08:05.836113+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:08:05.836113+0000","last_undegraded":"2024-03-04T06:08:05.836113+0000","last_fullsized":"2024-03-04T06:08:05.836113+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.818988+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:08:03.818988+0000","last_peered":"2024-03-04T06:08:03.818988+0000","last_clean":"2024-03-04T06:08:03.818988+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:08:03.818988+0000","last_undegraded":"2024-03-04T06:08:03.818988+0000","last_fullsized":"2024-03-04T06:08:03.818988+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"18'13","reported_seq":32,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.792825+0000","last_change":"2024-03-04T06:08:01.884009+0000","last_active":"2024-03-04T06:08:04.792825+0000","last_peered":"2024-03-04T06:08:04.792825+0000","last_clean":"2024-03-04T06:08:04.792825+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:08:04.792825+0000","last_undegraded":"2024-03-04T06:08:04.792825+0000","last_fullsized":"2024-03-04T06:08:04.792825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7949612,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":17,"num_write_kb":4330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.855484+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:08:03.855484+0000","last_peered":"2024-03-04T06:08:03.855484+0000","last_clean":"2024-03-04T06:08:03.855484+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:08:03.855484+0000","last_undegraded":"2024-03-04T06:08:03.855484+0000","last_fullsized":"2024-03-04T06:08:03.855484+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"18'6","reported_seq":27,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.875465+0000","last_change":"2024-03-04T06:08:01.878597+0000","last_active":"2024-03-04T06:08:05.875465+0000","last_peered":"2024-03-04T06:08:05.875465+0000","last_clean":"2024-03-04T06:08:05.875465+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:08:05.875465+0000","last_undegraded":"2024-03-04T06:08:05.875465+0000","last_fullsized":"2024-03-04T06:08:05.875465+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4890771,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5413,"num_write":8,"num_write_kb":2158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.810817+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:08:03.810817+0000","last_peered":"2024-03-04T06:08:03.810817+0000","last_clean":"2024-03-04T06:08:03.810817+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:08:03.810817+0000","last_undegraded":"2024-03-04T06:08:03.810817+0000","last_fullsized":"2024-03-04T06:08:03.810817+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"18'22","reported_seq":51,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.864521+0000","last_change":"2024-03-04T06:08:01.890481+0000","last_active":"2024-03-04T06:08:05.864521+0000","last_peered":"2024-03-04T06:08:05.864521+0000","last_clean":"2024-03-04T06:08:05.864521+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:08:05.864521+0000","last_undegraded":"2024-03-04T06:08:05.864521+0000","last_fullsized":"2024-03-04T06:08:05.864521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13075681,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":9709,"num_write":29,"num_write_kb":8088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.830698+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:08:03.830698+0000","last_peered":"2024-03-04T06:08:03.830698+0000","last_clean":"2024-03-04T06:08:03.830698+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:08:03.830698+0000","last_undegraded":"2024-03-04T06:08:03.830698+0000","last_fullsized":"2024-03-04T06:08:03.830698+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"18'22","reported_seq":43,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.807481+0000","last_change":"2024-03-04T06:08:01.906823+0000","last_active":"2024-03-04T06:08:04.807481+0000","last_peered":"2024-03-04T06:08:04.807481+0000","last_clean":"2024-03-04T06:08:04.807481+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:08:04.807481+0000","last_undegraded":"2024-03-04T06:08:04.807481+0000","last_fullsized":"2024-03-04T06:08:04.807481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12134489,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":30,"num_write_kb":7626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"18'9","reported_seq":30,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.917850+0000","last_change":"2024-03-04T06:08:01.908948+0000","last_active":"2024-03-04T06:08:04.917850+0000","last_peered":"2024-03-04T06:08:04.917850+0000","last_clean":"2024-03-04T06:08:04.917850+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:08:04.917850+0000","last_undegraded":"2024-03-04T06:08:04.917850+0000","last_fullsized":"2024-03-04T06:08:04.917850+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6655473,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":5283,"num_write":12,"num_write_kb":2532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.875772+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:08:03.875772+0000","last_peered":"2024-03-04T06:08:03.875772+0000","last_clean":"2024-03-04T06:08:03.875772+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:08:03.875772+0000","last_undegraded":"2024-03-04T06:08:03.875772+0000","last_fullsized":"2024-03-04T06:08:03.875772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"18'9","reported_seq":25,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.331862+0000","last_change":"2024-03-04T06:08:01.897227+0000","last_active":"2024-03-04T06:08:04.331862+0000","last_peered":"2024-03-04T06:08:04.331862+0000","last_clean":"2024-03-04T06:08:04.331862+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:08:04.331862+0000","last_undegraded":"2024-03-04T06:08:04.331862+0000","last_fullsized":"2024-03-04T06:08:04.331862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6957999,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":2888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.841301+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:08:05.841301+0000","last_peered":"2024-03-04T06:08:05.841301+0000","last_clean":"2024-03-04T06:08:05.841301+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:08:05.841301+0000","last_undegraded":"2024-03-04T06:08:05.841301+0000","last_fullsized":"2024-03-04T06:08:05.841301+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"18'16","reported_seq":37,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-03-04T06:08:05.832804+0000","last_change":"2024-03-04T06:08:01.896375+0000","last_active":"2024-03-04T06:08:05.832804+0000","last_peered":"2024-03-04T06:08:05.832804+0000","last_clean":"2024-03-04T06:08:05.832804+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:08:05.832804+0000","last_undegraded":"2024-03-04T06:08:05.832804+0000","last_fullsized":"2024-03-04T06:08:05.832804+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":21,"num_write_kb":4752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.865162+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:08:03.865162+0000","last_peered":"2024-03-04T06:08:03.865162+0000","last_clean":"2024-03-04T06:08:03.865162+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:08:03.865162+0000","last_undegraded":"2024-03-04T06:08:03.865162+0000","last_fullsized":"2024-03-04T06:08:03.865162+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"18'12","reported_seq":29,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:04.603753+0000","last_change":"2024-03-04T06:08:01.873075+0000","last_active":"2024-03-04T06:08:04.603753+0000","last_peered":"2024-03-04T06:08:04.603753+0000","last_clean":"2024-03-04T06:08:04.603753+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:08:04.603753+0000","last_undegraded":"2024-03-04T06:08:04.603753+0000","last_fullsized":"2024-03-04T06:08:04.603753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9848770,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-04T06:08:03.822537+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:08:03.822537+0000","last_peered":"2024-03-04T06:08:03.822537+0000","last_clean":"2024-03-04T06:08:03.822537+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:08:03.822537+0000","last_undegraded":"2024-03-04T06:08:03.822537+0000","last_fullsized":"2024-03-04T06:08:03.822537+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":115428958,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":25678,"num_write":232,"num_write_kb":61987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":174,"ondisk_log_size":174,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705685,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78500,"kb_used_data":51676,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94293340,"statfs":{"total":96636764160,"available":96556380160,"internally_reserved":0,"allocated":52916224,"data_stored":53506007,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705685,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67796,"kb_used_data":40972,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94304044,"statfs":{"total":96636764160,"available":96567341056,"internally_reserved":0,"allocated":41955328,"data_stored":42099627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":53024,"kb_used_data":26200,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94318816,"statfs":{"total":96636764160,"available":96582467584,"internally_reserved":0,"allocated":26828800,"data_stored":27043975,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":47888,"kb_used_data":21064,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94323952,"statfs":{"total":96636764160,"available":96587726848,"internally_reserved":0,"allocated":21569536,"data_stored":21817541,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:08:09.816 INFO:tasks.rados.rados.0.smithi064.stdout:97: done (7 left) 2024-03-04T06:08:09.816 INFO:tasks.rados.rados.0.smithi064.stdout:98: write (excl) oid 34 current snap is 3 2024-03-04T06:08:09.816 INFO:tasks.rados.rados.0.smithi064.stdout:95: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-04T06:08:09.901 INFO:tasks.rados.rados.0.smithi064.stdout:93: finishing write tid 1 to smithi06431743-27 2024-03-04T06:08:09.901 INFO:tasks.rados.rados.0.smithi064.stdout:90: finishing write tid 2 to smithi06431743-22 2024-03-04T06:08:09.901 INFO:tasks.rados.rados.0.smithi064.stdout:88: finishing write tid 4 to smithi06431743-6 2024-03-04T06:08:09.902 INFO:tasks.rados.rados.0.smithi064.stdout:88: finishing write tid 5 to smithi06431743-6 2024-03-04T06:08:09.902 INFO:tasks.rados.rados.0.smithi064.stdout:88: oid 6 updating version 0 to 13 2024-03-04T06:08:09.902 INFO:tasks.rados.rados.0.smithi064.stdout:88: oid 6 version 13 is already newer than 11 2024-03-04T06:08:09.902 INFO:tasks.rados.rados.0.smithi064.stdout:88: oid 6 version 13 is already newer than 13 2024-03-04T06:08:09.902 INFO:tasks.rados.rados.0.smithi064.stdout:88: oid 6 updating version 13 to 14 2024-03-04T06:08:09.902 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 14 (ObjNum 66 snap 3 seq_num 66) dirty exists 2024-03-04T06:08:09.902 INFO:tasks.rados.rados.0.smithi064.stdout:88: left oid 6 (ObjNum 66 snap 3 seq_num 66) 2024-03-04T06:08:09.902 INFO:tasks.rados.rados.0.smithi064.stdout:98: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-03-04T06:08:09.907 INFO:tasks.rados.rados.0.smithi064.stdout:98: writing smithi06431743-34 from 426911 to 967265 tid 1 2024-03-04T06:08:09.915 INFO:tasks.rados.rados.0.smithi064.stdout:98: writing smithi06431743-34 from 1622009 to 2348692 tid 2 2024-03-04T06:08:09.920 INFO:tasks.rados.rados.0.smithi064.stdout:98: writing smithi06431743-34 from 3121996 to 3226911 tid 3 2024-03-04T06:08:09.920 INFO:tasks.rados.rados.0.smithi064.stdout:88: done (7 left) 2024-03-04T06:08:09.921 INFO:tasks.rados.rados.0.smithi064.stdout:94: done (6 left) 2024-03-04T06:08:09.921 INFO:tasks.rados.rados.0.smithi064.stdout:95: done (5 left) 2024-03-04T06:08:09.921 INFO:tasks.rados.rados.0.smithi064.stdout:99: snap_remove snap 2 2024-03-04T06:08:10.820 INFO:tasks.rados.rados.0.smithi064.stdout:99: done (5 left) 2024-03-04T06:08:10.820 INFO:tasks.rados.rados.0.smithi064.stdout:100: write oid 49 current snap is 3 2024-03-04T06:08:10.820 INFO:tasks.rados.rados.0.smithi064.stdout:89: finishing write tid 3 to smithi06431743-7 2024-03-04T06:08:10.820 INFO:tasks.rados.rados.0.smithi064.stdout:100: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-03-04T06:08:10.825 INFO:tasks.rados.rados.0.smithi064.stdout:100: writing smithi06431743-49 from 443718 to 1059321 tid 1 2024-03-04T06:08:10.831 INFO:tasks.rados.rados.0.smithi064.stdout:100: writing smithi06431743-49 from 1564138 to 2034479 tid 2 2024-03-04T06:08:10.838 INFO:tasks.rados.rados.0.smithi064.stdout:100: writing smithi06431743-49 from 2633066 to 3243718 tid 3 2024-03-04T06:08:10.839 INFO:tasks.rados.rados.0.smithi064.stdout:101: read oid 8 snap -1 2024-03-04T06:08:10.839 INFO:tasks.rados.rados.0.smithi064.stdout:101: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-04T06:08:10.839 INFO:tasks.rados.rados.0.smithi064.stdout:102: read oid 31 snap -1 2024-03-04T06:08:10.839 INFO:tasks.rados.rados.0.smithi064.stdout:102: expect (ObjNum 30 snap 0 seq_num 30) 2024-03-04T06:08:10.839 INFO:tasks.rados.rados.0.smithi064.stdout:103: read oid 3 snap -1 2024-03-04T06:08:10.839 INFO:tasks.rados.rados.0.smithi064.stdout:103: expect (ObjNum 54 snap 0 seq_num 54) 2024-03-04T06:08:10.839 INFO:tasks.rados.rados.0.smithi064.stdout:89: finishing write tid 4 to smithi06431743-7 2024-03-04T06:08:10.839 INFO:tasks.rados.rados.0.smithi064.stdout:89: oid 7 updating version 0 to 10 2024-03-04T06:08:10.839 INFO:tasks.rados.rados.0.smithi064.stdout:89: oid 7 version 10 is already newer than 9 2024-03-04T06:08:10.839 INFO:tasks.rados.rados.0.smithi064.stdout:89: oid 7 version 10 is already newer than 8 2024-03-04T06:08:10.840 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 10 (ObjNum 67 snap 3 seq_num 67) dirty exists 2024-03-04T06:08:10.840 INFO:tasks.rados.rados.0.smithi064.stdout:89: left oid 7 (ObjNum 67 snap 3 seq_num 67) 2024-03-04T06:08:10.840 INFO:tasks.rados.rados.0.smithi064.stdout:104: read oid 2 snap -1 2024-03-04T06:08:10.840 INFO:tasks.rados.rados.0.smithi064.stdout:104: expect (ObjNum 65 snap 2 seq_num 65) 2024-03-04T06:08:10.843 INFO:tasks.rados.rados.0.smithi064.stdout:89: done (9 left) 2024-03-04T06:08:10.843 INFO:tasks.rados.rados.0.smithi064.stdout:105: read oid 33 snap -1 2024-03-04T06:08:10.843 INFO:tasks.rados.rados.0.smithi064.stdout:105: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-04T06:08:10.843 INFO:tasks.rados.rados.0.smithi064.stdout:106: read oid 37 snap -1 2024-03-04T06:08:10.843 INFO:tasks.rados.rados.0.smithi064.stdout:106: expect (ObjNum 56 snap 0 seq_num 56) 2024-03-04T06:08:10.843 INFO:tasks.rados.rados.0.smithi064.stdout:107: delete oid 15 current snap is 3 2024-03-04T06:08:10.843 INFO:tasks.rados.rados.0.smithi064.stdout:92: finishing write tid 1 to smithi06431743-45 2024-03-04T06:08:10.843 INFO:tasks.rados.rados.0.smithi064.stdout:90: finishing write tid 3 to smithi06431743-22 2024-03-04T06:08:10.843 INFO:tasks.rados.rados.0.smithi064.stdout:93: finishing write tid 2 to smithi06431743-27 2024-03-04T06:08:10.843 INFO:tasks.rados.rados.0.smithi064.stdout:92: finishing write tid 2 to smithi06431743-45 2024-03-04T06:08:10.844 INFO:tasks.rados.rados.0.smithi064.stdout:90: finishing write tid 4 to smithi06431743-22 2024-03-04T06:08:10.844 INFO:tasks.rados.rados.0.smithi064.stdout:90: finishing write tid 5 to smithi06431743-22 2024-03-04T06:08:10.844 INFO:tasks.rados.rados.0.smithi064.stdout:90: oid 22 updating version 0 to 15 2024-03-04T06:08:10.844 INFO:tasks.rados.rados.0.smithi064.stdout:90: oid 22 updating version 15 to 17 2024-03-04T06:08:10.844 INFO:tasks.rados.rados.0.smithi064.stdout:90: oid 22 version 17 is already newer than 16 2024-03-04T06:08:10.844 INFO:tasks.rados.rados.0.smithi064.stdout:90: oid 22 version 17 is already newer than 14 2024-03-04T06:08:10.844 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 17 (ObjNum 68 snap 3 seq_num 68) dirty exists 2024-03-04T06:08:10.844 INFO:tasks.rados.rados.0.smithi064.stdout:90: left oid 22 (ObjNum 68 snap 3 seq_num 68) 2024-03-04T06:08:10.844 INFO:tasks.rados.rados.0.smithi064.stdout:93: finishing write tid 3 to smithi06431743-27 2024-03-04T06:08:10.844 INFO:tasks.rados.rados.0.smithi064.stdout:92: finishing write tid 3 to smithi06431743-45 2024-03-04T06:08:10.844 INFO:tasks.rados.rados.0.smithi064.stdout:93: finishing write tid 4 to smithi06431743-27 2024-03-04T06:08:10.845 INFO:tasks.rados.rados.0.smithi064.stdout:93: oid 27 updating version 0 to 15 2024-03-04T06:08:10.845 INFO:tasks.rados.rados.0.smithi064.stdout:93: oid 27 updating version 15 to 17 2024-03-04T06:08:10.845 INFO:tasks.rados.rados.0.smithi064.stdout:93: oid 27 version 17 is already newer than 16 2024-03-04T06:08:10.845 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 17 (ObjNum 71 snap 3 seq_num 71) dirty exists 2024-03-04T06:08:10.845 INFO:tasks.rados.rados.0.smithi064.stdout:93: left oid 27 (ObjNum 71 snap 3 seq_num 71) 2024-03-04T06:08:10.845 INFO:tasks.rados.rados.0.smithi064.stdout:92: finishing write tid 4 to smithi06431743-45 2024-03-04T06:08:10.845 INFO:tasks.rados.rados.0.smithi064.stdout:92: oid 45 updating version 0 to 22 2024-03-04T06:08:10.845 INFO:tasks.rados.rados.0.smithi064.stdout:92: oid 45 version 22 is already newer than 21 2024-03-04T06:08:10.845 INFO:tasks.rados.rados.0.smithi064.stdout:92: oid 45 version 22 is already newer than 20 2024-03-04T06:08:10.845 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 22 (ObjNum 70 snap 3 seq_num 70) dirty exists 2024-03-04T06:08:10.845 INFO:tasks.rados.rados.0.smithi064.stdout:92: left oid 45 (ObjNum 70 snap 3 seq_num 70) 2024-03-04T06:08:10.846 INFO:tasks.rados.rados.0.smithi064.stdout:98: finishing write tid 1 to smithi06431743-34 2024-03-04T06:08:10.846 INFO:tasks.rados.rados.0.smithi064.stdout:98: finishing write tid 2 to smithi06431743-34 2024-03-04T06:08:10.846 INFO:tasks.rados.rados.0.smithi064.stdout:98: finishing write tid 3 to smithi06431743-34 2024-03-04T06:08:10.846 INFO:tasks.rados.rados.0.smithi064.stdout:98: finishing write tid 4 to smithi06431743-34 2024-03-04T06:08:10.846 INFO:tasks.rados.rados.0.smithi064.stdout:98: finishing write tid 5 to smithi06431743-34 2024-03-04T06:08:10.846 INFO:tasks.rados.rados.0.smithi064.stdout:98: oid 34 updating version 0 to 35 2024-03-04T06:08:10.846 INFO:tasks.rados.rados.0.smithi064.stdout:98: oid 34 version 35 is already newer than 32 2024-03-04T06:08:10.846 INFO:tasks.rados.rados.0.smithi064.stdout:98: oid 34 version 35 is already newer than 33 2024-03-04T06:08:10.846 INFO:tasks.rados.rados.0.smithi064.stdout:98: oid 34 version 35 is already newer than 34 2024-03-04T06:08:10.846 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 35 (ObjNum 73 snap 3 seq_num 73) dirty exists 2024-03-04T06:08:10.846 INFO:tasks.rados.rados.0.smithi064.stdout:98: left oid 34 (ObjNum 73 snap 3 seq_num 73) 2024-03-04T06:08:10.876 INFO:tasks.rados.rados.0.smithi064.stdout:101: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-04T06:08:10.967 INFO:tasks.rados.rados.0.smithi064.stdout:104: expect (ObjNum 65 snap 2 seq_num 65) 2024-03-04T06:08:11.152 INFO:tasks.rados.rados.0.smithi064.stdout:106: expect (ObjNum 56 snap 0 seq_num 56) 2024-03-04T06:08:11.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:11.301 INFO:tasks.rados.rados.0.smithi064.stdout:105: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-04T06:08:11.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:11.301+0000 7fac55b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:11.306 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:11.306+0000 7fac55b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:11.398 INFO:tasks.rados.rados.0.smithi064.stdout:103: expect (ObjNum 54 snap 0 seq_num 54) 2024-03-04T06:08:11.541 INFO:tasks.rados.rados.0.smithi064.stdout:100: finishing write tid 1 to smithi06431743-49 2024-03-04T06:08:11.541 INFO:tasks.rados.rados.0.smithi064.stdout:102: expect (ObjNum 30 snap 0 seq_num 30) 2024-03-04T06:08:11.634 INFO:tasks.rados.rados.0.smithi064.stdout:100: finishing write tid 2 to smithi06431743-49 2024-03-04T06:08:11.634 INFO:tasks.rados.rados.0.smithi064.stdout:100: finishing write tid 3 to smithi06431743-49 2024-03-04T06:08:11.635 INFO:tasks.rados.rados.0.smithi064.stdout:100: finishing write tid 4 to smithi06431743-49 2024-03-04T06:08:11.635 INFO:tasks.rados.rados.0.smithi064.stdout:100: finishing write tid 5 to smithi06431743-49 2024-03-04T06:08:11.635 INFO:tasks.rados.rados.0.smithi064.stdout:100: oid 49 updating version 0 to 12 2024-03-04T06:08:11.635 INFO:tasks.rados.rados.0.smithi064.stdout:100: oid 49 version 12 is already newer than 12 2024-03-04T06:08:11.635 INFO:tasks.rados.rados.0.smithi064.stdout:100: oid 49 updating version 12 to 14 2024-03-04T06:08:11.635 INFO:tasks.rados.rados.0.smithi064.stdout:100: oid 49 version 14 is already newer than 14 2024-03-04T06:08:11.635 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 14 (ObjNum 74 snap 3 seq_num 74) dirty exists 2024-03-04T06:08:11.635 INFO:tasks.rados.rados.0.smithi064.stdout:100: left oid 49 (ObjNum 74 snap 3 seq_num 74) 2024-03-04T06:08:11.635 INFO:tasks.rados.rados.0.smithi064.stdout:90: done (11 left) 2024-03-04T06:08:11.635 INFO:tasks.rados.rados.0.smithi064.stdout:92: done (10 left) 2024-03-04T06:08:11.635 INFO:tasks.rados.rados.0.smithi064.stdout:93: done (9 left) 2024-03-04T06:08:11.636 INFO:tasks.rados.rados.0.smithi064.stdout:98: done (8 left) 2024-03-04T06:08:11.636 INFO:tasks.rados.rados.0.smithi064.stdout:100: done (7 left) 2024-03-04T06:08:11.636 INFO:tasks.rados.rados.0.smithi064.stdout:101: done (6 left) 2024-03-04T06:08:11.636 INFO:tasks.rados.rados.0.smithi064.stdout:102: done (5 left) 2024-03-04T06:08:11.636 INFO:tasks.rados.rados.0.smithi064.stdout:103: done (4 left) 2024-03-04T06:08:11.636 INFO:tasks.rados.rados.0.smithi064.stdout:104: done (3 left) 2024-03-04T06:08:11.636 INFO:tasks.rados.rados.0.smithi064.stdout:105: done (2 left) 2024-03-04T06:08:11.636 INFO:tasks.rados.rados.0.smithi064.stdout:106: done (1 left) 2024-03-04T06:08:11.636 INFO:tasks.rados.rados.0.smithi064.stdout:107: done (0 left) 2024-03-04T06:08:11.636 INFO:tasks.rados.rados.0.smithi064.stdout:108: snap_create 2024-03-04T06:08:11.822 INFO:tasks.rados.rados.0.smithi064.stdout:108: done (0 left) 2024-03-04T06:08:11.822 INFO:tasks.rados.rados.0.smithi064.stdout:109: delete oid 42 current snap is 4 2024-03-04T06:08:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:109: done (0 left) 2024-03-04T06:08:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:110: read oid 48 snap 3 2024-03-04T06:08:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:110: expect (ObjNum 47 snap 0 seq_num 47) 2024-03-04T06:08:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:111: snap_create 2024-03-04T06:08:11.907 INFO:tasks.rados.rados.0.smithi064.stdout:110: expect (ObjNum 47 snap 0 seq_num 47) 2024-03-04T06:08:12.826 INFO:tasks.rados.rados.0.smithi064.stdout:110: done (1 left) 2024-03-04T06:08:12.826 INFO:tasks.rados.rados.0.smithi064.stdout:111: done (0 left) 2024-03-04T06:08:12.827 INFO:tasks.rados.rados.0.smithi064.stdout:112: snap_create 2024-03-04T06:08:13.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:13.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:13.493+0000 7fd9a28ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:13.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:13.498+0000 7fd9a28ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:13.828 INFO:tasks.rados.rados.0.smithi064.stdout:112: done (0 left) 2024-03-04T06:08:13.828 INFO:tasks.rados.rados.0.smithi064.stdout:113: read oid 35 snap -1 2024-03-04T06:08:13.828 INFO:tasks.rados.rados.0.smithi064.stdout:113: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-04T06:08:13.828 INFO:tasks.rados.rados.0.smithi064.stdout:114: read oid 9 snap -1 2024-03-04T06:08:13.828 INFO:tasks.rados.rados.0.smithi064.stdout:114: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-04T06:08:13.828 INFO:tasks.rados.rados.0.smithi064.stdout:115: delete oid 26 current snap is 6 2024-03-04T06:08:13.883 INFO:tasks.rados.rados.0.smithi064.stdout:114: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-04T06:08:13.964 INFO:tasks.rados.rados.0.smithi064.stdout:113: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-04T06:08:14.060 INFO:tasks.rados.rados.0.smithi064.stdout:113: done (2 left) 2024-03-04T06:08:14.060 INFO:tasks.rados.rados.0.smithi064.stdout:114: done (1 left) 2024-03-04T06:08:14.060 INFO:tasks.rados.rados.0.smithi064.stdout:115: done (0 left) 2024-03-04T06:08:14.060 INFO:tasks.rados.rados.0.smithi064.stdout:116: write oid 28 current snap is 6 2024-03-04T06:08:14.060 INFO:tasks.rados.rados.0.smithi064.stdout:116: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-03-04T06:08:14.066 INFO:tasks.rados.rados.0.smithi064.stdout:116: writing smithi06431743-28 from 510946 to 1143898 tid 1 2024-03-04T06:08:14.072 INFO:tasks.rados.rados.0.smithi064.stdout:116: writing smithi06431743-28 from 1798066 to 2275745 tid 2 2024-03-04T06:08:14.077 INFO:tasks.rados.rados.0.smithi064.stdout:116: writing smithi06431743-28 from 2742758 to 3199975 tid 3 2024-03-04T06:08:14.077 INFO:tasks.rados.rados.0.smithi064.stdout:117: write oid 50 current snap is 6 2024-03-04T06:08:14.077 INFO:tasks.rados.rados.0.smithi064.stdout:117: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-03-04T06:08:14.084 INFO:tasks.rados.rados.0.smithi064.stdout:117: writing smithi06431743-50 from 527753 to 1235954 tid 1 2024-03-04T06:08:14.091 INFO:tasks.rados.rados.0.smithi064.stdout:117: writing smithi06431743-50 from 1856548 to 2594238 tid 2 2024-03-04T06:08:14.096 INFO:tasks.rados.rados.0.smithi064.stdout:117: writing smithi06431743-50 from 3286534 to 3327753 tid 3 2024-03-04T06:08:14.096 INFO:tasks.rados.rados.0.smithi064.stdout:118: write oid 49 current snap is 6 2024-03-04T06:08:14.096 INFO:tasks.rados.rados.0.smithi064.stdout:116: finishing write tid 1 to smithi06431743-28 2024-03-04T06:08:14.096 INFO:tasks.rados.rados.0.smithi064.stdout:118: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-03-04T06:08:14.102 INFO:tasks.rados.rados.0.smithi064.stdout:118: writing smithi06431743-49 from 544560 to 1328010 tid 1 2024-03-04T06:08:14.108 INFO:tasks.rados.rados.0.smithi064.stdout:118: writing smithi06431743-49 from 1915030 to 2396378 tid 2 2024-03-04T06:08:14.113 INFO:tasks.rados.rados.0.smithi064.stdout:118: writing smithi06431743-49 from 2797604 to 3277365 tid 3 2024-03-04T06:08:14.113 INFO:tasks.rados.rados.0.smithi064.stdout:119: write (excl) oid 26 current snap is 6 2024-03-04T06:08:14.114 INFO:tasks.rados.rados.0.smithi064.stdout:119: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-03-04T06:08:14.119 INFO:tasks.rados.rados.0.smithi064.stdout:119: writing smithi06431743-26 from 561367 to 1020066 tid 1 2024-03-04T06:08:14.126 INFO:tasks.rados.rados.0.smithi064.stdout:119: writing smithi06431743-26 from 1573512 to 2314871 tid 2 2024-03-04T06:08:14.132 INFO:tasks.rados.rados.0.smithi064.stdout:119: writing smithi06431743-26 from 2941380 to 3361367 tid 3 2024-03-04T06:08:14.132 INFO:tasks.rados.rados.0.smithi064.stdout:120: write (excl) oid 44 current snap is 6 2024-03-04T06:08:14.132 INFO:tasks.rados.rados.0.smithi064.stdout:120: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-03-04T06:08:14.137 INFO:tasks.rados.rados.0.smithi064.stdout:120: writing smithi06431743-44 from 578174 to 1112122 tid 1 2024-03-04T06:08:14.139 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:08:14.139 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:08:14.139 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:08:14.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:08:14.144 INFO:tasks.rados.rados.0.smithi064.stdout:120: writing smithi06431743-44 from 1515641 to 2000658 tid 2 2024-03-04T06:08:14.151 INFO:tasks.rados.rados.0.smithi064.stdout:120: writing smithi06431743-44 from 2736097 to 3238402 tid 3 2024-03-04T06:08:14.151 INFO:tasks.rados.rados.0.smithi064.stdout:121: read oid 22 snap -1 2024-03-04T06:08:14.151 INFO:tasks.rados.rados.0.smithi064.stdout:121: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-04T06:08:14.151 INFO:tasks.rados.rados.0.smithi064.stdout:122: snap_remove snap 5 2024-03-04T06:08:14.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:14.264+0000 7f786c51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:14.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:14.271+0000 7f786c51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:14.284 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:14.284 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:14.284 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:14.284 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:14.284 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:14.284 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:14.284 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:14.284 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:14.285 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:14.285 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:14.285 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:14.285 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:14.287 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:14.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:08:14.442 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:14.437+0000 7f5c34624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:14.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:14.445+0000 7f5c34624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:14.458 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:14.458 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:14.458 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:14.458 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:14.458 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:14.458 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:14.458 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:14.458 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:14.458 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:14.458 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:14.459 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:14.459 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:14.461 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:14.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:08:14.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:14.565+0000 7ff56f39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:14.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:14.572+0000 7ff56f39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:14.585 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:14.585 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:14.585 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:14.585 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:14.585 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:14.588 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:14.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:08:14.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:14.691+0000 7f9dc6c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:14.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:14.698+0000 7f9dc6c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:14.711 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:14.711 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:14.711 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:14.711 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:14.712 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:14.712 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:14.712 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:14.712 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:14.712 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:14.712 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:14.712 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:14.712 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:14.714 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:14.714 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:14.842 INFO:tasks.rados.rados.0.smithi064.stdout:122: done (6 left) 2024-03-04T06:08:14.842 INFO:tasks.rados.rados.0.smithi064.stdout:123: read oid 37 snap -1 2024-03-04T06:08:14.842 INFO:tasks.rados.rados.0.smithi064.stdout:123: expect (ObjNum 56 snap 0 seq_num 56) 2024-03-04T06:08:14.842 INFO:tasks.rados.rados.0.smithi064.stdout:124: snap_remove snap 4 2024-03-04T06:08:15.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:15.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:15.680+0000 7fab35f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:15.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:15.684+0000 7fab35f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:15.840 INFO:tasks.rados.rados.0.smithi064.stdout:124: done (7 left) 2024-03-04T06:08:15.840 INFO:tasks.rados.rados.0.smithi064.stdout:125: snap_remove snap 3 2024-03-04T06:08:16.841 INFO:tasks.rados.rados.0.smithi064.stdout:125: done (7 left) 2024-03-04T06:08:16.841 INFO:tasks.rados.rados.0.smithi064.stdout:126: write oid 36 current snap is 6 2024-03-04T06:08:16.841 INFO:tasks.rados.rados.0.smithi064.stdout:116: finishing write tid 2 to smithi06431743-28 2024-03-04T06:08:16.841 INFO:tasks.rados.rados.0.smithi064.stdout:126: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-03-04T06:08:16.847 INFO:tasks.rados.rados.0.smithi064.stdout:126: writing smithi06431743-36 from 594981 to 1204178 tid 1 2024-03-04T06:08:16.855 INFO:tasks.rados.rados.0.smithi064.stdout:126: writing smithi06431743-36 from 1974123 to 2719151 tid 2 2024-03-04T06:08:16.860 INFO:tasks.rados.rados.0.smithi064.stdout:126: writing smithi06431743-36 from 3279873 to 3394981 tid 3 2024-03-04T06:08:16.860 INFO:tasks.rados.rados.0.smithi064.stdout:127: write (excl) oid 35 current snap is 6 2024-03-04T06:08:16.861 INFO:tasks.rados.rados.0.smithi064.stdout:117: finishing write tid 1 to smithi06431743-50 2024-03-04T06:08:16.861 INFO:tasks.rados.rados.0.smithi064.stdout:127: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-03-04T06:08:16.866 INFO:tasks.rados.rados.0.smithi064.stdout:127: writing smithi06431743-35 from 611788 to 1012587 tid 1 2024-03-04T06:08:16.875 INFO:tasks.rados.rados.0.smithi064.stdout:127: writing smithi06431743-35 from 1748958 to 2237644 tid 2 2024-03-04T06:08:16.883 INFO:tasks.rados.rados.0.smithi064.stdout:127: writing smithi06431743-35 from 2907296 to 3411788 tid 3 2024-03-04T06:08:16.883 INFO:tasks.rados.rados.0.smithi064.stdout:128: write oid 2 current snap is 6 2024-03-04T06:08:16.883 INFO:tasks.rados.rados.0.smithi064.stdout:128: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-03-04T06:08:16.890 INFO:tasks.rados.rados.0.smithi064.stdout:128: writing smithi06431743-2 from 628595 to 1104643 tid 1 2024-03-04T06:08:16.898 INFO:tasks.rados.rados.0.smithi064.stdout:128: writing smithi06431743-2 from 1807440 to 2439784 tid 2 2024-03-04T06:08:16.905 INFO:tasks.rados.rados.0.smithi064.stdout:128: writing smithi06431743-2 from 2934719 to 3387193 tid 3 2024-03-04T06:08:16.905 INFO:tasks.rados.rados.0.smithi064.stdout:129: read oid 9 snap -1 2024-03-04T06:08:16.905 INFO:tasks.rados.rados.0.smithi064.stdout:129: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-04T06:08:16.905 INFO:tasks.rados.rados.0.smithi064.stdout:130: read oid 46 snap -1 2024-03-04T06:08:16.905 INFO:tasks.rados.rados.0.smithi064.stdout:130: expect (ObjNum 62 snap 2 seq_num 62) 2024-03-04T06:08:16.905 INFO:tasks.rados.rados.0.smithi064.stdout:131: write oid 1 current snap is 6 2024-03-04T06:08:16.905 INFO:tasks.rados.rados.0.smithi064.stdout:131: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-03-04T06:08:16.911 INFO:tasks.rados.rados.0.smithi064.stdout:131: writing smithi06431743-1 from 645402 to 1196699 tid 1 2024-03-04T06:08:16.917 INFO:tasks.rados.rados.0.smithi064.stdout:131: writing smithi06431743-1 from 1749569 to 2241924 tid 2 2024-03-04T06:08:16.926 INFO:tasks.rados.rados.0.smithi064.stdout:131: writing smithi06431743-1 from 2845789 to 3445402 tid 3 2024-03-04T06:08:16.926 INFO:tasks.rados.rados.0.smithi064.stdout:132: read oid 19 snap -1 2024-03-04T06:08:16.926 INFO:tasks.rados.rados.0.smithi064.stdout:132: expect (ObjNum 55 snap 0 seq_num 55) 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:133: write oid 43 current snap is 6 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:116: finishing write tid 3 to smithi06431743-28 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:116: finishing write tid 4 to smithi06431743-28 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:117: finishing write tid 2 to smithi06431743-50 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:116: finishing write tid 5 to smithi06431743-28 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:116: oid 28 updating version 0 to 41 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:116: oid 28 updating version 41 to 44 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:116: oid 28 version 44 is already newer than 42 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:116: oid 28 version 44 is already newer than 44 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 44 (ObjNum 78 snap 6 seq_num 78) dirty exists 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:116: left oid 28 (ObjNum 78 snap 6 seq_num 78) 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:117: finishing write tid 3 to smithi06431743-50 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:119: finishing write tid 1 to smithi06431743-26 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:120: finishing write tid 1 to smithi06431743-44 2024-03-04T06:08:16.927 INFO:tasks.rados.rados.0.smithi064.stdout:117: finishing write tid 4 to smithi06431743-50 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:117: finishing write tid 5 to smithi06431743-50 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:117: oid 50 updating version 0 to 23 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:117: oid 50 updating version 23 to 25 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:117: oid 50 updating version 25 to 26 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:117: oid 50 version 26 is already newer than 25 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 26 (ObjNum 79 snap 6 seq_num 79) dirty exists 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:117: left oid 50 (ObjNum 79 snap 6 seq_num 79) 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:118: finishing write tid 1 to smithi06431743-49 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:119: finishing write tid 2 to smithi06431743-26 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:120: finishing write tid 2 to smithi06431743-44 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:118: finishing write tid 2 to smithi06431743-49 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:119: finishing write tid 3 to smithi06431743-26 2024-03-04T06:08:16.928 INFO:tasks.rados.rados.0.smithi064.stdout:133: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-03-04T06:08:16.934 INFO:tasks.rados.rados.0.smithi064.stdout:133: writing smithi06431743-43 from 662209 to 1288755 tid 1 2024-03-04T06:08:16.941 INFO:tasks.rados.rados.0.smithi064.stdout:133: writing smithi06431743-43 from 1808051 to 2444064 tid 2 2024-03-04T06:08:16.947 INFO:tasks.rados.rados.0.smithi064.stdout:133: writing smithi06431743-43 from 2873212 to 3462209 tid 3 2024-03-04T06:08:16.947 INFO:tasks.rados.rados.0.smithi064.stdout:116: done (14 left) 2024-03-04T06:08:16.947 INFO:tasks.rados.rados.0.smithi064.stdout:117: done (13 left) 2024-03-04T06:08:16.947 INFO:tasks.rados.rados.0.smithi064.stdout:134: snap_create 2024-03-04T06:08:16.947 INFO:tasks.rados.rados.0.smithi064.stdout:121: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-04T06:08:17.111 INFO:tasks.rados.rados.0.smithi064.stdout:119: finishing write tid 4 to smithi06431743-26 2024-03-04T06:08:17.111 INFO:tasks.rados.rados.0.smithi064.stdout:119: finishing write tid 5 to smithi06431743-26 2024-03-04T06:08:17.111 INFO:tasks.rados.rados.0.smithi064.stdout:119: oid 26 updating version 0 to 48 2024-03-04T06:08:17.111 INFO:tasks.rados.rados.0.smithi064.stdout:119: oid 26 version 48 is already newer than 47 2024-03-04T06:08:17.111 INFO:tasks.rados.rados.0.smithi064.stdout:119: oid 26 version 48 is already newer than 45 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:119: oid 26 version 48 is already newer than 47 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 48 (ObjNum 81 snap 6 seq_num 81) dirty exists 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:119: left oid 26 (ObjNum 81 snap 6 seq_num 81) 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:118: finishing write tid 3 to smithi06431743-49 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:120: finishing write tid 3 to smithi06431743-44 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:118: finishing write tid 4 to smithi06431743-49 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:118: finishing write tid 5 to smithi06431743-49 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:118: oid 49 updating version 0 to 18 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:118: oid 49 version 18 is already newer than 18 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:118: oid 49 updating version 18 to 19 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:118: oid 49 version 19 is already newer than 16 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 19 (ObjNum 80 snap 6 seq_num 80) dirty exists 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:118: left oid 49 (ObjNum 80 snap 6 seq_num 80) 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:120: finishing write tid 4 to smithi06431743-44 2024-03-04T06:08:17.112 INFO:tasks.rados.rados.0.smithi064.stdout:120: finishing write tid 5 to smithi06431743-44 2024-03-04T06:08:17.113 INFO:tasks.rados.rados.0.smithi064.stdout:120: oid 44 updating version 0 to 22 2024-03-04T06:08:17.113 INFO:tasks.rados.rados.0.smithi064.stdout:120: oid 44 version 22 is already newer than 21 2024-03-04T06:08:17.113 INFO:tasks.rados.rados.0.smithi064.stdout:120: oid 44 updating version 22 to 23 2024-03-04T06:08:17.113 INFO:tasks.rados.rados.0.smithi064.stdout:120: oid 44 version 23 is already newer than 20 2024-03-04T06:08:17.113 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 23 (ObjNum 82 snap 6 seq_num 82) dirty exists 2024-03-04T06:08:17.113 INFO:tasks.rados.rados.0.smithi064.stdout:120: left oid 44 (ObjNum 82 snap 6 seq_num 82) 2024-03-04T06:08:17.113 INFO:tasks.rados.rados.0.smithi064.stdout:123: expect (ObjNum 56 snap 0 seq_num 56) 2024-03-04T06:08:17.258 INFO:tasks.rados.rados.0.smithi064.stdout:128: finishing write tid 1 to smithi06431743-2 2024-03-04T06:08:17.259 INFO:tasks.rados.rados.0.smithi064.stdout:129: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-04T06:08:17.338 INFO:tasks.rados.rados.0.smithi064.stdout:132: expect (ObjNum 55 snap 0 seq_num 55) 2024-03-04T06:08:17.478 INFO:tasks.rados.rados.0.smithi064.stdout:126: finishing write tid 1 to smithi06431743-36 2024-03-04T06:08:17.478 INFO:tasks.rados.rados.0.smithi064.stdout:127: finishing write tid 1 to smithi06431743-35 2024-03-04T06:08:17.478 INFO:tasks.rados.rados.0.smithi064.stdout:128: finishing write tid 2 to smithi06431743-2 2024-03-04T06:08:17.478 INFO:tasks.rados.rados.0.smithi064.stdout:130: expect (ObjNum 62 snap 2 seq_num 62) 2024-03-04T06:08:17.633 INFO:tasks.rados.rados.0.smithi064.stdout:133: finishing write tid 1 to smithi06431743-43 2024-03-04T06:08:17.633 INFO:tasks.rados.rados.0.smithi064.stdout:127: finishing write tid 2 to smithi06431743-35 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:131: finishing write tid 1 to smithi06431743-1 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:128: finishing write tid 3 to smithi06431743-2 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:126: finishing write tid 2 to smithi06431743-36 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:127: finishing write tid 3 to smithi06431743-35 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:126: finishing write tid 3 to smithi06431743-36 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:128: finishing write tid 4 to smithi06431743-2 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:131: finishing write tid 2 to smithi06431743-1 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:128: finishing write tid 5 to smithi06431743-2 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:128: oid 2 updating version 0 to 20 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:128: oid 2 updating version 20 to 22 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:128: oid 2 updating version 22 to 23 2024-03-04T06:08:17.634 INFO:tasks.rados.rados.0.smithi064.stdout:128: oid 2 version 23 is already newer than 21 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 23 (ObjNum 85 snap 6 seq_num 85) dirty exists 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:128: left oid 2 (ObjNum 85 snap 6 seq_num 85) 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:127: finishing write tid 4 to smithi06431743-35 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:127: finishing write tid 5 to smithi06431743-35 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:127: oid 35 updating version 0 to 23 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:127: oid 35 updating version 23 to 26 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:127: oid 35 version 26 is already newer than 26 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:127: oid 35 version 26 is already newer than 24 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 26 (ObjNum 84 snap 6 seq_num 84) dirty exists 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:127: left oid 35 (ObjNum 84 snap 6 seq_num 84) 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:133: finishing write tid 2 to smithi06431743-43 2024-03-04T06:08:17.635 INFO:tasks.rados.rados.0.smithi064.stdout:126: finishing write tid 4 to smithi06431743-36 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:126: finishing write tid 5 to smithi06431743-36 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:126: oid 36 updating version 0 to 12 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:126: oid 36 updating version 12 to 13 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:126: oid 36 version 13 is already newer than 10 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:126: oid 36 version 13 is already newer than 11 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 13 (ObjNum 83 snap 6 seq_num 83) dirty exists 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:126: left oid 36 (ObjNum 83 snap 6 seq_num 83) 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:131: finishing write tid 3 to smithi06431743-1 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:131: finishing write tid 4 to smithi06431743-1 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:133: finishing write tid 3 to smithi06431743-43 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:131: finishing write tid 5 to smithi06431743-1 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:131: oid 1 updating version 0 to 59 2024-03-04T06:08:17.636 INFO:tasks.rados.rados.0.smithi064.stdout:131: oid 1 version 59 is already newer than 57 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:131: oid 1 version 59 is already newer than 58 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:131: oid 1 updating version 59 to 60 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 60 (ObjNum 86 snap 6 seq_num 86) dirty exists 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:131: left oid 1 (ObjNum 86 snap 6 seq_num 86) 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:133: finishing write tid 4 to smithi06431743-43 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:133: finishing write tid 5 to smithi06431743-43 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:133: oid 43 updating version 0 to 23 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:133: oid 43 version 23 is already newer than 22 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:133: oid 43 version 23 is already newer than 21 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:133: oid 43 updating version 23 to 24 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 24 (ObjNum 87 snap 6 seq_num 87) dirty exists 2024-03-04T06:08:17.637 INFO:tasks.rados.rados.0.smithi064.stdout:133: left oid 43 (ObjNum 87 snap 6 seq_num 87) 2024-03-04T06:08:17.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:17.853 INFO:tasks.rados.rados.0.smithi064.stdout:118: done (13 left) 2024-03-04T06:08:17.853 INFO:tasks.rados.rados.0.smithi064.stdout:119: done (12 left) 2024-03-04T06:08:17.853 INFO:tasks.rados.rados.0.smithi064.stdout:120: done (11 left) 2024-03-04T06:08:17.853 INFO:tasks.rados.rados.0.smithi064.stdout:121: done (10 left) 2024-03-04T06:08:17.853 INFO:tasks.rados.rados.0.smithi064.stdout:123: done (9 left) 2024-03-04T06:08:17.853 INFO:tasks.rados.rados.0.smithi064.stdout:126: done (8 left) 2024-03-04T06:08:17.853 INFO:tasks.rados.rados.0.smithi064.stdout:127: done (7 left) 2024-03-04T06:08:17.853 INFO:tasks.rados.rados.0.smithi064.stdout:128: done (6 left) 2024-03-04T06:08:17.853 INFO:tasks.rados.rados.0.smithi064.stdout:129: done (5 left) 2024-03-04T06:08:17.854 INFO:tasks.rados.rados.0.smithi064.stdout:130: done (4 left) 2024-03-04T06:08:17.854 INFO:tasks.rados.rados.0.smithi064.stdout:131: done (3 left) 2024-03-04T06:08:17.854 INFO:tasks.rados.rados.0.smithi064.stdout:132: done (2 left) 2024-03-04T06:08:17.854 INFO:tasks.rados.rados.0.smithi064.stdout:133: done (1 left) 2024-03-04T06:08:17.854 INFO:tasks.rados.rados.0.smithi064.stdout:134: done (0 left) 2024-03-04T06:08:17.854 INFO:tasks.rados.rados.0.smithi064.stdout:135: write (excl) oid 42 current snap is 7 2024-03-04T06:08:17.854 INFO:tasks.rados.rados.0.smithi064.stdout:135: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-03-04T06:08:17.860 INFO:tasks.rados.rados.0.smithi064.stdout:135: writing smithi06431743-42 from 679016 to 1380811 tid 1 2024-03-04T06:08:17.860 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:17.858+0000 7ff6cef9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:17.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:17.863+0000 7ff6cef9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:17.865 INFO:tasks.rados.rados.0.smithi064.stdout:135: writing smithi06431743-42 from 1866533 to 2362557 tid 2 2024-03-04T06:08:17.872 INFO:tasks.rados.rados.0.smithi064.stdout:135: writing smithi06431743-42 from 2900635 to 3303278 tid 3 2024-03-04T06:08:17.872 INFO:tasks.rados.rados.0.smithi064.stdout:136: snap_remove snap 6 2024-03-04T06:08:18.857 INFO:tasks.rados.rados.0.smithi064.stdout:136: done (1 left) 2024-03-04T06:08:18.857 INFO:tasks.rados.rados.0.smithi064.stdout:137: read oid 2 snap -1 2024-03-04T06:08:18.858 INFO:tasks.rados.rados.0.smithi064.stdout:137: expect (ObjNum 85 snap 6 seq_num 85) 2024-03-04T06:08:18.858 INFO:tasks.rados.rados.0.smithi064.stdout:138: read oid 16 snap -1 2024-03-04T06:08:18.858 INFO:tasks.rados.rados.0.smithi064.stdout:138: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-04T06:08:18.858 INFO:tasks.rados.rados.0.smithi064.stdout:139: read oid 32 snap -1 2024-03-04T06:08:18.858 INFO:tasks.rados.rados.0.smithi064.stdout:139: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-04T06:08:18.858 INFO:tasks.rados.rados.0.smithi064.stdout:140: read oid 22 snap -1 2024-03-04T06:08:18.858 INFO:tasks.rados.rados.0.smithi064.stdout:140: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-04T06:08:18.858 INFO:tasks.rados.rados.0.smithi064.stdout:141: write oid 17 current snap is 7 2024-03-04T06:08:18.858 INFO:tasks.rados.rados.0.smithi064.stdout:135: finishing write tid 1 to smithi06431743-42 2024-03-04T06:08:18.858 INFO:tasks.rados.rados.0.smithi064.stdout:141: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-03-04T06:08:18.864 INFO:tasks.rados.rados.0.smithi064.stdout:141: writing smithi06431743-17 from 695823 to 1472867 tid 1 2024-03-04T06:08:18.870 INFO:tasks.rados.rados.0.smithi064.stdout:141: writing smithi06431743-17 from 1925015 to 2564697 tid 2 2024-03-04T06:08:18.876 INFO:tasks.rados.rados.0.smithi064.stdout:141: writing smithi06431743-17 from 3328058 to 3495823 tid 3 2024-03-04T06:08:18.877 INFO:tasks.rados.rados.0.smithi064.stdout:142: write (excl) oid 36 current snap is 7 2024-03-04T06:08:18.877 INFO:tasks.rados.rados.0.smithi064.stdout:135: finishing write tid 2 to smithi06431743-42 2024-03-04T06:08:18.877 INFO:tasks.rados.rados.0.smithi064.stdout:135: finishing write tid 3 to smithi06431743-42 2024-03-04T06:08:18.877 INFO:tasks.rados.rados.0.smithi064.stdout:135: finishing write tid 4 to smithi06431743-42 2024-03-04T06:08:18.877 INFO:tasks.rados.rados.0.smithi064.stdout:135: finishing write tid 5 to smithi06431743-42 2024-03-04T06:08:18.877 INFO:tasks.rados.rados.0.smithi064.stdout:135: oid 42 updating version 0 to 28 2024-03-04T06:08:18.877 INFO:tasks.rados.rados.0.smithi064.stdout:135: oid 42 version 28 is already newer than 25 2024-03-04T06:08:18.877 INFO:tasks.rados.rados.0.smithi064.stdout:135: oid 42 version 28 is already newer than 27 2024-03-04T06:08:18.877 INFO:tasks.rados.rados.0.smithi064.stdout:135: oid 42 version 28 is already newer than 26 2024-03-04T06:08:18.877 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 28 (ObjNum 88 snap 7 seq_num 88) dirty exists 2024-03-04T06:08:18.877 INFO:tasks.rados.rados.0.smithi064.stdout:135: left oid 42 (ObjNum 88 snap 7 seq_num 88) 2024-03-04T06:08:18.878 INFO:tasks.rados.rados.0.smithi064.stdout:142: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-03-04T06:08:18.884 INFO:tasks.rados.rados.0.smithi064.stdout:142: writing smithi06431743-36 from 712630 to 1164923 tid 1 2024-03-04T06:08:18.890 INFO:tasks.rados.rados.0.smithi064.stdout:142: writing smithi06431743-36 from 1583497 to 2083190 tid 2 2024-03-04T06:08:18.897 INFO:tasks.rados.rados.0.smithi064.stdout:142: writing smithi06431743-36 from 2555481 to 2980668 tid 3 2024-03-04T06:08:18.897 INFO:tasks.rados.rados.0.smithi064.stdout:135: done (6 left) 2024-03-04T06:08:18.897 INFO:tasks.rados.rados.0.smithi064.stdout:143: write (excl) oid 29 current snap is 7 2024-03-04T06:08:18.897 INFO:tasks.rados.rados.0.smithi064.stdout:143: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-03-04T06:08:18.904 INFO:tasks.rados.rados.0.smithi064.stdout:143: writing smithi06431743-29 from 729437 to 1256979 tid 1 2024-03-04T06:08:18.911 INFO:tasks.rados.rados.0.smithi064.stdout:143: writing smithi06431743-29 from 1925626 to 2568977 tid 2 2024-03-04T06:08:18.917 INFO:tasks.rados.rados.0.smithi064.stdout:143: writing smithi06431743-29 from 3266551 to 3529437 tid 3 2024-03-04T06:08:18.917 INFO:tasks.rados.rados.0.smithi064.stdout:144: write oid 14 current snap is 7 2024-03-04T06:08:18.917 INFO:tasks.rados.rados.0.smithi064.stdout:144: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-03-04T06:08:18.926 INFO:tasks.rados.rados.0.smithi064.stdout:144: writing smithi06431743-14 from 746244 to 1465388 tid 1 2024-03-04T06:08:18.933 INFO:tasks.rados.rados.0.smithi064.stdout:144: writing smithi06431743-14 from 2100461 to 2603823 tid 2 2024-03-04T06:08:18.940 INFO:tasks.rados.rados.0.smithi064.stdout:144: writing smithi06431743-14 from 3126680 to 3546244 tid 3 2024-03-04T06:08:18.940 INFO:tasks.rados.rados.0.smithi064.stdout:145: write oid 13 current snap is 7 2024-03-04T06:08:18.940 INFO:tasks.rados.rados.0.smithi064.stdout:145: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-03-04T06:08:18.950 INFO:tasks.rados.rados.0.smithi064.stdout:145: writing smithi06431743-13 from 763051 to 1557444 tid 1 2024-03-04T06:08:18.958 INFO:tasks.rados.rados.0.smithi064.stdout:145: writing smithi06431743-13 from 2158943 to 2805963 tid 2 2024-03-04T06:08:18.963 INFO:tasks.rados.rados.0.smithi064.stdout:145: writing smithi06431743-13 from 3437750 to 3563051 tid 3 2024-03-04T06:08:18.963 INFO:tasks.rados.rados.0.smithi064.stdout:146: write oid 33 current snap is 7 2024-03-04T06:08:18.963 INFO:tasks.rados.rados.0.smithi064.stdout:146: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-03-04T06:08:18.971 INFO:tasks.rados.rados.0.smithi064.stdout:146: writing smithi06431743-33 from 779858 to 1249500 tid 1 2024-03-04T06:08:18.978 INFO:tasks.rados.rados.0.smithi064.stdout:146: writing smithi06431743-33 from 1817425 to 2324456 tid 2 2024-03-04T06:08:18.985 INFO:tasks.rados.rados.0.smithi064.stdout:146: writing smithi06431743-33 from 2781526 to 3368154 tid 3 2024-03-04T06:08:18.985 INFO:tasks.rados.rados.0.smithi064.stdout:139: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-04T06:08:19.139 INFO:tasks.rados.rados.0.smithi064.stdout:141: finishing write tid 1 to smithi06431743-17 2024-03-04T06:08:19.139 INFO:tasks.rados.rados.0.smithi064.stdout:137: expect (ObjNum 85 snap 6 seq_num 85) 2024-03-04T06:08:19.364 INFO:tasks.rados.rados.0.smithi064.stdout:138: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-04T06:08:19.449 INFO:tasks.rados.rados.0.smithi064.stdout:143: finishing write tid 1 to smithi06431743-29 2024-03-04T06:08:19.449 INFO:tasks.rados.rados.0.smithi064.stdout:140: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-04T06:08:19.607 INFO:tasks.rados.rados.0.smithi064.stdout:142: finishing write tid 1 to smithi06431743-36 2024-03-04T06:08:19.607 INFO:tasks.rados.rados.0.smithi064.stdout:147: read oid 34 snap -1 2024-03-04T06:08:19.607 INFO:tasks.rados.rados.0.smithi064.stdout:147: expect (ObjNum 73 snap 3 seq_num 73) 2024-03-04T06:08:19.607 INFO:tasks.rados.rados.0.smithi064.stdout:137: done (10 left) 2024-03-04T06:08:19.607 INFO:tasks.rados.rados.0.smithi064.stdout:138: done (9 left) 2024-03-04T06:08:19.607 INFO:tasks.rados.rados.0.smithi064.stdout:139: done (8 left) 2024-03-04T06:08:19.607 INFO:tasks.rados.rados.0.smithi064.stdout:140: done (7 left) 2024-03-04T06:08:19.607 INFO:tasks.rados.rados.0.smithi064.stdout:148: read oid 27 snap -1 2024-03-04T06:08:19.608 INFO:tasks.rados.rados.0.smithi064.stdout:148: expect (ObjNum 71 snap 3 seq_num 71) 2024-03-04T06:08:19.608 INFO:tasks.rados.rados.0.smithi064.stdout:149: delete oid 38 current snap is 7 2024-03-04T06:08:19.608 INFO:tasks.rados.rados.0.smithi064.stdout:144: finishing write tid 1 to smithi06431743-14 2024-03-04T06:08:19.608 INFO:tasks.rados.rados.0.smithi064.stdout:141: finishing write tid 2 to smithi06431743-17 2024-03-04T06:08:19.608 INFO:tasks.rados.rados.0.smithi064.stdout:143: finishing write tid 2 to smithi06431743-29 2024-03-04T06:08:19.608 INFO:tasks.rados.rados.0.smithi064.stdout:144: finishing write tid 2 to smithi06431743-14 2024-03-04T06:08:19.608 INFO:tasks.rados.rados.0.smithi064.stdout:141: finishing write tid 3 to smithi06431743-17 2024-03-04T06:08:19.608 INFO:tasks.rados.rados.0.smithi064.stdout:142: finishing write tid 2 to smithi06431743-36 2024-03-04T06:08:19.608 INFO:tasks.rados.rados.0.smithi064.stdout:144: finishing write tid 3 to smithi06431743-14 2024-03-04T06:08:19.608 INFO:tasks.rados.rados.0.smithi064.stdout:146: finishing write tid 1 to smithi06431743-33 2024-03-04T06:08:19.609 INFO:tasks.rados.rados.0.smithi064.stdout:143: finishing write tid 3 to smithi06431743-29 2024-03-04T06:08:19.609 INFO:tasks.rados.rados.0.smithi064.stdout:141: finishing write tid 4 to smithi06431743-17 2024-03-04T06:08:19.609 INFO:tasks.rados.rados.0.smithi064.stdout:144: finishing write tid 4 to smithi06431743-14 2024-03-04T06:08:19.609 INFO:tasks.rados.rados.0.smithi064.stdout:141: finishing write tid 5 to smithi06431743-17 2024-03-04T06:08:19.609 INFO:tasks.rados.rados.0.smithi064.stdout:141: oid 17 updating version 0 to 25 2024-03-04T06:08:19.609 INFO:tasks.rados.rados.0.smithi064.stdout:141: oid 17 version 25 is already newer than 23 2024-03-04T06:08:19.609 INFO:tasks.rados.rados.0.smithi064.stdout:141: oid 17 version 25 is already newer than 24 2024-03-04T06:08:19.609 INFO:tasks.rados.rados.0.smithi064.stdout:141: oid 17 version 25 is already newer than 22 2024-03-04T06:08:19.609 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 25 (ObjNum 89 snap 7 seq_num 89) dirty exists 2024-03-04T06:08:19.609 INFO:tasks.rados.rados.0.smithi064.stdout:141: left oid 17 (ObjNum 89 snap 7 seq_num 89) 2024-03-04T06:08:19.610 INFO:tasks.rados.rados.0.smithi064.stdout:142: finishing write tid 3 to smithi06431743-36 2024-03-04T06:08:19.610 INFO:tasks.rados.rados.0.smithi064.stdout:144: finishing write tid 5 to smithi06431743-14 2024-03-04T06:08:19.610 INFO:tasks.rados.rados.0.smithi064.stdout:144: oid 14 updating version 0 to 29 2024-03-04T06:08:19.610 INFO:tasks.rados.rados.0.smithi064.stdout:144: oid 14 version 29 is already newer than 29 2024-03-04T06:08:19.610 INFO:tasks.rados.rados.0.smithi064.stdout:144: oid 14 updating version 29 to 30 2024-03-04T06:08:19.610 INFO:tasks.rados.rados.0.smithi064.stdout:144: oid 14 version 30 is already newer than 27 2024-03-04T06:08:19.610 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 30 (ObjNum 92 snap 7 seq_num 92) dirty exists 2024-03-04T06:08:19.610 INFO:tasks.rados.rados.0.smithi064.stdout:144: left oid 14 (ObjNum 92 snap 7 seq_num 92) 2024-03-04T06:08:19.610 INFO:tasks.rados.rados.0.smithi064.stdout:142: finishing write tid 4 to smithi06431743-36 2024-03-04T06:08:19.610 INFO:tasks.rados.rados.0.smithi064.stdout:142: finishing write tid 5 to smithi06431743-36 2024-03-04T06:08:19.611 INFO:tasks.rados.rados.0.smithi064.stdout:142: oid 36 updating version 0 to 17 2024-03-04T06:08:19.611 INFO:tasks.rados.rados.0.smithi064.stdout:142: oid 36 version 17 is already newer than 17 2024-03-04T06:08:19.611 INFO:tasks.rados.rados.0.smithi064.stdout:142: oid 36 version 17 is already newer than 14 2024-03-04T06:08:19.611 INFO:tasks.rados.rados.0.smithi064.stdout:142: oid 36 version 17 is already newer than 15 2024-03-04T06:08:19.611 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 17 (ObjNum 90 snap 7 seq_num 90) dirty exists 2024-03-04T06:08:19.611 INFO:tasks.rados.rados.0.smithi064.stdout:142: left oid 36 (ObjNum 90 snap 7 seq_num 90) 2024-03-04T06:08:19.611 INFO:tasks.rados.rados.0.smithi064.stdout:143: finishing write tid 4 to smithi06431743-29 2024-03-04T06:08:19.611 INFO:tasks.rados.rados.0.smithi064.stdout:143: finishing write tid 5 to smithi06431743-29 2024-03-04T06:08:19.611 INFO:tasks.rados.rados.0.smithi064.stdout:143: oid 29 updating version 0 to 25 2024-03-04T06:08:19.611 INFO:tasks.rados.rados.0.smithi064.stdout:143: oid 29 updating version 25 to 27 2024-03-04T06:08:19.612 INFO:tasks.rados.rados.0.smithi064.stdout:143: oid 29 version 27 is already newer than 24 2024-03-04T06:08:19.612 INFO:tasks.rados.rados.0.smithi064.stdout:143: oid 29 version 27 is already newer than 26 2024-03-04T06:08:19.612 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 27 (ObjNum 91 snap 7 seq_num 91) dirty exists 2024-03-04T06:08:19.612 INFO:tasks.rados.rados.0.smithi064.stdout:143: left oid 29 (ObjNum 91 snap 7 seq_num 91) 2024-03-04T06:08:19.612 INFO:tasks.rados.rados.0.smithi064.stdout:146: finishing write tid 2 to smithi06431743-33 2024-03-04T06:08:19.612 INFO:tasks.rados.rados.0.smithi064.stdout:145: finishing write tid 1 to smithi06431743-13 2024-03-04T06:08:19.612 INFO:tasks.rados.rados.0.smithi064.stdout:146: finishing write tid 3 to smithi06431743-33 2024-03-04T06:08:19.612 INFO:tasks.rados.rados.0.smithi064.stdout:145: finishing write tid 2 to smithi06431743-13 2024-03-04T06:08:19.612 INFO:tasks.rados.rados.0.smithi064.stdout:145: finishing write tid 3 to smithi06431743-13 2024-03-04T06:08:19.613 INFO:tasks.rados.rados.0.smithi064.stdout:145: finishing write tid 4 to smithi06431743-13 2024-03-04T06:08:19.613 INFO:tasks.rados.rados.0.smithi064.stdout:145: finishing write tid 5 to smithi06431743-13 2024-03-04T06:08:19.613 INFO:tasks.rados.rados.0.smithi064.stdout:145: oid 13 updating version 0 to 31 2024-03-04T06:08:19.613 INFO:tasks.rados.rados.0.smithi064.stdout:145: oid 13 version 31 is already newer than 31 2024-03-04T06:08:19.613 INFO:tasks.rados.rados.0.smithi064.stdout:145: oid 13 version 31 is already newer than 31 2024-03-04T06:08:19.613 INFO:tasks.rados.rados.0.smithi064.stdout:145: oid 13 version 31 is already newer than 31 2024-03-04T06:08:19.613 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 31 (ObjNum 93 snap 7 seq_num 93) dirty exists 2024-03-04T06:08:19.613 INFO:tasks.rados.rados.0.smithi064.stdout:145: left oid 13 (ObjNum 93 snap 7 seq_num 93) 2024-03-04T06:08:19.613 INFO:tasks.rados.rados.0.smithi064.stdout:146: finishing write tid 4 to smithi06431743-33 2024-03-04T06:08:19.613 INFO:tasks.rados.rados.0.smithi064.stdout:146: finishing write tid 5 to smithi06431743-33 2024-03-04T06:08:19.614 INFO:tasks.rados.rados.0.smithi064.stdout:146: oid 33 updating version 0 to 31 2024-03-04T06:08:19.614 INFO:tasks.rados.rados.0.smithi064.stdout:146: oid 33 updating version 31 to 33 2024-03-04T06:08:19.614 INFO:tasks.rados.rados.0.smithi064.stdout:146: oid 33 updating version 33 to 34 2024-03-04T06:08:19.614 INFO:tasks.rados.rados.0.smithi064.stdout:146: oid 33 version 34 is already newer than 32 2024-03-04T06:08:19.614 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 34 (ObjNum 94 snap 7 seq_num 94) dirty exists 2024-03-04T06:08:19.614 INFO:tasks.rados.rados.0.smithi064.stdout:146: left oid 33 (ObjNum 94 snap 7 seq_num 94) 2024-03-04T06:08:19.618 INFO:tasks.rados.rados.0.smithi064.stdout:141: done (8 left) 2024-03-04T06:08:19.619 INFO:tasks.rados.rados.0.smithi064.stdout:142: done (7 left) 2024-03-04T06:08:19.619 INFO:tasks.rados.rados.0.smithi064.stdout:143: done (6 left) 2024-03-04T06:08:19.619 INFO:tasks.rados.rados.0.smithi064.stdout:144: done (5 left) 2024-03-04T06:08:19.619 INFO:tasks.rados.rados.0.smithi064.stdout:145: done (4 left) 2024-03-04T06:08:19.619 INFO:tasks.rados.rados.0.smithi064.stdout:146: done (3 left) 2024-03-04T06:08:19.619 INFO:tasks.rados.rados.0.smithi064.stdout:149: done (2 left) 2024-03-04T06:08:19.619 INFO:tasks.rados.rados.0.smithi064.stdout:150: write (excl) oid 26 current snap is 7 2024-03-04T06:08:19.619 INFO:tasks.rados.rados.0.smithi064.stdout:150: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-03-04T06:08:19.624 INFO:tasks.rados.rados.0.smithi064.stdout:150: writing smithi06431743-26 from 413472 to 1033612 tid 1 2024-03-04T06:08:19.631 INFO:tasks.rados.rados.0.smithi064.stdout:150: writing smithi06431743-26 from 1818036 to 2328736 tid 2 2024-03-04T06:08:19.639 INFO:tasks.rados.rados.0.smithi064.stdout:150: writing smithi06431743-26 from 3120019 to 3613472 tid 3 2024-03-04T06:08:19.639 INFO:tasks.rados.rados.0.smithi064.stdout:151: delete oid 39 current snap is 7 2024-03-04T06:08:19.639 INFO:tasks.rados.rados.0.smithi064.stdout:147: expect (ObjNum 73 snap 3 seq_num 73) 2024-03-04T06:08:19.715 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:08:19.715 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:08:19.715 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:08:19.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:08:19.802 INFO:tasks.rados.rados.0.smithi064.stdout:148: expect (ObjNum 71 snap 3 seq_num 71) 2024-03-04T06:08:19.824 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:19.819+0000 7fabe5c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:19.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:19.827+0000 7fabe5c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:19.840 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:19.841 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:19.843 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:19.843 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:19.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:08:19.922 INFO:tasks.rados.rados.0.smithi064.stdout:150: finishing write tid 1 to smithi06431743-26 2024-03-04T06:08:19.922 INFO:tasks.rados.rados.0.smithi064.stdout:150: finishing write tid 2 to smithi06431743-26 2024-03-04T06:08:19.922 INFO:tasks.rados.rados.0.smithi064.stdout:150: finishing write tid 3 to smithi06431743-26 2024-03-04T06:08:19.922 INFO:tasks.rados.rados.0.smithi064.stdout:150: finishing write tid 4 to smithi06431743-26 2024-03-04T06:08:19.922 INFO:tasks.rados.rados.0.smithi064.stdout:150: finishing write tid 5 to smithi06431743-26 2024-03-04T06:08:19.923 INFO:tasks.rados.rados.0.smithi064.stdout:150: oid 26 updating version 0 to 64 2024-03-04T06:08:19.923 INFO:tasks.rados.rados.0.smithi064.stdout:150: oid 26 version 64 is already newer than 61 2024-03-04T06:08:19.923 INFO:tasks.rados.rados.0.smithi064.stdout:150: oid 26 version 64 is already newer than 63 2024-03-04T06:08:19.923 INFO:tasks.rados.rados.0.smithi064.stdout:150: oid 26 version 64 is already newer than 62 2024-03-04T06:08:19.923 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 64 (ObjNum 96 snap 7 seq_num 96) dirty exists 2024-03-04T06:08:19.923 INFO:tasks.rados.rados.0.smithi064.stdout:150: left oid 26 (ObjNum 96 snap 7 seq_num 96) 2024-03-04T06:08:19.923 INFO:tasks.rados.rados.0.smithi064.stdout:147: done (3 left) 2024-03-04T06:08:19.923 INFO:tasks.rados.rados.0.smithi064.stdout:148: done (2 left) 2024-03-04T06:08:19.923 INFO:tasks.rados.rados.0.smithi064.stdout:150: done (1 left) 2024-03-04T06:08:19.923 INFO:tasks.rados.rados.0.smithi064.stdout:151: done (0 left) 2024-03-04T06:08:19.924 INFO:tasks.rados.rados.0.smithi064.stdout:152: write (excl) oid 40 current snap is 7 2024-03-04T06:08:19.924 INFO:tasks.rados.rados.0.smithi064.stdout:152: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-03-04T06:08:19.928 INFO:tasks.rados.rados.0.smithi064.stdout:152: writing smithi06431743-40 from 447086 to 1217724 tid 1 2024-03-04T06:08:19.938 INFO:tasks.rados.rados.0.smithi064.stdout:152: writing smithi06431743-40 from 1935000 to 2733016 tid 2 2024-03-04T06:08:19.945 INFO:tasks.rados.rados.0.smithi064.stdout:152: writing smithi06431743-40 from 3458512 to 3647086 tid 3 2024-03-04T06:08:19.945 INFO:tasks.rados.rados.0.smithi064.stdout:153: read oid 26 snap -1 2024-03-04T06:08:19.945 INFO:tasks.rados.rados.0.smithi064.stdout:153: expect (ObjNum 96 snap 7 seq_num 96) 2024-03-04T06:08:19.945 INFO:tasks.rados.rados.0.smithi064.stdout:154: snap_create 2024-03-04T06:08:19.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:19.951+0000 7f9eefd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:19.960 INFO:tasks.rados.rados.0.smithi064.stdout:152: finishing write tid 1 to smithi06431743-40 2024-03-04T06:08:19.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:19.961+0000 7f9eefd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:19.971 INFO:tasks.rados.rados.0.smithi064.stdout:153: expect (ObjNum 96 snap 7 seq_num 96) 2024-03-04T06:08:19.977 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:19.978 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:19.978 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:19.978 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:19.978 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:19.978 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:19.978 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:19.978 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:19.978 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:19.978 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:19.979 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:19.979 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:19.982 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:19.982 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:08:19.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:20.092 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:20.087+0000 7fca7d31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:20.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:20.095+0000 7fca7d31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:20.109 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:20.109 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:20.109 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:20.109 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:20.109 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:20.109 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:20.110 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:20.110 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:20.110 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:20.110 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:20.110 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:20.110 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:20.113 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:20.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:08:20.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:20.132+0000 7fc97e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:20.137 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:20.136+0000 7fc97e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:20.155 INFO:tasks.rados.rados.0.smithi064.stdout:152: finishing write tid 2 to smithi06431743-40 2024-03-04T06:08:20.155 INFO:tasks.rados.rados.0.smithi064.stdout:152: finishing write tid 3 to smithi06431743-40 2024-03-04T06:08:20.155 INFO:tasks.rados.rados.0.smithi064.stdout:152: finishing write tid 4 to smithi06431743-40 2024-03-04T06:08:20.155 INFO:tasks.rados.rados.0.smithi064.stdout:152: finishing write tid 5 to smithi06431743-40 2024-03-04T06:08:20.156 INFO:tasks.rados.rados.0.smithi064.stdout:152: oid 40 updating version 0 to 32 2024-03-04T06:08:20.156 INFO:tasks.rados.rados.0.smithi064.stdout:152: oid 40 version 32 is already newer than 29 2024-03-04T06:08:20.156 INFO:tasks.rados.rados.0.smithi064.stdout:152: oid 40 version 32 is already newer than 30 2024-03-04T06:08:20.156 INFO:tasks.rados.rados.0.smithi064.stdout:152: oid 40 version 32 is already newer than 31 2024-03-04T06:08:20.156 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 32 (ObjNum 98 snap 7 seq_num 98) dirty exists 2024-03-04T06:08:20.156 INFO:tasks.rados.rados.0.smithi064.stdout:152: left oid 40 (ObjNum 98 snap 7 seq_num 98) 2024-03-04T06:08:20.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:20.225+0000 7fceda0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:20.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:20.232+0000 7fceda0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:20.245 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:20.247 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:20.248 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:20.859 INFO:tasks.rados.rados.0.smithi064.stdout:152: done (2 left) 2024-03-04T06:08:20.859 INFO:tasks.rados.rados.0.smithi064.stdout:153: done (1 left) 2024-03-04T06:08:20.859 INFO:tasks.rados.rados.0.smithi064.stdout:154: done (0 left) 2024-03-04T06:08:20.859 INFO:tasks.rados.rados.0.smithi064.stdout:155: read oid 33 snap -1 2024-03-04T06:08:20.859 INFO:tasks.rados.rados.0.smithi064.stdout:155: expect (ObjNum 94 snap 7 seq_num 94) 2024-03-04T06:08:20.859 INFO:tasks.rados.rados.0.smithi064.stdout:156: write (excl) oid 26 current snap is 8 2024-03-04T06:08:20.859 INFO:tasks.rados.rados.0.smithi064.stdout:156: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-03-04T06:08:20.864 INFO:tasks.rados.rados.0.smithi064.stdout:156: writing smithi06431743-26 from 463893 to 1026133 tid 1 2024-03-04T06:08:20.871 INFO:tasks.rados.rados.0.smithi064.stdout:156: writing smithi06431743-26 from 1593482 to 2251509 tid 2 2024-03-04T06:08:20.876 INFO:tasks.rados.rados.0.smithi064.stdout:156: writing smithi06431743-26 from 2685935 to 3245276 tid 3 2024-03-04T06:08:20.876 INFO:tasks.rados.rados.0.smithi064.stdout:157: delete oid 5 current snap is 8 2024-03-04T06:08:20.906 INFO:tasks.rados.rados.0.smithi064.stdout:157: done (2 left) 2024-03-04T06:08:20.906 INFO:tasks.rados.rados.0.smithi064.stdout:158: read oid 45 snap 7 2024-03-04T06:08:20.906 INFO:tasks.rados.rados.0.smithi064.stdout:158: expect (ObjNum 70 snap 3 seq_num 70) 2024-03-04T06:08:20.906 INFO:tasks.rados.rados.0.smithi064.stdout:159: read oid 5 snap -1 2024-03-04T06:08:20.906 INFO:tasks.rados.rados.0.smithi064.stdout:159: expect deleted 2024-03-04T06:08:20.906 INFO:tasks.rados.rados.0.smithi064.stdout:160: snap_create 2024-03-04T06:08:20.926 INFO:tasks.rados.rados.0.smithi064.stdout:155: expect (ObjNum 94 snap 7 seq_num 94) 2024-03-04T06:08:21.117 INFO:tasks.rados.rados.0.smithi064.stdout:156: finishing write tid 1 to smithi06431743-26 2024-03-04T06:08:21.117 INFO:tasks.rados.rados.0.smithi064.stdout:156: finishing write tid 2 to smithi06431743-26 2024-03-04T06:08:21.117 INFO:tasks.rados.rados.0.smithi064.stdout:156: finishing write tid 3 to smithi06431743-26 2024-03-04T06:08:21.117 INFO:tasks.rados.rados.0.smithi064.stdout:158: expect (ObjNum 70 snap 3 seq_num 70) 2024-03-04T06:08:21.325 INFO:tasks.rados.rados.0.smithi064.stdout:156: finishing write tid 4 to smithi06431743-26 2024-03-04T06:08:21.325 INFO:tasks.rados.rados.0.smithi064.stdout:156: finishing write tid 5 to smithi06431743-26 2024-03-04T06:08:21.325 INFO:tasks.rados.rados.0.smithi064.stdout:156: oid 26 updating version 0 to 69 2024-03-04T06:08:21.325 INFO:tasks.rados.rados.0.smithi064.stdout:156: oid 26 version 69 is already newer than 68 2024-03-04T06:08:21.326 INFO:tasks.rados.rados.0.smithi064.stdout:156: oid 26 version 69 is already newer than 66 2024-03-04T06:08:21.326 INFO:tasks.rados.rados.0.smithi064.stdout:156: oid 26 version 69 is already newer than 67 2024-03-04T06:08:21.326 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 69 (ObjNum 99 snap 8 seq_num 99) dirty exists 2024-03-04T06:08:21.326 INFO:tasks.rados.rados.0.smithi064.stdout:156: left oid 26 (ObjNum 99 snap 8 seq_num 99) 2024-03-04T06:08:21.861 INFO:tasks.rados.rados.0.smithi064.stdout:155: done (4 left) 2024-03-04T06:08:21.861 INFO:tasks.rados.rados.0.smithi064.stdout:156: done (3 left) 2024-03-04T06:08:21.861 INFO:tasks.rados.rados.0.smithi064.stdout:158: done (2 left) 2024-03-04T06:08:21.861 INFO:tasks.rados.rados.0.smithi064.stdout:159: done (1 left) 2024-03-04T06:08:21.861 INFO:tasks.rados.rados.0.smithi064.stdout:160: done (0 left) 2024-03-04T06:08:21.861 INFO:tasks.rados.rados.0.smithi064.stdout:161: snap_remove snap 8 2024-03-04T06:08:22.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:22.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:22.323+0000 7f1b3d59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:22.329 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:22.328+0000 7f1b3d59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:22.864 INFO:tasks.rados.rados.0.smithi064.stdout:161: done (0 left) 2024-03-04T06:08:22.864 INFO:tasks.rados.rados.0.smithi064.stdout:162: write (excl) oid 26 current snap is 9 2024-03-04T06:08:22.865 INFO:tasks.rados.rados.0.smithi064.stdout:162: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-03-04T06:08:22.871 INFO:tasks.rados.rados.0.smithi064.stdout:162: writing smithi06431743-26 from 497507 to 1210245 tid 1 2024-03-04T06:08:22.877 INFO:tasks.rados.rados.0.smithi064.stdout:162: writing smithi06431743-26 from 1710446 to 2372142 tid 2 2024-03-04T06:08:22.885 INFO:tasks.rados.rados.0.smithi064.stdout:162: writing smithi06431743-26 from 3140781 to 3697507 tid 3 2024-03-04T06:08:22.885 INFO:tasks.rados.rados.0.smithi064.stdout:163: snap_remove snap 7 2024-03-04T06:08:23.867 INFO:tasks.rados.rados.0.smithi064.stdout:163: done (1 left) 2024-03-04T06:08:23.868 INFO:tasks.rados.rados.0.smithi064.stdout:164: read oid 7 snap -1 2024-03-04T06:08:23.868 INFO:tasks.rados.rados.0.smithi064.stdout:164: expect (ObjNum 67 snap 3 seq_num 67) 2024-03-04T06:08:23.868 INFO:tasks.rados.rados.0.smithi064.stdout:165: delete oid 12 current snap is 9 2024-03-04T06:08:23.868 INFO:tasks.rados.rados.0.smithi064.stdout:162: finishing write tid 1 to smithi06431743-26 2024-03-04T06:08:23.868 INFO:tasks.rados.rados.0.smithi064.stdout:162: finishing write tid 2 to smithi06431743-26 2024-03-04T06:08:23.868 INFO:tasks.rados.rados.0.smithi064.stdout:162: finishing write tid 3 to smithi06431743-26 2024-03-04T06:08:23.868 INFO:tasks.rados.rados.0.smithi064.stdout:162: finishing write tid 4 to smithi06431743-26 2024-03-04T06:08:23.868 INFO:tasks.rados.rados.0.smithi064.stdout:162: finishing write tid 5 to smithi06431743-26 2024-03-04T06:08:23.869 INFO:tasks.rados.rados.0.smithi064.stdout:162: oid 26 updating version 0 to 70 2024-03-04T06:08:23.869 INFO:tasks.rados.rados.0.smithi064.stdout:162: oid 26 updating version 70 to 72 2024-03-04T06:08:23.869 INFO:tasks.rados.rados.0.smithi064.stdout:162: oid 26 version 72 is already newer than 71 2024-03-04T06:08:23.869 INFO:tasks.rados.rados.0.smithi064.stdout:162: oid 26 updating version 72 to 73 2024-03-04T06:08:23.869 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 73 (ObjNum 101 snap 9 seq_num 101) dirty exists 2024-03-04T06:08:23.870 INFO:tasks.rados.rados.0.smithi064.stdout:162: left oid 26 (ObjNum 101 snap 9 seq_num 101) 2024-03-04T06:08:23.927 INFO:tasks.rados.rados.0.smithi064.stdout:164: expect (ObjNum 67 snap 3 seq_num 67) 2024-03-04T06:08:24.082 INFO:tasks.rados.rados.0.smithi064.stdout:162: done (2 left) 2024-03-04T06:08:24.083 INFO:tasks.rados.rados.0.smithi064.stdout:164: done (1 left) 2024-03-04T06:08:24.083 INFO:tasks.rados.rados.0.smithi064.stdout:165: done (0 left) 2024-03-04T06:08:24.083 INFO:tasks.rados.rados.0.smithi064.stdout:166: delete oid 48 current snap is 9 2024-03-04T06:08:24.093 INFO:tasks.rados.rados.0.smithi064.stdout:166: done (0 left) 2024-03-04T06:08:24.093 INFO:tasks.rados.rados.0.smithi064.stdout:167: delete oid 11 current snap is 9 2024-03-04T06:08:24.135 INFO:tasks.rados.rados.0.smithi064.stdout:167: done (0 left) 2024-03-04T06:08:24.135 INFO:tasks.rados.rados.0.smithi064.stdout:168: write (excl) oid 44 current snap is 9 2024-03-04T06:08:24.135 INFO:tasks.rados.rados.0.smithi064.stdout:168: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-03-04T06:08:24.141 INFO:tasks.rados.rados.0.smithi064.stdout:168: writing smithi06431743-44 from 564735 to 1178469 tid 1 2024-03-04T06:08:24.149 INFO:tasks.rados.rados.0.smithi064.stdout:168: writing smithi06431743-44 from 1828021 to 2497055 tid 2 2024-03-04T06:08:24.156 INFO:tasks.rados.rados.0.smithi064.stdout:168: writing smithi06431743-44 from 3134120 to 3764735 tid 3 2024-03-04T06:08:24.157 INFO:tasks.rados.rados.0.smithi064.stdout:169: write oid 2 current snap is 9 2024-03-04T06:08:24.157 INFO:tasks.rados.rados.0.smithi064.stdout:168: finishing write tid 1 to smithi06431743-44 2024-03-04T06:08:24.157 INFO:tasks.rados.rados.0.smithi064.stdout:169: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-03-04T06:08:24.164 INFO:tasks.rados.rados.0.smithi064.stdout:169: writing smithi06431743-2 from 581542 to 1270525 tid 1 2024-03-04T06:08:24.170 INFO:tasks.rados.rados.0.smithi064.stdout:169: writing smithi06431743-2 from 1886503 to 2299195 tid 2 2024-03-04T06:08:24.177 INFO:tasks.rados.rados.0.smithi064.stdout:169: writing smithi06431743-2 from 2761543 to 3432494 tid 3 2024-03-04T06:08:24.177 INFO:tasks.rados.rados.0.smithi064.stdout:170: read oid 21 snap -1 2024-03-04T06:08:24.177 INFO:tasks.rados.rados.0.smithi064.stdout:170: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:08:24.177 INFO:tasks.rados.rados.0.smithi064.stdout:171: write oid 17 current snap is 9 2024-03-04T06:08:24.177 INFO:tasks.rados.rados.0.smithi064.stdout:168: finishing write tid 2 to smithi06431743-44 2024-03-04T06:08:24.177 INFO:tasks.rados.rados.0.smithi064.stdout:171: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-03-04T06:08:24.183 INFO:tasks.rados.rados.0.smithi064.stdout:171: writing smithi06431743-17 from 598349 to 1078934 tid 1 2024-03-04T06:08:24.189 INFO:tasks.rados.rados.0.smithi064.stdout:171: writing smithi06431743-17 from 1544985 to 2217688 tid 2 2024-03-04T06:08:24.196 INFO:tasks.rados.rados.0.smithi064.stdout:171: writing smithi06431743-17 from 2905319 to 3387542 tid 3 2024-03-04T06:08:24.196 INFO:tasks.rados.rados.0.smithi064.stdout:172: write (excl) oid 15 current snap is 9 2024-03-04T06:08:24.196 INFO:tasks.rados.rados.0.smithi064.stdout:169: finishing write tid 1 to smithi06431743-2 2024-03-04T06:08:24.196 INFO:tasks.rados.rados.0.smithi064.stdout:172: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-03-04T06:08:24.203 INFO:tasks.rados.rados.0.smithi064.stdout:172: writing smithi06431743-15 from 615156 to 1170990 tid 1 2024-03-04T06:08:24.207 INFO:tasks.rados.rados.0.smithi064.stdout:172: writing smithi06431743-15 from 1603467 to 2019828 tid 2 2024-03-04T06:08:24.216 INFO:tasks.rados.rados.0.smithi064.stdout:172: writing smithi06431743-15 from 2816389 to 3509884 tid 3 2024-03-04T06:08:24.216 INFO:tasks.rados.rados.0.smithi064.stdout:173: read oid 9 snap -1 2024-03-04T06:08:24.216 INFO:tasks.rados.rados.0.smithi064.stdout:173: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-04T06:08:24.216 INFO:tasks.rados.rados.0.smithi064.stdout:174: write (excl) oid 39 current snap is 9 2024-03-04T06:08:24.216 INFO:tasks.rados.rados.0.smithi064.stdout:168: finishing write tid 3 to smithi06431743-44 2024-03-04T06:08:24.216 INFO:tasks.rados.rados.0.smithi064.stdout:174: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2024-03-04T06:08:24.222 INFO:tasks.rados.rados.0.smithi064.stdout:174: writing smithi06431743-39 from 631963 to 1263046 tid 1 2024-03-04T06:08:24.230 INFO:tasks.rados.rados.0.smithi064.stdout:174: writing smithi06431743-39 from 2061949 to 2621968 tid 2 2024-03-04T06:08:24.236 INFO:tasks.rados.rados.0.smithi064.stdout:174: writing smithi06431743-39 from 3243812 to 3748579 tid 3 2024-03-04T06:08:24.237 INFO:tasks.rados.rados.0.smithi064.stdout:175: read oid 3 snap -1 2024-03-04T06:08:24.237 INFO:tasks.rados.rados.0.smithi064.stdout:175: expect (ObjNum 54 snap 0 seq_num 54) 2024-03-04T06:08:24.237 INFO:tasks.rados.rados.0.smithi064.stdout:176: write oid 35 current snap is 9 2024-03-04T06:08:24.237 INFO:tasks.rados.rados.0.smithi064.stdout:176: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-03-04T06:08:24.244 INFO:tasks.rados.rados.0.smithi064.stdout:176: writing smithi06431743-35 from 648770 to 1355102 tid 1 2024-03-04T06:08:24.252 INFO:tasks.rados.rados.0.smithi064.stdout:176: writing smithi06431743-35 from 2120431 to 2540461 tid 2 2024-03-04T06:08:24.260 INFO:tasks.rados.rados.0.smithi064.stdout:176: writing smithi06431743-35 from 3271235 to 3703627 tid 3 2024-03-04T06:08:24.260 INFO:tasks.rados.rados.0.smithi064.stdout:177: snap_create 2024-03-04T06:08:24.260 INFO:tasks.rados.rados.0.smithi064.stdout:170: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:08:24.361 INFO:tasks.rados.rados.0.smithi064.stdout:171: finishing write tid 1 to smithi06431743-17 2024-03-04T06:08:24.361 INFO:tasks.rados.rados.0.smithi064.stdout:168: finishing write tid 4 to smithi06431743-44 2024-03-04T06:08:24.361 INFO:tasks.rados.rados.0.smithi064.stdout:168: finishing write tid 5 to smithi06431743-44 2024-03-04T06:08:24.361 INFO:tasks.rados.rados.0.smithi064.stdout:168: oid 44 updating version 0 to 29 2024-03-04T06:08:24.361 INFO:tasks.rados.rados.0.smithi064.stdout:168: oid 44 updating version 29 to 31 2024-03-04T06:08:24.361 INFO:tasks.rados.rados.0.smithi064.stdout:168: oid 44 updating version 31 to 32 2024-03-04T06:08:24.361 INFO:tasks.rados.rados.0.smithi064.stdout:168: oid 44 version 32 is already newer than 30 2024-03-04T06:08:24.361 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 32 (ObjNum 105 snap 9 seq_num 105) dirty exists 2024-03-04T06:08:24.362 INFO:tasks.rados.rados.0.smithi064.stdout:168: left oid 44 (ObjNum 105 snap 9 seq_num 105) 2024-03-04T06:08:24.362 INFO:tasks.rados.rados.0.smithi064.stdout:169: finishing write tid 2 to smithi06431743-2 2024-03-04T06:08:24.362 INFO:tasks.rados.rados.0.smithi064.stdout:171: finishing write tid 2 to smithi06431743-17 2024-03-04T06:08:24.362 INFO:tasks.rados.rados.0.smithi064.stdout:169: finishing write tid 3 to smithi06431743-2 2024-03-04T06:08:24.362 INFO:tasks.rados.rados.0.smithi064.stdout:172: finishing write tid 1 to smithi06431743-15 2024-03-04T06:08:24.362 INFO:tasks.rados.rados.0.smithi064.stdout:174: finishing write tid 1 to smithi06431743-39 2024-03-04T06:08:24.362 INFO:tasks.rados.rados.0.smithi064.stdout:175: expect (ObjNum 54 snap 0 seq_num 54) 2024-03-04T06:08:24.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:24.504 INFO:tasks.rados.rados.0.smithi064.stdout:171: finishing write tid 3 to smithi06431743-17 2024-03-04T06:08:24.504 INFO:tasks.rados.rados.0.smithi064.stdout:169: finishing write tid 4 to smithi06431743-2 2024-03-04T06:08:24.504 INFO:tasks.rados.rados.0.smithi064.stdout:169: finishing write tid 5 to smithi06431743-2 2024-03-04T06:08:24.504 INFO:tasks.rados.rados.0.smithi064.stdout:169: oid 2 updating version 0 to 27 2024-03-04T06:08:24.504 INFO:tasks.rados.rados.0.smithi064.stdout:169: oid 2 version 27 is already newer than 26 2024-03-04T06:08:24.504 INFO:tasks.rados.rados.0.smithi064.stdout:169: oid 2 updating version 27 to 28 2024-03-04T06:08:24.504 INFO:tasks.rados.rados.0.smithi064.stdout:169: oid 2 version 28 is already newer than 25 2024-03-04T06:08:24.504 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 28 (ObjNum 106 snap 9 seq_num 106) dirty exists 2024-03-04T06:08:24.505 INFO:tasks.rados.rados.0.smithi064.stdout:169: left oid 2 (ObjNum 106 snap 9 seq_num 106) 2024-03-04T06:08:24.505 INFO:tasks.rados.rados.0.smithi064.stdout:173: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-04T06:08:24.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:24.512+0000 7f30bf26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:24.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:24.516+0000 7f30bf26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:24.586 INFO:tasks.rados.rados.0.smithi064.stdout:174: finishing write tid 2 to smithi06431743-39 2024-03-04T06:08:24.586 INFO:tasks.rados.rados.0.smithi064.stdout:176: finishing write tid 1 to smithi06431743-35 2024-03-04T06:08:24.586 INFO:tasks.rados.rados.0.smithi064.stdout:172: finishing write tid 2 to smithi06431743-15 2024-03-04T06:08:24.586 INFO:tasks.rados.rados.0.smithi064.stdout:171: finishing write tid 4 to smithi06431743-17 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:171: finishing write tid 5 to smithi06431743-17 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:171: oid 17 updating version 0 to 28 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:171: oid 17 version 28 is already newer than 26 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:171: oid 17 updating version 28 to 29 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:171: oid 17 version 29 is already newer than 27 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 29 (ObjNum 107 snap 9 seq_num 107) dirty exists 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:171: left oid 17 (ObjNum 107 snap 9 seq_num 107) 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:174: finishing write tid 3 to smithi06431743-39 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:172: finishing write tid 3 to smithi06431743-15 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:176: finishing write tid 2 to smithi06431743-35 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:172: finishing write tid 4 to smithi06431743-15 2024-03-04T06:08:24.587 INFO:tasks.rados.rados.0.smithi064.stdout:174: finishing write tid 4 to smithi06431743-39 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:172: finishing write tid 5 to smithi06431743-15 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:172: oid 15 updating version 0 to 21 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:172: oid 15 updating version 21 to 22 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:172: oid 15 version 22 is already newer than 20 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:172: oid 15 updating version 22 to 23 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 23 (ObjNum 108 snap 9 seq_num 108) dirty exists 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:172: left oid 15 (ObjNum 108 snap 9 seq_num 108) 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:174: finishing write tid 5 to smithi06431743-39 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:174: oid 39 updating version 0 to 31 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:174: oid 39 version 31 is already newer than 30 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:174: oid 39 updating version 31 to 32 2024-03-04T06:08:24.588 INFO:tasks.rados.rados.0.smithi064.stdout:174: oid 39 version 32 is already newer than 29 2024-03-04T06:08:24.589 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 32 (ObjNum 109 snap 9 seq_num 109) dirty exists 2024-03-04T06:08:24.589 INFO:tasks.rados.rados.0.smithi064.stdout:174: left oid 39 (ObjNum 109 snap 9 seq_num 109) 2024-03-04T06:08:24.589 INFO:tasks.rados.rados.0.smithi064.stdout:176: finishing write tid 3 to smithi06431743-35 2024-03-04T06:08:24.589 INFO:tasks.rados.rados.0.smithi064.stdout:176: finishing write tid 4 to smithi06431743-35 2024-03-04T06:08:24.589 INFO:tasks.rados.rados.0.smithi064.stdout:176: finishing write tid 5 to smithi06431743-35 2024-03-04T06:08:24.589 INFO:tasks.rados.rados.0.smithi064.stdout:176: oid 35 updating version 0 to 32 2024-03-04T06:08:24.589 INFO:tasks.rados.rados.0.smithi064.stdout:176: oid 35 updating version 32 to 33 2024-03-04T06:08:24.589 INFO:tasks.rados.rados.0.smithi064.stdout:176: oid 35 updating version 33 to 34 2024-03-04T06:08:24.589 INFO:tasks.rados.rados.0.smithi064.stdout:176: oid 35 version 34 is already newer than 31 2024-03-04T06:08:24.589 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 34 (ObjNum 110 snap 9 seq_num 110) dirty exists 2024-03-04T06:08:24.589 INFO:tasks.rados.rados.0.smithi064.stdout:176: left oid 35 (ObjNum 110 snap 9 seq_num 110) 2024-03-04T06:08:24.871 INFO:tasks.rados.rados.0.smithi064.stdout:168: done (9 left) 2024-03-04T06:08:24.872 INFO:tasks.rados.rados.0.smithi064.stdout:169: done (8 left) 2024-03-04T06:08:24.872 INFO:tasks.rados.rados.0.smithi064.stdout:170: done (7 left) 2024-03-04T06:08:24.872 INFO:tasks.rados.rados.0.smithi064.stdout:171: done (6 left) 2024-03-04T06:08:24.872 INFO:tasks.rados.rados.0.smithi064.stdout:172: done (5 left) 2024-03-04T06:08:24.872 INFO:tasks.rados.rados.0.smithi064.stdout:173: done (4 left) 2024-03-04T06:08:24.872 INFO:tasks.rados.rados.0.smithi064.stdout:174: done (3 left) 2024-03-04T06:08:24.872 INFO:tasks.rados.rados.0.smithi064.stdout:175: done (2 left) 2024-03-04T06:08:24.872 INFO:tasks.rados.rados.0.smithi064.stdout:176: done (1 left) 2024-03-04T06:08:24.872 INFO:tasks.rados.rados.0.smithi064.stdout:177: done (0 left) 2024-03-04T06:08:24.872 INFO:tasks.rados.rados.0.smithi064.stdout:178: delete oid 23 current snap is 10 2024-03-04T06:08:24.909 INFO:tasks.rados.rados.0.smithi064.stdout:178: done (0 left) 2024-03-04T06:08:24.909 INFO:tasks.rados.rados.0.smithi064.stdout:179: delete oid 36 current snap is 10 2024-03-04T06:08:24.954 INFO:tasks.rados.rados.0.smithi064.stdout:179: done (0 left) 2024-03-04T06:08:24.954 INFO:tasks.rados.rados.0.smithi064.stdout:180: write oid 41 current snap is 10 2024-03-04T06:08:24.954 INFO:tasks.rados.rados.0.smithi064.stdout:180: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-03-04T06:08:24.961 INFO:tasks.rados.rados.0.smithi064.stdout:180: writing smithi06431743-41 from 699191 to 1231270 tid 1 2024-03-04T06:08:24.968 INFO:tasks.rados.rados.0.smithi064.stdout:180: writing smithi06431743-41 from 1779524 to 2346881 tid 2 2024-03-04T06:08:24.976 INFO:tasks.rados.rados.0.smithi064.stdout:180: writing smithi06431743-41 from 2837151 to 3503359 tid 3 2024-03-04T06:08:24.976 INFO:tasks.rados.rados.0.smithi064.stdout:181: write oid 20 current snap is 10 2024-03-04T06:08:24.976 INFO:tasks.rados.rados.0.smithi064.stdout:181: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-03-04T06:08:24.985 INFO:tasks.rados.rados.0.smithi064.stdout:181: writing smithi06431743-20 from 715998 to 1323326 tid 1 2024-03-04T06:08:24.992 INFO:tasks.rados.rados.0.smithi064.stdout:181: writing smithi06431743-20 from 1838006 to 2265374 tid 2 2024-03-04T06:08:24.999 INFO:tasks.rados.rados.0.smithi064.stdout:181: writing smithi06431743-20 from 2864574 to 3342054 tid 3 2024-03-04T06:08:25.002 INFO:tasks.rados.rados.0.smithi064.stdout:181: writing smithi06431743-20 from 3863835 to 3915998 tid 4 2024-03-04T06:08:25.002 INFO:tasks.rados.rados.0.smithi064.stdout:182: snap_remove snap 9 2024-03-04T06:08:25.249 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:08:25.249 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:08:25.250 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:08:25.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:08:25.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:25.353+0000 7f60ee26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:25.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:25.360+0000 7f60ee26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:25.374 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:25.374 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:25.374 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:25.374 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:25.374 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:25.374 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:25.375 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:25.375 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:25.375 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:25.375 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:25.375 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:25.375 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:25.377 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:25.377 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:25.378 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:08:25.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:25.480+0000 7f98a595d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:25.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:25.487+0000 7f98a595d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:25.501 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:25.501 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:25.502 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:25.502 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:25.502 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:25.502 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:25.502 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:25.502 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:25.502 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:25.502 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:25.503 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:25.503 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:25.503 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:25.504 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:25.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:08:25.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:25.606+0000 7f725b71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:25.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:25.613+0000 7f725b71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:25.628 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:25.628 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:25.628 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:25.628 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:25.628 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:25.628 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:25.628 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:25.629 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:25.629 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:25.629 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:25.629 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:25.629 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:25.631 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:25.631 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:25.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:08:25.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:25.737+0000 7f348386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:25.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:25.744+0000 7f348386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:25.758 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:25.758 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:25.758 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:25.758 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:25.758 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:25.758 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:25.758 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:25.758 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:25.759 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:25.759 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:25.759 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:25.759 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:25.761 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:25.761 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:25.873 INFO:tasks.rados.rados.0.smithi064.stdout:182: done (2 left) 2024-03-04T06:08:25.873 INFO:tasks.rados.rados.0.smithi064.stdout:183: delete oid 30 current snap is 10 2024-03-04T06:08:25.873 INFO:tasks.rados.rados.0.smithi064.stdout:180: finishing write tid 1 to smithi06431743-41 2024-03-04T06:08:25.873 INFO:tasks.rados.rados.0.smithi064.stdout:180: finishing write tid 2 to smithi06431743-41 2024-03-04T06:08:25.873 INFO:tasks.rados.rados.0.smithi064.stdout:181: finishing write tid 1 to smithi06431743-20 2024-03-04T06:08:25.873 INFO:tasks.rados.rados.0.smithi064.stdout:180: finishing write tid 3 to smithi06431743-41 2024-03-04T06:08:25.873 INFO:tasks.rados.rados.0.smithi064.stdout:181: finishing write tid 2 to smithi06431743-20 2024-03-04T06:08:25.873 INFO:tasks.rados.rados.0.smithi064.stdout:180: finishing write tid 4 to smithi06431743-41 2024-03-04T06:08:25.874 INFO:tasks.rados.rados.0.smithi064.stdout:180: finishing write tid 5 to smithi06431743-41 2024-03-04T06:08:25.874 INFO:tasks.rados.rados.0.smithi064.stdout:180: oid 41 updating version 0 to 20 2024-03-04T06:08:25.874 INFO:tasks.rados.rados.0.smithi064.stdout:180: oid 41 version 20 is already newer than 20 2024-03-04T06:08:25.874 INFO:tasks.rados.rados.0.smithi064.stdout:180: oid 41 version 20 is already newer than 17 2024-03-04T06:08:25.874 INFO:tasks.rados.rados.0.smithi064.stdout:180: oid 41 version 20 is already newer than 18 2024-03-04T06:08:25.874 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 20 (ObjNum 113 snap 10 seq_num 113) dirty exists 2024-03-04T06:08:25.874 INFO:tasks.rados.rados.0.smithi064.stdout:180: left oid 41 (ObjNum 113 snap 10 seq_num 113) 2024-03-04T06:08:25.874 INFO:tasks.rados.rados.0.smithi064.stdout:181: finishing write tid 3 to smithi06431743-20 2024-03-04T06:08:25.874 INFO:tasks.rados.rados.0.smithi064.stdout:181: finishing write tid 4 to smithi06431743-20 2024-03-04T06:08:25.874 INFO:tasks.rados.rados.0.smithi064.stdout:181: finishing write tid 5 to smithi06431743-20 2024-03-04T06:08:25.874 INFO:tasks.rados.rados.0.smithi064.stdout:181: finishing write tid 6 to smithi06431743-20 2024-03-04T06:08:25.875 INFO:tasks.rados.rados.0.smithi064.stdout:181: oid 20 updating version 0 to 39 2024-03-04T06:08:25.875 INFO:tasks.rados.rados.0.smithi064.stdout:181: oid 20 version 39 is already newer than 36 2024-03-04T06:08:25.875 INFO:tasks.rados.rados.0.smithi064.stdout:181: oid 20 version 39 is already newer than 38 2024-03-04T06:08:25.875 INFO:tasks.rados.rados.0.smithi064.stdout:181: oid 20 version 39 is already newer than 38 2024-03-04T06:08:25.875 INFO:tasks.rados.rados.0.smithi064.stdout:181: oid 20 version 39 is already newer than 35 2024-03-04T06:08:25.875 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 39 (ObjNum 114 snap 10 seq_num 114) dirty exists 2024-03-04T06:08:25.875 INFO:tasks.rados.rados.0.smithi064.stdout:181: left oid 20 (ObjNum 114 snap 10 seq_num 114) 2024-03-04T06:08:25.935 INFO:tasks.rados.rados.0.smithi064.stdout:180: done (2 left) 2024-03-04T06:08:25.935 INFO:tasks.rados.rados.0.smithi064.stdout:181: done (1 left) 2024-03-04T06:08:25.935 INFO:tasks.rados.rados.0.smithi064.stdout:183: done (0 left) 2024-03-04T06:08:25.935 INFO:tasks.rados.rados.0.smithi064.stdout:184: write (excl) oid 45 current snap is 10 2024-03-04T06:08:25.935 INFO:tasks.rados.rados.0.smithi064.stdout:184: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-03-04T06:08:25.942 INFO:tasks.rados.rados.0.smithi064.stdout:184: writing smithi06431743-45 from 749612 to 1223791 tid 1 2024-03-04T06:08:25.948 INFO:tasks.rados.rados.0.smithi064.stdout:184: writing smithi06431743-45 from 1954970 to 2386007 tid 2 2024-03-04T06:08:25.955 INFO:tasks.rados.rados.0.smithi064.stdout:184: writing smithi06431743-45 from 2919420 to 3535797 tid 3 2024-03-04T06:08:25.955 INFO:tasks.rados.rados.0.smithi064.stdout:185: write (excl) oid 4 current snap is 10 2024-03-04T06:08:25.955 INFO:tasks.rados.rados.0.smithi064.stdout:185: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-03-04T06:08:25.963 INFO:tasks.rados.rados.0.smithi064.stdout:185: writing smithi06431743-4 from 766419 to 1315847 tid 1 2024-03-04T06:08:25.970 INFO:tasks.rados.rados.0.smithi064.stdout:185: writing smithi06431743-4 from 2013452 to 2588147 tid 2 2024-03-04T06:08:25.977 INFO:tasks.rados.rados.0.smithi064.stdout:185: writing smithi06431743-4 from 3346843 to 3774492 tid 3 2024-03-04T06:08:25.977 INFO:tasks.rados.rados.0.smithi064.stdout:186: delete oid 30 current snap is 10 2024-03-04T06:08:25.977 INFO:tasks.rados.rados.0.smithi064.stdout:184: finishing write tid 1 to smithi06431743-45 2024-03-04T06:08:25.983 INFO:tasks.rados.rados.0.smithi064.stdout:185: finishing write tid 1 to smithi06431743-4 2024-03-04T06:08:25.987 INFO:tasks.rados.rados.0.smithi064.stdout:184: finishing write tid 2 to smithi06431743-45 2024-03-04T06:08:25.997 INFO:tasks.rados.rados.0.smithi064.stdout:185: finishing write tid 2 to smithi06431743-4 2024-03-04T06:08:25.999 INFO:tasks.rados.rados.0.smithi064.stdout:184: finishing write tid 3 to smithi06431743-45 2024-03-04T06:08:26.004 INFO:tasks.rados.rados.0.smithi064.stdout:184: finishing write tid 4 to smithi06431743-45 2024-03-04T06:08:26.005 INFO:tasks.rados.rados.0.smithi064.stdout:184: finishing write tid 5 to smithi06431743-45 2024-03-04T06:08:26.005 INFO:tasks.rados.rados.0.smithi064.stdout:184: oid 45 updating version 0 to 35 2024-03-04T06:08:26.005 INFO:tasks.rados.rados.0.smithi064.stdout:184: oid 45 version 35 is already newer than 33 2024-03-04T06:08:26.005 INFO:tasks.rados.rados.0.smithi064.stdout:184: oid 45 updating version 35 to 36 2024-03-04T06:08:26.006 INFO:tasks.rados.rados.0.smithi064.stdout:184: oid 45 version 36 is already newer than 34 2024-03-04T06:08:26.006 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 36 (ObjNum 116 snap 10 seq_num 116) dirty exists 2024-03-04T06:08:26.006 INFO:tasks.rados.rados.0.smithi064.stdout:184: left oid 45 (ObjNum 116 snap 10 seq_num 116) 2024-03-04T06:08:26.009 INFO:tasks.rados.rados.0.smithi064.stdout:185: finishing write tid 3 to smithi06431743-4 2024-03-04T06:08:26.012 INFO:tasks.rados.rados.0.smithi064.stdout:185: finishing write tid 4 to smithi06431743-4 2024-03-04T06:08:26.014 INFO:tasks.rados.rados.0.smithi064.stdout:185: finishing write tid 5 to smithi06431743-4 2024-03-04T06:08:26.014 INFO:tasks.rados.rados.0.smithi064.stdout:185: oid 4 updating version 0 to 37 2024-03-04T06:08:26.014 INFO:tasks.rados.rados.0.smithi064.stdout:185: oid 4 version 37 is already newer than 37 2024-03-04T06:08:26.014 INFO:tasks.rados.rados.0.smithi064.stdout:185: oid 4 version 37 is already newer than 34 2024-03-04T06:08:26.014 INFO:tasks.rados.rados.0.smithi064.stdout:185: oid 4 version 37 is already newer than 35 2024-03-04T06:08:26.014 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 37 (ObjNum 117 snap 10 seq_num 117) dirty exists 2024-03-04T06:08:26.014 INFO:tasks.rados.rados.0.smithi064.stdout:185: left oid 4 (ObjNum 117 snap 10 seq_num 117) 2024-03-04T06:08:26.017 INFO:tasks.rados.rados.0.smithi064.stdout:184: done (2 left) 2024-03-04T06:08:26.017 INFO:tasks.rados.rados.0.smithi064.stdout:185: done (1 left) 2024-03-04T06:08:26.017 INFO:tasks.rados.rados.0.smithi064.stdout:186: done (0 left) 2024-03-04T06:08:26.017 INFO:tasks.rados.rados.0.smithi064.stdout:187: write (excl) oid 46 current snap is 10 2024-03-04T06:08:26.018 INFO:tasks.rados.rados.0.smithi064.stdout:187: seq_num 119 ranges {400033=699926,1614063=385970} 2024-03-04T06:08:26.023 INFO:tasks.rados.rados.0.smithi064.stdout:187: writing smithi06431743-46 from 400033 to 1099959 tid 1 2024-03-04T06:08:26.028 INFO:tasks.rados.rados.0.smithi064.stdout:187: writing smithi06431743-46 from 1614063 to 2000033 tid 2 2024-03-04T06:08:26.028 INFO:tasks.rados.rados.0.smithi064.stdout:188: write oid 8 current snap is 10 2024-03-04T06:08:26.028 INFO:tasks.rados.rados.0.smithi064.stdout:188: seq_num 120 ranges {416840=775175,1672545=344295} 2024-03-04T06:08:26.034 INFO:tasks.rados.rados.0.smithi064.stdout:188: writing smithi06431743-8 from 416840 to 1192015 tid 1 2024-03-04T06:08:26.039 INFO:tasks.rados.rados.0.smithi064.stdout:188: writing smithi06431743-8 from 1672545 to 2016840 tid 2 2024-03-04T06:08:26.039 INFO:tasks.rados.rados.0.smithi064.stdout:189: snap_create 2024-03-04T06:08:26.039 INFO:tasks.rados.rados.0.smithi064.stdout:187: finishing write tid 1 to smithi06431743-46 2024-03-04T06:08:26.054 INFO:tasks.rados.rados.0.smithi064.stdout:188: finishing write tid 1 to smithi06431743-8 2024-03-04T06:08:26.056 INFO:tasks.rados.rados.0.smithi064.stdout:187: finishing write tid 2 to smithi06431743-46 2024-03-04T06:08:26.063 INFO:tasks.rados.rados.0.smithi064.stdout:187: finishing write tid 3 to smithi06431743-46 2024-03-04T06:08:26.065 INFO:tasks.rados.rados.0.smithi064.stdout:187: finishing write tid 4 to smithi06431743-46 2024-03-04T06:08:26.065 INFO:tasks.rados.rados.0.smithi064.stdout:187: oid 46 updating version 0 to 77 2024-03-04T06:08:26.066 INFO:tasks.rados.rados.0.smithi064.stdout:187: oid 46 version 77 is already newer than 76 2024-03-04T06:08:26.066 INFO:tasks.rados.rados.0.smithi064.stdout:187: oid 46 updating version 77 to 78 2024-03-04T06:08:26.066 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 78 (ObjNum 119 snap 10 seq_num 119) dirty exists 2024-03-04T06:08:26.066 INFO:tasks.rados.rados.0.smithi064.stdout:187: left oid 46 (ObjNum 119 snap 10 seq_num 119) 2024-03-04T06:08:26.067 INFO:tasks.rados.rados.0.smithi064.stdout:188: finishing write tid 2 to smithi06431743-8 2024-03-04T06:08:26.075 INFO:tasks.rados.rados.0.smithi064.stdout:188: finishing write tid 3 to smithi06431743-8 2024-03-04T06:08:26.075 INFO:tasks.rados.rados.0.smithi064.stdout:188: finishing write tid 4 to smithi06431743-8 2024-03-04T06:08:26.075 INFO:tasks.rados.rados.0.smithi064.stdout:188: oid 8 updating version 0 to 23 2024-03-04T06:08:26.075 INFO:tasks.rados.rados.0.smithi064.stdout:188: oid 8 updating version 23 to 24 2024-03-04T06:08:26.075 INFO:tasks.rados.rados.0.smithi064.stdout:188: oid 8 version 24 is already newer than 22 2024-03-04T06:08:26.075 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 24 (ObjNum 120 snap 10 seq_num 120) dirty exists 2024-03-04T06:08:26.075 INFO:tasks.rados.rados.0.smithi064.stdout:188: left oid 8 (ObjNum 120 snap 10 seq_num 120) 2024-03-04T06:08:26.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:26.689 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:26.687+0000 7ff8566dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:26.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:26.691+0000 7ff8566dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:26.877 INFO:tasks.rados.rados.0.smithi064.stdout:187: done (2 left) 2024-03-04T06:08:26.877 INFO:tasks.rados.rados.0.smithi064.stdout:188: done (1 left) 2024-03-04T06:08:26.877 INFO:tasks.rados.rados.0.smithi064.stdout:189: done (0 left) 2024-03-04T06:08:26.877 INFO:tasks.rados.rados.0.smithi064.stdout:190: snap_create 2024-03-04T06:08:27.879 INFO:tasks.rados.rados.0.smithi064.stdout:190: done (0 left) 2024-03-04T06:08:27.879 INFO:tasks.rados.rados.0.smithi064.stdout:191: read oid 2 snap -1 2024-03-04T06:08:27.879 INFO:tasks.rados.rados.0.smithi064.stdout:191: expect (ObjNum 106 snap 9 seq_num 106) 2024-03-04T06:08:27.879 INFO:tasks.rados.rados.0.smithi064.stdout:192: snap_create 2024-03-04T06:08:27.935 INFO:tasks.rados.rados.0.smithi064.stdout:191: expect (ObjNum 106 snap 9 seq_num 106) 2024-03-04T06:08:28.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:28.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:28.870+0000 7f248159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:28.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:28.874+0000 7f248159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:28.885 INFO:tasks.rados.rados.0.smithi064.stdout:191: done (1 left) 2024-03-04T06:08:28.885 INFO:tasks.rados.rados.0.smithi064.stdout:192: done (0 left) 2024-03-04T06:08:28.885 INFO:tasks.rados.rados.0.smithi064.stdout:193: snap_create 2024-03-04T06:08:29.886 INFO:tasks.rados.rados.0.smithi064.stdout:193: done (0 left) 2024-03-04T06:08:29.886 INFO:tasks.rados.rados.0.smithi064.stdout:194: delete oid 46 current snap is 14 2024-03-04T06:08:29.942 INFO:tasks.rados.rados.0.smithi064.stdout:194: done (0 left) 2024-03-04T06:08:29.942 INFO:tasks.rados.rados.0.smithi064.stdout:195: snap_remove snap 13 2024-03-04T06:08:30.762 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:08:30.762 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:08:30.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:08:30.884 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:30.880+0000 7fc8dff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:30.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:30.889+0000 7fc8dff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:30.893 INFO:tasks.rados.rados.0.smithi064.stdout:195: done (0 left) 2024-03-04T06:08:30.893 INFO:tasks.rados.rados.0.smithi064.stdout:196: write (excl) oid 40 current snap is 14 2024-03-04T06:08:30.893 INFO:tasks.rados.rados.0.smithi064.stdout:196: seq_num 122 ranges {450454=642026,1505862=544592} 2024-03-04T06:08:30.899 INFO:tasks.rados.rados.0.smithi064.stdout:196: writing smithi06431743-40 from 450454 to 1092480 tid 1 2024-03-04T06:08:30.904 INFO:tasks.rados.rados.0.smithi064.stdout:196: writing smithi06431743-40 from 1505862 to 2050454 tid 2 2024-03-04T06:08:30.904 INFO:tasks.rados.rados.0.smithi064.stdout:197: read oid 8 snap 12 2024-03-04T06:08:30.905 INFO:tasks.rados.rados.0.smithi064.stdout:197: expect (ObjNum 120 snap 10 seq_num 120) 2024-03-04T06:08:30.905 INFO:tasks.rados.rados.0.smithi064.stdout:198: delete oid 17 current snap is 14 2024-03-04T06:08:30.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:30.951 INFO:tasks.rados.rados.0.smithi064.stdout:198: done (2 left) 2024-03-04T06:08:30.951 INFO:tasks.rados.rados.0.smithi064.stdout:199: snap_create 2024-03-04T06:08:30.956 INFO:tasks.rados.rados.0.smithi064.stdout:197: expect (ObjNum 120 snap 10 seq_num 120) 2024-03-04T06:08:31.038 INFO:tasks.rados.rados.0.smithi064.stdout:196: finishing write tid 1 to smithi06431743-40 2024-03-04T06:08:31.039 INFO:tasks.rados.rados.0.smithi064.stdout:196: finishing write tid 2 to smithi06431743-40 2024-03-04T06:08:31.039 INFO:tasks.rados.rados.0.smithi064.stdout:196: finishing write tid 3 to smithi06431743-40 2024-03-04T06:08:31.039 INFO:tasks.rados.rados.0.smithi064.stdout:196: finishing write tid 4 to smithi06431743-40 2024-03-04T06:08:31.039 INFO:tasks.rados.rados.0.smithi064.stdout:196: oid 40 updating version 0 to 35 2024-03-04T06:08:31.039 INFO:tasks.rados.rados.0.smithi064.stdout:196: oid 40 version 35 is already newer than 34 2024-03-04T06:08:31.039 INFO:tasks.rados.rados.0.smithi064.stdout:196: oid 40 updating version 35 to 36 2024-03-04T06:08:31.039 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 36 (ObjNum 122 snap 14 seq_num 122) dirty exists 2024-03-04T06:08:31.039 INFO:tasks.rados.rados.0.smithi064.stdout:196: left oid 40 (ObjNum 122 snap 14 seq_num 122) 2024-03-04T06:08:31.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:31.056+0000 7f323f6d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:31.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:31.060+0000 7f323f6d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:31.128 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:08:31.129 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:08:31.140 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-03-04T06:08:30.891432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":133801330,"num_objects":71,"num_object_clones":22,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":558,"num_read_kb":266736,"num_write":600,"num_write_kb":131273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":549,"ondisk_log_size":549,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":327732,"kb_used_data":216468,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377159628,"statfs":{"total":386547056640,"available":386211459072,"internally_reserved":0,"allocated":221663232,"data_stored":220602318,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-672214,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10334,"num_write":10,"num_write_kb":2612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015509"},"pg_stats":[{"pgid":"3.f","version":"32'32","reported_seq":111,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-04T06:08:27.912007+0000","last_change":"2024-03-04T06:08:25.908400+0000","last_active":"2024-03-04T06:08:27.912007+0000","last_peered":"2024-03-04T06:08:27.912007+0000","last_clean":"2024-03-04T06:08:27.912007+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:08:27.912007+0000","last_undegraded":"2024-03-04T06:08:27.912007+0000","last_fullsized":"2024-03-04T06:08:27.912007+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10588311,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":19,"num_read_kb":6452,"num_write":33,"num_write_kb":9910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"22'17","reported_seq":122,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.896926+0000","last_change":"2024-03-04T06:08:25.906305+0000","last_active":"2024-03-04T06:08:29.896926+0000","last_peered":"2024-03-04T06:08:29.896926+0000","last_clean":"2024-03-04T06:08:29.896926+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:08:29.896926+0000","last_undegraded":"2024-03-04T06:08:29.896926+0000","last_fullsized":"2024-03-04T06:08:29.896926+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8420208,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":21064,"num_write":22,"num_write_kb":6447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"38'36","reported_seq":112,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.998121+0000","last_change":"2024-03-04T06:08:25.915326+0000","last_active":"2024-03-04T06:08:25.998121+0000","last_peered":"2024-03-04T06:08:25.998121+0000","last_clean":"2024-03-04T06:08:25.998121+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:08:25.998121+0000","last_undegraded":"2024-03-04T06:08:25.998121+0000","last_fullsized":"2024-03-04T06:08:25.998121+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10184943,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":5333,"num_write":36,"num_write_kb":10562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"36'28","reported_seq":116,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-04T06:08:27.928091+0000","last_change":"2024-03-04T06:08:25.896430+0000","last_active":"2024-03-04T06:08:27.928091+0000","last_peered":"2024-03-04T06:08:27.928091+0000","last_clean":"2024-03-04T06:08:27.928091+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:08:27.928091+0000","last_undegraded":"2024-03-04T06:08:27.928091+0000","last_fullsized":"2024-03-04T06:08:27.928091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554664,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":26791,"num_write":32,"num_write_kb":9241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"18'3","reported_seq":86,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.892425+0000","last_change":"2024-03-04T06:08:25.905902+0000","last_active":"2024-03-04T06:08:25.892425+0000","last_peered":"2024-03-04T06:08:25.892425+0000","last_clean":"2024-03-04T06:08:25.892425+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:08:25.892425+0000","last_undegraded":"2024-03-04T06:08:25.892425+0000","last_fullsized":"2024-03-04T06:08:25.892425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504210,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":6094,"num_write":4,"num_write_kb":1201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"36'32","reported_seq":106,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-04T06:08:27.913490+0000","last_change":"2024-03-04T06:08:25.908053+0000","last_active":"2024-03-04T06:08:27.913490+0000","last_peered":"2024-03-04T06:08:27.913490+0000","last_clean":"2024-03-04T06:08:27.913490+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:08:27.913490+0000","last_undegraded":"2024-03-04T06:08:27.913490+0000","last_fullsized":"2024-03-04T06:08:27.913490+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5915998,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":5235,"num_write":28,"num_write_kb":8182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":75,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-04T06:08:27.909866+0000","last_change":"2024-03-04T06:08:25.908657+0000","last_active":"2024-03-04T06:08:27.909866+0000","last_peered":"2024-03-04T06:08:27.909866+0000","last_clean":"2024-03-04T06:08:27.909866+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:08:27.909866+0000","last_undegraded":"2024-03-04T06:08:27.909866+0000","last_fullsized":"2024-03-04T06:08:27.909866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"36'29","reported_seq":103,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.906336+0000","last_change":"2024-03-04T06:08:25.915989+0000","last_active":"2024-03-04T06:08:25.906336+0000","last_peered":"2024-03-04T06:08:25.906336+0000","last_clean":"2024-03-04T06:08:25.906336+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:08:25.906336+0000","last_undegraded":"2024-03-04T06:08:25.906336+0000","last_fullsized":"2024-03-04T06:08:25.906336+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7142909,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":10,"num_read_kb":6,"num_write":29,"num_write_kb":8825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":211,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.887475+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:08:25.887475+0000","last_peered":"2024-03-04T06:08:25.887475+0000","last_clean":"2024-03-04T06:08:25.887475+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:08:25.887475+0000","last_undegraded":"2024-03-04T06:08:25.887475+0000","last_fullsized":"2024-03-04T06:08:25.887475+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":72,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.917003+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:08:25.917003+0000","last_peered":"2024-03-04T06:08:25.917003+0000","last_clean":"2024-03-04T06:08:25.917003+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:08:25.917003+0000","last_undegraded":"2024-03-04T06:08:25.917003+0000","last_fullsized":"2024-03-04T06:08:25.917003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"38'24","reported_seq":105,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.915553+0000","last_change":"2024-03-04T06:08:25.943291+0000","last_active":"2024-03-04T06:08:25.915553+0000","last_peered":"2024-03-04T06:08:25.915553+0000","last_clean":"2024-03-04T06:08:25.915553+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:08:25.915553+0000","last_undegraded":"2024-03-04T06:08:25.915553+0000","last_fullsized":"2024-03-04T06:08:25.915553+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6806769,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":27,"num_read_kb":20997,"num_write":23,"num_write_kb":6056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-04T06:08:27.910012+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:08:27.910012+0000","last_peered":"2024-03-04T06:08:27.910012+0000","last_clean":"2024-03-04T06:08:27.910012+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:08:27.910012+0000","last_undegraded":"2024-03-04T06:08:27.910012+0000","last_fullsized":"2024-03-04T06:08:27.910012+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"36'15","reported_seq":104,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.890790+0000","last_change":"2024-03-04T06:08:25.898197+0000","last_active":"2024-03-04T06:08:25.890790+0000","last_peered":"2024-03-04T06:08:25.890790+0000","last_clean":"2024-03-04T06:08:25.890790+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:08:25.890790+0000","last_undegraded":"2024-03-04T06:08:25.890790+0000","last_fullsized":"2024-03-04T06:08:25.890790+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3126069,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":24013,"num_write":13,"num_write_kb":3612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.906934+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:08:25.906934+0000","last_peered":"2024-03-04T06:08:25.906934+0000","last_clean":"2024-03-04T06:08:25.906934+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:08:25.906934+0000","last_undegraded":"2024-03-04T06:08:25.906934+0000","last_fullsized":"2024-03-04T06:08:25.906934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"38'78","reported_seq":176,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:26.054587+0000","last_change":"2024-03-04T06:08:25.905422+0000","last_active":"2024-03-04T06:08:26.054587+0000","last_peered":"2024-03-04T06:08:26.054587+0000","last_clean":"2024-03-04T06:08:26.054587+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:08:26.054587+0000","last_undegraded":"2024-03-04T06:08:26.054587+0000","last_fullsized":"2024-03-04T06:08:26.054587+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18571570,"num_objects":11,"num_object_clones":5,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":45719,"num_write":73,"num_write_kb":21757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":74,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-04T06:08:27.897654+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:08:27.897654+0000","last_peered":"2024-03-04T06:08:27.897654+0000","last_clean":"2024-03-04T06:08:27.897654+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:08:27.897654+0000","last_undegraded":"2024-03-04T06:08:27.897654+0000","last_fullsized":"2024-03-04T06:08:27.897654+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"38'38","reported_seq":149,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.899184+0000","last_change":"2024-03-04T06:08:25.906481+0000","last_active":"2024-03-04T06:08:29.899184+0000","last_peered":"2024-03-04T06:08:29.899184+0000","last_clean":"2024-03-04T06:08:29.899184+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:08:29.899184+0000","last_undegraded":"2024-03-04T06:08:29.899184+0000","last_fullsized":"2024-03-04T06:08:29.899184+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13983292,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":23489,"num_write":46,"num_write_kb":12076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"36'23","reported_seq":102,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.919671+0000","last_change":"2024-03-04T06:08:25.925139+0000","last_active":"2024-03-04T06:08:25.919671+0000","last_peered":"2024-03-04T06:08:25.919671+0000","last_clean":"2024-03-04T06:08:25.919671+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:08:25.919671+0000","last_undegraded":"2024-03-04T06:08:25.919671+0000","last_fullsized":"2024-03-04T06:08:25.919671+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9260558,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":9235,"num_write":23,"num_write_kb":5462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.912654+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:08:29.912654+0000","last_peered":"2024-03-04T06:08:29.912654+0000","last_clean":"2024-03-04T06:08:29.912654+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:08:29.912654+0000","last_undegraded":"2024-03-04T06:08:29.912654+0000","last_fullsized":"2024-03-04T06:08:29.912654+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"38'24","reported_seq":133,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.906492+0000","last_change":"2024-03-04T06:08:25.938811+0000","last_active":"2024-03-04T06:08:29.906492+0000","last_peered":"2024-03-04T06:08:29.906492+0000","last_clean":"2024-03-04T06:08:29.906492+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:08:29.906492+0000","last_undegraded":"2024-03-04T06:08:29.906492+0000","last_fullsized":"2024-03-04T06:08:29.906492+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4268945,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":26605,"num_write":27,"num_write_kb":6765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":78,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.893380+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:08:25.893380+0000","last_peered":"2024-03-04T06:08:25.893380+0000","last_clean":"2024-03-04T06:08:25.893380+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:08:25.893380+0000","last_undegraded":"2024-03-04T06:08:25.893380+0000","last_fullsized":"2024-03-04T06:08:25.893380+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"37'39","reported_seq":125,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.883618+0000","last_change":"2024-03-04T06:08:25.897766+0000","last_active":"2024-03-04T06:08:25.883618+0000","last_peered":"2024-03-04T06:08:25.883618+0000","last_clean":"2024-03-04T06:08:25.883618+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:08:25.883618+0000","last_undegraded":"2024-03-04T06:08:25.883618+0000","last_fullsized":"2024-03-04T06:08:25.883618+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13697573,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":22,"num_read_kb":4971,"num_write":43,"num_write_kb":10774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.900531+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:08:29.900531+0000","last_peered":"2024-03-04T06:08:29.900531+0000","last_clean":"2024-03-04T06:08:29.900531+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:08:29.900531+0000","last_undegraded":"2024-03-04T06:08:29.900531+0000","last_fullsized":"2024-03-04T06:08:29.900531+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"32'34","reported_seq":146,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.902666+0000","last_change":"2024-03-04T06:08:25.905105+0000","last_active":"2024-03-04T06:08:29.902666+0000","last_peered":"2024-03-04T06:08:29.902666+0000","last_clean":"2024-03-04T06:08:29.902666+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:08:29.902666+0000","last_undegraded":"2024-03-04T06:08:29.902666+0000","last_fullsized":"2024-03-04T06:08:29.902666+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12184943,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":40641,"num_write":31,"num_write_kb":8509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":72,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-04T06:08:25.917808+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:08:25.917808+0000","last_peered":"2024-03-04T06:08:25.917808+0000","last_clean":"2024-03-04T06:08:25.917808+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:08:25.917808+0000","last_undegraded":"2024-03-04T06:08:25.917808+0000","last_fullsized":"2024-03-04T06:08:25.917808+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":133210962,"num_objects":69,"num_object_clones":22,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":455,"num_read_kb":266645,"num_write":463,"num_write_kb":129379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96580,"kb_used_data":68156,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94275260,"statfs":{"total":96636764160,"available":96537866240,"internally_reserved":0,"allocated":69791744,"data_stored":69544668,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83924,"kb_used_data":56284,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94287916,"statfs":{"total":96636764160,"available":96550825984,"internally_reserved":0,"allocated":57634816,"data_stored":56926228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76012,"kb_used_data":49188,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94295828,"statfs":{"total":96636764160,"available":96558927872,"internally_reserved":0,"allocated":50368512,"data_stored":50273042,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71216,"kb_used_data":42840,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94300624,"statfs":{"total":96636764160,"available":96563838976,"internally_reserved":0,"allocated":43868160,"data_stored":43858380,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:08:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:196: done (2 left) 2024-03-04T06:08:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:197: done (1 left) 2024-03-04T06:08:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:199: done (0 left) 2024-03-04T06:08:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:200: write (excl) oid 33 current snap is 15 2024-03-04T06:08:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:200: seq_num 124 ranges {484068=792524,1906473=177595} 2024-03-04T06:08:31.908 INFO:tasks.rados.rados.0.smithi064.stdout:200: writing smithi06431743-33 from 484068 to 1276592 tid 1 2024-03-04T06:08:31.912 INFO:tasks.rados.rados.0.smithi064.stdout:200: writing smithi06431743-33 from 1906473 to 2084068 tid 2 2024-03-04T06:08:31.912 INFO:tasks.rados.rados.0.smithi064.stdout:201: read oid 46 snap 12 2024-03-04T06:08:31.912 INFO:tasks.rados.rados.0.smithi064.stdout:201: expect (ObjNum 119 snap 10 seq_num 119) 2024-03-04T06:08:31.913 INFO:tasks.rados.rados.0.smithi064.stdout:202: delete oid 19 current snap is 15 2024-03-04T06:08:31.969 INFO:tasks.rados.rados.0.smithi064.stdout:201: expect (ObjNum 119 snap 10 seq_num 119) 2024-03-04T06:08:32.088 INFO:tasks.rados.rados.0.smithi064.stdout:200: finishing write tid 1 to smithi06431743-33 2024-03-04T06:08:32.088 INFO:tasks.rados.rados.0.smithi064.stdout:200: finishing write tid 2 to smithi06431743-33 2024-03-04T06:08:32.088 INFO:tasks.rados.rados.0.smithi064.stdout:200: finishing write tid 3 to smithi06431743-33 2024-03-04T06:08:32.088 INFO:tasks.rados.rados.0.smithi064.stdout:200: finishing write tid 4 to smithi06431743-33 2024-03-04T06:08:32.088 INFO:tasks.rados.rados.0.smithi064.stdout:200: oid 33 updating version 0 to 38 2024-03-04T06:08:32.088 INFO:tasks.rados.rados.0.smithi064.stdout:200: oid 33 version 38 is already newer than 36 2024-03-04T06:08:32.088 INFO:tasks.rados.rados.0.smithi064.stdout:200: oid 33 version 38 is already newer than 37 2024-03-04T06:08:32.089 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 38 (ObjNum 124 snap 15 seq_num 124) dirty exists 2024-03-04T06:08:32.089 INFO:tasks.rados.rados.0.smithi064.stdout:200: left oid 33 (ObjNum 124 snap 15 seq_num 124) 2024-03-04T06:08:32.089 INFO:tasks.rados.rados.0.smithi064.stdout:200: done (2 left) 2024-03-04T06:08:32.089 INFO:tasks.rados.rados.0.smithi064.stdout:201: done (1 left) 2024-03-04T06:08:32.089 INFO:tasks.rados.rados.0.smithi064.stdout:202: done (0 left) 2024-03-04T06:08:32.089 INFO:tasks.rados.rados.0.smithi064.stdout:203: snap_remove snap 14 2024-03-04T06:08:32.910 INFO:tasks.rados.rados.0.smithi064.stdout:203: done (0 left) 2024-03-04T06:08:32.911 INFO:tasks.rados.rados.0.smithi064.stdout:204: read oid 16 snap -1 2024-03-04T06:08:32.911 INFO:tasks.rados.rados.0.smithi064.stdout:204: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-04T06:08:32.911 INFO:tasks.rados.rados.0.smithi064.stdout:205: write (excl) oid 7 current snap is 15 2024-03-04T06:08:32.911 INFO:tasks.rados.rados.0.smithi064.stdout:205: seq_num 126 ranges {517682=543022,1623437=494245} 2024-03-04T06:08:32.916 INFO:tasks.rados.rados.0.smithi064.stdout:205: writing smithi06431743-7 from 517682 to 1060704 tid 1 2024-03-04T06:08:32.922 INFO:tasks.rados.rados.0.smithi064.stdout:205: writing smithi06431743-7 from 1623437 to 2117682 tid 2 2024-03-04T06:08:32.923 INFO:tasks.rados.rados.0.smithi064.stdout:206: read oid 21 snap -1 2024-03-04T06:08:32.923 INFO:tasks.rados.rados.0.smithi064.stdout:206: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:08:32.923 INFO:tasks.rados.rados.0.smithi064.stdout:207: snap_remove snap 12 2024-03-04T06:08:33.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:33.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:33.241+0000 7fd822806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:33.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:33.246+0000 7fd822806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:33.910 INFO:tasks.rados.rados.0.smithi064.stdout:207: done (3 left) 2024-03-04T06:08:33.910 INFO:tasks.rados.rados.0.smithi064.stdout:208: read oid 20 snap -1 2024-03-04T06:08:33.910 INFO:tasks.rados.rados.0.smithi064.stdout:208: expect (ObjNum 114 snap 10 seq_num 114) 2024-03-04T06:08:33.910 INFO:tasks.rados.rados.0.smithi064.stdout:209: write (excl) oid 34 current snap is 15 2024-03-04T06:08:33.910 INFO:tasks.rados.rados.0.smithi064.stdout:209: seq_num 127 ranges {534489=618271,1565566=568923} 2024-03-04T06:08:33.918 INFO:tasks.rados.rados.0.smithi064.stdout:209: writing smithi06431743-34 from 534489 to 1152760 tid 1 2024-03-04T06:08:33.924 INFO:tasks.rados.rados.0.smithi064.stdout:209: writing smithi06431743-34 from 1565566 to 2134489 tid 2 2024-03-04T06:08:33.924 INFO:tasks.rados.rados.0.smithi064.stdout:210: read oid 22 snap -1 2024-03-04T06:08:33.924 INFO:tasks.rados.rados.0.smithi064.stdout:210: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-04T06:08:33.924 INFO:tasks.rados.rados.0.smithi064.stdout:211: delete oid 48 current snap is 15 2024-03-04T06:08:33.924 INFO:tasks.rados.rados.0.smithi064.stdout:206: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:08:34.023 INFO:tasks.rados.rados.0.smithi064.stdout:204: expect (ObjNum 15 snap 0 seq_num 15) 2024-03-04T06:08:34.109 INFO:tasks.rados.rados.0.smithi064.stdout:205: finishing write tid 1 to smithi06431743-7 2024-03-04T06:08:34.109 INFO:tasks.rados.rados.0.smithi064.stdout:205: finishing write tid 2 to smithi06431743-7 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:205: finishing write tid 3 to smithi06431743-7 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:205: finishing write tid 4 to smithi06431743-7 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:205: oid 7 updating version 0 to 19 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:205: oid 7 version 19 is already newer than 19 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:205: oid 7 version 19 is already newer than 17 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 19 (ObjNum 126 snap 15 seq_num 126) dirty exists 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:205: left oid 7 (ObjNum 126 snap 15 seq_num 126) 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:204: done (6 left) 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:205: done (5 left) 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:206: done (4 left) 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:211: done (3 left) 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:212: read oid 1 snap -1 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:212: expect (ObjNum 86 snap 6 seq_num 86) 2024-03-04T06:08:34.110 INFO:tasks.rados.rados.0.smithi064.stdout:213: write oid 15 current snap is 15 2024-03-04T06:08:34.111 INFO:tasks.rados.rados.0.smithi064.stdout:213: seq_num 129 ranges {568103=768769,2082530=85573} 2024-03-04T06:08:34.116 INFO:tasks.rados.rados.0.smithi064.stdout:213: writing smithi06431743-15 from 568103 to 1336872 tid 1 2024-03-04T06:08:34.121 INFO:tasks.rados.rados.0.smithi064.stdout:213: writing smithi06431743-15 from 2082530 to 2168103 tid 2 2024-03-04T06:08:34.121 INFO:tasks.rados.rados.0.smithi064.stdout:214: write oid 25 current snap is 15 2024-03-04T06:08:34.121 INFO:tasks.rados.rados.0.smithi064.stdout:210: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-04T06:08:34.283 INFO:tasks.rados.rados.0.smithi064.stdout:208: expect (ObjNum 114 snap 10 seq_num 114) 2024-03-04T06:08:34.432 INFO:tasks.rados.rados.0.smithi064.stdout:209: finishing write tid 1 to smithi06431743-34 2024-03-04T06:08:34.432 INFO:tasks.rados.rados.0.smithi064.stdout:209: finishing write tid 2 to smithi06431743-34 2024-03-04T06:08:34.432 INFO:tasks.rados.rados.0.smithi064.stdout:209: finishing write tid 3 to smithi06431743-34 2024-03-04T06:08:34.432 INFO:tasks.rados.rados.0.smithi064.stdout:209: finishing write tid 4 to smithi06431743-34 2024-03-04T06:08:34.432 INFO:tasks.rados.rados.0.smithi064.stdout:209: oid 34 updating version 0 to 88 2024-03-04T06:08:34.432 INFO:tasks.rados.rados.0.smithi064.stdout:209: oid 34 version 88 is already newer than 87 2024-03-04T06:08:34.432 INFO:tasks.rados.rados.0.smithi064.stdout:209: oid 34 version 88 is already newer than 86 2024-03-04T06:08:34.432 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 88 (ObjNum 127 snap 15 seq_num 127) dirty exists 2024-03-04T06:08:34.432 INFO:tasks.rados.rados.0.smithi064.stdout:209: left oid 34 (ObjNum 127 snap 15 seq_num 127) 2024-03-04T06:08:34.432 INFO:tasks.rados.rados.0.smithi064.stdout:214: seq_num 130 ranges {584910=560371,1857365=327545} 2024-03-04T06:08:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:214: writing smithi06431743-25 from 584910 to 1145281 tid 1 2024-03-04T06:08:34.444 INFO:tasks.rados.rados.0.smithi064.stdout:214: writing smithi06431743-25 from 1857365 to 2184910 tid 2 2024-03-04T06:08:34.444 INFO:tasks.rados.rados.0.smithi064.stdout:208: done (5 left) 2024-03-04T06:08:34.444 INFO:tasks.rados.rados.0.smithi064.stdout:209: done (4 left) 2024-03-04T06:08:34.444 INFO:tasks.rados.rados.0.smithi064.stdout:210: done (3 left) 2024-03-04T06:08:34.444 INFO:tasks.rados.rados.0.smithi064.stdout:215: snap_remove snap 10 2024-03-04T06:08:34.913 INFO:tasks.rados.rados.0.smithi064.stdout:215: done (3 left) 2024-03-04T06:08:34.913 INFO:tasks.rados.rados.0.smithi064.stdout:216: delete oid 14 current snap is 15 2024-03-04T06:08:34.913 INFO:tasks.rados.rados.0.smithi064.stdout:212: expect (ObjNum 86 snap 6 seq_num 86) 2024-03-04T06:08:35.041 INFO:tasks.rados.rados.0.smithi064.stdout:213: finishing write tid 1 to smithi06431743-15 2024-03-04T06:08:35.041 INFO:tasks.rados.rados.0.smithi064.stdout:213: finishing write tid 2 to smithi06431743-15 2024-03-04T06:08:35.041 INFO:tasks.rados.rados.0.smithi064.stdout:213: finishing write tid 3 to smithi06431743-15 2024-03-04T06:08:35.041 INFO:tasks.rados.rados.0.smithi064.stdout:213: finishing write tid 4 to smithi06431743-15 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:213: oid 15 updating version 0 to 26 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:213: oid 15 version 26 is already newer than 25 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:213: oid 15 updating version 26 to 27 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 27 (ObjNum 129 snap 15 seq_num 129) dirty exists 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:213: left oid 15 (ObjNum 129 snap 15 seq_num 129) 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:212: done (3 left) 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:213: done (2 left) 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:216: done (1 left) 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:217: write (excl) oid 1 current snap is 15 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:214: finishing write tid 1 to smithi06431743-25 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:214: finishing write tid 2 to smithi06431743-25 2024-03-04T06:08:35.042 INFO:tasks.rados.rados.0.smithi064.stdout:217: seq_num 132 ranges {618524=710869,1857976=360548} 2024-03-04T06:08:35.048 INFO:tasks.rados.rados.0.smithi064.stdout:217: writing smithi06431743-1 from 618524 to 1329393 tid 1 2024-03-04T06:08:35.053 INFO:tasks.rados.rados.0.smithi064.stdout:217: writing smithi06431743-1 from 1857976 to 2218524 tid 2 2024-03-04T06:08:35.053 INFO:tasks.rados.rados.0.smithi064.stdout:218: delete oid 12 current snap is 15 2024-03-04T06:08:35.053 INFO:tasks.rados.rados.0.smithi064.stdout:214: finishing write tid 3 to smithi06431743-25 2024-03-04T06:08:35.054 INFO:tasks.rados.rados.0.smithi064.stdout:214: finishing write tid 4 to smithi06431743-25 2024-03-04T06:08:35.054 INFO:tasks.rados.rados.0.smithi064.stdout:214: oid 25 updating version 0 to 39 2024-03-04T06:08:35.054 INFO:tasks.rados.rados.0.smithi064.stdout:214: oid 25 version 39 is already newer than 38 2024-03-04T06:08:35.054 INFO:tasks.rados.rados.0.smithi064.stdout:214: oid 25 updating version 39 to 40 2024-03-04T06:08:35.054 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 40 (ObjNum 130 snap 15 seq_num 130) dirty exists 2024-03-04T06:08:35.054 INFO:tasks.rados.rados.0.smithi064.stdout:214: left oid 25 (ObjNum 130 snap 15 seq_num 130) 2024-03-04T06:08:35.074 INFO:tasks.rados.rados.0.smithi064.stdout:217: finishing write tid 1 to smithi06431743-1 2024-03-04T06:08:35.081 INFO:tasks.rados.rados.0.smithi064.stdout:214: done (2 left) 2024-03-04T06:08:35.082 INFO:tasks.rados.rados.0.smithi064.stdout:218: done (1 left) 2024-03-04T06:08:35.082 INFO:tasks.rados.rados.0.smithi064.stdout:219: read oid 17 snap -1 2024-03-04T06:08:35.082 INFO:tasks.rados.rados.0.smithi064.stdout:219: expect deleted 2024-03-04T06:08:35.082 INFO:tasks.rados.rados.0.smithi064.stdout:220: write (excl) oid 16 current snap is 15 2024-03-04T06:08:35.082 INFO:tasks.rados.rados.0.smithi064.stdout:220: seq_num 134 ranges {652138=461367,1574940=677198} 2024-03-04T06:08:35.088 INFO:tasks.rados.rados.0.smithi064.stdout:220: writing smithi06431743-16 from 652138 to 1113505 tid 1 2024-03-04T06:08:35.095 INFO:tasks.rados.rados.0.smithi064.stdout:220: writing smithi06431743-16 from 1574940 to 2252138 tid 2 2024-03-04T06:08:35.095 INFO:tasks.rados.rados.0.smithi064.stdout:221: snap_create 2024-03-04T06:08:35.095 INFO:tasks.rados.rados.0.smithi064.stdout:217: finishing write tid 2 to smithi06431743-1 2024-03-04T06:08:35.110 INFO:tasks.rados.rados.0.smithi064.stdout:217: finishing write tid 3 to smithi06431743-1 2024-03-04T06:08:35.111 INFO:tasks.rados.rados.0.smithi064.stdout:217: finishing write tid 4 to smithi06431743-1 2024-03-04T06:08:35.111 INFO:tasks.rados.rados.0.smithi064.stdout:217: oid 1 updating version 0 to 94 2024-03-04T06:08:35.111 INFO:tasks.rados.rados.0.smithi064.stdout:217: oid 1 updating version 94 to 95 2024-03-04T06:08:35.111 INFO:tasks.rados.rados.0.smithi064.stdout:217: oid 1 updating version 95 to 96 2024-03-04T06:08:35.111 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 96 (ObjNum 132 snap 15 seq_num 132) dirty exists 2024-03-04T06:08:35.111 INFO:tasks.rados.rados.0.smithi064.stdout:217: left oid 1 (ObjNum 132 snap 15 seq_num 132) 2024-03-04T06:08:35.144 INFO:tasks.rados.rados.0.smithi064.stdout:220: finishing write tid 1 to smithi06431743-16 2024-03-04T06:08:35.157 INFO:tasks.rados.rados.0.smithi064.stdout:220: finishing write tid 2 to smithi06431743-16 2024-03-04T06:08:35.159 INFO:tasks.rados.rados.0.smithi064.stdout:220: finishing write tid 3 to smithi06431743-16 2024-03-04T06:08:35.160 INFO:tasks.rados.rados.0.smithi064.stdout:220: finishing write tid 4 to smithi06431743-16 2024-03-04T06:08:35.160 INFO:tasks.rados.rados.0.smithi064.stdout:220: oid 16 updating version 0 to 26 2024-03-04T06:08:35.161 INFO:tasks.rados.rados.0.smithi064.stdout:220: oid 16 updating version 26 to 28 2024-03-04T06:08:35.161 INFO:tasks.rados.rados.0.smithi064.stdout:220: oid 16 version 28 is already newer than 28 2024-03-04T06:08:35.161 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 28 (ObjNum 134 snap 15 seq_num 134) dirty exists 2024-03-04T06:08:35.161 INFO:tasks.rados.rados.0.smithi064.stdout:220: left oid 16 (ObjNum 134 snap 15 seq_num 134) 2024-03-04T06:08:35.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:35.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:35.420+0000 7f7cf2cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:35.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:35.424+0000 7f7cf2cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:35.914 INFO:tasks.rados.rados.0.smithi064.stdout:217: done (3 left) 2024-03-04T06:08:35.915 INFO:tasks.rados.rados.0.smithi064.stdout:219: done (2 left) 2024-03-04T06:08:35.915 INFO:tasks.rados.rados.0.smithi064.stdout:220: done (1 left) 2024-03-04T06:08:35.915 INFO:tasks.rados.rados.0.smithi064.stdout:221: done (0 left) 2024-03-04T06:08:35.915 INFO:tasks.rados.rados.0.smithi064.stdout:222: snap_create 2024-03-04T06:08:36.142 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:08:36.142 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:08:36.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:08:36.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:36.250+0000 7f14f6e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:36.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:36.257+0000 7f14f6e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:36.451 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:08:36.451 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:08:36.463 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-03-04T06:08:35.913732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121986174,"num_objects":77,"num_object_clones":28,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":584,"num_read_kb":279117,"num_write":615,"num_write_kb":134493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":579,"ondisk_log_size":579,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":331412,"kb_used_data":224116,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377155948,"statfs":{"total":386547056640,"available":386207690752,"internally_reserved":0,"allocated":229494784,"data_stored":230211353,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6016774,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":12381,"num_write":13,"num_write_kb":3220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013677"},"pg_stats":[{"pgid":"3.f","version":"47'40","reported_seq":138,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.073632+0000","last_change":"2024-03-04T06:08:35.094780+0000","last_active":"2024-03-04T06:08:35.073632+0000","last_peered":"2024-03-04T06:08:35.073632+0000","last_clean":"2024-03-04T06:08:35.073632+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:08:35.073632+0000","last_undegraded":"2024-03-04T06:08:35.073632+0000","last_fullsized":"2024-03-04T06:08:35.073632+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8991679,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":21,"num_read_kb":6453,"num_write":37,"num_write_kb":11069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"22'17","reported_seq":141,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.107673+0000","last_change":"2024-03-04T06:08:35.117270+0000","last_active":"2024-03-04T06:08:35.107673+0000","last_peered":"2024-03-04T06:08:35.107673+0000","last_clean":"2024-03-04T06:08:35.107673+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:08:35.107673+0000","last_undegraded":"2024-03-04T06:08:35.107673+0000","last_fullsized":"2024-03-04T06:08:35.107673+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8420208,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":21064,"num_write":22,"num_write_kb":6447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"38'36","reported_seq":124,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T06:08:30.902058+0000","last_change":"2024-03-04T06:08:30.911431+0000","last_active":"2024-03-04T06:08:30.902058+0000","last_peered":"2024-03-04T06:08:30.902058+0000","last_clean":"2024-03-04T06:08:30.902058+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:08:30.902058+0000","last_undegraded":"2024-03-04T06:08:30.902058+0000","last_fullsized":"2024-03-04T06:08:30.902058+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10184943,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":5333,"num_write":36,"num_write_kb":10562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"36'28","reported_seq":139,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.067349+0000","last_change":"2024-03-04T06:08:35.069945+0000","last_active":"2024-03-04T06:08:35.067349+0000","last_peered":"2024-03-04T06:08:35.067349+0000","last_clean":"2024-03-04T06:08:35.067349+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:08:35.067349+0000","last_undegraded":"2024-03-04T06:08:35.067349+0000","last_fullsized":"2024-03-04T06:08:35.067349+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554664,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":26791,"num_write":32,"num_write_kb":9241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"18'3","reported_seq":94,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.918323+0000","last_change":"2024-03-04T06:08:25.905902+0000","last_active":"2024-03-04T06:08:29.918323+0000","last_peered":"2024-03-04T06:08:29.918323+0000","last_clean":"2024-03-04T06:08:29.918323+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:08:29.918323+0000","last_undegraded":"2024-03-04T06:08:29.918323+0000","last_fullsized":"2024-03-04T06:08:29.918323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504210,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":6094,"num_write":4,"num_write_kb":1201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"36'32","reported_seq":129,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.071269+0000","last_change":"2024-03-04T06:08:35.078028+0000","last_active":"2024-03-04T06:08:35.071269+0000","last_peered":"2024-03-04T06:08:35.071269+0000","last_clean":"2024-03-04T06:08:35.071269+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:08:35.071269+0000","last_undegraded":"2024-03-04T06:08:35.071269+0000","last_fullsized":"2024-03-04T06:08:35.071269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5915998,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":5235,"num_write":28,"num_write_kb":8182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":98,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.070446+0000","last_change":"2024-03-04T06:08:35.077516+0000","last_active":"2024-03-04T06:08:35.070446+0000","last_peered":"2024-03-04T06:08:35.070446+0000","last_clean":"2024-03-04T06:08:35.070446+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:08:35.070446+0000","last_undegraded":"2024-03-04T06:08:35.070446+0000","last_fullsized":"2024-03-04T06:08:35.070446+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"36'29","reported_seq":116,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T06:08:30.911335+0000","last_change":"2024-03-04T06:08:30.911335+0000","last_active":"2024-03-04T06:08:30.911335+0000","last_peered":"2024-03-04T06:08:30.911335+0000","last_clean":"2024-03-04T06:08:30.911335+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:08:30.911335+0000","last_undegraded":"2024-03-04T06:08:30.911335+0000","last_fullsized":"2024-03-04T06:08:30.911335+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3344560,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":10,"num_read_kb":6,"num_write":30,"num_write_kb":8825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":219,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.911308+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:08:29.911308+0000","last_peered":"2024-03-04T06:08:29.911308+0000","last_clean":"2024-03-04T06:08:29.911308+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:08:29.911308+0000","last_undegraded":"2024-03-04T06:08:29.911308+0000","last_fullsized":"2024-03-04T06:08:29.911308+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":83,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T06:08:30.909448+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:08:30.909448+0000","last_peered":"2024-03-04T06:08:30.909448+0000","last_clean":"2024-03-04T06:08:30.909448+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:08:30.909448+0000","last_undegraded":"2024-03-04T06:08:30.909448+0000","last_fullsized":"2024-03-04T06:08:30.909448+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"38'24","reported_seq":117,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T06:08:30.917201+0000","last_change":"2024-03-04T06:08:30.923963+0000","last_active":"2024-03-04T06:08:30.917201+0000","last_peered":"2024-03-04T06:08:30.917201+0000","last_clean":"2024-03-04T06:08:30.917201+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:08:30.917201+0000","last_undegraded":"2024-03-04T06:08:30.917201+0000","last_fullsized":"2024-03-04T06:08:30.917201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6806769,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":27,"num_read_kb":20997,"num_write":23,"num_write_kb":6056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":93,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.071227+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:08:35.071227+0000","last_peered":"2024-03-04T06:08:35.071227+0000","last_clean":"2024-03-04T06:08:35.071227+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:08:35.071227+0000","last_undegraded":"2024-03-04T06:08:35.071227+0000","last_fullsized":"2024-03-04T06:08:35.071227+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"36'15","reported_seq":112,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.910288+0000","last_change":"2024-03-04T06:08:25.898197+0000","last_active":"2024-03-04T06:08:29.910288+0000","last_peered":"2024-03-04T06:08:29.910288+0000","last_clean":"2024-03-04T06:08:29.910288+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:08:29.910288+0000","last_undegraded":"2024-03-04T06:08:29.910288+0000","last_fullsized":"2024-03-04T06:08:29.910288+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3126069,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":24013,"num_write":13,"num_write_kb":3612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":83,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T06:08:30.903336+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:08:30.903336+0000","last_peered":"2024-03-04T06:08:30.903336+0000","last_clean":"2024-03-04T06:08:30.903336+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:08:30.903336+0000","last_undegraded":"2024-03-04T06:08:30.903336+0000","last_fullsized":"2024-03-04T06:08:30.903336+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"38'78","reported_seq":185,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.921886+0000","last_change":"2024-03-04T06:08:25.905422+0000","last_active":"2024-03-04T06:08:29.921886+0000","last_peered":"2024-03-04T06:08:29.921886+0000","last_clean":"2024-03-04T06:08:29.921886+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:08:29.921886+0000","last_undegraded":"2024-03-04T06:08:29.921886+0000","last_fullsized":"2024-03-04T06:08:29.921886+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16571537,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":78,"num_read_kb":45719,"num_write":74,"num_write_kb":21757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":93,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.066588+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:08:35.066588+0000","last_peered":"2024-03-04T06:08:35.066588+0000","last_clean":"2024-03-04T06:08:35.066588+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:08:35.066588+0000","last_undegraded":"2024-03-04T06:08:35.066588+0000","last_fullsized":"2024-03-04T06:08:35.066588+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"47'46","reported_seq":169,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.101428+0000","last_change":"2024-03-04T06:08:35.139954+0000","last_active":"2024-03-04T06:08:35.101428+0000","last_peered":"2024-03-04T06:08:35.101428+0000","last_clean":"2024-03-04T06:08:35.101428+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:08:35.101428+0000","last_undegraded":"2024-03-04T06:08:35.101428+0000","last_fullsized":"2024-03-04T06:08:35.101428+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11058907,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":45,"num_read_kb":23489,"num_write":47,"num_write_kb":12076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"36'23","reported_seq":114,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T06:08:30.919974+0000","last_change":"2024-03-04T06:08:30.925738+0000","last_active":"2024-03-04T06:08:30.919974+0000","last_peered":"2024-03-04T06:08:30.919974+0000","last_clean":"2024-03-04T06:08:30.919974+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:08:30.919974+0000","last_undegraded":"2024-03-04T06:08:30.919974+0000","last_fullsized":"2024-03-04T06:08:30.919974+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9260558,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":9235,"num_write":23,"num_write_kb":5462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":105,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.115773+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:08:35.115773+0000","last_peered":"2024-03-04T06:08:35.115773+0000","last_clean":"2024-03-04T06:08:35.115773+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:08:35.115773+0000","last_undegraded":"2024-03-04T06:08:35.115773+0000","last_fullsized":"2024-03-04T06:08:35.115773+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"47'28","reported_seq":162,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.147797+0000","last_change":"2024-03-04T06:08:35.118583+0000","last_active":"2024-03-04T06:08:35.147797+0000","last_peered":"2024-03-04T06:08:35.147797+0000","last_clean":"2024-03-04T06:08:35.147797+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:08:35.147797+0000","last_undegraded":"2024-03-04T06:08:35.147797+0000","last_fullsized":"2024-03-04T06:08:35.147797+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4268978,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":55,"num_read_kb":33398,"num_write":31,"num_write_kb":7878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":86,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.919073+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:08:29.919073+0000","last_peered":"2024-03-04T06:08:29.919073+0000","last_clean":"2024-03-04T06:08:29.919073+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:08:29.919073+0000","last_undegraded":"2024-03-04T06:08:29.919073+0000","last_fullsized":"2024-03-04T06:08:29.919073+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"37'39","reported_seq":133,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-03-04T06:08:29.909519+0000","last_change":"2024-03-04T06:08:25.897766+0000","last_active":"2024-03-04T06:08:29.909519+0000","last_peered":"2024-03-04T06:08:29.909519+0000","last_clean":"2024-03-04T06:08:29.909519+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:08:29.909519+0000","last_undegraded":"2024-03-04T06:08:29.909519+0000","last_fullsized":"2024-03-04T06:08:29.909519+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13697573,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":22,"num_read_kb":4971,"num_write":43,"num_write_kb":10774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":105,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.106046+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:08:35.106046+0000","last_peered":"2024-03-04T06:08:35.106046+0000","last_clean":"2024-03-04T06:08:35.106046+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:08:35.106046+0000","last_undegraded":"2024-03-04T06:08:35.106046+0000","last_fullsized":"2024-03-04T06:08:35.106046+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"47'44","reported_seq":172,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-04T06:08:35.099835+0000","last_change":"2024-03-04T06:08:35.137489+0000","last_active":"2024-03-04T06:08:35.099835+0000","last_peered":"2024-03-04T06:08:35.099835+0000","last_clean":"2024-03-04T06:08:35.099835+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:08:35.099835+0000","last_undegraded":"2024-03-04T06:08:35.099835+0000","last_fullsized":"2024-03-04T06:08:35.099835+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10689153,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":46228,"num_write":35,"num_write_kb":9457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":83,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-04T06:08:30.917941+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:08:30.917941+0000","last_peered":"2024-03-04T06:08:30.917941+0000","last_clean":"2024-03-04T06:08:30.917941+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:08:30.917941+0000","last_undegraded":"2024-03-04T06:08:30.917941+0000","last_fullsized":"2024-03-04T06:08:30.917941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":121395806,"num_objects":75,"num_object_clones":28,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":481,"num_read_kb":279026,"num_write":478,"num_write_kb":132599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":482,"ondisk_log_size":482,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96180,"kb_used_data":69356,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94275660,"statfs":{"total":96636764160,"available":96538275840,"internally_reserved":0,"allocated":71020544,"data_stored":71028543,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705710,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84316,"kb_used_data":57492,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94287524,"statfs":{"total":96636764160,"available":96550424576,"internally_reserved":0,"allocated":58871808,"data_stored":58413780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79016,"kb_used_data":52192,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94292824,"statfs":{"total":96636764160,"available":96555851776,"internally_reserved":0,"allocated":53444608,"data_stored":53927354,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71900,"kb_used_data":45076,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94299940,"statfs":{"total":96636764160,"available":96563138560,"internally_reserved":0,"allocated":46157824,"data_stored":46841676,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:08:36.920 INFO:tasks.rados.rados.0.smithi064.stdout:222: done (0 left) 2024-03-04T06:08:36.921 INFO:tasks.rados.rados.0.smithi064.stdout:223: write (excl) oid 45 current snap is 17 2024-03-04T06:08:36.921 INFO:tasks.rados.rados.0.smithi064.stdout:223: seq_num 135 ranges {668945=536616,1633422=491363} 2024-03-04T06:08:36.927 INFO:tasks.rados.rados.0.smithi064.stdout:223: writing smithi06431743-45 from 668945 to 1205561 tid 1 2024-03-04T06:08:36.932 INFO:tasks.rados.rados.0.smithi064.stdout:223: writing smithi06431743-45 from 1633422 to 2124785 tid 2 2024-03-04T06:08:36.932 INFO:tasks.rados.rados.0.smithi064.stdout:224: write oid 25 current snap is 17 2024-03-04T06:08:36.932 INFO:tasks.rados.rados.0.smithi064.stdout:224: seq_num 136 ranges {685752=611865,1975551=310201} 2024-03-04T06:08:36.939 INFO:tasks.rados.rados.0.smithi064.stdout:224: writing smithi06431743-25 from 685752 to 1297617 tid 1 2024-03-04T06:08:36.944 INFO:tasks.rados.rados.0.smithi064.stdout:224: writing smithi06431743-25 from 1975551 to 2285752 tid 2 2024-03-04T06:08:36.945 INFO:tasks.rados.rados.0.smithi064.stdout:225: write (excl) oid 19 current snap is 17 2024-03-04T06:08:36.945 INFO:tasks.rados.rados.0.smithi064.stdout:225: seq_num 137 ranges {702559=403467,1750386=495032} 2024-03-04T06:08:36.950 INFO:tasks.rados.rados.0.smithi064.stdout:225: writing smithi06431743-19 from 702559 to 1106026 tid 1 2024-03-04T06:08:36.958 INFO:tasks.rados.rados.0.smithi064.stdout:225: writing smithi06431743-19 from 1750386 to 2245418 tid 2 2024-03-04T06:08:36.958 INFO:tasks.rados.rados.0.smithi064.stdout:226: read oid 44 snap -1 2024-03-04T06:08:36.958 INFO:tasks.rados.rados.0.smithi064.stdout:226: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-04T06:08:36.958 INFO:tasks.rados.rados.0.smithi064.stdout:227: read oid 32 snap -1 2024-03-04T06:08:36.958 INFO:tasks.rados.rados.0.smithi064.stdout:227: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-04T06:08:36.958 INFO:tasks.rados.rados.0.smithi064.stdout:228: read oid 36 snap -1 2024-03-04T06:08:36.958 INFO:tasks.rados.rados.0.smithi064.stdout:228: expect deleted 2024-03-04T06:08:36.958 INFO:tasks.rados.rados.0.smithi064.stdout:229: read oid 6 snap -1 2024-03-04T06:08:36.958 INFO:tasks.rados.rados.0.smithi064.stdout:229: expect (ObjNum 66 snap 3 seq_num 66) 2024-03-04T06:08:36.958 INFO:tasks.rados.rados.0.smithi064.stdout:230: snap_remove snap 15 2024-03-04T06:08:37.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:37.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:37.596+0000 7f45c9aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:37.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:37.601+0000 7f45c9aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:37.919 INFO:tasks.rados.rados.0.smithi064.stdout:230: done (7 left) 2024-03-04T06:08:37.919 INFO:tasks.rados.rados.0.smithi064.stdout:231: read oid 31 snap -1 2024-03-04T06:08:37.920 INFO:tasks.rados.rados.0.smithi064.stdout:231: expect (ObjNum 30 snap 0 seq_num 30) 2024-03-04T06:08:37.920 INFO:tasks.rados.rados.0.smithi064.stdout:232: write oid 2 current snap is 17 2024-03-04T06:08:37.920 INFO:tasks.rados.rados.0.smithi064.stdout:232: seq_num 138 ranges {719366=478716,1808868=510498} 2024-03-04T06:08:37.925 INFO:tasks.rados.rados.0.smithi064.stdout:232: writing smithi06431743-2 from 719366 to 1198082 tid 1 2024-03-04T06:08:37.932 INFO:tasks.rados.rados.0.smithi064.stdout:232: writing smithi06431743-2 from 1808868 to 2319366 tid 2 2024-03-04T06:08:37.932 INFO:tasks.rados.rados.0.smithi064.stdout:233: read oid 29 snap -1 2024-03-04T06:08:37.932 INFO:tasks.rados.rados.0.smithi064.stdout:233: expect (ObjNum 91 snap 7 seq_num 91) 2024-03-04T06:08:37.933 INFO:tasks.rados.rados.0.smithi064.stdout:234: write oid 14 current snap is 17 2024-03-04T06:08:37.933 INFO:tasks.rados.rados.0.smithi064.stdout:234: seq_num 139 ranges {736173=553965,1867350=468823} 2024-03-04T06:08:37.939 INFO:tasks.rados.rados.0.smithi064.stdout:234: writing smithi06431743-14 from 736173 to 1290138 tid 1 2024-03-04T06:08:37.946 INFO:tasks.rados.rados.0.smithi064.stdout:234: writing smithi06431743-14 from 1867350 to 2336173 tid 2 2024-03-04T06:08:37.947 INFO:tasks.rados.rados.0.smithi064.stdout:235: delete oid 16 current snap is 17 2024-03-04T06:08:37.947 INFO:tasks.rados.rados.0.smithi064.stdout:229: expect (ObjNum 66 snap 3 seq_num 66) 2024-03-04T06:08:38.113 INFO:tasks.rados.rados.0.smithi064.stdout:223: finishing write tid 1 to smithi06431743-45 2024-03-04T06:08:38.113 INFO:tasks.rados.rados.0.smithi064.stdout:223: finishing write tid 2 to smithi06431743-45 2024-03-04T06:08:38.113 INFO:tasks.rados.rados.0.smithi064.stdout:226: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-04T06:08:38.338 INFO:tasks.rados.rados.0.smithi064.stdout:225: finishing write tid 1 to smithi06431743-19 2024-03-04T06:08:38.339 INFO:tasks.rados.rados.0.smithi064.stdout:223: finishing write tid 3 to smithi06431743-45 2024-03-04T06:08:38.339 INFO:tasks.rados.rados.0.smithi064.stdout:223: finishing write tid 4 to smithi06431743-45 2024-03-04T06:08:38.339 INFO:tasks.rados.rados.0.smithi064.stdout:223: oid 45 updating version 0 to 46 2024-03-04T06:08:38.339 INFO:tasks.rados.rados.0.smithi064.stdout:223: oid 45 version 46 is already newer than 44 2024-03-04T06:08:38.339 INFO:tasks.rados.rados.0.smithi064.stdout:223: oid 45 version 46 is already newer than 46 2024-03-04T06:08:38.339 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 46 (ObjNum 135 snap 17 seq_num 135) dirty exists 2024-03-04T06:08:38.339 INFO:tasks.rados.rados.0.smithi064.stdout:223: left oid 45 (ObjNum 135 snap 17 seq_num 135) 2024-03-04T06:08:38.339 INFO:tasks.rados.rados.0.smithi064.stdout:227: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-04T06:08:38.493 INFO:tasks.rados.rados.0.smithi064.stdout:225: finishing write tid 2 to smithi06431743-19 2024-03-04T06:08:38.493 INFO:tasks.rados.rados.0.smithi064.stdout:224: finishing write tid 1 to smithi06431743-25 2024-03-04T06:08:38.493 INFO:tasks.rados.rados.0.smithi064.stdout:225: finishing write tid 3 to smithi06431743-19 2024-03-04T06:08:38.493 INFO:tasks.rados.rados.0.smithi064.stdout:225: finishing write tid 4 to smithi06431743-19 2024-03-04T06:08:38.493 INFO:tasks.rados.rados.0.smithi064.stdout:225: oid 19 updating version 0 to 48 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:225: oid 19 updating version 48 to 49 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:225: oid 19 version 49 is already newer than 47 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 49 (ObjNum 137 snap 17 seq_num 137) dirty exists 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:225: left oid 19 (ObjNum 137 snap 17 seq_num 137) 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:224: finishing write tid 2 to smithi06431743-25 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:224: finishing write tid 3 to smithi06431743-25 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:224: finishing write tid 4 to smithi06431743-25 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:224: oid 25 updating version 0 to 49 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:224: oid 25 updating version 49 to 50 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:224: oid 25 version 50 is already newer than 49 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 50 (ObjNum 136 snap 17 seq_num 136) dirty exists 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:224: left oid 25 (ObjNum 136 snap 17 seq_num 136) 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:223: done (11 left) 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:224: done (10 left) 2024-03-04T06:08:38.494 INFO:tasks.rados.rados.0.smithi064.stdout:225: done (9 left) 2024-03-04T06:08:38.495 INFO:tasks.rados.rados.0.smithi064.stdout:226: done (8 left) 2024-03-04T06:08:38.495 INFO:tasks.rados.rados.0.smithi064.stdout:227: done (7 left) 2024-03-04T06:08:38.495 INFO:tasks.rados.rados.0.smithi064.stdout:228: done (6 left) 2024-03-04T06:08:38.495 INFO:tasks.rados.rados.0.smithi064.stdout:229: done (5 left) 2024-03-04T06:08:38.495 INFO:tasks.rados.rados.0.smithi064.stdout:235: done (4 left) 2024-03-04T06:08:38.495 INFO:tasks.rados.rados.0.smithi064.stdout:236: read oid 21 snap -1 2024-03-04T06:08:38.495 INFO:tasks.rados.rados.0.smithi064.stdout:236: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:08:38.495 INFO:tasks.rados.rados.0.smithi064.stdout:237: delete oid 43 current snap is 17 2024-03-04T06:08:38.495 INFO:tasks.rados.rados.0.smithi064.stdout:231: expect (ObjNum 30 snap 0 seq_num 30) 2024-03-04T06:08:38.591 INFO:tasks.rados.rados.0.smithi064.stdout:234: finishing write tid 1 to smithi06431743-14 2024-03-04T06:08:38.591 INFO:tasks.rados.rados.0.smithi064.stdout:232: finishing write tid 1 to smithi06431743-2 2024-03-04T06:08:38.591 INFO:tasks.rados.rados.0.smithi064.stdout:233: expect (ObjNum 91 snap 7 seq_num 91) 2024-03-04T06:08:38.762 INFO:tasks.rados.rados.0.smithi064.stdout:234: finishing write tid 2 to smithi06431743-14 2024-03-04T06:08:38.762 INFO:tasks.rados.rados.0.smithi064.stdout:232: finishing write tid 2 to smithi06431743-2 2024-03-04T06:08:38.762 INFO:tasks.rados.rados.0.smithi064.stdout:231: done (5 left) 2024-03-04T06:08:38.762 INFO:tasks.rados.rados.0.smithi064.stdout:233: done (4 left) 2024-03-04T06:08:38.762 INFO:tasks.rados.rados.0.smithi064.stdout:237: done (3 left) 2024-03-04T06:08:38.762 INFO:tasks.rados.rados.0.smithi064.stdout:238: snap_remove snap 11 2024-03-04T06:08:38.930 INFO:tasks.rados.rados.0.smithi064.stdout:238: done (3 left) 2024-03-04T06:08:38.930 INFO:tasks.rados.rados.0.smithi064.stdout:239: read oid 11 snap -1 2024-03-04T06:08:38.930 INFO:tasks.rados.rados.0.smithi064.stdout:239: expect deleted 2024-03-04T06:08:38.930 INFO:tasks.rados.rados.0.smithi064.stdout:240: snap_remove snap 16 2024-03-04T06:08:39.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:39.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:39.778+0000 7febf5d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:39.783 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:39.782+0000 7febf5d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:39.933 INFO:tasks.rados.rados.0.smithi064.stdout:240: done (4 left) 2024-03-04T06:08:39.933 INFO:tasks.rados.rados.0.smithi064.stdout:241: read oid 16 snap -1 2024-03-04T06:08:39.933 INFO:tasks.rados.rados.0.smithi064.stdout:241: expect deleted 2024-03-04T06:08:39.933 INFO:tasks.rados.rados.0.smithi064.stdout:242: read oid 20 snap -1 2024-03-04T06:08:39.933 INFO:tasks.rados.rados.0.smithi064.stdout:242: expect (ObjNum 114 snap 10 seq_num 114) 2024-03-04T06:08:39.933 INFO:tasks.rados.rados.0.smithi064.stdout:243: snap_create 2024-03-04T06:08:39.933 INFO:tasks.rados.rados.0.smithi064.stdout:234: finishing write tid 3 to smithi06431743-14 2024-03-04T06:08:39.933 INFO:tasks.rados.rados.0.smithi064.stdout:234: finishing write tid 4 to smithi06431743-14 2024-03-04T06:08:39.934 INFO:tasks.rados.rados.0.smithi064.stdout:234: oid 14 updating version 0 to 44 2024-03-04T06:08:39.934 INFO:tasks.rados.rados.0.smithi064.stdout:234: oid 14 version 44 is already newer than 43 2024-03-04T06:08:39.934 INFO:tasks.rados.rados.0.smithi064.stdout:234: oid 14 version 44 is already newer than 42 2024-03-04T06:08:39.934 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 44 (ObjNum 139 snap 17 seq_num 139) dirty exists 2024-03-04T06:08:39.934 INFO:tasks.rados.rados.0.smithi064.stdout:234: left oid 14 (ObjNum 139 snap 17 seq_num 139) 2024-03-04T06:08:39.934 INFO:tasks.rados.rados.0.smithi064.stdout:232: finishing write tid 3 to smithi06431743-2 2024-03-04T06:08:39.934 INFO:tasks.rados.rados.0.smithi064.stdout:232: finishing write tid 4 to smithi06431743-2 2024-03-04T06:08:39.934 INFO:tasks.rados.rados.0.smithi064.stdout:232: oid 2 updating version 0 to 33 2024-03-04T06:08:39.934 INFO:tasks.rados.rados.0.smithi064.stdout:232: oid 2 version 33 is already newer than 32 2024-03-04T06:08:39.934 INFO:tasks.rados.rados.0.smithi064.stdout:232: oid 2 version 33 is already newer than 31 2024-03-04T06:08:39.934 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 33 (ObjNum 138 snap 17 seq_num 138) dirty exists 2024-03-04T06:08:39.935 INFO:tasks.rados.rados.0.smithi064.stdout:232: left oid 2 (ObjNum 138 snap 17 seq_num 138) 2024-03-04T06:08:39.935 INFO:tasks.rados.rados.0.smithi064.stdout:236: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:08:40.027 INFO:tasks.rados.rados.0.smithi064.stdout:242: expect (ObjNum 114 snap 10 seq_num 114) 2024-03-04T06:08:40.930 INFO:tasks.rados.rados.0.smithi064.stdout:232: done (6 left) 2024-03-04T06:08:40.931 INFO:tasks.rados.rados.0.smithi064.stdout:234: done (5 left) 2024-03-04T06:08:40.931 INFO:tasks.rados.rados.0.smithi064.stdout:236: done (4 left) 2024-03-04T06:08:40.931 INFO:tasks.rados.rados.0.smithi064.stdout:239: done (3 left) 2024-03-04T06:08:40.931 INFO:tasks.rados.rados.0.smithi064.stdout:241: done (2 left) 2024-03-04T06:08:40.931 INFO:tasks.rados.rados.0.smithi064.stdout:242: done (1 left) 2024-03-04T06:08:40.931 INFO:tasks.rados.rados.0.smithi064.stdout:243: done (0 left) 2024-03-04T06:08:40.931 INFO:tasks.rados.rados.0.smithi064.stdout:244: write (excl) oid 49 current snap is 18 2024-03-04T06:08:40.931 INFO:tasks.rados.rados.0.smithi064.stdout:244: seq_num 142 ranges {786594=779712,2326443=60151} 2024-03-04T06:08:40.938 INFO:tasks.rados.rados.0.smithi064.stdout:244: writing smithi06431743-49 from 786594 to 1566306 tid 1 2024-03-04T06:08:40.943 INFO:tasks.rados.rados.0.smithi064.stdout:244: writing smithi06431743-49 from 2326443 to 2386594 tid 2 2024-03-04T06:08:40.944 INFO:tasks.rados.rados.0.smithi064.stdout:245: snap_create 2024-03-04T06:08:41.021 INFO:tasks.rados.rados.0.smithi064.stdout:244: finishing write tid 1 to smithi06431743-49 2024-03-04T06:08:41.030 INFO:tasks.rados.rados.0.smithi064.stdout:244: finishing write tid 2 to smithi06431743-49 2024-03-04T06:08:41.037 INFO:tasks.rados.rados.0.smithi064.stdout:244: finishing write tid 3 to smithi06431743-49 2024-03-04T06:08:41.038 INFO:tasks.rados.rados.0.smithi064.stdout:244: finishing write tid 4 to smithi06431743-49 2024-03-04T06:08:41.038 INFO:tasks.rados.rados.0.smithi064.stdout:244: oid 49 updating version 0 to 39 2024-03-04T06:08:41.038 INFO:tasks.rados.rados.0.smithi064.stdout:244: oid 49 updating version 39 to 40 2024-03-04T06:08:41.038 INFO:tasks.rados.rados.0.smithi064.stdout:244: oid 49 updating version 40 to 41 2024-03-04T06:08:41.038 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 41 (ObjNum 142 snap 18 seq_num 142) dirty exists 2024-03-04T06:08:41.038 INFO:tasks.rados.rados.0.smithi064.stdout:244: left oid 49 (ObjNum 142 snap 18 seq_num 142) 2024-03-04T06:08:41.465 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:08:41.465 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:08:41.466 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:08:41.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:08:41.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:41.573+0000 7f58d226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:41.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:41.581+0000 7f58d226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:41.594 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:41.594 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:41.594 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:41.594 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:41.594 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:41.594 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:41.594 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:41.595 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:41.595 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:41.595 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:41.595 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:41.595 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:41.597 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:41.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:08:41.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:41.701+0000 7f5c4199d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:41.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:41.708+0000 7f5c4199d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:41.722 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:41.722 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:41.722 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:41.722 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:41.722 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:41.722 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:41.722 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:41.722 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:41.722 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:41.722 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:41.723 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:41.723 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:41.725 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:41.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:08:41.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:41.828+0000 7f894339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:41.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:41.835+0000 7f894339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:41.848 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:41.848 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:41.849 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:41.849 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:41.849 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:41.849 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:41.849 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:41.849 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:41.849 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:41.849 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:41.849 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:41.849 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:41.851 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:41.851 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:41.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:08:41.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:244: done (1 left) 2024-03-04T06:08:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:245: done (0 left) 2024-03-04T06:08:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:246: read oid 49 snap -1 2024-03-04T06:08:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:246: expect (ObjNum 142 snap 18 seq_num 142) 2024-03-04T06:08:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:247: write oid 37 current snap is 19 2024-03-04T06:08:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:247: seq_num 143 ranges {403401=454961,1584925=506039} 2024-03-04T06:08:41.935 INFO:tasks.rados.rados.0.smithi064.stdout:247: writing smithi06431743-37 from 403401 to 858362 tid 1 2024-03-04T06:08:41.942 INFO:tasks.rados.rados.0.smithi064.stdout:247: writing smithi06431743-37 from 1584925 to 2090964 tid 2 2024-03-04T06:08:41.943 INFO:tasks.rados.rados.0.smithi064.stdout:248: write oid 48 current snap is 19 2024-03-04T06:08:41.943 INFO:tasks.rados.rados.0.smithi064.stdout:248: seq_num 144 ranges {420208=530210,1527054=766050} 2024-03-04T06:08:41.948 INFO:tasks.rados.rados.0.smithi064.stdout:248: writing smithi06431743-48 from 420208 to 950418 tid 1 2024-03-04T06:08:41.955 INFO:tasks.rados.rados.0.smithi064.stdout:248: writing smithi06431743-48 from 1527054 to 2293104 tid 2 2024-03-04T06:08:41.955 INFO:tasks.rados.rados.0.smithi064.stdout:249: snap_create 2024-03-04T06:08:41.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:41.977+0000 7f0b0066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:41.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:41.981+0000 7f3fc635d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:41.987 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:41.986+0000 7f0b0066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:41.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:41.987+0000 7f3fc635d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:41.993 INFO:tasks.rados.rados.0.smithi064.stdout:247: finishing write tid 1 to smithi06431743-37 2024-03-04T06:08:42.006 INFO:tasks.rados.rados.0.smithi064.stdout:247: finishing write tid 2 to smithi06431743-37 2024-03-04T06:08:42.006 INFO:tasks.rados.rados.0.smithi064.stdout:246: expect (ObjNum 142 snap 18 seq_num 142) 2024-03-04T06:08:42.009 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:42.009 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:42.009 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:42.009 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:42.009 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:42.009 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:42.009 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:42.009 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:42.010 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:42.010 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:42.010 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:42.010 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:42.012 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:42.013 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:42.171 INFO:tasks.rados.rados.0.smithi064.stdout:248: finishing write tid 1 to smithi06431743-48 2024-03-04T06:08:42.171 INFO:tasks.rados.rados.0.smithi064.stdout:247: finishing write tid 3 to smithi06431743-37 2024-03-04T06:08:42.171 INFO:tasks.rados.rados.0.smithi064.stdout:247: finishing write tid 4 to smithi06431743-37 2024-03-04T06:08:42.171 INFO:tasks.rados.rados.0.smithi064.stdout:247: oid 37 updating version 0 to 21 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:247: oid 37 version 21 is already newer than 20 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:247: oid 37 version 21 is already newer than 20 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 21 (ObjNum 143 snap 19 seq_num 143) dirty exists 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:247: left oid 37 (ObjNum 143 snap 19 seq_num 143) 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:248: finishing write tid 2 to smithi06431743-48 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:248: finishing write tid 3 to smithi06431743-48 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:248: finishing write tid 4 to smithi06431743-48 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:248: oid 48 updating version 0 to 27 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:248: oid 48 updating version 27 to 29 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:248: oid 48 version 29 is already newer than 29 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 29 (ObjNum 144 snap 19 seq_num 144) dirty exists 2024-03-04T06:08:42.172 INFO:tasks.rados.rados.0.smithi064.stdout:248: left oid 48 (ObjNum 144 snap 19 seq_num 144) 2024-03-04T06:08:42.963 INFO:tasks.rados.rados.0.smithi064.stdout:246: done (3 left) 2024-03-04T06:08:42.963 INFO:tasks.rados.rados.0.smithi064.stdout:247: done (2 left) 2024-03-04T06:08:42.964 INFO:tasks.rados.rados.0.smithi064.stdout:248: done (1 left) 2024-03-04T06:08:42.964 INFO:tasks.rados.rados.0.smithi064.stdout:249: done (0 left) 2024-03-04T06:08:42.964 INFO:tasks.rados.rados.0.smithi064.stdout:250: read oid 44 snap -1 2024-03-04T06:08:42.964 INFO:tasks.rados.rados.0.smithi064.stdout:250: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-04T06:08:42.964 INFO:tasks.rados.rados.0.smithi064.stdout:251: write (excl) oid 32 current snap is 20 2024-03-04T06:08:42.964 INFO:tasks.rados.rados.0.smithi064.stdout:251: seq_num 145 ranges {437015=721812,1701889=509708} 2024-03-04T06:08:42.969 INFO:tasks.rados.rados.0.smithi064.stdout:251: writing smithi06431743-32 from 437015 to 1158827 tid 1 2024-03-04T06:08:42.975 INFO:tasks.rados.rados.0.smithi064.stdout:251: writing smithi06431743-32 from 1701889 to 2211597 tid 2 2024-03-04T06:08:42.976 INFO:tasks.rados.rados.0.smithi064.stdout:252: write oid 10 current snap is 20 2024-03-04T06:08:42.976 INFO:tasks.rados.rados.0.smithi064.stdout:252: seq_num 146 ranges {453822=797061,1760371=653366} 2024-03-04T06:08:42.982 INFO:tasks.rados.rados.0.smithi064.stdout:252: writing smithi06431743-10 from 453822 to 1250883 tid 1 2024-03-04T06:08:42.991 INFO:tasks.rados.rados.0.smithi064.stdout:252: writing smithi06431743-10 from 1760371 to 2413737 tid 2 2024-03-04T06:08:42.991 INFO:tasks.rados.rados.0.smithi064.stdout:253: delete oid 46 current snap is 20 2024-03-04T06:08:43.010 INFO:tasks.rados.rados.0.smithi064.stdout:250: expect (ObjNum 105 snap 9 seq_num 105) 2024-03-04T06:08:43.237 INFO:tasks.rados.rados.0.smithi064.stdout:250: done (3 left) 2024-03-04T06:08:43.237 INFO:tasks.rados.rados.0.smithi064.stdout:253: done (2 left) 2024-03-04T06:08:43.237 INFO:tasks.rados.rados.0.smithi064.stdout:254: snap_remove snap 19 2024-03-04T06:08:44.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:44.182 INFO:tasks.rados.rados.0.smithi064.stdout:254: done (2 left) 2024-03-04T06:08:44.182 INFO:tasks.rados.rados.0.smithi064.stdout:255: delete oid 50 current snap is 20 2024-03-04T06:08:44.182 INFO:tasks.rados.rados.0.smithi064.stdout:252: finishing write tid 1 to smithi06431743-10 2024-03-04T06:08:44.182 INFO:tasks.rados.rados.0.smithi064.stdout:251: finishing write tid 1 to smithi06431743-32 2024-03-04T06:08:44.182 INFO:tasks.rados.rados.0.smithi064.stdout:252: finishing write tid 2 to smithi06431743-10 2024-03-04T06:08:44.182 INFO:tasks.rados.rados.0.smithi064.stdout:251: finishing write tid 2 to smithi06431743-32 2024-03-04T06:08:44.183 INFO:tasks.rados.rados.0.smithi064.stdout:252: finishing write tid 3 to smithi06431743-10 2024-03-04T06:08:44.183 INFO:tasks.rados.rados.0.smithi064.stdout:252: finishing write tid 4 to smithi06431743-10 2024-03-04T06:08:44.183 INFO:tasks.rados.rados.0.smithi064.stdout:252: oid 10 updating version 0 to 34 2024-03-04T06:08:44.183 INFO:tasks.rados.rados.0.smithi064.stdout:252: oid 10 updating version 34 to 35 2024-03-04T06:08:44.183 INFO:tasks.rados.rados.0.smithi064.stdout:252: oid 10 updating version 35 to 36 2024-03-04T06:08:44.183 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 36 (ObjNum 146 snap 20 seq_num 146) dirty exists 2024-03-04T06:08:44.183 INFO:tasks.rados.rados.0.smithi064.stdout:252: left oid 10 (ObjNum 146 snap 20 seq_num 146) 2024-03-04T06:08:44.183 INFO:tasks.rados.rados.0.smithi064.stdout:251: finishing write tid 3 to smithi06431743-32 2024-03-04T06:08:44.183 INFO:tasks.rados.rados.0.smithi064.stdout:251: finishing write tid 4 to smithi06431743-32 2024-03-04T06:08:44.183 INFO:tasks.rados.rados.0.smithi064.stdout:251: oid 32 updating version 0 to 106 2024-03-04T06:08:44.184 INFO:tasks.rados.rados.0.smithi064.stdout:251: oid 32 version 106 is already newer than 105 2024-03-04T06:08:44.184 INFO:tasks.rados.rados.0.smithi064.stdout:251: oid 32 version 106 is already newer than 104 2024-03-04T06:08:44.184 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 106 (ObjNum 145 snap 20 seq_num 145) dirty exists 2024-03-04T06:08:44.184 INFO:tasks.rados.rados.0.smithi064.stdout:251: left oid 32 (ObjNum 145 snap 20 seq_num 145) 2024-03-04T06:08:44.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:44.208+0000 7f9c09d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:44.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:44.213+0000 7f9c09d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:44.229 INFO:tasks.rados.rados.0.smithi064.stdout:251: done (2 left) 2024-03-04T06:08:44.229 INFO:tasks.rados.rados.0.smithi064.stdout:252: done (1 left) 2024-03-04T06:08:44.229 INFO:tasks.rados.rados.0.smithi064.stdout:255: done (0 left) 2024-03-04T06:08:44.229 INFO:tasks.rados.rados.0.smithi064.stdout:256: read oid 25 snap 18 2024-03-04T06:08:44.229 INFO:tasks.rados.rados.0.smithi064.stdout:256: expect (ObjNum 136 snap 17 seq_num 136) 2024-03-04T06:08:44.229 INFO:tasks.rados.rados.0.smithi064.stdout:257: write (excl) oid 30 current snap is 20 2024-03-04T06:08:44.229 INFO:tasks.rados.rados.0.smithi064.stdout:257: seq_num 149 ranges {504243=622808,1819464=517046} 2024-03-04T06:08:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:257: writing smithi06431743-30 from 504243 to 1127051 tid 1 2024-03-04T06:08:44.242 INFO:tasks.rados.rados.0.smithi064.stdout:257: writing smithi06431743-30 from 1819464 to 2336510 tid 2 2024-03-04T06:08:44.242 INFO:tasks.rados.rados.0.smithi064.stdout:258: write oid 3 current snap is 20 2024-03-04T06:08:44.242 INFO:tasks.rados.rados.0.smithi064.stdout:258: seq_num 150 ranges {521050=698057,1877946=643104} 2024-03-04T06:08:44.248 INFO:tasks.rados.rados.0.smithi064.stdout:258: writing smithi06431743-3 from 521050 to 1219107 tid 1 2024-03-04T06:08:44.256 INFO:tasks.rados.rados.0.smithi064.stdout:258: writing smithi06431743-3 from 1877946 to 2521050 tid 2 2024-03-04T06:08:44.256 INFO:tasks.rados.rados.0.smithi064.stdout:259: snap_create 2024-03-04T06:08:44.257 INFO:tasks.rados.rados.0.smithi064.stdout:257: finishing write tid 1 to smithi06431743-30 2024-03-04T06:08:44.263 INFO:tasks.rados.rados.0.smithi064.stdout:257: finishing write tid 2 to smithi06431743-30 2024-03-04T06:08:44.275 INFO:tasks.rados.rados.0.smithi064.stdout:257: finishing write tid 3 to smithi06431743-30 2024-03-04T06:08:44.276 INFO:tasks.rados.rados.0.smithi064.stdout:257: finishing write tid 4 to smithi06431743-30 2024-03-04T06:08:44.276 INFO:tasks.rados.rados.0.smithi064.stdout:257: oid 30 updating version 0 to 53 2024-03-04T06:08:44.277 INFO:tasks.rados.rados.0.smithi064.stdout:257: oid 30 updating version 53 to 54 2024-03-04T06:08:44.277 INFO:tasks.rados.rados.0.smithi064.stdout:257: oid 30 version 54 is already newer than 52 2024-03-04T06:08:44.277 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 54 (ObjNum 149 snap 20 seq_num 149) dirty exists 2024-03-04T06:08:44.277 INFO:tasks.rados.rados.0.smithi064.stdout:257: left oid 30 (ObjNum 149 snap 20 seq_num 149) 2024-03-04T06:08:44.311 INFO:tasks.rados.rados.0.smithi064.stdout:258: finishing write tid 1 to smithi06431743-3 2024-03-04T06:08:44.312 INFO:tasks.rados.rados.0.smithi064.stdout:256: expect (ObjNum 136 snap 17 seq_num 136) 2024-03-04T06:08:44.455 INFO:tasks.rados.rados.0.smithi064.stdout:258: finishing write tid 2 to smithi06431743-3 2024-03-04T06:08:44.455 INFO:tasks.rados.rados.0.smithi064.stdout:258: finishing write tid 3 to smithi06431743-3 2024-03-04T06:08:44.455 INFO:tasks.rados.rados.0.smithi064.stdout:258: finishing write tid 4 to smithi06431743-3 2024-03-04T06:08:44.455 INFO:tasks.rados.rados.0.smithi064.stdout:258: oid 3 updating version 0 to 26 2024-03-04T06:08:44.455 INFO:tasks.rados.rados.0.smithi064.stdout:258: oid 3 updating version 26 to 28 2024-03-04T06:08:44.456 INFO:tasks.rados.rados.0.smithi064.stdout:258: oid 3 version 28 is already newer than 27 2024-03-04T06:08:44.456 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 28 (ObjNum 150 snap 20 seq_num 150) dirty exists 2024-03-04T06:08:44.456 INFO:tasks.rados.rados.0.smithi064.stdout:258: left oid 3 (ObjNum 150 snap 20 seq_num 150) 2024-03-04T06:08:45.182 INFO:tasks.rados.rados.0.smithi064.stdout:256: done (3 left) 2024-03-04T06:08:45.182 INFO:tasks.rados.rados.0.smithi064.stdout:257: done (2 left) 2024-03-04T06:08:45.182 INFO:tasks.rados.rados.0.smithi064.stdout:258: done (1 left) 2024-03-04T06:08:45.182 INFO:tasks.rados.rados.0.smithi064.stdout:259: done (0 left) 2024-03-04T06:08:45.182 INFO:tasks.rados.rados.0.smithi064.stdout:260: write (excl) oid 45 current snap is 21 2024-03-04T06:08:45.182 INFO:tasks.rados.rados.0.smithi064.stdout:260: seq_num 151 ranges {537857=773306,1936428=520715} 2024-03-04T06:08:45.189 INFO:tasks.rados.rados.0.smithi064.stdout:260: writing smithi06431743-45 from 537857 to 1311163 tid 1 2024-03-04T06:08:45.195 INFO:tasks.rados.rados.0.smithi064.stdout:260: writing smithi06431743-45 from 1936428 to 2457143 tid 2 2024-03-04T06:08:45.195 INFO:tasks.rados.rados.0.smithi064.stdout:261: snap_remove snap 17 2024-03-04T06:08:46.185 INFO:tasks.rados.rados.0.smithi064.stdout:261: done (1 left) 2024-03-04T06:08:46.185 INFO:tasks.rados.rados.0.smithi064.stdout:262: write (excl) oid 9 current snap is 21 2024-03-04T06:08:46.185 INFO:tasks.rados.rados.0.smithi064.stdout:260: finishing write tid 1 to smithi06431743-45 2024-03-04T06:08:46.185 INFO:tasks.rados.rados.0.smithi064.stdout:262: seq_num 152 ranges {554664=448555,1478557=664373} 2024-03-04T06:08:46.190 INFO:tasks.rados.rados.0.smithi064.stdout:262: writing smithi06431743-9 from 554664 to 1003219 tid 1 2024-03-04T06:08:46.196 INFO:tasks.rados.rados.0.smithi064.stdout:262: writing smithi06431743-9 from 1478557 to 2142930 tid 2 2024-03-04T06:08:46.196 INFO:tasks.rados.rados.0.smithi064.stdout:263: write (excl) oid 20 current snap is 21 2024-03-04T06:08:46.196 INFO:tasks.rados.rados.0.smithi064.stdout:263: seq_num 153 ranges {571471=640157,1653392=524384} 2024-03-04T06:08:46.203 INFO:tasks.rados.rados.0.smithi064.stdout:263: writing smithi06431743-20 from 571471 to 1211628 tid 1 2024-03-04T06:08:46.208 INFO:tasks.rados.rados.0.smithi064.stdout:263: writing smithi06431743-20 from 1653392 to 2177776 tid 2 2024-03-04T06:08:46.208 INFO:tasks.rados.rados.0.smithi064.stdout:264: snap_create 2024-03-04T06:08:46.208 INFO:tasks.rados.rados.0.smithi064.stdout:260: finishing write tid 2 to smithi06431743-45 2024-03-04T06:08:46.208 INFO:tasks.rados.rados.0.smithi064.stdout:260: finishing write tid 3 to smithi06431743-45 2024-03-04T06:08:46.209 INFO:tasks.rados.rados.0.smithi064.stdout:260: finishing write tid 4 to smithi06431743-45 2024-03-04T06:08:46.209 INFO:tasks.rados.rados.0.smithi064.stdout:260: oid 45 updating version 0 to 64 2024-03-04T06:08:46.209 INFO:tasks.rados.rados.0.smithi064.stdout:260: oid 45 updating version 64 to 65 2024-03-04T06:08:46.209 INFO:tasks.rados.rados.0.smithi064.stdout:260: oid 45 updating version 65 to 66 2024-03-04T06:08:46.209 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 66 (ObjNum 151 snap 21 seq_num 151) dirty exists 2024-03-04T06:08:46.209 INFO:tasks.rados.rados.0.smithi064.stdout:260: left oid 45 (ObjNum 151 snap 21 seq_num 151) 2024-03-04T06:08:46.271 INFO:tasks.rados.rados.0.smithi064.stdout:262: finishing write tid 1 to smithi06431743-9 2024-03-04T06:08:46.273 INFO:tasks.rados.rados.0.smithi064.stdout:263: finishing write tid 1 to smithi06431743-20 2024-03-04T06:08:46.289 INFO:tasks.rados.rados.0.smithi064.stdout:262: finishing write tid 2 to smithi06431743-9 2024-03-04T06:08:46.294 INFO:tasks.rados.rados.0.smithi064.stdout:263: finishing write tid 2 to smithi06431743-20 2024-03-04T06:08:46.296 INFO:tasks.rados.rados.0.smithi064.stdout:262: finishing write tid 3 to smithi06431743-9 2024-03-04T06:08:46.298 INFO:tasks.rados.rados.0.smithi064.stdout:262: finishing write tid 4 to smithi06431743-9 2024-03-04T06:08:46.298 INFO:tasks.rados.rados.0.smithi064.stdout:262: oid 9 updating version 0 to 53 2024-03-04T06:08:46.298 INFO:tasks.rados.rados.0.smithi064.stdout:262: oid 9 updating version 53 to 54 2024-03-04T06:08:46.298 INFO:tasks.rados.rados.0.smithi064.stdout:262: oid 9 version 54 is already newer than 52 2024-03-04T06:08:46.298 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 54 (ObjNum 152 snap 21 seq_num 152) dirty exists 2024-03-04T06:08:46.298 INFO:tasks.rados.rados.0.smithi064.stdout:262: left oid 9 (ObjNum 152 snap 21 seq_num 152) 2024-03-04T06:08:46.301 INFO:tasks.rados.rados.0.smithi064.stdout:263: finishing write tid 3 to smithi06431743-20 2024-03-04T06:08:46.302 INFO:tasks.rados.rados.0.smithi064.stdout:263: finishing write tid 4 to smithi06431743-20 2024-03-04T06:08:46.303 INFO:tasks.rados.rados.0.smithi064.stdout:263: oid 20 updating version 0 to 49 2024-03-04T06:08:46.303 INFO:tasks.rados.rados.0.smithi064.stdout:263: oid 20 updating version 49 to 50 2024-03-04T06:08:46.303 INFO:tasks.rados.rados.0.smithi064.stdout:263: oid 20 version 50 is already newer than 48 2024-03-04T06:08:46.303 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 50 (ObjNum 153 snap 21 seq_num 153) dirty exists 2024-03-04T06:08:46.303 INFO:tasks.rados.rados.0.smithi064.stdout:263: left oid 20 (ObjNum 153 snap 21 seq_num 153) 2024-03-04T06:08:46.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:46.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:46.445+0000 7fd47b2dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:46.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:46.449+0000 7fd47b2dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:47.014 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:08:47.014 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:08:47.015 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:08:47.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:08:47.127 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:47.122+0000 7faa92e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:47.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:47.129+0000 7faa92e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:47.142 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:47.142 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:47.142 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:47.142 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:47.142 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:47.142 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:47.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:47.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:47.143 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:47.143 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:47.143 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:47.143 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:47.145 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:47.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:08:47.187 INFO:tasks.rados.rados.0.smithi064.stdout:260: done (3 left) 2024-03-04T06:08:47.187 INFO:tasks.rados.rados.0.smithi064.stdout:262: done (2 left) 2024-03-04T06:08:47.187 INFO:tasks.rados.rados.0.smithi064.stdout:263: done (1 left) 2024-03-04T06:08:47.188 INFO:tasks.rados.rados.0.smithi064.stdout:264: done (0 left) 2024-03-04T06:08:47.188 INFO:tasks.rados.rados.0.smithi064.stdout:265: snap_create 2024-03-04T06:08:47.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:47.254+0000 7f1c1c62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:47.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:47.261+0000 7f1c1c62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:47.274 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:47.274 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:47.274 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:47.274 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:47.274 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:47.274 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:47.274 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:47.274 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:47.274 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:47.274 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:47.274 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:47.275 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:47.277 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:47.277 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:47.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:08:47.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:47.420+0000 7ff6e5f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:47.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:47.427+0000 7ff6e5f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:47.441 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:47.444 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:47.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:08:47.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:47.546+0000 7f33fc95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:47.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:47.554+0000 7f33fc95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:47.567 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:47.567 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:47.567 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:47.567 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:47.567 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:47.567 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:47.567 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:47.567 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:47.567 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:47.567 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:47.567 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:47.568 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:47.570 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:48.188 INFO:tasks.rados.rados.0.smithi064.stdout:265: done (0 left) 2024-03-04T06:08:48.189 INFO:tasks.rados.rados.0.smithi064.stdout:266: snap_remove snap 22 2024-03-04T06:08:48.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:48.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:48.633+0000 7fef80c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:48.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:48.637+0000 7fef80c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:49.200 INFO:tasks.rados.rados.0.smithi064.stdout:266: done (0 left) 2024-03-04T06:08:49.200 INFO:tasks.rados.rados.0.smithi064.stdout:267: write (excl) oid 9 current snap is 23 2024-03-04T06:08:49.200 INFO:tasks.rados.rados.0.smithi064.stdout:267: seq_num 154 ranges {588278=715406,1711874=668042} 2024-03-04T06:08:49.206 INFO:tasks.rados.rados.0.smithi064.stdout:267: writing smithi06431743-9 from 588278 to 1303684 tid 1 2024-03-04T06:08:49.212 INFO:tasks.rados.rados.0.smithi064.stdout:267: writing smithi06431743-9 from 1711874 to 2379916 tid 2 2024-03-04T06:08:49.212 INFO:tasks.rados.rados.0.smithi064.stdout:268: snap_create 2024-03-04T06:08:49.275 INFO:tasks.rados.rados.0.smithi064.stdout:267: finishing write tid 1 to smithi06431743-9 2024-03-04T06:08:49.291 INFO:tasks.rados.rados.0.smithi064.stdout:267: finishing write tid 2 to smithi06431743-9 2024-03-04T06:08:49.295 INFO:tasks.rados.rados.0.smithi064.stdout:267: finishing write tid 3 to smithi06431743-9 2024-03-04T06:08:49.296 INFO:tasks.rados.rados.0.smithi064.stdout:267: finishing write tid 4 to smithi06431743-9 2024-03-04T06:08:49.296 INFO:tasks.rados.rados.0.smithi064.stdout:267: oid 9 updating version 0 to 57 2024-03-04T06:08:49.296 INFO:tasks.rados.rados.0.smithi064.stdout:267: oid 9 updating version 57 to 58 2024-03-04T06:08:49.296 INFO:tasks.rados.rados.0.smithi064.stdout:267: oid 9 version 58 is already newer than 56 2024-03-04T06:08:49.296 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 58 (ObjNum 154 snap 23 seq_num 154) dirty exists 2024-03-04T06:08:49.296 INFO:tasks.rados.rados.0.smithi064.stdout:267: left oid 9 (ObjNum 154 snap 23 seq_num 154) 2024-03-04T06:08:50.203 INFO:tasks.rados.rados.0.smithi064.stdout:267: done (1 left) 2024-03-04T06:08:50.203 INFO:tasks.rados.rados.0.smithi064.stdout:268: done (0 left) 2024-03-04T06:08:50.203 INFO:tasks.rados.rados.0.smithi064.stdout:269: read oid 33 snap -1 2024-03-04T06:08:50.203 INFO:tasks.rados.rados.0.smithi064.stdout:269: expect (ObjNum 124 snap 15 seq_num 124) 2024-03-04T06:08:50.203 INFO:tasks.rados.rados.0.smithi064.stdout:270: delete oid 39 current snap is 24 2024-03-04T06:08:50.246 INFO:tasks.rados.rados.0.smithi064.stdout:270: done (1 left) 2024-03-04T06:08:50.246 INFO:tasks.rados.rados.0.smithi064.stdout:271: snap_remove snap 21 2024-03-04T06:08:50.708 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:50.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:50.825+0000 7fd2a80ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:50.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:50.829+0000 7fd2a80ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:51.209 INFO:tasks.rados.rados.0.smithi064.stdout:271: done (1 left) 2024-03-04T06:08:51.209 INFO:tasks.rados.rados.0.smithi064.stdout:272: write (excl) oid 31 current snap is 24 2024-03-04T06:08:51.209 INFO:tasks.rados.rados.0.smithi064.stdout:272: seq_num 156 ranges {621892=465904,1712485=671711} 2024-03-04T06:08:51.215 INFO:tasks.rados.rados.0.smithi064.stdout:272: writing smithi06431743-31 from 621892 to 1087796 tid 1 2024-03-04T06:08:51.222 INFO:tasks.rados.rados.0.smithi064.stdout:272: writing smithi06431743-31 from 1712485 to 2384196 tid 2 2024-03-04T06:08:51.222 INFO:tasks.rados.rados.0.smithi064.stdout:273: delete oid 28 current snap is 24 2024-03-04T06:08:51.222 INFO:tasks.rados.rados.0.smithi064.stdout:269: expect (ObjNum 124 snap 15 seq_num 124) 2024-03-04T06:08:51.364 INFO:tasks.rados.rados.0.smithi064.stdout:272: finishing write tid 1 to smithi06431743-31 2024-03-04T06:08:51.364 INFO:tasks.rados.rados.0.smithi064.stdout:272: finishing write tid 2 to smithi06431743-31 2024-03-04T06:08:51.364 INFO:tasks.rados.rados.0.smithi064.stdout:272: finishing write tid 3 to smithi06431743-31 2024-03-04T06:08:51.365 INFO:tasks.rados.rados.0.smithi064.stdout:272: finishing write tid 4 to smithi06431743-31 2024-03-04T06:08:51.365 INFO:tasks.rados.rados.0.smithi064.stdout:272: oid 31 updating version 0 to 6 2024-03-04T06:08:51.365 INFO:tasks.rados.rados.0.smithi064.stdout:272: oid 31 version 6 is already newer than 5 2024-03-04T06:08:51.365 INFO:tasks.rados.rados.0.smithi064.stdout:272: oid 31 updating version 6 to 7 2024-03-04T06:08:51.365 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 7 (ObjNum 156 snap 24 seq_num 156) dirty exists 2024-03-04T06:08:51.365 INFO:tasks.rados.rados.0.smithi064.stdout:272: left oid 31 (ObjNum 156 snap 24 seq_num 156) 2024-03-04T06:08:51.365 INFO:tasks.rados.rados.0.smithi064.stdout:269: done (2 left) 2024-03-04T06:08:51.365 INFO:tasks.rados.rados.0.smithi064.stdout:272: done (1 left) 2024-03-04T06:08:51.365 INFO:tasks.rados.rados.0.smithi064.stdout:273: done (0 left) 2024-03-04T06:08:51.365 INFO:tasks.rados.rados.0.smithi064.stdout:274: delete oid 29 current snap is 24 2024-03-04T06:08:51.410 INFO:tasks.rados.rados.0.smithi064.stdout:274: done (0 left) 2024-03-04T06:08:51.410 INFO:tasks.rados.rados.0.smithi064.stdout:275: read oid 3 snap -1 2024-03-04T06:08:51.411 INFO:tasks.rados.rados.0.smithi064.stdout:275: expect (ObjNum 150 snap 20 seq_num 150) 2024-03-04T06:08:51.411 INFO:tasks.rados.rados.0.smithi064.stdout:276: snap_remove snap 20 2024-03-04T06:08:52.208 INFO:tasks.rados.rados.0.smithi064.stdout:276: done (1 left) 2024-03-04T06:08:52.208 INFO:tasks.rados.rados.0.smithi064.stdout:277: delete oid 40 current snap is 24 2024-03-04T06:08:52.208 INFO:tasks.rados.rados.0.smithi064.stdout:275: expect (ObjNum 150 snap 20 seq_num 150) 2024-03-04T06:08:52.352 INFO:tasks.rados.rados.0.smithi064.stdout:275: done (1 left) 2024-03-04T06:08:52.352 INFO:tasks.rados.rados.0.smithi064.stdout:277: done (0 left) 2024-03-04T06:08:52.352 INFO:tasks.rados.rados.0.smithi064.stdout:278: write (excl) oid 37 current snap is 24 2024-03-04T06:08:52.352 INFO:tasks.rados.rados.0.smithi064.stdout:278: seq_num 160 ranges {689120=483253,1946413=679049} 2024-03-04T06:08:52.358 INFO:tasks.rados.rados.0.smithi064.stdout:278: writing smithi06431743-37 from 689120 to 1172373 tid 1 2024-03-04T06:08:52.367 INFO:tasks.rados.rados.0.smithi064.stdout:278: writing smithi06431743-37 from 1946413 to 2625462 tid 2 2024-03-04T06:08:52.367 INFO:tasks.rados.rados.0.smithi064.stdout:279: write (excl) oid 50 current snap is 24 2024-03-04T06:08:52.367 INFO:tasks.rados.rados.0.smithi064.stdout:279: seq_num 161 ranges {705927=558502,2004895=422707} 2024-03-04T06:08:52.374 INFO:tasks.rados.rados.0.smithi064.stdout:279: writing smithi06431743-50 from 705927 to 1264429 tid 1 2024-03-04T06:08:52.381 INFO:tasks.rados.rados.0.smithi064.stdout:279: writing smithi06431743-50 from 2004895 to 2427602 tid 2 2024-03-04T06:08:52.381 INFO:tasks.rados.rados.0.smithi064.stdout:280: snap_remove snap 18 2024-03-04T06:08:52.571 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:08:52.571 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:08:52.571 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:08:52.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:08:52.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:52.676+0000 7f91f771d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:52.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:52.683+0000 7f91f771d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:52.696 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:52.696 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:52.696 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:52.696 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:52.696 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:52.696 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:52.696 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:52.696 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:52.697 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:52.697 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:52.697 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:52.697 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:52.699 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:52.699 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:08:52.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:52.803+0000 7f7c5a35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:52.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:52.810+0000 7f7c5a35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:52.823 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:52.823 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:52.823 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:52.824 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:52.824 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:52.824 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:52.824 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:52.824 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:52.824 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:52.824 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:52.824 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:52.824 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:52.826 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:52.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:08:52.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:52.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:52.931+0000 7fe03cd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:52.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:52.939+0000 7fe03cd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:52.952 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:52.952 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:52.952 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:52.952 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:52.952 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:52.952 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:52.952 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:52.952 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:52.953 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:52.953 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:52.953 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:52.953 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:52.955 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:52.955 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:52.956 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:08:53.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:53.006+0000 7f69a98d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:53.011 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:53.011+0000 7f69a98d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:53.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:53.102+0000 7f548b35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:53.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:53.109+0000 7f548b35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:53.122 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:53.122 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:53.122 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:53.122 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:53.122 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:53.123 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:53.123 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:53.123 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:53.123 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:53.123 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:53.123 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:53.123 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:53.125 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:53.222 INFO:tasks.rados.rados.0.smithi064.stdout:280: done (2 left) 2024-03-04T06:08:53.222 INFO:tasks.rados.rados.0.smithi064.stdout:281: write (excl) oid 6 current snap is 24 2024-03-04T06:08:53.222 INFO:tasks.rados.rados.0.smithi064.stdout:279: finishing write tid 1 to smithi06431743-50 2024-03-04T06:08:53.222 INFO:tasks.rados.rados.0.smithi064.stdout:281: seq_num 162 ranges {722734=633751,2063377=659357} 2024-03-04T06:08:53.229 INFO:tasks.rados.rados.0.smithi064.stdout:281: writing smithi06431743-6 from 722734 to 1356485 tid 1 2024-03-04T06:08:53.237 INFO:tasks.rados.rados.0.smithi064.stdout:281: writing smithi06431743-6 from 2063377 to 2722734 tid 2 2024-03-04T06:08:53.237 INFO:tasks.rados.rados.0.smithi064.stdout:282: write oid 7 current snap is 24 2024-03-04T06:08:53.237 INFO:tasks.rados.rados.0.smithi064.stdout:278: finishing write tid 1 to smithi06431743-37 2024-03-04T06:08:53.237 INFO:tasks.rados.rados.0.smithi064.stdout:279: finishing write tid 2 to smithi06431743-50 2024-03-04T06:08:53.237 INFO:tasks.rados.rados.0.smithi064.stdout:278: finishing write tid 2 to smithi06431743-37 2024-03-04T06:08:53.238 INFO:tasks.rados.rados.0.smithi064.stdout:282: seq_num 163 ranges {739541=709000,2121859=426376} 2024-03-04T06:08:53.245 INFO:tasks.rados.rados.0.smithi064.stdout:282: writing smithi06431743-7 from 739541 to 1448541 tid 1 2024-03-04T06:08:53.251 INFO:tasks.rados.rados.0.smithi064.stdout:282: writing smithi06431743-7 from 2121859 to 2548235 tid 2 2024-03-04T06:08:53.251 INFO:tasks.rados.rados.0.smithi064.stdout:283: snap_create 2024-03-04T06:08:53.251 INFO:tasks.rados.rados.0.smithi064.stdout:279: finishing write tid 3 to smithi06431743-50 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:279: finishing write tid 4 to smithi06431743-50 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:279: oid 50 updating version 0 to 65 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:279: oid 50 version 65 is already newer than 64 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:279: oid 50 version 65 is already newer than 63 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 65 (ObjNum 161 snap 24 seq_num 161) dirty exists 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:279: left oid 50 (ObjNum 161 snap 24 seq_num 161) 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:278: finishing write tid 3 to smithi06431743-37 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:278: finishing write tid 4 to smithi06431743-37 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:278: oid 37 updating version 0 to 27 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:278: oid 37 version 27 is already newer than 25 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:278: oid 37 version 27 is already newer than 26 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 27 (ObjNum 160 snap 24 seq_num 160) dirty exists 2024-03-04T06:08:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:278: left oid 37 (ObjNum 160 snap 24 seq_num 160) 2024-03-04T06:08:53.352 INFO:tasks.rados.rados.0.smithi064.stdout:282: finishing write tid 1 to smithi06431743-7 2024-03-04T06:08:53.359 INFO:tasks.rados.rados.0.smithi064.stdout:281: finishing write tid 1 to smithi06431743-6 2024-03-04T06:08:53.363 INFO:tasks.rados.rados.0.smithi064.stdout:282: finishing write tid 2 to smithi06431743-7 2024-03-04T06:08:53.371 INFO:tasks.rados.rados.0.smithi064.stdout:282: finishing write tid 3 to smithi06431743-7 2024-03-04T06:08:53.374 INFO:tasks.rados.rados.0.smithi064.stdout:282: finishing write tid 4 to smithi06431743-7 2024-03-04T06:08:53.374 INFO:tasks.rados.rados.0.smithi064.stdout:282: oid 7 updating version 0 to 33 2024-03-04T06:08:53.374 INFO:tasks.rados.rados.0.smithi064.stdout:282: oid 7 version 33 is already newer than 32 2024-03-04T06:08:53.374 INFO:tasks.rados.rados.0.smithi064.stdout:282: oid 7 version 33 is already newer than 31 2024-03-04T06:08:53.374 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 33 (ObjNum 163 snap 24 seq_num 163) dirty exists 2024-03-04T06:08:53.374 INFO:tasks.rados.rados.0.smithi064.stdout:282: left oid 7 (ObjNum 163 snap 24 seq_num 163) 2024-03-04T06:08:53.379 INFO:tasks.rados.rados.0.smithi064.stdout:281: finishing write tid 2 to smithi06431743-6 2024-03-04T06:08:53.382 INFO:tasks.rados.rados.0.smithi064.stdout:281: finishing write tid 3 to smithi06431743-6 2024-03-04T06:08:53.383 INFO:tasks.rados.rados.0.smithi064.stdout:281: finishing write tid 4 to smithi06431743-6 2024-03-04T06:08:53.383 INFO:tasks.rados.rados.0.smithi064.stdout:281: oid 6 updating version 0 to 35 2024-03-04T06:08:53.383 INFO:tasks.rados.rados.0.smithi064.stdout:281: oid 6 version 35 is already newer than 35 2024-03-04T06:08:53.383 INFO:tasks.rados.rados.0.smithi064.stdout:281: oid 6 version 35 is already newer than 33 2024-03-04T06:08:53.383 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 35 (ObjNum 162 snap 24 seq_num 162) dirty exists 2024-03-04T06:08:53.383 INFO:tasks.rados.rados.0.smithi064.stdout:281: left oid 6 (ObjNum 162 snap 24 seq_num 162) 2024-03-04T06:08:54.221 INFO:tasks.rados.rados.0.smithi064.stdout:278: done (4 left) 2024-03-04T06:08:54.221 INFO:tasks.rados.rados.0.smithi064.stdout:279: done (3 left) 2024-03-04T06:08:54.221 INFO:tasks.rados.rados.0.smithi064.stdout:281: done (2 left) 2024-03-04T06:08:54.221 INFO:tasks.rados.rados.0.smithi064.stdout:282: done (1 left) 2024-03-04T06:08:54.221 INFO:tasks.rados.rados.0.smithi064.stdout:283: done (0 left) 2024-03-04T06:08:54.221 INFO:tasks.rados.rados.0.smithi064.stdout:284: read oid 13 snap -1 2024-03-04T06:08:54.221 INFO:tasks.rados.rados.0.smithi064.stdout:284: expect (ObjNum 93 snap 7 seq_num 93) 2024-03-04T06:08:54.221 INFO:tasks.rados.rados.0.smithi064.stdout:285: snap_remove snap 24 2024-03-04T06:08:55.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:55.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:55.198+0000 7f21d595d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:55.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:55.203+0000 7f21d595d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:55.226 INFO:tasks.rados.rados.0.smithi064.stdout:285: done (1 left) 2024-03-04T06:08:55.226 INFO:tasks.rados.rados.0.smithi064.stdout:286: write oid 9 current snap is 25 2024-03-04T06:08:55.226 INFO:tasks.rados.rados.0.smithi064.stdout:286: seq_num 164 ranges {756348=784249,2063988=686387} 2024-03-04T06:08:55.234 INFO:tasks.rados.rados.0.smithi064.stdout:286: writing smithi06431743-9 from 756348 to 1540597 tid 1 2024-03-04T06:08:55.244 INFO:tasks.rados.rados.0.smithi064.stdout:286: writing smithi06431743-9 from 2063988 to 2750375 tid 2 2024-03-04T06:08:55.244 INFO:tasks.rados.rados.0.smithi064.stdout:287: delete oid 26 current snap is 25 2024-03-04T06:08:55.244 INFO:tasks.rados.rados.0.smithi064.stdout:284: expect (ObjNum 93 snap 7 seq_num 93) 2024-03-04T06:08:55.432 INFO:tasks.rados.rados.0.smithi064.stdout:286: finishing write tid 1 to smithi06431743-9 2024-03-04T06:08:55.432 INFO:tasks.rados.rados.0.smithi064.stdout:286: finishing write tid 2 to smithi06431743-9 2024-03-04T06:08:55.432 INFO:tasks.rados.rados.0.smithi064.stdout:286: finishing write tid 3 to smithi06431743-9 2024-03-04T06:08:55.432 INFO:tasks.rados.rados.0.smithi064.stdout:286: finishing write tid 4 to smithi06431743-9 2024-03-04T06:08:55.432 INFO:tasks.rados.rados.0.smithi064.stdout:286: oid 9 updating version 0 to 72 2024-03-04T06:08:55.433 INFO:tasks.rados.rados.0.smithi064.stdout:286: oid 9 version 72 is already newer than 71 2024-03-04T06:08:55.433 INFO:tasks.rados.rados.0.smithi064.stdout:286: oid 9 updating version 72 to 73 2024-03-04T06:08:55.433 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 73 (ObjNum 164 snap 25 seq_num 164) dirty exists 2024-03-04T06:08:55.433 INFO:tasks.rados.rados.0.smithi064.stdout:286: left oid 9 (ObjNum 164 snap 25 seq_num 164) 2024-03-04T06:08:55.433 INFO:tasks.rados.rados.0.smithi064.stdout:284: done (2 left) 2024-03-04T06:08:55.433 INFO:tasks.rados.rados.0.smithi064.stdout:286: done (1 left) 2024-03-04T06:08:55.433 INFO:tasks.rados.rados.0.smithi064.stdout:287: done (0 left) 2024-03-04T06:08:55.433 INFO:tasks.rados.rados.0.smithi064.stdout:288: read oid 31 snap 23 2024-03-04T06:08:55.433 INFO:tasks.rados.rados.0.smithi064.stdout:288: expect (ObjNum 30 snap 0 seq_num 30) 2024-03-04T06:08:55.433 INFO:tasks.rados.rados.0.smithi064.stdout:289: snap_create 2024-03-04T06:08:55.450 INFO:tasks.rados.rados.0.smithi064.stdout:288: expect (ObjNum 30 snap 0 seq_num 30) 2024-03-04T06:08:56.226 INFO:tasks.rados.rados.0.smithi064.stdout:288: done (1 left) 2024-03-04T06:08:56.226 INFO:tasks.rados.rados.0.smithi064.stdout:289: done (0 left) 2024-03-04T06:08:56.226 INFO:tasks.rados.rados.0.smithi064.stdout:290: write (excl) oid 7 current snap is 26 2024-03-04T06:08:56.226 INFO:tasks.rados.rados.0.smithi064.stdout:290: seq_num 166 ranges {789962=534747,1780952=690056} 2024-03-04T06:08:56.233 INFO:tasks.rados.rados.0.smithi064.stdout:290: writing smithi06431743-7 from 789962 to 1324709 tid 1 2024-03-04T06:08:56.241 INFO:tasks.rados.rados.0.smithi064.stdout:290: writing smithi06431743-7 from 1780952 to 2471008 tid 2 2024-03-04T06:08:56.241 INFO:tasks.rados.rados.0.smithi064.stdout:291: read oid 50 snap -1 2024-03-04T06:08:56.241 INFO:tasks.rados.rados.0.smithi064.stdout:291: expect (ObjNum 161 snap 24 seq_num 161) 2024-03-04T06:08:56.242 INFO:tasks.rados.rados.0.smithi064.stdout:292: read oid 36 snap -1 2024-03-04T06:08:56.242 INFO:tasks.rados.rados.0.smithi064.stdout:292: expect deleted 2024-03-04T06:08:56.242 INFO:tasks.rados.rados.0.smithi064.stdout:293: read oid 20 snap 23 2024-03-04T06:08:56.242 INFO:tasks.rados.rados.0.smithi064.stdout:293: expect (ObjNum 153 snap 21 seq_num 153) 2024-03-04T06:08:56.242 INFO:tasks.rados.rados.0.smithi064.stdout:294: delete oid 23 current snap is 26 2024-03-04T06:08:56.257 INFO:tasks.rados.rados.0.smithi064.stdout:294: done (4 left) 2024-03-04T06:08:56.257 INFO:tasks.rados.rados.0.smithi064.stdout:295: write oid 3 current snap is 26 2024-03-04T06:08:56.257 INFO:tasks.rados.rados.0.smithi064.stdout:295: seq_num 168 ranges {423576=401598,1497916=693725} 2024-03-04T06:08:56.261 INFO:tasks.rados.rados.0.smithi064.stdout:295: writing smithi06431743-3 from 423576 to 825174 tid 1 2024-03-04T06:08:56.270 INFO:tasks.rados.rados.0.smithi064.stdout:295: writing smithi06431743-3 from 1497916 to 2191641 tid 2 2024-03-04T06:08:56.270 INFO:tasks.rados.rados.0.smithi064.stdout:296: write oid 9 current snap is 26 2024-03-04T06:08:56.270 INFO:tasks.rados.rados.0.smithi064.stdout:296: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-03-04T06:08:56.275 INFO:tasks.rados.rados.0.smithi064.stdout:296: writing smithi06431743-9 from 440383 to 917230 tid 1 2024-03-04T06:08:56.283 INFO:tasks.rados.rados.0.smithi064.stdout:296: writing smithi06431743-9 from 1556398 to 1993781 tid 2 2024-03-04T06:08:56.288 INFO:tasks.rados.rados.0.smithi064.stdout:296: writing smithi06431743-9 from 2474721 to 2840383 tid 3 2024-03-04T06:08:56.288 INFO:tasks.rados.rados.0.smithi064.stdout:297: snap_create 2024-03-04T06:08:56.306 INFO:tasks.rados.rados.0.smithi064.stdout:291: expect (ObjNum 161 snap 24 seq_num 161) 2024-03-04T06:08:56.426 INFO:tasks.rados.rados.0.smithi064.stdout:295: finishing write tid 1 to smithi06431743-3 2024-03-04T06:08:56.426 INFO:tasks.rados.rados.0.smithi064.stdout:290: finishing write tid 1 to smithi06431743-7 2024-03-04T06:08:56.426 INFO:tasks.rados.rados.0.smithi064.stdout:293: expect (ObjNum 153 snap 21 seq_num 153) 2024-03-04T06:08:56.557 INFO:tasks.rados.rados.0.smithi064.stdout:290: finishing write tid 2 to smithi06431743-7 2024-03-04T06:08:56.557 INFO:tasks.rados.rados.0.smithi064.stdout:295: finishing write tid 2 to smithi06431743-3 2024-03-04T06:08:56.557 INFO:tasks.rados.rados.0.smithi064.stdout:290: finishing write tid 3 to smithi06431743-7 2024-03-04T06:08:56.557 INFO:tasks.rados.rados.0.smithi064.stdout:290: finishing write tid 4 to smithi06431743-7 2024-03-04T06:08:56.557 INFO:tasks.rados.rados.0.smithi064.stdout:290: oid 7 updating version 0 to 37 2024-03-04T06:08:56.557 INFO:tasks.rados.rados.0.smithi064.stdout:290: oid 7 version 37 is already newer than 35 2024-03-04T06:08:56.557 INFO:tasks.rados.rados.0.smithi064.stdout:290: oid 7 version 37 is already newer than 37 2024-03-04T06:08:56.557 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 37 (ObjNum 166 snap 26 seq_num 166) dirty exists 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:290: left oid 7 (ObjNum 166 snap 26 seq_num 166) 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:296: finishing write tid 1 to smithi06431743-9 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:295: finishing write tid 3 to smithi06431743-3 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:295: finishing write tid 4 to smithi06431743-3 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:295: oid 3 updating version 0 to 35 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:295: oid 3 updating version 35 to 37 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:295: oid 3 version 37 is already newer than 36 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 37 (ObjNum 168 snap 26 seq_num 168) dirty exists 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:295: left oid 3 (ObjNum 168 snap 26 seq_num 168) 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:296: finishing write tid 2 to smithi06431743-9 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:296: finishing write tid 3 to smithi06431743-9 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:296: finishing write tid 4 to smithi06431743-9 2024-03-04T06:08:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:296: finishing write tid 5 to smithi06431743-9 2024-03-04T06:08:56.559 INFO:tasks.rados.rados.0.smithi064.stdout:296: oid 9 updating version 0 to 78 2024-03-04T06:08:56.559 INFO:tasks.rados.rados.0.smithi064.stdout:296: oid 9 version 78 is already newer than 78 2024-03-04T06:08:56.559 INFO:tasks.rados.rados.0.smithi064.stdout:296: oid 9 version 78 is already newer than 75 2024-03-04T06:08:56.559 INFO:tasks.rados.rados.0.smithi064.stdout:296: oid 9 version 78 is already newer than 76 2024-03-04T06:08:56.559 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 78 (ObjNum 169 snap 26 seq_num 169) dirty exists 2024-03-04T06:08:56.559 INFO:tasks.rados.rados.0.smithi064.stdout:296: left oid 9 (ObjNum 169 snap 26 seq_num 169) 2024-03-04T06:08:57.234 INFO:tasks.rados.rados.0.smithi064.stdout:290: done (6 left) 2024-03-04T06:08:57.234 INFO:tasks.rados.rados.0.smithi064.stdout:291: done (5 left) 2024-03-04T06:08:57.234 INFO:tasks.rados.rados.0.smithi064.stdout:292: done (4 left) 2024-03-04T06:08:57.234 INFO:tasks.rados.rados.0.smithi064.stdout:293: done (3 left) 2024-03-04T06:08:57.234 INFO:tasks.rados.rados.0.smithi064.stdout:295: done (2 left) 2024-03-04T06:08:57.234 INFO:tasks.rados.rados.0.smithi064.stdout:296: done (1 left) 2024-03-04T06:08:57.234 INFO:tasks.rados.rados.0.smithi064.stdout:297: done (0 left) 2024-03-04T06:08:57.234 INFO:tasks.rados.rados.0.smithi064.stdout:298: read oid 24 snap 25 2024-03-04T06:08:57.235 INFO:tasks.rados.rados.0.smithi064.stdout:298: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-04T06:08:57.235 INFO:tasks.rados.rados.0.smithi064.stdout:299: snap_remove snap 23 2024-03-04T06:08:57.349 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:08:57.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:57.459+0000 7f009fd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:57.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:57.463+0000 7f009fd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:58.126 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:08:58.126 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:08:58.127 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:08:58.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:08:58.180 INFO:tasks.rados.rados.0.smithi064.stdout:299: done (1 left) 2024-03-04T06:08:58.180 INFO:tasks.rados.rados.0.smithi064.stdout:300: read oid 6 snap -1 2024-03-04T06:08:58.181 INFO:tasks.rados.rados.0.smithi064.stdout:300: expect (ObjNum 162 snap 24 seq_num 162) 2024-03-04T06:08:58.181 INFO:tasks.rados.rados.0.smithi064.stdout:301: snap_remove snap 26 2024-03-04T06:08:58.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:58.245+0000 7f8402b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:58.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:58.252+0000 7f8402b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:58.265 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:58.265 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:58.265 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:58.265 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:58.265 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:58.265 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:58.265 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:58.265 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:58.265 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:58.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:58.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:58.266 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:58.267 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:58.268 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:58.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:08:58.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:58.373+0000 7f53956ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:58.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:58.380+0000 7f53956ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:58.393 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:58.396 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:58.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:08:58.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:58.498+0000 7f47428ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:58.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:58.506+0000 7f47428ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:58.519 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:58.520 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:58.521 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:58.523 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:58.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:08:58.631 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:58.626+0000 7feb6e39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:58.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:58.633+0000 7feb6e39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:58.646 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:08:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:08:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:08:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:08:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:08:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:08:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:08:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:08:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:08:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:08:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:08:58.647 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:08:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:08:58.650 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:08:59.181 INFO:tasks.rados.rados.0.smithi064.stdout:301: done (2 left) 2024-03-04T06:08:59.181 INFO:tasks.rados.rados.0.smithi064.stdout:302: write oid 31 current snap is 27 2024-03-04T06:08:59.181 INFO:tasks.rados.rados.0.smithi064.stdout:302: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-03-04T06:08:59.186 INFO:tasks.rados.rados.0.smithi064.stdout:302: writing smithi06431743-31 from 457190 to 1009286 tid 1 2024-03-04T06:08:59.193 INFO:tasks.rados.rados.0.smithi064.stdout:302: writing smithi06431743-31 from 1614880 to 2195921 tid 2 2024-03-04T06:08:59.198 INFO:tasks.rados.rados.0.smithi064.stdout:302: writing smithi06431743-31 from 2785791 to 2857190 tid 3 2024-03-04T06:08:59.198 INFO:tasks.rados.rados.0.smithi064.stdout:303: read oid 39 snap -1 2024-03-04T06:08:59.198 INFO:tasks.rados.rados.0.smithi064.stdout:303: expect deleted 2024-03-04T06:08:59.199 INFO:tasks.rados.rados.0.smithi064.stdout:304: write (excl) oid 17 current snap is 27 2024-03-04T06:08:59.199 INFO:tasks.rados.rados.0.smithi064.stdout:304: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-03-04T06:08:59.204 INFO:tasks.rados.rados.0.smithi064.stdout:304: writing smithi06431743-17 from 473997 to 1101342 tid 1 2024-03-04T06:08:59.211 INFO:tasks.rados.rados.0.smithi064.stdout:304: writing smithi06431743-17 from 1673362 to 2114414 tid 2 2024-03-04T06:08:59.216 INFO:tasks.rados.rados.0.smithi064.stdout:304: writing smithi06431743-17 from 2529567 to 2873997 tid 3 2024-03-04T06:08:59.216 INFO:tasks.rados.rados.0.smithi064.stdout:305: snap_create 2024-03-04T06:08:59.216 INFO:tasks.rados.rados.0.smithi064.stdout:298: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-04T06:08:59.318 INFO:tasks.rados.rados.0.smithi064.stdout:300: expect (ObjNum 162 snap 24 seq_num 162) 2024-03-04T06:08:59.461 INFO:tasks.rados.rados.0.smithi064.stdout:304: finishing write tid 1 to smithi06431743-17 2024-03-04T06:08:59.461 INFO:tasks.rados.rados.0.smithi064.stdout:302: finishing write tid 1 to smithi06431743-31 2024-03-04T06:08:59.461 INFO:tasks.rados.rados.0.smithi064.stdout:302: finishing write tid 2 to smithi06431743-31 2024-03-04T06:08:59.461 INFO:tasks.rados.rados.0.smithi064.stdout:304: finishing write tid 2 to smithi06431743-17 2024-03-04T06:08:59.461 INFO:tasks.rados.rados.0.smithi064.stdout:302: finishing write tid 3 to smithi06431743-31 2024-03-04T06:08:59.461 INFO:tasks.rados.rados.0.smithi064.stdout:302: finishing write tid 4 to smithi06431743-31 2024-03-04T06:08:59.461 INFO:tasks.rados.rados.0.smithi064.stdout:304: finishing write tid 3 to smithi06431743-17 2024-03-04T06:08:59.461 INFO:tasks.rados.rados.0.smithi064.stdout:302: finishing write tid 5 to smithi06431743-31 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:302: oid 31 updating version 0 to 16 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:302: oid 31 version 16 is already newer than 14 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:302: oid 31 version 16 is already newer than 16 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:302: oid 31 version 16 is already newer than 14 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 16 (ObjNum 170 snap 27 seq_num 170) dirty exists 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:302: left oid 31 (ObjNum 170 snap 27 seq_num 170) 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:304: finishing write tid 4 to smithi06431743-17 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:304: finishing write tid 5 to smithi06431743-17 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:304: oid 17 updating version 0 to 44 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:304: oid 17 updating version 44 to 46 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:304: oid 17 updating version 46 to 47 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:304: oid 17 version 47 is already newer than 46 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 47 (ObjNum 171 snap 27 seq_num 171) dirty exists 2024-03-04T06:08:59.462 INFO:tasks.rados.rados.0.smithi064.stdout:304: left oid 17 (ObjNum 171 snap 27 seq_num 171) 2024-03-04T06:08:59.533 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:08:59.641 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:59.638+0000 7ff85f95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:08:59.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:08:59.642+0000 7ff85f95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:00.182 INFO:tasks.rados.rados.0.smithi064.stdout:298: done (5 left) 2024-03-04T06:09:00.182 INFO:tasks.rados.rados.0.smithi064.stdout:300: done (4 left) 2024-03-04T06:09:00.182 INFO:tasks.rados.rados.0.smithi064.stdout:302: done (3 left) 2024-03-04T06:09:00.182 INFO:tasks.rados.rados.0.smithi064.stdout:303: done (2 left) 2024-03-04T06:09:00.182 INFO:tasks.rados.rados.0.smithi064.stdout:304: done (1 left) 2024-03-04T06:09:00.182 INFO:tasks.rados.rados.0.smithi064.stdout:305: done (0 left) 2024-03-04T06:09:00.182 INFO:tasks.rados.rados.0.smithi064.stdout:306: read oid 16 snap -1 2024-03-04T06:09:00.182 INFO:tasks.rados.rados.0.smithi064.stdout:306: expect deleted 2024-03-04T06:09:00.183 INFO:tasks.rados.rados.0.smithi064.stdout:307: delete oid 37 current snap is 28 2024-03-04T06:09:00.276 INFO:tasks.rados.rados.0.smithi064.stdout:306: done (1 left) 2024-03-04T06:09:00.276 INFO:tasks.rados.rados.0.smithi064.stdout:307: done (0 left) 2024-03-04T06:09:00.276 INFO:tasks.rados.rados.0.smithi064.stdout:308: snap_remove snap 27 2024-03-04T06:09:01.186 INFO:tasks.rados.rados.0.smithi064.stdout:308: done (0 left) 2024-03-04T06:09:01.187 INFO:tasks.rados.rados.0.smithi064.stdout:309: snap_create 2024-03-04T06:09:01.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:01.830 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:01.828+0000 7fe4cb154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:01.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:01.832+0000 7fe4cb154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:02.186 INFO:tasks.rados.rados.0.smithi064.stdout:309: done (0 left) 2024-03-04T06:09:02.186 INFO:tasks.rados.rados.0.smithi064.stdout:310: write oid 32 current snap is 29 2024-03-04T06:09:02.186 INFO:tasks.rados.rados.0.smithi064.stdout:310: seq_num 173 ranges {507611=777843,2073973=444721} 2024-03-04T06:09:02.195 INFO:tasks.rados.rados.0.smithi064.stdout:310: writing smithi06431743-32 from 507611 to 1285454 tid 1 2024-03-04T06:09:02.203 INFO:tasks.rados.rados.0.smithi064.stdout:310: writing smithi06431743-32 from 2073973 to 2518694 tid 2 2024-03-04T06:09:02.203 INFO:tasks.rados.rados.0.smithi064.stdout:311: read oid 36 snap 28 2024-03-04T06:09:02.203 INFO:tasks.rados.rados.0.smithi064.stdout:311: expect deleted 2024-03-04T06:09:02.203 INFO:tasks.rados.rados.0.smithi064.stdout:312: snap_create 2024-03-04T06:09:02.275 INFO:tasks.rados.rados.0.smithi064.stdout:310: finishing write tid 1 to smithi06431743-32 2024-03-04T06:09:02.290 INFO:tasks.rados.rados.0.smithi064.stdout:310: finishing write tid 2 to smithi06431743-32 2024-03-04T06:09:02.290 INFO:tasks.rados.rados.0.smithi064.stdout:310: finishing write tid 3 to smithi06431743-32 2024-03-04T06:09:02.292 INFO:tasks.rados.rados.0.smithi064.stdout:310: finishing write tid 4 to smithi06431743-32 2024-03-04T06:09:02.292 INFO:tasks.rados.rados.0.smithi064.stdout:310: oid 32 updating version 0 to 129 2024-03-04T06:09:02.292 INFO:tasks.rados.rados.0.smithi064.stdout:310: oid 32 version 129 is already newer than 129 2024-03-04T06:09:02.292 INFO:tasks.rados.rados.0.smithi064.stdout:310: oid 32 version 129 is already newer than 127 2024-03-04T06:09:02.292 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 129 (ObjNum 173 snap 29 seq_num 173) dirty exists 2024-03-04T06:09:02.292 INFO:tasks.rados.rados.0.smithi064.stdout:310: left oid 32 (ObjNum 173 snap 29 seq_num 173) 2024-03-04T06:09:03.200 INFO:tasks.rados.rados.0.smithi064.stdout:310: done (2 left) 2024-03-04T06:09:03.200 INFO:tasks.rados.rados.0.smithi064.stdout:311: done (1 left) 2024-03-04T06:09:03.200 INFO:tasks.rados.rados.0.smithi064.stdout:312: done (0 left) 2024-03-04T06:09:03.200 INFO:tasks.rados.rados.0.smithi064.stdout:313: read oid 23 snap -1 2024-03-04T06:09:03.200 INFO:tasks.rados.rados.0.smithi064.stdout:313: expect deleted 2024-03-04T06:09:03.200 INFO:tasks.rados.rados.0.smithi064.stdout:314: snap_create 2024-03-04T06:09:03.650 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:09:03.651 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:09:03.651 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:09:03.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:09:03.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:03.768+0000 7f4a60cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:03.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:03.776+0000 7f4a60cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:03.789 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:03.789 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:03.789 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:03.790 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:03.790 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:03.790 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:03.790 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:03.790 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:03.790 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:03.790 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:03.790 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:03.790 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:03.792 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:03.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:09:03.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:03.896+0000 7f5f5926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:03.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:03.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:03.904+0000 7f5f5926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:03.917 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:03.917 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:03.917 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:03.917 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:03.917 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:03.917 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:03.917 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:03.917 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:03.917 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:03.917 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:03.918 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:03.918 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:03.920 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:03.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:09:04.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:04.008+0000 7f2856d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:04.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:04.013+0000 7f2856d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:04.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:04.071+0000 7fb85859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:04.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:04.078+0000 7fb85859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:04.092 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:04.094 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:04.095 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:04.095 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:09:04.198 INFO:tasks.rados.rados.0.smithi064.stdout:313: done (1 left) 2024-03-04T06:09:04.198 INFO:tasks.rados.rados.0.smithi064.stdout:314: done (0 left) 2024-03-04T06:09:04.199 INFO:tasks.rados.rados.0.smithi064.stdout:315: snap_remove snap 30 2024-03-04T06:09:04.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:04.245+0000 7f778395d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:04.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:04.252+0000 7f778395d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:04.265 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:04.265 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:04.265 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:04.265 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:04.265 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:04.265 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:04.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:04.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:04.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:04.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:04.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:04.266 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:04.268 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:04.268 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:05.210 INFO:tasks.rados.rados.0.smithi064.stdout:315: done (0 left) 2024-03-04T06:09:05.210 INFO:tasks.rados.rados.0.smithi064.stdout:316: snap_create 2024-03-04T06:09:06.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:06.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:06.204+0000 7fdd72cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:06.209 INFO:tasks.rados.rados.0.smithi064.stdout:316: done (0 left) 2024-03-04T06:09:06.209 INFO:tasks.rados.rados.0.smithi064.stdout:317: read oid 34 snap -1 2024-03-04T06:09:06.209 INFO:tasks.rados.rados.0.smithi064.stdout:317: expect (ObjNum 127 snap 15 seq_num 127) 2024-03-04T06:09:06.209 INFO:tasks.rados.rados.0.smithi064.stdout:318: delete oid 2 current snap is 32 2024-03-04T06:09:06.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:06.210+0000 7fdd72cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:06.251 INFO:tasks.rados.rados.0.smithi064.stdout:317: expect (ObjNum 127 snap 15 seq_num 127) 2024-03-04T06:09:06.341 INFO:tasks.rados.rados.0.smithi064.stdout:317: done (1 left) 2024-03-04T06:09:06.341 INFO:tasks.rados.rados.0.smithi064.stdout:318: done (0 left) 2024-03-04T06:09:06.341 INFO:tasks.rados.rados.0.smithi064.stdout:319: snap_create 2024-03-04T06:09:07.218 INFO:tasks.rados.rados.0.smithi064.stdout:319: done (0 left) 2024-03-04T06:09:07.218 INFO:tasks.rados.rados.0.smithi064.stdout:320: read oid 2 snap -1 2024-03-04T06:09:07.218 INFO:tasks.rados.rados.0.smithi064.stdout:320: expect deleted 2024-03-04T06:09:07.218 INFO:tasks.rados.rados.0.smithi064.stdout:321: write (excl) oid 16 current snap is 33 2024-03-04T06:09:07.218 INFO:tasks.rados.rados.0.smithi064.stdout:321: seq_num 175 ranges {541225=644694,1907290=448390} 2024-03-04T06:09:07.224 INFO:tasks.rados.rados.0.smithi064.stdout:321: writing smithi06431743-16 from 541225 to 1185919 tid 1 2024-03-04T06:09:07.231 INFO:tasks.rados.rados.0.smithi064.stdout:321: writing smithi06431743-16 from 1907290 to 2355680 tid 2 2024-03-04T06:09:07.231 INFO:tasks.rados.rados.0.smithi064.stdout:322: write oid 13 current snap is 33 2024-03-04T06:09:07.231 INFO:tasks.rados.rados.0.smithi064.stdout:322: seq_num 176 ranges {558032=719943,1849419=592048} 2024-03-04T06:09:07.240 INFO:tasks.rados.rados.0.smithi064.stdout:322: writing smithi06431743-13 from 558032 to 1277975 tid 1 2024-03-04T06:09:07.247 INFO:tasks.rados.rados.0.smithi064.stdout:322: writing smithi06431743-13 from 1849419 to 2441467 tid 2 2024-03-04T06:09:07.247 INFO:tasks.rados.rados.0.smithi064.stdout:323: delete oid 22 current snap is 33 2024-03-04T06:09:07.286 INFO:tasks.rados.rados.0.smithi064.stdout:321: finishing write tid 1 to smithi06431743-16 2024-03-04T06:09:07.303 INFO:tasks.rados.rados.0.smithi064.stdout:322: finishing write tid 1 to smithi06431743-13 2024-03-04T06:09:07.309 INFO:tasks.rados.rados.0.smithi064.stdout:320: done (3 left) 2024-03-04T06:09:07.309 INFO:tasks.rados.rados.0.smithi064.stdout:323: done (2 left) 2024-03-04T06:09:07.310 INFO:tasks.rados.rados.0.smithi064.stdout:324: delete oid 33 current snap is 33 2024-03-04T06:09:07.324 INFO:tasks.rados.rados.0.smithi064.stdout:322: finishing write tid 2 to smithi06431743-13 2024-03-04T06:09:07.324 INFO:tasks.rados.rados.0.smithi064.stdout:321: finishing write tid 2 to smithi06431743-16 2024-03-04T06:09:07.329 INFO:tasks.rados.rados.0.smithi064.stdout:324: done (2 left) 2024-03-04T06:09:07.329 INFO:tasks.rados.rados.0.smithi064.stdout:325: delete oid 28 current snap is 33 2024-03-04T06:09:07.331 INFO:tasks.rados.rados.0.smithi064.stdout:321: finishing write tid 3 to smithi06431743-16 2024-03-04T06:09:07.333 INFO:tasks.rados.rados.0.smithi064.stdout:321: finishing write tid 4 to smithi06431743-16 2024-03-04T06:09:07.333 INFO:tasks.rados.rados.0.smithi064.stdout:321: oid 16 updating version 0 to 35 2024-03-04T06:09:07.333 INFO:tasks.rados.rados.0.smithi064.stdout:321: oid 16 updating version 35 to 36 2024-03-04T06:09:07.333 INFO:tasks.rados.rados.0.smithi064.stdout:321: oid 16 updating version 36 to 37 2024-03-04T06:09:07.333 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 37 (ObjNum 175 snap 33 seq_num 175) dirty exists 2024-03-04T06:09:07.333 INFO:tasks.rados.rados.0.smithi064.stdout:321: left oid 16 (ObjNum 175 snap 33 seq_num 175) 2024-03-04T06:09:07.333 INFO:tasks.rados.rados.0.smithi064.stdout:322: finishing write tid 3 to smithi06431743-13 2024-03-04T06:09:07.334 INFO:tasks.rados.rados.0.smithi064.stdout:322: finishing write tid 4 to smithi06431743-13 2024-03-04T06:09:07.334 INFO:tasks.rados.rados.0.smithi064.stdout:322: oid 13 updating version 0 to 64 2024-03-04T06:09:07.334 INFO:tasks.rados.rados.0.smithi064.stdout:322: oid 13 updating version 64 to 65 2024-03-04T06:09:07.334 INFO:tasks.rados.rados.0.smithi064.stdout:322: oid 13 updating version 65 to 66 2024-03-04T06:09:07.334 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 66 (ObjNum 176 snap 33 seq_num 176) dirty exists 2024-03-04T06:09:07.334 INFO:tasks.rados.rados.0.smithi064.stdout:322: left oid 13 (ObjNum 176 snap 33 seq_num 176) 2024-03-04T06:09:07.358 INFO:tasks.rados.rados.0.smithi064.stdout:321: done (2 left) 2024-03-04T06:09:07.358 INFO:tasks.rados.rados.0.smithi064.stdout:322: done (1 left) 2024-03-04T06:09:07.358 INFO:tasks.rados.rados.0.smithi064.stdout:325: done (0 left) 2024-03-04T06:09:07.358 INFO:tasks.rados.rados.0.smithi064.stdout:326: write oid 26 current snap is 33 2024-03-04T06:09:07.358 INFO:tasks.rados.rados.0.smithi064.stdout:326: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-03-04T06:09:07.365 INFO:tasks.rados.rados.0.smithi064.stdout:326: writing smithi06431743-26 from 625260 to 1246199 tid 1 2024-03-04T06:09:07.370 INFO:tasks.rados.rados.0.smithi064.stdout:326: writing smithi06431743-26 from 1683347 to 2282733 tid 2 2024-03-04T06:09:07.374 INFO:tasks.rados.rados.0.smithi064.stdout:326: writing smithi06431743-26 from 2943668 to 3025260 tid 3 2024-03-04T06:09:07.375 INFO:tasks.rados.rados.0.smithi064.stdout:327: snap_remove snap 29 2024-03-04T06:09:08.180 INFO:tasks.rados.rados.0.smithi064.stdout:327: done (1 left) 2024-03-04T06:09:08.180 INFO:tasks.rados.rados.0.smithi064.stdout:328: snap_create 2024-03-04T06:09:08.180 INFO:tasks.rados.rados.0.smithi064.stdout:326: finishing write tid 1 to smithi06431743-26 2024-03-04T06:09:08.180 INFO:tasks.rados.rados.0.smithi064.stdout:326: finishing write tid 2 to smithi06431743-26 2024-03-04T06:09:08.181 INFO:tasks.rados.rados.0.smithi064.stdout:326: finishing write tid 3 to smithi06431743-26 2024-03-04T06:09:08.181 INFO:tasks.rados.rados.0.smithi064.stdout:326: finishing write tid 4 to smithi06431743-26 2024-03-04T06:09:08.181 INFO:tasks.rados.rados.0.smithi064.stdout:326: finishing write tid 5 to smithi06431743-26 2024-03-04T06:09:08.181 INFO:tasks.rados.rados.0.smithi064.stdout:326: oid 26 updating version 0 to 131 2024-03-04T06:09:08.181 INFO:tasks.rados.rados.0.smithi064.stdout:326: oid 26 updating version 131 to 134 2024-03-04T06:09:08.181 INFO:tasks.rados.rados.0.smithi064.stdout:326: oid 26 version 134 is already newer than 133 2024-03-04T06:09:08.181 INFO:tasks.rados.rados.0.smithi064.stdout:326: oid 26 version 134 is already newer than 132 2024-03-04T06:09:08.181 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 134 (ObjNum 180 snap 33 seq_num 180) dirty exists 2024-03-04T06:09:08.182 INFO:tasks.rados.rados.0.smithi064.stdout:326: left oid 26 (ObjNum 180 snap 33 seq_num 180) 2024-03-04T06:09:08.304 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:08.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:08.429+0000 7f6c9586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:08.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:08.433+0000 7f6c9586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:09.184 INFO:tasks.rados.rados.0.smithi064.stdout:326: done (1 left) 2024-03-04T06:09:09.184 INFO:tasks.rados.rados.0.smithi064.stdout:328: done (0 left) 2024-03-04T06:09:09.184 INFO:tasks.rados.rados.0.smithi064.stdout:329: write (excl) oid 29 current snap is 34 2024-03-04T06:09:09.184 INFO:tasks.rados.rados.0.smithi064.stdout:329: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-03-04T06:09:09.190 INFO:tasks.rados.rados.0.smithi064.stdout:329: writing smithi06431743-29 from 642067 to 1338255 tid 1 2024-03-04T06:09:09.196 INFO:tasks.rados.rados.0.smithi064.stdout:329: writing smithi06431743-29 from 2025476 to 2484873 tid 2 2024-03-04T06:09:09.199 INFO:tasks.rados.rados.0.smithi064.stdout:329: writing smithi06431743-29 from 2971091 to 3042067 tid 3 2024-03-04T06:09:09.199 INFO:tasks.rados.rados.0.smithi064.stdout:330: snap_remove snap 25 2024-03-04T06:09:09.269 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:09:09.269 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:09:09.269 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:09:09.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:09:09.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:09.380+0000 7f568be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:09.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:09.388+0000 7f568be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:09.401 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:09.404 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:09.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:09:09.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:09.507+0000 7ff129424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:09.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:09.514+0000 7ff129424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:09.527 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:09.527 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:09.528 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:09.528 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:09.528 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:09.528 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:09.528 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:09.528 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:09.528 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:09.528 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:09.528 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:09.528 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:09.530 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:09.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:09:09.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:09.633+0000 7f8993754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:09.643 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:09.641+0000 7f8993754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:09.658 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:09.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:09:09.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:09.760+0000 7f1bd98dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:09.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:09.767+0000 7f1bd98dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:09.781 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:09.781 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:09.781 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:09.781 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:09.781 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:09.781 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:09.781 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:09.781 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:09.782 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:09.782 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:09.782 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:09.782 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:09.784 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:10.183 INFO:tasks.rados.rados.0.smithi064.stdout:330: done (1 left) 2024-03-04T06:09:10.184 INFO:tasks.rados.rados.0.smithi064.stdout:331: write (excl) oid 28 current snap is 34 2024-03-04T06:09:10.184 INFO:tasks.rados.rados.0.smithi064.stdout:331: seq_num 182 ranges {658874=771437,2083958=603055} 2024-03-04T06:09:10.193 INFO:tasks.rados.rados.0.smithi064.stdout:331: writing smithi06431743-28 from 658874 to 1430311 tid 1 2024-03-04T06:09:10.202 INFO:tasks.rados.rados.0.smithi064.stdout:331: writing smithi06431743-28 from 2083958 to 2687013 tid 2 2024-03-04T06:09:10.203 INFO:tasks.rados.rados.0.smithi064.stdout:332: write oid 19 current snap is 34 2024-03-04T06:09:10.203 INFO:tasks.rados.rados.0.smithi064.stdout:329: finishing write tid 1 to smithi06431743-29 2024-03-04T06:09:10.203 INFO:tasks.rados.rados.0.smithi064.stdout:332: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-03-04T06:09:10.211 INFO:tasks.rados.rados.0.smithi064.stdout:332: writing smithi06431743-19 from 675681 to 1238720 tid 1 2024-03-04T06:09:10.220 INFO:tasks.rados.rados.0.smithi064.stdout:332: writing smithi06431743-19 from 1858793 to 2605506 tid 2 2024-03-04T06:09:10.223 INFO:tasks.rados.rados.0.smithi064.stdout:332: writing smithi06431743-19 from 3025937 to 3075681 tid 3 2024-03-04T06:09:10.223 INFO:tasks.rados.rados.0.smithi064.stdout:333: read oid 41 snap 32 2024-03-04T06:09:10.223 INFO:tasks.rados.rados.0.smithi064.stdout:333: expect (ObjNum 113 snap 10 seq_num 113) 2024-03-04T06:09:10.223 INFO:tasks.rados.rados.0.smithi064.stdout:334: snap_remove snap 33 2024-03-04T06:09:10.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:10.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:10.606+0000 7f51d535d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:10.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:10.610+0000 7f51d535d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:11.223 INFO:tasks.rados.rados.0.smithi064.stdout:334: done (4 left) 2024-03-04T06:09:11.224 INFO:tasks.rados.rados.0.smithi064.stdout:335: write oid 18 current snap is 34 2024-03-04T06:09:11.224 INFO:tasks.rados.rados.0.smithi064.stdout:329: finishing write tid 2 to smithi06431743-29 2024-03-04T06:09:11.224 INFO:tasks.rados.rados.0.smithi064.stdout:335: seq_num 184 ranges {692488=638288,1917275=606724} 2024-03-04T06:09:11.230 INFO:tasks.rados.rados.0.smithi064.stdout:335: writing smithi06431743-18 from 692488 to 1330776 tid 1 2024-03-04T06:09:11.237 INFO:tasks.rados.rados.0.smithi064.stdout:335: writing smithi06431743-18 from 1917275 to 2523999 tid 2 2024-03-04T06:09:11.238 INFO:tasks.rados.rados.0.smithi064.stdout:336: read oid 47 snap -1 2024-03-04T06:09:11.238 INFO:tasks.rados.rados.0.smithi064.stdout:336: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-04T06:09:11.238 INFO:tasks.rados.rados.0.smithi064.stdout:337: write (excl) oid 17 current snap is 34 2024-03-04T06:09:11.238 INFO:tasks.rados.rados.0.smithi064.stdout:337: seq_num 185 ranges {709295=713537,1859404=750382} 2024-03-04T06:09:11.245 INFO:tasks.rados.rados.0.smithi064.stdout:337: writing smithi06431743-17 from 709295 to 1422832 tid 1 2024-03-04T06:09:11.254 INFO:tasks.rados.rados.0.smithi064.stdout:337: writing smithi06431743-17 from 1859404 to 2609786 tid 2 2024-03-04T06:09:11.254 INFO:tasks.rados.rados.0.smithi064.stdout:338: snap_remove snap 28 2024-03-04T06:09:12.227 INFO:tasks.rados.rados.0.smithi064.stdout:338: done (7 left) 2024-03-04T06:09:12.227 INFO:tasks.rados.rados.0.smithi064.stdout:339: read oid 45 snap -1 2024-03-04T06:09:12.227 INFO:tasks.rados.rados.0.smithi064.stdout:339: expect (ObjNum 151 snap 21 seq_num 151) 2024-03-04T06:09:12.228 INFO:tasks.rados.rados.0.smithi064.stdout:340: snap_create 2024-03-04T06:09:12.228 INFO:tasks.rados.rados.0.smithi064.stdout:329: finishing write tid 3 to smithi06431743-29 2024-03-04T06:09:12.228 INFO:tasks.rados.rados.0.smithi064.stdout:329: finishing write tid 4 to smithi06431743-29 2024-03-04T06:09:12.228 INFO:tasks.rados.rados.0.smithi064.stdout:329: finishing write tid 5 to smithi06431743-29 2024-03-04T06:09:12.228 INFO:tasks.rados.rados.0.smithi064.stdout:329: oid 29 updating version 0 to 71 2024-03-04T06:09:12.228 INFO:tasks.rados.rados.0.smithi064.stdout:329: oid 29 updating version 71 to 72 2024-03-04T06:09:12.228 INFO:tasks.rados.rados.0.smithi064.stdout:329: oid 29 version 72 is already newer than 69 2024-03-04T06:09:12.228 INFO:tasks.rados.rados.0.smithi064.stdout:329: oid 29 version 72 is already newer than 71 2024-03-04T06:09:12.228 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 72 (ObjNum 181 snap 34 seq_num 181) dirty exists 2024-03-04T06:09:12.228 INFO:tasks.rados.rados.0.smithi064.stdout:329: left oid 29 (ObjNum 181 snap 34 seq_num 181) 2024-03-04T06:09:12.229 INFO:tasks.rados.rados.0.smithi064.stdout:333: expect (ObjNum 113 snap 10 seq_num 113) 2024-03-04T06:09:12.429 INFO:tasks.rados.rados.0.smithi064.stdout:331: finishing write tid 1 to smithi06431743-28 2024-03-04T06:09:12.429 INFO:tasks.rados.rados.0.smithi064.stdout:332: finishing write tid 1 to smithi06431743-19 2024-03-04T06:09:12.429 INFO:tasks.rados.rados.0.smithi064.stdout:332: finishing write tid 2 to smithi06431743-19 2024-03-04T06:09:12.429 INFO:tasks.rados.rados.0.smithi064.stdout:331: finishing write tid 2 to smithi06431743-28 2024-03-04T06:09:12.430 INFO:tasks.rados.rados.0.smithi064.stdout:332: finishing write tid 3 to smithi06431743-19 2024-03-04T06:09:12.430 INFO:tasks.rados.rados.0.smithi064.stdout:331: finishing write tid 3 to smithi06431743-28 2024-03-04T06:09:12.430 INFO:tasks.rados.rados.0.smithi064.stdout:331: finishing write tid 4 to smithi06431743-28 2024-03-04T06:09:12.430 INFO:tasks.rados.rados.0.smithi064.stdout:331: oid 28 updating version 0 to 139 2024-03-04T06:09:12.430 INFO:tasks.rados.rados.0.smithi064.stdout:331: oid 28 version 139 is already newer than 139 2024-03-04T06:09:12.430 INFO:tasks.rados.rados.0.smithi064.stdout:331: oid 28 version 139 is already newer than 137 2024-03-04T06:09:12.430 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 139 (ObjNum 182 snap 34 seq_num 182) dirty exists 2024-03-04T06:09:12.430 INFO:tasks.rados.rados.0.smithi064.stdout:331: left oid 28 (ObjNum 182 snap 34 seq_num 182) 2024-03-04T06:09:12.430 INFO:tasks.rados.rados.0.smithi064.stdout:332: finishing write tid 4 to smithi06431743-19 2024-03-04T06:09:12.430 INFO:tasks.rados.rados.0.smithi064.stdout:332: finishing write tid 5 to smithi06431743-19 2024-03-04T06:09:12.431 INFO:tasks.rados.rados.0.smithi064.stdout:332: oid 19 updating version 0 to 79 2024-03-04T06:09:12.431 INFO:tasks.rados.rados.0.smithi064.stdout:332: oid 19 version 79 is already newer than 76 2024-03-04T06:09:12.431 INFO:tasks.rados.rados.0.smithi064.stdout:332: oid 19 version 79 is already newer than 78 2024-03-04T06:09:12.431 INFO:tasks.rados.rados.0.smithi064.stdout:332: oid 19 version 79 is already newer than 78 2024-03-04T06:09:12.431 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 79 (ObjNum 183 snap 34 seq_num 183) dirty exists 2024-03-04T06:09:12.431 INFO:tasks.rados.rados.0.smithi064.stdout:332: left oid 19 (ObjNum 183 snap 34 seq_num 183) 2024-03-04T06:09:12.431 INFO:tasks.rados.rados.0.smithi064.stdout:336: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-04T06:09:12.532 INFO:tasks.rados.rados.0.smithi064.stdout:335: finishing write tid 1 to smithi06431743-18 2024-03-04T06:09:12.532 INFO:tasks.rados.rados.0.smithi064.stdout:337: finishing write tid 1 to smithi06431743-17 2024-03-04T06:09:12.532 INFO:tasks.rados.rados.0.smithi064.stdout:335: finishing write tid 2 to smithi06431743-18 2024-03-04T06:09:12.532 INFO:tasks.rados.rados.0.smithi064.stdout:335: finishing write tid 3 to smithi06431743-18 2024-03-04T06:09:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:337: finishing write tid 2 to smithi06431743-17 2024-03-04T06:09:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:335: finishing write tid 4 to smithi06431743-18 2024-03-04T06:09:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:335: oid 18 updating version 0 to 41 2024-03-04T06:09:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:335: oid 18 version 41 is already newer than 41 2024-03-04T06:09:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:335: oid 18 version 41 is already newer than 39 2024-03-04T06:09:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 41 (ObjNum 184 snap 34 seq_num 184) dirty exists 2024-03-04T06:09:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:335: left oid 18 (ObjNum 184 snap 34 seq_num 184) 2024-03-04T06:09:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:337: finishing write tid 3 to smithi06431743-17 2024-03-04T06:09:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:337: finishing write tid 4 to smithi06431743-17 2024-03-04T06:09:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:337: oid 17 updating version 0 to 51 2024-03-04T06:09:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:337: oid 17 version 51 is already newer than 49 2024-03-04T06:09:12.534 INFO:tasks.rados.rados.0.smithi064.stdout:337: oid 17 version 51 is already newer than 51 2024-03-04T06:09:12.534 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 51 (ObjNum 185 snap 34 seq_num 185) dirty exists 2024-03-04T06:09:12.534 INFO:tasks.rados.rados.0.smithi064.stdout:337: left oid 17 (ObjNum 185 snap 34 seq_num 185) 2024-03-04T06:09:12.534 INFO:tasks.rados.rados.0.smithi064.stdout:339: expect (ObjNum 151 snap 21 seq_num 151) 2024-03-04T06:09:12.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:12.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:12.780+0000 7f7445c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:12.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:12.784+0000 7f7445c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:13.232 INFO:tasks.rados.rados.0.smithi064.stdout:329: done (8 left) 2024-03-04T06:09:13.232 INFO:tasks.rados.rados.0.smithi064.stdout:331: done (7 left) 2024-03-04T06:09:13.232 INFO:tasks.rados.rados.0.smithi064.stdout:332: done (6 left) 2024-03-04T06:09:13.233 INFO:tasks.rados.rados.0.smithi064.stdout:333: done (5 left) 2024-03-04T06:09:13.233 INFO:tasks.rados.rados.0.smithi064.stdout:335: done (4 left) 2024-03-04T06:09:13.233 INFO:tasks.rados.rados.0.smithi064.stdout:336: done (3 left) 2024-03-04T06:09:13.233 INFO:tasks.rados.rados.0.smithi064.stdout:337: done (2 left) 2024-03-04T06:09:13.233 INFO:tasks.rados.rados.0.smithi064.stdout:339: done (1 left) 2024-03-04T06:09:13.233 INFO:tasks.rados.rados.0.smithi064.stdout:340: done (0 left) 2024-03-04T06:09:13.233 INFO:tasks.rados.rados.0.smithi064.stdout:341: snap_create 2024-03-04T06:09:14.232 INFO:tasks.rados.rados.0.smithi064.stdout:341: done (0 left) 2024-03-04T06:09:14.232 INFO:tasks.rados.rados.0.smithi064.stdout:342: read oid 24 snap 32 2024-03-04T06:09:14.232 INFO:tasks.rados.rados.0.smithi064.stdout:342: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-04T06:09:14.232 INFO:tasks.rados.rados.0.smithi064.stdout:343: delete oid 27 current snap is 36 2024-03-04T06:09:14.290 INFO:tasks.rados.rados.0.smithi064.stdout:342: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-04T06:09:14.380 INFO:tasks.rados.rados.0.smithi064.stdout:342: done (1 left) 2024-03-04T06:09:14.381 INFO:tasks.rados.rados.0.smithi064.stdout:343: done (0 left) 2024-03-04T06:09:14.381 INFO:tasks.rados.rados.0.smithi064.stdout:344: write (excl) oid 14 current snap is 36 2024-03-04T06:09:14.381 INFO:tasks.rados.rados.0.smithi064.stdout:344: seq_num 187 ranges {742909=464035,1976368=754051} 2024-03-04T06:09:14.387 INFO:tasks.rados.rados.0.smithi064.stdout:344: writing smithi06431743-14 from 742909 to 1206944 tid 1 2024-03-04T06:09:14.396 INFO:tasks.rados.rados.0.smithi064.stdout:344: writing smithi06431743-14 from 1976368 to 2730419 tid 2 2024-03-04T06:09:14.396 INFO:tasks.rados.rados.0.smithi064.stdout:345: snap_create 2024-03-04T06:09:14.406 INFO:tasks.rados.rados.0.smithi064.stdout:344: finishing write tid 1 to smithi06431743-14 2024-03-04T06:09:14.419 INFO:tasks.rados.rados.0.smithi064.stdout:344: finishing write tid 2 to smithi06431743-14 2024-03-04T06:09:14.422 INFO:tasks.rados.rados.0.smithi064.stdout:344: finishing write tid 3 to smithi06431743-14 2024-03-04T06:09:14.423 INFO:tasks.rados.rados.0.smithi064.stdout:344: finishing write tid 4 to smithi06431743-14 2024-03-04T06:09:14.423 INFO:tasks.rados.rados.0.smithi064.stdout:344: oid 14 updating version 0 to 56 2024-03-04T06:09:14.423 INFO:tasks.rados.rados.0.smithi064.stdout:344: oid 14 updating version 56 to 58 2024-03-04T06:09:14.423 INFO:tasks.rados.rados.0.smithi064.stdout:344: oid 14 version 58 is already newer than 58 2024-03-04T06:09:14.424 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 58 (ObjNum 187 snap 36 seq_num 187) dirty exists 2024-03-04T06:09:14.424 INFO:tasks.rados.rados.0.smithi064.stdout:344: left oid 14 (ObjNum 187 snap 36 seq_num 187) 2024-03-04T06:09:14.784 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:09:14.784 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:09:14.785 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:09:14.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:09:14.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:14.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:14.890+0000 7fab07f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:14.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:14.897+0000 7fab07f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:14.912 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:14.913 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:14.913 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:14.913 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:14.913 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:14.913 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:14.913 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:14.913 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:14.913 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:14.913 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:14.914 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:14.914 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:14.916 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:14.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:09:14.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:14.961+0000 7fc475954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:14.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:14.965+0000 7fc475954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:15.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:15.062+0000 7f5a59a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:15.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:15.069+0000 7f5a59a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:15.082 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:15.082 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:15.082 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:15.082 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:15.083 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:15.085 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:15.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:09:15.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:15.228+0000 7f606922d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:15.238 INFO:tasks.rados.rados.0.smithi064.stdout:344: done (1 left) 2024-03-04T06:09:15.238 INFO:tasks.rados.rados.0.smithi064.stdout:345: done (0 left) 2024-03-04T06:09:15.239 INFO:tasks.rados.rados.0.smithi064.stdout:346: write (excl) oid 13 current snap is 37 2024-03-04T06:09:15.239 INFO:tasks.rados.rados.0.smithi064.stdout:346: seq_num 188 ranges {759716=539284,2034850=614062} 2024-03-04T06:09:15.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:15.237+0000 7f606922d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:15.246 INFO:tasks.rados.rados.0.smithi064.stdout:346: writing smithi06431743-13 from 759716 to 1299000 tid 1 2024-03-04T06:09:15.253 INFO:tasks.rados.rados.0.smithi064.stdout:346: writing smithi06431743-13 from 2034850 to 2648912 tid 2 2024-03-04T06:09:15.253 INFO:tasks.rados.rados.0.smithi064.stdout:347: snap_create 2024-03-04T06:09:15.255 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:15.255 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:15.255 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:15.256 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:15.256 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:15.256 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:15.256 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:15.256 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:15.256 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:15.256 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:15.256 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:15.256 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:15.258 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:15.258 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:15.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:09:15.307 INFO:tasks.rados.rados.0.smithi064.stdout:346: finishing write tid 1 to smithi06431743-13 2024-03-04T06:09:15.320 INFO:tasks.rados.rados.0.smithi064.stdout:346: finishing write tid 2 to smithi06431743-13 2024-03-04T06:09:15.327 INFO:tasks.rados.rados.0.smithi064.stdout:346: finishing write tid 3 to smithi06431743-13 2024-03-04T06:09:15.328 INFO:tasks.rados.rados.0.smithi064.stdout:346: finishing write tid 4 to smithi06431743-13 2024-03-04T06:09:15.328 INFO:tasks.rados.rados.0.smithi064.stdout:346: oid 13 updating version 0 to 87 2024-03-04T06:09:15.328 INFO:tasks.rados.rados.0.smithi064.stdout:346: oid 13 updating version 87 to 88 2024-03-04T06:09:15.328 INFO:tasks.rados.rados.0.smithi064.stdout:346: oid 13 updating version 88 to 89 2024-03-04T06:09:15.328 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 89 (ObjNum 188 snap 37 seq_num 188) dirty exists 2024-03-04T06:09:15.328 INFO:tasks.rados.rados.0.smithi064.stdout:346: left oid 13 (ObjNum 188 snap 37 seq_num 188) 2024-03-04T06:09:15.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:15.362+0000 7f693bd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:15.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:15.369+0000 7f693bd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:15.383 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:15.386 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:16.249 INFO:tasks.rados.rados.0.smithi064.stdout:346: done (1 left) 2024-03-04T06:09:16.249 INFO:tasks.rados.rados.0.smithi064.stdout:347: done (0 left) 2024-03-04T06:09:16.249 INFO:tasks.rados.rados.0.smithi064.stdout:348: read oid 49 snap -1 2024-03-04T06:09:16.249 INFO:tasks.rados.rados.0.smithi064.stdout:348: expect (ObjNum 142 snap 18 seq_num 142) 2024-03-04T06:09:16.250 INFO:tasks.rados.rados.0.smithi064.stdout:349: read oid 33 snap 37 2024-03-04T06:09:16.250 INFO:tasks.rados.rados.0.smithi064.stdout:349: expect deleted 2024-03-04T06:09:16.250 INFO:tasks.rados.rados.0.smithi064.stdout:350: read oid 38 snap -1 2024-03-04T06:09:16.250 INFO:tasks.rados.rados.0.smithi064.stdout:350: expect deleted 2024-03-04T06:09:16.250 INFO:tasks.rados.rados.0.smithi064.stdout:351: snap_remove snap 35 2024-03-04T06:09:17.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:17.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:17.158+0000 7fcbcfe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:17.163 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:17.162+0000 7fcbcfe2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:17.257 INFO:tasks.rados.rados.0.smithi064.stdout:351: done (3 left) 2024-03-04T06:09:17.257 INFO:tasks.rados.rados.0.smithi064.stdout:352: snap_remove snap 31 2024-03-04T06:09:18.183 INFO:tasks.rados.rados.0.smithi064.stdout:352: done (3 left) 2024-03-04T06:09:18.183 INFO:tasks.rados.rados.0.smithi064.stdout:353: snap_remove snap 36 2024-03-04T06:09:19.183 INFO:tasks.rados.rados.0.smithi064.stdout:353: done (3 left) 2024-03-04T06:09:19.183 INFO:tasks.rados.rados.0.smithi064.stdout:354: delete oid 15 current snap is 38 2024-03-04T06:09:19.183 INFO:tasks.rados.rados.0.smithi064.stdout:348: expect (ObjNum 142 snap 18 seq_num 142) 2024-03-04T06:09:19.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:19.345 INFO:tasks.rados.rados.0.smithi064.stdout:348: done (3 left) 2024-03-04T06:09:19.345 INFO:tasks.rados.rados.0.smithi064.stdout:349: done (2 left) 2024-03-04T06:09:19.345 INFO:tasks.rados.rados.0.smithi064.stdout:350: done (1 left) 2024-03-04T06:09:19.345 INFO:tasks.rados.rados.0.smithi064.stdout:354: done (0 left) 2024-03-04T06:09:19.346 INFO:tasks.rados.rados.0.smithi064.stdout:355: write (excl) oid 24 current snap is 38 2024-03-04T06:09:19.346 INFO:tasks.rados.rados.0.smithi064.stdout:355: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-03-04T06:09:19.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:19.348+0000 7fee76606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:19.353 INFO:tasks.rados.rados.0.smithi064.stdout:355: writing smithi06431743-24 from 793330 to 1483112 tid 1 2024-03-04T06:09:19.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:19.353+0000 7fee76606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:19.360 INFO:tasks.rados.rados.0.smithi064.stdout:355: writing smithi06431743-24 from 2035461 to 2653192 tid 2 2024-03-04T06:09:19.364 INFO:tasks.rados.rados.0.smithi064.stdout:355: writing smithi06431743-24 from 3101545 to 3193330 tid 3 2024-03-04T06:09:19.364 INFO:tasks.rados.rados.0.smithi064.stdout:356: read oid 7 snap -1 2024-03-04T06:09:19.364 INFO:tasks.rados.rados.0.smithi064.stdout:356: expect (ObjNum 166 snap 26 seq_num 166) 2024-03-04T06:09:19.364 INFO:tasks.rados.rados.0.smithi064.stdout:357: read oid 14 snap -1 2024-03-04T06:09:19.364 INFO:tasks.rados.rados.0.smithi064.stdout:357: expect (ObjNum 187 snap 36 seq_num 187) 2024-03-04T06:09:19.364 INFO:tasks.rados.rados.0.smithi064.stdout:358: write oid 36 current snap is 38 2024-03-04T06:09:19.364 INFO:tasks.rados.rados.0.smithi064.stdout:358: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-03-04T06:09:19.368 INFO:tasks.rados.rados.0.smithi064.stdout:358: writing smithi06431743-36 from 410137 to 891521 tid 1 2024-03-04T06:09:19.375 INFO:tasks.rados.rados.0.smithi064.stdout:358: writing smithi06431743-36 from 1410296 to 2171685 tid 2 2024-03-04T06:09:19.381 INFO:tasks.rados.rados.0.smithi064.stdout:358: writing smithi06431743-36 from 2728968 to 3210137 tid 3 2024-03-04T06:09:19.381 INFO:tasks.rados.rados.0.smithi064.stdout:359: read oid 4 snap -1 2024-03-04T06:09:19.382 INFO:tasks.rados.rados.0.smithi064.stdout:359: expect (ObjNum 117 snap 10 seq_num 117) 2024-03-04T06:09:19.382 INFO:tasks.rados.rados.0.smithi064.stdout:360: read oid 45 snap -1 2024-03-04T06:09:19.382 INFO:tasks.rados.rados.0.smithi064.stdout:360: expect (ObjNum 151 snap 21 seq_num 151) 2024-03-04T06:09:19.382 INFO:tasks.rados.rados.0.smithi064.stdout:361: write oid 47 current snap is 38 2024-03-04T06:09:19.382 INFO:tasks.rados.rados.0.smithi064.stdout:361: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-03-04T06:09:19.386 INFO:tasks.rados.rados.0.smithi064.stdout:361: writing smithi06431743-47 from 426944 to 983577 tid 1 2024-03-04T06:09:19.393 INFO:tasks.rados.rados.0.smithi064.stdout:361: writing smithi06431743-47 from 1468778 to 2090178 tid 2 2024-03-04T06:09:19.400 INFO:tasks.rados.rados.0.smithi064.stdout:361: writing smithi06431743-47 from 2872744 to 3226944 tid 3 2024-03-04T06:09:19.400 INFO:tasks.rados.rados.0.smithi064.stdout:362: snap_create 2024-03-04T06:09:19.400 INFO:tasks.rados.rados.0.smithi064.stdout:360: expect (ObjNum 151 snap 21 seq_num 151) 2024-03-04T06:09:19.579 INFO:tasks.rados.rados.0.smithi064.stdout:355: finishing write tid 1 to smithi06431743-24 2024-03-04T06:09:19.580 INFO:tasks.rados.rados.0.smithi064.stdout:361: finishing write tid 1 to smithi06431743-47 2024-03-04T06:09:19.580 INFO:tasks.rados.rados.0.smithi064.stdout:358: finishing write tid 1 to smithi06431743-36 2024-03-04T06:09:19.580 INFO:tasks.rados.rados.0.smithi064.stdout:355: finishing write tid 2 to smithi06431743-24 2024-03-04T06:09:19.580 INFO:tasks.rados.rados.0.smithi064.stdout:359: expect (ObjNum 117 snap 10 seq_num 117) 2024-03-04T06:09:19.785 INFO:tasks.rados.rados.0.smithi064.stdout:358: finishing write tid 2 to smithi06431743-36 2024-03-04T06:09:19.785 INFO:tasks.rados.rados.0.smithi064.stdout:356: expect (ObjNum 166 snap 26 seq_num 166) 2024-03-04T06:09:20.001 INFO:tasks.rados.rados.0.smithi064.stdout:355: finishing write tid 3 to smithi06431743-24 2024-03-04T06:09:20.002 INFO:tasks.rados.rados.0.smithi064.stdout:358: finishing write tid 3 to smithi06431743-36 2024-03-04T06:09:20.002 INFO:tasks.rados.rados.0.smithi064.stdout:361: finishing write tid 2 to smithi06431743-47 2024-03-04T06:09:20.002 INFO:tasks.rados.rados.0.smithi064.stdout:355: finishing write tid 4 to smithi06431743-24 2024-03-04T06:09:20.002 INFO:tasks.rados.rados.0.smithi064.stdout:355: finishing write tid 5 to smithi06431743-24 2024-03-04T06:09:20.002 INFO:tasks.rados.rados.0.smithi064.stdout:355: oid 24 updating version 0 to 67 2024-03-04T06:09:20.002 INFO:tasks.rados.rados.0.smithi064.stdout:355: oid 24 version 67 is already newer than 64 2024-03-04T06:09:20.002 INFO:tasks.rados.rados.0.smithi064.stdout:355: oid 24 version 67 is already newer than 66 2024-03-04T06:09:20.002 INFO:tasks.rados.rados.0.smithi064.stdout:355: oid 24 version 67 is already newer than 65 2024-03-04T06:09:20.002 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 67 (ObjNum 190 snap 38 seq_num 190) dirty exists 2024-03-04T06:09:20.003 INFO:tasks.rados.rados.0.smithi064.stdout:355: left oid 24 (ObjNum 190 snap 38 seq_num 190) 2024-03-04T06:09:20.003 INFO:tasks.rados.rados.0.smithi064.stdout:358: finishing write tid 4 to smithi06431743-36 2024-03-04T06:09:20.003 INFO:tasks.rados.rados.0.smithi064.stdout:358: finishing write tid 5 to smithi06431743-36 2024-03-04T06:09:20.003 INFO:tasks.rados.rados.0.smithi064.stdout:358: oid 36 updating version 0 to 39 2024-03-04T06:09:20.003 INFO:tasks.rados.rados.0.smithi064.stdout:358: oid 36 updating version 39 to 40 2024-03-04T06:09:20.003 INFO:tasks.rados.rados.0.smithi064.stdout:358: oid 36 version 40 is already newer than 39 2024-03-04T06:09:20.003 INFO:tasks.rados.rados.0.smithi064.stdout:358: oid 36 updating version 40 to 41 2024-03-04T06:09:20.003 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 41 (ObjNum 191 snap 38 seq_num 191) dirty exists 2024-03-04T06:09:20.003 INFO:tasks.rados.rados.0.smithi064.stdout:358: left oid 36 (ObjNum 191 snap 38 seq_num 191) 2024-03-04T06:09:20.004 INFO:tasks.rados.rados.0.smithi064.stdout:361: finishing write tid 3 to smithi06431743-47 2024-03-04T06:09:20.004 INFO:tasks.rados.rados.0.smithi064.stdout:361: finishing write tid 4 to smithi06431743-47 2024-03-04T06:09:20.004 INFO:tasks.rados.rados.0.smithi064.stdout:357: expect (ObjNum 187 snap 36 seq_num 187) 2024-03-04T06:09:20.162 INFO:tasks.rados.rados.0.smithi064.stdout:361: finishing write tid 5 to smithi06431743-47 2024-03-04T06:09:20.162 INFO:tasks.rados.rados.0.smithi064.stdout:361: oid 47 updating version 0 to 48 2024-03-04T06:09:20.162 INFO:tasks.rados.rados.0.smithi064.stdout:361: oid 47 updating version 48 to 50 2024-03-04T06:09:20.162 INFO:tasks.rados.rados.0.smithi064.stdout:361: oid 47 version 50 is already newer than 47 2024-03-04T06:09:20.163 INFO:tasks.rados.rados.0.smithi064.stdout:361: oid 47 version 50 is already newer than 49 2024-03-04T06:09:20.163 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 50 (ObjNum 192 snap 38 seq_num 192) dirty exists 2024-03-04T06:09:20.163 INFO:tasks.rados.rados.0.smithi064.stdout:361: left oid 47 (ObjNum 192 snap 38 seq_num 192) 2024-03-04T06:09:20.254 INFO:tasks.rados.rados.0.smithi064.stdout:355: done (7 left) 2024-03-04T06:09:20.255 INFO:tasks.rados.rados.0.smithi064.stdout:356: done (6 left) 2024-03-04T06:09:20.255 INFO:tasks.rados.rados.0.smithi064.stdout:357: done (5 left) 2024-03-04T06:09:20.255 INFO:tasks.rados.rados.0.smithi064.stdout:358: done (4 left) 2024-03-04T06:09:20.255 INFO:tasks.rados.rados.0.smithi064.stdout:359: done (3 left) 2024-03-04T06:09:20.255 INFO:tasks.rados.rados.0.smithi064.stdout:360: done (2 left) 2024-03-04T06:09:20.255 INFO:tasks.rados.rados.0.smithi064.stdout:361: done (1 left) 2024-03-04T06:09:20.255 INFO:tasks.rados.rados.0.smithi064.stdout:362: done (0 left) 2024-03-04T06:09:20.255 INFO:tasks.rados.rados.0.smithi064.stdout:363: snap_remove snap 34 2024-03-04T06:09:20.386 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:09:20.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:09:20.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:20.490+0000 7fa9b2b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:20.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:20.498+0000 7fa9b2b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:21.255 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:09:21.259 INFO:tasks.rados.rados.0.smithi064.stdout:363: done (0 left) 2024-03-04T06:09:21.259 INFO:tasks.rados.rados.0.smithi064.stdout:364: snap_create 2024-03-04T06:09:21.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:09:21.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:21.379+0000 7fdaf4f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:21.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:21.387+0000 7fdaf4f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:21.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:21.625 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:21.623+0000 7fd5c2c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:21.628 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:21.627+0000 7fd5c2c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:22.256 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:09:22.259 INFO:tasks.rados.rados.0.smithi064.stdout:364: done (0 left) 2024-03-04T06:09:22.260 INFO:tasks.rados.rados.0.smithi064.stdout:365: write (excl) oid 4 current snap is 40 2024-03-04T06:09:22.260 INFO:tasks.rados.rados.0.smithi064.stdout:365: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-03-04T06:09:22.265 INFO:tasks.rados.rados.0.smithi064.stdout:365: writing smithi06431743-4 from 443751 to 1075633 tid 1 2024-03-04T06:09:22.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:09:22.275 INFO:tasks.rados.rados.0.smithi064.stdout:365: writing smithi06431743-4 from 1810907 to 2575965 tid 2 2024-03-04T06:09:22.278 INFO:tasks.rados.rados.0.smithi064.stdout:365: writing smithi06431743-4 from 3067461 to 3243751 tid 3 2024-03-04T06:09:22.278 INFO:tasks.rados.rados.0.smithi064.stdout:366: write (excl) oid 22 current snap is 40 2024-03-04T06:09:22.278 INFO:tasks.rados.rados.0.smithi064.stdout:366: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-03-04T06:09:22.286 INFO:tasks.rados.rados.0.smithi064.stdout:366: writing smithi06431743-22 from 460558 to 1167689 tid 1 2024-03-04T06:09:22.294 INFO:tasks.rados.rados.0.smithi064.stdout:366: writing smithi06431743-22 from 1869389 to 2378105 tid 2 2024-03-04T06:09:22.299 INFO:tasks.rados.rados.0.smithi064.stdout:366: writing smithi06431743-22 from 3094884 to 3260558 tid 3 2024-03-04T06:09:22.299 INFO:tasks.rados.rados.0.smithi064.stdout:367: write oid 34 current snap is 40 2024-03-04T06:09:22.300 INFO:tasks.rados.rados.0.smithi064.stdout:367: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-03-04T06:09:22.306 INFO:tasks.rados.rados.0.smithi064.stdout:367: writing smithi06431743-34 from 477365 to 1259745 tid 1 2024-03-04T06:09:22.314 INFO:tasks.rados.rados.0.smithi064.stdout:367: writing smithi06431743-34 from 1927871 to 2696598 tid 2 2024-03-04T06:09:22.317 INFO:tasks.rados.rados.0.smithi064.stdout:367: writing smithi06431743-34 from 3122307 to 3277365 tid 3 2024-03-04T06:09:22.318 INFO:tasks.rados.rados.0.smithi064.stdout:368: write oid 15 current snap is 40 2024-03-04T06:09:22.318 INFO:tasks.rados.rados.0.smithi064.stdout:368: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-03-04T06:09:22.322 INFO:tasks.rados.rados.0.smithi064.stdout:368: writing smithi06431743-15 from 494172 to 951801 tid 1 2024-03-04T06:09:22.329 INFO:tasks.rados.rados.0.smithi064.stdout:368: writing smithi06431743-15 from 1586353 to 2098738 tid 2 2024-03-04T06:09:22.336 INFO:tasks.rados.rados.0.smithi064.stdout:368: writing smithi06431743-15 from 2749730 to 3294172 tid 3 2024-03-04T06:09:22.336 INFO:tasks.rados.rados.0.smithi064.stdout:369: read oid 10 snap 38 2024-03-04T06:09:22.336 INFO:tasks.rados.rados.0.smithi064.stdout:369: expect (ObjNum 146 snap 20 seq_num 146) 2024-03-04T06:09:22.336 INFO:tasks.rados.rados.0.smithi064.stdout:370: write (excl) oid 12 current snap is 40 2024-03-04T06:09:22.336 INFO:tasks.rados.rados.0.smithi064.stdout:366: finishing write tid 1 to smithi06431743-22 2024-03-04T06:09:22.336 INFO:tasks.rados.rados.0.smithi064.stdout:370: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-03-04T06:09:22.342 INFO:tasks.rados.rados.0.smithi064.stdout:370: writing smithi06431743-12 from 510979 to 1043857 tid 1 2024-03-04T06:09:22.349 INFO:tasks.rados.rados.0.smithi064.stdout:370: writing smithi06431743-12 from 1528482 to 2300878 tid 2 2024-03-04T06:09:22.355 INFO:tasks.rados.rados.0.smithi064.stdout:370: writing smithi06431743-12 from 3060800 to 3310979 tid 3 2024-03-04T06:09:22.356 INFO:tasks.rados.rados.0.smithi064.stdout:371: read oid 37 snap 38 2024-03-04T06:09:22.356 INFO:tasks.rados.rados.0.smithi064.stdout:371: expect deleted 2024-03-04T06:09:22.356 INFO:tasks.rados.rados.0.smithi064.stdout:372: write (excl) oid 19 current snap is 40 2024-03-04T06:09:22.357 INFO:tasks.rados.rados.0.smithi064.stdout:365: finishing write tid 1 to smithi06431743-4 2024-03-04T06:09:22.357 INFO:tasks.rados.rados.0.smithi064.stdout:372: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-03-04T06:09:22.363 INFO:tasks.rados.rados.0.smithi064.stdout:372: writing smithi06431743-19 from 527786 to 1252266 tid 1 2024-03-04T06:09:22.370 INFO:tasks.rados.rados.0.smithi064.stdout:372: writing smithi06431743-19 from 1703317 to 2219371 tid 2 2024-03-04T06:09:22.377 INFO:tasks.rados.rados.0.smithi064.stdout:372: writing smithi06431743-19 from 2804576 to 3239611 tid 3 2024-03-04T06:09:22.377 INFO:tasks.rados.rados.0.smithi064.stdout:373: snap_create 2024-03-04T06:09:22.386 INFO:tasks.rados.rados.0.smithi064.stdout:370: finishing write tid 1 to smithi06431743-12 2024-03-04T06:09:22.387 INFO:tasks.rados.rados.0.smithi064.stdout:366: finishing write tid 2 to smithi06431743-22 2024-03-04T06:09:22.390 INFO:tasks.rados.rados.0.smithi064.stdout:369: expect (ObjNum 146 snap 20 seq_num 146) 2024-03-04T06:09:22.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:22.412+0000 7fd313b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:22.424 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:22.423+0000 7fd313b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:22.539 INFO:tasks.rados.rados.0.smithi064.stdout:365: finishing write tid 2 to smithi06431743-4 2024-03-04T06:09:22.539 INFO:tasks.rados.rados.0.smithi064.stdout:368: finishing write tid 1 to smithi06431743-15 2024-03-04T06:09:22.539 INFO:tasks.rados.rados.0.smithi064.stdout:365: finishing write tid 3 to smithi06431743-4 2024-03-04T06:09:22.539 INFO:tasks.rados.rados.0.smithi064.stdout:368: finishing write tid 2 to smithi06431743-15 2024-03-04T06:09:22.539 INFO:tasks.rados.rados.0.smithi064.stdout:367: finishing write tid 1 to smithi06431743-34 2024-03-04T06:09:22.539 INFO:tasks.rados.rados.0.smithi064.stdout:366: finishing write tid 3 to smithi06431743-22 2024-03-04T06:09:22.539 INFO:tasks.rados.rados.0.smithi064.stdout:365: finishing write tid 4 to smithi06431743-4 2024-03-04T06:09:22.540 INFO:tasks.rados.rados.0.smithi064.stdout:370: finishing write tid 2 to smithi06431743-12 2024-03-04T06:09:22.540 INFO:tasks.rados.rados.0.smithi064.stdout:365: finishing write tid 5 to smithi06431743-4 2024-03-04T06:09:22.540 INFO:tasks.rados.rados.0.smithi064.stdout:365: oid 4 updating version 0 to 101 2024-03-04T06:09:22.540 INFO:tasks.rados.rados.0.smithi064.stdout:365: oid 4 updating version 101 to 102 2024-03-04T06:09:22.540 INFO:tasks.rados.rados.0.smithi064.stdout:365: oid 4 version 102 is already newer than 99 2024-03-04T06:09:22.540 INFO:tasks.rados.rados.0.smithi064.stdout:365: oid 4 version 102 is already newer than 101 2024-03-04T06:09:22.540 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 102 (ObjNum 193 snap 40 seq_num 193) dirty exists 2024-03-04T06:09:22.540 INFO:tasks.rados.rados.0.smithi064.stdout:365: left oid 4 (ObjNum 193 snap 40 seq_num 193) 2024-03-04T06:09:22.540 INFO:tasks.rados.rados.0.smithi064.stdout:367: finishing write tid 2 to smithi06431743-34 2024-03-04T06:09:22.540 INFO:tasks.rados.rados.0.smithi064.stdout:366: finishing write tid 4 to smithi06431743-22 2024-03-04T06:09:22.541 INFO:tasks.rados.rados.0.smithi064.stdout:368: finishing write tid 3 to smithi06431743-15 2024-03-04T06:09:22.541 INFO:tasks.rados.rados.0.smithi064.stdout:368: finishing write tid 4 to smithi06431743-15 2024-03-04T06:09:22.541 INFO:tasks.rados.rados.0.smithi064.stdout:368: finishing write tid 5 to smithi06431743-15 2024-03-04T06:09:22.541 INFO:tasks.rados.rados.0.smithi064.stdout:368: oid 15 updating version 0 to 43 2024-03-04T06:09:22.541 INFO:tasks.rados.rados.0.smithi064.stdout:368: oid 15 updating version 43 to 45 2024-03-04T06:09:22.541 INFO:tasks.rados.rados.0.smithi064.stdout:368: oid 15 version 45 is already newer than 45 2024-03-04T06:09:22.541 INFO:tasks.rados.rados.0.smithi064.stdout:368: oid 15 version 45 is already newer than 43 2024-03-04T06:09:22.541 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 45 (ObjNum 196 snap 40 seq_num 196) dirty exists 2024-03-04T06:09:22.541 INFO:tasks.rados.rados.0.smithi064.stdout:368: left oid 15 (ObjNum 196 snap 40 seq_num 196) 2024-03-04T06:09:22.541 INFO:tasks.rados.rados.0.smithi064.stdout:366: finishing write tid 5 to smithi06431743-22 2024-03-04T06:09:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:366: oid 22 updating version 0 to 103 2024-03-04T06:09:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:366: oid 22 updating version 103 to 104 2024-03-04T06:09:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:366: oid 22 updating version 104 to 106 2024-03-04T06:09:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:366: oid 22 version 106 is already newer than 105 2024-03-04T06:09:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 106 (ObjNum 194 snap 40 seq_num 194) dirty exists 2024-03-04T06:09:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:366: left oid 22 (ObjNum 194 snap 40 seq_num 194) 2024-03-04T06:09:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:370: finishing write tid 3 to smithi06431743-12 2024-03-04T06:09:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:367: finishing write tid 3 to smithi06431743-34 2024-03-04T06:09:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:370: finishing write tid 4 to smithi06431743-12 2024-03-04T06:09:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:372: finishing write tid 1 to smithi06431743-19 2024-03-04T06:09:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:370: finishing write tid 5 to smithi06431743-12 2024-03-04T06:09:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:370: oid 12 updating version 0 to 53 2024-03-04T06:09:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:370: oid 12 updating version 53 to 56 2024-03-04T06:09:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:370: oid 12 version 56 is already newer than 55 2024-03-04T06:09:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:370: oid 12 version 56 is already newer than 54 2024-03-04T06:09:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 56 (ObjNum 197 snap 40 seq_num 197) dirty exists 2024-03-04T06:09:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:370: left oid 12 (ObjNum 197 snap 40 seq_num 197) 2024-03-04T06:09:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:367: finishing write tid 4 to smithi06431743-34 2024-03-04T06:09:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:367: finishing write tid 5 to smithi06431743-34 2024-03-04T06:09:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:367: oid 34 updating version 0 to 143 2024-03-04T06:09:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:367: oid 34 updating version 143 to 146 2024-03-04T06:09:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:367: oid 34 version 146 is already newer than 145 2024-03-04T06:09:22.544 INFO:tasks.rados.rados.0.smithi064.stdout:367: oid 34 version 146 is already newer than 144 2024-03-04T06:09:22.544 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 146 (ObjNum 195 snap 40 seq_num 195) dirty exists 2024-03-04T06:09:22.544 INFO:tasks.rados.rados.0.smithi064.stdout:367: left oid 34 (ObjNum 195 snap 40 seq_num 195) 2024-03-04T06:09:22.544 INFO:tasks.rados.rados.0.smithi064.stdout:372: finishing write tid 2 to smithi06431743-19 2024-03-04T06:09:22.544 INFO:tasks.rados.rados.0.smithi064.stdout:372: finishing write tid 3 to smithi06431743-19 2024-03-04T06:09:22.544 INFO:tasks.rados.rados.0.smithi064.stdout:372: finishing write tid 4 to smithi06431743-19 2024-03-04T06:09:22.544 INFO:tasks.rados.rados.0.smithi064.stdout:372: finishing write tid 5 to smithi06431743-19 2024-03-04T06:09:22.544 INFO:tasks.rados.rados.0.smithi064.stdout:372: oid 19 updating version 0 to 105 2024-03-04T06:09:22.544 INFO:tasks.rados.rados.0.smithi064.stdout:372: oid 19 updating version 105 to 107 2024-03-04T06:09:22.544 INFO:tasks.rados.rados.0.smithi064.stdout:372: oid 19 version 107 is already newer than 104 2024-03-04T06:09:22.545 INFO:tasks.rados.rados.0.smithi064.stdout:372: oid 19 version 107 is already newer than 106 2024-03-04T06:09:22.545 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 107 (ObjNum 198 snap 40 seq_num 198) dirty exists 2024-03-04T06:09:22.545 INFO:tasks.rados.rados.0.smithi064.stdout:372: left oid 19 (ObjNum 198 snap 40 seq_num 198) 2024-03-04T06:09:23.275 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:09:23.283 INFO:tasks.rados.rados.0.smithi064.stdout:365: done (8 left) 2024-03-04T06:09:23.283 INFO:tasks.rados.rados.0.smithi064.stdout:366: done (7 left) 2024-03-04T06:09:23.283 INFO:tasks.rados.rados.0.smithi064.stdout:367: done (6 left) 2024-03-04T06:09:23.283 INFO:tasks.rados.rados.0.smithi064.stdout:368: done (5 left) 2024-03-04T06:09:23.283 INFO:tasks.rados.rados.0.smithi064.stdout:369: done (4 left) 2024-03-04T06:09:23.284 INFO:tasks.rados.rados.0.smithi064.stdout:370: done (3 left) 2024-03-04T06:09:23.284 INFO:tasks.rados.rados.0.smithi064.stdout:371: done (2 left) 2024-03-04T06:09:23.284 INFO:tasks.rados.rados.0.smithi064.stdout:372: done (1 left) 2024-03-04T06:09:23.284 INFO:tasks.rados.rados.0.smithi064.stdout:373: done (0 left) 2024-03-04T06:09:23.284 INFO:tasks.rados.rados.0.smithi064.stdout:374: write oid 33 current snap is 41 2024-03-04T06:09:23.284 INFO:tasks.rados.rados.0.smithi064.stdout:374: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-03-04T06:09:23.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:09:23.292 INFO:tasks.rados.rados.0.smithi064.stdout:374: writing smithi06431743-33 from 544593 to 1344322 tid 1 2024-03-04T06:09:23.297 INFO:tasks.rados.rados.0.smithi064.stdout:374: writing smithi06431743-33 from 1761799 to 2537864 tid 2 2024-03-04T06:09:23.302 INFO:tasks.rados.rados.0.smithi064.stdout:374: writing smithi06431743-33 from 2948352 to 3344593 tid 3 2024-03-04T06:09:23.302 INFO:tasks.rados.rados.0.smithi064.stdout:375: read oid 5 snap -1 2024-03-04T06:09:23.303 INFO:tasks.rados.rados.0.smithi064.stdout:375: expect deleted 2024-03-04T06:09:23.303 INFO:tasks.rados.rados.0.smithi064.stdout:376: read oid 32 snap -1 2024-03-04T06:09:23.303 INFO:tasks.rados.rados.0.smithi064.stdout:376: expect (ObjNum 173 snap 29 seq_num 173) 2024-03-04T06:09:23.303 INFO:tasks.rados.rados.0.smithi064.stdout:377: read oid 38 snap -1 2024-03-04T06:09:23.303 INFO:tasks.rados.rados.0.smithi064.stdout:377: expect deleted 2024-03-04T06:09:23.303 INFO:tasks.rados.rados.0.smithi064.stdout:378: snap_create 2024-03-04T06:09:23.343 INFO:tasks.rados.rados.0.smithi064.stdout:376: expect (ObjNum 173 snap 29 seq_num 173) 2024-03-04T06:09:23.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:23.420+0000 7fc271c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:23.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:23.428+0000 7fc271c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:23.511 INFO:tasks.rados.rados.0.smithi064.stdout:374: finishing write tid 1 to smithi06431743-33 2024-03-04T06:09:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:374: finishing write tid 2 to smithi06431743-33 2024-03-04T06:09:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:374: finishing write tid 3 to smithi06431743-33 2024-03-04T06:09:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:374: finishing write tid 4 to smithi06431743-33 2024-03-04T06:09:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:374: finishing write tid 5 to smithi06431743-33 2024-03-04T06:09:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:374: oid 33 updating version 0 to 109 2024-03-04T06:09:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:374: oid 33 version 109 is already newer than 107 2024-03-04T06:09:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:374: oid 33 updating version 109 to 110 2024-03-04T06:09:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:374: oid 33 version 110 is already newer than 109 2024-03-04T06:09:23.513 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 110 (ObjNum 199 snap 41 seq_num 199) dirty exists 2024-03-04T06:09:23.513 INFO:tasks.rados.rados.0.smithi064.stdout:374: left oid 33 (ObjNum 199 snap 41 seq_num 199) 2024-03-04T06:09:23.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:23.802 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:23.800+0000 7f15e826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:23.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:23.804+0000 7f15e826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:24.280 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:09:24.283 INFO:tasks.rados.rados.0.smithi064.stdout:374: done (4 left) 2024-03-04T06:09:24.283 INFO:tasks.rados.rados.0.smithi064.stdout:375: done (3 left) 2024-03-04T06:09:24.283 INFO:tasks.rados.rados.0.smithi064.stdout:376: done (2 left) 2024-03-04T06:09:24.283 INFO:tasks.rados.rados.0.smithi064.stdout:377: done (1 left) 2024-03-04T06:09:24.283 INFO:tasks.rados.rados.0.smithi064.stdout:378: done (0 left) 2024-03-04T06:09:24.283 INFO:tasks.rados.rados.0.smithi064.stdout:379: delete oid 8 current snap is 42 2024-03-04T06:09:24.294 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:09:24.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:09:24.335 INFO:tasks.rados.rados.0.smithi064.stdout:379: done (0 left) 2024-03-04T06:09:24.335 INFO:tasks.rados.rados.0.smithi064.stdout:380: read oid 14 snap -1 2024-03-04T06:09:24.335 INFO:tasks.rados.rados.0.smithi064.stdout:380: expect (ObjNum 187 snap 36 seq_num 187) 2024-03-04T06:09:24.335 INFO:tasks.rados.rados.0.smithi064.stdout:381: snap_create 2024-03-04T06:09:24.352 INFO:tasks.rados.rados.0.smithi064.stdout:380: expect (ObjNum 187 snap 36 seq_num 187) 2024-03-04T06:09:24.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:24.413+0000 7f3f56d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:24.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:24.420+0000 7f3f56d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:24.639 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:09:24.640 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:09:24.651 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-03-04T06:09:24.281088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":102339748,"num_objects":135,"num_object_clones":78,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":852,"num_read_kb":484240,"num_write":823,"num_write_kb":188387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1014,"ondisk_log_size":1014,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":314080,"kb_used_data":202624,"kb_used_omap":0,"kb_used_meta":111424,"kb_avail":377173280,"statfs":{"total":386547056640,"available":386225438720,"internally_reserved":0,"allocated":207486976,"data_stored":215561168,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":114098176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2504309,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":29,"num_read_kb":29040,"num_write":18,"num_write_kb":4550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027750"},"pg_stats":[{"pgid":"3.f","version":"71'52","reported_seq":268,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.316656+0000","last_change":"2024-03-04T06:09:19.211410+0000","last_active":"2024-03-04T06:09:20.316656+0000","last_peered":"2024-03-04T06:09:20.316656+0000","last_clean":"2024-03-04T06:09:20.316656+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:09:20.316656+0000","last_undegraded":"2024-03-04T06:09:20.316656+0000","last_fullsized":"2024-03-04T06:09:20.316656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3479016,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":22,"num_read_kb":6453,"num_write":39,"num_write_kb":11069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"90'49","reported_seq":290,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.267858+0000","last_change":"2024-03-04T06:09:19.419802+0000","last_active":"2024-03-04T06:09:20.267858+0000","last_peered":"2024-03-04T06:09:20.267858+0000","last_clean":"2024-03-04T06:09:20.267858+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:09:20.267858+0000","last_undegraded":"2024-03-04T06:09:20.267858+0000","last_fullsized":"2024-03-04T06:09:20.267858+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092488,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":33,"num_read_kb":21067,"num_write":36,"num_write_kb":9740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"70'82","reported_seq":301,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.310454+0000","last_change":"2024-03-04T06:09:19.240252+0000","last_active":"2024-03-04T06:09:20.310454+0000","last_peered":"2024-03-04T06:09:20.310454+0000","last_clean":"2024-03-04T06:09:20.310454+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:09:20.310454+0000","last_undegraded":"2024-03-04T06:09:20.310454+0000","last_fullsized":"2024-03-04T06:09:20.310454+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4823609,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":45,"num_read_kb":23587,"num_write":53,"num_write_kb":14601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"91'50","reported_seq":280,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.311089+0000","last_change":"2024-03-04T06:09:19.203276+0000","last_active":"2024-03-04T06:09:20.311089+0000","last_peered":"2024-03-04T06:09:20.311089+0000","last_clean":"2024-03-04T06:09:20.311089+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:09:20.311089+0000","last_undegraded":"2024-03-04T06:09:20.311089+0000","last_fullsized":"2024-03-04T06:09:20.311089+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226944,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":39,"num_read_kb":31868,"num_write":42,"num_write_kb":11705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"82'20","reported_seq":264,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.318585+0000","last_change":"2024-03-04T06:09:19.387398+0000","last_active":"2024-03-04T06:09:20.318585+0000","last_peered":"2024-03-04T06:09:20.318585+0000","last_clean":"2024-03-04T06:09:20.318585+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:09:20.318585+0000","last_undegraded":"2024-03-04T06:09:20.318585+0000","last_fullsized":"2024-03-04T06:09:20.318585+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857190,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":17037,"num_write":13,"num_write_kb":3490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"66'42","reported_seq":267,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.317509+0000","last_change":"2024-03-04T06:09:19.210320+0000","last_active":"2024-03-04T06:09:20.317509+0000","last_peered":"2024-03-04T06:09:20.317509+0000","last_clean":"2024-03-04T06:09:20.317509+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:09:20.317509+0000","last_undegraded":"2024-03-04T06:09:20.317509+0000","last_fullsized":"2024-03-04T06:09:20.317509+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218557,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":22192,"num_write":32,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":226,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.318142+0000","last_change":"2024-03-04T06:09:19.210868+0000","last_active":"2024-03-04T06:09:20.318142+0000","last_peered":"2024-03-04T06:09:20.318142+0000","last_clean":"2024-03-04T06:09:20.318142+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:09:20.318142+0000","last_undegraded":"2024-03-04T06:09:20.318142+0000","last_fullsized":"2024-03-04T06:09:20.318142+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"90'55","reported_seq":286,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.311252+0000","last_change":"2024-03-04T06:09:19.245772+0000","last_active":"2024-03-04T06:09:20.311252+0000","last_peered":"2024-03-04T06:09:20.311252+0000","last_clean":"2024-03-04T06:09:20.311252+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:09:20.311252+0000","last_undegraded":"2024-03-04T06:09:20.311252+0000","last_fullsized":"2024-03-04T06:09:20.311252+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5495889,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":28,"num_read_kb":13129,"num_write":43,"num_write_kb":12457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":349,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.310530+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:09:20.310530+0000","last_peered":"2024-03-04T06:09:20.310530+0000","last_clean":"2024-03-04T06:09:20.310530+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:09:20.310530+0000","last_undegraded":"2024-03-04T06:09:20.310530+0000","last_fullsized":"2024-03-04T06:09:20.310530+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":210,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.322535+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:09:20.322535+0000","last_peered":"2024-03-04T06:09:20.322535+0000","last_clean":"2024-03-04T06:09:20.322535+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:09:20.322535+0000","last_undegraded":"2024-03-04T06:09:20.322535+0000","last_fullsized":"2024-03-04T06:09:20.322535+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"82'41","reported_seq":282,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.318656+0000","last_change":"2024-03-04T06:09:19.246494+0000","last_active":"2024-03-04T06:09:20.318656+0000","last_peered":"2024-03-04T06:09:20.318656+0000","last_clean":"2024-03-04T06:09:20.318656+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:09:20.318656+0000","last_undegraded":"2024-03-04T06:09:20.318656+0000","last_fullsized":"2024-03-04T06:09:20.318656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722767,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":36817,"num_write":31,"num_write_kb":8438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":198,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:19.204523+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:09:19.204523+0000","last_peered":"2024-03-04T06:09:19.204523+0000","last_clean":"2024-03-04T06:09:19.204523+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:09:19.204523+0000","last_undegraded":"2024-03-04T06:09:19.204523+0000","last_fullsized":"2024-03-04T06:09:19.204523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"82'41","reported_seq":286,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.317853+0000","last_change":"2024-03-04T06:09:19.380497+0000","last_active":"2024-03-04T06:09:20.317853+0000","last_peered":"2024-03-04T06:09:20.317853+0000","last_clean":"2024-03-04T06:09:20.317853+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:09:20.317853+0000","last_undegraded":"2024-03-04T06:09:20.317853+0000","last_fullsized":"2024-03-04T06:09:20.317853+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210170,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31345,"num_write":29,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":210,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.309729+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:09:20.309729+0000","last_peered":"2024-03-04T06:09:20.309729+0000","last_clean":"2024-03-04T06:09:20.309729+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:09:20.309729+0000","last_undegraded":"2024-03-04T06:09:20.309729+0000","last_fullsized":"2024-03-04T06:09:20.309729+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"84'141","reported_seq":379,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.319446+0000","last_change":"2024-03-04T06:09:19.387805+0000","last_active":"2024-03-04T06:09:20.319446+0000","last_peered":"2024-03-04T06:09:20.319446+0000","last_clean":"2024-03-04T06:09:20.319446+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:09:20.319446+0000","last_undegraded":"2024-03-04T06:09:20.319446+0000","last_fullsized":"2024-03-04T06:09:20.319446+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13344758,"num_objects":20,"num_object_clones":12,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":112,"num_read_kb":72051,"num_write":101,"num_write_kb":28979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":200,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.318967+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:09:20.318967+0000","last_peered":"2024-03-04T06:09:20.318967+0000","last_clean":"2024-03-04T06:09:20.318967+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:09:20.318967+0000","last_undegraded":"2024-03-04T06:09:20.318967+0000","last_fullsized":"2024-03-04T06:09:20.318967+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"91'95","reported_seq":339,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.266599+0000","last_change":"2024-03-04T06:09:20.273491+0000","last_active":"2024-03-04T06:09:20.266599+0000","last_peered":"2024-03-04T06:09:20.266599+0000","last_clean":"2024-03-04T06:09:20.266599+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:09:20.266599+0000","last_undegraded":"2024-03-04T06:09:20.266599+0000","last_fullsized":"2024-03-04T06:09:20.266599+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15748126,"num_objects":11,"num_object_clones":5,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":73,"num_read_kb":47357,"num_write":74,"num_write_kb":19007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"91'39","reported_seq":286,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.323607+0000","last_change":"2024-03-04T06:09:19.252483+0000","last_active":"2024-03-04T06:09:20.323607+0000","last_peered":"2024-03-04T06:09:20.323607+0000","last_clean":"2024-03-04T06:09:20.323607+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:09:20.323607+0000","last_undegraded":"2024-03-04T06:09:20.323607+0000","last_fullsized":"2024-03-04T06:09:20.323607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5058874,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":49,"num_read_kb":33025,"num_write":32,"num_write_kb":7560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":220,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.276850+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:09:20.276850+0000","last_peered":"2024-03-04T06:09:20.276850+0000","last_clean":"2024-03-04T06:09:20.276850+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:09:20.276850+0000","last_undegraded":"2024-03-04T06:09:20.276850+0000","last_fullsized":"2024-03-04T06:09:20.276850+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"91'41","reported_seq":308,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.275051+0000","last_change":"2024-03-04T06:09:19.405449+0000","last_active":"2024-03-04T06:09:20.275051+0000","last_peered":"2024-03-04T06:09:20.275051+0000","last_clean":"2024-03-04T06:09:20.275051+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:09:20.275051+0000","last_undegraded":"2024-03-04T06:09:20.275051+0000","last_fullsized":"2024-03-04T06:09:20.275051+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8168202,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":58,"num_read_kb":33400,"num_write":41,"num_write_kb":10631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":216,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.321088+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:09:20.321088+0000","last_peered":"2024-03-04T06:09:20.321088+0000","last_clean":"2024-03-04T06:09:20.321088+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:09:20.321088+0000","last_undegraded":"2024-03-04T06:09:20.321088+0000","last_fullsized":"2024-03-04T06:09:20.321088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"91'67","reported_seq":324,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.317091+0000","last_change":"2024-03-04T06:09:19.388037+0000","last_active":"2024-03-04T06:09:20.317091+0000","last_peered":"2024-03-04T06:09:20.317091+0000","last_clean":"2024-03-04T06:09:20.317091+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:09:20.317091+0000","last_undegraded":"2024-03-04T06:09:20.317091+0000","last_fullsized":"2024-03-04T06:09:20.317091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12756480,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":66,"num_read_kb":37225,"num_write":61,"num_write_kb":15471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":220,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.265532+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:09:20.265532+0000","last_peered":"2024-03-04T06:09:20.265532+0000","last_clean":"2024-03-04T06:09:20.265532+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:09:20.265532+0000","last_undegraded":"2024-03-04T06:09:20.265532+0000","last_fullsized":"2024-03-04T06:09:20.265532+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"90'102","reported_seq":337,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.264699+0000","last_change":"2024-03-04T06:09:19.426736+0000","last_active":"2024-03-04T06:09:20.264699+0000","last_peered":"2024-03-04T06:09:20.264699+0000","last_clean":"2024-03-04T06:09:20.264699+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:09:20.264699+0000","last_undegraded":"2024-03-04T06:09:20.264699+0000","last_fullsized":"2024-03-04T06:09:20.264699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546310,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":88,"num_read_kb":57596,"num_write":59,"num_write_kb":15545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":210,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.320634+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:09:20.320634+0000","last_peered":"2024-03-04T06:09:20.320634+0000","last_clean":"2024-03-04T06:09:20.320634+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:09:20.320634+0000","last_undegraded":"2024-03-04T06:09:20.320634+0000","last_fullsized":"2024-03-04T06:09:20.320634+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":101749380,"num_objects":133,"num_object_clones":78,"num_object_copies":399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":749,"num_read_kb":484149,"num_write":686,"num_write_kb":186493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":917,"ondisk_log_size":917,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96508,"kb_used_data":65524,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94275332,"statfs":{"total":96636764160,"available":96537939968,"internally_reserved":0,"allocated":67096576,"data_stored":70636189,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77208,"kb_used_data":50384,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94294632,"statfs":{"total":96636764160,"available":96557703168,"internally_reserved":0,"allocated":51593216,"data_stored":51918675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72968,"kb_used_data":46144,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298872,"statfs":{"total":96636764160,"available":96562044928,"internally_reserved":0,"allocated":47251456,"data_stored":49648286,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67396,"kb_used_data":40572,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94304444,"statfs":{"total":96636764160,"available":96567750656,"internally_reserved":0,"allocated":41545728,"data_stored":43358018,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:09:24.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:09:24.799 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:24.795+0000 7f2ba866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:24.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:24.801+0000 7f2ba866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:24.990 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:09:24.990 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:09:25.002 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-03-04T06:09:24.725401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":102339748,"num_objects":135,"num_object_clones":78,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":852,"num_read_kb":484240,"num_write":823,"num_write_kb":188387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1014,"ondisk_log_size":1014,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":314080,"kb_used_data":202624,"kb_used_omap":0,"kb_used_meta":111424,"kb_avail":377173280,"statfs":{"total":386547056640,"available":386225438720,"internally_reserved":0,"allocated":207486976,"data_stored":215561168,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":114098176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1361334,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":20,"num_read_kb":19118,"num_write":6,"num_write_kb":1368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000790"},"pg_stats":[{"pgid":"3.f","version":"71'52","reported_seq":268,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.316656+0000","last_change":"2024-03-04T06:09:19.211410+0000","last_active":"2024-03-04T06:09:20.316656+0000","last_peered":"2024-03-04T06:09:20.316656+0000","last_clean":"2024-03-04T06:09:20.316656+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:09:20.316656+0000","last_undegraded":"2024-03-04T06:09:20.316656+0000","last_fullsized":"2024-03-04T06:09:20.316656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3479016,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":22,"num_read_kb":6453,"num_write":39,"num_write_kb":11069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"90'49","reported_seq":290,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.267858+0000","last_change":"2024-03-04T06:09:19.419802+0000","last_active":"2024-03-04T06:09:20.267858+0000","last_peered":"2024-03-04T06:09:20.267858+0000","last_clean":"2024-03-04T06:09:20.267858+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:09:20.267858+0000","last_undegraded":"2024-03-04T06:09:20.267858+0000","last_fullsized":"2024-03-04T06:09:20.267858+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092488,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":33,"num_read_kb":21067,"num_write":36,"num_write_kb":9740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"70'82","reported_seq":301,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.310454+0000","last_change":"2024-03-04T06:09:19.240252+0000","last_active":"2024-03-04T06:09:20.310454+0000","last_peered":"2024-03-04T06:09:20.310454+0000","last_clean":"2024-03-04T06:09:20.310454+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:09:20.310454+0000","last_undegraded":"2024-03-04T06:09:20.310454+0000","last_fullsized":"2024-03-04T06:09:20.310454+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4823609,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":45,"num_read_kb":23587,"num_write":53,"num_write_kb":14601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"91'50","reported_seq":280,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.311089+0000","last_change":"2024-03-04T06:09:19.203276+0000","last_active":"2024-03-04T06:09:20.311089+0000","last_peered":"2024-03-04T06:09:20.311089+0000","last_clean":"2024-03-04T06:09:20.311089+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:09:20.311089+0000","last_undegraded":"2024-03-04T06:09:20.311089+0000","last_fullsized":"2024-03-04T06:09:20.311089+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226944,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":39,"num_read_kb":31868,"num_write":42,"num_write_kb":11705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"82'20","reported_seq":264,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.318585+0000","last_change":"2024-03-04T06:09:19.387398+0000","last_active":"2024-03-04T06:09:20.318585+0000","last_peered":"2024-03-04T06:09:20.318585+0000","last_clean":"2024-03-04T06:09:20.318585+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:09:20.318585+0000","last_undegraded":"2024-03-04T06:09:20.318585+0000","last_fullsized":"2024-03-04T06:09:20.318585+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857190,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":17037,"num_write":13,"num_write_kb":3490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"66'42","reported_seq":267,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.317509+0000","last_change":"2024-03-04T06:09:19.210320+0000","last_active":"2024-03-04T06:09:20.317509+0000","last_peered":"2024-03-04T06:09:20.317509+0000","last_clean":"2024-03-04T06:09:20.317509+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:09:20.317509+0000","last_undegraded":"2024-03-04T06:09:20.317509+0000","last_fullsized":"2024-03-04T06:09:20.317509+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218557,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":22192,"num_write":32,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":226,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.318142+0000","last_change":"2024-03-04T06:09:19.210868+0000","last_active":"2024-03-04T06:09:20.318142+0000","last_peered":"2024-03-04T06:09:20.318142+0000","last_clean":"2024-03-04T06:09:20.318142+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:09:20.318142+0000","last_undegraded":"2024-03-04T06:09:20.318142+0000","last_fullsized":"2024-03-04T06:09:20.318142+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"90'55","reported_seq":286,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.311252+0000","last_change":"2024-03-04T06:09:19.245772+0000","last_active":"2024-03-04T06:09:20.311252+0000","last_peered":"2024-03-04T06:09:20.311252+0000","last_clean":"2024-03-04T06:09:20.311252+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:09:20.311252+0000","last_undegraded":"2024-03-04T06:09:20.311252+0000","last_fullsized":"2024-03-04T06:09:20.311252+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5495889,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":28,"num_read_kb":13129,"num_write":43,"num_write_kb":12457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":349,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.310530+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:09:20.310530+0000","last_peered":"2024-03-04T06:09:20.310530+0000","last_clean":"2024-03-04T06:09:20.310530+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:09:20.310530+0000","last_undegraded":"2024-03-04T06:09:20.310530+0000","last_fullsized":"2024-03-04T06:09:20.310530+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":210,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.322535+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:09:20.322535+0000","last_peered":"2024-03-04T06:09:20.322535+0000","last_clean":"2024-03-04T06:09:20.322535+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:09:20.322535+0000","last_undegraded":"2024-03-04T06:09:20.322535+0000","last_fullsized":"2024-03-04T06:09:20.322535+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"82'41","reported_seq":282,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.318656+0000","last_change":"2024-03-04T06:09:19.246494+0000","last_active":"2024-03-04T06:09:20.318656+0000","last_peered":"2024-03-04T06:09:20.318656+0000","last_clean":"2024-03-04T06:09:20.318656+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:09:20.318656+0000","last_undegraded":"2024-03-04T06:09:20.318656+0000","last_fullsized":"2024-03-04T06:09:20.318656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722767,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":36817,"num_write":31,"num_write_kb":8438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":198,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:19.204523+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:09:19.204523+0000","last_peered":"2024-03-04T06:09:19.204523+0000","last_clean":"2024-03-04T06:09:19.204523+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:09:19.204523+0000","last_undegraded":"2024-03-04T06:09:19.204523+0000","last_fullsized":"2024-03-04T06:09:19.204523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"82'41","reported_seq":286,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.317853+0000","last_change":"2024-03-04T06:09:19.380497+0000","last_active":"2024-03-04T06:09:20.317853+0000","last_peered":"2024-03-04T06:09:20.317853+0000","last_clean":"2024-03-04T06:09:20.317853+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:09:20.317853+0000","last_undegraded":"2024-03-04T06:09:20.317853+0000","last_fullsized":"2024-03-04T06:09:20.317853+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210170,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31345,"num_write":29,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":210,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.309729+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:09:20.309729+0000","last_peered":"2024-03-04T06:09:20.309729+0000","last_clean":"2024-03-04T06:09:20.309729+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:09:20.309729+0000","last_undegraded":"2024-03-04T06:09:20.309729+0000","last_fullsized":"2024-03-04T06:09:20.309729+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"84'141","reported_seq":379,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.319446+0000","last_change":"2024-03-04T06:09:19.387805+0000","last_active":"2024-03-04T06:09:20.319446+0000","last_peered":"2024-03-04T06:09:20.319446+0000","last_clean":"2024-03-04T06:09:20.319446+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:09:20.319446+0000","last_undegraded":"2024-03-04T06:09:20.319446+0000","last_fullsized":"2024-03-04T06:09:20.319446+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13344758,"num_objects":20,"num_object_clones":12,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":112,"num_read_kb":72051,"num_write":101,"num_write_kb":28979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":200,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.318967+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:09:20.318967+0000","last_peered":"2024-03-04T06:09:20.318967+0000","last_clean":"2024-03-04T06:09:20.318967+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:09:20.318967+0000","last_undegraded":"2024-03-04T06:09:20.318967+0000","last_fullsized":"2024-03-04T06:09:20.318967+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"91'95","reported_seq":339,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.266599+0000","last_change":"2024-03-04T06:09:20.273491+0000","last_active":"2024-03-04T06:09:20.266599+0000","last_peered":"2024-03-04T06:09:20.266599+0000","last_clean":"2024-03-04T06:09:20.266599+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:09:20.266599+0000","last_undegraded":"2024-03-04T06:09:20.266599+0000","last_fullsized":"2024-03-04T06:09:20.266599+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15748126,"num_objects":11,"num_object_clones":5,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":73,"num_read_kb":47357,"num_write":74,"num_write_kb":19007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"91'39","reported_seq":286,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.323607+0000","last_change":"2024-03-04T06:09:19.252483+0000","last_active":"2024-03-04T06:09:20.323607+0000","last_peered":"2024-03-04T06:09:20.323607+0000","last_clean":"2024-03-04T06:09:20.323607+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:09:20.323607+0000","last_undegraded":"2024-03-04T06:09:20.323607+0000","last_fullsized":"2024-03-04T06:09:20.323607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5058874,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":49,"num_read_kb":33025,"num_write":32,"num_write_kb":7560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":220,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.276850+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:09:20.276850+0000","last_peered":"2024-03-04T06:09:20.276850+0000","last_clean":"2024-03-04T06:09:20.276850+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:09:20.276850+0000","last_undegraded":"2024-03-04T06:09:20.276850+0000","last_fullsized":"2024-03-04T06:09:20.276850+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"91'41","reported_seq":308,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.275051+0000","last_change":"2024-03-04T06:09:19.405449+0000","last_active":"2024-03-04T06:09:20.275051+0000","last_peered":"2024-03-04T06:09:20.275051+0000","last_clean":"2024-03-04T06:09:20.275051+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:09:20.275051+0000","last_undegraded":"2024-03-04T06:09:20.275051+0000","last_fullsized":"2024-03-04T06:09:20.275051+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8168202,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":58,"num_read_kb":33400,"num_write":41,"num_write_kb":10631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":216,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.321088+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:09:20.321088+0000","last_peered":"2024-03-04T06:09:20.321088+0000","last_clean":"2024-03-04T06:09:20.321088+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:09:20.321088+0000","last_undegraded":"2024-03-04T06:09:20.321088+0000","last_fullsized":"2024-03-04T06:09:20.321088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"91'67","reported_seq":324,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.317091+0000","last_change":"2024-03-04T06:09:19.388037+0000","last_active":"2024-03-04T06:09:20.317091+0000","last_peered":"2024-03-04T06:09:20.317091+0000","last_clean":"2024-03-04T06:09:20.317091+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:09:20.317091+0000","last_undegraded":"2024-03-04T06:09:20.317091+0000","last_fullsized":"2024-03-04T06:09:20.317091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12756480,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":66,"num_read_kb":37225,"num_write":61,"num_write_kb":15471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":220,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.265532+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:09:20.265532+0000","last_peered":"2024-03-04T06:09:20.265532+0000","last_clean":"2024-03-04T06:09:20.265532+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:09:20.265532+0000","last_undegraded":"2024-03-04T06:09:20.265532+0000","last_fullsized":"2024-03-04T06:09:20.265532+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"90'102","reported_seq":337,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.264699+0000","last_change":"2024-03-04T06:09:19.426736+0000","last_active":"2024-03-04T06:09:20.264699+0000","last_peered":"2024-03-04T06:09:20.264699+0000","last_clean":"2024-03-04T06:09:20.264699+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:09:20.264699+0000","last_undegraded":"2024-03-04T06:09:20.264699+0000","last_fullsized":"2024-03-04T06:09:20.264699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546310,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":88,"num_read_kb":57596,"num_write":59,"num_write_kb":15545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":210,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-04T06:09:20.320634+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:09:20.320634+0000","last_peered":"2024-03-04T06:09:20.320634+0000","last_clean":"2024-03-04T06:09:20.320634+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:09:20.320634+0000","last_undegraded":"2024-03-04T06:09:20.320634+0000","last_fullsized":"2024-03-04T06:09:20.320634+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":101749380,"num_objects":133,"num_object_clones":78,"num_object_copies":399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":749,"num_read_kb":484149,"num_write":686,"num_write_kb":186493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":917,"ondisk_log_size":917,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96508,"kb_used_data":65524,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94275332,"statfs":{"total":96636764160,"available":96537939968,"internally_reserved":0,"allocated":67096576,"data_stored":70636189,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77208,"kb_used_data":50384,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94294632,"statfs":{"total":96636764160,"available":96557703168,"internally_reserved":0,"allocated":51593216,"data_stored":51918675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72968,"kb_used_data":46144,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94298872,"statfs":{"total":96636764160,"available":96562044928,"internally_reserved":0,"allocated":47251456,"data_stored":49648286,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67396,"kb_used_data":40572,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94304444,"statfs":{"total":96636764160,"available":96567750656,"internally_reserved":0,"allocated":41545728,"data_stored":43358018,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:09:25.003 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:09:25.003 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:09:25.004 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:09:25.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:09:25.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:25.107+0000 7f6598b10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:25.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:25.114+0000 7f6598b10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:25.127 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:25.127 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:25.127 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:25.127 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:25.127 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:25.127 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:25.127 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:25.128 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:25.128 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:25.128 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:25.128 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:25.128 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:25.130 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:25.130 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:25.130 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:09:25.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:25.233+0000 7fa67919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:25.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:25.240+0000 7fa67919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:25.253 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:25.254 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:25.254 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:25.254 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:25.254 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:25.254 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:25.254 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:25.254 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:25.254 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:25.254 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:25.254 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:25.255 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:25.257 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:25.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:09:25.286 INFO:tasks.rados.rados.0.smithi064.stdout:380: done (1 left) 2024-03-04T06:09:25.286 INFO:tasks.rados.rados.0.smithi064.stdout:381: done (0 left) 2024-03-04T06:09:25.286 INFO:tasks.rados.rados.0.smithi064.stdout:382: snap_remove snap 40 2024-03-04T06:09:25.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:25.360+0000 7f62ebf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:25.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:25.368+0000 7f62ebf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:25.381 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:25.381 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:25.381 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:25.381 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:25.381 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:25.381 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:25.381 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:25.381 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:25.381 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:25.381 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:25.382 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:25.382 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:25.384 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:25.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:09:25.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:25.527+0000 7f0e48314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:25.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:25.534+0000 7f0e48314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:25.549 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:25.551 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:25.872 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:25.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:25.977+0000 7fa27ec2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:25.982 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:25.981+0000 7fa27ec2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:26.287 INFO:tasks.rados.rados.0.smithi064.stdout:382: done (0 left) 2024-03-04T06:09:26.287 INFO:tasks.rados.rados.0.smithi064.stdout:383: write (excl) oid 17 current snap is 43 2024-03-04T06:09:26.287 INFO:tasks.rados.rados.0.smithi064.stdout:383: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-03-04T06:09:26.292 INFO:tasks.rados.rados.0.smithi064.stdout:383: writing smithi06431743-17 from 578207 to 1128434 tid 1 2024-03-04T06:09:26.300 INFO:tasks.rados.rados.0.smithi064.stdout:383: writing smithi06431743-17 from 1762410 to 2542144 tid 2 2024-03-04T06:09:26.305 INFO:tasks.rados.rados.0.smithi064.stdout:383: writing smithi06431743-17 from 3286845 to 3378207 tid 3 2024-03-04T06:09:26.305 INFO:tasks.rados.rados.0.smithi064.stdout:384: read oid 20 snap -1 2024-03-04T06:09:26.305 INFO:tasks.rados.rados.0.smithi064.stdout:384: expect (ObjNum 153 snap 21 seq_num 153) 2024-03-04T06:09:26.306 INFO:tasks.rados.rados.0.smithi064.stdout:385: delete oid 42 current snap is 43 2024-03-04T06:09:26.350 INFO:tasks.rados.rados.0.smithi064.stdout:384: expect (ObjNum 153 snap 21 seq_num 153) 2024-03-04T06:09:26.489 INFO:tasks.rados.rados.0.smithi064.stdout:383: finishing write tid 1 to smithi06431743-17 2024-03-04T06:09:26.489 INFO:tasks.rados.rados.0.smithi064.stdout:383: finishing write tid 2 to smithi06431743-17 2024-03-04T06:09:26.489 INFO:tasks.rados.rados.0.smithi064.stdout:383: finishing write tid 3 to smithi06431743-17 2024-03-04T06:09:26.489 INFO:tasks.rados.rados.0.smithi064.stdout:383: finishing write tid 4 to smithi06431743-17 2024-03-04T06:09:26.490 INFO:tasks.rados.rados.0.smithi064.stdout:383: finishing write tid 5 to smithi06431743-17 2024-03-04T06:09:26.490 INFO:tasks.rados.rados.0.smithi064.stdout:383: oid 17 updating version 0 to 57 2024-03-04T06:09:26.490 INFO:tasks.rados.rados.0.smithi064.stdout:383: oid 17 updating version 57 to 58 2024-03-04T06:09:26.490 INFO:tasks.rados.rados.0.smithi064.stdout:383: oid 17 updating version 58 to 59 2024-03-04T06:09:26.490 INFO:tasks.rados.rados.0.smithi064.stdout:383: oid 17 updating version 59 to 60 2024-03-04T06:09:26.490 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 60 (ObjNum 201 snap 43 seq_num 201) dirty exists 2024-03-04T06:09:26.490 INFO:tasks.rados.rados.0.smithi064.stdout:383: left oid 17 (ObjNum 201 snap 43 seq_num 201) 2024-03-04T06:09:26.490 INFO:tasks.rados.rados.0.smithi064.stdout:383: done (2 left) 2024-03-04T06:09:26.490 INFO:tasks.rados.rados.0.smithi064.stdout:384: done (1 left) 2024-03-04T06:09:26.490 INFO:tasks.rados.rados.0.smithi064.stdout:385: done (0 left) 2024-03-04T06:09:26.490 INFO:tasks.rados.rados.0.smithi064.stdout:386: read oid 32 snap 41 2024-03-04T06:09:26.491 INFO:tasks.rados.rados.0.smithi064.stdout:386: expect (ObjNum 173 snap 29 seq_num 173) 2024-03-04T06:09:26.491 INFO:tasks.rados.rados.0.smithi064.stdout:387: read oid 12 snap -1 2024-03-04T06:09:26.491 INFO:tasks.rados.rados.0.smithi064.stdout:387: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-04T06:09:26.491 INFO:tasks.rados.rados.0.smithi064.stdout:388: read oid 43 snap -1 2024-03-04T06:09:26.491 INFO:tasks.rados.rados.0.smithi064.stdout:388: expect deleted 2024-03-04T06:09:26.491 INFO:tasks.rados.rados.0.smithi064.stdout:389: read oid 25 snap -1 2024-03-04T06:09:26.491 INFO:tasks.rados.rados.0.smithi064.stdout:389: expect (ObjNum 136 snap 17 seq_num 136) 2024-03-04T06:09:26.491 INFO:tasks.rados.rados.0.smithi064.stdout:390: read oid 33 snap -1 2024-03-04T06:09:26.491 INFO:tasks.rados.rados.0.smithi064.stdout:390: expect (ObjNum 199 snap 41 seq_num 199) 2024-03-04T06:09:26.491 INFO:tasks.rados.rados.0.smithi064.stdout:391: snap_create 2024-03-04T06:09:26.503 INFO:tasks.rados.rados.0.smithi064.stdout:387: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-04T06:09:26.626 INFO:tasks.rados.rados.0.smithi064.stdout:389: expect (ObjNum 136 snap 17 seq_num 136) 2024-03-04T06:09:26.767 INFO:tasks.rados.rados.0.smithi064.stdout:386: expect (ObjNum 173 snap 29 seq_num 173) 2024-03-04T06:09:26.934 INFO:tasks.rados.rados.0.smithi064.stdout:390: expect (ObjNum 199 snap 41 seq_num 199) 2024-03-04T06:09:27.310 INFO:tasks.rados.rados.0.smithi064.stdout:386: done (5 left) 2024-03-04T06:09:27.310 INFO:tasks.rados.rados.0.smithi064.stdout:387: done (4 left) 2024-03-04T06:09:27.310 INFO:tasks.rados.rados.0.smithi064.stdout:388: done (3 left) 2024-03-04T06:09:27.310 INFO:tasks.rados.rados.0.smithi064.stdout:389: done (2 left) 2024-03-04T06:09:27.310 INFO:tasks.rados.rados.0.smithi064.stdout:390: done (1 left) 2024-03-04T06:09:27.310 INFO:tasks.rados.rados.0.smithi064.stdout:391: done (0 left) 2024-03-04T06:09:27.310 INFO:tasks.rados.rados.0.smithi064.stdout:392: read oid 26 snap 37 2024-03-04T06:09:27.310 INFO:tasks.rados.rados.0.smithi064.stdout:392: expect (ObjNum 180 snap 33 seq_num 180) 2024-03-04T06:09:27.310 INFO:tasks.rados.rados.0.smithi064.stdout:393: delete oid 29 current snap is 44 2024-03-04T06:09:27.368 INFO:tasks.rados.rados.0.smithi064.stdout:393: done (1 left) 2024-03-04T06:09:27.368 INFO:tasks.rados.rados.0.smithi064.stdout:394: write oid 23 current snap is 44 2024-03-04T06:09:27.368 INFO:tasks.rados.rados.0.smithi064.stdout:394: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-03-04T06:09:27.375 INFO:tasks.rados.rados.0.smithi064.stdout:394: writing smithi06431743-23 from 628628 to 1404602 tid 1 2024-03-04T06:09:27.381 INFO:tasks.rados.rados.0.smithi064.stdout:394: writing smithi06431743-23 from 1937856 to 2464917 tid 2 2024-03-04T06:09:27.387 INFO:tasks.rados.rados.0.smithi064.stdout:394: writing smithi06431743-23 from 3252761 to 3428628 tid 3 2024-03-04T06:09:27.387 INFO:tasks.rados.rados.0.smithi064.stdout:395: read oid 1 snap -1 2024-03-04T06:09:27.387 INFO:tasks.rados.rados.0.smithi064.stdout:395: expect (ObjNum 132 snap 15 seq_num 132) 2024-03-04T06:09:27.387 INFO:tasks.rados.rados.0.smithi064.stdout:396: read oid 12 snap -1 2024-03-04T06:09:27.387 INFO:tasks.rados.rados.0.smithi064.stdout:396: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-04T06:09:27.387 INFO:tasks.rados.rados.0.smithi064.stdout:397: write (excl) oid 30 current snap is 44 2024-03-04T06:09:27.387 INFO:tasks.rados.rados.0.smithi064.stdout:392: expect (ObjNum 180 snap 33 seq_num 180) 2024-03-04T06:09:27.503 INFO:tasks.rados.rados.0.smithi064.stdout:396: expect (ObjNum 197 snap 40 seq_num 197) 2024-03-04T06:09:27.626 INFO:tasks.rados.rados.0.smithi064.stdout:395: expect (ObjNum 132 snap 15 seq_num 132) 2024-03-04T06:09:27.737 INFO:tasks.rados.rados.0.smithi064.stdout:394: finishing write tid 1 to smithi06431743-23 2024-03-04T06:09:27.737 INFO:tasks.rados.rados.0.smithi064.stdout:394: finishing write tid 2 to smithi06431743-23 2024-03-04T06:09:27.737 INFO:tasks.rados.rados.0.smithi064.stdout:397: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-03-04T06:09:27.743 INFO:tasks.rados.rados.0.smithi064.stdout:397: writing smithi06431743-30 from 645435 to 1096658 tid 1 2024-03-04T06:09:27.752 INFO:tasks.rados.rados.0.smithi064.stdout:397: writing smithi06431743-30 from 1879985 to 2667057 tid 2 2024-03-04T06:09:27.757 INFO:tasks.rados.rados.0.smithi064.stdout:397: writing smithi06431743-30 from 3280184 to 3445435 tid 3 2024-03-04T06:09:27.757 INFO:tasks.rados.rados.0.smithi064.stdout:392: done (4 left) 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:395: done (3 left) 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:396: done (2 left) 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:398: delete oid 42 current snap is 44 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:394: finishing write tid 3 to smithi06431743-23 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:394: finishing write tid 4 to smithi06431743-23 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:394: finishing write tid 5 to smithi06431743-23 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:394: oid 23 updating version 0 to 42 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:394: oid 23 updating version 42 to 45 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:394: oid 23 version 45 is already newer than 44 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:394: oid 23 version 45 is already newer than 44 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 45 (ObjNum 204 snap 44 seq_num 204) dirty exists 2024-03-04T06:09:27.758 INFO:tasks.rados.rados.0.smithi064.stdout:394: left oid 23 (ObjNum 204 snap 44 seq_num 204) 2024-03-04T06:09:27.759 INFO:tasks.rados.rados.0.smithi064.stdout:394: done (2 left) 2024-03-04T06:09:27.760 INFO:tasks.rados.rados.0.smithi064.stdout:398: done (1 left) 2024-03-04T06:09:27.760 INFO:tasks.rados.rados.0.smithi064.stdout:399: write oid 50 current snap is 44 2024-03-04T06:09:27.760 INFO:tasks.rados.rados.0.smithi064.stdout:399: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-03-04T06:09:27.767 INFO:tasks.rados.rados.0.smithi064.stdout:399: writing smithi06431743-50 from 679049 to 1397123 tid 1 2024-03-04T06:09:27.775 INFO:tasks.rados.rados.0.smithi064.stdout:399: writing smithi06431743-50 from 2113302 to 2787690 tid 2 2024-03-04T06:09:27.778 INFO:tasks.rados.rados.0.smithi064.stdout:399: writing smithi06431743-50 from 3335030 to 3479049 tid 3 2024-03-04T06:09:27.779 INFO:tasks.rados.rados.0.smithi064.stdout:400: write oid 32 current snap is 44 2024-03-04T06:09:27.779 INFO:tasks.rados.rados.0.smithi064.stdout:397: finishing write tid 1 to smithi06431743-30 2024-03-04T06:09:27.779 INFO:tasks.rados.rados.0.smithi064.stdout:400: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-03-04T06:09:27.786 INFO:tasks.rados.rados.0.smithi064.stdout:400: writing smithi06431743-32 from 695856 to 1489179 tid 1 2024-03-04T06:09:27.793 INFO:tasks.rados.rados.0.smithi064.stdout:400: writing smithi06431743-32 from 2171784 to 2706183 tid 2 2024-03-04T06:09:27.798 INFO:tasks.rados.rados.0.smithi064.stdout:400: writing smithi06431743-32 from 3362453 to 3495856 tid 3 2024-03-04T06:09:27.798 INFO:tasks.rados.rados.0.smithi064.stdout:401: read oid 21 snap -1 2024-03-04T06:09:27.798 INFO:tasks.rados.rados.0.smithi064.stdout:401: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:09:27.798 INFO:tasks.rados.rados.0.smithi064.stdout:402: snap_create 2024-03-04T06:09:27.798 INFO:tasks.rados.rados.0.smithi064.stdout:397: finishing write tid 2 to smithi06431743-30 2024-03-04T06:09:27.807 INFO:tasks.rados.rados.0.smithi064.stdout:399: finishing write tid 1 to smithi06431743-50 2024-03-04T06:09:27.816 INFO:tasks.rados.rados.0.smithi064.stdout:401: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:09:27.905 INFO:tasks.rados.rados.0.smithi064.stdout:397: finishing write tid 3 to smithi06431743-30 2024-03-04T06:09:27.905 INFO:tasks.rados.rados.0.smithi064.stdout:400: finishing write tid 1 to smithi06431743-32 2024-03-04T06:09:27.906 INFO:tasks.rados.rados.0.smithi064.stdout:399: finishing write tid 2 to smithi06431743-50 2024-03-04T06:09:27.906 INFO:tasks.rados.rados.0.smithi064.stdout:397: finishing write tid 4 to smithi06431743-30 2024-03-04T06:09:27.906 INFO:tasks.rados.rados.0.smithi064.stdout:397: finishing write tid 5 to smithi06431743-30 2024-03-04T06:09:27.906 INFO:tasks.rados.rados.0.smithi064.stdout:397: oid 30 updating version 0 to 111 2024-03-04T06:09:27.906 INFO:tasks.rados.rados.0.smithi064.stdout:397: oid 30 updating version 111 to 112 2024-03-04T06:09:27.906 INFO:tasks.rados.rados.0.smithi064.stdout:397: oid 30 updating version 112 to 114 2024-03-04T06:09:27.906 INFO:tasks.rados.rados.0.smithi064.stdout:397: oid 30 version 114 is already newer than 113 2024-03-04T06:09:27.906 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 114 (ObjNum 205 snap 44 seq_num 205) dirty exists 2024-03-04T06:09:27.906 INFO:tasks.rados.rados.0.smithi064.stdout:397: left oid 30 (ObjNum 205 snap 44 seq_num 205) 2024-03-04T06:09:27.906 INFO:tasks.rados.rados.0.smithi064.stdout:400: finishing write tid 2 to smithi06431743-32 2024-03-04T06:09:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:399: finishing write tid 3 to smithi06431743-50 2024-03-04T06:09:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:399: finishing write tid 4 to smithi06431743-50 2024-03-04T06:09:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:399: finishing write tid 5 to smithi06431743-50 2024-03-04T06:09:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:399: oid 50 updating version 0 to 113 2024-03-04T06:09:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:399: oid 50 updating version 113 to 114 2024-03-04T06:09:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:399: oid 50 version 114 is already newer than 112 2024-03-04T06:09:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:399: oid 50 updating version 114 to 115 2024-03-04T06:09:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 115 (ObjNum 207 snap 44 seq_num 207) dirty exists 2024-03-04T06:09:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:399: left oid 50 (ObjNum 207 snap 44 seq_num 207) 2024-03-04T06:09:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:400: finishing write tid 3 to smithi06431743-32 2024-03-04T06:09:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:400: finishing write tid 4 to smithi06431743-32 2024-03-04T06:09:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:400: finishing write tid 5 to smithi06431743-32 2024-03-04T06:09:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:400: oid 32 updating version 0 to 148 2024-03-04T06:09:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:400: oid 32 updating version 148 to 150 2024-03-04T06:09:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:400: oid 32 updating version 150 to 151 2024-03-04T06:09:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:400: oid 32 version 151 is already newer than 150 2024-03-04T06:09:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 151 (ObjNum 208 snap 44 seq_num 208) dirty exists 2024-03-04T06:09:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:400: left oid 32 (ObjNum 208 snap 44 seq_num 208) 2024-03-04T06:09:28.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:28.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:28.154+0000 7f2c7575d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:28.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:28.158+0000 7f2c7575d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:28.185 INFO:tasks.rados.rados.0.smithi064.stdout:397: done (4 left) 2024-03-04T06:09:28.185 INFO:tasks.rados.rados.0.smithi064.stdout:399: done (3 left) 2024-03-04T06:09:28.186 INFO:tasks.rados.rados.0.smithi064.stdout:400: done (2 left) 2024-03-04T06:09:28.186 INFO:tasks.rados.rados.0.smithi064.stdout:401: done (1 left) 2024-03-04T06:09:28.186 INFO:tasks.rados.rados.0.smithi064.stdout:402: done (0 left) 2024-03-04T06:09:28.186 INFO:tasks.rados.rados.0.smithi064.stdout:403: read oid 17 snap -1 2024-03-04T06:09:28.186 INFO:tasks.rados.rados.0.smithi064.stdout:403: expect (ObjNum 201 snap 43 seq_num 201) 2024-03-04T06:09:28.186 INFO:tasks.rados.rados.0.smithi064.stdout:404: read oid 35 snap 37 2024-03-04T06:09:28.186 INFO:tasks.rados.rados.0.smithi064.stdout:404: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-04T06:09:28.186 INFO:tasks.rados.rados.0.smithi064.stdout:405: read oid 8 snap -1 2024-03-04T06:09:28.186 INFO:tasks.rados.rados.0.smithi064.stdout:405: expect deleted 2024-03-04T06:09:28.186 INFO:tasks.rados.rados.0.smithi064.stdout:406: snap_remove snap 44 2024-03-04T06:09:29.184 INFO:tasks.rados.rados.0.smithi064.stdout:406: done (3 left) 2024-03-04T06:09:29.184 INFO:tasks.rados.rados.0.smithi064.stdout:407: delete oid 4 current snap is 45 2024-03-04T06:09:29.184 INFO:tasks.rados.rados.0.smithi064.stdout:403: expect (ObjNum 201 snap 43 seq_num 201) 2024-03-04T06:09:29.401 INFO:tasks.rados.rados.0.smithi064.stdout:404: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-04T06:09:29.642 INFO:tasks.rados.rados.0.smithi064.stdout:403: done (3 left) 2024-03-04T06:09:29.642 INFO:tasks.rados.rados.0.smithi064.stdout:404: done (2 left) 2024-03-04T06:09:29.642 INFO:tasks.rados.rados.0.smithi064.stdout:405: done (1 left) 2024-03-04T06:09:29.642 INFO:tasks.rados.rados.0.smithi064.stdout:407: done (0 left) 2024-03-04T06:09:29.642 INFO:tasks.rados.rados.0.smithi064.stdout:408: write oid 13 current snap is 45 2024-03-04T06:09:29.642 INFO:tasks.rados.rados.0.smithi064.stdout:408: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-03-04T06:09:29.648 INFO:tasks.rados.rados.0.smithi064.stdout:408: writing smithi06431743-13 from 729470 to 1273291 tid 1 2024-03-04T06:09:29.654 INFO:tasks.rados.rados.0.smithi064.stdout:408: writing smithi06431743-13 from 1772395 to 2310463 tid 2 2024-03-04T06:09:29.659 INFO:tasks.rados.rados.0.smithi064.stdout:408: writing smithi06431743-13 from 2900946 to 3303951 tid 3 2024-03-04T06:09:29.660 INFO:tasks.rados.rados.0.smithi064.stdout:409: write (excl) oid 14 current snap is 45 2024-03-04T06:09:29.660 INFO:tasks.rados.rados.0.smithi064.stdout:409: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-03-04T06:09:29.667 INFO:tasks.rados.rados.0.smithi064.stdout:409: writing smithi06431743-14 from 746277 to 1365347 tid 1 2024-03-04T06:09:29.673 INFO:tasks.rados.rados.0.smithi064.stdout:409: writing smithi06431743-14 from 1830877 to 2512603 tid 2 2024-03-04T06:09:29.679 INFO:tasks.rados.rados.0.smithi064.stdout:409: writing smithi06431743-14 from 2928369 to 3546277 tid 3 2024-03-04T06:09:29.679 INFO:tasks.rados.rados.0.smithi064.stdout:410: read oid 27 snap -1 2024-03-04T06:09:29.679 INFO:tasks.rados.rados.0.smithi064.stdout:410: expect deleted 2024-03-04T06:09:29.679 INFO:tasks.rados.rados.0.smithi064.stdout:411: snap_remove snap 37 2024-03-04T06:09:30.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:30.306 INFO:tasks.rados.rados.0.smithi064.stdout:411: done (3 left) 2024-03-04T06:09:30.306 INFO:tasks.rados.rados.0.smithi064.stdout:412: write (excl) oid 31 current snap is 45 2024-03-04T06:09:30.306 INFO:tasks.rados.rados.0.smithi064.stdout:412: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-03-04T06:09:30.317 INFO:tasks.rados.rados.0.smithi064.stdout:412: writing smithi06431743-31 from 763084 to 1457403 tid 1 2024-03-04T06:09:30.325 INFO:tasks.rados.rados.0.smithi064.stdout:412: writing smithi06431743-31 from 1889359 to 2431096 tid 2 2024-03-04T06:09:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:412: writing smithi06431743-31 from 2955792 to 3497694 tid 3 2024-03-04T06:09:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:413: snap_create 2024-03-04T06:09:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:408: finishing write tid 1 to smithi06431743-13 2024-03-04T06:09:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:408: finishing write tid 2 to smithi06431743-13 2024-03-04T06:09:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:409: finishing write tid 1 to smithi06431743-14 2024-03-04T06:09:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:408: finishing write tid 3 to smithi06431743-13 2024-03-04T06:09:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:408: finishing write tid 4 to smithi06431743-13 2024-03-04T06:09:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:408: finishing write tid 5 to smithi06431743-13 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:408: oid 13 updating version 0 to 119 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:408: oid 13 version 119 is already newer than 118 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:408: oid 13 updating version 119 to 121 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:408: oid 13 version 121 is already newer than 120 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 121 (ObjNum 210 snap 45 seq_num 210) dirty exists 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:408: left oid 13 (ObjNum 210 snap 45 seq_num 210) 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:409: finishing write tid 2 to smithi06431743-14 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:409: finishing write tid 3 to smithi06431743-14 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:409: finishing write tid 4 to smithi06431743-14 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:409: finishing write tid 5 to smithi06431743-14 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:409: oid 14 updating version 0 to 76 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:409: oid 14 version 76 is already newer than 74 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:409: oid 14 version 76 is already newer than 73 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:409: oid 14 version 76 is already newer than 75 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 76 (ObjNum 211 snap 45 seq_num 211) dirty exists 2024-03-04T06:09:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:409: left oid 14 (ObjNum 211 snap 45 seq_num 211) 2024-03-04T06:09:30.385 INFO:tasks.rados.rados.0.smithi064.stdout:412: finishing write tid 1 to smithi06431743-31 2024-03-04T06:09:30.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:30.386+0000 7ffa3962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:30.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:30.391+0000 7ffa3962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:30.397 INFO:tasks.rados.rados.0.smithi064.stdout:412: finishing write tid 2 to smithi06431743-31 2024-03-04T06:09:30.414 INFO:tasks.rados.rados.0.smithi064.stdout:412: finishing write tid 3 to smithi06431743-31 2024-03-04T06:09:30.418 INFO:tasks.rados.rados.0.smithi064.stdout:412: finishing write tid 4 to smithi06431743-31 2024-03-04T06:09:30.418 INFO:tasks.rados.rados.0.smithi064.stdout:412: finishing write tid 5 to smithi06431743-31 2024-03-04T06:09:30.418 INFO:tasks.rados.rados.0.smithi064.stdout:412: oid 31 updating version 0 to 25 2024-03-04T06:09:30.418 INFO:tasks.rados.rados.0.smithi064.stdout:412: oid 31 version 25 is already newer than 22 2024-03-04T06:09:30.418 INFO:tasks.rados.rados.0.smithi064.stdout:412: oid 31 version 25 is already newer than 23 2024-03-04T06:09:30.419 INFO:tasks.rados.rados.0.smithi064.stdout:412: oid 31 version 25 is already newer than 24 2024-03-04T06:09:30.419 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 25 (ObjNum 212 snap 45 seq_num 212) dirty exists 2024-03-04T06:09:30.419 INFO:tasks.rados.rados.0.smithi064.stdout:412: left oid 31 (ObjNum 212 snap 45 seq_num 212) 2024-03-04T06:09:30.552 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:09:30.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:09:30.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:30.655+0000 7f825559d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:30.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:30.663+0000 7f825559d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:31.306 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:09:31.309 INFO:tasks.rados.rados.0.smithi064.stdout:408: done (4 left) 2024-03-04T06:09:31.309 INFO:tasks.rados.rados.0.smithi064.stdout:409: done (3 left) 2024-03-04T06:09:31.309 INFO:tasks.rados.rados.0.smithi064.stdout:410: done (2 left) 2024-03-04T06:09:31.309 INFO:tasks.rados.rados.0.smithi064.stdout:412: done (1 left) 2024-03-04T06:09:31.310 INFO:tasks.rados.rados.0.smithi064.stdout:413: done (0 left) 2024-03-04T06:09:31.310 INFO:tasks.rados.rados.0.smithi064.stdout:414: snap_remove snap 43 2024-03-04T06:09:31.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:09:31.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:31.425+0000 7f988219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:31.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:31.433+0000 7f988219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:32.308 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:09:32.313 INFO:tasks.rados.rados.0.smithi064.stdout:414: done (0 left) 2024-03-04T06:09:32.314 INFO:tasks.rados.rados.0.smithi064.stdout:415: read oid 22 snap -1 2024-03-04T06:09:32.314 INFO:tasks.rados.rados.0.smithi064.stdout:415: expect (ObjNum 194 snap 40 seq_num 194) 2024-03-04T06:09:32.314 INFO:tasks.rados.rados.0.smithi064.stdout:416: snap_remove snap 38 2024-03-04T06:09:32.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:09:32.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:32.442+0000 7f6578e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:32.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:32.450+0000 7f6578e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:32.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:32.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:32.567+0000 7fedf246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:32.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:32.571+0000 7fedf246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:33.314 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:09:33.318 INFO:tasks.rados.rados.0.smithi064.stdout:416: done (1 left) 2024-03-04T06:09:33.318 INFO:tasks.rados.rados.0.smithi064.stdout:417: write (excl) oid 14 current snap is 46 2024-03-04T06:09:33.319 INFO:tasks.rados.rados.0.smithi064.stdout:417: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-03-04T06:09:33.325 INFO:tasks.rados.rados.0.smithi064.stdout:417: writing smithi06431743-14 from 779891 to 1265812 tid 1 2024-03-04T06:09:33.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:09:33.333 INFO:tasks.rados.rados.0.smithi064.stdout:417: writing smithi06431743-14 from 1947841 to 2633236 tid 2 2024-03-04T06:09:33.338 INFO:tasks.rados.rados.0.smithi064.stdout:417: writing smithi06431743-14 from 3383215 to 3579891 tid 3 2024-03-04T06:09:33.338 INFO:tasks.rados.rados.0.smithi064.stdout:418: write oid 11 current snap is 46 2024-03-04T06:09:33.339 INFO:tasks.rados.rados.0.smithi064.stdout:418: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-03-04T06:09:33.345 INFO:tasks.rados.rados.0.smithi064.stdout:418: writing smithi06431743-11 from 796698 to 1357868 tid 1 2024-03-04T06:09:33.352 INFO:tasks.rados.rados.0.smithi064.stdout:418: writing smithi06431743-11 from 2006323 to 2551729 tid 2 2024-03-04T06:09:33.359 INFO:tasks.rados.rados.0.smithi064.stdout:418: writing smithi06431743-11 from 3126991 to 3596698 tid 3 2024-03-04T06:09:33.360 INFO:tasks.rados.rados.0.smithi064.stdout:419: read oid 30 snap -1 2024-03-04T06:09:33.360 INFO:tasks.rados.rados.0.smithi064.stdout:419: expect (ObjNum 205 snap 44 seq_num 205) 2024-03-04T06:09:33.360 INFO:tasks.rados.rados.0.smithi064.stdout:420: snap_create 2024-03-04T06:09:33.360 INFO:tasks.rados.rados.0.smithi064.stdout:415: expect (ObjNum 194 snap 40 seq_num 194) 2024-03-04T06:09:33.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:33.448+0000 7f123a99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:33.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:33.463+0000 7f123a99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:33.502 INFO:tasks.rados.rados.0.smithi064.stdout:419: expect (ObjNum 205 snap 44 seq_num 205) 2024-03-04T06:09:33.674 INFO:tasks.rados.rados.0.smithi064.stdout:418: finishing write tid 1 to smithi06431743-11 2024-03-04T06:09:33.674 INFO:tasks.rados.rados.0.smithi064.stdout:417: finishing write tid 1 to smithi06431743-14 2024-03-04T06:09:33.674 INFO:tasks.rados.rados.0.smithi064.stdout:418: finishing write tid 2 to smithi06431743-11 2024-03-04T06:09:33.674 INFO:tasks.rados.rados.0.smithi064.stdout:418: finishing write tid 3 to smithi06431743-11 2024-03-04T06:09:33.674 INFO:tasks.rados.rados.0.smithi064.stdout:417: finishing write tid 2 to smithi06431743-14 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:418: finishing write tid 4 to smithi06431743-11 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:418: finishing write tid 5 to smithi06431743-11 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:418: oid 11 updating version 0 to 85 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:418: oid 11 updating version 85 to 86 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:418: oid 11 version 86 is already newer than 85 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:418: oid 11 version 86 is already newer than 83 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 86 (ObjNum 214 snap 46 seq_num 214) dirty exists 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:418: left oid 11 (ObjNum 214 snap 46 seq_num 214) 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:417: finishing write tid 3 to smithi06431743-14 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:417: finishing write tid 4 to smithi06431743-14 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:417: finishing write tid 5 to smithi06431743-14 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:417: oid 14 updating version 0 to 87 2024-03-04T06:09:33.675 INFO:tasks.rados.rados.0.smithi064.stdout:417: oid 14 updating version 87 to 89 2024-03-04T06:09:33.676 INFO:tasks.rados.rados.0.smithi064.stdout:417: oid 14 version 89 is already newer than 89 2024-03-04T06:09:33.676 INFO:tasks.rados.rados.0.smithi064.stdout:417: oid 14 version 89 is already newer than 86 2024-03-04T06:09:33.676 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 89 (ObjNum 213 snap 46 seq_num 213) dirty exists 2024-03-04T06:09:33.676 INFO:tasks.rados.rados.0.smithi064.stdout:417: left oid 14 (ObjNum 213 snap 46 seq_num 213) 2024-03-04T06:09:34.318 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:09:34.321 INFO:tasks.rados.rados.0.smithi064.stdout:415: done (4 left) 2024-03-04T06:09:34.321 INFO:tasks.rados.rados.0.smithi064.stdout:417: done (3 left) 2024-03-04T06:09:34.321 INFO:tasks.rados.rados.0.smithi064.stdout:418: done (2 left) 2024-03-04T06:09:34.321 INFO:tasks.rados.rados.0.smithi064.stdout:419: done (1 left) 2024-03-04T06:09:34.321 INFO:tasks.rados.rados.0.smithi064.stdout:420: done (0 left) 2024-03-04T06:09:34.321 INFO:tasks.rados.rados.0.smithi064.stdout:421: write (excl) oid 13 current snap is 47 2024-03-04T06:09:34.322 INFO:tasks.rados.rados.0.smithi064.stdout:421: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-03-04T06:09:34.329 INFO:tasks.rados.rados.0.smithi064.stdout:421: writing smithi06431743-13 from 413505 to 1049924 tid 1 2024-03-04T06:09:34.336 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:09:34.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:09:34.337 INFO:tasks.rados.rados.0.smithi064.stdout:421: writing smithi06431743-13 from 1664805 to 2353869 tid 2 2024-03-04T06:09:34.343 INFO:tasks.rados.rados.0.smithi064.stdout:421: writing smithi06431743-13 from 3038061 to 3530132 tid 3 2024-03-04T06:09:34.344 INFO:tasks.rados.rados.0.smithi064.stdout:422: write oid 48 current snap is 47 2024-03-04T06:09:34.344 INFO:tasks.rados.rados.0.smithi064.stdout:422: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-03-04T06:09:34.349 INFO:tasks.rados.rados.0.smithi064.stdout:422: writing smithi06431743-48 from 430312 to 1141980 tid 1 2024-03-04T06:09:34.356 INFO:tasks.rados.rados.0.smithi064.stdout:422: writing smithi06431743-48 from 1723287 to 2272362 tid 2 2024-03-04T06:09:34.363 INFO:tasks.rados.rados.0.smithi064.stdout:422: writing smithi06431743-48 from 2781837 to 3485180 tid 3 2024-03-04T06:09:34.363 INFO:tasks.rados.rados.0.smithi064.stdout:423: write (excl) oid 12 current snap is 47 2024-03-04T06:09:34.363 INFO:tasks.rados.rados.0.smithi064.stdout:423: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-03-04T06:09:34.369 INFO:tasks.rados.rados.0.smithi064.stdout:423: writing smithi06431743-12 from 447119 to 1234036 tid 1 2024-03-04T06:09:34.376 INFO:tasks.rados.rados.0.smithi064.stdout:423: writing smithi06431743-12 from 1665416 to 2358149 tid 2 2024-03-04T06:09:34.384 INFO:tasks.rados.rados.0.smithi064.stdout:423: writing smithi06431743-12 from 2976554 to 3491169 tid 3 2024-03-04T06:09:34.384 INFO:tasks.rados.rados.0.smithi064.stdout:424: snap_remove snap 45 2024-03-04T06:09:34.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:34.467+0000 7f54f546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:34.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:34.474+0000 7f54f546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:34.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:34.671 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:09:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:09:34.684 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-03-04T06:09:34.319851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":111919804,"num_objects":151,"num_object_clones":92,"num_object_copies":453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":960,"num_read_kb":579698,"num_write":902,"num_write_kb":211711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1114,"ondisk_log_size":1114,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":384016,"kb_used_data":268400,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377103344,"statfs":{"total":386547056640,"available":386153824256,"internally_reserved":0,"allocated":274841600,"data_stored":291409182,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2285719,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":73,"num_read_kb":71452,"num_write":43,"num_write_kb":12203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015210"},"pg_stats":[{"pgid":"3.f","version":"98'54","reported_seq":300,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.204540+0000","last_change":"2024-03-04T06:09:29.211869+0000","last_active":"2024-03-04T06:09:29.204540+0000","last_peered":"2024-03-04T06:09:29.204540+0000","last_clean":"2024-03-04T06:09:29.204540+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:09:29.204540+0000","last_undegraded":"2024-03-04T06:09:29.204540+0000","last_fullsized":"2024-03-04T06:09:29.204540+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":22,"num_read_kb":6453,"num_write":40,"num_write_kb":11069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"93'51","reported_seq":321,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.196312+0000","last_change":"2024-03-04T06:09:29.206727+0000","last_active":"2024-03-04T06:09:29.196312+0000","last_peered":"2024-03-04T06:09:29.196312+0000","last_clean":"2024-03-04T06:09:29.196312+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:09:29.196312+0000","last_undegraded":"2024-03-04T06:09:29.196312+0000","last_fullsized":"2024-03-04T06:09:29.196312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092488,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":33,"num_read_kb":21067,"num_write":36,"num_write_kb":9740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"70'82","reported_seq":330,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.875645+0000","last_change":"2024-03-04T06:09:29.885491+0000","last_active":"2024-03-04T06:09:29.875645+0000","last_peered":"2024-03-04T06:09:29.875645+0000","last_clean":"2024-03-04T06:09:29.875645+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:09:29.875645+0000","last_undegraded":"2024-03-04T06:09:29.875645+0000","last_fullsized":"2024-03-04T06:09:29.875645+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4823609,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":51,"num_read_kb":29739,"num_write":53,"num_write_kb":14601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"94'56","reported_seq":321,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.196630+0000","last_change":"2024-03-04T06:09:29.200543+0000","last_active":"2024-03-04T06:09:29.196630+0000","last_peered":"2024-03-04T06:09:29.196630+0000","last_clean":"2024-03-04T06:09:29.196630+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:09:29.196630+0000","last_undegraded":"2024-03-04T06:09:29.196630+0000","last_fullsized":"2024-03-04T06:09:29.196630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6537923,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":53,"num_read_kb":46128,"num_write":47,"num_write_kb":13226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"102'25","reported_seq":304,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.329339+0000","last_change":"2024-03-04T06:09:30.351227+0000","last_active":"2024-03-04T06:09:31.329339+0000","last_peered":"2024-03-04T06:09:31.329339+0000","last_clean":"2024-03-04T06:09:31.329339+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:09:31.329339+0000","last_undegraded":"2024-03-04T06:09:31.329339+0000","last_fullsized":"2024-03-04T06:09:31.329339+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3563084,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":17038,"num_write":18,"num_write_kb":5229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"66'42","reported_seq":300,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.206235+0000","last_change":"2024-03-04T06:09:29.211420+0000","last_active":"2024-03-04T06:09:29.206235+0000","last_peered":"2024-03-04T06:09:29.206235+0000","last_clean":"2024-03-04T06:09:29.206235+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:09:29.206235+0000","last_undegraded":"2024-03-04T06:09:29.206235+0000","last_fullsized":"2024-03-04T06:09:29.206235+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218557,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":28495,"num_write":32,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":256,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.205384+0000","last_change":"2024-03-04T06:09:29.212129+0000","last_active":"2024-03-04T06:09:29.205384+0000","last_peered":"2024-03-04T06:09:29.205384+0000","last_clean":"2024-03-04T06:09:29.205384+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:09:29.205384+0000","last_undegraded":"2024-03-04T06:09:29.205384+0000","last_fullsized":"2024-03-04T06:09:29.205384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"98'60","reported_seq":320,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.881156+0000","last_change":"2024-03-04T06:09:29.886098+0000","last_active":"2024-03-04T06:09:29.881156+0000","last_peered":"2024-03-04T06:09:29.881156+0000","last_clean":"2024-03-04T06:09:29.881156+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:09:29.881156+0000","last_undegraded":"2024-03-04T06:09:29.881156+0000","last_fullsized":"2024-03-04T06:09:29.881156+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764801,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":21919,"num_write":48,"num_write_kb":13847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":380,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.319040+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:09:31.319040+0000","last_peered":"2024-03-04T06:09:31.319040+0000","last_clean":"2024-03-04T06:09:31.319040+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:09:31.319040+0000","last_undegraded":"2024-03-04T06:09:31.319040+0000","last_fullsized":"2024-03-04T06:09:31.319040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":233,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.887300+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:09:29.887300+0000","last_peered":"2024-03-04T06:09:29.887300+0000","last_clean":"2024-03-04T06:09:29.887300+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:09:29.887300+0000","last_undegraded":"2024-03-04T06:09:29.887300+0000","last_fullsized":"2024-03-04T06:09:29.887300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"99'45","reported_seq":313,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.883589+0000","last_change":"2024-03-04T06:09:29.888313+0000","last_active":"2024-03-04T06:09:29.883589+0000","last_peered":"2024-03-04T06:09:29.883589+0000","last_clean":"2024-03-04T06:09:29.883589+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:09:29.883589+0000","last_undegraded":"2024-03-04T06:09:29.883589+0000","last_fullsized":"2024-03-04T06:09:29.883589+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10151395,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":42,"num_read_kb":36818,"num_write":36,"num_write_kb":9883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":227,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.203574+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:09:29.203574+0000","last_peered":"2024-03-04T06:09:29.203574+0000","last_clean":"2024-03-04T06:09:29.203574+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:09:29.203574+0000","last_undegraded":"2024-03-04T06:09:29.203574+0000","last_fullsized":"2024-03-04T06:09:29.203574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"82'41","reported_seq":321,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.320772+0000","last_change":"2024-03-04T06:09:30.347017+0000","last_active":"2024-03-04T06:09:31.320772+0000","last_peered":"2024-03-04T06:09:31.320772+0000","last_clean":"2024-03-04T06:09:31.320772+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:09:31.320772+0000","last_undegraded":"2024-03-04T06:09:31.320772+0000","last_fullsized":"2024-03-04T06:09:31.320772+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210170,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31345,"num_write":29,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":233,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.878672+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:09:29.878672+0000","last_peered":"2024-03-04T06:09:29.878672+0000","last_clean":"2024-03-04T06:09:29.878672+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:09:29.878672+0000","last_undegraded":"2024-03-04T06:09:29.878672+0000","last_fullsized":"2024-03-04T06:09:29.878672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"102'155","reported_seq":436,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.331619+0000","last_change":"2024-03-04T06:09:30.388017+0000","last_active":"2024-03-04T06:09:31.331619+0000","last_peered":"2024-03-04T06:09:31.331619+0000","last_clean":"2024-03-04T06:09:31.331619+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:09:31.331619+0000","last_undegraded":"2024-03-04T06:09:31.331619+0000","last_fullsized":"2024-03-04T06:09:31.331619+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15075879,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":138,"num_read_kb":100582,"num_write":111,"num_write_kb":32075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":227,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.193086+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:09:29.193086+0000","last_peered":"2024-03-04T06:09:29.193086+0000","last_clean":"2024-03-04T06:09:29.193086+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:09:29.193086+0000","last_undegraded":"2024-03-04T06:09:29.193086+0000","last_fullsized":"2024-03-04T06:09:29.193086+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"101'121","reported_seq":393,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.737064+0000","last_change":"2024-03-04T06:09:29.207592+0000","last_active":"2024-03-04T06:09:29.737064+0000","last_peered":"2024-03-04T06:09:29.737064+0000","last_clean":"2024-03-04T06:09:29.737064+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:09:29.737064+0000","last_undegraded":"2024-03-04T06:09:29.737064+0000","last_fullsized":"2024-03-04T06:09:29.737064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10302691,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":80,"num_read_kb":47361,"num_write":96,"num_write_kb":25007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"94'45","reported_seq":320,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.886605+0000","last_change":"2024-03-04T06:09:29.889466+0000","last_active":"2024-03-04T06:09:29.886605+0000","last_peered":"2024-03-04T06:09:29.886605+0000","last_clean":"2024-03-04T06:09:29.886605+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:09:29.886605+0000","last_undegraded":"2024-03-04T06:09:29.886605+0000","last_fullsized":"2024-03-04T06:09:29.886605+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8353046,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":36976,"num_write":37,"num_write_kb":9040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":247,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.204730+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:09:29.204730+0000","last_peered":"2024-03-04T06:09:29.204730+0000","last_clean":"2024-03-04T06:09:29.204730+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:09:29.204730+0000","last_undegraded":"2024-03-04T06:09:29.204730+0000","last_fullsized":"2024-03-04T06:09:29.204730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"98'45","reported_seq":339,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.209385+0000","last_change":"2024-03-04T06:09:29.213320+0000","last_active":"2024-03-04T06:09:29.209385+0000","last_peered":"2024-03-04T06:09:29.209385+0000","last_clean":"2024-03-04T06:09:29.209385+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:09:29.209385+0000","last_undegraded":"2024-03-04T06:09:29.209385+0000","last_fullsized":"2024-03-04T06:09:29.209385+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151362,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":59,"num_read_kb":33400,"num_write":42,"num_write_kb":10631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":247,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.325969+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:09:31.325969+0000","last_peered":"2024-03-04T06:09:31.325969+0000","last_clean":"2024-03-04T06:09:31.325969+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:09:31.325969+0000","last_undegraded":"2024-03-04T06:09:31.325969+0000","last_fullsized":"2024-03-04T06:09:31.325969+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"102'80","reported_seq":373,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.318185+0000","last_change":"2024-03-04T06:09:30.383642+0000","last_active":"2024-03-04T06:09:31.318185+0000","last_peered":"2024-03-04T06:09:31.318185+0000","last_clean":"2024-03-04T06:09:31.318185+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:09:31.318185+0000","last_undegraded":"2024-03-04T06:09:31.318185+0000","last_fullsized":"2024-03-04T06:09:31.318185+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13159848,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":86,"num_read_kb":57568,"num_write":66,"num_write_kb":17346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":247,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.199692+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:09:29.199692+0000","last_peered":"2024-03-04T06:09:29.199692+0000","last_clean":"2024-03-04T06:09:29.199692+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:09:29.199692+0000","last_undegraded":"2024-03-04T06:09:29.199692+0000","last_fullsized":"2024-03-04T06:09:29.199692+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"99'115","reported_seq":386,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.198773+0000","last_change":"2024-03-04T06:09:29.207714+0000","last_active":"2024-03-04T06:09:29.198773+0000","last_peered":"2024-03-04T06:09:29.198773+0000","last_clean":"2024-03-04T06:09:29.198773+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:09:29.198773+0000","last_undegraded":"2024-03-04T06:09:29.198773+0000","last_fullsized":"2024-03-04T06:09:29.198773+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12924583,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":98,"num_read_kb":64718,"num_write":74,"num_write_kb":20323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":233,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.886670+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:09:29.886670+0000","last_peered":"2024-03-04T06:09:29.886670+0000","last_clean":"2024-03-04T06:09:29.886670+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:09:29.886670+0000","last_undegraded":"2024-03-04T06:09:29.886670+0000","last_fullsized":"2024-03-04T06:09:29.886670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":111329436,"num_objects":149,"num_object_clones":92,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":857,"num_read_kb":579607,"num_write":765,"num_write_kb":209817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1017,"ondisk_log_size":1017,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":117416,"kb_used_data":86432,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94254424,"statfs":{"total":96636764160,"available":96516530176,"internally_reserved":0,"allocated":88506368,"data_stored":94994064,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91448,"kb_used_data":64624,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94280392,"statfs":{"total":96636764160,"available":96543121408,"internally_reserved":0,"allocated":66174976,"data_stored":68639587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96444,"kb_used_data":65460,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94275396,"statfs":{"total":96636764160,"available":96538005504,"internally_reserved":0,"allocated":67031040,"data_stored":72003803,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78708,"kb_used_data":51884,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94293132,"statfs":{"total":96636764160,"available":96556167168,"internally_reserved":0,"allocated":53129216,"data_stored":55771728,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:09:34.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:09:34.744 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:34.742+0000 7f5c4159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:34.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:34.746+0000 7f5c4159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:34.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:34.831+0000 7f661cd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:34.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:34.837+0000 7f661cd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:35.025 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:09:35.037 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-03-04T06:09:34.727186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":111919804,"num_objects":151,"num_object_clones":92,"num_object_copies":453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":960,"num_read_kb":579698,"num_write":902,"num_write_kb":211711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1114,"ondisk_log_size":1114,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":384016,"kb_used_data":268400,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377103344,"statfs":{"total":386547056640,"available":386153824256,"internally_reserved":0,"allocated":274841600,"data_stored":291409182,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5395047,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":51,"num_read_kb":50071,"num_write":25,"num_write_kb":7877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000657"},"pg_stats":[{"pgid":"3.f","version":"98'54","reported_seq":300,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.204540+0000","last_change":"2024-03-04T06:09:29.211869+0000","last_active":"2024-03-04T06:09:29.204540+0000","last_peered":"2024-03-04T06:09:29.204540+0000","last_clean":"2024-03-04T06:09:29.204540+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:09:29.204540+0000","last_undegraded":"2024-03-04T06:09:29.204540+0000","last_fullsized":"2024-03-04T06:09:29.204540+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":22,"num_read_kb":6453,"num_write":40,"num_write_kb":11069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"93'51","reported_seq":321,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.196312+0000","last_change":"2024-03-04T06:09:29.206727+0000","last_active":"2024-03-04T06:09:29.196312+0000","last_peered":"2024-03-04T06:09:29.196312+0000","last_clean":"2024-03-04T06:09:29.196312+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:09:29.196312+0000","last_undegraded":"2024-03-04T06:09:29.196312+0000","last_fullsized":"2024-03-04T06:09:29.196312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092488,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":33,"num_read_kb":21067,"num_write":36,"num_write_kb":9740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"70'82","reported_seq":330,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.875645+0000","last_change":"2024-03-04T06:09:29.885491+0000","last_active":"2024-03-04T06:09:29.875645+0000","last_peered":"2024-03-04T06:09:29.875645+0000","last_clean":"2024-03-04T06:09:29.875645+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:09:29.875645+0000","last_undegraded":"2024-03-04T06:09:29.875645+0000","last_fullsized":"2024-03-04T06:09:29.875645+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4823609,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":51,"num_read_kb":29739,"num_write":53,"num_write_kb":14601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"94'56","reported_seq":321,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.196630+0000","last_change":"2024-03-04T06:09:29.200543+0000","last_active":"2024-03-04T06:09:29.196630+0000","last_peered":"2024-03-04T06:09:29.196630+0000","last_clean":"2024-03-04T06:09:29.196630+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:09:29.196630+0000","last_undegraded":"2024-03-04T06:09:29.196630+0000","last_fullsized":"2024-03-04T06:09:29.196630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6537923,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":53,"num_read_kb":46128,"num_write":47,"num_write_kb":13226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"102'25","reported_seq":304,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.329339+0000","last_change":"2024-03-04T06:09:30.351227+0000","last_active":"2024-03-04T06:09:31.329339+0000","last_peered":"2024-03-04T06:09:31.329339+0000","last_clean":"2024-03-04T06:09:31.329339+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:09:31.329339+0000","last_undegraded":"2024-03-04T06:09:31.329339+0000","last_fullsized":"2024-03-04T06:09:31.329339+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3563084,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":17038,"num_write":18,"num_write_kb":5229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"66'42","reported_seq":300,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.206235+0000","last_change":"2024-03-04T06:09:29.211420+0000","last_active":"2024-03-04T06:09:29.206235+0000","last_peered":"2024-03-04T06:09:29.206235+0000","last_clean":"2024-03-04T06:09:29.206235+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:09:29.206235+0000","last_undegraded":"2024-03-04T06:09:29.206235+0000","last_fullsized":"2024-03-04T06:09:29.206235+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218557,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":28495,"num_write":32,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":256,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.205384+0000","last_change":"2024-03-04T06:09:29.212129+0000","last_active":"2024-03-04T06:09:29.205384+0000","last_peered":"2024-03-04T06:09:29.205384+0000","last_clean":"2024-03-04T06:09:29.205384+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:09:29.205384+0000","last_undegraded":"2024-03-04T06:09:29.205384+0000","last_fullsized":"2024-03-04T06:09:29.205384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"98'60","reported_seq":320,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.881156+0000","last_change":"2024-03-04T06:09:29.886098+0000","last_active":"2024-03-04T06:09:29.881156+0000","last_peered":"2024-03-04T06:09:29.881156+0000","last_clean":"2024-03-04T06:09:29.881156+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:09:29.881156+0000","last_undegraded":"2024-03-04T06:09:29.881156+0000","last_fullsized":"2024-03-04T06:09:29.881156+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764801,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":21919,"num_write":48,"num_write_kb":13847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":380,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.319040+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:09:31.319040+0000","last_peered":"2024-03-04T06:09:31.319040+0000","last_clean":"2024-03-04T06:09:31.319040+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:09:31.319040+0000","last_undegraded":"2024-03-04T06:09:31.319040+0000","last_fullsized":"2024-03-04T06:09:31.319040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":233,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.887300+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:09:29.887300+0000","last_peered":"2024-03-04T06:09:29.887300+0000","last_clean":"2024-03-04T06:09:29.887300+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:09:29.887300+0000","last_undegraded":"2024-03-04T06:09:29.887300+0000","last_fullsized":"2024-03-04T06:09:29.887300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"99'45","reported_seq":313,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.883589+0000","last_change":"2024-03-04T06:09:29.888313+0000","last_active":"2024-03-04T06:09:29.883589+0000","last_peered":"2024-03-04T06:09:29.883589+0000","last_clean":"2024-03-04T06:09:29.883589+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:09:29.883589+0000","last_undegraded":"2024-03-04T06:09:29.883589+0000","last_fullsized":"2024-03-04T06:09:29.883589+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10151395,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":42,"num_read_kb":36818,"num_write":36,"num_write_kb":9883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":227,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.203574+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:09:29.203574+0000","last_peered":"2024-03-04T06:09:29.203574+0000","last_clean":"2024-03-04T06:09:29.203574+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:09:29.203574+0000","last_undegraded":"2024-03-04T06:09:29.203574+0000","last_fullsized":"2024-03-04T06:09:29.203574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"82'41","reported_seq":321,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.320772+0000","last_change":"2024-03-04T06:09:30.347017+0000","last_active":"2024-03-04T06:09:31.320772+0000","last_peered":"2024-03-04T06:09:31.320772+0000","last_clean":"2024-03-04T06:09:31.320772+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:09:31.320772+0000","last_undegraded":"2024-03-04T06:09:31.320772+0000","last_fullsized":"2024-03-04T06:09:31.320772+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210170,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31345,"num_write":29,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":233,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.878672+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:09:29.878672+0000","last_peered":"2024-03-04T06:09:29.878672+0000","last_clean":"2024-03-04T06:09:29.878672+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:09:29.878672+0000","last_undegraded":"2024-03-04T06:09:29.878672+0000","last_fullsized":"2024-03-04T06:09:29.878672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"102'155","reported_seq":436,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.331619+0000","last_change":"2024-03-04T06:09:30.388017+0000","last_active":"2024-03-04T06:09:31.331619+0000","last_peered":"2024-03-04T06:09:31.331619+0000","last_clean":"2024-03-04T06:09:31.331619+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:09:31.331619+0000","last_undegraded":"2024-03-04T06:09:31.331619+0000","last_fullsized":"2024-03-04T06:09:31.331619+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15075879,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":138,"num_read_kb":100582,"num_write":111,"num_write_kb":32075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":227,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.193086+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:09:29.193086+0000","last_peered":"2024-03-04T06:09:29.193086+0000","last_clean":"2024-03-04T06:09:29.193086+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:09:29.193086+0000","last_undegraded":"2024-03-04T06:09:29.193086+0000","last_fullsized":"2024-03-04T06:09:29.193086+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"101'121","reported_seq":393,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.737064+0000","last_change":"2024-03-04T06:09:29.207592+0000","last_active":"2024-03-04T06:09:29.737064+0000","last_peered":"2024-03-04T06:09:29.737064+0000","last_clean":"2024-03-04T06:09:29.737064+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:09:29.737064+0000","last_undegraded":"2024-03-04T06:09:29.737064+0000","last_fullsized":"2024-03-04T06:09:29.737064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10302691,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":80,"num_read_kb":47361,"num_write":96,"num_write_kb":25007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"94'45","reported_seq":320,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.886605+0000","last_change":"2024-03-04T06:09:29.889466+0000","last_active":"2024-03-04T06:09:29.886605+0000","last_peered":"2024-03-04T06:09:29.886605+0000","last_clean":"2024-03-04T06:09:29.886605+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:09:29.886605+0000","last_undegraded":"2024-03-04T06:09:29.886605+0000","last_fullsized":"2024-03-04T06:09:29.886605+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8353046,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":36976,"num_write":37,"num_write_kb":9040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":247,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.204730+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:09:29.204730+0000","last_peered":"2024-03-04T06:09:29.204730+0000","last_clean":"2024-03-04T06:09:29.204730+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:09:29.204730+0000","last_undegraded":"2024-03-04T06:09:29.204730+0000","last_fullsized":"2024-03-04T06:09:29.204730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"98'45","reported_seq":339,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.209385+0000","last_change":"2024-03-04T06:09:29.213320+0000","last_active":"2024-03-04T06:09:29.209385+0000","last_peered":"2024-03-04T06:09:29.209385+0000","last_clean":"2024-03-04T06:09:29.209385+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:09:29.209385+0000","last_undegraded":"2024-03-04T06:09:29.209385+0000","last_fullsized":"2024-03-04T06:09:29.209385+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151362,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":59,"num_read_kb":33400,"num_write":42,"num_write_kb":10631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":247,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.325969+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:09:31.325969+0000","last_peered":"2024-03-04T06:09:31.325969+0000","last_clean":"2024-03-04T06:09:31.325969+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:09:31.325969+0000","last_undegraded":"2024-03-04T06:09:31.325969+0000","last_fullsized":"2024-03-04T06:09:31.325969+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"102'80","reported_seq":373,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.318185+0000","last_change":"2024-03-04T06:09:30.383642+0000","last_active":"2024-03-04T06:09:31.318185+0000","last_peered":"2024-03-04T06:09:31.318185+0000","last_clean":"2024-03-04T06:09:31.318185+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:09:31.318185+0000","last_undegraded":"2024-03-04T06:09:31.318185+0000","last_fullsized":"2024-03-04T06:09:31.318185+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13159848,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":86,"num_read_kb":57568,"num_write":66,"num_write_kb":17346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":247,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.199692+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:09:29.199692+0000","last_peered":"2024-03-04T06:09:29.199692+0000","last_clean":"2024-03-04T06:09:29.199692+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:09:29.199692+0000","last_undegraded":"2024-03-04T06:09:29.199692+0000","last_fullsized":"2024-03-04T06:09:29.199692+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"99'115","reported_seq":386,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.198773+0000","last_change":"2024-03-04T06:09:29.207714+0000","last_active":"2024-03-04T06:09:29.198773+0000","last_peered":"2024-03-04T06:09:29.198773+0000","last_clean":"2024-03-04T06:09:29.198773+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:09:29.198773+0000","last_undegraded":"2024-03-04T06:09:29.198773+0000","last_fullsized":"2024-03-04T06:09:29.198773+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12924583,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":98,"num_read_kb":64718,"num_write":74,"num_write_kb":20323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":233,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.886670+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:09:29.886670+0000","last_peered":"2024-03-04T06:09:29.886670+0000","last_clean":"2024-03-04T06:09:29.886670+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:09:29.886670+0000","last_undegraded":"2024-03-04T06:09:29.886670+0000","last_fullsized":"2024-03-04T06:09:29.886670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":111329436,"num_objects":149,"num_object_clones":92,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":857,"num_read_kb":579607,"num_write":765,"num_write_kb":209817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1017,"ondisk_log_size":1017,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":117416,"kb_used_data":86432,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94254424,"statfs":{"total":96636764160,"available":96516530176,"internally_reserved":0,"allocated":88506368,"data_stored":94994064,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91448,"kb_used_data":64624,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94280392,"statfs":{"total":96636764160,"available":96543121408,"internally_reserved":0,"allocated":66174976,"data_stored":68639587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96444,"kb_used_data":65460,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94275396,"statfs":{"total":96636764160,"available":96538005504,"internally_reserved":0,"allocated":67031040,"data_stored":72003803,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78708,"kb_used_data":51884,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94293132,"statfs":{"total":96636764160,"available":96556167168,"internally_reserved":0,"allocated":53129216,"data_stored":55771728,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:09:35.038 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:09:35.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:09:35.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:09:35.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:35.181+0000 7fb289b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:35.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:35.187+0000 7fb289b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:35.325 INFO:tasks.rados.rados.0.smithi064.stdout:424: done (3 left) 2024-03-04T06:09:35.325 INFO:tasks.rados.rados.0.smithi064.stdout:425: snap_create 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:421: finishing write tid 1 to smithi06431743-13 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:422: finishing write tid 1 to smithi06431743-48 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:421: finishing write tid 2 to smithi06431743-13 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:422: finishing write tid 2 to smithi06431743-48 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:421: finishing write tid 3 to smithi06431743-13 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:423: finishing write tid 1 to smithi06431743-12 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:422: finishing write tid 3 to smithi06431743-48 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:422: finishing write tid 4 to smithi06431743-48 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:421: finishing write tid 4 to smithi06431743-13 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:421: finishing write tid 5 to smithi06431743-13 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:421: oid 13 updating version 0 to 151 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:421: oid 13 updating version 151 to 152 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:421: oid 13 version 152 is already newer than 150 2024-03-04T06:09:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:421: oid 13 version 152 is already newer than 149 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 152 (ObjNum 215 snap 47 seq_num 215) dirty exists 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:421: left oid 13 (ObjNum 215 snap 47 seq_num 215) 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:422: finishing write tid 5 to smithi06431743-48 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:422: oid 48 updating version 0 to 46 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:422: oid 48 version 46 is already newer than 46 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:422: oid 48 version 46 is already newer than 44 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:422: oid 48 version 46 is already newer than 43 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 46 (ObjNum 216 snap 47 seq_num 216) dirty exists 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:422: left oid 48 (ObjNum 216 snap 47 seq_num 216) 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:423: finishing write tid 2 to smithi06431743-12 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:423: finishing write tid 3 to smithi06431743-12 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:423: finishing write tid 4 to smithi06431743-12 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:423: finishing write tid 5 to smithi06431743-12 2024-03-04T06:09:35.327 INFO:tasks.rados.rados.0.smithi064.stdout:423: oid 12 updating version 0 to 61 2024-03-04T06:09:35.328 INFO:tasks.rados.rados.0.smithi064.stdout:423: oid 12 version 61 is already newer than 60 2024-03-04T06:09:35.328 INFO:tasks.rados.rados.0.smithi064.stdout:423: oid 12 updating version 61 to 63 2024-03-04T06:09:35.328 INFO:tasks.rados.rados.0.smithi064.stdout:423: oid 12 version 63 is already newer than 63 2024-03-04T06:09:35.328 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 63 (ObjNum 217 snap 47 seq_num 217) dirty exists 2024-03-04T06:09:35.328 INFO:tasks.rados.rados.0.smithi064.stdout:423: left oid 12 (ObjNum 217 snap 47 seq_num 217) 2024-03-04T06:09:35.378 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:09:35.378 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:09:35.390 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-03-04T06:09:35.323416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112339979,"num_objects":153,"num_object_clones":94,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":976,"num_read_kb":595680,"num_write":912,"num_write_kb":215436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1166,"ondisk_log_size":1166,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":395276,"kb_used_data":279660,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377092084,"statfs":{"total":386547056640,"available":386142294016,"internally_reserved":0,"allocated":286371840,"data_stored":307364313,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5815222,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":66053,"num_write":35,"num_write_kb":11602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016169"},"pg_stats":[{"pgid":"3.f","version":"106'58","reported_seq":316,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.408220+0000","last_change":"2024-03-04T06:09:34.454469+0000","last_active":"2024-03-04T06:09:34.408220+0000","last_peered":"2024-03-04T06:09:34.408220+0000","last_clean":"2024-03-04T06:09:34.408220+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:09:34.408220+0000","last_undegraded":"2024-03-04T06:09:34.408220+0000","last_fullsized":"2024-03-04T06:09:34.408220+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":22,"num_read_kb":6453,"num_write":40,"num_write_kb":11069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"93'51","reported_seq":339,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.335510+0000","last_change":"2024-03-04T06:09:33.338052+0000","last_active":"2024-03-04T06:09:34.335510+0000","last_peered":"2024-03-04T06:09:34.335510+0000","last_clean":"2024-03-04T06:09:34.335510+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:09:34.335510+0000","last_undegraded":"2024-03-04T06:09:34.335510+0000","last_fullsized":"2024-03-04T06:09:34.335510+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092488,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":33,"num_read_kb":21067,"num_write":36,"num_write_kb":9740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"70'82","reported_seq":330,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.875645+0000","last_change":"2024-03-04T06:09:29.885491+0000","last_active":"2024-03-04T06:09:29.875645+0000","last_peered":"2024-03-04T06:09:29.875645+0000","last_clean":"2024-03-04T06:09:29.875645+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:09:29.875645+0000","last_undegraded":"2024-03-04T06:09:29.875645+0000","last_fullsized":"2024-03-04T06:09:29.875645+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4823609,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":51,"num_read_kb":29739,"num_write":53,"num_write_kb":14601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'63","reported_seq":342,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.500262+0000","last_change":"2024-03-04T06:09:34.412923+0000","last_active":"2024-03-04T06:09:34.500262+0000","last_peered":"2024-03-04T06:09:34.500262+0000","last_clean":"2024-03-04T06:09:34.500262+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:09:34.500262+0000","last_undegraded":"2024-03-04T06:09:34.500262+0000","last_fullsized":"2024-03-04T06:09:34.500262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6874063,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":55,"num_read_kb":46129,"num_write":52,"num_write_kb":15175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"102'25","reported_seq":304,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.329339+0000","last_change":"2024-03-04T06:09:30.351227+0000","last_active":"2024-03-04T06:09:31.329339+0000","last_peered":"2024-03-04T06:09:31.329339+0000","last_clean":"2024-03-04T06:09:31.329339+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:09:31.329339+0000","last_undegraded":"2024-03-04T06:09:31.329339+0000","last_fullsized":"2024-03-04T06:09:31.329339+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3563084,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":17038,"num_write":18,"num_write_kb":5229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"66'42","reported_seq":316,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.413242+0000","last_change":"2024-03-04T06:09:34.418012+0000","last_active":"2024-03-04T06:09:34.413242+0000","last_peered":"2024-03-04T06:09:34.413242+0000","last_clean":"2024-03-04T06:09:34.413242+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:09:34.413242+0000","last_undegraded":"2024-03-04T06:09:34.413242+0000","last_fullsized":"2024-03-04T06:09:34.413242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218557,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":28495,"num_write":32,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":272,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.407424+0000","last_change":"2024-03-04T06:09:34.418133+0000","last_active":"2024-03-04T06:09:34.407424+0000","last_peered":"2024-03-04T06:09:34.407424+0000","last_clean":"2024-03-04T06:09:34.407424+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:09:34.407424+0000","last_undegraded":"2024-03-04T06:09:34.407424+0000","last_fullsized":"2024-03-04T06:09:34.407424+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"98'60","reported_seq":320,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.881156+0000","last_change":"2024-03-04T06:09:29.886098+0000","last_active":"2024-03-04T06:09:29.881156+0000","last_peered":"2024-03-04T06:09:29.881156+0000","last_clean":"2024-03-04T06:09:29.881156+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:09:29.881156+0000","last_undegraded":"2024-03-04T06:09:29.881156+0000","last_fullsized":"2024-03-04T06:09:29.881156+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764801,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":21919,"num_write":48,"num_write_kb":13847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":380,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.319040+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:09:31.319040+0000","last_peered":"2024-03-04T06:09:31.319040+0000","last_clean":"2024-03-04T06:09:31.319040+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:09:31.319040+0000","last_undegraded":"2024-03-04T06:09:31.319040+0000","last_fullsized":"2024-03-04T06:09:31.319040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":233,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.887300+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:09:29.887300+0000","last_peered":"2024-03-04T06:09:29.887300+0000","last_clean":"2024-03-04T06:09:29.887300+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:09:29.887300+0000","last_undegraded":"2024-03-04T06:09:29.887300+0000","last_fullsized":"2024-03-04T06:09:29.887300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"99'45","reported_seq":313,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.883589+0000","last_change":"2024-03-04T06:09:29.888313+0000","last_active":"2024-03-04T06:09:29.883589+0000","last_peered":"2024-03-04T06:09:29.883589+0000","last_clean":"2024-03-04T06:09:29.883589+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:09:29.883589+0000","last_undegraded":"2024-03-04T06:09:29.883589+0000","last_fullsized":"2024-03-04T06:09:29.883589+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10151395,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":42,"num_read_kb":36818,"num_write":36,"num_write_kb":9883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":240,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.412456+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:09:34.412456+0000","last_peered":"2024-03-04T06:09:34.412456+0000","last_clean":"2024-03-04T06:09:34.412456+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:09:34.412456+0000","last_undegraded":"2024-03-04T06:09:34.412456+0000","last_fullsized":"2024-03-04T06:09:34.412456+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"82'41","reported_seq":321,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.320772+0000","last_change":"2024-03-04T06:09:30.347017+0000","last_active":"2024-03-04T06:09:31.320772+0000","last_peered":"2024-03-04T06:09:31.320772+0000","last_clean":"2024-03-04T06:09:31.320772+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:09:31.320772+0000","last_undegraded":"2024-03-04T06:09:31.320772+0000","last_fullsized":"2024-03-04T06:09:31.320772+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210170,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31345,"num_write":29,"num_write_kb":8200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":233,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.878672+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:09:29.878672+0000","last_peered":"2024-03-04T06:09:29.878672+0000","last_clean":"2024-03-04T06:09:29.878672+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:09:29.878672+0000","last_undegraded":"2024-03-04T06:09:29.878672+0000","last_fullsized":"2024-03-04T06:09:29.878672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"102'155","reported_seq":436,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.331619+0000","last_change":"2024-03-04T06:09:30.388017+0000","last_active":"2024-03-04T06:09:31.331619+0000","last_peered":"2024-03-04T06:09:31.331619+0000","last_clean":"2024-03-04T06:09:31.331619+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:09:31.331619+0000","last_undegraded":"2024-03-04T06:09:31.331619+0000","last_fullsized":"2024-03-04T06:09:31.331619+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15075879,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":138,"num_read_kb":100582,"num_write":111,"num_write_kb":32075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":240,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.391490+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:09:34.391490+0000","last_peered":"2024-03-04T06:09:34.391490+0000","last_clean":"2024-03-04T06:09:34.391490+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:09:34.391490+0000","last_undegraded":"2024-03-04T06:09:34.391490+0000","last_fullsized":"2024-03-04T06:09:34.391490+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"106'152","reported_seq":419,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.477332+0000","last_change":"2024-03-04T06:09:33.434013+0000","last_active":"2024-03-04T06:09:34.477332+0000","last_peered":"2024-03-04T06:09:34.477332+0000","last_clean":"2024-03-04T06:09:34.477332+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:09:34.477332+0000","last_undegraded":"2024-03-04T06:09:34.477332+0000","last_fullsized":"2024-03-04T06:09:34.477332+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10386726,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":88,"num_read_kb":57457,"num_write":101,"num_write_kb":26783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"94'45","reported_seq":320,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.886605+0000","last_change":"2024-03-04T06:09:29.889466+0000","last_active":"2024-03-04T06:09:29.886605+0000","last_peered":"2024-03-04T06:09:29.886605+0000","last_clean":"2024-03-04T06:09:29.886605+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:09:29.886605+0000","last_undegraded":"2024-03-04T06:09:29.886605+0000","last_fullsized":"2024-03-04T06:09:29.886605+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8353046,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":36976,"num_write":37,"num_write_kb":9040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":262,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.348242+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:09:34.348242+0000","last_peered":"2024-03-04T06:09:34.348242+0000","last_clean":"2024-03-04T06:09:34.348242+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:09:34.348242+0000","last_undegraded":"2024-03-04T06:09:34.348242+0000","last_fullsized":"2024-03-04T06:09:34.348242+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"105'49","reported_seq":357,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.349683+0000","last_change":"2024-03-04T06:09:33.386362+0000","last_active":"2024-03-04T06:09:34.349683+0000","last_peered":"2024-03-04T06:09:34.349683+0000","last_clean":"2024-03-04T06:09:34.349683+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:09:34.349683+0000","last_undegraded":"2024-03-04T06:09:34.349683+0000","last_fullsized":"2024-03-04T06:09:34.349683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151362,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":59,"num_read_kb":33400,"num_write":42,"num_write_kb":10631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":247,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.325969+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:09:31.325969+0000","last_peered":"2024-03-04T06:09:31.325969+0000","last_clean":"2024-03-04T06:09:31.325969+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:09:31.325969+0000","last_undegraded":"2024-03-04T06:09:31.325969+0000","last_fullsized":"2024-03-04T06:09:31.325969+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"102'80","reported_seq":373,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T06:09:31.318185+0000","last_change":"2024-03-04T06:09:30.383642+0000","last_active":"2024-03-04T06:09:31.318185+0000","last_peered":"2024-03-04T06:09:31.318185+0000","last_clean":"2024-03-04T06:09:31.318185+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:09:31.318185+0000","last_undegraded":"2024-03-04T06:09:31.318185+0000","last_fullsized":"2024-03-04T06:09:31.318185+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13159848,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":86,"num_read_kb":57568,"num_write":66,"num_write_kb":17346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":262,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.336600+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:09:34.336600+0000","last_peered":"2024-03-04T06:09:34.336600+0000","last_clean":"2024-03-04T06:09:34.336600+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:09:34.336600+0000","last_undegraded":"2024-03-04T06:09:34.336600+0000","last_fullsized":"2024-03-04T06:09:34.336600+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"105'121","reported_seq":407,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.333875+0000","last_change":"2024-03-04T06:09:33.375588+0000","last_active":"2024-03-04T06:09:34.333875+0000","last_peered":"2024-03-04T06:09:34.333875+0000","last_clean":"2024-03-04T06:09:34.333875+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:09:34.333875+0000","last_undegraded":"2024-03-04T06:09:34.333875+0000","last_fullsized":"2024-03-04T06:09:34.333875+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12924583,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":70603,"num_write":74,"num_write_kb":20323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":233,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-03-04T06:09:29.886670+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:09:29.886670+0000","last_peered":"2024-03-04T06:09:29.886670+0000","last_clean":"2024-03-04T06:09:29.886670+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:09:29.886670+0000","last_undegraded":"2024-03-04T06:09:29.886670+0000","last_fullsized":"2024-03-04T06:09:29.886670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":111749611,"num_objects":151,"num_object_clones":94,"num_object_copies":453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":873,"num_read_kb":595589,"num_write":775,"num_write_kb":213542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1069,"ondisk_log_size":1069,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":117416,"kb_used_data":86432,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94254424,"statfs":{"total":96636764160,"available":96516530176,"internally_reserved":0,"allocated":88506368,"data_stored":94994064,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91448,"kb_used_data":64624,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94280392,"statfs":{"total":96636764160,"available":96543121408,"internally_reserved":0,"allocated":66174976,"data_stored":68639587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99688,"kb_used_data":68704,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94272152,"statfs":{"total":96636764160,"available":96534683648,"internally_reserved":0,"allocated":70352896,"data_stored":77033485,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86724,"kb_used_data":59900,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94285116,"statfs":{"total":96636764160,"available":96547958784,"internally_reserved":0,"allocated":61337600,"data_stored":66697177,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:09:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:421: done (3 left) 2024-03-04T06:09:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:422: done (2 left) 2024-03-04T06:09:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:423: done (1 left) 2024-03-04T06:09:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:425: done (0 left) 2024-03-04T06:09:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:426: snap_create 2024-03-04T06:09:36.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:36.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:36.933+0000 7ff42b754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:36.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:36.937+0000 7ff42b754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:37.326 INFO:tasks.rados.rados.0.smithi064.stdout:426: done (0 left) 2024-03-04T06:09:37.326 INFO:tasks.rados.rados.0.smithi064.stdout:427: write (excl) oid 14 current snap is 49 2024-03-04T06:09:37.327 INFO:tasks.rados.rados.0.smithi064.stdout:427: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-03-04T06:09:37.331 INFO:tasks.rados.rados.0.smithi064.stdout:427: writing smithi06431743-14 from 463926 to 926092 tid 1 2024-03-04T06:09:37.338 INFO:tasks.rados.rados.0.smithi064.stdout:427: writing smithi06431743-14 from 1723898 to 2160289 tid 2 2024-03-04T06:09:37.344 INFO:tasks.rados.rados.0.smithi064.stdout:427: writing smithi06431743-14 from 2603977 to 3329864 tid 3 2024-03-04T06:09:37.344 INFO:tasks.rados.rados.0.smithi064.stdout:428: write (excl) oid 17 current snap is 49 2024-03-04T06:09:37.345 INFO:tasks.rados.rados.0.smithi064.stdout:428: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-03-04T06:09:37.350 INFO:tasks.rados.rados.0.smithi064.stdout:428: writing smithi06431743-17 from 480733 to 1018148 tid 1 2024-03-04T06:09:37.358 INFO:tasks.rados.rados.0.smithi064.stdout:428: writing smithi06431743-17 from 1782380 to 2478782 tid 2 2024-03-04T06:09:37.365 INFO:tasks.rados.rados.0.smithi064.stdout:428: writing smithi06431743-17 from 3031400 to 3568559 tid 3 2024-03-04T06:09:37.365 INFO:tasks.rados.rados.0.smithi064.stdout:429: write oid 24 current snap is 49 2024-03-04T06:09:37.365 INFO:tasks.rados.rados.0.smithi064.stdout:429: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-03-04T06:09:37.372 INFO:tasks.rados.rados.0.smithi064.stdout:429: writing smithi06431743-24 from 497540 to 1110204 tid 1 2024-03-04T06:09:37.380 INFO:tasks.rados.rados.0.smithi064.stdout:429: writing smithi06431743-24 from 1840862 to 2280922 tid 2 2024-03-04T06:09:37.386 INFO:tasks.rados.rados.0.smithi064.stdout:429: writing smithi06431743-24 from 3058823 to 3523607 tid 3 2024-03-04T06:09:37.387 INFO:tasks.rados.rados.0.smithi064.stdout:430: snap_remove snap 41 2024-03-04T06:09:38.185 INFO:tasks.rados.rados.0.smithi064.stdout:430: done (3 left) 2024-03-04T06:09:38.185 INFO:tasks.rados.rados.0.smithi064.stdout:431: snap_create 2024-03-04T06:09:38.185 INFO:tasks.rados.rados.0.smithi064.stdout:427: finishing write tid 1 to smithi06431743-14 2024-03-04T06:09:38.185 INFO:tasks.rados.rados.0.smithi064.stdout:427: finishing write tid 2 to smithi06431743-14 2024-03-04T06:09:38.185 INFO:tasks.rados.rados.0.smithi064.stdout:428: finishing write tid 1 to smithi06431743-17 2024-03-04T06:09:38.185 INFO:tasks.rados.rados.0.smithi064.stdout:427: finishing write tid 3 to smithi06431743-14 2024-03-04T06:09:38.185 INFO:tasks.rados.rados.0.smithi064.stdout:428: finishing write tid 2 to smithi06431743-17 2024-03-04T06:09:38.185 INFO:tasks.rados.rados.0.smithi064.stdout:427: finishing write tid 4 to smithi06431743-14 2024-03-04T06:09:38.185 INFO:tasks.rados.rados.0.smithi064.stdout:427: finishing write tid 5 to smithi06431743-14 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:427: oid 14 updating version 0 to 96 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:427: oid 14 version 96 is already newer than 93 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:427: oid 14 version 96 is already newer than 95 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:427: oid 14 version 96 is already newer than 95 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 96 (ObjNum 218 snap 49 seq_num 218) dirty exists 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:427: left oid 14 (ObjNum 218 snap 49 seq_num 218) 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:428: finishing write tid 3 to smithi06431743-17 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:428: finishing write tid 4 to smithi06431743-17 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:428: finishing write tid 5 to smithi06431743-17 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:428: oid 17 updating version 0 to 69 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:428: oid 17 version 69 is already newer than 69 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:428: oid 17 version 69 is already newer than 66 2024-03-04T06:09:38.186 INFO:tasks.rados.rados.0.smithi064.stdout:428: oid 17 version 69 is already newer than 67 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 69 (ObjNum 219 snap 49 seq_num 219) dirty exists 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:428: left oid 17 (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:429: finishing write tid 1 to smithi06431743-24 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:429: finishing write tid 2 to smithi06431743-24 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:429: finishing write tid 3 to smithi06431743-24 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:429: finishing write tid 4 to smithi06431743-24 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:429: finishing write tid 5 to smithi06431743-24 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:429: oid 24 updating version 0 to 99 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:429: oid 24 version 99 is already newer than 98 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:429: oid 24 updating version 99 to 101 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:429: oid 24 version 101 is already newer than 101 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 101 (ObjNum 220 snap 49 seq_num 220) dirty exists 2024-03-04T06:09:38.187 INFO:tasks.rados.rados.0.smithi064.stdout:429: left oid 24 (ObjNum 220 snap 49 seq_num 220) 2024-03-04T06:09:39.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:39.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:39.118+0000 7f728ae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:39.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:39.122+0000 7f728ae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:39.195 INFO:tasks.rados.rados.0.smithi064.stdout:427: done (3 left) 2024-03-04T06:09:39.195 INFO:tasks.rados.rados.0.smithi064.stdout:428: done (2 left) 2024-03-04T06:09:39.195 INFO:tasks.rados.rados.0.smithi064.stdout:429: done (1 left) 2024-03-04T06:09:39.195 INFO:tasks.rados.rados.0.smithi064.stdout:431: done (0 left) 2024-03-04T06:09:39.195 INFO:tasks.rados.rados.0.smithi064.stdout:432: delete oid 12 current snap is 50 2024-03-04T06:09:39.236 INFO:tasks.rados.rados.0.smithi064.stdout:432: done (0 left) 2024-03-04T06:09:39.236 INFO:tasks.rados.rados.0.smithi064.stdout:433: write (excl) oid 43 current snap is 50 2024-03-04T06:09:39.236 INFO:tasks.rados.rados.0.smithi064.stdout:433: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-03-04T06:09:39.242 INFO:tasks.rados.rados.0.smithi064.stdout:433: writing smithi06431743-43 from 531154 to 1010669 tid 1 2024-03-04T06:09:39.247 INFO:tasks.rados.rados.0.smithi064.stdout:433: writing smithi06431743-43 from 1557826 to 2001555 tid 2 2024-03-04T06:09:39.254 INFO:tasks.rados.rados.0.smithi064.stdout:433: writing smithi06431743-43 from 2713669 to 3200997 tid 3 2024-03-04T06:09:39.257 INFO:tasks.rados.rados.0.smithi064.stdout:433: writing smithi06431743-43 from 3679295 to 3731154 tid 4 2024-03-04T06:09:39.258 INFO:tasks.rados.rados.0.smithi064.stdout:434: snap_create 2024-03-04T06:09:39.268 INFO:tasks.rados.rados.0.smithi064.stdout:433: finishing write tid 1 to smithi06431743-43 2024-03-04T06:09:39.276 INFO:tasks.rados.rados.0.smithi064.stdout:433: finishing write tid 2 to smithi06431743-43 2024-03-04T06:09:39.287 INFO:tasks.rados.rados.0.smithi064.stdout:433: finishing write tid 3 to smithi06431743-43 2024-03-04T06:09:39.293 INFO:tasks.rados.rados.0.smithi064.stdout:433: finishing write tid 4 to smithi06431743-43 2024-03-04T06:09:39.298 INFO:tasks.rados.rados.0.smithi064.stdout:433: finishing write tid 5 to smithi06431743-43 2024-03-04T06:09:39.299 INFO:tasks.rados.rados.0.smithi064.stdout:433: finishing write tid 6 to smithi06431743-43 2024-03-04T06:09:39.299 INFO:tasks.rados.rados.0.smithi064.stdout:433: oid 43 updating version 0 to 64 2024-03-04T06:09:39.299 INFO:tasks.rados.rados.0.smithi064.stdout:433: oid 43 version 64 is already newer than 61 2024-03-04T06:09:39.299 INFO:tasks.rados.rados.0.smithi064.stdout:433: oid 43 version 64 is already newer than 63 2024-03-04T06:09:39.299 INFO:tasks.rados.rados.0.smithi064.stdout:433: oid 43 updating version 64 to 65 2024-03-04T06:09:39.299 INFO:tasks.rados.rados.0.smithi064.stdout:433: oid 43 version 65 is already newer than 62 2024-03-04T06:09:39.299 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 65 (ObjNum 222 snap 50 seq_num 222) dirty exists 2024-03-04T06:09:39.299 INFO:tasks.rados.rados.0.smithi064.stdout:433: left oid 43 (ObjNum 222 snap 50 seq_num 222) 2024-03-04T06:09:40.195 INFO:tasks.rados.rados.0.smithi064.stdout:433: done (1 left) 2024-03-04T06:09:40.195 INFO:tasks.rados.rados.0.smithi064.stdout:434: done (0 left) 2024-03-04T06:09:40.195 INFO:tasks.rados.rados.0.smithi064.stdout:435: snap_create 2024-03-04T06:09:40.393 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:09:40.393 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:09:40.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:09:40.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:40.511+0000 7f929a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:40.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:40.517+0000 7f929a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:40.708 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:09:40.709 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:09:40.720 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-03-04T06:09:40.193729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117180395,"num_objects":156,"num_object_clones":96,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":980,"num_read_kb":595683,"num_write":927,"num_write_kb":220234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1200,"ondisk_log_size":1200,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":410488,"kb_used_data":294872,"kb_used_omap":0,"kb_used_meta":115584,"kb_avail":377076872,"statfs":{"total":386547056640,"available":386126716928,"internally_reserved":0,"allocated":301948928,"data_stored":326810303,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":118358016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4840416,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":15,"num_write_kb":4798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.869398"},"pg_stats":[{"pgid":"3.f","version":"106'58","reported_seq":316,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.408220+0000","last_change":"2024-03-04T06:09:34.454469+0000","last_active":"2024-03-04T06:09:34.408220+0000","last_peered":"2024-03-04T06:09:34.408220+0000","last_clean":"2024-03-04T06:09:34.408220+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:09:34.408220+0000","last_undegraded":"2024-03-04T06:09:34.408220+0000","last_fullsized":"2024-03-04T06:09:34.408220+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":22,"num_read_kb":6453,"num_write":40,"num_write_kb":11069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"93'51","reported_seq":339,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.335510+0000","last_change":"2024-03-04T06:09:33.338052+0000","last_active":"2024-03-04T06:09:34.335510+0000","last_peered":"2024-03-04T06:09:34.335510+0000","last_clean":"2024-03-04T06:09:34.335510+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:09:34.335510+0000","last_undegraded":"2024-03-04T06:09:34.335510+0000","last_fullsized":"2024-03-04T06:09:34.335510+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092488,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":33,"num_read_kb":21067,"num_write":36,"num_write_kb":9740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"105'86","reported_seq":351,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.389995+0000","last_change":"2024-03-04T06:09:33.399451+0000","last_active":"2024-03-04T06:09:34.389995+0000","last_peered":"2024-03-04T06:09:34.389995+0000","last_clean":"2024-03-04T06:09:34.389995+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:09:34.389995+0000","last_undegraded":"2024-03-04T06:09:34.389995+0000","last_fullsized":"2024-03-04T06:09:34.389995+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8420307,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":52,"num_read_kb":29740,"num_write":58,"num_write_kb":16142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"106'63","reported_seq":342,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.500262+0000","last_change":"2024-03-04T06:09:34.412923+0000","last_active":"2024-03-04T06:09:34.500262+0000","last_peered":"2024-03-04T06:09:34.500262+0000","last_clean":"2024-03-04T06:09:34.500262+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:09:34.500262+0000","last_undegraded":"2024-03-04T06:09:34.500262+0000","last_fullsized":"2024-03-04T06:09:34.500262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6874063,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":55,"num_read_kb":46129,"num_write":52,"num_write_kb":15175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"105'29","reported_seq":316,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.380143+0000","last_change":"2024-03-04T06:09:33.389489+0000","last_active":"2024-03-04T06:09:34.380143+0000","last_peered":"2024-03-04T06:09:34.380143+0000","last_clean":"2024-03-04T06:09:34.380143+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:09:34.380143+0000","last_undegraded":"2024-03-04T06:09:34.380143+0000","last_fullsized":"2024-03-04T06:09:34.380143+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3563084,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":17038,"num_write":18,"num_write_kb":5229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"66'42","reported_seq":316,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.413242+0000","last_change":"2024-03-04T06:09:34.418012+0000","last_active":"2024-03-04T06:09:34.413242+0000","last_peered":"2024-03-04T06:09:34.413242+0000","last_clean":"2024-03-04T06:09:34.413242+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:09:34.413242+0000","last_undegraded":"2024-03-04T06:09:34.413242+0000","last_fullsized":"2024-03-04T06:09:34.413242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218557,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":28495,"num_write":32,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":272,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.407424+0000","last_change":"2024-03-04T06:09:34.418133+0000","last_active":"2024-03-04T06:09:34.407424+0000","last_peered":"2024-03-04T06:09:34.407424+0000","last_clean":"2024-03-04T06:09:34.407424+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:09:34.407424+0000","last_undegraded":"2024-03-04T06:09:34.407424+0000","last_fullsized":"2024-03-04T06:09:34.407424+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"105'64","reported_seq":336,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.392802+0000","last_change":"2024-03-04T06:09:34.392802+0000","last_active":"2024-03-04T06:09:34.392802+0000","last_peered":"2024-03-04T06:09:34.392802+0000","last_clean":"2024-03-04T06:09:34.392802+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:09:34.392802+0000","last_undegraded":"2024-03-04T06:09:34.392802+0000","last_fullsized":"2024-03-04T06:09:34.392802+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764801,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":21919,"num_write":48,"num_write_kb":13847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":393,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T06:09:36.342558+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:09:36.342558+0000","last_peered":"2024-03-04T06:09:36.342558+0000","last_clean":"2024-03-04T06:09:36.342558+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:09:36.342558+0000","last_undegraded":"2024-03-04T06:09:36.342558+0000","last_fullsized":"2024-03-04T06:09:36.342558+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":246,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.397558+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:09:34.397558+0000","last_peered":"2024-03-04T06:09:34.397558+0000","last_clean":"2024-03-04T06:09:34.397558+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:09:34.397558+0000","last_undegraded":"2024-03-04T06:09:34.397558+0000","last_fullsized":"2024-03-04T06:09:34.397558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"99'45","reported_seq":329,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.394992+0000","last_change":"2024-03-04T06:09:33.402889+0000","last_active":"2024-03-04T06:09:34.394992+0000","last_peered":"2024-03-04T06:09:34.394992+0000","last_clean":"2024-03-04T06:09:34.394992+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:09:34.394992+0000","last_undegraded":"2024-03-04T06:09:34.394992+0000","last_fullsized":"2024-03-04T06:09:34.394992+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10151395,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":42,"num_read_kb":36818,"num_write":36,"num_write_kb":9883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":240,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.412456+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:09:34.412456+0000","last_peered":"2024-03-04T06:09:34.412456+0000","last_clean":"2024-03-04T06:09:34.412456+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:09:34.412456+0000","last_undegraded":"2024-03-04T06:09:34.412456+0000","last_fullsized":"2024-03-04T06:09:34.412456+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"106'46","reported_seq":338,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.493700+0000","last_change":"2024-03-04T06:09:33.357285+0000","last_active":"2024-03-04T06:09:34.493700+0000","last_peered":"2024-03-04T06:09:34.493700+0000","last_clean":"2024-03-04T06:09:34.493700+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:09:34.493700+0000","last_undegraded":"2024-03-04T06:09:34.493700+0000","last_fullsized":"2024-03-04T06:09:34.493700+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6420274,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":31346,"num_write":34,"num_write_kb":10119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":246,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.387380+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:09:34.387380+0000","last_peered":"2024-03-04T06:09:34.387380+0000","last_clean":"2024-03-04T06:09:34.387380+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:09:34.387380+0000","last_undegraded":"2024-03-04T06:09:34.387380+0000","last_fullsized":"2024-03-04T06:09:34.387380+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"105'161","reported_seq":448,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.379937+0000","last_change":"2024-03-04T06:09:33.403563+0000","last_active":"2024-03-04T06:09:34.379937+0000","last_peered":"2024-03-04T06:09:34.379937+0000","last_clean":"2024-03-04T06:09:34.379937+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:09:34.379937+0000","last_undegraded":"2024-03-04T06:09:34.379937+0000","last_fullsized":"2024-03-04T06:09:34.379937+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15075879,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":138,"num_read_kb":100582,"num_write":111,"num_write_kb":32075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":240,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.391490+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:09:34.391490+0000","last_peered":"2024-03-04T06:09:34.391490+0000","last_clean":"2024-03-04T06:09:34.391490+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:09:34.391490+0000","last_undegraded":"2024-03-04T06:09:34.391490+0000","last_fullsized":"2024-03-04T06:09:34.391490+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"106'152","reported_seq":419,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.477332+0000","last_change":"2024-03-04T06:09:33.434013+0000","last_active":"2024-03-04T06:09:34.477332+0000","last_peered":"2024-03-04T06:09:34.477332+0000","last_clean":"2024-03-04T06:09:34.477332+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:09:34.477332+0000","last_undegraded":"2024-03-04T06:09:34.477332+0000","last_fullsized":"2024-03-04T06:09:34.477332+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10386726,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":88,"num_read_kb":57457,"num_write":101,"num_write_kb":26783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"102'47","reported_seq":336,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.399611+0000","last_change":"2024-03-04T06:09:33.396125+0000","last_active":"2024-03-04T06:09:34.399611+0000","last_peered":"2024-03-04T06:09:34.399611+0000","last_clean":"2024-03-04T06:09:34.399611+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:09:34.399611+0000","last_undegraded":"2024-03-04T06:09:34.399611+0000","last_fullsized":"2024-03-04T06:09:34.399611+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8353046,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":36976,"num_write":37,"num_write_kb":9040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":262,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.348242+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:09:34.348242+0000","last_peered":"2024-03-04T06:09:34.348242+0000","last_clean":"2024-03-04T06:09:34.348242+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:09:34.348242+0000","last_undegraded":"2024-03-04T06:09:34.348242+0000","last_fullsized":"2024-03-04T06:09:34.348242+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"105'49","reported_seq":357,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.349683+0000","last_change":"2024-03-04T06:09:33.386362+0000","last_active":"2024-03-04T06:09:34.349683+0000","last_peered":"2024-03-04T06:09:34.349683+0000","last_clean":"2024-03-04T06:09:34.349683+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:09:34.349683+0000","last_undegraded":"2024-03-04T06:09:34.349683+0000","last_fullsized":"2024-03-04T06:09:34.349683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151362,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":59,"num_read_kb":33400,"num_write":42,"num_write_kb":10631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":257,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.377175+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:09:34.377175+0000","last_peered":"2024-03-04T06:09:34.377175+0000","last_clean":"2024-03-04T06:09:34.377175+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:09:34.377175+0000","last_undegraded":"2024-03-04T06:09:34.377175+0000","last_fullsized":"2024-03-04T06:09:34.377175+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"105'89","reported_seq":393,"reported_epoch":108,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:09:36.346487+0000","last_change":"2024-03-04T06:09:36.346487+0000","last_active":"2024-03-04T06:09:36.346487+0000","last_peered":"2024-03-04T06:09:36.346487+0000","last_clean":"2024-03-04T06:09:36.346487+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:09:36.346487+0000","last_undegraded":"2024-03-04T06:09:36.346487+0000","last_fullsized":"2024-03-04T06:09:36.346487+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13193462,"num_objects":12,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":88,"num_read_kb":57569,"num_write":71,"num_write_kb":18684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":262,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.336600+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:09:34.336600+0000","last_peered":"2024-03-04T06:09:34.336600+0000","last_clean":"2024-03-04T06:09:34.336600+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:09:34.336600+0000","last_undegraded":"2024-03-04T06:09:34.336600+0000","last_fullsized":"2024-03-04T06:09:34.336600+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"105'121","reported_seq":407,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.333875+0000","last_change":"2024-03-04T06:09:33.375588+0000","last_active":"2024-03-04T06:09:34.333875+0000","last_peered":"2024-03-04T06:09:34.333875+0000","last_clean":"2024-03-04T06:09:34.333875+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:09:34.333875+0000","last_undegraded":"2024-03-04T06:09:34.333875+0000","last_fullsized":"2024-03-04T06:09:34.333875+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12924583,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":70603,"num_write":74,"num_write_kb":20323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":246,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-04T06:09:34.400363+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:09:34.400363+0000","last_peered":"2024-03-04T06:09:34.400363+0000","last_clean":"2024-03-04T06:09:34.400363+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:09:34.400363+0000","last_undegraded":"2024-03-04T06:09:34.400363+0000","last_fullsized":"2024-03-04T06:09:34.400363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":116590027,"num_objects":154,"num_object_clones":96,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":877,"num_read_kb":595592,"num_write":790,"num_write_kb":218340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1103,"ondisk_log_size":1103,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":127764,"kb_used_data":96780,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94244076,"statfs":{"total":96636764160,"available":96505933824,"internally_reserved":0,"allocated":99102720,"data_stored":108473903,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96312,"kb_used_data":69488,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94275528,"statfs":{"total":96636764160,"available":96538140672,"internally_reserved":0,"allocated":71155712,"data_stored":74605738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99688,"kb_used_data":68704,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94272152,"statfs":{"total":96636764160,"available":96534683648,"internally_reserved":0,"allocated":70352896,"data_stored":77033485,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86724,"kb_used_data":59900,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94285116,"statfs":{"total":96636764160,"available":96547958784,"internally_reserved":0,"allocated":61337600,"data_stored":66697177,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:09:41.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:41.197 INFO:tasks.rados.rados.0.smithi064.stdout:435: done (0 left) 2024-03-04T06:09:41.198 INFO:tasks.rados.rados.0.smithi064.stdout:436: write oid 45 current snap is 52 2024-03-04T06:09:41.198 INFO:tasks.rados.rados.0.smithi064.stdout:436: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-03-04T06:09:41.204 INFO:tasks.rados.rados.0.smithi064.stdout:436: writing smithi06431743-45 from 547961 to 1102725 tid 1 2024-03-04T06:09:41.210 INFO:tasks.rados.rados.0.smithi064.stdout:436: writing smithi06431743-45 from 1616308 to 2320048 tid 2 2024-03-04T06:09:41.217 INFO:tasks.rados.rados.0.smithi064.stdout:436: writing smithi06431743-45 from 2741092 to 3439692 tid 3 2024-03-04T06:09:41.217 INFO:tasks.rados.rados.0.smithi064.stdout:437: snap_create 2024-03-04T06:09:41.274 INFO:tasks.rados.rados.0.smithi064.stdout:436: finishing write tid 1 to smithi06431743-45 2024-03-04T06:09:41.283 INFO:tasks.rados.rados.0.smithi064.stdout:436: finishing write tid 2 to smithi06431743-45 2024-03-04T06:09:41.301 INFO:tasks.rados.rados.0.smithi064.stdout:436: finishing write tid 3 to smithi06431743-45 2024-03-04T06:09:41.305 INFO:tasks.rados.rados.0.smithi064.stdout:436: finishing write tid 4 to smithi06431743-45 2024-03-04T06:09:41.306 INFO:tasks.rados.rados.0.smithi064.stdout:436: finishing write tid 5 to smithi06431743-45 2024-03-04T06:09:41.306 INFO:tasks.rados.rados.0.smithi064.stdout:436: oid 45 updating version 0 to 91 2024-03-04T06:09:41.307 INFO:tasks.rados.rados.0.smithi064.stdout:436: oid 45 version 91 is already newer than 89 2024-03-04T06:09:41.307 INFO:tasks.rados.rados.0.smithi064.stdout:436: oid 45 version 91 is already newer than 89 2024-03-04T06:09:41.307 INFO:tasks.rados.rados.0.smithi064.stdout:436: oid 45 version 91 is already newer than 90 2024-03-04T06:09:41.307 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 91 (ObjNum 223 snap 52 seq_num 223) dirty exists 2024-03-04T06:09:41.307 INFO:tasks.rados.rados.0.smithi064.stdout:436: left oid 45 (ObjNum 223 snap 52 seq_num 223) 2024-03-04T06:09:41.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:41.312+0000 7f9d7ca2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:41.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:41.317+0000 7f9d7ca2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:42.206 INFO:tasks.rados.rados.0.smithi064.stdout:436: done (1 left) 2024-03-04T06:09:42.206 INFO:tasks.rados.rados.0.smithi064.stdout:437: done (0 left) 2024-03-04T06:09:42.206 INFO:tasks.rados.rados.0.smithi064.stdout:438: snap_create 2024-03-04T06:09:43.216 INFO:tasks.rados.rados.0.smithi064.stdout:438: done (0 left) 2024-03-04T06:09:43.216 INFO:tasks.rados.rados.0.smithi064.stdout:439: delete oid 13 current snap is 54 2024-03-04T06:09:43.268 INFO:tasks.rados.rados.0.smithi064.stdout:439: done (0 left) 2024-03-04T06:09:43.269 INFO:tasks.rados.rados.0.smithi064.stdout:440: write (excl) oid 36 current snap is 54 2024-03-04T06:09:43.269 INFO:tasks.rados.rados.0.smithi064.stdout:440: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-03-04T06:09:43.275 INFO:tasks.rados.rados.0.smithi064.stdout:440: writing smithi06431743-36 from 581575 to 1286837 tid 1 2024-03-04T06:09:43.282 INFO:tasks.rados.rados.0.smithi064.stdout:440: writing smithi06431743-36 from 1733272 to 2440681 tid 2 2024-03-04T06:09:43.290 INFO:tasks.rados.rados.0.smithi064.stdout:440: writing smithi06431743-36 from 3195938 to 3633435 tid 3 2024-03-04T06:09:43.290 INFO:tasks.rados.rados.0.smithi064.stdout:441: snap_create 2024-03-04T06:09:43.303 INFO:tasks.rados.rados.0.smithi064.stdout:440: finishing write tid 1 to smithi06431743-36 2024-03-04T06:09:43.314 INFO:tasks.rados.rados.0.smithi064.stdout:440: finishing write tid 2 to smithi06431743-36 2024-03-04T06:09:43.333 INFO:tasks.rados.rados.0.smithi064.stdout:440: finishing write tid 3 to smithi06431743-36 2024-03-04T06:09:43.338 INFO:tasks.rados.rados.0.smithi064.stdout:440: finishing write tid 4 to smithi06431743-36 2024-03-04T06:09:43.338 INFO:tasks.rados.rados.0.smithi064.stdout:440: finishing write tid 5 to smithi06431743-36 2024-03-04T06:09:43.338 INFO:tasks.rados.rados.0.smithi064.stdout:440: oid 36 updating version 0 to 55 2024-03-04T06:09:43.338 INFO:tasks.rados.rados.0.smithi064.stdout:440: oid 36 version 55 is already newer than 53 2024-03-04T06:09:43.338 INFO:tasks.rados.rados.0.smithi064.stdout:440: oid 36 version 55 is already newer than 54 2024-03-04T06:09:43.338 INFO:tasks.rados.rados.0.smithi064.stdout:440: oid 36 updating version 55 to 56 2024-03-04T06:09:43.338 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 56 (ObjNum 225 snap 54 seq_num 225) dirty exists 2024-03-04T06:09:43.338 INFO:tasks.rados.rados.0.smithi064.stdout:440: left oid 36 (ObjNum 225 snap 54 seq_num 225) 2024-03-04T06:09:43.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:43.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:43.495+0000 7ff03682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:43.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:43.499+0000 7ff03682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:44.215 INFO:tasks.rados.rados.0.smithi064.stdout:440: done (1 left) 2024-03-04T06:09:44.215 INFO:tasks.rados.rados.0.smithi064.stdout:441: done (0 left) 2024-03-04T06:09:44.215 INFO:tasks.rados.rados.0.smithi064.stdout:442: read oid 48 snap -1 2024-03-04T06:09:44.215 INFO:tasks.rados.rados.0.smithi064.stdout:442: expect (ObjNum 216 snap 47 seq_num 216) 2024-03-04T06:09:44.216 INFO:tasks.rados.rados.0.smithi064.stdout:443: read oid 43 snap 42 2024-03-04T06:09:44.216 INFO:tasks.rados.rados.0.smithi064.stdout:443: expect deleted 2024-03-04T06:09:44.216 INFO:tasks.rados.rados.0.smithi064.stdout:444: write oid 50 current snap is 55 2024-03-04T06:09:44.216 INFO:tasks.rados.rados.0.smithi064.stdout:444: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-03-04T06:09:44.222 INFO:tasks.rados.rados.0.smithi064.stdout:444: writing smithi06431743-50 from 598382 to 1378893 tid 1 2024-03-04T06:09:44.229 INFO:tasks.rados.rados.0.smithi064.stdout:444: writing smithi06431743-50 from 2075401 to 2526468 tid 2 2024-03-04T06:09:44.238 INFO:tasks.rados.rados.0.smithi064.stdout:444: writing smithi06431743-50 from 3107008 to 3755777 tid 3 2024-03-04T06:09:44.238 INFO:tasks.rados.rados.0.smithi064.stdout:445: read oid 1 snap -1 2024-03-04T06:09:44.238 INFO:tasks.rados.rados.0.smithi064.stdout:445: expect (ObjNum 132 snap 15 seq_num 132) 2024-03-04T06:09:44.239 INFO:tasks.rados.rados.0.smithi064.stdout:446: write oid 29 current snap is 55 2024-03-04T06:09:44.239 INFO:tasks.rados.rados.0.smithi064.stdout:446: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-03-04T06:09:44.245 INFO:tasks.rados.rados.0.smithi064.stdout:446: writing smithi06431743-29 from 615189 to 1070949 tid 1 2024-03-04T06:09:44.255 INFO:tasks.rados.rados.0.smithi064.stdout:446: writing smithi06431743-29 from 1733883 to 2444961 tid 2 2024-03-04T06:09:44.262 INFO:tasks.rados.rados.0.smithi064.stdout:446: writing smithi06431743-29 from 3134431 to 3594472 tid 3 2024-03-04T06:09:44.263 INFO:tasks.rados.rados.0.smithi064.stdout:447: snap_remove snap 49 2024-03-04T06:09:45.224 INFO:tasks.rados.rados.0.smithi064.stdout:447: done (5 left) 2024-03-04T06:09:45.224 INFO:tasks.rados.rados.0.smithi064.stdout:448: delete oid 12 current snap is 55 2024-03-04T06:09:45.224 INFO:tasks.rados.rados.0.smithi064.stdout:446: finishing write tid 1 to smithi06431743-29 2024-03-04T06:09:45.224 INFO:tasks.rados.rados.0.smithi064.stdout:445: expect (ObjNum 132 snap 15 seq_num 132) 2024-03-04T06:09:45.343 INFO:tasks.rados.rados.0.smithi064.stdout:444: finishing write tid 1 to smithi06431743-50 2024-03-04T06:09:45.343 INFO:tasks.rados.rados.0.smithi064.stdout:443: done (5 left) 2024-03-04T06:09:45.344 INFO:tasks.rados.rados.0.smithi064.stdout:445: done (4 left) 2024-03-04T06:09:45.344 INFO:tasks.rados.rados.0.smithi064.stdout:448: done (3 left) 2024-03-04T06:09:45.344 INFO:tasks.rados.rados.0.smithi064.stdout:449: read oid 15 snap 47 2024-03-04T06:09:45.344 INFO:tasks.rados.rados.0.smithi064.stdout:449: expect (ObjNum 196 snap 40 seq_num 196) 2024-03-04T06:09:45.344 INFO:tasks.rados.rados.0.smithi064.stdout:450: read oid 10 snap -1 2024-03-04T06:09:45.344 INFO:tasks.rados.rados.0.smithi064.stdout:450: expect (ObjNum 146 snap 20 seq_num 146) 2024-03-04T06:09:45.344 INFO:tasks.rados.rados.0.smithi064.stdout:451: write (excl) oid 7 current snap is 55 2024-03-04T06:09:45.344 INFO:tasks.rados.rados.0.smithi064.stdout:446: finishing write tid 2 to smithi06431743-29 2024-03-04T06:09:45.344 INFO:tasks.rados.rados.0.smithi064.stdout:442: expect (ObjNum 216 snap 47 seq_num 216) 2024-03-04T06:09:45.534 INFO:tasks.rados.rados.0.smithi064.stdout:446: finishing write tid 3 to smithi06431743-29 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:444: finishing write tid 2 to smithi06431743-50 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:446: finishing write tid 4 to smithi06431743-29 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:446: finishing write tid 5 to smithi06431743-29 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:446: oid 29 updating version 0 to 167 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:446: oid 29 version 167 is already newer than 165 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:446: oid 29 updating version 167 to 168 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:446: oid 29 version 168 is already newer than 167 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 168 (ObjNum 227 snap 55 seq_num 227) dirty exists 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:446: left oid 29 (ObjNum 227 snap 55 seq_num 227) 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:444: finishing write tid 3 to smithi06431743-50 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:444: finishing write tid 4 to smithi06431743-50 2024-03-04T06:09:45.535 INFO:tasks.rados.rados.0.smithi064.stdout:444: finishing write tid 5 to smithi06431743-50 2024-03-04T06:09:45.536 INFO:tasks.rados.rados.0.smithi064.stdout:444: oid 50 updating version 0 to 127 2024-03-04T06:09:45.536 INFO:tasks.rados.rados.0.smithi064.stdout:444: oid 50 version 127 is already newer than 125 2024-03-04T06:09:45.536 INFO:tasks.rados.rados.0.smithi064.stdout:444: oid 50 version 127 is already newer than 127 2024-03-04T06:09:45.536 INFO:tasks.rados.rados.0.smithi064.stdout:444: oid 50 updating version 127 to 128 2024-03-04T06:09:45.536 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 128 (ObjNum 226 snap 55 seq_num 226) dirty exists 2024-03-04T06:09:45.536 INFO:tasks.rados.rados.0.smithi064.stdout:444: left oid 50 (ObjNum 226 snap 55 seq_num 226) 2024-03-04T06:09:45.536 INFO:tasks.rados.rados.0.smithi064.stdout:451: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-03-04T06:09:45.542 INFO:tasks.rados.rados.0.smithi064.stdout:451: writing smithi06431743-7 from 648803 to 1371414 tid 1 2024-03-04T06:09:45.549 INFO:tasks.rados.rados.0.smithi064.stdout:451: writing smithi06431743-7 from 1967200 to 2681947 tid 2 2024-03-04T06:09:45.556 INFO:tasks.rados.rados.0.smithi064.stdout:451: writing smithi06431743-7 from 3421983 to 3848803 tid 3 2024-03-04T06:09:45.556 INFO:tasks.rados.rados.0.smithi064.stdout:442: done (5 left) 2024-03-04T06:09:45.556 INFO:tasks.rados.rados.0.smithi064.stdout:444: done (4 left) 2024-03-04T06:09:45.556 INFO:tasks.rados.rados.0.smithi064.stdout:446: done (3 left) 2024-03-04T06:09:45.556 INFO:tasks.rados.rados.0.smithi064.stdout:452: read oid 43 snap 47 2024-03-04T06:09:45.556 INFO:tasks.rados.rados.0.smithi064.stdout:452: expect deleted 2024-03-04T06:09:45.556 INFO:tasks.rados.rados.0.smithi064.stdout:453: write (excl) oid 38 current snap is 55 2024-03-04T06:09:45.556 INFO:tasks.rados.rados.0.smithi064.stdout:453: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-03-04T06:09:45.565 INFO:tasks.rados.rados.0.smithi064.stdout:453: writing smithi06431743-38 from 665610 to 1463470 tid 1 2024-03-04T06:09:45.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:45.571 INFO:tasks.rados.rados.0.smithi064.stdout:453: writing smithi06431743-38 from 1909329 to 2367734 tid 2 2024-03-04T06:09:45.576 INFO:tasks.rados.rados.0.smithi064.stdout:453: writing smithi06431743-38 from 2816700 to 3226910 tid 3 2024-03-04T06:09:45.576 INFO:tasks.rados.rados.0.smithi064.stdout:454: write oid 14 current snap is 55 2024-03-04T06:09:45.576 INFO:tasks.rados.rados.0.smithi064.stdout:454: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-03-04T06:09:45.583 INFO:tasks.rados.rados.0.smithi064.stdout:454: writing smithi06431743-14 from 682417 to 1155526 tid 1 2024-03-04T06:09:45.589 INFO:tasks.rados.rados.0.smithi064.stdout:454: writing smithi06431743-14 from 1567811 to 2169874 tid 2 2024-03-04T06:09:45.598 INFO:tasks.rados.rados.0.smithi064.stdout:454: writing smithi06431743-14 from 2844123 to 3465605 tid 3 2024-03-04T06:09:45.598 INFO:tasks.rados.rados.0.smithi064.stdout:455: snap_create 2024-03-04T06:09:45.598 INFO:tasks.rados.rados.0.smithi064.stdout:450: expect (ObjNum 146 snap 20 seq_num 146) 2024-03-04T06:09:45.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:45.695+0000 7fcf0399d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:45.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:45.704+0000 7fcf0399d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:45.722 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:09:45.722 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:09:45.723 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:09:45.751 INFO:tasks.rados.rados.0.smithi064.stdout:449: expect (ObjNum 196 snap 40 seq_num 196) 2024-03-04T06:09:45.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:45.831+0000 7fd292d14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:45.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:45.838+0000 7fd292d14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:45.875 INFO:tasks.rados.rados.0.smithi064.stdout:453: finishing write tid 1 to smithi06431743-38 2024-03-04T06:09:45.875 INFO:tasks.rados.rados.0.smithi064.stdout:451: finishing write tid 1 to smithi06431743-7 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:453: finishing write tid 2 to smithi06431743-38 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:454: finishing write tid 1 to smithi06431743-14 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:453: finishing write tid 3 to smithi06431743-38 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:453: finishing write tid 4 to smithi06431743-38 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:453: finishing write tid 5 to smithi06431743-38 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:453: oid 38 updating version 0 to 46 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:453: oid 38 version 46 is already newer than 46 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:453: oid 38 version 46 is already newer than 44 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:453: oid 38 version 46 is already newer than 43 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 46 (ObjNum 230 snap 55 seq_num 230) dirty exists 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:453: left oid 38 (ObjNum 230 snap 55 seq_num 230) 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:451: finishing write tid 2 to smithi06431743-7 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:454: finishing write tid 2 to smithi06431743-14 2024-03-04T06:09:45.876 INFO:tasks.rados.rados.0.smithi064.stdout:451: finishing write tid 3 to smithi06431743-7 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:454: finishing write tid 3 to smithi06431743-14 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:451: finishing write tid 4 to smithi06431743-7 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:451: finishing write tid 5 to smithi06431743-7 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:451: oid 7 updating version 0 to 52 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:451: oid 7 updating version 52 to 54 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:451: oid 7 updating version 54 to 55 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:451: oid 7 version 55 is already newer than 54 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 55 (ObjNum 229 snap 55 seq_num 229) dirty exists 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:451: left oid 7 (ObjNum 229 snap 55 seq_num 229) 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:454: finishing write tid 4 to smithi06431743-14 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:454: finishing write tid 5 to smithi06431743-14 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:454: oid 14 updating version 0 to 110 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:454: oid 14 version 110 is already newer than 107 2024-03-04T06:09:45.877 INFO:tasks.rados.rados.0.smithi064.stdout:454: oid 14 version 110 is already newer than 110 2024-03-04T06:09:45.878 INFO:tasks.rados.rados.0.smithi064.stdout:454: oid 14 version 110 is already newer than 108 2024-03-04T06:09:45.878 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 110 (ObjNum 231 snap 55 seq_num 231) dirty exists 2024-03-04T06:09:45.878 INFO:tasks.rados.rados.0.smithi064.stdout:454: left oid 14 (ObjNum 231 snap 55 seq_num 231) 2024-03-04T06:09:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:09:46.045 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:09:46.057 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-03-04T06:09:45.219386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119365305,"num_objects":162,"num_object_clones":101,"num_object_copies":486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":989,"num_read_kb":595688,"num_write":954,"num_write_kb":228379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1261,"ondisk_log_size":1261,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":443680,"kb_used_data":319744,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":377043680,"statfs":{"total":386547056640,"available":386092728320,"internally_reserved":0,"allocated":327417856,"data_stored":360375431,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2100875,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":20,"num_write_kb":6715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.023051"},"pg_stats":[{"pgid":"3.f","version":"111'65","reported_seq":335,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-04T06:09:39.293111+0000","last_change":"2024-03-04T06:09:38.347165+0000","last_active":"2024-03-04T06:09:39.293111+0000","last_peered":"2024-03-04T06:09:39.293111+0000","last_clean":"2024-03-04T06:09:39.293111+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:09:39.293111+0000","last_undegraded":"2024-03-04T06:09:39.293111+0000","last_fullsized":"2024-03-04T06:09:39.293111+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731154,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":24,"num_read_kb":6454,"num_write":46,"num_write_kb":12499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"93'51","reported_seq":357,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-04T06:09:40.208696+0000","last_change":"2024-03-04T06:09:38.203815+0000","last_active":"2024-03-04T06:09:40.208696+0000","last_peered":"2024-03-04T06:09:40.208696+0000","last_clean":"2024-03-04T06:09:40.208696+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:09:40.208696+0000","last_undegraded":"2024-03-04T06:09:40.208696+0000","last_fullsized":"2024-03-04T06:09:40.208696+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092488,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":33,"num_read_kb":21067,"num_write":36,"num_write_kb":9740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"113'91","reported_seq":371,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.300214+0000","last_change":"2024-03-04T06:09:38.319236+0000","last_active":"2024-03-04T06:09:41.300214+0000","last_peered":"2024-03-04T06:09:41.300214+0000","last_clean":"2024-03-04T06:09:41.300214+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:09:41.300214+0000","last_undegraded":"2024-03-04T06:09:41.300214+0000","last_fullsized":"2024-03-04T06:09:41.300214+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9630411,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":53,"num_read_kb":29741,"num_write":63,"num_write_kb":18055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"110'67","reported_seq":356,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-04T06:09:39.205821+0000","last_change":"2024-03-04T06:09:38.344211+0000","last_active":"2024-03-04T06:09:39.205821+0000","last_peered":"2024-03-04T06:09:39.205821+0000","last_clean":"2024-03-04T06:09:39.205821+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:09:39.205821+0000","last_undegraded":"2024-03-04T06:09:39.205821+0000","last_fullsized":"2024-03-04T06:09:39.205821+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226944,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":56,"num_read_kb":46129,"num_write":53,"num_write_kb":15175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"110'31","reported_seq":334,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.218741+0000","last_change":"2024-03-04T06:09:38.350261+0000","last_active":"2024-03-04T06:09:41.218741+0000","last_peered":"2024-03-04T06:09:41.218741+0000","last_clean":"2024-03-04T06:09:41.218741+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:09:41.218741+0000","last_undegraded":"2024-03-04T06:09:41.218741+0000","last_fullsized":"2024-03-04T06:09:41.218741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3563084,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":17038,"num_write":18,"num_write_kb":5229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"66'42","reported_seq":329,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-04T06:09:39.216114+0000","last_change":"2024-03-04T06:09:38.325027+0000","last_active":"2024-03-04T06:09:39.216114+0000","last_peered":"2024-03-04T06:09:39.216114+0000","last_clean":"2024-03-04T06:09:39.216114+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:09:39.216114+0000","last_undegraded":"2024-03-04T06:09:39.216114+0000","last_fullsized":"2024-03-04T06:09:39.216114+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218557,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":37,"num_read_kb":28495,"num_write":32,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":285,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-04T06:09:39.211076+0000","last_change":"2024-03-04T06:09:38.327731+0000","last_active":"2024-03-04T06:09:39.211076+0000","last_peered":"2024-03-04T06:09:39.211076+0000","last_clean":"2024-03-04T06:09:39.211076+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:09:39.211076+0000","last_undegraded":"2024-03-04T06:09:39.211076+0000","last_fullsized":"2024-03-04T06:09:39.211076+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"110'71","reported_seq":356,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.229164+0000","last_change":"2024-03-04T06:09:38.343988+0000","last_active":"2024-03-04T06:09:41.229164+0000","last_peered":"2024-03-04T06:09:41.229164+0000","last_clean":"2024-03-04T06:09:41.229164+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:09:41.229164+0000","last_undegraded":"2024-03-04T06:09:41.229164+0000","last_fullsized":"2024-03-04T06:09:41.229164+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067327,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":38,"num_read_kb":21920,"num_write":53,"num_write_kb":15578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":406,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.205628+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:09:41.205628+0000","last_peered":"2024-03-04T06:09:41.205628+0000","last_clean":"2024-03-04T06:09:41.205628+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:09:41.205628+0000","last_undegraded":"2024-03-04T06:09:41.205628+0000","last_fullsized":"2024-03-04T06:09:41.205628+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":259,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.245052+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:09:41.245052+0000","last_peered":"2024-03-04T06:09:41.245052+0000","last_clean":"2024-03-04T06:09:41.245052+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:09:41.245052+0000","last_undegraded":"2024-03-04T06:09:41.245052+0000","last_fullsized":"2024-03-04T06:09:41.245052+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"99'45","reported_seq":344,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.243924+0000","last_change":"2024-03-04T06:09:38.326195+0000","last_active":"2024-03-04T06:09:41.243924+0000","last_peered":"2024-03-04T06:09:41.243924+0000","last_clean":"2024-03-04T06:09:41.243924+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:09:41.243924+0000","last_undegraded":"2024-03-04T06:09:41.243924+0000","last_fullsized":"2024-03-04T06:09:41.243924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10151395,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":42,"num_read_kb":36818,"num_write":36,"num_write_kb":9883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":251,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-04T06:09:39.211585+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:09:39.211585+0000","last_peered":"2024-03-04T06:09:39.211585+0000","last_clean":"2024-03-04T06:09:39.211585+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:09:39.211585+0000","last_undegraded":"2024-03-04T06:09:39.211585+0000","last_fullsized":"2024-03-04T06:09:39.211585+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"110'50","reported_seq":356,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.211680+0000","last_change":"2024-03-04T06:09:38.346218+0000","last_active":"2024-03-04T06:09:41.211680+0000","last_peered":"2024-03-04T06:09:41.211680+0000","last_clean":"2024-03-04T06:09:41.211680+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:09:41.211680+0000","last_undegraded":"2024-03-04T06:09:41.211680+0000","last_fullsized":"2024-03-04T06:09:41.211680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6420274,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":31346,"num_write":34,"num_write_kb":10119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":259,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.226714+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:09:41.226714+0000","last_peered":"2024-03-04T06:09:41.226714+0000","last_clean":"2024-03-04T06:09:41.226714+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:09:41.226714+0000","last_undegraded":"2024-03-04T06:09:41.226714+0000","last_fullsized":"2024-03-04T06:09:41.226714+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"110'163","reported_seq":466,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.214670+0000","last_change":"2024-03-04T06:09:38.350526+0000","last_active":"2024-03-04T06:09:41.214670+0000","last_peered":"2024-03-04T06:09:41.214670+0000","last_clean":"2024-03-04T06:09:41.214670+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:09:41.214670+0000","last_undegraded":"2024-03-04T06:09:41.214670+0000","last_fullsized":"2024-03-04T06:09:41.214670+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15075879,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":138,"num_read_kb":100582,"num_write":111,"num_write_kb":32075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":251,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-04T06:09:39.205682+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:09:39.205682+0000","last_peered":"2024-03-04T06:09:39.205682+0000","last_clean":"2024-03-04T06:09:39.205682+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:09:39.205682+0000","last_undegraded":"2024-03-04T06:09:39.205682+0000","last_fullsized":"2024-03-04T06:09:39.205682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"110'162","reported_seq":437,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-04T06:09:40.213898+0000","last_change":"2024-03-04T06:09:38.239901+0000","last_active":"2024-03-04T06:09:40.213898+0000","last_peered":"2024-03-04T06:09:40.213898+0000","last_clean":"2024-03-04T06:09:40.213898+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:09:40.213898+0000","last_undegraded":"2024-03-04T06:09:40.213898+0000","last_fullsized":"2024-03-04T06:09:40.213898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10386726,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":88,"num_read_kb":57457,"num_write":101,"num_write_kb":26783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"102'47","reported_seq":351,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.244380+0000","last_change":"2024-03-04T06:09:38.331375+0000","last_active":"2024-03-04T06:09:41.244380+0000","last_peered":"2024-03-04T06:09:41.244380+0000","last_clean":"2024-03-04T06:09:41.244380+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:09:41.244380+0000","last_undegraded":"2024-03-04T06:09:41.244380+0000","last_fullsized":"2024-03-04T06:09:41.244380+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8353046,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":36976,"num_write":37,"num_write_kb":9040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":278,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-04T06:09:40.218100+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:09:40.218100+0000","last_peered":"2024-03-04T06:09:40.218100+0000","last_clean":"2024-03-04T06:09:40.218100+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:09:40.218100+0000","last_undegraded":"2024-03-04T06:09:40.218100+0000","last_fullsized":"2024-03-04T06:09:40.218100+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"110'51","reported_seq":375,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-04T06:09:40.219656+0000","last_change":"2024-03-04T06:09:38.225339+0000","last_active":"2024-03-04T06:09:40.219656+0000","last_peered":"2024-03-04T06:09:40.219656+0000","last_clean":"2024-03-04T06:09:40.219656+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:09:40.219656+0000","last_undegraded":"2024-03-04T06:09:40.219656+0000","last_fullsized":"2024-03-04T06:09:40.219656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151362,"num_objects":9,"num_object_clones":4,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":59,"num_read_kb":33400,"num_write":42,"num_write_kb":10631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":273,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.219396+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:09:41.219396+0000","last_peered":"2024-03-04T06:09:41.219396+0000","last_clean":"2024-03-04T06:09:41.219396+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:09:41.219396+0000","last_undegraded":"2024-03-04T06:09:41.219396+0000","last_fullsized":"2024-03-04T06:09:41.219396+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"110'105","reported_seq":418,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.212519+0000","last_change":"2024-03-04T06:09:38.350198+0000","last_active":"2024-03-04T06:09:41.212519+0000","last_peered":"2024-03-04T06:09:41.212519+0000","last_clean":"2024-03-04T06:09:41.212519+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:09:41.212519+0000","last_undegraded":"2024-03-04T06:09:41.212519+0000","last_fullsized":"2024-03-04T06:09:41.212519+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13781707,"num_objects":14,"num_object_clones":9,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":91,"num_read_kb":57571,"num_write":81,"num_write_kb":21755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":278,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-04T06:09:40.212839+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:09:40.212839+0000","last_peered":"2024-03-04T06:09:40.212839+0000","last_clean":"2024-03-04T06:09:40.212839+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:09:40.212839+0000","last_undegraded":"2024-03-04T06:09:40.212839+0000","last_fullsized":"2024-03-04T06:09:40.212839+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"110'123","reported_seq":425,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-03-04T06:09:40.210727+0000","last_change":"2024-03-04T06:09:38.222912+0000","last_active":"2024-03-04T06:09:40.210727+0000","last_peered":"2024-03-04T06:09:40.210727+0000","last_clean":"2024-03-04T06:09:40.210727+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:09:40.210727+0000","last_undegraded":"2024-03-04T06:09:40.210727+0000","last_fullsized":"2024-03-04T06:09:40.210727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12924583,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":70603,"num_write":74,"num_write_kb":20323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":259,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-04T06:09:41.243169+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:09:41.243169+0000","last_peered":"2024-03-04T06:09:41.243169+0000","last_clean":"2024-03-04T06:09:41.243169+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:09:41.243169+0000","last_undegraded":"2024-03-04T06:09:41.243169+0000","last_fullsized":"2024-03-04T06:09:41.243169+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":118774937,"num_objects":160,"num_object_clones":101,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":886,"num_read_kb":595597,"num_write":817,"num_write_kb":226485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1164,"ondisk_log_size":1164,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":132904,"kb_used_data":101920,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94238936,"statfs":{"total":96636764160,"available":96500670464,"internally_reserved":0,"allocated":104366080,"data_stored":115563304,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105596,"kb_used_data":74612,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94266244,"statfs":{"total":96636764160,"available":96528633856,"internally_reserved":0,"allocated":76402688,"data_stored":81687833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105216,"kb_used_data":74232,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94266624,"statfs":{"total":96636764160,"available":96529022976,"internally_reserved":0,"allocated":76013568,"data_stored":84783973,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705778,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99964,"kb_used_data":68980,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94271876,"statfs":{"total":96636764160,"available":96534401024,"internally_reserved":0,"allocated":70635520,"data_stored":78340321,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:09:46.229 INFO:tasks.rados.rados.0.smithi064.stdout:449: done (6 left) 2024-03-04T06:09:46.229 INFO:tasks.rados.rados.0.smithi064.stdout:450: done (5 left) 2024-03-04T06:09:46.229 INFO:tasks.rados.rados.0.smithi064.stdout:451: done (4 left) 2024-03-04T06:09:46.229 INFO:tasks.rados.rados.0.smithi064.stdout:452: done (3 left) 2024-03-04T06:09:46.229 INFO:tasks.rados.rados.0.smithi064.stdout:453: done (2 left) 2024-03-04T06:09:46.230 INFO:tasks.rados.rados.0.smithi064.stdout:454: done (1 left) 2024-03-04T06:09:46.230 INFO:tasks.rados.rados.0.smithi064.stdout:455: done (0 left) 2024-03-04T06:09:46.230 INFO:tasks.rados.rados.0.smithi064.stdout:456: read oid 33 snap 55 2024-03-04T06:09:46.230 INFO:tasks.rados.rados.0.smithi064.stdout:456: expect (ObjNum 199 snap 41 seq_num 199) 2024-03-04T06:09:46.230 INFO:tasks.rados.rados.0.smithi064.stdout:457: snap_create 2024-03-04T06:09:46.273 INFO:tasks.rados.rados.0.smithi064.stdout:456: expect (ObjNum 199 snap 41 seq_num 199) 2024-03-04T06:09:47.234 INFO:tasks.rados.rados.0.smithi064.stdout:456: done (1 left) 2024-03-04T06:09:47.234 INFO:tasks.rados.rados.0.smithi064.stdout:457: done (0 left) 2024-03-04T06:09:47.235 INFO:tasks.rados.rados.0.smithi064.stdout:458: write (excl) oid 45 current snap is 57 2024-03-04T06:09:47.235 INFO:tasks.rados.rados.0.smithi064.stdout:458: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-03-04T06:09:47.241 INFO:tasks.rados.rados.0.smithi064.stdout:458: writing smithi06431743-45 from 699224 to 1247582 tid 1 2024-03-04T06:09:47.248 INFO:tasks.rados.rados.0.smithi064.stdout:458: writing smithi06431743-45 from 2026293 to 2488367 tid 2 2024-03-04T06:09:47.255 INFO:tasks.rados.rados.0.smithi064.stdout:458: writing smithi06431743-45 from 3271546 to 3704300 tid 3 2024-03-04T06:09:47.255 INFO:tasks.rados.rados.0.smithi064.stdout:459: snap_create 2024-03-04T06:09:47.312 INFO:tasks.rados.rados.0.smithi064.stdout:458: finishing write tid 1 to smithi06431743-45 2024-03-04T06:09:47.323 INFO:tasks.rados.rados.0.smithi064.stdout:458: finishing write tid 2 to smithi06431743-45 2024-03-04T06:09:47.331 INFO:tasks.rados.rados.0.smithi064.stdout:458: finishing write tid 3 to smithi06431743-45 2024-03-04T06:09:47.336 INFO:tasks.rados.rados.0.smithi064.stdout:458: finishing write tid 4 to smithi06431743-45 2024-03-04T06:09:47.338 INFO:tasks.rados.rados.0.smithi064.stdout:458: finishing write tid 5 to smithi06431743-45 2024-03-04T06:09:47.338 INFO:tasks.rados.rados.0.smithi064.stdout:458: oid 45 updating version 0 to 98 2024-03-04T06:09:47.338 INFO:tasks.rados.rados.0.smithi064.stdout:458: oid 45 version 98 is already newer than 96 2024-03-04T06:09:47.338 INFO:tasks.rados.rados.0.smithi064.stdout:458: oid 45 version 98 is already newer than 95 2024-03-04T06:09:47.338 INFO:tasks.rados.rados.0.smithi064.stdout:458: oid 45 version 98 is already newer than 98 2024-03-04T06:09:47.338 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 98 (ObjNum 232 snap 57 seq_num 232) dirty exists 2024-03-04T06:09:47.338 INFO:tasks.rados.rados.0.smithi064.stdout:458: left oid 45 (ObjNum 232 snap 57 seq_num 232) 2024-03-04T06:09:47.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:47.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:47.888+0000 7f46da91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:47.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:47.893+0000 7f46da91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:48.189 INFO:tasks.rados.rados.0.smithi064.stdout:458: done (1 left) 2024-03-04T06:09:48.189 INFO:tasks.rados.rados.0.smithi064.stdout:459: done (0 left) 2024-03-04T06:09:48.189 INFO:tasks.rados.rados.0.smithi064.stdout:460: read oid 35 snap 52 2024-03-04T06:09:48.189 INFO:tasks.rados.rados.0.smithi064.stdout:460: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-04T06:09:48.189 INFO:tasks.rados.rados.0.smithi064.stdout:461: read oid 2 snap -1 2024-03-04T06:09:48.190 INFO:tasks.rados.rados.0.smithi064.stdout:461: expect deleted 2024-03-04T06:09:48.190 INFO:tasks.rados.rados.0.smithi064.stdout:462: snap_remove snap 48 2024-03-04T06:09:49.189 INFO:tasks.rados.rados.0.smithi064.stdout:462: done (2 left) 2024-03-04T06:09:49.189 INFO:tasks.rados.rados.0.smithi064.stdout:463: read oid 9 snap -1 2024-03-04T06:09:49.189 INFO:tasks.rados.rados.0.smithi064.stdout:463: expect (ObjNum 169 snap 26 seq_num 169) 2024-03-04T06:09:49.189 INFO:tasks.rados.rados.0.smithi064.stdout:464: read oid 40 snap -1 2024-03-04T06:09:49.190 INFO:tasks.rados.rados.0.smithi064.stdout:464: expect deleted 2024-03-04T06:09:49.190 INFO:tasks.rados.rados.0.smithi064.stdout:465: write (excl) oid 24 current snap is 58 2024-03-04T06:09:49.190 INFO:tasks.rados.rados.0.smithi064.stdout:465: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-03-04T06:09:49.196 INFO:tasks.rados.rados.0.smithi064.stdout:465: writing smithi06431743-24 from 716031 to 1339638 tid 1 2024-03-04T06:09:49.203 INFO:tasks.rados.rados.0.smithi064.stdout:465: writing smithi06431743-24 from 2084775 to 2690507 tid 2 2024-03-04T06:09:49.211 INFO:tasks.rados.rados.0.smithi064.stdout:465: writing smithi06431743-24 from 3298969 to 3916031 tid 3 2024-03-04T06:09:49.211 INFO:tasks.rados.rados.0.smithi064.stdout:466: read oid 17 snap -1 2024-03-04T06:09:49.211 INFO:tasks.rados.rados.0.smithi064.stdout:466: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:09:49.211 INFO:tasks.rados.rados.0.smithi064.stdout:467: snap_remove snap 32 2024-03-04T06:09:49.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:50.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:50.074+0000 7fd5ff26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:50.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:50.079+0000 7fd5ff26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:50.189 INFO:tasks.rados.rados.0.smithi064.stdout:467: done (6 left) 2024-03-04T06:09:50.190 INFO:tasks.rados.rados.0.smithi064.stdout:468: delete oid 20 current snap is 58 2024-03-04T06:09:50.190 INFO:tasks.rados.rados.0.smithi064.stdout:460: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-04T06:09:50.466 INFO:tasks.rados.rados.0.smithi064.stdout:463: expect (ObjNum 169 snap 26 seq_num 169) 2024-03-04T06:09:50.662 INFO:tasks.rados.rados.0.smithi064.stdout:466: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:09:50.938 INFO:tasks.rados.rados.0.smithi064.stdout:465: finishing write tid 1 to smithi06431743-24 2024-03-04T06:09:50.938 INFO:tasks.rados.rados.0.smithi064.stdout:465: finishing write tid 2 to smithi06431743-24 2024-03-04T06:09:50.938 INFO:tasks.rados.rados.0.smithi064.stdout:465: finishing write tid 3 to smithi06431743-24 2024-03-04T06:09:50.939 INFO:tasks.rados.rados.0.smithi064.stdout:465: finishing write tid 4 to smithi06431743-24 2024-03-04T06:09:50.939 INFO:tasks.rados.rados.0.smithi064.stdout:465: finishing write tid 5 to smithi06431743-24 2024-03-04T06:09:50.939 INFO:tasks.rados.rados.0.smithi064.stdout:465: oid 24 updating version 0 to 118 2024-03-04T06:09:50.939 INFO:tasks.rados.rados.0.smithi064.stdout:465: oid 24 version 118 is already newer than 116 2024-03-04T06:09:50.939 INFO:tasks.rados.rados.0.smithi064.stdout:465: oid 24 updating version 118 to 119 2024-03-04T06:09:50.939 INFO:tasks.rados.rados.0.smithi064.stdout:465: oid 24 version 119 is already newer than 118 2024-03-04T06:09:50.939 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 119 (ObjNum 233 snap 58 seq_num 233) dirty exists 2024-03-04T06:09:50.939 INFO:tasks.rados.rados.0.smithi064.stdout:465: left oid 24 (ObjNum 233 snap 58 seq_num 233) 2024-03-04T06:09:50.939 INFO:tasks.rados.rados.0.smithi064.stdout:460: done (6 left) 2024-03-04T06:09:50.939 INFO:tasks.rados.rados.0.smithi064.stdout:461: done (5 left) 2024-03-04T06:09:50.940 INFO:tasks.rados.rados.0.smithi064.stdout:463: done (4 left) 2024-03-04T06:09:50.940 INFO:tasks.rados.rados.0.smithi064.stdout:464: done (3 left) 2024-03-04T06:09:50.940 INFO:tasks.rados.rados.0.smithi064.stdout:465: done (2 left) 2024-03-04T06:09:50.940 INFO:tasks.rados.rados.0.smithi064.stdout:466: done (1 left) 2024-03-04T06:09:50.940 INFO:tasks.rados.rados.0.smithi064.stdout:468: done (0 left) 2024-03-04T06:09:50.940 INFO:tasks.rados.rados.0.smithi064.stdout:469: read oid 15 snap 55 2024-03-04T06:09:50.940 INFO:tasks.rados.rados.0.smithi064.stdout:469: expect (ObjNum 196 snap 40 seq_num 196) 2024-03-04T06:09:50.940 INFO:tasks.rados.rados.0.smithi064.stdout:470: snap_create 2024-03-04T06:09:50.950 INFO:tasks.rados.rados.0.smithi064.stdout:469: expect (ObjNum 196 snap 40 seq_num 196) 2024-03-04T06:09:51.058 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:09:51.059 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:09:51.059 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:09:51.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:09:51.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:51.163+0000 7f6d6eb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:51.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:51.170+0000 7f6d6eb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:51.183 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:51.183 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:51.183 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:51.184 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:51.184 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:51.184 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:51.184 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:51.184 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:51.184 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:51.184 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:51.184 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:51.184 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:51.186 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:51.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:09:51.201 INFO:tasks.rados.rados.0.smithi064.stdout:469: done (1 left) 2024-03-04T06:09:51.201 INFO:tasks.rados.rados.0.smithi064.stdout:470: done (0 left) 2024-03-04T06:09:51.201 INFO:tasks.rados.rados.0.smithi064.stdout:471: delete oid 12 current snap is 59 2024-03-04T06:09:51.227 INFO:tasks.rados.rados.0.smithi064.stdout:471: done (0 left) 2024-03-04T06:09:51.227 INFO:tasks.rados.rados.0.smithi064.stdout:472: snap_remove snap 52 2024-03-04T06:09:51.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:51.293+0000 7f558ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:51.301 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:51.300+0000 7f558ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:51.313 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:51.313 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:51.313 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:51.313 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:51.313 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:51.314 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:51.314 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:51.314 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:51.314 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:51.314 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:51.314 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:51.314 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:51.316 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:51.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:09:51.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:51.459+0000 7f453b42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:51.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:51.466+0000 7f453b42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:51.480 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:51.483 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:51.484 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:51.484 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:51.484 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:51.484 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:51.485 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:51.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:09:51.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:51.588+0000 7fee8f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:51.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:51.595+0000 7fee8f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:51.609 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:51.609 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:51.609 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:51.609 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:51.609 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:51.609 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:51.609 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:51.609 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:51.610 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:51.610 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:51.610 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:51.610 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:51.612 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:52.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:52.206 INFO:tasks.rados.rados.0.smithi064.stdout:472: done (0 left) 2024-03-04T06:09:52.206 INFO:tasks.rados.rados.0.smithi064.stdout:473: snap_create 2024-03-04T06:09:52.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:52.262+0000 7f0d6815d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:52.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:52.267+0000 7f0d6815d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:53.209 INFO:tasks.rados.rados.0.smithi064.stdout:473: done (0 left) 2024-03-04T06:09:53.209 INFO:tasks.rados.rados.0.smithi064.stdout:474: snap_remove snap 47 2024-03-04T06:09:54.212 INFO:tasks.rados.rados.0.smithi064.stdout:474: done (0 left) 2024-03-04T06:09:54.212 INFO:tasks.rados.rados.0.smithi064.stdout:475: read oid 6 snap -1 2024-03-04T06:09:54.213 INFO:tasks.rados.rados.0.smithi064.stdout:475: expect (ObjNum 162 snap 24 seq_num 162) 2024-03-04T06:09:54.213 INFO:tasks.rados.rados.0.smithi064.stdout:476: snap_remove snap 55 2024-03-04T06:09:54.337 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:54.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:54.448+0000 7fd78c15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:54.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:54.453+0000 7fd78c15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:55.209 INFO:tasks.rados.rados.0.smithi064.stdout:476: done (1 left) 2024-03-04T06:09:55.209 INFO:tasks.rados.rados.0.smithi064.stdout:477: read oid 25 snap 46 2024-03-04T06:09:55.209 INFO:tasks.rados.rados.0.smithi064.stdout:477: expect (ObjNum 136 snap 17 seq_num 136) 2024-03-04T06:09:55.209 INFO:tasks.rados.rados.0.smithi064.stdout:478: delete oid 2 current snap is 60 2024-03-04T06:09:55.209 INFO:tasks.rados.rados.0.smithi064.stdout:475: expect (ObjNum 162 snap 24 seq_num 162) 2024-03-04T06:09:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:477: expect (ObjNum 136 snap 17 seq_num 136) 2024-03-04T06:09:55.499 INFO:tasks.rados.rados.0.smithi064.stdout:475: done (2 left) 2024-03-04T06:09:55.499 INFO:tasks.rados.rados.0.smithi064.stdout:477: done (1 left) 2024-03-04T06:09:55.499 INFO:tasks.rados.rados.0.smithi064.stdout:478: done (0 left) 2024-03-04T06:09:55.499 INFO:tasks.rados.rados.0.smithi064.stdout:479: read oid 16 snap -1 2024-03-04T06:09:55.499 INFO:tasks.rados.rados.0.smithi064.stdout:479: expect (ObjNum 175 snap 33 seq_num 175) 2024-03-04T06:09:55.499 INFO:tasks.rados.rados.0.smithi064.stdout:480: snap_create 2024-03-04T06:09:55.561 INFO:tasks.rados.rados.0.smithi064.stdout:479: expect (ObjNum 175 snap 33 seq_num 175) 2024-03-04T06:09:56.228 INFO:tasks.rados.rados.0.smithi064.stdout:479: done (1 left) 2024-03-04T06:09:56.228 INFO:tasks.rados.rados.0.smithi064.stdout:480: done (0 left) 2024-03-04T06:09:56.228 INFO:tasks.rados.rados.0.smithi064.stdout:481: snap_create 2024-03-04T06:09:56.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:09:56.613 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:09:56.613 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:09:56.613 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:09:56.613 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:09:56.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:56.633+0000 7fd1dcb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:56.638 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:56.637+0000 7fd1dcb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:56.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:56.727+0000 7fabf031d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:56.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:56.735+0000 7fabf031d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:56.747 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:56.747 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:56.747 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:56.748 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:56.748 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:56.748 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:56.748 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:56.748 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:56.748 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:56.748 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:56.748 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:56.748 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:56.750 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:56.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:09:56.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:56.894+0000 7feeae8dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:56.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:56.901+0000 7feeae8dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:56.914 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:56.917 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:56.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:09:57.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:57.020+0000 7fc6a759d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:57.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:57.027+0000 7fc6a759d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:57.041 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:57.041 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:57.041 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:57.041 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:57.042 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:57.042 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:57.042 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:57.042 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:57.042 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:57.042 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:57.042 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:57.042 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:57.044 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:57.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:09:57.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:57.148+0000 7ff19311d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:57.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:57.155+0000 7ff19311d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:57.169 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:09:57.169 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:09:57.169 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:09:57.169 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:09:57.169 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:09:57.169 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:09:57.169 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:09:57.169 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:09:57.169 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:09:57.169 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:09:57.170 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:09:57.170 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:09:57.171 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:09:57.172 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:09:57.225 INFO:tasks.rados.rados.0.smithi064.stdout:481: done (0 left) 2024-03-04T06:09:57.225 INFO:tasks.rados.rados.0.smithi064.stdout:482: snap_remove snap 61 2024-03-04T06:09:58.189 INFO:tasks.rados.rados.0.smithi064.stdout:482: done (0 left) 2024-03-04T06:09:58.189 INFO:tasks.rados.rados.0.smithi064.stdout:483: delete oid 48 current snap is 62 2024-03-04T06:09:58.256 INFO:tasks.rados.rados.0.smithi064.stdout:483: done (0 left) 2024-03-04T06:09:58.256 INFO:tasks.rados.rados.0.smithi064.stdout:484: read oid 46 snap -1 2024-03-04T06:09:58.256 INFO:tasks.rados.rados.0.smithi064.stdout:484: expect deleted 2024-03-04T06:09:58.256 INFO:tasks.rados.rados.0.smithi064.stdout:485: write oid 40 current snap is 62 2024-03-04T06:09:58.256 INFO:tasks.rados.rados.0.smithi064.stdout:485: seq_num 238 ranges {400066=716205,1860832=139234} 2024-03-04T06:09:58.262 INFO:tasks.rados.rados.0.smithi064.stdout:485: writing smithi06431743-40 from 400066 to 1116271 tid 1 2024-03-04T06:09:58.267 INFO:tasks.rados.rados.0.smithi064.stdout:485: writing smithi06431743-40 from 1860832 to 2000066 tid 2 2024-03-04T06:09:58.267 INFO:tasks.rados.rados.0.smithi064.stdout:486: snap_create 2024-03-04T06:09:58.306 INFO:tasks.rados.rados.0.smithi064.stdout:485: finishing write tid 1 to smithi06431743-40 2024-03-04T06:09:58.315 INFO:tasks.rados.rados.0.smithi064.stdout:485: finishing write tid 2 to smithi06431743-40 2024-03-04T06:09:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:485: finishing write tid 3 to smithi06431743-40 2024-03-04T06:09:58.321 INFO:tasks.rados.rados.0.smithi064.stdout:485: finishing write tid 4 to smithi06431743-40 2024-03-04T06:09:58.321 INFO:tasks.rados.rados.0.smithi064.stdout:485: oid 40 updating version 0 to 70 2024-03-04T06:09:58.321 INFO:tasks.rados.rados.0.smithi064.stdout:485: oid 40 version 70 is already newer than 69 2024-03-04T06:09:58.321 INFO:tasks.rados.rados.0.smithi064.stdout:485: oid 40 version 70 is already newer than 68 2024-03-04T06:09:58.321 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 70 (ObjNum 238 snap 62 seq_num 238) dirty exists 2024-03-04T06:09:58.321 INFO:tasks.rados.rados.0.smithi064.stdout:485: left oid 40 (ObjNum 238 snap 62 seq_num 238) 2024-03-04T06:09:58.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:09:58.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:58.824+0000 7fea1a22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:58.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:09:58.828+0000 7fea1a22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:09:59.245 INFO:tasks.rados.rados.0.smithi064.stdout:484: done (2 left) 2024-03-04T06:09:59.246 INFO:tasks.rados.rados.0.smithi064.stdout:485: done (1 left) 2024-03-04T06:09:59.246 INFO:tasks.rados.rados.0.smithi064.stdout:486: done (0 left) 2024-03-04T06:09:59.246 INFO:tasks.rados.rados.0.smithi064.stdout:487: snap_create 2024-03-04T06:10:00.247 INFO:tasks.rados.rados.0.smithi064.stdout:487: done (0 left) 2024-03-04T06:10:00.247 INFO:tasks.rados.rados.0.smithi064.stdout:488: write (excl) oid 9 current snap is 64 2024-03-04T06:10:00.248 INFO:tasks.rados.rados.0.smithi064.stdout:488: seq_num 239 ranges {416873=791454,1919314=97559} 2024-03-04T06:10:00.254 INFO:tasks.rados.rados.0.smithi064.stdout:488: writing smithi06431743-9 from 416873 to 1208327 tid 1 2024-03-04T06:10:00.258 INFO:tasks.rados.rados.0.smithi064.stdout:488: writing smithi06431743-9 from 1919314 to 2016873 tid 2 2024-03-04T06:10:00.258 INFO:tasks.rados.rados.0.smithi064.stdout:489: write oid 41 current snap is 64 2024-03-04T06:10:00.258 INFO:tasks.rados.rados.0.smithi064.stdout:489: seq_num 240 ranges {433680=466703,1577796=455884} 2024-03-04T06:10:00.263 INFO:tasks.rados.rados.0.smithi064.stdout:489: writing smithi06431743-41 from 433680 to 900383 tid 1 2024-03-04T06:10:00.269 INFO:tasks.rados.rados.0.smithi064.stdout:489: writing smithi06431743-41 from 1577796 to 2033680 tid 2 2024-03-04T06:10:00.269 INFO:tasks.rados.rados.0.smithi064.stdout:490: snap_create 2024-03-04T06:10:00.317 INFO:tasks.rados.rados.0.smithi064.stdout:489: finishing write tid 1 to smithi06431743-41 2024-03-04T06:10:00.331 INFO:tasks.rados.rados.0.smithi064.stdout:488: finishing write tid 1 to smithi06431743-9 2024-03-04T06:10:00.332 INFO:tasks.rados.rados.0.smithi064.stdout:489: finishing write tid 2 to smithi06431743-41 2024-03-04T06:10:00.341 INFO:tasks.rados.rados.0.smithi064.stdout:489: finishing write tid 3 to smithi06431743-41 2024-03-04T06:10:00.342 INFO:tasks.rados.rados.0.smithi064.stdout:488: finishing write tid 2 to smithi06431743-9 2024-03-04T06:10:00.343 INFO:tasks.rados.rados.0.smithi064.stdout:489: finishing write tid 4 to smithi06431743-41 2024-03-04T06:10:00.343 INFO:tasks.rados.rados.0.smithi064.stdout:489: oid 41 updating version 0 to 47 2024-03-04T06:10:00.343 INFO:tasks.rados.rados.0.smithi064.stdout:489: oid 41 updating version 47 to 49 2024-03-04T06:10:00.343 INFO:tasks.rados.rados.0.smithi064.stdout:489: oid 41 version 49 is already newer than 49 2024-03-04T06:10:00.343 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 49 (ObjNum 240 snap 64 seq_num 240) dirty exists 2024-03-04T06:10:00.343 INFO:tasks.rados.rados.0.smithi064.stdout:489: left oid 41 (ObjNum 240 snap 64 seq_num 240) 2024-03-04T06:10:00.352 INFO:tasks.rados.rados.0.smithi064.stdout:488: finishing write tid 3 to smithi06431743-9 2024-03-04T06:10:00.353 INFO:tasks.rados.rados.0.smithi064.stdout:488: finishing write tid 4 to smithi06431743-9 2024-03-04T06:10:00.353 INFO:tasks.rados.rados.0.smithi064.stdout:488: oid 9 updating version 0 to 144 2024-03-04T06:10:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:488: oid 9 updating version 144 to 146 2024-03-04T06:10:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:488: oid 9 version 146 is already newer than 145 2024-03-04T06:10:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 146 (ObjNum 239 snap 64 seq_num 239) dirty exists 2024-03-04T06:10:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:488: left oid 9 (ObjNum 239 snap 64 seq_num 239) 2024-03-04T06:10:00.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:01.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:01.015+0000 7f1c55a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:01.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:01.019+0000 7f1c55a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:01.259 INFO:tasks.rados.rados.0.smithi064.stdout:488: done (2 left) 2024-03-04T06:10:01.259 INFO:tasks.rados.rados.0.smithi064.stdout:489: done (1 left) 2024-03-04T06:10:01.259 INFO:tasks.rados.rados.0.smithi064.stdout:490: done (0 left) 2024-03-04T06:10:01.259 INFO:tasks.rados.rados.0.smithi064.stdout:491: snap_remove snap 59 2024-03-04T06:10:02.173 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:10:02.173 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:10:02.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:10:02.257 INFO:tasks.rados.rados.0.smithi064.stdout:491: done (0 left) 2024-03-04T06:10:02.257 INFO:tasks.rados.rados.0.smithi064.stdout:492: snap_create 2024-03-04T06:10:02.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:02.282+0000 7f77cd307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:02.289 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:02.288+0000 7f77cd307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:02.492 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:10:02.492 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:10:02.504 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-03-04T06:10:02.255721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119903228,"num_objects":175,"num_object_clones":112,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":24,"num_read":1081,"num_read_kb":677084,"num_write":1006,"num_write_kb":244450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1427,"ondisk_log_size":1427,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":458484,"kb_used_data":334548,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":377028876,"statfs":{"total":386547056640,"available":386077569024,"internally_reserved":0,"allocated":342577152,"data_stored":386246305,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3495757,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":10,"num_read_kb":5011,"num_write":10,"num_write_kb":2511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.067693"},"pg_stats":[{"pgid":"3.f","version":"130'70","reported_seq":393,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-04T06:09:58.315196+0000","last_change":"2024-03-04T06:09:58.295872+0000","last_active":"2024-03-04T06:09:58.315196+0000","last_peered":"2024-03-04T06:09:58.315196+0000","last_clean":"2024-03-04T06:09:58.315196+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:09:58.315196+0000","last_undegraded":"2024-03-04T06:09:58.315196+0000","last_fullsized":"2024-03-04T06:09:58.315196+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5731220,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":25,"num_read_kb":6455,"num_write":50,"num_write_kb":13335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"122'55","reported_seq":412,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-04T06:10:00.280436+0000","last_change":"2024-03-04T06:09:58.397463+0000","last_active":"2024-03-04T06:10:00.280436+0000","last_peered":"2024-03-04T06:10:00.280436+0000","last_clean":"2024-03-04T06:10:00.280436+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:10:00.280436+0000","last_undegraded":"2024-03-04T06:10:00.280436+0000","last_fullsized":"2024-03-04T06:10:00.280436+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092488,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":33,"num_read_kb":21067,"num_write":36,"num_write_kb":9740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"127'108","reported_seq":436,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-04T06:10:01.266475+0000","last_change":"2024-03-04T06:09:58.211988+0000","last_active":"2024-03-04T06:10:01.266475+0000","last_peered":"2024-03-04T06:10:01.266475+0000","last_clean":"2024-03-04T06:10:01.266475+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:10:01.266475+0000","last_undegraded":"2024-03-04T06:10:01.266475+0000","last_fullsized":"2024-03-04T06:10:01.266475+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9781674,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":61,"num_read_kb":35347,"num_write":68,"num_write_kb":19466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"127'78","reported_seq":412,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-04T06:09:58.277225+0000","last_change":"2024-03-04T06:09:58.284183+0000","last_active":"2024-03-04T06:09:58.277225+0000","last_peered":"2024-03-04T06:09:58.277225+0000","last_clean":"2024-03-04T06:09:58.277225+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:09:58.277225+0000","last_undegraded":"2024-03-04T06:09:58.277225+0000","last_fullsized":"2024-03-04T06:09:58.277225+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226944,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":58,"num_read_kb":46129,"num_write":53,"num_write_kb":15175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"122'33","reported_seq":379,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-04T06:09:59.267757+0000","last_change":"2024-03-04T06:09:58.232339+0000","last_active":"2024-03-04T06:09:59.267757+0000","last_peered":"2024-03-04T06:09:59.267757+0000","last_clean":"2024-03-04T06:09:59.267757+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:09:59.267757+0000","last_undegraded":"2024-03-04T06:09:59.267757+0000","last_fullsized":"2024-03-04T06:09:59.267757+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3563084,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":17038,"num_write":18,"num_write_kb":5229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"117'46","reported_seq":391,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-04T06:09:58.287688+0000","last_change":"2024-03-04T06:09:58.296207+0000","last_active":"2024-03-04T06:09:58.287688+0000","last_peered":"2024-03-04T06:09:58.287688+0000","last_clean":"2024-03-04T06:09:58.287688+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:09:58.287688+0000","last_undegraded":"2024-03-04T06:09:58.287688+0000","last_fullsized":"2024-03-04T06:09:58.287688+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10084167,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":45,"num_read_kb":33911,"num_write":37,"num_write_kb":11229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":339,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-04T06:09:58.289297+0000","last_change":"2024-03-04T06:09:58.296713+0000","last_active":"2024-03-04T06:09:58.289297+0000","last_peered":"2024-03-04T06:09:58.289297+0000","last_clean":"2024-03-04T06:09:58.289297+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:09:58.289297+0000","last_undegraded":"2024-03-04T06:09:58.289297+0000","last_fullsized":"2024-03-04T06:09:58.289297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"126'77","reported_seq":416,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-04T06:10:01.267122+0000","last_change":"2024-03-04T06:09:58.211801+0000","last_active":"2024-03-04T06:10:01.267122+0000","last_peered":"2024-03-04T06:10:01.267122+0000","last_clean":"2024-03-04T06:10:01.267122+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:10:01.267122+0000","last_undegraded":"2024-03-04T06:10:01.267122+0000","last_fullsized":"2024-03-04T06:10:01.267122+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067327,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":44,"num_read_kb":31734,"num_write":53,"num_write_kb":15578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":444,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-04T06:09:59.254759+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:09:59.254759+0000","last_peered":"2024-03-04T06:09:59.254759+0000","last_clean":"2024-03-04T06:09:59.254759+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:09:59.254759+0000","last_undegraded":"2024-03-04T06:09:59.254759+0000","last_fullsized":"2024-03-04T06:09:59.254759+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":309,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-04T06:10:01.273634+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:10:01.273634+0000","last_peered":"2024-03-04T06:10:01.273634+0000","last_clean":"2024-03-04T06:10:01.273634+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:10:01.273634+0000","last_undegraded":"2024-03-04T06:10:01.273634+0000","last_fullsized":"2024-03-04T06:10:01.273634+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"132'49","reported_seq":405,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-04T06:10:01.273297+0000","last_change":"2024-03-04T06:09:58.223270+0000","last_active":"2024-03-04T06:10:01.273297+0000","last_peered":"2024-03-04T06:10:01.273297+0000","last_clean":"2024-03-04T06:10:01.273297+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:10:01.273297+0000","last_undegraded":"2024-03-04T06:10:01.273297+0000","last_fullsized":"2024-03-04T06:10:01.273297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8285884,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":43,"num_read_kb":36819,"num_write":40,"num_write_kb":10785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":298,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-04T06:09:58.285869+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:09:58.285869+0000","last_peered":"2024-03-04T06:09:58.285869+0000","last_clean":"2024-03-04T06:09:58.285869+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:09:58.285869+0000","last_undegraded":"2024-03-04T06:09:58.285869+0000","last_fullsized":"2024-03-04T06:09:58.285869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"130'67","reported_seq":410,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-04T06:09:59.259686+0000","last_change":"2024-03-04T06:09:58.231241+0000","last_active":"2024-03-04T06:09:59.259686+0000","last_peered":"2024-03-04T06:09:59.259686+0000","last_clean":"2024-03-04T06:09:59.259686+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:09:59.259686+0000","last_undegraded":"2024-03-04T06:09:59.259686+0000","last_fullsized":"2024-03-04T06:09:59.259686+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3848803,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":51,"num_read_kb":40579,"num_write":40,"num_write_kb":11940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":309,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-04T06:10:01.263127+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:10:01.263127+0000","last_peered":"2024-03-04T06:10:01.263127+0000","last_clean":"2024-03-04T06:10:01.263127+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:10:01.263127+0000","last_undegraded":"2024-03-04T06:10:01.263127+0000","last_fullsized":"2024-03-04T06:10:01.263127+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"122'167","reported_seq":514,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-04T06:09:59.268580+0000","last_change":"2024-03-04T06:09:58.232563+0000","last_active":"2024-03-04T06:09:59.268580+0000","last_peered":"2024-03-04T06:09:59.268580+0000","last_clean":"2024-03-04T06:09:59.268580+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:09:59.268580+0000","last_undegraded":"2024-03-04T06:09:59.268580+0000","last_fullsized":"2024-03-04T06:09:59.268580+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15075879,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":144,"num_read_kb":105055,"num_write":111,"num_write_kb":32075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":298,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-04T06:09:58.280067+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:09:58.280067+0000","last_peered":"2024-03-04T06:09:58.280067+0000","last_clean":"2024-03-04T06:09:58.280067+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:09:58.280067+0000","last_undegraded":"2024-03-04T06:09:58.280067+0000","last_fullsized":"2024-03-04T06:09:58.280067+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"127'184","reported_seq":498,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-04T06:10:00.279567+0000","last_change":"2024-03-04T06:09:58.397005+0000","last_active":"2024-03-04T06:10:00.279567+0000","last_peered":"2024-03-04T06:10:00.279567+0000","last_clean":"2024-03-04T06:10:00.279567+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:10:00.279567+0000","last_undegraded":"2024-03-04T06:10:00.279567+0000","last_fullsized":"2024-03-04T06:10:00.279567+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10588410,"num_objects":19,"num_object_clones":13,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":90,"num_read_kb":57458,"num_write":107,"num_write_kb":28374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"122'49","reported_seq":417,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-04T06:10:01.271136+0000","last_change":"2024-03-04T06:09:58.221201+0000","last_active":"2024-03-04T06:10:01.271136+0000","last_peered":"2024-03-04T06:10:01.271136+0000","last_clean":"2024-03-04T06:10:01.271136+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:10:01.271136+0000","last_undegraded":"2024-03-04T06:10:01.271136+0000","last_fullsized":"2024-03-04T06:10:01.271136+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8353046,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":74,"num_read_kb":54501,"num_write":37,"num_write_kb":9040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":326,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-04T06:10:00.287923+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:10:00.287923+0000","last_peered":"2024-03-04T06:10:00.287923+0000","last_clean":"2024-03-04T06:10:00.287923+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:10:00.287923+0000","last_undegraded":"2024-03-04T06:10:00.287923+0000","last_fullsized":"2024-03-04T06:10:00.287923+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"127'66","reported_seq":438,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-04T06:10:00.289942+0000","last_change":"2024-03-04T06:09:58.401630+0000","last_active":"2024-03-04T06:10:00.289942+0000","last_peered":"2024-03-04T06:10:00.289942+0000","last_clean":"2024-03-04T06:10:00.289942+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:10:00.289942+0000","last_undegraded":"2024-03-04T06:10:00.289942+0000","last_fullsized":"2024-03-04T06:10:00.289942+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722800,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":67,"num_read_kb":38410,"num_write":47,"num_write_kb":12439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":311,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-04T06:09:59.267436+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:09:59.267436+0000","last_peered":"2024-03-04T06:09:59.267436+0000","last_clean":"2024-03-04T06:09:59.267436+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:09:59.267436+0000","last_undegraded":"2024-03-04T06:09:59.267436+0000","last_fullsized":"2024-03-04T06:09:59.267436+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"130'139","reported_seq":477,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-04T06:09:59.260597+0000","last_change":"2024-03-04T06:09:58.295543+0000","last_active":"2024-03-04T06:09:59.260597+0000","last_peered":"2024-03-04T06:09:59.260597+0000","last_clean":"2024-03-04T06:09:59.260597+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:09:59.260597+0000","last_undegraded":"2024-03-04T06:09:59.260597+0000","last_fullsized":"2024-03-04T06:09:59.260597+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11647218,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":100,"num_read_kb":65964,"num_write":92,"num_write_kb":25217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":326,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-04T06:10:00.275678+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:10:00.275678+0000","last_peered":"2024-03-04T06:10:00.275678+0000","last_clean":"2024-03-04T06:10:00.275678+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:10:00.275678+0000","last_undegraded":"2024-03-04T06:10:00.275678+0000","last_fullsized":"2024-03-04T06:10:00.275678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"127'142","reported_seq":492,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-03-04T06:10:00.274927+0000","last_change":"2024-03-04T06:09:58.396725+0000","last_active":"2024-03-04T06:10:00.274927+0000","last_peered":"2024-03-04T06:10:00.274927+0000","last_clean":"2024-03-04T06:10:00.274927+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:10:00.274927+0000","last_undegraded":"2024-03-04T06:10:00.274927+0000","last_fullsized":"2024-03-04T06:10:00.274927+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13243916,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":86526,"num_write":80,"num_write_kb":22934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":309,"reported_epoch":133,"state":"active+clean","last_fresh":"2024-03-04T06:10:01.274042+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:10:01.274042+0000","last_peered":"2024-03-04T06:10:01.274042+0000","last_clean":"2024-03-04T06:10:01.274042+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:10:01.274042+0000","last_undegraded":"2024-03-04T06:10:01.274042+0000","last_fullsized":"2024-03-04T06:10:01.274042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":119312860,"num_objects":173,"num_object_clones":112,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":24,"num_read":978,"num_read_kb":676993,"num_write":869,"num_write_kb":242556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1330,"ondisk_log_size":1330,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":136736,"kb_used_data":105752,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94235104,"statfs":{"total":96636764160,"available":96496746496,"internally_reserved":0,"allocated":108290048,"data_stored":123522180,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705799,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112304,"kb_used_data":81320,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94259536,"statfs":{"total":96636764160,"available":96521764864,"internally_reserved":0,"allocated":83271680,"data_stored":93344657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100876,"kb_used_data":69892,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94270964,"statfs":{"total":96636764160,"available":96533467136,"internally_reserved":0,"allocated":71569408,"data_stored":79022859,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705798,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":108568,"kb_used_data":77584,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94263272,"statfs":{"total":96636764160,"available":96525590528,"internally_reserved":0,"allocated":79446016,"data_stored":90356609,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:10:03.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:03.199 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:03.196+0000 7fb38e11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:03.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:03.200+0000 7fb38e11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:03.262 INFO:tasks.rados.rados.0.smithi064.stdout:492: done (0 left) 2024-03-04T06:10:03.262 INFO:tasks.rados.rados.0.smithi064.stdout:493: read oid 32 snap -1 2024-03-04T06:10:03.262 INFO:tasks.rados.rados.0.smithi064.stdout:493: expect (ObjNum 208 snap 44 seq_num 208) 2024-03-04T06:10:03.262 INFO:tasks.rados.rados.0.smithi064.stdout:494: write oid 40 current snap is 66 2024-03-04T06:10:03.262 INFO:tasks.rados.rados.0.smithi064.stdout:494: seq_num 241 ranges {450487=541952,1636278=414209} 2024-03-04T06:10:03.267 INFO:tasks.rados.rados.0.smithi064.stdout:494: writing smithi06431743-40 from 450487 to 992439 tid 1 2024-03-04T06:10:03.274 INFO:tasks.rados.rados.0.smithi064.stdout:494: writing smithi06431743-40 from 1636278 to 2050487 tid 2 2024-03-04T06:10:03.274 INFO:tasks.rados.rados.0.smithi064.stdout:495: snap_create 2024-03-04T06:10:03.315 INFO:tasks.rados.rados.0.smithi064.stdout:493: expect (ObjNum 208 snap 44 seq_num 208) 2024-03-04T06:10:03.532 INFO:tasks.rados.rados.0.smithi064.stdout:494: finishing write tid 1 to smithi06431743-40 2024-03-04T06:10:03.532 INFO:tasks.rados.rados.0.smithi064.stdout:494: finishing write tid 2 to smithi06431743-40 2024-03-04T06:10:03.532 INFO:tasks.rados.rados.0.smithi064.stdout:494: finishing write tid 3 to smithi06431743-40 2024-03-04T06:10:03.532 INFO:tasks.rados.rados.0.smithi064.stdout:494: finishing write tid 4 to smithi06431743-40 2024-03-04T06:10:03.532 INFO:tasks.rados.rados.0.smithi064.stdout:494: oid 40 updating version 0 to 72 2024-03-04T06:10:03.533 INFO:tasks.rados.rados.0.smithi064.stdout:494: oid 40 updating version 72 to 74 2024-03-04T06:10:03.533 INFO:tasks.rados.rados.0.smithi064.stdout:494: oid 40 version 74 is already newer than 74 2024-03-04T06:10:03.533 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 74 (ObjNum 241 snap 66 seq_num 241) dirty exists 2024-03-04T06:10:03.533 INFO:tasks.rados.rados.0.smithi064.stdout:494: left oid 40 (ObjNum 241 snap 66 seq_num 241) 2024-03-04T06:10:04.266 INFO:tasks.rados.rados.0.smithi064.stdout:493: done (2 left) 2024-03-04T06:10:04.266 INFO:tasks.rados.rados.0.smithi064.stdout:494: done (1 left) 2024-03-04T06:10:04.266 INFO:tasks.rados.rados.0.smithi064.stdout:495: done (0 left) 2024-03-04T06:10:04.266 INFO:tasks.rados.rados.0.smithi064.stdout:496: read oid 38 snap -1 2024-03-04T06:10:04.266 INFO:tasks.rados.rados.0.smithi064.stdout:496: expect (ObjNum 230 snap 55 seq_num 230) 2024-03-04T06:10:04.266 INFO:tasks.rados.rados.0.smithi064.stdout:497: snap_remove snap 56 2024-03-04T06:10:05.273 INFO:tasks.rados.rados.0.smithi064.stdout:497: done (1 left) 2024-03-04T06:10:05.273 INFO:tasks.rados.rados.0.smithi064.stdout:498: write oid 9 current snap is 67 2024-03-04T06:10:05.273 INFO:tasks.rados.rados.0.smithi064.stdout:498: seq_num 242 ranges {467294=617201,1578407=488887} 2024-03-04T06:10:05.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:05.279 INFO:tasks.rados.rados.0.smithi064.stdout:498: writing smithi06431743-9 from 467294 to 1084495 tid 1 2024-03-04T06:10:05.285 INFO:tasks.rados.rados.0.smithi064.stdout:498: writing smithi06431743-9 from 1578407 to 2067294 tid 2 2024-03-04T06:10:05.285 INFO:tasks.rados.rados.0.smithi064.stdout:499: write oid 34 current snap is 67 2024-03-04T06:10:05.285 INFO:tasks.rados.rados.0.smithi064.stdout:499: seq_num 243 ranges {484101=692450,1636889=447212} 2024-03-04T06:10:05.293 INFO:tasks.rados.rados.0.smithi064.stdout:499: writing smithi06431743-34 from 484101 to 1176551 tid 1 2024-03-04T06:10:05.298 INFO:tasks.rados.rados.0.smithi064.stdout:499: writing smithi06431743-34 from 1636889 to 2084101 tid 2 2024-03-04T06:10:05.298 INFO:tasks.rados.rados.0.smithi064.stdout:500: snap_create 2024-03-04T06:10:05.299 INFO:tasks.rados.rados.0.smithi064.stdout:496: expect (ObjNum 230 snap 55 seq_num 230) 2024-03-04T06:10:05.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:05.423+0000 7facc359d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:05.428 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:05.427+0000 7facc359d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:05.463 INFO:tasks.rados.rados.0.smithi064.stdout:498: finishing write tid 1 to smithi06431743-9 2024-03-04T06:10:05.464 INFO:tasks.rados.rados.0.smithi064.stdout:499: finishing write tid 1 to smithi06431743-34 2024-03-04T06:10:05.464 INFO:tasks.rados.rados.0.smithi064.stdout:498: finishing write tid 2 to smithi06431743-9 2024-03-04T06:10:05.464 INFO:tasks.rados.rados.0.smithi064.stdout:498: finishing write tid 3 to smithi06431743-9 2024-03-04T06:10:05.464 INFO:tasks.rados.rados.0.smithi064.stdout:498: finishing write tid 4 to smithi06431743-9 2024-03-04T06:10:05.464 INFO:tasks.rados.rados.0.smithi064.stdout:498: oid 9 updating version 0 to 154 2024-03-04T06:10:05.464 INFO:tasks.rados.rados.0.smithi064.stdout:498: oid 9 version 154 is already newer than 154 2024-03-04T06:10:05.464 INFO:tasks.rados.rados.0.smithi064.stdout:498: oid 9 version 154 is already newer than 152 2024-03-04T06:10:05.464 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 154 (ObjNum 242 snap 67 seq_num 242) dirty exists 2024-03-04T06:10:05.464 INFO:tasks.rados.rados.0.smithi064.stdout:498: left oid 9 (ObjNum 242 snap 67 seq_num 242) 2024-03-04T06:10:05.465 INFO:tasks.rados.rados.0.smithi064.stdout:499: finishing write tid 2 to smithi06431743-34 2024-03-04T06:10:05.465 INFO:tasks.rados.rados.0.smithi064.stdout:499: finishing write tid 3 to smithi06431743-34 2024-03-04T06:10:05.465 INFO:tasks.rados.rados.0.smithi064.stdout:499: finishing write tid 4 to smithi06431743-34 2024-03-04T06:10:05.465 INFO:tasks.rados.rados.0.smithi064.stdout:499: oid 34 updating version 0 to 169 2024-03-04T06:10:05.465 INFO:tasks.rados.rados.0.smithi064.stdout:499: oid 34 updating version 169 to 170 2024-03-04T06:10:05.465 INFO:tasks.rados.rados.0.smithi064.stdout:499: oid 34 updating version 170 to 171 2024-03-04T06:10:05.465 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 171 (ObjNum 243 snap 67 seq_num 243) dirty exists 2024-03-04T06:10:05.465 INFO:tasks.rados.rados.0.smithi064.stdout:499: left oid 34 (ObjNum 243 snap 67 seq_num 243) 2024-03-04T06:10:06.281 INFO:tasks.rados.rados.0.smithi064.stdout:496: done (3 left) 2024-03-04T06:10:06.281 INFO:tasks.rados.rados.0.smithi064.stdout:498: done (2 left) 2024-03-04T06:10:06.281 INFO:tasks.rados.rados.0.smithi064.stdout:499: done (1 left) 2024-03-04T06:10:06.281 INFO:tasks.rados.rados.0.smithi064.stdout:500: done (0 left) 2024-03-04T06:10:06.281 INFO:tasks.rados.rados.0.smithi064.stdout:501: write (excl) oid 44 current snap is 68 2024-03-04T06:10:06.281 INFO:tasks.rados.rados.0.smithi064.stdout:501: seq_num 244 ranges {500908=484052,1411724=689184} 2024-03-04T06:10:06.285 INFO:tasks.rados.rados.0.smithi064.stdout:501: writing smithi06431743-44 from 500908 to 984960 tid 1 2024-03-04T06:10:06.292 INFO:tasks.rados.rados.0.smithi064.stdout:501: writing smithi06431743-44 from 1411724 to 2100908 tid 2 2024-03-04T06:10:06.293 INFO:tasks.rados.rados.0.smithi064.stdout:502: write oid 49 current snap is 68 2024-03-04T06:10:06.293 INFO:tasks.rados.rados.0.smithi064.stdout:502: seq_num 245 ranges {517715=559301,1870206=247509} 2024-03-04T06:10:06.299 INFO:tasks.rados.rados.0.smithi064.stdout:502: writing smithi06431743-49 from 517715 to 1077016 tid 1 2024-03-04T06:10:06.307 INFO:tasks.rados.rados.0.smithi064.stdout:502: writing smithi06431743-49 from 1870206 to 2117715 tid 2 2024-03-04T06:10:06.307 INFO:tasks.rados.rados.0.smithi064.stdout:503: read oid 17 snap -1 2024-03-04T06:10:06.307 INFO:tasks.rados.rados.0.smithi064.stdout:503: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:10:06.307 INFO:tasks.rados.rados.0.smithi064.stdout:504: read oid 34 snap -1 2024-03-04T06:10:06.307 INFO:tasks.rados.rados.0.smithi064.stdout:504: expect (ObjNum 243 snap 67 seq_num 243) 2024-03-04T06:10:06.307 INFO:tasks.rados.rados.0.smithi064.stdout:505: read oid 21 snap -1 2024-03-04T06:10:06.307 INFO:tasks.rados.rados.0.smithi064.stdout:505: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:10:06.307 INFO:tasks.rados.rados.0.smithi064.stdout:506: delete oid 30 current snap is 68 2024-03-04T06:10:06.330 INFO:tasks.rados.rados.0.smithi064.stdout:504: expect (ObjNum 243 snap 67 seq_num 243) 2024-03-04T06:10:06.481 INFO:tasks.rados.rados.0.smithi064.stdout:505: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:10:06.569 INFO:tasks.rados.rados.0.smithi064.stdout:502: finishing write tid 1 to smithi06431743-49 2024-03-04T06:10:06.569 INFO:tasks.rados.rados.0.smithi064.stdout:502: finishing write tid 2 to smithi06431743-49 2024-03-04T06:10:06.569 INFO:tasks.rados.rados.0.smithi064.stdout:501: finishing write tid 1 to smithi06431743-44 2024-03-04T06:10:06.569 INFO:tasks.rados.rados.0.smithi064.stdout:502: finishing write tid 3 to smithi06431743-49 2024-03-04T06:10:06.570 INFO:tasks.rados.rados.0.smithi064.stdout:502: finishing write tid 4 to smithi06431743-49 2024-03-04T06:10:06.570 INFO:tasks.rados.rados.0.smithi064.stdout:502: oid 49 updating version 0 to 80 2024-03-04T06:10:06.570 INFO:tasks.rados.rados.0.smithi064.stdout:502: oid 49 version 80 is already newer than 80 2024-03-04T06:10:06.570 INFO:tasks.rados.rados.0.smithi064.stdout:502: oid 49 updating version 80 to 81 2024-03-04T06:10:06.570 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 81 (ObjNum 245 snap 68 seq_num 245) dirty exists 2024-03-04T06:10:06.570 INFO:tasks.rados.rados.0.smithi064.stdout:502: left oid 49 (ObjNum 245 snap 68 seq_num 245) 2024-03-04T06:10:06.570 INFO:tasks.rados.rados.0.smithi064.stdout:501: finishing write tid 2 to smithi06431743-44 2024-03-04T06:10:06.570 INFO:tasks.rados.rados.0.smithi064.stdout:503: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:10:06.844 INFO:tasks.rados.rados.0.smithi064.stdout:501: finishing write tid 3 to smithi06431743-44 2024-03-04T06:10:06.844 INFO:tasks.rados.rados.0.smithi064.stdout:501: finishing write tid 4 to smithi06431743-44 2024-03-04T06:10:06.844 INFO:tasks.rados.rados.0.smithi064.stdout:501: oid 44 updating version 0 to 49 2024-03-04T06:10:06.844 INFO:tasks.rados.rados.0.smithi064.stdout:501: oid 44 updating version 49 to 50 2024-03-04T06:10:06.844 INFO:tasks.rados.rados.0.smithi064.stdout:501: oid 44 version 50 is already newer than 48 2024-03-04T06:10:06.845 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 50 (ObjNum 244 snap 68 seq_num 244) dirty exists 2024-03-04T06:10:06.845 INFO:tasks.rados.rados.0.smithi064.stdout:501: left oid 44 (ObjNum 244 snap 68 seq_num 244) 2024-03-04T06:10:06.845 INFO:tasks.rados.rados.0.smithi064.stdout:501: done (5 left) 2024-03-04T06:10:06.845 INFO:tasks.rados.rados.0.smithi064.stdout:502: done (4 left) 2024-03-04T06:10:06.845 INFO:tasks.rados.rados.0.smithi064.stdout:503: done (3 left) 2024-03-04T06:10:06.845 INFO:tasks.rados.rados.0.smithi064.stdout:504: done (2 left) 2024-03-04T06:10:06.845 INFO:tasks.rados.rados.0.smithi064.stdout:505: done (1 left) 2024-03-04T06:10:06.845 INFO:tasks.rados.rados.0.smithi064.stdout:506: done (0 left) 2024-03-04T06:10:06.845 INFO:tasks.rados.rados.0.smithi064.stdout:507: write (excl) oid 7 current snap is 68 2024-03-04T06:10:06.845 INFO:tasks.rados.rados.0.smithi064.stdout:507: seq_num 247 ranges {551329=709799,1870817=280512} 2024-03-04T06:10:06.851 INFO:tasks.rados.rados.0.smithi064.stdout:507: writing smithi06431743-7 from 551329 to 1261128 tid 1 2024-03-04T06:10:06.856 INFO:tasks.rados.rados.0.smithi064.stdout:507: writing smithi06431743-7 from 1870817 to 2151329 tid 2 2024-03-04T06:10:06.856 INFO:tasks.rados.rados.0.smithi064.stdout:508: write (excl) oid 3 current snap is 68 2024-03-04T06:10:06.856 INFO:tasks.rados.rados.0.smithi064.stdout:508: seq_num 248 ranges {568136=785048,1929299=238837} 2024-03-04T06:10:06.863 INFO:tasks.rados.rados.0.smithi064.stdout:508: writing smithi06431743-3 from 568136 to 1353184 tid 1 2024-03-04T06:10:06.868 INFO:tasks.rados.rados.0.smithi064.stdout:508: writing smithi06431743-3 from 1929299 to 2168136 tid 2 2024-03-04T06:10:06.868 INFO:tasks.rados.rados.0.smithi064.stdout:509: snap_create 2024-03-04T06:10:06.880 INFO:tasks.rados.rados.0.smithi064.stdout:507: finishing write tid 1 to smithi06431743-7 2024-03-04T06:10:06.903 INFO:tasks.rados.rados.0.smithi064.stdout:507: finishing write tid 2 to smithi06431743-7 2024-03-04T06:10:06.911 INFO:tasks.rados.rados.0.smithi064.stdout:508: finishing write tid 1 to smithi06431743-3 2024-03-04T06:10:06.914 INFO:tasks.rados.rados.0.smithi064.stdout:507: finishing write tid 3 to smithi06431743-7 2024-03-04T06:10:06.916 INFO:tasks.rados.rados.0.smithi064.stdout:507: finishing write tid 4 to smithi06431743-7 2024-03-04T06:10:06.916 INFO:tasks.rados.rados.0.smithi064.stdout:507: oid 7 updating version 0 to 75 2024-03-04T06:10:06.916 INFO:tasks.rados.rados.0.smithi064.stdout:507: oid 7 version 75 is already newer than 73 2024-03-04T06:10:06.916 INFO:tasks.rados.rados.0.smithi064.stdout:507: oid 7 version 75 is already newer than 74 2024-03-04T06:10:06.916 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 75 (ObjNum 247 snap 68 seq_num 247) dirty exists 2024-03-04T06:10:06.916 INFO:tasks.rados.rados.0.smithi064.stdout:507: left oid 7 (ObjNum 247 snap 68 seq_num 247) 2024-03-04T06:10:06.924 INFO:tasks.rados.rados.0.smithi064.stdout:508: finishing write tid 2 to smithi06431743-3 2024-03-04T06:10:06.929 INFO:tasks.rados.rados.0.smithi064.stdout:508: finishing write tid 3 to smithi06431743-3 2024-03-04T06:10:06.930 INFO:tasks.rados.rados.0.smithi064.stdout:508: finishing write tid 4 to smithi06431743-3 2024-03-04T06:10:06.930 INFO:tasks.rados.rados.0.smithi064.stdout:508: oid 3 updating version 0 to 56 2024-03-04T06:10:06.930 INFO:tasks.rados.rados.0.smithi064.stdout:508: oid 3 version 56 is already newer than 55 2024-03-04T06:10:06.931 INFO:tasks.rados.rados.0.smithi064.stdout:508: oid 3 updating version 56 to 57 2024-03-04T06:10:06.931 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 57 (ObjNum 248 snap 68 seq_num 248) dirty exists 2024-03-04T06:10:06.931 INFO:tasks.rados.rados.0.smithi064.stdout:508: left oid 3 (ObjNum 248 snap 68 seq_num 248) 2024-03-04T06:10:07.285 INFO:tasks.rados.rados.0.smithi064.stdout:507: done (2 left) 2024-03-04T06:10:07.285 INFO:tasks.rados.rados.0.smithi064.stdout:508: done (1 left) 2024-03-04T06:10:07.285 INFO:tasks.rados.rados.0.smithi064.stdout:509: done (0 left) 2024-03-04T06:10:07.285 INFO:tasks.rados.rados.0.smithi064.stdout:510: write oid 18 current snap is 69 2024-03-04T06:10:07.286 INFO:tasks.rados.rados.0.smithi064.stdout:510: seq_num 249 ranges {584943=460297,1587781=597162} 2024-03-04T06:10:07.291 INFO:tasks.rados.rados.0.smithi064.stdout:510: writing smithi06431743-18 from 584943 to 1045240 tid 1 2024-03-04T06:10:07.297 INFO:tasks.rados.rados.0.smithi064.stdout:510: writing smithi06431743-18 from 1587781 to 2184943 tid 2 2024-03-04T06:10:07.297 INFO:tasks.rados.rados.0.smithi064.stdout:511: read oid 47 snap -1 2024-03-04T06:10:07.297 INFO:tasks.rados.rados.0.smithi064.stdout:511: expect (ObjNum 192 snap 38 seq_num 192) 2024-03-04T06:10:07.297 INFO:tasks.rados.rados.0.smithi064.stdout:512: snap_remove snap 62 2024-03-04T06:10:07.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:07.506 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:10:07.506 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:10:07.506 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:10:07.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:10:07.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:07.605+0000 7f9ce8f14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:07.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:07.609+0000 7f9ce8f14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:07.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:07.650+0000 7f6c7e75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:07.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:07.657+0000 7f6c7e75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:07.670 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:07.670 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:07.670 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:07.670 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:07.671 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:07.671 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:07.671 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:07.671 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:07.671 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:07.671 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:07.671 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:07.671 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:07.673 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:07.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:10:07.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:07.777+0000 7f46d38ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:07.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:07.784+0000 7f46d38ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:07.798 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:07.799 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:07.800 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:07.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:10:07.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:07.903+0000 7f6fc319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:07.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:07.910+0000 7f6fc319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:07.924 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:07.925 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:07.925 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:07.927 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:07.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:10:08.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:08.030+0000 7f8b1dcd4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:08.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:08.038+0000 7f8b1dcd4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:08.051 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:08.051 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:08.051 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:08.051 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:08.051 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:08.051 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:08.051 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:08.051 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:08.052 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:08.052 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:08.052 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:08.052 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:08.053 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:08.054 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:08.189 INFO:tasks.rados.rados.0.smithi064.stdout:512: done (2 left) 2024-03-04T06:10:08.189 INFO:tasks.rados.rados.0.smithi064.stdout:513: read oid 41 snap -1 2024-03-04T06:10:08.189 INFO:tasks.rados.rados.0.smithi064.stdout:513: expect (ObjNum 240 snap 64 seq_num 240) 2024-03-04T06:10:08.189 INFO:tasks.rados.rados.0.smithi064.stdout:514: delete oid 31 current snap is 69 2024-03-04T06:10:08.189 INFO:tasks.rados.rados.0.smithi064.stdout:511: expect (ObjNum 192 snap 38 seq_num 192) 2024-03-04T06:10:08.362 INFO:tasks.rados.rados.0.smithi064.stdout:510: finishing write tid 1 to smithi06431743-18 2024-03-04T06:10:08.362 INFO:tasks.rados.rados.0.smithi064.stdout:510: finishing write tid 2 to smithi06431743-18 2024-03-04T06:10:08.362 INFO:tasks.rados.rados.0.smithi064.stdout:510: finishing write tid 3 to smithi06431743-18 2024-03-04T06:10:08.362 INFO:tasks.rados.rados.0.smithi064.stdout:510: finishing write tid 4 to smithi06431743-18 2024-03-04T06:10:08.362 INFO:tasks.rados.rados.0.smithi064.stdout:510: oid 18 updating version 0 to 58 2024-03-04T06:10:08.362 INFO:tasks.rados.rados.0.smithi064.stdout:510: oid 18 version 58 is already newer than 57 2024-03-04T06:10:08.362 INFO:tasks.rados.rados.0.smithi064.stdout:510: oid 18 updating version 58 to 59 2024-03-04T06:10:08.362 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 59 (ObjNum 249 snap 69 seq_num 249) dirty exists 2024-03-04T06:10:08.362 INFO:tasks.rados.rados.0.smithi064.stdout:510: left oid 18 (ObjNum 249 snap 69 seq_num 249) 2024-03-04T06:10:08.363 INFO:tasks.rados.rados.0.smithi064.stdout:510: done (3 left) 2024-03-04T06:10:08.363 INFO:tasks.rados.rados.0.smithi064.stdout:511: done (2 left) 2024-03-04T06:10:08.363 INFO:tasks.rados.rados.0.smithi064.stdout:514: done (1 left) 2024-03-04T06:10:08.363 INFO:tasks.rados.rados.0.smithi064.stdout:515: write oid 40 current snap is 69 2024-03-04T06:10:08.363 INFO:tasks.rados.rados.0.smithi064.stdout:513: expect (ObjNum 240 snap 64 seq_num 240) 2024-03-04T06:10:08.476 INFO:tasks.rados.rados.0.smithi064.stdout:515: seq_num 251 ranges {618557=727148,2104745=113812} 2024-03-04T06:10:08.483 INFO:tasks.rados.rados.0.smithi064.stdout:515: writing smithi06431743-40 from 618557 to 1345705 tid 1 2024-03-04T06:10:08.488 INFO:tasks.rados.rados.0.smithi064.stdout:515: writing smithi06431743-40 from 2104745 to 2218557 tid 2 2024-03-04T06:10:08.488 INFO:tasks.rados.rados.0.smithi064.stdout:513: done (1 left) 2024-03-04T06:10:08.488 INFO:tasks.rados.rados.0.smithi064.stdout:516: snap_create 2024-03-04T06:10:08.502 INFO:tasks.rados.rados.0.smithi064.stdout:515: finishing write tid 1 to smithi06431743-40 2024-03-04T06:10:08.512 INFO:tasks.rados.rados.0.smithi064.stdout:515: finishing write tid 2 to smithi06431743-40 2024-03-04T06:10:08.516 INFO:tasks.rados.rados.0.smithi064.stdout:515: finishing write tid 3 to smithi06431743-40 2024-03-04T06:10:08.517 INFO:tasks.rados.rados.0.smithi064.stdout:515: finishing write tid 4 to smithi06431743-40 2024-03-04T06:10:08.517 INFO:tasks.rados.rados.0.smithi064.stdout:515: oid 40 updating version 0 to 79 2024-03-04T06:10:08.517 INFO:tasks.rados.rados.0.smithi064.stdout:515: oid 40 updating version 79 to 80 2024-03-04T06:10:08.517 INFO:tasks.rados.rados.0.smithi064.stdout:515: oid 40 version 80 is already newer than 78 2024-03-04T06:10:08.517 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 80 (ObjNum 251 snap 69 seq_num 251) dirty exists 2024-03-04T06:10:08.517 INFO:tasks.rados.rados.0.smithi064.stdout:515: left oid 40 (ObjNum 251 snap 69 seq_num 251) 2024-03-04T06:10:09.295 INFO:tasks.rados.rados.0.smithi064.stdout:515: done (1 left) 2024-03-04T06:10:09.295 INFO:tasks.rados.rados.0.smithi064.stdout:516: done (0 left) 2024-03-04T06:10:09.295 INFO:tasks.rados.rados.0.smithi064.stdout:517: snap_remove snap 60 2024-03-04T06:10:09.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:09.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:09.793+0000 7f344a51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:09.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:09.797+0000 7f344a51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:10.294 INFO:tasks.rados.rados.0.smithi064.stdout:517: done (0 left) 2024-03-04T06:10:10.294 INFO:tasks.rados.rados.0.smithi064.stdout:518: snap_remove snap 39 2024-03-04T06:10:11.293 INFO:tasks.rados.rados.0.smithi064.stdout:518: done (0 left) 2024-03-04T06:10:11.293 INFO:tasks.rados.rados.0.smithi064.stdout:519: write (excl) oid 22 current snap is 70 2024-03-04T06:10:11.293 INFO:tasks.rados.rados.0.smithi064.stdout:519: seq_num 252 ranges {635364=402397,1763227=472137} 2024-03-04T06:10:11.298 INFO:tasks.rados.rados.0.smithi064.stdout:519: writing smithi06431743-22 from 635364 to 1037761 tid 1 2024-03-04T06:10:11.305 INFO:tasks.rados.rados.0.smithi064.stdout:519: writing smithi06431743-22 from 1763227 to 2235364 tid 2 2024-03-04T06:10:11.306 INFO:tasks.rados.rados.0.smithi064.stdout:520: read oid 23 snap 50 2024-03-04T06:10:11.306 INFO:tasks.rados.rados.0.smithi064.stdout:520: expect (ObjNum 204 snap 44 seq_num 204) 2024-03-04T06:10:11.306 INFO:tasks.rados.rados.0.smithi064.stdout:521: read oid 16 snap -1 2024-03-04T06:10:11.306 INFO:tasks.rados.rados.0.smithi064.stdout:521: expect (ObjNum 175 snap 33 seq_num 175) 2024-03-04T06:10:11.306 INFO:tasks.rados.rados.0.smithi064.stdout:522: read oid 1 snap -1 2024-03-04T06:10:11.306 INFO:tasks.rados.rados.0.smithi064.stdout:522: expect (ObjNum 132 snap 15 seq_num 132) 2024-03-04T06:10:11.306 INFO:tasks.rados.rados.0.smithi064.stdout:523: delete oid 45 current snap is 70 2024-03-04T06:10:11.372 INFO:tasks.rados.rados.0.smithi064.stdout:520: expect (ObjNum 204 snap 44 seq_num 204) 2024-03-04T06:10:11.505 INFO:tasks.rados.rados.0.smithi064.stdout:522: expect (ObjNum 132 snap 15 seq_num 132) 2024-03-04T06:10:11.616 INFO:tasks.rados.rados.0.smithi064.stdout:521: expect (ObjNum 175 snap 33 seq_num 175) 2024-03-04T06:10:11.728 INFO:tasks.rados.rados.0.smithi064.stdout:519: finishing write tid 1 to smithi06431743-22 2024-03-04T06:10:11.728 INFO:tasks.rados.rados.0.smithi064.stdout:519: finishing write tid 2 to smithi06431743-22 2024-03-04T06:10:11.728 INFO:tasks.rados.rados.0.smithi064.stdout:519: finishing write tid 3 to smithi06431743-22 2024-03-04T06:10:11.728 INFO:tasks.rados.rados.0.smithi064.stdout:519: finishing write tid 4 to smithi06431743-22 2024-03-04T06:10:11.728 INFO:tasks.rados.rados.0.smithi064.stdout:519: oid 22 updating version 0 to 162 2024-03-04T06:10:11.728 INFO:tasks.rados.rados.0.smithi064.stdout:519: oid 22 updating version 162 to 163 2024-03-04T06:10:11.728 INFO:tasks.rados.rados.0.smithi064.stdout:519: oid 22 updating version 163 to 164 2024-03-04T06:10:11.729 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 164 (ObjNum 252 snap 70 seq_num 252) dirty exists 2024-03-04T06:10:11.729 INFO:tasks.rados.rados.0.smithi064.stdout:519: left oid 22 (ObjNum 252 snap 70 seq_num 252) 2024-03-04T06:10:11.729 INFO:tasks.rados.rados.0.smithi064.stdout:519: done (4 left) 2024-03-04T06:10:11.729 INFO:tasks.rados.rados.0.smithi064.stdout:520: done (3 left) 2024-03-04T06:10:11.729 INFO:tasks.rados.rados.0.smithi064.stdout:521: done (2 left) 2024-03-04T06:10:11.729 INFO:tasks.rados.rados.0.smithi064.stdout:522: done (1 left) 2024-03-04T06:10:11.729 INFO:tasks.rados.rados.0.smithi064.stdout:523: done (0 left) 2024-03-04T06:10:11.729 INFO:tasks.rados.rados.0.smithi064.stdout:524: write (excl) oid 18 current snap is 70 2024-03-04T06:10:11.729 INFO:tasks.rados.rados.0.smithi064.stdout:524: seq_num 254 ranges {668978=552895,1763838=505140} 2024-03-04T06:10:11.734 INFO:tasks.rados.rados.0.smithi064.stdout:524: writing smithi06431743-18 from 668978 to 1221873 tid 1 2024-03-04T06:10:11.739 INFO:tasks.rados.rados.0.smithi064.stdout:524: writing smithi06431743-18 from 1763838 to 2268978 tid 2 2024-03-04T06:10:11.739 INFO:tasks.rados.rados.0.smithi064.stdout:525: snap_remove snap 65 2024-03-04T06:10:11.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:11.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:11.971+0000 7f15e139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:11.977 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:11.976+0000 7f15e139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:12.314 INFO:tasks.rados.rados.0.smithi064.stdout:525: done (1 left) 2024-03-04T06:10:12.314 INFO:tasks.rados.rados.0.smithi064.stdout:526: read oid 41 snap 66 2024-03-04T06:10:12.314 INFO:tasks.rados.rados.0.smithi064.stdout:526: expect (ObjNum 240 snap 64 seq_num 240) 2024-03-04T06:10:12.314 INFO:tasks.rados.rados.0.smithi064.stdout:527: read oid 15 snap -1 2024-03-04T06:10:12.314 INFO:tasks.rados.rados.0.smithi064.stdout:527: expect (ObjNum 196 snap 40 seq_num 196) 2024-03-04T06:10:12.314 INFO:tasks.rados.rados.0.smithi064.stdout:528: read oid 7 snap 46 2024-03-04T06:10:12.314 INFO:tasks.rados.rados.0.smithi064.stdout:528: expect (ObjNum 166 snap 26 seq_num 166) 2024-03-04T06:10:12.314 INFO:tasks.rados.rados.0.smithi064.stdout:529: write oid 26 current snap is 70 2024-03-04T06:10:12.315 INFO:tasks.rados.rados.0.smithi064.stdout:524: finishing write tid 1 to smithi06431743-18 2024-03-04T06:10:12.315 INFO:tasks.rados.rados.0.smithi064.stdout:524: finishing write tid 2 to smithi06431743-18 2024-03-04T06:10:12.315 INFO:tasks.rados.rados.0.smithi064.stdout:529: seq_num 255 ranges {685785=628144,1822320=463465} 2024-03-04T06:10:12.322 INFO:tasks.rados.rados.0.smithi064.stdout:529: writing smithi06431743-26 from 685785 to 1313929 tid 1 2024-03-04T06:10:12.329 INFO:tasks.rados.rados.0.smithi064.stdout:529: writing smithi06431743-26 from 1822320 to 2285785 tid 2 2024-03-04T06:10:12.329 INFO:tasks.rados.rados.0.smithi064.stdout:530: snap_remove snap 64 2024-03-04T06:10:13.054 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:10:13.054 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:10:13.055 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:10:13.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:10:13.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:13.162+0000 7ff2ade6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:13.170 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:13.170+0000 7ff2ade6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:13.183 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:13.183 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:13.183 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:13.183 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:13.183 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:13.183 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:13.183 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:13.183 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:13.183 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:13.183 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:13.184 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:13.184 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:13.186 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:13.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:10:13.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:13.289+0000 7f615486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:13.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:13.296+0000 7f615486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:13.310 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:13.311 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:13.311 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:13.313 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:13.313 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:13.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:10:13.314 INFO:tasks.rados.rados.0.smithi064.stdout:530: done (5 left) 2024-03-04T06:10:13.315 INFO:tasks.rados.rados.0.smithi064.stdout:531: write oid 9 current snap is 70 2024-03-04T06:10:13.315 INFO:tasks.rados.rados.0.smithi064.stdout:524: finishing write tid 3 to smithi06431743-18 2024-03-04T06:10:13.315 INFO:tasks.rados.rados.0.smithi064.stdout:531: seq_num 256 ranges {702592=703393,1880802=421790} 2024-03-04T06:10:13.322 INFO:tasks.rados.rados.0.smithi064.stdout:531: writing smithi06431743-9 from 702592 to 1405985 tid 1 2024-03-04T06:10:13.327 INFO:tasks.rados.rados.0.smithi064.stdout:531: writing smithi06431743-9 from 1880802 to 2302592 tid 2 2024-03-04T06:10:13.327 INFO:tasks.rados.rados.0.smithi064.stdout:532: read oid 48 snap 54 2024-03-04T06:10:13.328 INFO:tasks.rados.rados.0.smithi064.stdout:532: expect (ObjNum 216 snap 47 seq_num 216) 2024-03-04T06:10:13.328 INFO:tasks.rados.rados.0.smithi064.stdout:533: delete oid 47 current snap is 70 2024-03-04T06:10:13.328 INFO:tasks.rados.rados.0.smithi064.stdout:524: finishing write tid 4 to smithi06431743-18 2024-03-04T06:10:13.328 INFO:tasks.rados.rados.0.smithi064.stdout:524: oid 18 updating version 0 to 65 2024-03-04T06:10:13.328 INFO:tasks.rados.rados.0.smithi064.stdout:524: oid 18 updating version 65 to 66 2024-03-04T06:10:13.328 INFO:tasks.rados.rados.0.smithi064.stdout:524: oid 18 updating version 66 to 67 2024-03-04T06:10:13.328 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 67 (ObjNum 254 snap 70 seq_num 254) dirty exists 2024-03-04T06:10:13.328 INFO:tasks.rados.rados.0.smithi064.stdout:524: left oid 18 (ObjNum 254 snap 70 seq_num 254) 2024-03-04T06:10:13.328 INFO:tasks.rados.rados.0.smithi064.stdout:526: expect (ObjNum 240 snap 64 seq_num 240) 2024-03-04T06:10:13.443 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:13.438+0000 7f33ecedd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:13.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:13.446+0000 7f33ecedd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:13.451 INFO:tasks.rados.rados.0.smithi064.stdout:527: expect (ObjNum 196 snap 40 seq_num 196) 2024-03-04T06:10:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:13.460 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:13.461 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:13.463 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:13.464 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:13.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:10:13.576 INFO:tasks.rados.rados.0.smithi064.stdout:528: expect (ObjNum 166 snap 26 seq_num 166) 2024-03-04T06:10:13.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:13.607+0000 7fe6de46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:13.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:13.615+0000 7fe6de46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:13.629 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:13.630 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:13.632 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:13.793 INFO:tasks.rados.rados.0.smithi064.stdout:529: finishing write tid 1 to smithi06431743-26 2024-03-04T06:10:13.794 INFO:tasks.rados.rados.0.smithi064.stdout:529: finishing write tid 2 to smithi06431743-26 2024-03-04T06:10:13.794 INFO:tasks.rados.rados.0.smithi064.stdout:529: finishing write tid 3 to smithi06431743-26 2024-03-04T06:10:13.794 INFO:tasks.rados.rados.0.smithi064.stdout:529: finishing write tid 4 to smithi06431743-26 2024-03-04T06:10:13.794 INFO:tasks.rados.rados.0.smithi064.stdout:529: oid 26 updating version 0 to 182 2024-03-04T06:10:13.794 INFO:tasks.rados.rados.0.smithi064.stdout:529: oid 26 updating version 182 to 183 2024-03-04T06:10:13.794 INFO:tasks.rados.rados.0.smithi064.stdout:529: oid 26 version 183 is already newer than 182 2024-03-04T06:10:13.794 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 183 (ObjNum 255 snap 70 seq_num 255) dirty exists 2024-03-04T06:10:13.794 INFO:tasks.rados.rados.0.smithi064.stdout:529: left oid 26 (ObjNum 255 snap 70 seq_num 255) 2024-03-04T06:10:13.794 INFO:tasks.rados.rados.0.smithi064.stdout:524: done (7 left) 2024-03-04T06:10:13.794 INFO:tasks.rados.rados.0.smithi064.stdout:526: done (6 left) 2024-03-04T06:10:13.795 INFO:tasks.rados.rados.0.smithi064.stdout:527: done (5 left) 2024-03-04T06:10:13.795 INFO:tasks.rados.rados.0.smithi064.stdout:528: done (4 left) 2024-03-04T06:10:13.795 INFO:tasks.rados.rados.0.smithi064.stdout:529: done (3 left) 2024-03-04T06:10:13.795 INFO:tasks.rados.rados.0.smithi064.stdout:533: done (2 left) 2024-03-04T06:10:13.795 INFO:tasks.rados.rados.0.smithi064.stdout:534: write (excl) oid 26 current snap is 70 2024-03-04T06:10:13.795 INFO:tasks.rados.rados.0.smithi064.stdout:531: finishing write tid 1 to smithi06431743-9 2024-03-04T06:10:13.795 INFO:tasks.rados.rados.0.smithi064.stdout:532: expect (ObjNum 216 snap 47 seq_num 216) 2024-03-04T06:10:13.979 INFO:tasks.rados.rados.0.smithi064.stdout:531: finishing write tid 2 to smithi06431743-9 2024-03-04T06:10:13.979 INFO:tasks.rados.rados.0.smithi064.stdout:531: finishing write tid 3 to smithi06431743-9 2024-03-04T06:10:13.980 INFO:tasks.rados.rados.0.smithi064.stdout:531: finishing write tid 4 to smithi06431743-9 2024-03-04T06:10:13.980 INFO:tasks.rados.rados.0.smithi064.stdout:531: oid 9 updating version 0 to 176 2024-03-04T06:10:13.980 INFO:tasks.rados.rados.0.smithi064.stdout:531: oid 9 version 176 is already newer than 174 2024-03-04T06:10:13.980 INFO:tasks.rados.rados.0.smithi064.stdout:531: oid 9 version 176 is already newer than 175 2024-03-04T06:10:13.980 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 176 (ObjNum 256 snap 70 seq_num 256) dirty exists 2024-03-04T06:10:13.980 INFO:tasks.rados.rados.0.smithi064.stdout:531: left oid 9 (ObjNum 256 snap 70 seq_num 256) 2024-03-04T06:10:13.980 INFO:tasks.rados.rados.0.smithi064.stdout:534: seq_num 258 ranges {736206=453891,1881413=454793} 2024-03-04T06:10:13.985 INFO:tasks.rados.rados.0.smithi064.stdout:534: writing smithi06431743-26 from 736206 to 1190097 tid 1 2024-03-04T06:10:13.992 INFO:tasks.rados.rados.0.smithi064.stdout:534: writing smithi06431743-26 from 1881413 to 2336206 tid 2 2024-03-04T06:10:13.992 INFO:tasks.rados.rados.0.smithi064.stdout:531: done (2 left) 2024-03-04T06:10:13.992 INFO:tasks.rados.rados.0.smithi064.stdout:532: done (1 left) 2024-03-04T06:10:13.992 INFO:tasks.rados.rados.0.smithi064.stdout:535: write (excl) oid 6 current snap is 70 2024-03-04T06:10:13.992 INFO:tasks.rados.rados.0.smithi064.stdout:535: seq_num 259 ranges {753013=645493,2056248=296765} 2024-03-04T06:10:13.999 INFO:tasks.rados.rados.0.smithi064.stdout:535: writing smithi06431743-6 from 753013 to 1398506 tid 1 2024-03-04T06:10:14.005 INFO:tasks.rados.rados.0.smithi064.stdout:535: writing smithi06431743-6 from 2056248 to 2353013 tid 2 2024-03-04T06:10:14.005 INFO:tasks.rados.rados.0.smithi064.stdout:536: delete oid 31 current snap is 70 2024-03-04T06:10:14.005 INFO:tasks.rados.rados.0.smithi064.stdout:534: finishing write tid 1 to smithi06431743-26 2024-03-04T06:10:14.009 INFO:tasks.rados.rados.0.smithi064.stdout:536: done (2 left) 2024-03-04T06:10:14.010 INFO:tasks.rados.rados.0.smithi064.stdout:537: read oid 13 snap -1 2024-03-04T06:10:14.010 INFO:tasks.rados.rados.0.smithi064.stdout:537: expect deleted 2024-03-04T06:10:14.010 INFO:tasks.rados.rados.0.smithi064.stdout:538: write oid 5 current snap is 70 2024-03-04T06:10:14.010 INFO:tasks.rados.rados.0.smithi064.stdout:538: seq_num 261 ranges {786627=795991,2173212=213415} 2024-03-04T06:10:14.018 INFO:tasks.rados.rados.0.smithi064.stdout:538: writing smithi06431743-5 from 786627 to 1582618 tid 1 2024-03-04T06:10:14.023 INFO:tasks.rados.rados.0.smithi064.stdout:538: writing smithi06431743-5 from 2173212 to 2386627 tid 2 2024-03-04T06:10:14.023 INFO:tasks.rados.rados.0.smithi064.stdout:539: snap_remove snap 54 2024-03-04T06:10:14.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:14.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:14.151+0000 7fe5a50ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:14.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:14.156+0000 7fe5a50ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:14.322 INFO:tasks.rados.rados.0.smithi064.stdout:539: done (4 left) 2024-03-04T06:10:14.322 INFO:tasks.rados.rados.0.smithi064.stdout:540: delete oid 16 current snap is 70 2024-03-04T06:10:14.322 INFO:tasks.rados.rados.0.smithi064.stdout:534: finishing write tid 2 to smithi06431743-26 2024-03-04T06:10:14.322 INFO:tasks.rados.rados.0.smithi064.stdout:538: finishing write tid 1 to smithi06431743-5 2024-03-04T06:10:14.322 INFO:tasks.rados.rados.0.smithi064.stdout:534: finishing write tid 3 to smithi06431743-26 2024-03-04T06:10:14.322 INFO:tasks.rados.rados.0.smithi064.stdout:534: finishing write tid 4 to smithi06431743-26 2024-03-04T06:10:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:534: oid 26 updating version 0 to 189 2024-03-04T06:10:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:534: oid 26 updating version 189 to 190 2024-03-04T06:10:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:534: oid 26 version 190 is already newer than 188 2024-03-04T06:10:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 190 (ObjNum 258 snap 70 seq_num 258) dirty exists 2024-03-04T06:10:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:534: left oid 26 (ObjNum 258 snap 70 seq_num 258) 2024-03-04T06:10:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:538: finishing write tid 2 to smithi06431743-5 2024-03-04T06:10:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:535: finishing write tid 1 to smithi06431743-6 2024-03-04T06:10:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:538: finishing write tid 3 to smithi06431743-5 2024-03-04T06:10:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:538: finishing write tid 4 to smithi06431743-5 2024-03-04T06:10:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:538: oid 5 updating version 0 to 203 2024-03-04T06:10:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:538: oid 5 version 203 is already newer than 202 2024-03-04T06:10:14.324 INFO:tasks.rados.rados.0.smithi064.stdout:538: oid 5 version 203 is already newer than 201 2024-03-04T06:10:14.324 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 203 (ObjNum 261 snap 70 seq_num 261) dirty exists 2024-03-04T06:10:14.324 INFO:tasks.rados.rados.0.smithi064.stdout:538: left oid 5 (ObjNum 261 snap 70 seq_num 261) 2024-03-04T06:10:14.324 INFO:tasks.rados.rados.0.smithi064.stdout:535: finishing write tid 2 to smithi06431743-6 2024-03-04T06:10:14.324 INFO:tasks.rados.rados.0.smithi064.stdout:535: finishing write tid 3 to smithi06431743-6 2024-03-04T06:10:14.324 INFO:tasks.rados.rados.0.smithi064.stdout:535: finishing write tid 4 to smithi06431743-6 2024-03-04T06:10:14.324 INFO:tasks.rados.rados.0.smithi064.stdout:535: oid 6 updating version 0 to 53 2024-03-04T06:10:14.324 INFO:tasks.rados.rados.0.smithi064.stdout:535: oid 6 version 53 is already newer than 51 2024-03-04T06:10:14.324 INFO:tasks.rados.rados.0.smithi064.stdout:535: oid 6 version 53 is already newer than 53 2024-03-04T06:10:14.324 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 53 (ObjNum 259 snap 70 seq_num 259) dirty exists 2024-03-04T06:10:14.325 INFO:tasks.rados.rados.0.smithi064.stdout:535: left oid 6 (ObjNum 259 snap 70 seq_num 259) 2024-03-04T06:10:14.414 INFO:tasks.rados.rados.0.smithi064.stdout:534: done (4 left) 2024-03-04T06:10:14.414 INFO:tasks.rados.rados.0.smithi064.stdout:535: done (3 left) 2024-03-04T06:10:14.414 INFO:tasks.rados.rados.0.smithi064.stdout:537: done (2 left) 2024-03-04T06:10:14.414 INFO:tasks.rados.rados.0.smithi064.stdout:538: done (1 left) 2024-03-04T06:10:14.414 INFO:tasks.rados.rados.0.smithi064.stdout:540: done (0 left) 2024-03-04T06:10:14.414 INFO:tasks.rados.rados.0.smithi064.stdout:541: write (excl) oid 44 current snap is 70 2024-03-04T06:10:14.414 INFO:tasks.rados.rados.0.smithi064.stdout:541: seq_num 263 ranges {420241=546489,1373823=544414} 2024-03-04T06:10:14.419 INFO:tasks.rados.rados.0.smithi064.stdout:541: writing smithi06431743-44 from 420241 to 966730 tid 1 2024-03-04T06:10:14.427 INFO:tasks.rados.rados.0.smithi064.stdout:541: writing smithi06431743-44 from 1373823 to 1918237 tid 2 2024-03-04T06:10:14.427 INFO:tasks.rados.rados.0.smithi064.stdout:542: write oid 31 current snap is 70 2024-03-04T06:10:14.427 INFO:tasks.rados.rados.0.smithi064.stdout:542: seq_num 264 ranges {437048=621738,1832305=404425} 2024-03-04T06:10:14.433 INFO:tasks.rados.rados.0.smithi064.stdout:542: writing smithi06431743-31 from 437048 to 1058786 tid 1 2024-03-04T06:10:14.440 INFO:tasks.rados.rados.0.smithi064.stdout:542: writing smithi06431743-31 from 1832305 to 2236730 tid 2 2024-03-04T06:10:14.440 INFO:tasks.rados.rados.0.smithi064.stdout:543: write oid 21 current snap is 70 2024-03-04T06:10:14.440 INFO:tasks.rados.rados.0.smithi064.stdout:543: seq_num 265 ranges {453855=696987,1890787=548083} 2024-03-04T06:10:14.446 INFO:tasks.rados.rados.0.smithi064.stdout:543: writing smithi06431743-21 from 453855 to 1150842 tid 1 2024-03-04T06:10:14.454 INFO:tasks.rados.rados.0.smithi064.stdout:543: writing smithi06431743-21 from 1890787 to 2438870 tid 2 2024-03-04T06:10:14.454 INFO:tasks.rados.rados.0.smithi064.stdout:544: delete oid 45 current snap is 70 2024-03-04T06:10:14.454 INFO:tasks.rados.rados.0.smithi064.stdout:542: finishing write tid 1 to smithi06431743-31 2024-03-04T06:10:14.462 INFO:tasks.rados.rados.0.smithi064.stdout:542: finishing write tid 2 to smithi06431743-31 2024-03-04T06:10:14.464 INFO:tasks.rados.rados.0.smithi064.stdout:544: done (3 left) 2024-03-04T06:10:14.465 INFO:tasks.rados.rados.0.smithi064.stdout:545: write oid 7 current snap is 70 2024-03-04T06:10:14.465 INFO:tasks.rados.rados.0.smithi064.stdout:545: seq_num 267 ranges {487469=563838,1607751=551752} 2024-03-04T06:10:14.470 INFO:tasks.rados.rados.0.smithi064.stdout:545: writing smithi06431743-7 from 487469 to 1051307 tid 1 2024-03-04T06:10:14.478 INFO:tasks.rados.rados.0.smithi064.stdout:545: writing smithi06431743-7 from 1607751 to 2159503 tid 2 2024-03-04T06:10:14.478 INFO:tasks.rados.rados.0.smithi064.stdout:546: read oid 47 snap 63 2024-03-04T06:10:14.478 INFO:tasks.rados.rados.0.smithi064.stdout:546: expect (ObjNum 192 snap 38 seq_num 192) 2024-03-04T06:10:14.478 INFO:tasks.rados.rados.0.smithi064.stdout:547: write oid 40 current snap is 70 2024-03-04T06:10:14.478 INFO:tasks.rados.rados.0.smithi064.stdout:541: finishing write tid 1 to smithi06431743-44 2024-03-04T06:10:14.478 INFO:tasks.rados.rados.0.smithi064.stdout:543: finishing write tid 1 to smithi06431743-21 2024-03-04T06:10:14.478 INFO:tasks.rados.rados.0.smithi064.stdout:542: finishing write tid 3 to smithi06431743-31 2024-03-04T06:10:14.478 INFO:tasks.rados.rados.0.smithi064.stdout:542: finishing write tid 4 to smithi06431743-31 2024-03-04T06:10:14.479 INFO:tasks.rados.rados.0.smithi064.stdout:542: oid 31 updating version 0 to 45 2024-03-04T06:10:14.479 INFO:tasks.rados.rados.0.smithi064.stdout:542: oid 31 version 45 is already newer than 44 2024-03-04T06:10:14.479 INFO:tasks.rados.rados.0.smithi064.stdout:542: oid 31 updating version 45 to 46 2024-03-04T06:10:14.479 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 46 (ObjNum 264 snap 70 seq_num 264) dirty exists 2024-03-04T06:10:14.479 INFO:tasks.rados.rados.0.smithi064.stdout:542: left oid 31 (ObjNum 264 snap 70 seq_num 264) 2024-03-04T06:10:14.479 INFO:tasks.rados.rados.0.smithi064.stdout:547: seq_num 268 ranges {504276=639087,1666233=695410} 2024-03-04T06:10:14.483 INFO:tasks.rados.rados.0.smithi064.stdout:547: writing smithi06431743-40 from 504276 to 1143363 tid 1 2024-03-04T06:10:14.490 INFO:tasks.rados.rados.0.smithi064.stdout:547: writing smithi06431743-40 from 1666233 to 2361643 tid 2 2024-03-04T06:10:14.490 INFO:tasks.rados.rados.0.smithi064.stdout:542: done (5 left) 2024-03-04T06:10:14.491 INFO:tasks.rados.rados.0.smithi064.stdout:548: snap_remove snap 66 2024-03-04T06:10:15.324 INFO:tasks.rados.rados.0.smithi064.stdout:548: done (5 left) 2024-03-04T06:10:15.324 INFO:tasks.rados.rados.0.smithi064.stdout:549: write oid 24 current snap is 70 2024-03-04T06:10:15.325 INFO:tasks.rados.rados.0.smithi064.stdout:549: seq_num 269 ranges {521083=714336,1724715=555421} 2024-03-04T06:10:15.331 INFO:tasks.rados.rados.0.smithi064.stdout:549: writing smithi06431743-24 from 521083 to 1235419 tid 1 2024-03-04T06:10:15.336 INFO:tasks.rados.rados.0.smithi064.stdout:549: writing smithi06431743-24 from 1724715 to 2280136 tid 2 2024-03-04T06:10:15.337 INFO:tasks.rados.rados.0.smithi064.stdout:550: write (excl) oid 30 current snap is 70 2024-03-04T06:10:15.337 INFO:tasks.rados.rados.0.smithi064.stdout:550: seq_num 270 ranges {537890=789585,1783197=699079} 2024-03-04T06:10:15.343 INFO:tasks.rados.rados.0.smithi064.stdout:550: writing smithi06431743-30 from 537890 to 1327475 tid 1 2024-03-04T06:10:15.350 INFO:tasks.rados.rados.0.smithi064.stdout:550: writing smithi06431743-30 from 1783197 to 2482276 tid 2 2024-03-04T06:10:15.350 INFO:tasks.rados.rados.0.smithi064.stdout:551: snap_remove snap 46 2024-03-04T06:10:16.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:16.332 INFO:tasks.rados.rados.0.smithi064.stdout:551: done (7 left) 2024-03-04T06:10:16.332 INFO:tasks.rados.rados.0.smithi064.stdout:552: write (excl) oid 39 current snap is 70 2024-03-04T06:10:16.332 INFO:tasks.rados.rados.0.smithi064.stdout:541: finishing write tid 2 to smithi06431743-44 2024-03-04T06:10:16.332 INFO:tasks.rados.rados.0.smithi064.stdout:552: seq_num 271 ranges {554697=464834,1725326=559090} 2024-03-04T06:10:16.338 INFO:tasks.rados.rados.0.smithi064.stdout:552: writing smithi06431743-39 from 554697 to 1019531 tid 1 2024-03-04T06:10:16.346 INFO:tasks.rados.rados.0.smithi064.stdout:552: writing smithi06431743-39 from 1725326 to 2284416 tid 2 2024-03-04T06:10:16.346 INFO:tasks.rados.rados.0.smithi064.stdout:553: read oid 3 snap 58 2024-03-04T06:10:16.346 INFO:tasks.rados.rados.0.smithi064.stdout:553: expect (ObjNum 168 snap 26 seq_num 168) 2024-03-04T06:10:16.346 INFO:tasks.rados.rados.0.smithi064.stdout:554: read oid 22 snap -1 2024-03-04T06:10:16.346 INFO:tasks.rados.rados.0.smithi064.stdout:554: expect (ObjNum 252 snap 70 seq_num 252) 2024-03-04T06:10:16.346 INFO:tasks.rados.rados.0.smithi064.stdout:555: delete oid 12 current snap is 70 2024-03-04T06:10:16.346 INFO:tasks.rados.rados.0.smithi064.stdout:543: finishing write tid 2 to smithi06431743-21 2024-03-04T06:10:16.346 INFO:tasks.rados.rados.0.smithi064.stdout:541: finishing write tid 3 to smithi06431743-44 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:541: finishing write tid 4 to smithi06431743-44 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:541: oid 44 updating version 0 to 62 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:541: oid 44 version 62 is already newer than 60 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:541: oid 44 version 62 is already newer than 61 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 62 (ObjNum 263 snap 70 seq_num 263) dirty exists 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:541: left oid 44 (ObjNum 263 snap 70 seq_num 263) 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:545: finishing write tid 1 to smithi06431743-7 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:543: finishing write tid 3 to smithi06431743-21 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:543: finishing write tid 4 to smithi06431743-21 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:543: oid 21 updating version 0 to 59 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:543: oid 21 version 59 is already newer than 58 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:543: oid 21 version 59 is already newer than 57 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 59 (ObjNum 265 snap 70 seq_num 265) dirty exists 2024-03-04T06:10:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:543: left oid 21 (ObjNum 265 snap 70 seq_num 265) 2024-03-04T06:10:16.348 INFO:tasks.rados.rados.0.smithi064.stdout:546: expect (ObjNum 192 snap 38 seq_num 192) 2024-03-04T06:10:16.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:16.344+0000 7f172326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:16.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:16.352+0000 7f172326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:16.532 INFO:tasks.rados.rados.0.smithi064.stdout:547: finishing write tid 1 to smithi06431743-40 2024-03-04T06:10:16.532 INFO:tasks.rados.rados.0.smithi064.stdout:545: finishing write tid 2 to smithi06431743-7 2024-03-04T06:10:16.532 INFO:tasks.rados.rados.0.smithi064.stdout:545: finishing write tid 3 to smithi06431743-7 2024-03-04T06:10:16.532 INFO:tasks.rados.rados.0.smithi064.stdout:545: finishing write tid 4 to smithi06431743-7 2024-03-04T06:10:16.532 INFO:tasks.rados.rados.0.smithi064.stdout:545: oid 7 updating version 0 to 95 2024-03-04T06:10:16.532 INFO:tasks.rados.rados.0.smithi064.stdout:545: oid 7 version 95 is already newer than 93 2024-03-04T06:10:16.532 INFO:tasks.rados.rados.0.smithi064.stdout:545: oid 7 version 95 is already newer than 95 2024-03-04T06:10:16.533 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 95 (ObjNum 267 snap 70 seq_num 267) dirty exists 2024-03-04T06:10:16.533 INFO:tasks.rados.rados.0.smithi064.stdout:545: left oid 7 (ObjNum 267 snap 70 seq_num 267) 2024-03-04T06:10:16.533 INFO:tasks.rados.rados.0.smithi064.stdout:547: finishing write tid 2 to smithi06431743-40 2024-03-04T06:10:16.533 INFO:tasks.rados.rados.0.smithi064.stdout:547: finishing write tid 3 to smithi06431743-40 2024-03-04T06:10:16.533 INFO:tasks.rados.rados.0.smithi064.stdout:541: done (10 left) 2024-03-04T06:10:16.533 INFO:tasks.rados.rados.0.smithi064.stdout:543: done (9 left) 2024-03-04T06:10:16.533 INFO:tasks.rados.rados.0.smithi064.stdout:545: done (8 left) 2024-03-04T06:10:16.533 INFO:tasks.rados.rados.0.smithi064.stdout:546: done (7 left) 2024-03-04T06:10:16.533 INFO:tasks.rados.rados.0.smithi064.stdout:555: done (6 left) 2024-03-04T06:10:16.533 INFO:tasks.rados.rados.0.smithi064.stdout:556: snap_remove snap 57 2024-03-04T06:10:17.335 INFO:tasks.rados.rados.0.smithi064.stdout:556: done (6 left) 2024-03-04T06:10:17.335 INFO:tasks.rados.rados.0.smithi064.stdout:557: read oid 21 snap -1 2024-03-04T06:10:17.335 INFO:tasks.rados.rados.0.smithi064.stdout:557: expect (ObjNum 265 snap 70 seq_num 265) 2024-03-04T06:10:17.335 INFO:tasks.rados.rados.0.smithi064.stdout:558: snap_remove snap 68 2024-03-04T06:10:18.194 INFO:tasks.rados.rados.0.smithi064.stdout:558: done (7 left) 2024-03-04T06:10:18.195 INFO:tasks.rados.rados.0.smithi064.stdout:559: read oid 4 snap -1 2024-03-04T06:10:18.195 INFO:tasks.rados.rados.0.smithi064.stdout:559: expect deleted 2024-03-04T06:10:18.195 INFO:tasks.rados.rados.0.smithi064.stdout:560: write oid 38 current snap is 70 2024-03-04T06:10:18.195 INFO:tasks.rados.rados.0.smithi064.stdout:547: finishing write tid 4 to smithi06431743-40 2024-03-04T06:10:18.195 INFO:tasks.rados.rados.0.smithi064.stdout:547: oid 40 updating version 0 to 87 2024-03-04T06:10:18.195 INFO:tasks.rados.rados.0.smithi064.stdout:547: oid 40 version 87 is already newer than 86 2024-03-04T06:10:18.195 INFO:tasks.rados.rados.0.smithi064.stdout:547: oid 40 updating version 87 to 88 2024-03-04T06:10:18.195 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 88 (ObjNum 268 snap 70 seq_num 268) dirty exists 2024-03-04T06:10:18.195 INFO:tasks.rados.rados.0.smithi064.stdout:547: left oid 40 (ObjNum 268 snap 70 seq_num 268) 2024-03-04T06:10:18.196 INFO:tasks.rados.rados.0.smithi064.stdout:549: finishing write tid 1 to smithi06431743-24 2024-03-04T06:10:18.196 INFO:tasks.rados.rados.0.smithi064.stdout:549: finishing write tid 2 to smithi06431743-24 2024-03-04T06:10:18.196 INFO:tasks.rados.rados.0.smithi064.stdout:549: finishing write tid 3 to smithi06431743-24 2024-03-04T06:10:18.196 INFO:tasks.rados.rados.0.smithi064.stdout:549: finishing write tid 4 to smithi06431743-24 2024-03-04T06:10:18.196 INFO:tasks.rados.rados.0.smithi064.stdout:549: oid 24 updating version 0 to 157 2024-03-04T06:10:18.196 INFO:tasks.rados.rados.0.smithi064.stdout:549: oid 24 updating version 157 to 158 2024-03-04T06:10:18.196 INFO:tasks.rados.rados.0.smithi064.stdout:549: oid 24 updating version 158 to 159 2024-03-04T06:10:18.196 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 159 (ObjNum 269 snap 70 seq_num 269) dirty exists 2024-03-04T06:10:18.196 INFO:tasks.rados.rados.0.smithi064.stdout:549: left oid 24 (ObjNum 269 snap 70 seq_num 269) 2024-03-04T06:10:18.197 INFO:tasks.rados.rados.0.smithi064.stdout:550: finishing write tid 1 to smithi06431743-30 2024-03-04T06:10:18.197 INFO:tasks.rados.rados.0.smithi064.stdout:560: seq_num 273 ranges {588311=615332,1842290=562759} 2024-03-04T06:10:18.200 INFO:tasks.rados.rados.0.smithi064.stdout:560: writing smithi06431743-38 from 588311 to 1203643 tid 1 2024-03-04T06:10:18.207 INFO:tasks.rados.rados.0.smithi064.stdout:560: writing smithi06431743-38 from 1842290 to 2405049 tid 2 2024-03-04T06:10:18.207 INFO:tasks.rados.rados.0.smithi064.stdout:547: done (8 left) 2024-03-04T06:10:18.207 INFO:tasks.rados.rados.0.smithi064.stdout:549: done (7 left) 2024-03-04T06:10:18.207 INFO:tasks.rados.rados.0.smithi064.stdout:561: snap_remove snap 53 2024-03-04T06:10:18.512 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:18.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:18.621+0000 7fec5bb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:18.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:18.626+0000 7fec5bb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:18.633 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:10:18.633 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:10:18.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:10:18.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:18.739+0000 7f2a99e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:18.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:18.745+0000 7f2a99e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:18.936 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:10:18.937 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:10:18.948 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-03-04T06:10:18.737307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98609287,"num_objects":200,"num_object_clones":136,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":28,"num_read":1207,"num_read_kb":776486,"num_write":1102,"num_write_kb":267212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1712,"ondisk_log_size":1712,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":503468,"kb_used_data":377996,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":376983892,"statfs":{"total":386547056640,"available":386031505408,"internally_reserved":0,"allocated":387067904,"data_stored":450482290,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7344395,"num_objects":14,"num_object_clones":13,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":69,"num_read_kb":51331,"num_write":55,"num_write_kb":13774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000917"},"pg_stats":[{"pgid":"3.f","version":"146'88","reported_seq":450,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.539842+0000","last_change":"2024-03-04T06:10:14.401704+0000","last_active":"2024-03-04T06:10:14.539842+0000","last_peered":"2024-03-04T06:10:14.539842+0000","last_clean":"2024-03-04T06:10:14.539842+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:10:14.539842+0000","last_undegraded":"2024-03-04T06:10:14.539842+0000","last_fullsized":"2024-03-04T06:10:14.539842+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6235430,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":28,"num_read_kb":6458,"num_write":62,"num_write_kb":16398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"146'73","reported_seq":466,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.335317+0000","last_change":"2024-03-04T06:10:14.379213+0000","last_active":"2024-03-04T06:10:14.335317+0000","last_peered":"2024-03-04T06:10:14.335317+0000","last_clean":"2024-03-04T06:10:14.335317+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:10:14.335317+0000","last_undegraded":"2024-03-04T06:10:14.335317+0000","last_fullsized":"2024-03-04T06:10:14.335317+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2268978,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":36,"num_read_kb":21069,"num_write":44,"num_write_kb":11808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"146'120","reported_seq":481,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.424044+0000","last_change":"2024-03-04T06:10:14.424044+0000","last_active":"2024-03-04T06:10:14.424044+0000","last_peered":"2024-03-04T06:10:14.424044+0000","last_clean":"2024-03-04T06:10:14.424044+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:10:14.424044+0000","last_undegraded":"2024-03-04T06:10:14.424044+0000","last_fullsized":"2024-03-04T06:10:14.424044+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882450,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":63,"num_read_kb":35347,"num_write":69,"num_write_kb":19466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"146'82","reported_seq":465,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.511650+0000","last_change":"2024-03-04T06:10:14.412269+0000","last_active":"2024-03-04T06:10:14.511650+0000","last_peered":"2024-03-04T06:10:14.511650+0000","last_clean":"2024-03-04T06:10:14.511650+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:10:14.511650+0000","last_undegraded":"2024-03-04T06:10:14.511650+0000","last_fullsized":"2024-03-04T06:10:14.511650+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":70,"num_read_kb":62719,"num_write":54,"num_write_kb":15175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"147'48","reported_seq":438,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-04T06:10:15.398248+0000","last_change":"2024-03-04T06:10:15.398248+0000","last_active":"2024-03-04T06:10:15.398248+0000","last_peered":"2024-03-04T06:10:15.398248+0000","last_clean":"2024-03-04T06:10:15.398248+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:10:15.398248+0000","last_undegraded":"2024-03-04T06:10:15.398248+0000","last_fullsized":"2024-03-04T06:10:15.398248+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2437048,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":17039,"num_write":23,"num_write_kb":6232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"146'62","reported_seq":446,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.480589+0000","last_change":"2024-03-04T06:10:14.426191+0000","last_active":"2024-03-04T06:10:14.480589+0000","last_peered":"2024-03-04T06:10:14.480589+0000","last_clean":"2024-03-04T06:10:14.480589+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:10:14.480589+0000","last_undegraded":"2024-03-04T06:10:14.480589+0000","last_fullsized":"2024-03-04T06:10:14.480589+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8739673,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":40945,"num_write":45,"num_write_kb":13442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":385,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.391348+0000","last_change":"2024-03-04T06:10:14.397198+0000","last_active":"2024-03-04T06:10:14.391348+0000","last_peered":"2024-03-04T06:10:14.391348+0000","last_clean":"2024-03-04T06:10:14.391348+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:10:14.391348+0000","last_undegraded":"2024-03-04T06:10:14.391348+0000","last_fullsized":"2024-03-04T06:10:14.391348+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"146'95","reported_seq":466,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.388874+0000","last_change":"2024-03-04T06:10:14.420634+0000","last_active":"2024-03-04T06:10:14.388874+0000","last_peered":"2024-03-04T06:10:14.388874+0000","last_clean":"2024-03-04T06:10:14.388874+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:10:14.388874+0000","last_undegraded":"2024-03-04T06:10:14.388874+0000","last_fullsized":"2024-03-04T06:10:14.388874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5798448,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":51,"num_read_kb":39607,"num_write":57,"num_write_kb":16367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":488,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-04T06:10:15.338378+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:10:15.338378+0000","last_peered":"2024-03-04T06:10:15.338378+0000","last_clean":"2024-03-04T06:10:15.338378+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:10:15.338378+0000","last_undegraded":"2024-03-04T06:10:15.338378+0000","last_fullsized":"2024-03-04T06:10:15.338378+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":344,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.402401+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:10:14.402401+0000","last_peered":"2024-03-04T06:10:14.402401+0000","last_clean":"2024-03-04T06:10:14.402401+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:10:14.402401+0000","last_undegraded":"2024-03-04T06:10:14.402401+0000","last_fullsized":"2024-03-04T06:10:14.402401+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"146'77","reported_seq":461,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.399864+0000","last_change":"2024-03-04T06:10:14.454426+0000","last_active":"2024-03-04T06:10:14.399864+0000","last_peered":"2024-03-04T06:10:14.399864+0000","last_clean":"2024-03-04T06:10:14.399864+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:10:14.399864+0000","last_undegraded":"2024-03-04T06:10:14.399864+0000","last_fullsized":"2024-03-04T06:10:14.399864+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7630444,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":63,"num_read_kb":55280,"num_write":44,"num_write_kb":11786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":336,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.385891+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:10:14.385891+0000","last_peered":"2024-03-04T06:10:14.385891+0000","last_clean":"2024-03-04T06:10:14.385891+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:10:14.385891+0000","last_undegraded":"2024-03-04T06:10:14.385891+0000","last_fullsized":"2024-03-04T06:10:14.385891+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"147'97","reported_seq":477,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-04T06:10:15.380930+0000","last_change":"2024-03-04T06:10:15.380930+0000","last_active":"2024-03-04T06:10:15.380930+0000","last_peered":"2024-03-04T06:10:15.380930+0000","last_clean":"2024-03-04T06:10:15.380930+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:10:15.380930+0000","last_undegraded":"2024-03-04T06:10:15.380930+0000","last_fullsized":"2024-03-04T06:10:15.380930+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487469,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":66,"num_read_kb":54041,"num_write":48,"num_write_kb":13998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":344,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.387092+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:10:14.387092+0000","last_peered":"2024-03-04T06:10:14.387092+0000","last_clean":"2024-03-04T06:10:14.387092+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:10:14.387092+0000","last_undegraded":"2024-03-04T06:10:14.387092+0000","last_fullsized":"2024-03-04T06:10:14.387092+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"147'198","reported_seq":588,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-04T06:10:15.399635+0000","last_change":"2024-03-04T06:10:15.399635+0000","last_active":"2024-03-04T06:10:15.399635+0000","last_peered":"2024-03-04T06:10:15.399635+0000","last_clean":"2024-03-04T06:10:15.399635+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:10:15.399635+0000","last_undegraded":"2024-03-04T06:10:15.399635+0000","last_fullsized":"2024-03-04T06:10:15.399635+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13193561,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":166,"num_read_kb":124151,"num_write":123,"num_write_kb":35145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":336,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.381502+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:10:14.381502+0000","last_peered":"2024-03-04T06:10:14.381502+0000","last_clean":"2024-03-04T06:10:14.381502+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:10:14.381502+0000","last_undegraded":"2024-03-04T06:10:14.381502+0000","last_fullsized":"2024-03-04T06:10:14.381502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"146'205","reported_seq":546,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.402301+0000","last_change":"2024-03-04T06:10:14.404267+0000","last_active":"2024-03-04T06:10:14.402301+0000","last_peered":"2024-03-04T06:10:14.402301+0000","last_clean":"2024-03-04T06:10:14.402301+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:10:14.402301+0000","last_undegraded":"2024-03-04T06:10:14.402301+0000","last_fullsized":"2024-03-04T06:10:14.402301+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9529602,"num_objects":21,"num_object_clones":14,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":92,"num_read_kb":57459,"num_write":112,"num_write_kb":29361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"146'59","reported_seq":474,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.498477+0000","last_change":"2024-03-04T06:10:14.428812+0000","last_active":"2024-03-04T06:10:14.498477+0000","last_peered":"2024-03-04T06:10:14.498477+0000","last_clean":"2024-03-04T06:10:14.498477+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:10:14.498477+0000","last_undegraded":"2024-03-04T06:10:14.498477+0000","last_fullsized":"2024-03-04T06:10:14.498477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101040,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":89,"num_read_kb":66366,"num_write":45,"num_write_kb":11178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":364,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.349040+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:10:14.349040+0000","last_peered":"2024-03-04T06:10:14.349040+0000","last_clean":"2024-03-04T06:10:14.349040+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:10:14.349040+0000","last_undegraded":"2024-03-04T06:10:14.349040+0000","last_fullsized":"2024-03-04T06:10:14.349040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"143'70","reported_seq":488,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.353525+0000","last_change":"2024-03-04T06:10:14.353525+0000","last_active":"2024-03-04T06:10:14.353525+0000","last_peered":"2024-03-04T06:10:14.353525+0000","last_clean":"2024-03-04T06:10:14.353525+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:10:14.353525+0000","last_undegraded":"2024-03-04T06:10:14.353525+0000","last_fullsized":"2024-03-04T06:10:14.353525+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781575,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":74,"num_read_kb":43419,"num_write":48,"num_write_kb":12439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":355,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-04T06:10:15.343484+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:10:15.343484+0000","last_peered":"2024-03-04T06:10:15.343484+0000","last_clean":"2024-03-04T06:10:15.343484+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:10:15.343484+0000","last_undegraded":"2024-03-04T06:10:15.343484+0000","last_fullsized":"2024-03-04T06:10:15.343484+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"147'159","reported_seq":534,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-04T06:10:15.427761+0000","last_change":"2024-03-04T06:10:15.346995+0000","last_active":"2024-03-04T06:10:15.427761+0000","last_peered":"2024-03-04T06:10:15.427761+0000","last_clean":"2024-03-04T06:10:15.427761+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:10:15.427761+0000","last_undegraded":"2024-03-04T06:10:15.427761+0000","last_fullsized":"2024-03-04T06:10:15.427761+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10252270,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":101,"num_read_kb":65965,"num_write":96,"num_write_kb":26458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":364,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.337290+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:10:14.337290+0000","last_peered":"2024-03-04T06:10:14.337290+0000","last_clean":"2024-03-04T06:10:14.337290+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:10:14.337290+0000","last_undegraded":"2024-03-04T06:10:14.337290+0000","last_fullsized":"2024-03-04T06:10:14.337290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"146'182","reported_seq":553,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.330614+0000","last_change":"2024-03-04T06:10:14.422823+0000","last_active":"2024-03-04T06:10:14.330614+0000","last_peered":"2024-03-04T06:10:14.330614+0000","last_clean":"2024-03-04T06:10:14.330614+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:10:14.330614+0000","last_undegraded":"2024-03-04T06:10:14.330614+0000","last_fullsized":"2024-03-04T06:10:14.330614+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11680931,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":123,"num_read_kb":86530,"num_write":95,"num_write_kb":26065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":344,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-04T06:10:14.398786+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:10:14.398786+0000","last_peered":"2024-03-04T06:10:14.398786+0000","last_clean":"2024-03-04T06:10:14.398786+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:10:14.398786+0000","last_undegraded":"2024-03-04T06:10:14.398786+0000","last_fullsized":"2024-03-04T06:10:14.398786+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98018919,"num_objects":198,"num_object_clones":136,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":28,"num_read":1104,"num_read_kb":776395,"num_write":965,"num_write_kb":265318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1615,"ondisk_log_size":1615,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":151792,"kb_used_data":120808,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94220048,"statfs":{"total":96636764160,"available":96481329152,"internally_reserved":0,"allocated":123707392,"data_stored":145902278,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":130968,"kb_used_data":99284,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94240872,"statfs":{"total":96636764160,"available":96502652928,"internally_reserved":0,"allocated":101666816,"data_stored":118858124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":109156,"kb_used_data":77336,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94262684,"statfs":{"total":96636764160,"available":96524988416,"internally_reserved":0,"allocated":79192064,"data_stored":90365170,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":111552,"kb_used_data":80568,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94260288,"statfs":{"total":96636764160,"available":96522534912,"internally_reserved":0,"allocated":82501632,"data_stored":95356718,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:10:19.194 INFO:tasks.rados.rados.0.smithi064.stdout:561: done (7 left) 2024-03-04T06:10:19.194 INFO:tasks.rados.rados.0.smithi064.stdout:562: snap_create 2024-03-04T06:10:19.194 INFO:tasks.rados.rados.0.smithi064.stdout:550: finishing write tid 2 to smithi06431743-30 2024-03-04T06:10:19.194 INFO:tasks.rados.rados.0.smithi064.stdout:550: finishing write tid 3 to smithi06431743-30 2024-03-04T06:10:19.194 INFO:tasks.rados.rados.0.smithi064.stdout:550: finishing write tid 4 to smithi06431743-30 2024-03-04T06:10:19.195 INFO:tasks.rados.rados.0.smithi064.stdout:550: oid 30 updating version 0 to 210 2024-03-04T06:10:19.195 INFO:tasks.rados.rados.0.smithi064.stdout:550: oid 30 version 210 is already newer than 208 2024-03-04T06:10:19.195 INFO:tasks.rados.rados.0.smithi064.stdout:550: oid 30 version 210 is already newer than 210 2024-03-04T06:10:19.195 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 210 (ObjNum 270 snap 70 seq_num 270) dirty exists 2024-03-04T06:10:19.195 INFO:tasks.rados.rados.0.smithi064.stdout:550: left oid 30 (ObjNum 270 snap 70 seq_num 270) 2024-03-04T06:10:19.195 INFO:tasks.rados.rados.0.smithi064.stdout:554: expect (ObjNum 252 snap 70 seq_num 252) 2024-03-04T06:10:19.303 INFO:tasks.rados.rados.0.smithi064.stdout:552: finishing write tid 1 to smithi06431743-39 2024-03-04T06:10:19.304 INFO:tasks.rados.rados.0.smithi064.stdout:553: expect (ObjNum 168 snap 26 seq_num 168) 2024-03-04T06:10:19.444 INFO:tasks.rados.rados.0.smithi064.stdout:552: finishing write tid 2 to smithi06431743-39 2024-03-04T06:10:19.444 INFO:tasks.rados.rados.0.smithi064.stdout:552: finishing write tid 3 to smithi06431743-39 2024-03-04T06:10:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:552: finishing write tid 4 to smithi06431743-39 2024-03-04T06:10:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:552: oid 39 updating version 0 to 124 2024-03-04T06:10:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:552: oid 39 version 124 is already newer than 123 2024-03-04T06:10:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:552: oid 39 updating version 124 to 125 2024-03-04T06:10:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 125 (ObjNum 271 snap 70 seq_num 271) dirty exists 2024-03-04T06:10:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:552: left oid 39 (ObjNum 271 snap 70 seq_num 271) 2024-03-04T06:10:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:557: expect (ObjNum 265 snap 70 seq_num 265) 2024-03-04T06:10:19.572 INFO:tasks.rados.rados.0.smithi064.stdout:560: finishing write tid 1 to smithi06431743-38 2024-03-04T06:10:19.572 INFO:tasks.rados.rados.0.smithi064.stdout:560: finishing write tid 2 to smithi06431743-38 2024-03-04T06:10:19.572 INFO:tasks.rados.rados.0.smithi064.stdout:560: finishing write tid 3 to smithi06431743-38 2024-03-04T06:10:19.572 INFO:tasks.rados.rados.0.smithi064.stdout:560: finishing write tid 4 to smithi06431743-38 2024-03-04T06:10:19.572 INFO:tasks.rados.rados.0.smithi064.stdout:560: oid 38 updating version 0 to 70 2024-03-04T06:10:19.572 INFO:tasks.rados.rados.0.smithi064.stdout:560: oid 38 updating version 70 to 71 2024-03-04T06:10:19.572 INFO:tasks.rados.rados.0.smithi064.stdout:560: oid 38 updating version 71 to 72 2024-03-04T06:10:19.572 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 72 (ObjNum 273 snap 70 seq_num 273) dirty exists 2024-03-04T06:10:19.572 INFO:tasks.rados.rados.0.smithi064.stdout:560: left oid 38 (ObjNum 273 snap 70 seq_num 273) 2024-03-04T06:10:20.193 INFO:tasks.rados.rados.0.smithi064.stdout:550: done (7 left) 2024-03-04T06:10:20.193 INFO:tasks.rados.rados.0.smithi064.stdout:552: done (6 left) 2024-03-04T06:10:20.194 INFO:tasks.rados.rados.0.smithi064.stdout:553: done (5 left) 2024-03-04T06:10:20.194 INFO:tasks.rados.rados.0.smithi064.stdout:554: done (4 left) 2024-03-04T06:10:20.194 INFO:tasks.rados.rados.0.smithi064.stdout:557: done (3 left) 2024-03-04T06:10:20.194 INFO:tasks.rados.rados.0.smithi064.stdout:559: done (2 left) 2024-03-04T06:10:20.194 INFO:tasks.rados.rados.0.smithi064.stdout:560: done (1 left) 2024-03-04T06:10:20.194 INFO:tasks.rados.rados.0.smithi064.stdout:562: done (0 left) 2024-03-04T06:10:20.194 INFO:tasks.rados.rados.0.smithi064.stdout:563: write oid 6 current snap is 71 2024-03-04T06:10:20.194 INFO:tasks.rados.rados.0.smithi064.stdout:563: seq_num 274 ranges {605118=406934,1617125=706417} 2024-03-04T06:10:20.198 INFO:tasks.rados.rados.0.smithi064.stdout:563: writing smithi06431743-6 from 605118 to 1012052 tid 1 2024-03-04T06:10:20.205 INFO:tasks.rados.rados.0.smithi064.stdout:563: writing smithi06431743-6 from 1617125 to 2323542 tid 2 2024-03-04T06:10:20.205 INFO:tasks.rados.rados.0.smithi064.stdout:564: snap_create 2024-03-04T06:10:20.274 INFO:tasks.rados.rados.0.smithi064.stdout:563: finishing write tid 1 to smithi06431743-6 2024-03-04T06:10:20.292 INFO:tasks.rados.rados.0.smithi064.stdout:563: finishing write tid 2 to smithi06431743-6 2024-03-04T06:10:20.300 INFO:tasks.rados.rados.0.smithi064.stdout:563: finishing write tid 3 to smithi06431743-6 2024-03-04T06:10:20.328 INFO:tasks.rados.rados.0.smithi064.stdout:563: finishing write tid 4 to smithi06431743-6 2024-03-04T06:10:20.328 INFO:tasks.rados.rados.0.smithi064.stdout:563: oid 6 updating version 0 to 71 2024-03-04T06:10:20.328 INFO:tasks.rados.rados.0.smithi064.stdout:563: oid 6 version 71 is already newer than 69 2024-03-04T06:10:20.328 INFO:tasks.rados.rados.0.smithi064.stdout:563: oid 6 version 71 is already newer than 70 2024-03-04T06:10:20.328 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 71 (ObjNum 274 snap 71 seq_num 274) dirty exists 2024-03-04T06:10:20.328 INFO:tasks.rados.rados.0.smithi064.stdout:563: left oid 6 (ObjNum 274 snap 71 seq_num 274) 2024-03-04T06:10:20.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:20.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:20.799+0000 7ff60c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:20.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:20.804+0000 7ff60c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:21.196 INFO:tasks.rados.rados.0.smithi064.stdout:563: done (1 left) 2024-03-04T06:10:21.196 INFO:tasks.rados.rados.0.smithi064.stdout:564: done (0 left) 2024-03-04T06:10:21.196 INFO:tasks.rados.rados.0.smithi064.stdout:565: write oid 22 current snap is 72 2024-03-04T06:10:21.196 INFO:tasks.rados.rados.0.smithi064.stdout:565: seq_num 275 ranges {621925=482183,1559254=566428} 2024-03-04T06:10:21.201 INFO:tasks.rados.rados.0.smithi064.stdout:565: writing smithi06431743-22 from 621925 to 1104108 tid 1 2024-03-04T06:10:21.207 INFO:tasks.rados.rados.0.smithi064.stdout:565: writing smithi06431743-22 from 1559254 to 2125682 tid 2 2024-03-04T06:10:21.208 INFO:tasks.rados.rados.0.smithi064.stdout:566: snap_remove snap 42 2024-03-04T06:10:22.197 INFO:tasks.rados.rados.0.smithi064.stdout:566: done (1 left) 2024-03-04T06:10:22.198 INFO:tasks.rados.rados.0.smithi064.stdout:567: snap_create 2024-03-04T06:10:22.198 INFO:tasks.rados.rados.0.smithi064.stdout:565: finishing write tid 1 to smithi06431743-22 2024-03-04T06:10:22.198 INFO:tasks.rados.rados.0.smithi064.stdout:565: finishing write tid 2 to smithi06431743-22 2024-03-04T06:10:22.198 INFO:tasks.rados.rados.0.smithi064.stdout:565: finishing write tid 3 to smithi06431743-22 2024-03-04T06:10:22.198 INFO:tasks.rados.rados.0.smithi064.stdout:565: finishing write tid 4 to smithi06431743-22 2024-03-04T06:10:22.198 INFO:tasks.rados.rados.0.smithi064.stdout:565: oid 22 updating version 0 to 210 2024-03-04T06:10:22.198 INFO:tasks.rados.rados.0.smithi064.stdout:565: oid 22 version 210 is already newer than 209 2024-03-04T06:10:22.198 INFO:tasks.rados.rados.0.smithi064.stdout:565: oid 22 version 210 is already newer than 208 2024-03-04T06:10:22.198 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 210 (ObjNum 275 snap 72 seq_num 275) dirty exists 2024-03-04T06:10:22.198 INFO:tasks.rados.rados.0.smithi064.stdout:565: left oid 22 (ObjNum 275 snap 72 seq_num 275) 2024-03-04T06:10:22.870 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:22.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:22.980+0000 7f165d35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:22.985 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:22.985+0000 7f165d35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:23.201 INFO:tasks.rados.rados.0.smithi064.stdout:565: done (1 left) 2024-03-04T06:10:23.201 INFO:tasks.rados.rados.0.smithi064.stdout:567: done (0 left) 2024-03-04T06:10:23.201 INFO:tasks.rados.rados.0.smithi064.stdout:568: snap_create 2024-03-04T06:10:23.950 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:10:23.950 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:10:23.950 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:10:23.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:10:24.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:24.063+0000 7f995362d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:24.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:24.070+0000 7f995362d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:24.083 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:24.083 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:24.083 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:24.083 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:24.083 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:24.083 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:24.083 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:24.083 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:24.083 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:24.083 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:24.084 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:24.084 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:24.086 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:24.086 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:24.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:10:24.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:24.189+0000 7fc204d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:24.197 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:24.196+0000 7fc204d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:24.208 INFO:tasks.rados.rados.0.smithi064.stdout:568: done (0 left) 2024-03-04T06:10:24.208 INFO:tasks.rados.rados.0.smithi064.stdout:569: snap_remove snap 69 2024-03-04T06:10:24.217 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:24.217 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:24.217 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:24.217 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:24.217 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:24.218 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:24.218 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:24.218 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:24.218 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:24.218 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:24.218 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:24.219 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:24.222 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:24.222 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:24.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:10:24.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:24.325+0000 7fa3f995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:24.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:24.332+0000 7fa3f995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:24.347 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:24.349 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:24.349 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:10:24.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:24.452+0000 7f10e219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:24.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:24.459+0000 7f10e219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:24.475 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:25.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:25.165 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:25.163+0000 7f53a079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:25.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:25.167+0000 7f53a079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:25.210 INFO:tasks.rados.rados.0.smithi064.stdout:569: done (0 left) 2024-03-04T06:10:25.210 INFO:tasks.rados.rados.0.smithi064.stdout:570: write oid 46 current snap is 74 2024-03-04T06:10:25.210 INFO:tasks.rados.rados.0.smithi064.stdout:570: seq_num 276 ranges {638732=557432,1617736=710086} 2024-03-04T06:10:25.216 INFO:tasks.rados.rados.0.smithi064.stdout:570: writing smithi06431743-46 from 638732 to 1196164 tid 1 2024-03-04T06:10:25.224 INFO:tasks.rados.rados.0.smithi064.stdout:570: writing smithi06431743-46 from 1617736 to 2327822 tid 2 2024-03-04T06:10:25.225 INFO:tasks.rados.rados.0.smithi064.stdout:571: snap_remove snap 50 2024-03-04T06:10:26.212 INFO:tasks.rados.rados.0.smithi064.stdout:571: done (1 left) 2024-03-04T06:10:26.212 INFO:tasks.rados.rados.0.smithi064.stdout:572: delete oid 37 current snap is 74 2024-03-04T06:10:26.212 INFO:tasks.rados.rados.0.smithi064.stdout:570: finishing write tid 1 to smithi06431743-46 2024-03-04T06:10:26.213 INFO:tasks.rados.rados.0.smithi064.stdout:570: finishing write tid 2 to smithi06431743-46 2024-03-04T06:10:26.213 INFO:tasks.rados.rados.0.smithi064.stdout:570: finishing write tid 3 to smithi06431743-46 2024-03-04T06:10:26.213 INFO:tasks.rados.rados.0.smithi064.stdout:570: finishing write tid 4 to smithi06431743-46 2024-03-04T06:10:26.213 INFO:tasks.rados.rados.0.smithi064.stdout:570: oid 46 updating version 0 to 219 2024-03-04T06:10:26.213 INFO:tasks.rados.rados.0.smithi064.stdout:570: oid 46 version 219 is already newer than 218 2024-03-04T06:10:26.213 INFO:tasks.rados.rados.0.smithi064.stdout:570: oid 46 version 219 is already newer than 217 2024-03-04T06:10:26.213 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 219 (ObjNum 276 snap 74 seq_num 276) dirty exists 2024-03-04T06:10:26.213 INFO:tasks.rados.rados.0.smithi064.stdout:570: left oid 46 (ObjNum 276 snap 74 seq_num 276) 2024-03-04T06:10:26.268 INFO:tasks.rados.rados.0.smithi064.stdout:570: done (1 left) 2024-03-04T06:10:26.268 INFO:tasks.rados.rados.0.smithi064.stdout:572: done (0 left) 2024-03-04T06:10:26.269 INFO:tasks.rados.rados.0.smithi064.stdout:573: delete oid 48 current snap is 74 2024-03-04T06:10:26.278 INFO:tasks.rados.rados.0.smithi064.stdout:573: done (0 left) 2024-03-04T06:10:26.279 INFO:tasks.rados.rados.0.smithi064.stdout:574: read oid 9 snap -1 2024-03-04T06:10:26.279 INFO:tasks.rados.rados.0.smithi064.stdout:574: expect (ObjNum 256 snap 70 seq_num 256) 2024-03-04T06:10:26.279 INFO:tasks.rados.rados.0.smithi064.stdout:575: snap_create 2024-03-04T06:10:26.300 INFO:tasks.rados.rados.0.smithi064.stdout:574: expect (ObjNum 256 snap 70 seq_num 256) 2024-03-04T06:10:27.225 INFO:tasks.rados.rados.0.smithi064.stdout:574: done (1 left) 2024-03-04T06:10:27.226 INFO:tasks.rados.rados.0.smithi064.stdout:575: done (0 left) 2024-03-04T06:10:27.226 INFO:tasks.rados.rados.0.smithi064.stdout:576: write oid 45 current snap is 75 2024-03-04T06:10:27.226 INFO:tasks.rados.rados.0.smithi064.stdout:576: seq_num 279 ranges {689153=783179,2076829=457413} 2024-03-04T06:10:27.234 INFO:tasks.rados.rados.0.smithi064.stdout:576: writing smithi06431743-45 from 689153 to 1472332 tid 1 2024-03-04T06:10:27.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:27.243 INFO:tasks.rados.rados.0.smithi064.stdout:576: writing smithi06431743-45 from 2076829 to 2534242 tid 2 2024-03-04T06:10:27.243 INFO:tasks.rados.rados.0.smithi064.stdout:577: read oid 2 snap 58 2024-03-04T06:10:27.243 INFO:tasks.rados.rados.0.smithi064.stdout:577: expect deleted 2024-03-04T06:10:27.243 INFO:tasks.rados.rados.0.smithi064.stdout:578: snap_remove snap 70 2024-03-04T06:10:27.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:27.352+0000 7f246af9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:27.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:27.357+0000 7f246af9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:28.195 INFO:tasks.rados.rados.0.smithi064.stdout:578: done (2 left) 2024-03-04T06:10:28.195 INFO:tasks.rados.rados.0.smithi064.stdout:579: snap_remove snap 74 2024-03-04T06:10:29.194 INFO:tasks.rados.rados.0.smithi064.stdout:579: done (2 left) 2024-03-04T06:10:29.194 INFO:tasks.rados.rados.0.smithi064.stdout:580: write (excl) oid 40 current snap is 75 2024-03-04T06:10:29.194 INFO:tasks.rados.rados.0.smithi064.stdout:580: seq_num 280 ranges {705960=458428,1735311=717424} 2024-03-04T06:10:29.200 INFO:tasks.rados.rados.0.smithi064.stdout:580: writing smithi06431743-40 from 705960 to 1164388 tid 1 2024-03-04T06:10:29.208 INFO:tasks.rados.rados.0.smithi064.stdout:580: writing smithi06431743-40 from 1735311 to 2452735 tid 2 2024-03-04T06:10:29.208 INFO:tasks.rados.rados.0.smithi064.stdout:581: read oid 50 snap -1 2024-03-04T06:10:29.208 INFO:tasks.rados.rados.0.smithi064.stdout:581: expect (ObjNum 226 snap 55 seq_num 226) 2024-03-04T06:10:29.208 INFO:tasks.rados.rados.0.smithi064.stdout:582: write oid 19 current snap is 75 2024-03-04T06:10:29.208 INFO:tasks.rados.rados.0.smithi064.stdout:582: seq_num 281 ranges {722767=533677,1793793=461082} 2024-03-04T06:10:29.215 INFO:tasks.rados.rados.0.smithi064.stdout:582: writing smithi06431743-19 from 722767 to 1256444 tid 1 2024-03-04T06:10:29.222 INFO:tasks.rados.rados.0.smithi064.stdout:582: writing smithi06431743-19 from 1793793 to 2254875 tid 2 2024-03-04T06:10:29.222 INFO:tasks.rados.rados.0.smithi064.stdout:583: snap_create 2024-03-04T06:10:29.222 INFO:tasks.rados.rados.0.smithi064.stdout:576: finishing write tid 1 to smithi06431743-45 2024-03-04T06:10:29.222 INFO:tasks.rados.rados.0.smithi064.stdout:576: finishing write tid 2 to smithi06431743-45 2024-03-04T06:10:29.222 INFO:tasks.rados.rados.0.smithi064.stdout:576: finishing write tid 3 to smithi06431743-45 2024-03-04T06:10:29.222 INFO:tasks.rados.rados.0.smithi064.stdout:576: finishing write tid 4 to smithi06431743-45 2024-03-04T06:10:29.222 INFO:tasks.rados.rados.0.smithi064.stdout:576: oid 45 updating version 0 to 140 2024-03-04T06:10:29.223 INFO:tasks.rados.rados.0.smithi064.stdout:576: oid 45 version 140 is already newer than 138 2024-03-04T06:10:29.223 INFO:tasks.rados.rados.0.smithi064.stdout:576: oid 45 version 140 is already newer than 140 2024-03-04T06:10:29.223 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 140 (ObjNum 279 snap 75 seq_num 279) dirty exists 2024-03-04T06:10:29.223 INFO:tasks.rados.rados.0.smithi064.stdout:576: left oid 45 (ObjNum 279 snap 75 seq_num 279) 2024-03-04T06:10:29.266 INFO:tasks.rados.rados.0.smithi064.stdout:580: finishing write tid 1 to smithi06431743-40 2024-03-04T06:10:29.282 INFO:tasks.rados.rados.0.smithi064.stdout:580: finishing write tid 2 to smithi06431743-40 2024-03-04T06:10:29.296 INFO:tasks.rados.rados.0.smithi064.stdout:582: finishing write tid 1 to smithi06431743-19 2024-03-04T06:10:29.297 INFO:tasks.rados.rados.0.smithi064.stdout:580: finishing write tid 3 to smithi06431743-40 2024-03-04T06:10:29.299 INFO:tasks.rados.rados.0.smithi064.stdout:580: finishing write tid 4 to smithi06431743-40 2024-03-04T06:10:29.299 INFO:tasks.rados.rados.0.smithi064.stdout:580: oid 40 updating version 0 to 100 2024-03-04T06:10:29.299 INFO:tasks.rados.rados.0.smithi064.stdout:580: oid 40 version 100 is already newer than 98 2024-03-04T06:10:29.299 INFO:tasks.rados.rados.0.smithi064.stdout:580: oid 40 version 100 is already newer than 99 2024-03-04T06:10:29.299 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 100 (ObjNum 280 snap 75 seq_num 280) dirty exists 2024-03-04T06:10:29.299 INFO:tasks.rados.rados.0.smithi064.stdout:580: left oid 40 (ObjNum 280 snap 75 seq_num 280) 2024-03-04T06:10:29.303 INFO:tasks.rados.rados.0.smithi064.stdout:581: expect (ObjNum 226 snap 55 seq_num 226) 2024-03-04T06:10:29.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:29.476 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:10:29.476 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:10:29.476 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:10:29.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:10:29.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:29.531+0000 7fa09886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:29.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:29.536+0000 7fa09886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:29.548 INFO:tasks.rados.rados.0.smithi064.stdout:582: finishing write tid 2 to smithi06431743-19 2024-03-04T06:10:29.548 INFO:tasks.rados.rados.0.smithi064.stdout:582: finishing write tid 3 to smithi06431743-19 2024-03-04T06:10:29.548 INFO:tasks.rados.rados.0.smithi064.stdout:582: finishing write tid 4 to smithi06431743-19 2024-03-04T06:10:29.548 INFO:tasks.rados.rados.0.smithi064.stdout:582: oid 19 updating version 0 to 238 2024-03-04T06:10:29.548 INFO:tasks.rados.rados.0.smithi064.stdout:582: oid 19 version 238 is already newer than 236 2024-03-04T06:10:29.548 INFO:tasks.rados.rados.0.smithi064.stdout:582: oid 19 version 238 is already newer than 237 2024-03-04T06:10:29.549 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 238 (ObjNum 281 snap 75 seq_num 281) dirty exists 2024-03-04T06:10:29.549 INFO:tasks.rados.rados.0.smithi064.stdout:582: left oid 19 (ObjNum 281 snap 75 seq_num 281) 2024-03-04T06:10:29.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:29.587+0000 7f2931314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:29.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:29.594+0000 7f2931314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:29.609 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:29.609 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:29.611 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:29.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:10:29.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:29.756+0000 7f7f5862d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:29.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:29.763+0000 7f7f5862d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:29.776 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:29.776 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:29.776 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:29.776 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:29.776 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:29.777 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:29.777 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:29.777 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:29.777 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:29.777 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:29.777 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:29.777 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:29.779 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:29.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:10:29.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:29.882+0000 7f12a6a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:29.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:29.889+0000 7f12a6a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:29.903 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:29.903 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:29.903 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:29.903 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:29.903 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:29.903 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:29.903 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:29.903 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:29.903 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:29.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:29.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:29.904 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:29.906 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:29.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:10:30.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:30.010+0000 7fb9d6a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:30.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:30.017+0000 7fb9d6a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:30.030 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:30.030 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:30.030 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:30.030 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:30.031 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:30.031 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:30.031 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:30.031 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:30.031 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:30.031 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:30.031 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:30.031 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:30.033 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:30.226 INFO:tasks.rados.rados.0.smithi064.stdout:576: done (5 left) 2024-03-04T06:10:30.226 INFO:tasks.rados.rados.0.smithi064.stdout:577: done (4 left) 2024-03-04T06:10:30.226 INFO:tasks.rados.rados.0.smithi064.stdout:580: done (3 left) 2024-03-04T06:10:30.226 INFO:tasks.rados.rados.0.smithi064.stdout:581: done (2 left) 2024-03-04T06:10:30.226 INFO:tasks.rados.rados.0.smithi064.stdout:582: done (1 left) 2024-03-04T06:10:30.226 INFO:tasks.rados.rados.0.smithi064.stdout:583: done (0 left) 2024-03-04T06:10:30.226 INFO:tasks.rados.rados.0.smithi064.stdout:584: write (excl) oid 28 current snap is 76 2024-03-04T06:10:30.226 INFO:tasks.rados.rados.0.smithi064.stdout:584: seq_num 282 ranges {739574=725279,1968628=721093} 2024-03-04T06:10:30.233 INFO:tasks.rados.rados.0.smithi064.stdout:584: writing smithi06431743-28 from 739574 to 1464853 tid 1 2024-03-04T06:10:30.240 INFO:tasks.rados.rados.0.smithi064.stdout:584: writing smithi06431743-28 from 1968628 to 2689721 tid 2 2024-03-04T06:10:30.240 INFO:tasks.rados.rados.0.smithi064.stdout:585: snap_create 2024-03-04T06:10:30.291 INFO:tasks.rados.rados.0.smithi064.stdout:584: finishing write tid 1 to smithi06431743-28 2024-03-04T06:10:30.306 INFO:tasks.rados.rados.0.smithi064.stdout:584: finishing write tid 2 to smithi06431743-28 2024-03-04T06:10:30.310 INFO:tasks.rados.rados.0.smithi064.stdout:584: finishing write tid 3 to smithi06431743-28 2024-03-04T06:10:30.311 INFO:tasks.rados.rados.0.smithi064.stdout:584: finishing write tid 4 to smithi06431743-28 2024-03-04T06:10:30.312 INFO:tasks.rados.rados.0.smithi064.stdout:584: oid 28 updating version 0 to 227 2024-03-04T06:10:30.312 INFO:tasks.rados.rados.0.smithi064.stdout:584: oid 28 version 227 is already newer than 226 2024-03-04T06:10:30.312 INFO:tasks.rados.rados.0.smithi064.stdout:584: oid 28 version 227 is already newer than 225 2024-03-04T06:10:30.312 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 227 (ObjNum 282 snap 76 seq_num 282) dirty exists 2024-03-04T06:10:30.312 INFO:tasks.rados.rados.0.smithi064.stdout:584: left oid 28 (ObjNum 282 snap 76 seq_num 282) 2024-03-04T06:10:31.228 INFO:tasks.rados.rados.0.smithi064.stdout:584: done (1 left) 2024-03-04T06:10:31.228 INFO:tasks.rados.rados.0.smithi064.stdout:585: done (0 left) 2024-03-04T06:10:31.228 INFO:tasks.rados.rados.0.smithi064.stdout:586: write (excl) oid 19 current snap is 77 2024-03-04T06:10:31.228 INFO:tasks.rados.rados.0.smithi064.stdout:586: seq_num 283 ranges {756381=400528,1910757=464751} 2024-03-04T06:10:31.234 INFO:tasks.rados.rados.0.smithi064.stdout:586: writing smithi06431743-19 from 756381 to 1156909 tid 1 2024-03-04T06:10:31.241 INFO:tasks.rados.rados.0.smithi064.stdout:586: writing smithi06431743-19 from 1910757 to 2375508 tid 2 2024-03-04T06:10:31.241 INFO:tasks.rados.rados.0.smithi064.stdout:587: snap_remove snap 72 2024-03-04T06:10:31.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:31.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:31.727+0000 7fc5b18dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:31.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:31.732+0000 7fc5b18dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:32.230 INFO:tasks.rados.rados.0.smithi064.stdout:587: done (1 left) 2024-03-04T06:10:32.230 INFO:tasks.rados.rados.0.smithi064.stdout:588: write (excl) oid 50 current snap is 77 2024-03-04T06:10:32.230 INFO:tasks.rados.rados.0.smithi064.stdout:586: finishing write tid 1 to smithi06431743-19 2024-03-04T06:10:32.230 INFO:tasks.rados.rados.0.smithi064.stdout:588: seq_num 284 ranges {773188=475777,1969239=724762} 2024-03-04T06:10:32.236 INFO:tasks.rados.rados.0.smithi064.stdout:588: writing smithi06431743-50 from 773188 to 1248965 tid 1 2024-03-04T06:10:32.245 INFO:tasks.rados.rados.0.smithi064.stdout:588: writing smithi06431743-50 from 1969239 to 2694001 tid 2 2024-03-04T06:10:32.245 INFO:tasks.rados.rados.0.smithi064.stdout:589: snap_create 2024-03-04T06:10:32.245 INFO:tasks.rados.rados.0.smithi064.stdout:586: finishing write tid 2 to smithi06431743-19 2024-03-04T06:10:32.245 INFO:tasks.rados.rados.0.smithi064.stdout:586: finishing write tid 3 to smithi06431743-19 2024-03-04T06:10:32.246 INFO:tasks.rados.rados.0.smithi064.stdout:586: finishing write tid 4 to smithi06431743-19 2024-03-04T06:10:32.246 INFO:tasks.rados.rados.0.smithi064.stdout:586: oid 19 updating version 0 to 240 2024-03-04T06:10:32.246 INFO:tasks.rados.rados.0.smithi064.stdout:586: oid 19 updating version 240 to 242 2024-03-04T06:10:32.246 INFO:tasks.rados.rados.0.smithi064.stdout:586: oid 19 version 242 is already newer than 242 2024-03-04T06:10:32.246 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 242 (ObjNum 283 snap 77 seq_num 283) dirty exists 2024-03-04T06:10:32.246 INFO:tasks.rados.rados.0.smithi064.stdout:586: left oid 19 (ObjNum 283 snap 77 seq_num 283) 2024-03-04T06:10:32.298 INFO:tasks.rados.rados.0.smithi064.stdout:588: finishing write tid 1 to smithi06431743-50 2024-03-04T06:10:32.320 INFO:tasks.rados.rados.0.smithi064.stdout:588: finishing write tid 2 to smithi06431743-50 2024-03-04T06:10:32.346 INFO:tasks.rados.rados.0.smithi064.stdout:588: finishing write tid 3 to smithi06431743-50 2024-03-04T06:10:32.348 INFO:tasks.rados.rados.0.smithi064.stdout:588: finishing write tid 4 to smithi06431743-50 2024-03-04T06:10:32.348 INFO:tasks.rados.rados.0.smithi064.stdout:588: oid 50 updating version 0 to 231 2024-03-04T06:10:32.348 INFO:tasks.rados.rados.0.smithi064.stdout:588: oid 50 updating version 231 to 232 2024-03-04T06:10:32.348 INFO:tasks.rados.rados.0.smithi064.stdout:588: oid 50 version 232 is already newer than 230 2024-03-04T06:10:32.348 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 232 (ObjNum 284 snap 77 seq_num 284) dirty exists 2024-03-04T06:10:32.348 INFO:tasks.rados.rados.0.smithi064.stdout:588: left oid 50 (ObjNum 284 snap 77 seq_num 284) 2024-03-04T06:10:33.236 INFO:tasks.rados.rados.0.smithi064.stdout:586: done (2 left) 2024-03-04T06:10:33.236 INFO:tasks.rados.rados.0.smithi064.stdout:588: done (1 left) 2024-03-04T06:10:33.237 INFO:tasks.rados.rados.0.smithi064.stdout:589: done (0 left) 2024-03-04T06:10:33.237 INFO:tasks.rados.rados.0.smithi064.stdout:590: write (excl) oid 2 current snap is 78 2024-03-04T06:10:33.237 INFO:tasks.rados.rados.0.smithi064.stdout:590: seq_num 285 ranges {789995=551026,2027721=468420} 2024-03-04T06:10:33.243 INFO:tasks.rados.rados.0.smithi064.stdout:590: writing smithi06431743-2 from 789995 to 1341021 tid 1 2024-03-04T06:10:33.250 INFO:tasks.rados.rados.0.smithi064.stdout:590: writing smithi06431743-2 from 2027721 to 2496141 tid 2 2024-03-04T06:10:33.251 INFO:tasks.rados.rados.0.smithi064.stdout:591: read oid 35 snap -1 2024-03-04T06:10:33.251 INFO:tasks.rados.rados.0.smithi064.stdout:591: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-04T06:10:33.251 INFO:tasks.rados.rados.0.smithi064.stdout:592: delete oid 8 current snap is 78 2024-03-04T06:10:33.268 INFO:tasks.rados.rados.0.smithi064.stdout:592: done (2 left) 2024-03-04T06:10:33.269 INFO:tasks.rados.rados.0.smithi064.stdout:593: write (excl) oid 32 current snap is 78 2024-03-04T06:10:33.269 INFO:tasks.rados.rados.0.smithi064.stdout:593: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-03-04T06:10:33.276 INFO:tasks.rados.rados.0.smithi064.stdout:593: writing smithi06431743-32 from 423609 to 1125133 tid 1 2024-03-04T06:10:33.282 INFO:tasks.rados.rados.0.smithi064.stdout:593: writing smithi06431743-32 from 1628332 to 2100421 tid 2 2024-03-04T06:10:33.287 INFO:tasks.rados.rados.0.smithi064.stdout:593: writing smithi06431743-32 from 2648987 to 2823609 tid 3 2024-03-04T06:10:33.287 INFO:tasks.rados.rados.0.smithi064.stdout:594: read oid 26 snap -1 2024-03-04T06:10:33.287 INFO:tasks.rados.rados.0.smithi064.stdout:594: expect (ObjNum 258 snap 70 seq_num 258) 2024-03-04T06:10:33.287 INFO:tasks.rados.rados.0.smithi064.stdout:595: write oid 30 current snap is 78 2024-03-04T06:10:33.287 INFO:tasks.rados.rados.0.smithi064.stdout:595: seq_num 288 ranges {440416=776773,1686814=732100} 2024-03-04T06:10:33.293 INFO:tasks.rados.rados.0.smithi064.stdout:595: writing smithi06431743-30 from 440416 to 1217189 tid 1 2024-03-04T06:10:33.300 INFO:tasks.rados.rados.0.smithi064.stdout:595: writing smithi06431743-30 from 1686814 to 2418914 tid 2 2024-03-04T06:10:33.300 INFO:tasks.rados.rados.0.smithi064.stdout:596: write (excl) oid 5 current snap is 78 2024-03-04T06:10:33.300 INFO:tasks.rados.rados.0.smithi064.stdout:596: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-03-04T06:10:33.305 INFO:tasks.rados.rados.0.smithi064.stdout:596: writing smithi06431743-5 from 457223 to 1025598 tid 1 2024-03-04T06:10:33.313 INFO:tasks.rados.rados.0.smithi064.stdout:596: writing smithi06431743-5 from 1461649 to 1937407 tid 2 2024-03-04T06:10:33.318 INFO:tasks.rados.rados.0.smithi064.stdout:596: writing smithi06431743-5 from 2420186 to 2857223 tid 3 2024-03-04T06:10:33.319 INFO:tasks.rados.rados.0.smithi064.stdout:597: write (excl) oid 6 current snap is 78 2024-03-04T06:10:33.319 INFO:tasks.rados.rados.0.smithi064.stdout:597: seq_num 290 ranges {474030=643624,1520131=735769} 2024-03-04T06:10:33.324 INFO:tasks.rados.rados.0.smithi064.stdout:597: writing smithi06431743-6 from 474030 to 1117654 tid 1 2024-03-04T06:10:33.331 INFO:tasks.rados.rados.0.smithi064.stdout:597: writing smithi06431743-6 from 1520131 to 2255900 tid 2 2024-03-04T06:10:33.331 INFO:tasks.rados.rados.0.smithi064.stdout:598: read oid 39 snap 76 2024-03-04T06:10:33.332 INFO:tasks.rados.rados.0.smithi064.stdout:598: expect (ObjNum 271 snap 70 seq_num 271) 2024-03-04T06:10:33.332 INFO:tasks.rados.rados.0.smithi064.stdout:599: delete oid 21 current snap is 78 2024-03-04T06:10:33.332 INFO:tasks.rados.rados.0.smithi064.stdout:590: finishing write tid 1 to smithi06431743-2 2024-03-04T06:10:33.335 INFO:tasks.rados.rados.0.smithi064.stdout:591: expect (ObjNum 110 snap 9 seq_num 110) 2024-03-04T06:10:33.580 INFO:tasks.rados.rados.0.smithi064.stdout:593: finishing write tid 1 to smithi06431743-32 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:595: finishing write tid 1 to smithi06431743-30 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:590: finishing write tid 2 to smithi06431743-2 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:590: finishing write tid 3 to smithi06431743-2 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:593: finishing write tid 2 to smithi06431743-32 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:590: finishing write tid 4 to smithi06431743-2 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:590: oid 2 updating version 0 to 103 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:590: oid 2 version 103 is already newer than 101 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:590: oid 2 version 103 is already newer than 102 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 103 (ObjNum 285 snap 78 seq_num 285) dirty exists 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:590: left oid 2 (ObjNum 285 snap 78 seq_num 285) 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:595: finishing write tid 2 to smithi06431743-30 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:590: done (8 left) 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:591: done (7 left) 2024-03-04T06:10:33.581 INFO:tasks.rados.rados.0.smithi064.stdout:599: done (6 left) 2024-03-04T06:10:33.582 INFO:tasks.rados.rados.0.smithi064.stdout:600: snap_remove snap 71 2024-03-04T06:10:33.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:33.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:33.906+0000 7feef871d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:33.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:33.911+0000 7feef871d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:34.238 INFO:tasks.rados.rados.0.smithi064.stdout:600: done (6 left) 2024-03-04T06:10:34.238 INFO:tasks.rados.rados.0.smithi064.stdout:601: snap_create 2024-03-04T06:10:34.238 INFO:tasks.rados.rados.0.smithi064.stdout:593: finishing write tid 3 to smithi06431743-32 2024-03-04T06:10:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:595: finishing write tid 3 to smithi06431743-30 2024-03-04T06:10:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:593: finishing write tid 4 to smithi06431743-32 2024-03-04T06:10:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:598: expect (ObjNum 271 snap 70 seq_num 271) 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:595: finishing write tid 4 to smithi06431743-30 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:595: oid 30 updating version 0 to 247 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:595: oid 30 version 247 is already newer than 246 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:595: oid 30 updating version 247 to 248 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 248 (ObjNum 288 snap 78 seq_num 288) dirty exists 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:595: left oid 30 (ObjNum 288 snap 78 seq_num 288) 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:597: finishing write tid 1 to smithi06431743-6 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:593: finishing write tid 5 to smithi06431743-32 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:593: oid 32 updating version 0 to 231 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:593: oid 32 updating version 231 to 234 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:593: oid 32 version 234 is already newer than 232 2024-03-04T06:10:34.335 INFO:tasks.rados.rados.0.smithi064.stdout:593: oid 32 version 234 is already newer than 234 2024-03-04T06:10:34.336 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 234 (ObjNum 287 snap 78 seq_num 287) dirty exists 2024-03-04T06:10:34.336 INFO:tasks.rados.rados.0.smithi064.stdout:593: left oid 32 (ObjNum 287 snap 78 seq_num 287) 2024-03-04T06:10:34.336 INFO:tasks.rados.rados.0.smithi064.stdout:597: finishing write tid 2 to smithi06431743-6 2024-03-04T06:10:34.336 INFO:tasks.rados.rados.0.smithi064.stdout:594: expect (ObjNum 258 snap 70 seq_num 258) 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:596: finishing write tid 1 to smithi06431743-5 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:596: finishing write tid 2 to smithi06431743-5 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:597: finishing write tid 3 to smithi06431743-6 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:597: finishing write tid 4 to smithi06431743-6 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:597: oid 6 updating version 0 to 91 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:597: oid 6 updating version 91 to 92 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:597: oid 6 updating version 92 to 93 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 93 (ObjNum 290 snap 78 seq_num 290) dirty exists 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:597: left oid 6 (ObjNum 290 snap 78 seq_num 290) 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:596: finishing write tid 3 to smithi06431743-5 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:596: finishing write tid 4 to smithi06431743-5 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:596: finishing write tid 5 to smithi06431743-5 2024-03-04T06:10:34.480 INFO:tasks.rados.rados.0.smithi064.stdout:596: oid 5 updating version 0 to 250 2024-03-04T06:10:34.481 INFO:tasks.rados.rados.0.smithi064.stdout:596: oid 5 updating version 250 to 253 2024-03-04T06:10:34.481 INFO:tasks.rados.rados.0.smithi064.stdout:596: oid 5 version 253 is already newer than 253 2024-03-04T06:10:34.481 INFO:tasks.rados.rados.0.smithi064.stdout:596: oid 5 version 253 is already newer than 253 2024-03-04T06:10:34.481 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 253 (ObjNum 289 snap 78 seq_num 289) dirty exists 2024-03-04T06:10:34.481 INFO:tasks.rados.rados.0.smithi064.stdout:596: left oid 5 (ObjNum 289 snap 78 seq_num 289) 2024-03-04T06:10:35.034 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:10:35.034 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:10:35.034 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:10:35.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:10:35.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:35.144+0000 7f4ae8e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:35.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:35.151+0000 7f4ae8e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:35.164 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:35.164 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:35.164 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:35.164 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:35.164 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:35.164 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:35.164 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:35.164 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:35.165 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:35.165 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:35.165 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:35.165 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:35.167 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:35.167 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:35.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:10:35.243 INFO:tasks.rados.rados.0.smithi064.stdout:593: done (6 left) 2024-03-04T06:10:35.243 INFO:tasks.rados.rados.0.smithi064.stdout:594: done (5 left) 2024-03-04T06:10:35.243 INFO:tasks.rados.rados.0.smithi064.stdout:595: done (4 left) 2024-03-04T06:10:35.243 INFO:tasks.rados.rados.0.smithi064.stdout:596: done (3 left) 2024-03-04T06:10:35.243 INFO:tasks.rados.rados.0.smithi064.stdout:597: done (2 left) 2024-03-04T06:10:35.243 INFO:tasks.rados.rados.0.smithi064.stdout:598: done (1 left) 2024-03-04T06:10:35.243 INFO:tasks.rados.rados.0.smithi064.stdout:601: done (0 left) 2024-03-04T06:10:35.244 INFO:tasks.rados.rados.0.smithi064.stdout:602: snap_create 2024-03-04T06:10:35.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:35.271+0000 7f5ca942d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:35.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:35.279+0000 7f5ca942d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:35.297 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:35.297 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:10:35.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:35.400+0000 7f09bb107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:35.407 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:35.407+0000 7f09bb107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:35.420 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:35.420 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:35.421 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:35.421 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:35.421 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:35.421 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:35.421 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:35.421 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:35.421 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:35.421 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:35.421 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:35.421 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:35.423 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:35.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:10:35.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:35.527+0000 7f4116d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:35.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:35.534+0000 7f4116d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:35.548 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:35.548 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:35.548 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:35.548 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:35.548 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:35.548 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:35.549 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:35.549 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:35.549 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:35.549 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:35.549 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:35.549 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:35.551 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:35.551 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:35.986 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:36.096 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:36.094+0000 7fe84acad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:36.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:36.098+0000 7fe84acad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:36.243 INFO:tasks.rados.rados.0.smithi064.stdout:602: done (0 left) 2024-03-04T06:10:36.243 INFO:tasks.rados.rados.0.smithi064.stdout:603: write oid 29 current snap is 80 2024-03-04T06:10:36.243 INFO:tasks.rados.rados.0.smithi064.stdout:603: seq_num 292 ranges {507644=794122,1920742=623085} 2024-03-04T06:10:36.249 INFO:tasks.rados.rados.0.smithi064.stdout:603: writing smithi06431743-29 from 507644 to 1301766 tid 1 2024-03-04T06:10:36.256 INFO:tasks.rados.rados.0.smithi064.stdout:603: writing smithi06431743-29 from 1920742 to 2543827 tid 2 2024-03-04T06:10:36.257 INFO:tasks.rados.rados.0.smithi064.stdout:604: write (excl) oid 19 current snap is 80 2024-03-04T06:10:36.257 INFO:tasks.rados.rados.0.smithi064.stdout:604: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2024-03-04T06:10:36.262 INFO:tasks.rados.rados.0.smithi064.stdout:604: writing smithi06431743-19 from 524451 to 993822 tid 1 2024-03-04T06:10:36.267 INFO:tasks.rados.rados.0.smithi064.stdout:604: writing smithi06431743-19 from 1579224 to 2062320 tid 2 2024-03-04T06:10:36.272 INFO:tasks.rados.rados.0.smithi064.stdout:604: writing smithi06431743-19 from 2529878 to 2924451 tid 3 2024-03-04T06:10:36.272 INFO:tasks.rados.rados.0.smithi064.stdout:605: write (excl) oid 31 current snap is 80 2024-03-04T06:10:36.272 INFO:tasks.rados.rados.0.smithi064.stdout:605: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-03-04T06:10:36.278 INFO:tasks.rados.rados.0.smithi064.stdout:605: writing smithi06431743-31 from 541258 to 1085878 tid 1 2024-03-04T06:10:36.284 INFO:tasks.rados.rados.0.smithi064.stdout:605: writing smithi06431743-31 from 1637706 to 2264460 tid 2 2024-03-04T06:10:36.289 INFO:tasks.rados.rados.0.smithi064.stdout:605: writing smithi06431743-31 from 2840948 to 2941258 tid 3 2024-03-04T06:10:36.289 INFO:tasks.rados.rados.0.smithi064.stdout:606: read oid 22 snap -1 2024-03-04T06:10:36.289 INFO:tasks.rados.rados.0.smithi064.stdout:606: expect (ObjNum 275 snap 72 seq_num 275) 2024-03-04T06:10:36.289 INFO:tasks.rados.rados.0.smithi064.stdout:607: read oid 12 snap 51 2024-03-04T06:10:36.289 INFO:tasks.rados.rados.0.smithi064.stdout:607: expect deleted 2024-03-04T06:10:36.289 INFO:tasks.rados.rados.0.smithi064.stdout:608: read oid 9 snap -1 2024-03-04T06:10:36.289 INFO:tasks.rados.rados.0.smithi064.stdout:608: expect (ObjNum 256 snap 70 seq_num 256) 2024-03-04T06:10:36.289 INFO:tasks.rados.rados.0.smithi064.stdout:609: write (excl) oid 24 current snap is 80 2024-03-04T06:10:36.289 INFO:tasks.rados.rados.0.smithi064.stdout:609: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-03-04T06:10:36.296 INFO:tasks.rados.rados.0.smithi064.stdout:609: writing smithi06431743-24 from 558065 to 1177934 tid 1 2024-03-04T06:10:36.303 INFO:tasks.rados.rados.0.smithi064.stdout:609: writing smithi06431743-24 from 1579835 to 2066600 tid 2 2024-03-04T06:10:36.309 INFO:tasks.rados.rados.0.smithi064.stdout:609: writing smithi06431743-24 from 2468371 to 2958065 tid 3 2024-03-04T06:10:36.309 INFO:tasks.rados.rados.0.smithi064.stdout:610: read oid 41 snap -1 2024-03-04T06:10:36.309 INFO:tasks.rados.rados.0.smithi064.stdout:610: expect (ObjNum 240 snap 64 seq_num 240) 2024-03-04T06:10:36.309 INFO:tasks.rados.rados.0.smithi064.stdout:611: read oid 17 snap -1 2024-03-04T06:10:36.309 INFO:tasks.rados.rados.0.smithi064.stdout:611: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:10:36.309 INFO:tasks.rados.rados.0.smithi064.stdout:612: delete oid 40 current snap is 80 2024-03-04T06:10:36.351 INFO:tasks.rados.rados.0.smithi064.stdout:605: finishing write tid 1 to smithi06431743-31 2024-03-04T06:10:36.359 INFO:tasks.rados.rados.0.smithi064.stdout:610: expect (ObjNum 240 snap 64 seq_num 240) 2024-03-04T06:10:36.481 INFO:tasks.rados.rados.0.smithi064.stdout:609: finishing write tid 1 to smithi06431743-24 2024-03-04T06:10:36.481 INFO:tasks.rados.rados.0.smithi064.stdout:606: expect (ObjNum 275 snap 72 seq_num 275) 2024-03-04T06:10:36.632 INFO:tasks.rados.rados.0.smithi064.stdout:605: finishing write tid 2 to smithi06431743-31 2024-03-04T06:10:36.632 INFO:tasks.rados.rados.0.smithi064.stdout:611: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:10:36.905 INFO:tasks.rados.rados.0.smithi064.stdout:603: finishing write tid 1 to smithi06431743-29 2024-03-04T06:10:36.906 INFO:tasks.rados.rados.0.smithi064.stdout:609: finishing write tid 2 to smithi06431743-24 2024-03-04T06:10:36.906 INFO:tasks.rados.rados.0.smithi064.stdout:608: expect (ObjNum 256 snap 70 seq_num 256) 2024-03-04T06:10:37.098 INFO:tasks.rados.rados.0.smithi064.stdout:603: finishing write tid 2 to smithi06431743-29 2024-03-04T06:10:37.098 INFO:tasks.rados.rados.0.smithi064.stdout:605: finishing write tid 3 to smithi06431743-31 2024-03-04T06:10:37.098 INFO:tasks.rados.rados.0.smithi064.stdout:603: finishing write tid 3 to smithi06431743-29 2024-03-04T06:10:37.098 INFO:tasks.rados.rados.0.smithi064.stdout:603: finishing write tid 4 to smithi06431743-29 2024-03-04T06:10:37.098 INFO:tasks.rados.rados.0.smithi064.stdout:603: oid 29 updating version 0 to 261 2024-03-04T06:10:37.098 INFO:tasks.rados.rados.0.smithi064.stdout:603: oid 29 version 261 is already newer than 259 2024-03-04T06:10:37.098 INFO:tasks.rados.rados.0.smithi064.stdout:603: oid 29 version 261 is already newer than 261 2024-03-04T06:10:37.098 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 261 (ObjNum 292 snap 80 seq_num 292) dirty exists 2024-03-04T06:10:37.098 INFO:tasks.rados.rados.0.smithi064.stdout:603: left oid 29 (ObjNum 292 snap 80 seq_num 292) 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:609: finishing write tid 3 to smithi06431743-24 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:603: done (9 left) 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:606: done (8 left) 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:607: done (7 left) 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:608: done (6 left) 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:610: done (5 left) 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:611: done (4 left) 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:612: done (3 left) 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:613: read oid 21 snap 67 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:613: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:614: write (excl) oid 29 current snap is 80 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:605: finishing write tid 4 to smithi06431743-31 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:605: finishing write tid 5 to smithi06431743-31 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:605: oid 31 updating version 0 to 60 2024-03-04T06:10:37.099 INFO:tasks.rados.rados.0.smithi064.stdout:605: oid 31 updating version 60 to 62 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:605: oid 31 version 62 is already newer than 61 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:605: oid 31 updating version 62 to 63 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 63 (ObjNum 294 snap 80 seq_num 294) dirty exists 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:605: left oid 31 (ObjNum 294 snap 80 seq_num 294) 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:609: finishing write tid 4 to smithi06431743-24 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:609: finishing write tid 5 to smithi06431743-24 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:609: oid 24 updating version 0 to 191 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:609: oid 24 updating version 191 to 193 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:609: oid 24 version 193 is already newer than 192 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:609: oid 24 updating version 193 to 194 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 194 (ObjNum 295 snap 80 seq_num 295) dirty exists 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:609: left oid 24 (ObjNum 295 snap 80 seq_num 295) 2024-03-04T06:10:37.100 INFO:tasks.rados.rados.0.smithi064.stdout:604: finishing write tid 1 to smithi06431743-19 2024-03-04T06:10:37.101 INFO:tasks.rados.rados.0.smithi064.stdout:614: seq_num 297 ranges {591679=486720,1813152=490434} 2024-03-04T06:10:37.103 INFO:tasks.rados.rados.0.smithi064.stdout:614: writing smithi06431743-29 from 591679 to 1078399 tid 1 2024-03-04T06:10:37.111 INFO:tasks.rados.rados.0.smithi064.stdout:614: writing smithi06431743-29 from 1813152 to 2303586 tid 2 2024-03-04T06:10:37.111 INFO:tasks.rados.rados.0.smithi064.stdout:605: done (4 left) 2024-03-04T06:10:37.111 INFO:tasks.rados.rados.0.smithi064.stdout:609: done (3 left) 2024-03-04T06:10:37.111 INFO:tasks.rados.rados.0.smithi064.stdout:615: delete oid 18 current snap is 80 2024-03-04T06:10:37.111 INFO:tasks.rados.rados.0.smithi064.stdout:604: finishing write tid 2 to smithi06431743-19 2024-03-04T06:10:37.111 INFO:tasks.rados.rados.0.smithi064.stdout:604: finishing write tid 3 to smithi06431743-19 2024-03-04T06:10:37.112 INFO:tasks.rados.rados.0.smithi064.stdout:604: finishing write tid 4 to smithi06431743-19 2024-03-04T06:10:37.112 INFO:tasks.rados.rados.0.smithi064.stdout:604: finishing write tid 5 to smithi06431743-19 2024-03-04T06:10:37.112 INFO:tasks.rados.rados.0.smithi064.stdout:604: oid 19 updating version 0 to 264 2024-03-04T06:10:37.112 INFO:tasks.rados.rados.0.smithi064.stdout:604: oid 19 version 264 is already newer than 263 2024-03-04T06:10:37.112 INFO:tasks.rados.rados.0.smithi064.stdout:604: oid 19 updating version 264 to 265 2024-03-04T06:10:37.112 INFO:tasks.rados.rados.0.smithi064.stdout:604: oid 19 updating version 265 to 266 2024-03-04T06:10:37.112 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 266 (ObjNum 293 snap 80 seq_num 293) dirty exists 2024-03-04T06:10:37.112 INFO:tasks.rados.rados.0.smithi064.stdout:604: left oid 19 (ObjNum 293 snap 80 seq_num 293) 2024-03-04T06:10:37.112 INFO:tasks.rados.rados.0.smithi064.stdout:613: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-04T06:10:37.199 INFO:tasks.rados.rados.0.smithi064.stdout:614: finishing write tid 1 to smithi06431743-29 2024-03-04T06:10:37.199 INFO:tasks.rados.rados.0.smithi064.stdout:614: finishing write tid 2 to smithi06431743-29 2024-03-04T06:10:37.199 INFO:tasks.rados.rados.0.smithi064.stdout:614: finishing write tid 3 to smithi06431743-29 2024-03-04T06:10:37.199 INFO:tasks.rados.rados.0.smithi064.stdout:614: finishing write tid 4 to smithi06431743-29 2024-03-04T06:10:37.199 INFO:tasks.rados.rados.0.smithi064.stdout:614: oid 29 updating version 0 to 270 2024-03-04T06:10:37.199 INFO:tasks.rados.rados.0.smithi064.stdout:614: oid 29 version 270 is already newer than 269 2024-03-04T06:10:37.199 INFO:tasks.rados.rados.0.smithi064.stdout:614: oid 29 updating version 270 to 271 2024-03-04T06:10:37.199 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 271 (ObjNum 297 snap 80 seq_num 297) dirty exists 2024-03-04T06:10:37.200 INFO:tasks.rados.rados.0.smithi064.stdout:614: left oid 29 (ObjNum 297 snap 80 seq_num 297) 2024-03-04T06:10:37.200 INFO:tasks.rados.rados.0.smithi064.stdout:604: done (3 left) 2024-03-04T06:10:37.200 INFO:tasks.rados.rados.0.smithi064.stdout:613: done (2 left) 2024-03-04T06:10:37.200 INFO:tasks.rados.rados.0.smithi064.stdout:614: done (1 left) 2024-03-04T06:10:37.200 INFO:tasks.rados.rados.0.smithi064.stdout:615: done (0 left) 2024-03-04T06:10:37.200 INFO:tasks.rados.rados.0.smithi064.stdout:616: write oid 6 current snap is 80 2024-03-04T06:10:37.200 INFO:tasks.rados.rados.0.smithi064.stdout:616: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-03-04T06:10:37.205 INFO:tasks.rados.rados.0.smithi064.stdout:616: writing smithi06431743-6 from 625293 to 1262511 tid 1 2024-03-04T06:10:37.213 INFO:tasks.rados.rados.0.smithi064.stdout:616: writing smithi06431743-6 from 1813763 to 2307866 tid 2 2024-03-04T06:10:37.217 INFO:tasks.rados.rados.0.smithi064.stdout:616: writing smithi06431743-6 from 2978063 to 3025293 tid 3 2024-03-04T06:10:37.218 INFO:tasks.rados.rados.0.smithi064.stdout:617: write oid 50 current snap is 80 2024-03-04T06:10:37.218 INFO:tasks.rados.rados.0.smithi064.stdout:617: seq_num 300 ranges {642100=712467,1872245=637761} 2024-03-04T06:10:37.226 INFO:tasks.rados.rados.0.smithi064.stdout:617: writing smithi06431743-50 from 642100 to 1354567 tid 1 2024-03-04T06:10:37.232 INFO:tasks.rados.rados.0.smithi064.stdout:617: writing smithi06431743-50 from 1872245 to 2510006 tid 2 2024-03-04T06:10:37.232 INFO:tasks.rados.rados.0.smithi064.stdout:618: delete oid 33 current snap is 80 2024-03-04T06:10:37.234 INFO:tasks.rados.rados.0.smithi064.stdout:616: finishing write tid 1 to smithi06431743-6 2024-03-04T06:10:37.257 INFO:tasks.rados.rados.0.smithi064.stdout:617: finishing write tid 1 to smithi06431743-50 2024-03-04T06:10:37.264 INFO:tasks.rados.rados.0.smithi064.stdout:616: finishing write tid 2 to smithi06431743-6 2024-03-04T06:10:37.274 INFO:tasks.rados.rados.0.smithi064.stdout:616: finishing write tid 3 to smithi06431743-6 2024-03-04T06:10:37.274 INFO:tasks.rados.rados.0.smithi064.stdout:617: finishing write tid 2 to smithi06431743-50 2024-03-04T06:10:37.280 INFO:tasks.rados.rados.0.smithi064.stdout:616: finishing write tid 4 to smithi06431743-6 2024-03-04T06:10:37.281 INFO:tasks.rados.rados.0.smithi064.stdout:616: finishing write tid 5 to smithi06431743-6 2024-03-04T06:10:37.281 INFO:tasks.rados.rados.0.smithi064.stdout:616: oid 6 updating version 0 to 104 2024-03-04T06:10:37.281 INFO:tasks.rados.rados.0.smithi064.stdout:616: oid 6 version 104 is already newer than 103 2024-03-04T06:10:37.281 INFO:tasks.rados.rados.0.smithi064.stdout:616: oid 6 version 104 is already newer than 101 2024-03-04T06:10:37.281 INFO:tasks.rados.rados.0.smithi064.stdout:616: oid 6 version 104 is already newer than 102 2024-03-04T06:10:37.281 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 104 (ObjNum 299 snap 80 seq_num 299) dirty exists 2024-03-04T06:10:37.282 INFO:tasks.rados.rados.0.smithi064.stdout:616: left oid 6 (ObjNum 299 snap 80 seq_num 299) 2024-03-04T06:10:37.282 INFO:tasks.rados.rados.0.smithi064.stdout:617: finishing write tid 3 to smithi06431743-50 2024-03-04T06:10:37.283 INFO:tasks.rados.rados.0.smithi064.stdout:617: finishing write tid 4 to smithi06431743-50 2024-03-04T06:10:37.283 INFO:tasks.rados.rados.0.smithi064.stdout:617: oid 50 updating version 0 to 238 2024-03-04T06:10:37.283 INFO:tasks.rados.rados.0.smithi064.stdout:617: oid 50 updating version 238 to 239 2024-03-04T06:10:37.283 INFO:tasks.rados.rados.0.smithi064.stdout:617: oid 50 updating version 239 to 240 2024-03-04T06:10:37.283 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 240 (ObjNum 300 snap 80 seq_num 300) dirty exists 2024-03-04T06:10:37.283 INFO:tasks.rados.rados.0.smithi064.stdout:617: left oid 50 (ObjNum 300 snap 80 seq_num 300) 2024-03-04T06:10:37.297 INFO:tasks.rados.rados.0.smithi064.stdout:616: done (2 left) 2024-03-04T06:10:37.297 INFO:tasks.rados.rados.0.smithi064.stdout:617: done (1 left) 2024-03-04T06:10:37.297 INFO:tasks.rados.rados.0.smithi064.stdout:618: done (0 left) 2024-03-04T06:10:37.297 INFO:tasks.rados.rados.0.smithi064.stdout:619: write oid 45 current snap is 80 2024-03-04T06:10:37.297 INFO:tasks.rados.rados.0.smithi064.stdout:619: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-03-04T06:10:37.303 INFO:tasks.rados.rados.0.smithi064.stdout:619: writing smithi06431743-45 from 675714 to 1138679 tid 1 2024-03-04T06:10:37.309 INFO:tasks.rados.rados.0.smithi064.stdout:619: writing smithi06431743-45 from 1589209 to 2230639 tid 2 2024-03-04T06:10:37.314 INFO:tasks.rados.rados.0.smithi064.stdout:619: writing smithi06431743-45 from 2943979 to 3075714 tid 3 2024-03-04T06:10:37.315 INFO:tasks.rados.rados.0.smithi064.stdout:620: read oid 2 snap -1 2024-03-04T06:10:37.315 INFO:tasks.rados.rados.0.smithi064.stdout:620: expect (ObjNum 285 snap 78 seq_num 285) 2024-03-04T06:10:37.315 INFO:tasks.rados.rados.0.smithi064.stdout:621: write (excl) oid 1 current snap is 80 2024-03-04T06:10:37.315 INFO:tasks.rados.rados.0.smithi064.stdout:621: seq_num 303 ranges {692521=538214,1931338=785088} 2024-03-04T06:10:37.321 INFO:tasks.rados.rados.0.smithi064.stdout:621: writing smithi06431743-1 from 692521 to 1230735 tid 1 2024-03-04T06:10:37.330 INFO:tasks.rados.rados.0.smithi064.stdout:621: writing smithi06431743-1 from 1931338 to 2716426 tid 2 2024-03-04T06:10:37.330 INFO:tasks.rados.rados.0.smithi064.stdout:622: read oid 11 snap 78 2024-03-04T06:10:37.330 INFO:tasks.rados.rados.0.smithi064.stdout:622: expect (ObjNum 214 snap 46 seq_num 214) 2024-03-04T06:10:37.330 INFO:tasks.rados.rados.0.smithi064.stdout:623: delete oid 7 current snap is 80 2024-03-04T06:10:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:619: finishing write tid 1 to smithi06431743-45 2024-03-04T06:10:37.337 INFO:tasks.rados.rados.0.smithi064.stdout:620: expect (ObjNum 285 snap 78 seq_num 285) 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:619: finishing write tid 2 to smithi06431743-45 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:621: finishing write tid 1 to smithi06431743-1 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:619: finishing write tid 3 to smithi06431743-45 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:621: finishing write tid 2 to smithi06431743-1 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:619: finishing write tid 4 to smithi06431743-45 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:620: done (4 left) 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:623: done (3 left) 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:624: write oid 3 current snap is 80 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:619: finishing write tid 5 to smithi06431743-45 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:619: oid 45 updating version 0 to 143 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:619: oid 45 updating version 143 to 145 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:619: oid 45 version 145 is already newer than 144 2024-03-04T06:10:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:619: oid 45 version 145 is already newer than 142 2024-03-04T06:10:37.449 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 145 (ObjNum 302 snap 80 seq_num 302) dirty exists 2024-03-04T06:10:37.449 INFO:tasks.rados.rados.0.smithi064.stdout:619: left oid 45 (ObjNum 302 snap 80 seq_num 302) 2024-03-04T06:10:37.449 INFO:tasks.rados.rados.0.smithi064.stdout:621: finishing write tid 3 to smithi06431743-1 2024-03-04T06:10:37.449 INFO:tasks.rados.rados.0.smithi064.stdout:624: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-03-04T06:10:37.453 INFO:tasks.rados.rados.0.smithi064.stdout:624: writing smithi06431743-3 from 726135 to 1131200 tid 1 2024-03-04T06:10:37.461 INFO:tasks.rados.rados.0.smithi064.stdout:624: writing smithi06431743-3 from 1764655 to 2553412 tid 2 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:624: writing smithi06431743-3 from 3026248 to 3126135 tid 3 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:619: done (3 left) 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:625: read oid 44 snap -1 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:625: expect (ObjNum 263 snap 70 seq_num 263) 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:626: snap_create 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:621: finishing write tid 4 to smithi06431743-1 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:621: oid 1 updating version 0 to 241 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:621: oid 1 updating version 241 to 242 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:621: oid 1 version 242 is already newer than 240 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 242 (ObjNum 303 snap 80 seq_num 303) dirty exists 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:621: left oid 1 (ObjNum 303 snap 80 seq_num 303) 2024-03-04T06:10:37.465 INFO:tasks.rados.rados.0.smithi064.stdout:622: expect (ObjNum 214 snap 46 seq_num 214) 2024-03-04T06:10:37.615 INFO:tasks.rados.rados.0.smithi064.stdout:624: finishing write tid 1 to smithi06431743-3 2024-03-04T06:10:37.615 INFO:tasks.rados.rados.0.smithi064.stdout:625: expect (ObjNum 263 snap 70 seq_num 263) 2024-03-04T06:10:37.774 INFO:tasks.rados.rados.0.smithi064.stdout:624: finishing write tid 2 to smithi06431743-3 2024-03-04T06:10:37.775 INFO:tasks.rados.rados.0.smithi064.stdout:624: finishing write tid 3 to smithi06431743-3 2024-03-04T06:10:37.775 INFO:tasks.rados.rados.0.smithi064.stdout:624: finishing write tid 4 to smithi06431743-3 2024-03-04T06:10:37.775 INFO:tasks.rados.rados.0.smithi064.stdout:624: finishing write tid 5 to smithi06431743-3 2024-03-04T06:10:37.775 INFO:tasks.rados.rados.0.smithi064.stdout:624: oid 3 updating version 0 to 101 2024-03-04T06:10:37.775 INFO:tasks.rados.rados.0.smithi064.stdout:624: oid 3 version 101 is already newer than 99 2024-03-04T06:10:37.775 INFO:tasks.rados.rados.0.smithi064.stdout:624: oid 3 updating version 101 to 102 2024-03-04T06:10:37.775 INFO:tasks.rados.rados.0.smithi064.stdout:624: oid 3 version 102 is already newer than 100 2024-03-04T06:10:37.775 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 102 (ObjNum 305 snap 80 seq_num 305) dirty exists 2024-03-04T06:10:37.775 INFO:tasks.rados.rados.0.smithi064.stdout:624: left oid 3 (ObjNum 305 snap 80 seq_num 305) 2024-03-04T06:10:38.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:38.252 INFO:tasks.rados.rados.0.smithi064.stdout:621: done (4 left) 2024-03-04T06:10:38.252 INFO:tasks.rados.rados.0.smithi064.stdout:622: done (3 left) 2024-03-04T06:10:38.252 INFO:tasks.rados.rados.0.smithi064.stdout:624: done (2 left) 2024-03-04T06:10:38.252 INFO:tasks.rados.rados.0.smithi064.stdout:625: done (1 left) 2024-03-04T06:10:38.252 INFO:tasks.rados.rados.0.smithi064.stdout:626: done (0 left) 2024-03-04T06:10:38.252 INFO:tasks.rados.rados.0.smithi064.stdout:627: delete oid 4 current snap is 81 2024-03-04T06:10:38.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:38.273+0000 7f24ef59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:38.277 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:38.277+0000 7f24ef59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:38.289 INFO:tasks.rados.rados.0.smithi064.stdout:627: done (0 left) 2024-03-04T06:10:38.289 INFO:tasks.rados.rados.0.smithi064.stdout:628: delete oid 1 current snap is 81 2024-03-04T06:10:38.309 INFO:tasks.rados.rados.0.smithi064.stdout:628: done (0 left) 2024-03-04T06:10:38.309 INFO:tasks.rados.rados.0.smithi064.stdout:629: read oid 41 snap 51 2024-03-04T06:10:38.309 INFO:tasks.rados.rados.0.smithi064.stdout:629: expect (ObjNum 113 snap 10 seq_num 113) 2024-03-04T06:10:38.309 INFO:tasks.rados.rados.0.smithi064.stdout:630: read oid 28 snap -1 2024-03-04T06:10:38.309 INFO:tasks.rados.rados.0.smithi064.stdout:630: expect (ObjNum 282 snap 76 seq_num 282) 2024-03-04T06:10:38.309 INFO:tasks.rados.rados.0.smithi064.stdout:631: snap_remove snap 67 2024-03-04T06:10:39.257 INFO:tasks.rados.rados.0.smithi064.stdout:631: done (2 left) 2024-03-04T06:10:39.258 INFO:tasks.rados.rados.0.smithi064.stdout:632: delete oid 4 current snap is 81 2024-03-04T06:10:39.258 INFO:tasks.rados.rados.0.smithi064.stdout:630: expect (ObjNum 282 snap 76 seq_num 282) 2024-03-04T06:10:39.375 INFO:tasks.rados.rados.0.smithi064.stdout:629: expect (ObjNum 113 snap 10 seq_num 113) 2024-03-04T06:10:39.572 INFO:tasks.rados.rados.0.smithi064.stdout:629: done (2 left) 2024-03-04T06:10:39.572 INFO:tasks.rados.rados.0.smithi064.stdout:630: done (1 left) 2024-03-04T06:10:39.572 INFO:tasks.rados.rados.0.smithi064.stdout:632: done (0 left) 2024-03-04T06:10:39.572 INFO:tasks.rados.rados.0.smithi064.stdout:633: write oid 44 current snap is 81 2024-03-04T06:10:39.572 INFO:tasks.rados.rados.0.smithi064.stdout:633: seq_num 309 ranges {793363=706061,2282230=796095} 2024-03-04T06:10:39.580 INFO:tasks.rados.rados.0.smithi064.stdout:633: writing smithi06431743-44 from 793363 to 1499424 tid 1 2024-03-04T06:10:39.589 INFO:tasks.rados.rados.0.smithi064.stdout:633: writing smithi06431743-44 from 2282230 to 3078325 tid 2 2024-03-04T06:10:39.589 INFO:tasks.rados.rados.0.smithi064.stdout:634: delete oid 19 current snap is 81 2024-03-04T06:10:39.610 INFO:tasks.rados.rados.0.smithi064.stdout:633: finishing write tid 1 to smithi06431743-44 2024-03-04T06:10:39.613 INFO:tasks.rados.rados.0.smithi064.stdout:634: done (1 left) 2024-03-04T06:10:39.613 INFO:tasks.rados.rados.0.smithi064.stdout:635: read oid 42 snap -1 2024-03-04T06:10:39.613 INFO:tasks.rados.rados.0.smithi064.stdout:635: expect deleted 2024-03-04T06:10:39.613 INFO:tasks.rados.rados.0.smithi064.stdout:636: read oid 13 snap -1 2024-03-04T06:10:39.614 INFO:tasks.rados.rados.0.smithi064.stdout:636: expect deleted 2024-03-04T06:10:39.614 INFO:tasks.rados.rados.0.smithi064.stdout:637: read oid 15 snap -1 2024-03-04T06:10:39.614 INFO:tasks.rados.rados.0.smithi064.stdout:637: expect (ObjNum 196 snap 40 seq_num 196) 2024-03-04T06:10:39.614 INFO:tasks.rados.rados.0.smithi064.stdout:638: read oid 22 snap -1 2024-03-04T06:10:39.614 INFO:tasks.rados.rados.0.smithi064.stdout:638: expect (ObjNum 275 snap 72 seq_num 275) 2024-03-04T06:10:39.614 INFO:tasks.rados.rados.0.smithi064.stdout:639: write (excl) oid 32 current snap is 81 2024-03-04T06:10:39.614 INFO:tasks.rados.rados.0.smithi064.stdout:639: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-03-04T06:10:39.618 INFO:tasks.rados.rados.0.smithi064.stdout:639: writing smithi06431743-32 from 426977 to 883536 tid 1 2024-03-04T06:10:39.625 INFO:tasks.rados.rados.0.smithi064.stdout:639: writing smithi06431743-32 from 1482841 to 2282605 tid 2 2024-03-04T06:10:39.631 INFO:tasks.rados.rados.0.smithi064.stdout:639: writing smithi06431743-32 from 2958080 to 3226977 tid 3 2024-03-04T06:10:39.631 INFO:tasks.rados.rados.0.smithi064.stdout:640: delete oid 31 current snap is 81 2024-03-04T06:10:39.635 INFO:tasks.rados.rados.0.smithi064.stdout:633: finishing write tid 2 to smithi06431743-44 2024-03-04T06:10:39.639 INFO:tasks.rados.rados.0.smithi064.stdout:638: expect (ObjNum 275 snap 72 seq_num 275) 2024-03-04T06:10:39.818 INFO:tasks.rados.rados.0.smithi064.stdout:633: finishing write tid 3 to smithi06431743-44 2024-03-04T06:10:39.818 INFO:tasks.rados.rados.0.smithi064.stdout:633: finishing write tid 4 to smithi06431743-44 2024-03-04T06:10:39.818 INFO:tasks.rados.rados.0.smithi064.stdout:633: oid 44 updating version 0 to 88 2024-03-04T06:10:39.818 INFO:tasks.rados.rados.0.smithi064.stdout:633: oid 44 updating version 88 to 89 2024-03-04T06:10:39.818 INFO:tasks.rados.rados.0.smithi064.stdout:633: oid 44 updating version 89 to 90 2024-03-04T06:10:39.818 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 90 (ObjNum 309 snap 81 seq_num 309) dirty exists 2024-03-04T06:10:39.818 INFO:tasks.rados.rados.0.smithi064.stdout:633: left oid 44 (ObjNum 309 snap 81 seq_num 309) 2024-03-04T06:10:39.818 INFO:tasks.rados.rados.0.smithi064.stdout:633: done (6 left) 2024-03-04T06:10:39.818 INFO:tasks.rados.rados.0.smithi064.stdout:635: done (5 left) 2024-03-04T06:10:39.818 INFO:tasks.rados.rados.0.smithi064.stdout:636: done (4 left) 2024-03-04T06:10:39.818 INFO:tasks.rados.rados.0.smithi064.stdout:638: done (3 left) 2024-03-04T06:10:39.819 INFO:tasks.rados.rados.0.smithi064.stdout:640: done (2 left) 2024-03-04T06:10:39.819 INFO:tasks.rados.rados.0.smithi064.stdout:641: read oid 48 snap 63 2024-03-04T06:10:39.819 INFO:tasks.rados.rados.0.smithi064.stdout:641: expect deleted 2024-03-04T06:10:39.819 INFO:tasks.rados.rados.0.smithi064.stdout:642: delete oid 49 current snap is 81 2024-03-04T06:10:39.819 INFO:tasks.rados.rados.0.smithi064.stdout:637: expect (ObjNum 196 snap 40 seq_num 196) 2024-03-04T06:10:39.941 INFO:tasks.rados.rados.0.smithi064.stdout:639: finishing write tid 1 to smithi06431743-32 2024-03-04T06:10:39.941 INFO:tasks.rados.rados.0.smithi064.stdout:639: finishing write tid 2 to smithi06431743-32 2024-03-04T06:10:39.941 INFO:tasks.rados.rados.0.smithi064.stdout:639: finishing write tid 3 to smithi06431743-32 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:639: finishing write tid 4 to smithi06431743-32 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:639: finishing write tid 5 to smithi06431743-32 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:639: oid 32 updating version 0 to 252 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:639: oid 32 updating version 252 to 253 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:639: oid 32 version 253 is already newer than 252 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:639: oid 32 version 253 is already newer than 250 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 253 (ObjNum 311 snap 81 seq_num 311) dirty exists 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:639: left oid 32 (ObjNum 311 snap 81 seq_num 311) 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:637: done (3 left) 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:639: done (2 left) 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:642: done (1 left) 2024-03-04T06:10:39.942 INFO:tasks.rados.rados.0.smithi064.stdout:643: snap_remove snap 75 2024-03-04T06:10:40.257 INFO:tasks.rados.rados.0.smithi064.stdout:643: done (1 left) 2024-03-04T06:10:40.257 INFO:tasks.rados.rados.0.smithi064.stdout:644: snap_remove snap 73 2024-03-04T06:10:40.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:40.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:40.481+0000 7fbccfa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:40.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:40.486+0000 7fbccfa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:40.552 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:10:40.552 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:10:40.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:10:40.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:40.655+0000 7fb90826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:40.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:40.662+0000 7fb90826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:40.855 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:10:40.855 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:10:40.866 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-03-04T06:10:40.741846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":96525450,"num_objects":226,"num_object_clones":159,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":31,"num_read":1325,"num_read_kb":855021,"num_write":1201,"num_write_kb":295718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2179,"ondisk_log_size":2179,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":515232,"kb_used_data":391296,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":376972128,"statfs":{"total":386547056640,"available":386019459072,"internally_reserved":0,"allocated":400687104,"data_stored":444659266,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13714578,"num_objects":13,"num_object_clones":13,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":63,"num_read_kb":45985,"num_write":38,"num_write_kb":10862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.500140"},"pg_stats":[{"pgid":"3.f","version":"171'110","reported_seq":531,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.280792+0000","last_change":"2024-03-04T06:10:40.296818+0000","last_active":"2024-03-04T06:10:40.280792+0000","last_peered":"2024-03-04T06:10:40.280792+0000","last_clean":"2024-03-04T06:10:40.280792+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:10:40.280792+0000","last_undegraded":"2024-03-04T06:10:40.280792+0000","last_fullsized":"2024-03-04T06:10:40.280792+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731154,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":30,"num_read_kb":6459,"num_write":67,"num_write_kb":17547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"170'94","reported_seq":540,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.271613+0000","last_change":"2024-03-04T06:10:39.321718+0000","last_active":"2024-03-04T06:10:39.271613+0000","last_peered":"2024-03-04T06:10:39.271613+0000","last_clean":"2024-03-04T06:10:39.271613+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:10:39.271613+0000","last_undegraded":"2024-03-04T06:10:39.271613+0000","last_fullsized":"2024-03-04T06:10:39.271613+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":36,"num_read_kb":21069,"num_write":45,"num_write_kb":11808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"159'140","reported_seq":552,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.332928+0000","last_change":"2024-03-04T06:10:35.262726+0000","last_active":"2024-03-04T06:10:36.332928+0000","last_peered":"2024-03-04T06:10:36.332928+0000","last_clean":"2024-03-04T06:10:36.332928+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:10:36.332928+0000","last_undegraded":"2024-03-04T06:10:36.332928+0000","last_fullsized":"2024-03-04T06:10:36.332928+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11126300,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":72,"num_read_kb":41339,"num_write":77,"num_write_kb":21678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"170'105","reported_seq":548,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.273289+0000","last_change":"2024-03-04T06:10:40.275741+0000","last_active":"2024-03-04T06:10:40.273289+0000","last_peered":"2024-03-04T06:10:40.273289+0000","last_clean":"2024-03-04T06:10:40.273289+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:10:40.273289+0000","last_undegraded":"2024-03-04T06:10:40.273289+0000","last_fullsized":"2024-03-04T06:10:40.273289+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2789995,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":79,"num_read_kb":67437,"num_write":58,"num_write_kb":16172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"168'60","reported_seq":500,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.286683+0000","last_change":"2024-03-04T06:10:34.264903+0000","last_active":"2024-03-04T06:10:36.286683+0000","last_peered":"2024-03-04T06:10:36.286683+0000","last_clean":"2024-03-04T06:10:36.286683+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:10:36.286683+0000","last_undegraded":"2024-03-04T06:10:36.286683+0000","last_fullsized":"2024-03-04T06:10:36.286683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2437048,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":17039,"num_write":25,"num_write_kb":7377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"171'92","reported_seq":534,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.279440+0000","last_change":"2024-03-04T06:10:40.296522+0000","last_active":"2024-03-04T06:10:40.279440+0000","last_peered":"2024-03-04T06:10:40.279440+0000","last_clean":"2024-03-04T06:10:40.279440+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:10:40.279440+0000","last_undegraded":"2024-03-04T06:10:40.279440+0000","last_fullsized":"2024-03-04T06:10:40.279440+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8235496,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":46529,"num_write":53,"num_write_kb":16061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":460,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.278032+0000","last_change":"2024-03-04T06:10:40.284092+0000","last_active":"2024-03-04T06:10:40.278032+0000","last_peered":"2024-03-04T06:10:40.278032+0000","last_clean":"2024-03-04T06:10:40.278032+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:10:40.278032+0000","last_undegraded":"2024-03-04T06:10:40.278032+0000","last_fullsized":"2024-03-04T06:10:40.278032+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"158'109","reported_seq":526,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.335707+0000","last_change":"2024-03-04T06:10:35.262305+0000","last_active":"2024-03-04T06:10:36.335707+0000","last_peered":"2024-03-04T06:10:36.335707+0000","last_clean":"2024-03-04T06:10:36.335707+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:10:36.335707+0000","last_undegraded":"2024-03-04T06:10:36.335707+0000","last_fullsized":"2024-03-04T06:10:36.335707+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5798448,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":51,"num_read_kb":39607,"num_write":57,"num_write_kb":16367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":537,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.262331+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:10:36.262331+0000","last_peered":"2024-03-04T06:10:36.262331+0000","last_clean":"2024-03-04T06:10:36.262331+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:10:36.262331+0000","last_undegraded":"2024-03-04T06:10:36.262331+0000","last_fullsized":"2024-03-04T06:10:36.262331+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":392,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.343300+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:10:36.343300+0000","last_peered":"2024-03-04T06:10:36.343300+0000","last_clean":"2024-03-04T06:10:36.343300+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:10:36.343300+0000","last_undegraded":"2024-03-04T06:10:36.343300+0000","last_fullsized":"2024-03-04T06:10:36.343300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"158'97","reported_seq":524,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.340877+0000","last_change":"2024-03-04T06:10:35.269122+0000","last_active":"2024-03-04T06:10:36.340877+0000","last_peered":"2024-03-04T06:10:36.340877+0000","last_clean":"2024-03-04T06:10:36.340877+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:10:36.340877+0000","last_undegraded":"2024-03-04T06:10:36.340877+0000","last_fullsized":"2024-03-04T06:10:36.340877+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7630444,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":69,"num_read_kb":62573,"num_write":44,"num_write_kb":11786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":397,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.276068+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:10:40.276068+0000","last_peered":"2024-03-04T06:10:40.276068+0000","last_clean":"2024-03-04T06:10:40.276068+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:10:40.276068+0000","last_undegraded":"2024-03-04T06:10:40.276068+0000","last_fullsized":"2024-03-04T06:10:40.276068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"158'121","reported_seq":538,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.264334+0000","last_change":"2024-03-04T06:10:34.254757+0000","last_active":"2024-03-04T06:10:36.264334+0000","last_peered":"2024-03-04T06:10:36.264334+0000","last_clean":"2024-03-04T06:10:36.264334+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:10:36.264334+0000","last_undegraded":"2024-03-04T06:10:36.264334+0000","last_fullsized":"2024-03-04T06:10:36.264334+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487469,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":67,"num_read_kb":54041,"num_write":48,"num_write_kb":13998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":392,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.332229+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:10:36.332229+0000","last_peered":"2024-03-04T06:10:36.332229+0000","last_clean":"2024-03-04T06:10:36.332229+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:10:36.332229+0000","last_undegraded":"2024-03-04T06:10:36.332229+0000","last_fullsized":"2024-03-04T06:10:36.332229+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"166'238","reported_seq":664,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.278808+0000","last_change":"2024-03-04T06:10:34.274204+0000","last_active":"2024-03-04T06:10:36.278808+0000","last_peered":"2024-03-04T06:10:36.278808+0000","last_clean":"2024-03-04T06:10:36.278808+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:10:36.278808+0000","last_undegraded":"2024-03-04T06:10:36.278808+0000","last_fullsized":"2024-03-04T06:10:36.278808+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14840746,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":177,"num_read_kb":130029,"num_write":136,"num_write_kb":39117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":397,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.267558+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:10:40.267558+0000","last_peered":"2024-03-04T06:10:40.267558+0000","last_clean":"2024-03-04T06:10:40.267558+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:10:40.267558+0000","last_undegraded":"2024-03-04T06:10:40.267558+0000","last_fullsized":"2024-03-04T06:10:40.267558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"170'279","reported_seq":657,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.356825+0000","last_change":"2024-03-04T06:10:39.356825+0000","last_active":"2024-03-04T06:10:39.356825+0000","last_peered":"2024-03-04T06:10:39.356825+0000","last_clean":"2024-03-04T06:10:39.356825+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:10:39.356825+0000","last_undegraded":"2024-03-04T06:10:39.356825+0000","last_fullsized":"2024-03-04T06:10:39.356825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8689318,"num_objects":28,"num_object_clones":21,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":105,"num_read_kb":57467,"num_write":147,"num_write_kb":39214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"167'99","reported_seq":546,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.342269+0000","last_change":"2024-03-04T06:10:35.290138+0000","last_active":"2024-03-04T06:10:36.342269+0000","last_peered":"2024-03-04T06:10:36.342269+0000","last_clean":"2024-03-04T06:10:36.342269+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:10:36.342269+0000","last_undegraded":"2024-03-04T06:10:36.342269+0000","last_fullsized":"2024-03-04T06:10:36.342269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168202,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":99,"num_read_kb":72498,"num_write":54,"num_write_kb":13614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":424,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.278278+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:10:39.278278+0000","last_peered":"2024-03-04T06:10:39.278278+0000","last_clean":"2024-03-04T06:10:39.278278+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:10:39.278278+0000","last_undegraded":"2024-03-04T06:10:39.278278+0000","last_fullsized":"2024-03-04T06:10:39.278278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"170'99","reported_seq":561,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.285027+0000","last_change":"2024-03-04T06:10:39.323767+0000","last_active":"2024-03-04T06:10:39.285027+0000","last_peered":"2024-03-04T06:10:39.285027+0000","last_clean":"2024-03-04T06:10:39.285027+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:10:39.285027+0000","last_undegraded":"2024-03-04T06:10:39.285027+0000","last_fullsized":"2024-03-04T06:10:39.285027+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781575,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":74,"num_read_kb":43419,"num_write":48,"num_write_kb":12439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":404,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.277500+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:10:36.277500+0000","last_peered":"2024-03-04T06:10:36.277500+0000","last_clean":"2024-03-04T06:10:36.277500+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:10:36.277500+0000","last_undegraded":"2024-03-04T06:10:36.277500+0000","last_fullsized":"2024-03-04T06:10:36.277500+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"158'189","reported_seq":598,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.265367+0000","last_change":"2024-03-04T06:10:34.254934+0000","last_active":"2024-03-04T06:10:36.265367+0000","last_peered":"2024-03-04T06:10:36.265367+0000","last_clean":"2024-03-04T06:10:36.265367+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:10:36.265367+0000","last_undegraded":"2024-03-04T06:10:36.265367+0000","last_fullsized":"2024-03-04T06:10:36.265367+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10252270,"num_objects":19,"num_object_clones":14,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":108,"num_read_kb":72913,"num_write":97,"num_write_kb":27064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":424,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.274282+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:10:39.274282+0000","last_peered":"2024-03-04T06:10:39.274282+0000","last_clean":"2024-03-04T06:10:39.274282+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:10:39.274282+0000","last_undegraded":"2024-03-04T06:10:39.274282+0000","last_fullsized":"2024-03-04T06:10:39.274282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"170'250","reported_seq":657,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.633179+0000","last_change":"2024-03-04T06:10:39.355324+0000","last_active":"2024-03-04T06:10:39.633179+0000","last_peered":"2024-03-04T06:10:39.633179+0000","last_clean":"2024-03-04T06:10:39.633179+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:10:39.633179+0000","last_undegraded":"2024-03-04T06:10:39.633179+0000","last_fullsized":"2024-03-04T06:10:39.633179+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7966617,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":164,"num_read_kb":122511,"num_write":108,"num_write_kb":29582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":392,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-04T06:10:36.339574+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:10:36.339574+0000","last_peered":"2024-03-04T06:10:36.339574+0000","last_clean":"2024-03-04T06:10:36.339574+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:10:36.339574+0000","last_undegraded":"2024-03-04T06:10:36.339574+0000","last_fullsized":"2024-03-04T06:10:36.339574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":95935082,"num_objects":224,"num_object_clones":159,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":31,"num_read":1222,"num_read_kb":854930,"num_write":1064,"num_write_kb":293824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2082,"ondisk_log_size":2082,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":147920,"kb_used_data":116936,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94223920,"statfs":{"total":96636764160,"available":96485294080,"internally_reserved":0,"allocated":119742464,"data_stored":132934898,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":129508,"kb_used_data":98524,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94242332,"statfs":{"total":96636764160,"available":96504147968,"internally_reserved":0,"allocated":100888576,"data_stored":111144622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":119104,"kb_used_data":88120,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94252736,"statfs":{"total":96636764160,"available":96514801664,"internally_reserved":0,"allocated":90234880,"data_stored":99416009,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":118700,"kb_used_data":87716,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94253140,"statfs":{"total":96636764160,"available":96515215360,"internally_reserved":0,"allocated":89821184,"data_stored":101163737,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:10:41.257 INFO:tasks.rados.rados.0.smithi064.stdout:644: done (1 left) 2024-03-04T06:10:41.257 INFO:tasks.rados.rados.0.smithi064.stdout:645: read oid 5 snap -1 2024-03-04T06:10:41.257 INFO:tasks.rados.rados.0.smithi064.stdout:645: expect (ObjNum 289 snap 78 seq_num 289) 2024-03-04T06:10:41.258 INFO:tasks.rados.rados.0.smithi064.stdout:646: write oid 6 current snap is 81 2024-03-04T06:10:41.258 INFO:tasks.rados.rados.0.smithi064.stdout:646: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-03-04T06:10:41.265 INFO:tasks.rados.rados.0.smithi064.stdout:646: writing smithi06431743-6 from 477398 to 1276057 tid 1 2024-03-04T06:10:41.272 INFO:tasks.rados.rados.0.smithi064.stdout:646: writing smithi06431743-6 from 1774640 to 2438084 tid 2 2024-03-04T06:10:41.277 INFO:tasks.rados.rados.0.smithi064.stdout:646: writing smithi06431743-6 from 2873055 to 3277398 tid 3 2024-03-04T06:10:41.277 INFO:tasks.rados.rados.0.smithi064.stdout:647: delete oid 40 current snap is 81 2024-03-04T06:10:41.292 INFO:tasks.rados.rados.0.smithi064.stdout:641: done (3 left) 2024-03-04T06:10:41.292 INFO:tasks.rados.rados.0.smithi064.stdout:647: done (2 left) 2024-03-04T06:10:41.292 INFO:tasks.rados.rados.0.smithi064.stdout:648: snap_remove snap 79 2024-03-04T06:10:42.267 INFO:tasks.rados.rados.0.smithi064.stdout:648: done (2 left) 2024-03-04T06:10:42.267 INFO:tasks.rados.rados.0.smithi064.stdout:649: write oid 37 current snap is 81 2024-03-04T06:10:42.267 INFO:tasks.rados.rados.0.smithi064.stdout:649: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-03-04T06:10:42.273 INFO:tasks.rados.rados.0.smithi064.stdout:649: writing smithi06431743-37 from 511012 to 1060169 tid 1 2024-03-04T06:10:42.279 INFO:tasks.rados.rados.0.smithi064.stdout:649: writing smithi06431743-37 from 1775251 to 2326011 tid 2 2024-03-04T06:10:42.285 INFO:tasks.rados.rados.0.smithi064.stdout:649: writing smithi06431743-37 from 3095195 to 3311012 tid 3 2024-03-04T06:10:42.285 INFO:tasks.rados.rados.0.smithi064.stdout:650: delete oid 45 current snap is 81 2024-03-04T06:10:42.286 INFO:tasks.rados.rados.0.smithi064.stdout:645: expect (ObjNum 289 snap 78 seq_num 289) 2024-03-04T06:10:42.428 INFO:tasks.rados.rados.0.smithi064.stdout:646: finishing write tid 1 to smithi06431743-6 2024-03-04T06:10:42.428 INFO:tasks.rados.rados.0.smithi064.stdout:646: finishing write tid 2 to smithi06431743-6 2024-03-04T06:10:42.428 INFO:tasks.rados.rados.0.smithi064.stdout:646: finishing write tid 3 to smithi06431743-6 2024-03-04T06:10:42.428 INFO:tasks.rados.rados.0.smithi064.stdout:646: finishing write tid 4 to smithi06431743-6 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:646: finishing write tid 5 to smithi06431743-6 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:646: oid 6 updating version 0 to 115 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:646: oid 6 updating version 115 to 117 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:646: oid 6 version 117 is already newer than 114 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:646: oid 6 version 117 is already newer than 117 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 117 (ObjNum 314 snap 81 seq_num 314) dirty exists 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:646: left oid 6 (ObjNum 314 snap 81 seq_num 314) 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:645: done (3 left) 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:646: done (2 left) 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:650: done (1 left) 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:651: read oid 17 snap -1 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:651: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:652: delete oid 13 current snap is 81 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:649: finishing write tid 1 to smithi06431743-37 2024-03-04T06:10:42.429 INFO:tasks.rados.rados.0.smithi064.stdout:649: finishing write tid 2 to smithi06431743-37 2024-03-04T06:10:42.430 INFO:tasks.rados.rados.0.smithi064.stdout:649: finishing write tid 3 to smithi06431743-37 2024-03-04T06:10:42.430 INFO:tasks.rados.rados.0.smithi064.stdout:649: finishing write tid 4 to smithi06431743-37 2024-03-04T06:10:42.430 INFO:tasks.rados.rados.0.smithi064.stdout:649: finishing write tid 5 to smithi06431743-37 2024-03-04T06:10:42.430 INFO:tasks.rados.rados.0.smithi064.stdout:649: oid 37 updating version 0 to 101 2024-03-04T06:10:42.430 INFO:tasks.rados.rados.0.smithi064.stdout:649: oid 37 version 101 is already newer than 100 2024-03-04T06:10:42.430 INFO:tasks.rados.rados.0.smithi064.stdout:649: oid 37 version 101 is already newer than 100 2024-03-04T06:10:42.430 INFO:tasks.rados.rados.0.smithi064.stdout:649: oid 37 updating version 101 to 102 2024-03-04T06:10:42.430 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 102 (ObjNum 316 snap 81 seq_num 316) dirty exists 2024-03-04T06:10:42.430 INFO:tasks.rados.rados.0.smithi064.stdout:649: left oid 37 (ObjNum 316 snap 81 seq_num 316) 2024-03-04T06:10:42.431 INFO:tasks.rados.rados.0.smithi064.stdout:649: done (2 left) 2024-03-04T06:10:42.431 INFO:tasks.rados.rados.0.smithi064.stdout:652: done (1 left) 2024-03-04T06:10:42.431 INFO:tasks.rados.rados.0.smithi064.stdout:653: read oid 13 snap -1 2024-03-04T06:10:42.431 INFO:tasks.rados.rados.0.smithi064.stdout:653: expect deleted 2024-03-04T06:10:42.431 INFO:tasks.rados.rados.0.smithi064.stdout:654: delete oid 16 current snap is 81 2024-03-04T06:10:42.438 INFO:tasks.rados.rados.0.smithi064.stdout:653: done (2 left) 2024-03-04T06:10:42.438 INFO:tasks.rados.rados.0.smithi064.stdout:654: done (1 left) 2024-03-04T06:10:42.438 INFO:tasks.rados.rados.0.smithi064.stdout:655: read oid 38 snap -1 2024-03-04T06:10:42.438 INFO:tasks.rados.rados.0.smithi064.stdout:655: expect (ObjNum 273 snap 70 seq_num 273) 2024-03-04T06:10:42.438 INFO:tasks.rados.rados.0.smithi064.stdout:656: snap_remove snap 63 2024-03-04T06:10:42.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:42.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:42.657+0000 7f71f191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:42.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:42.661+0000 7f71f191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:43.193 INFO:tasks.rados.rados.0.smithi064.stdout:656: done (2 left) 2024-03-04T06:10:43.194 INFO:tasks.rados.rados.0.smithi064.stdout:657: read oid 40 snap -1 2024-03-04T06:10:43.194 INFO:tasks.rados.rados.0.smithi064.stdout:657: expect deleted 2024-03-04T06:10:43.194 INFO:tasks.rados.rados.0.smithi064.stdout:658: write oid 24 current snap is 81 2024-03-04T06:10:43.194 INFO:tasks.rados.rados.0.smithi064.stdout:658: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-03-04T06:10:43.199 INFO:tasks.rados.rados.0.smithi064.stdout:658: writing smithi06431743-24 from 578240 to 1144746 tid 1 2024-03-04T06:10:43.207 INFO:tasks.rados.rados.0.smithi064.stdout:658: writing smithi06431743-24 from 1609179 to 2167277 tid 2 2024-03-04T06:10:43.216 INFO:tasks.rados.rados.0.smithi064.stdout:658: writing smithi06431743-24 from 2804887 to 3356637 tid 3 2024-03-04T06:10:43.216 INFO:tasks.rados.rados.0.smithi064.stdout:659: read oid 14 snap -1 2024-03-04T06:10:43.216 INFO:tasks.rados.rados.0.smithi064.stdout:659: expect (ObjNum 231 snap 55 seq_num 231) 2024-03-04T06:10:43.216 INFO:tasks.rados.rados.0.smithi064.stdout:660: read oid 28 snap -1 2024-03-04T06:10:43.216 INFO:tasks.rados.rados.0.smithi064.stdout:660: expect (ObjNum 282 snap 76 seq_num 282) 2024-03-04T06:10:43.216 INFO:tasks.rados.rados.0.smithi064.stdout:661: write (excl) oid 16 current snap is 81 2024-03-04T06:10:43.216 INFO:tasks.rados.rados.0.smithi064.stdout:661: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-03-04T06:10:43.224 INFO:tasks.rados.rados.0.smithi064.stdout:661: writing smithi06431743-16 from 595047 to 1236802 tid 1 2024-03-04T06:10:43.230 INFO:tasks.rados.rados.0.smithi064.stdout:661: writing smithi06431743-16 from 1667661 to 2085770 tid 2 2024-03-04T06:10:43.237 INFO:tasks.rados.rados.0.smithi064.stdout:661: writing smithi06431743-16 from 2548663 to 3311685 tid 3 2024-03-04T06:10:43.237 INFO:tasks.rados.rados.0.smithi064.stdout:662: read oid 6 snap -1 2024-03-04T06:10:43.237 INFO:tasks.rados.rados.0.smithi064.stdout:662: expect (ObjNum 314 snap 81 seq_num 314) 2024-03-04T06:10:43.237 INFO:tasks.rados.rados.0.smithi064.stdout:663: write (excl) oid 45 current snap is 81 2024-03-04T06:10:43.237 INFO:tasks.rados.rados.0.smithi064.stdout:651: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:10:43.535 INFO:tasks.rados.rados.0.smithi064.stdout:655: expect (ObjNum 273 snap 70 seq_num 273) 2024-03-04T06:10:43.654 INFO:tasks.rados.rados.0.smithi064.stdout:660: expect (ObjNum 282 snap 76 seq_num 282) 2024-03-04T06:10:43.766 INFO:tasks.rados.rados.0.smithi064.stdout:661: finishing write tid 1 to smithi06431743-16 2024-03-04T06:10:43.766 INFO:tasks.rados.rados.0.smithi064.stdout:662: expect (ObjNum 314 snap 81 seq_num 314) 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:661: finishing write tid 2 to smithi06431743-16 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:661: finishing write tid 3 to smithi06431743-16 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:658: finishing write tid 1 to smithi06431743-24 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:661: finishing write tid 4 to smithi06431743-16 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:661: finishing write tid 5 to smithi06431743-16 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:661: oid 16 updating version 0 to 104 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:661: oid 16 version 104 is already newer than 102 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:661: oid 16 updating version 104 to 105 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:661: oid 16 version 105 is already newer than 103 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 105 (ObjNum 321 snap 81 seq_num 321) dirty exists 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:661: left oid 16 (ObjNum 321 snap 81 seq_num 321) 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:658: finishing write tid 2 to smithi06431743-24 2024-03-04T06:10:44.019 INFO:tasks.rados.rados.0.smithi064.stdout:663: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-03-04T06:10:44.026 INFO:tasks.rados.rados.0.smithi064.stdout:663: writing smithi06431743-45 from 611854 to 1328858 tid 1 2024-03-04T06:10:44.034 INFO:tasks.rados.rados.0.smithi064.stdout:663: writing smithi06431743-45 from 2126143 to 2687910 tid 2 2024-03-04T06:10:44.040 INFO:tasks.rados.rados.0.smithi064.stdout:663: writing smithi06431743-45 from 3259733 to 3411854 tid 3 2024-03-04T06:10:44.040 INFO:tasks.rados.rados.0.smithi064.stdout:651: done (8 left) 2024-03-04T06:10:44.040 INFO:tasks.rados.rados.0.smithi064.stdout:655: done (7 left) 2024-03-04T06:10:44.040 INFO:tasks.rados.rados.0.smithi064.stdout:657: done (6 left) 2024-03-04T06:10:44.040 INFO:tasks.rados.rados.0.smithi064.stdout:660: done (5 left) 2024-03-04T06:10:44.040 INFO:tasks.rados.rados.0.smithi064.stdout:661: done (4 left) 2024-03-04T06:10:44.040 INFO:tasks.rados.rados.0.smithi064.stdout:662: done (3 left) 2024-03-04T06:10:44.040 INFO:tasks.rados.rados.0.smithi064.stdout:664: snap_remove snap 76 2024-03-04T06:10:44.268 INFO:tasks.rados.rados.0.smithi064.stdout:664: done (3 left) 2024-03-04T06:10:44.269 INFO:tasks.rados.rados.0.smithi064.stdout:665: delete oid 36 current snap is 81 2024-03-04T06:10:44.269 INFO:tasks.rados.rados.0.smithi064.stdout:658: finishing write tid 3 to smithi06431743-24 2024-03-04T06:10:44.269 INFO:tasks.rados.rados.0.smithi064.stdout:658: finishing write tid 4 to smithi06431743-24 2024-03-04T06:10:44.269 INFO:tasks.rados.rados.0.smithi064.stdout:658: finishing write tid 5 to smithi06431743-24 2024-03-04T06:10:44.269 INFO:tasks.rados.rados.0.smithi064.stdout:658: oid 24 updating version 0 to 205 2024-03-04T06:10:44.269 INFO:tasks.rados.rados.0.smithi064.stdout:658: oid 24 version 205 is already newer than 202 2024-03-04T06:10:44.269 INFO:tasks.rados.rados.0.smithi064.stdout:658: oid 24 version 205 is already newer than 203 2024-03-04T06:10:44.269 INFO:tasks.rados.rados.0.smithi064.stdout:658: oid 24 version 205 is already newer than 205 2024-03-04T06:10:44.269 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 205 (ObjNum 320 snap 81 seq_num 320) dirty exists 2024-03-04T06:10:44.269 INFO:tasks.rados.rados.0.smithi064.stdout:658: left oid 24 (ObjNum 320 snap 81 seq_num 320) 2024-03-04T06:10:44.269 INFO:tasks.rados.rados.0.smithi064.stdout:659: expect (ObjNum 231 snap 55 seq_num 231) 2024-03-04T06:10:44.563 INFO:tasks.rados.rados.0.smithi064.stdout:663: finishing write tid 1 to smithi06431743-45 2024-03-04T06:10:44.563 INFO:tasks.rados.rados.0.smithi064.stdout:663: finishing write tid 2 to smithi06431743-45 2024-03-04T06:10:44.563 INFO:tasks.rados.rados.0.smithi064.stdout:663: finishing write tid 3 to smithi06431743-45 2024-03-04T06:10:44.563 INFO:tasks.rados.rados.0.smithi064.stdout:663: finishing write tid 4 to smithi06431743-45 2024-03-04T06:10:44.563 INFO:tasks.rados.rados.0.smithi064.stdout:663: finishing write tid 5 to smithi06431743-45 2024-03-04T06:10:44.563 INFO:tasks.rados.rados.0.smithi064.stdout:663: oid 45 updating version 0 to 158 2024-03-04T06:10:44.563 INFO:tasks.rados.rados.0.smithi064.stdout:663: oid 45 version 158 is already newer than 157 2024-03-04T06:10:44.563 INFO:tasks.rados.rados.0.smithi064.stdout:663: oid 45 version 158 is already newer than 156 2024-03-04T06:10:44.563 INFO:tasks.rados.rados.0.smithi064.stdout:663: oid 45 updating version 158 to 159 2024-03-04T06:10:44.564 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 159 (ObjNum 322 snap 81 seq_num 322) dirty exists 2024-03-04T06:10:44.564 INFO:tasks.rados.rados.0.smithi064.stdout:663: left oid 45 (ObjNum 322 snap 81 seq_num 322) 2024-03-04T06:10:44.564 INFO:tasks.rados.rados.0.smithi064.stdout:658: done (3 left) 2024-03-04T06:10:44.564 INFO:tasks.rados.rados.0.smithi064.stdout:659: done (2 left) 2024-03-04T06:10:44.564 INFO:tasks.rados.rados.0.smithi064.stdout:663: done (1 left) 2024-03-04T06:10:44.564 INFO:tasks.rados.rados.0.smithi064.stdout:665: done (0 left) 2024-03-04T06:10:44.564 INFO:tasks.rados.rados.0.smithi064.stdout:666: delete oid 1 current snap is 81 2024-03-04T06:10:44.566 INFO:tasks.rados.rados.0.smithi064.stdout:666: done (0 left) 2024-03-04T06:10:44.566 INFO:tasks.rados.rados.0.smithi064.stdout:667: read oid 17 snap -1 2024-03-04T06:10:44.566 INFO:tasks.rados.rados.0.smithi064.stdout:667: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:10:44.567 INFO:tasks.rados.rados.0.smithi064.stdout:668: read oid 6 snap -1 2024-03-04T06:10:44.567 INFO:tasks.rados.rados.0.smithi064.stdout:668: expect (ObjNum 314 snap 81 seq_num 314) 2024-03-04T06:10:44.567 INFO:tasks.rados.rados.0.smithi064.stdout:669: read oid 42 snap -1 2024-03-04T06:10:44.567 INFO:tasks.rados.rados.0.smithi064.stdout:669: expect deleted 2024-03-04T06:10:44.567 INFO:tasks.rados.rados.0.smithi064.stdout:670: delete oid 14 current snap is 81 2024-03-04T06:10:44.595 INFO:tasks.rados.rados.0.smithi064.stdout:667: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:10:44.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:44.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:44.835+0000 7f2394f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:44.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:44.839+0000 7f2394f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:44.876 INFO:tasks.rados.rados.0.smithi064.stdout:668: expect (ObjNum 314 snap 81 seq_num 314) 2024-03-04T06:10:45.127 INFO:tasks.rados.rados.0.smithi064.stdout:667: done (3 left) 2024-03-04T06:10:45.127 INFO:tasks.rados.rados.0.smithi064.stdout:668: done (2 left) 2024-03-04T06:10:45.127 INFO:tasks.rados.rados.0.smithi064.stdout:669: done (1 left) 2024-03-04T06:10:45.127 INFO:tasks.rados.rados.0.smithi064.stdout:670: done (0 left) 2024-03-04T06:10:45.127 INFO:tasks.rados.rados.0.smithi064.stdout:671: read oid 2 snap 80 2024-03-04T06:10:45.127 INFO:tasks.rados.rados.0.smithi064.stdout:671: expect (ObjNum 285 snap 78 seq_num 285) 2024-03-04T06:10:45.127 INFO:tasks.rados.rados.0.smithi064.stdout:672: delete oid 3 current snap is 81 2024-03-04T06:10:45.145 INFO:tasks.rados.rados.0.smithi064.stdout:672: done (1 left) 2024-03-04T06:10:45.145 INFO:tasks.rados.rados.0.smithi064.stdout:673: read oid 47 snap -1 2024-03-04T06:10:45.145 INFO:tasks.rados.rados.0.smithi064.stdout:673: expect deleted 2024-03-04T06:10:45.145 INFO:tasks.rados.rados.0.smithi064.stdout:674: delete oid 44 current snap is 81 2024-03-04T06:10:45.146 INFO:tasks.rados.rados.0.smithi064.stdout:671: expect (ObjNum 285 snap 78 seq_num 285) 2024-03-04T06:10:45.253 INFO:tasks.rados.rados.0.smithi064.stdout:671: done (2 left) 2024-03-04T06:10:45.254 INFO:tasks.rados.rados.0.smithi064.stdout:673: done (1 left) 2024-03-04T06:10:45.254 INFO:tasks.rados.rados.0.smithi064.stdout:674: done (0 left) 2024-03-04T06:10:45.254 INFO:tasks.rados.rados.0.smithi064.stdout:675: snap_remove snap 51 2024-03-04T06:10:45.271 INFO:tasks.rados.rados.0.smithi064.stdout:675: done (0 left) 2024-03-04T06:10:45.271 INFO:tasks.rados.rados.0.smithi064.stdout:676: snap_create 2024-03-04T06:10:45.868 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:10:45.868 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:10:45.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:10:45.980 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:45.976+0000 7fe3bd39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:45.983 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:45.982+0000 7fe3bd39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:10:46.171 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:10:46.183 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-03-04T06:10:45.270242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89600867,"num_objects":236,"num_object_clones":169,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":35,"num_read":1377,"num_read_kb":901439,"num_write":1237,"num_write_kb":303962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2249,"ondisk_log_size":2249,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":534996,"kb_used_data":411060,"kb_used_omap":0,"kb_used_meta":123904,"kb_avail":376952364,"statfs":{"total":386547056640,"available":385999220736,"internally_reserved":0,"allocated":420925440,"data_stored":472661053,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":126877696},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6924583,"num_objects":10,"num_object_clones":10,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":52,"num_read_kb":46418,"num_write":36,"num_write_kb":8244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014138"},"pg_stats":[{"pgid":"3.f","version":"171'110","reported_seq":531,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.280792+0000","last_change":"2024-03-04T06:10:40.296818+0000","last_active":"2024-03-04T06:10:40.280792+0000","last_peered":"2024-03-04T06:10:40.280792+0000","last_clean":"2024-03-04T06:10:40.280792+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:10:40.280792+0000","last_undegraded":"2024-03-04T06:10:40.280792+0000","last_fullsized":"2024-03-04T06:10:40.280792+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731154,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":30,"num_read_kb":6459,"num_write":67,"num_write_kb":17547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"170'94","reported_seq":540,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.271613+0000","last_change":"2024-03-04T06:10:39.321718+0000","last_active":"2024-03-04T06:10:39.271613+0000","last_peered":"2024-03-04T06:10:39.271613+0000","last_clean":"2024-03-04T06:10:39.271613+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:10:39.271613+0000","last_undegraded":"2024-03-04T06:10:39.271613+0000","last_fullsized":"2024-03-04T06:10:39.271613+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":36,"num_read_kb":21069,"num_write":45,"num_write_kb":11808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"172'149","reported_seq":573,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T06:10:41.301267+0000","last_change":"2024-03-04T06:10:41.336923+0000","last_active":"2024-03-04T06:10:41.301267+0000","last_peered":"2024-03-04T06:10:41.301267+0000","last_clean":"2024-03-04T06:10:41.301267+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:10:41.301267+0000","last_undegraded":"2024-03-04T06:10:41.301267+0000","last_fullsized":"2024-03-04T06:10:41.301267+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11512861,"num_objects":16,"num_object_clones":11,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":79,"num_read_kb":51879,"num_write":82,"num_write_kb":22887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"170'105","reported_seq":548,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.273289+0000","last_change":"2024-03-04T06:10:40.275741+0000","last_active":"2024-03-04T06:10:40.273289+0000","last_peered":"2024-03-04T06:10:40.273289+0000","last_clean":"2024-03-04T06:10:40.273289+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:10:40.273289+0000","last_undegraded":"2024-03-04T06:10:40.273289+0000","last_fullsized":"2024-03-04T06:10:40.273289+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2789995,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":79,"num_read_kb":67437,"num_write":58,"num_write_kb":16172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"170'67","reported_seq":511,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.667574+0000","last_change":"2024-03-04T06:10:39.681472+0000","last_active":"2024-03-04T06:10:39.667574+0000","last_peered":"2024-03-04T06:10:39.667574+0000","last_clean":"2024-03-04T06:10:39.667574+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:10:39.667574+0000","last_undegraded":"2024-03-04T06:10:39.667574+0000","last_fullsized":"2024-03-04T06:10:39.667574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":29,"num_read_kb":17040,"num_write":29,"num_write_kb":7475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"171'92","reported_seq":534,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.279440+0000","last_change":"2024-03-04T06:10:40.296522+0000","last_active":"2024-03-04T06:10:40.279440+0000","last_peered":"2024-03-04T06:10:40.279440+0000","last_clean":"2024-03-04T06:10:40.279440+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:10:40.279440+0000","last_undegraded":"2024-03-04T06:10:40.279440+0000","last_fullsized":"2024-03-04T06:10:40.279440+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8235496,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":46529,"num_write":53,"num_write_kb":16061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":460,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.278032+0000","last_change":"2024-03-04T06:10:40.284092+0000","last_active":"2024-03-04T06:10:40.278032+0000","last_peered":"2024-03-04T06:10:40.278032+0000","last_clean":"2024-03-04T06:10:40.278032+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:10:40.278032+0000","last_undegraded":"2024-03-04T06:10:40.278032+0000","last_fullsized":"2024-03-04T06:10:40.278032+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"170'113","reported_seq":541,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:10:41.289638+0000","last_change":"2024-03-04T06:10:41.289638+0000","last_active":"2024-03-04T06:10:41.289638+0000","last_peered":"2024-03-04T06:10:41.289638+0000","last_clean":"2024-03-04T06:10:41.289638+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:10:41.289638+0000","last_undegraded":"2024-03-04T06:10:41.289638+0000","last_fullsized":"2024-03-04T06:10:41.289638+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3680733,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":57,"num_read_kb":48450,"num_write":58,"num_write_kb":16367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":543,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.639653+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:10:39.639653+0000","last_peered":"2024-03-04T06:10:39.639653+0000","last_clean":"2024-03-04T06:10:39.639653+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:10:39.639653+0000","last_undegraded":"2024-03-04T06:10:39.639653+0000","last_fullsized":"2024-03-04T06:10:39.639653+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":402,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T06:10:41.304679+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:10:41.304679+0000","last_peered":"2024-03-04T06:10:41.304679+0000","last_clean":"2024-03-04T06:10:41.304679+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:10:41.304679+0000","last_undegraded":"2024-03-04T06:10:41.304679+0000","last_fullsized":"2024-03-04T06:10:41.304679+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"170'104","reported_seq":546,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:10:41.308005+0000","last_change":"2024-03-04T06:10:41.308005+0000","last_active":"2024-03-04T06:10:41.308005+0000","last_peered":"2024-03-04T06:10:41.308005+0000","last_clean":"2024-03-04T06:10:41.308005+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:10:41.308005+0000","last_undegraded":"2024-03-04T06:10:41.308005+0000","last_fullsized":"2024-03-04T06:10:41.308005+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8588443,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":82,"num_read_kb":73818,"num_write":49,"num_write_kb":13051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":397,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.276068+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:10:40.276068+0000","last_peered":"2024-03-04T06:10:40.276068+0000","last_clean":"2024-03-04T06:10:40.276068+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:10:40.276068+0000","last_undegraded":"2024-03-04T06:10:40.276068+0000","last_fullsized":"2024-03-04T06:10:40.276068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"170'125","reported_seq":546,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.638560+0000","last_change":"2024-03-04T06:10:39.686724+0000","last_active":"2024-03-04T06:10:39.638560+0000","last_peered":"2024-03-04T06:10:39.638560+0000","last_clean":"2024-03-04T06:10:39.638560+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:10:39.638560+0000","last_undegraded":"2024-03-04T06:10:39.638560+0000","last_fullsized":"2024-03-04T06:10:39.638560+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":68,"num_read_kb":54041,"num_write":49,"num_write_kb":13998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":402,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T06:10:41.294500+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:10:41.294500+0000","last_peered":"2024-03-04T06:10:41.294500+0000","last_clean":"2024-03-04T06:10:41.294500+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:10:41.294500+0000","last_undegraded":"2024-03-04T06:10:41.294500+0000","last_fullsized":"2024-03-04T06:10:41.294500+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"170'257","reported_seq":684,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.734688+0000","last_change":"2024-03-04T06:10:39.752953+0000","last_active":"2024-03-04T06:10:39.734688+0000","last_peered":"2024-03-04T06:10:39.734688+0000","last_clean":"2024-03-04T06:10:39.734688+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:10:39.734688+0000","last_undegraded":"2024-03-04T06:10:39.734688+0000","last_fullsized":"2024-03-04T06:10:39.734688+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13025590,"num_objects":30,"num_object_clones":21,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":187,"num_read_kb":135691,"num_write":146,"num_write_kb":41901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":397,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-04T06:10:40.267558+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:10:40.267558+0000","last_peered":"2024-03-04T06:10:40.267558+0000","last_clean":"2024-03-04T06:10:40.267558+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:10:40.267558+0000","last_undegraded":"2024-03-04T06:10:40.267558+0000","last_fullsized":"2024-03-04T06:10:40.267558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"170'279","reported_seq":657,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.356825+0000","last_change":"2024-03-04T06:10:39.356825+0000","last_active":"2024-03-04T06:10:39.356825+0000","last_peered":"2024-03-04T06:10:39.356825+0000","last_clean":"2024-03-04T06:10:39.356825+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:10:39.356825+0000","last_undegraded":"2024-03-04T06:10:39.356825+0000","last_fullsized":"2024-03-04T06:10:39.356825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8689318,"num_objects":28,"num_object_clones":21,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":105,"num_read_kb":57467,"num_write":147,"num_write_kb":39214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"172'112","reported_seq":569,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:10:41.307421+0000","last_change":"2024-03-04T06:10:41.307262+0000","last_active":"2024-03-04T06:10:41.307421+0000","last_peered":"2024-03-04T06:10:41.307421+0000","last_clean":"2024-03-04T06:10:41.307421+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:10:41.307421+0000","last_undegraded":"2024-03-04T06:10:41.307421+0000","last_fullsized":"2024-03-04T06:10:41.307421+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6319465,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":112,"num_read_kb":82624,"num_write":60,"num_write_kb":15547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":424,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.278278+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:10:39.278278+0000","last_peered":"2024-03-04T06:10:39.278278+0000","last_clean":"2024-03-04T06:10:39.278278+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:10:39.278278+0000","last_undegraded":"2024-03-04T06:10:39.278278+0000","last_fullsized":"2024-03-04T06:10:39.278278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"170'99","reported_seq":561,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.285027+0000","last_change":"2024-03-04T06:10:39.323767+0000","last_active":"2024-03-04T06:10:39.285027+0000","last_peered":"2024-03-04T06:10:39.285027+0000","last_clean":"2024-03-04T06:10:39.285027+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:10:39.285027+0000","last_undegraded":"2024-03-04T06:10:39.285027+0000","last_fullsized":"2024-03-04T06:10:39.285027+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781575,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":74,"num_read_kb":43419,"num_write":48,"num_write_kb":12439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":410,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.644913+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:10:39.644913+0000","last_peered":"2024-03-04T06:10:39.644913+0000","last_clean":"2024-03-04T06:10:39.644913+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:10:39.644913+0000","last_undegraded":"2024-03-04T06:10:39.644913+0000","last_fullsized":"2024-03-04T06:10:39.644913+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"170'196","reported_seq":609,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.630145+0000","last_change":"2024-03-04T06:10:39.688511+0000","last_active":"2024-03-04T06:10:39.630145+0000","last_peered":"2024-03-04T06:10:39.630145+0000","last_clean":"2024-03-04T06:10:39.630145+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:10:39.630145+0000","last_undegraded":"2024-03-04T06:10:39.630145+0000","last_fullsized":"2024-03-04T06:10:39.630145+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10689252,"num_objects":19,"num_object_clones":14,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":109,"num_read_kb":72914,"num_write":101,"num_write_kb":28019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":424,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.274282+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:10:39.274282+0000","last_peered":"2024-03-04T06:10:39.274282+0000","last_clean":"2024-03-04T06:10:39.274282+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:10:39.274282+0000","last_undegraded":"2024-03-04T06:10:39.274282+0000","last_fullsized":"2024-03-04T06:10:39.274282+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"170'250","reported_seq":657,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-04T06:10:39.633179+0000","last_change":"2024-03-04T06:10:39.355324+0000","last_active":"2024-03-04T06:10:39.633179+0000","last_peered":"2024-03-04T06:10:39.633179+0000","last_clean":"2024-03-04T06:10:39.633179+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:10:39.633179+0000","last_undegraded":"2024-03-04T06:10:39.633179+0000","last_fullsized":"2024-03-04T06:10:39.633179+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7966617,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":164,"num_read_kb":122511,"num_write":108,"num_write_kb":29582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":402,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T06:10:41.309316+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:10:41.309316+0000","last_peered":"2024-03-04T06:10:41.309316+0000","last_clean":"2024-03-04T06:10:41.309316+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:10:41.309316+0000","last_undegraded":"2024-03-04T06:10:41.309316+0000","last_fullsized":"2024-03-04T06:10:41.309316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":89010499,"num_objects":234,"num_object_clones":169,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":35,"num_read":1274,"num_read_kb":901348,"num_write":1100,"num_write_kb":302068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2152,"ondisk_log_size":2152,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":159108,"kb_used_data":128124,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94212732,"statfs":{"total":96636764160,"available":96473837568,"internally_reserved":0,"allocated":131198976,"data_stored":148227843,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705839,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":138084,"kb_used_data":107100,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94233756,"statfs":{"total":96636764160,"available":96495366144,"internally_reserved":0,"allocated":109670400,"data_stored":123853464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":119104,"kb_used_data":88120,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94252736,"statfs":{"total":96636764160,"available":96514801664,"internally_reserved":0,"allocated":90234880,"data_stored":99416009,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705838,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":118700,"kb_used_data":87716,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94253140,"statfs":{"total":96636764160,"available":96515215360,"internally_reserved":0,"allocated":89821184,"data_stored":101163737,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:10:46.272 INFO:tasks.rados.rados.0.smithi064.stdout:676: done (0 left) 2024-03-04T06:10:46.272 INFO:tasks.rados.rados.0.smithi064.stdout:677: read oid 9 snap -1 2024-03-04T06:10:46.272 INFO:tasks.rados.rados.0.smithi064.stdout:677: expect (ObjNum 256 snap 70 seq_num 256) 2024-03-04T06:10:46.272 INFO:tasks.rados.rados.0.smithi064.stdout:678: delete oid 16 current snap is 82 2024-03-04T06:10:46.365 INFO:tasks.rados.rados.0.smithi064.stdout:677: expect (ObjNum 256 snap 70 seq_num 256) 2024-03-04T06:10:46.583 INFO:tasks.rados.rados.0.smithi064.stdout:677: done (1 left) 2024-03-04T06:10:46.583 INFO:tasks.rados.rados.0.smithi064.stdout:678: done (0 left) 2024-03-04T06:10:46.583 INFO:tasks.rados.rados.0.smithi064.stdout:679: read oid 21 snap 80 2024-03-04T06:10:46.583 INFO:tasks.rados.rados.0.smithi064.stdout:679: expect deleted 2024-03-04T06:10:46.584 INFO:tasks.rados.rados.0.smithi064.stdout:680: read oid 14 snap 77 2024-03-04T06:10:46.584 INFO:tasks.rados.rados.0.smithi064.stdout:680: expect (ObjNum 231 snap 55 seq_num 231) 2024-03-04T06:10:46.584 INFO:tasks.rados.rados.0.smithi064.stdout:681: snap_remove snap 58 2024-03-04T06:10:46.911 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:47.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:47.015+0000 7ff525107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:47.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:47.019+0000 7ff525107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:47.278 INFO:tasks.rados.rados.0.smithi064.stdout:681: done (2 left) 2024-03-04T06:10:47.278 INFO:tasks.rados.rados.0.smithi064.stdout:682: snap_create 2024-03-04T06:10:47.278 INFO:tasks.rados.rados.0.smithi064.stdout:680: expect (ObjNum 231 snap 55 seq_num 231) 2024-03-04T06:10:48.279 INFO:tasks.rados.rados.0.smithi064.stdout:679: done (2 left) 2024-03-04T06:10:48.279 INFO:tasks.rados.rados.0.smithi064.stdout:680: done (1 left) 2024-03-04T06:10:48.279 INFO:tasks.rados.rados.0.smithi064.stdout:682: done (0 left) 2024-03-04T06:10:48.279 INFO:tasks.rados.rados.0.smithi064.stdout:683: read oid 46 snap -1 2024-03-04T06:10:48.279 INFO:tasks.rados.rados.0.smithi064.stdout:683: expect (ObjNum 276 snap 74 seq_num 276) 2024-03-04T06:10:48.279 INFO:tasks.rados.rados.0.smithi064.stdout:684: write (excl) oid 9 current snap is 83 2024-03-04T06:10:48.279 INFO:tasks.rados.rados.0.smithi064.stdout:684: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-03-04T06:10:48.285 INFO:tasks.rados.rados.0.smithi064.stdout:684: writing smithi06431743-9 from 729503 to 1289603 tid 1 2024-03-04T06:10:48.293 INFO:tasks.rados.rados.0.smithi064.stdout:684: writing smithi06431743-9 from 2019164 to 2735596 tid 2 2024-03-04T06:10:48.298 INFO:tasks.rados.rados.0.smithi064.stdout:684: writing smithi06431743-9 from 3335341 to 3529503 tid 3 2024-03-04T06:10:48.298 INFO:tasks.rados.rados.0.smithi064.stdout:685: write oid 28 current snap is 83 2024-03-04T06:10:48.298 INFO:tasks.rados.rados.0.smithi064.stdout:685: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-03-04T06:10:48.306 INFO:tasks.rados.rados.0.smithi064.stdout:685: writing smithi06431743-28 from 746310 to 1381659 tid 1 2024-03-04T06:10:48.312 INFO:tasks.rados.rados.0.smithi064.stdout:685: writing smithi06431743-28 from 2077646 to 2654089 tid 2 2024-03-04T06:10:48.318 INFO:tasks.rados.rados.0.smithi064.stdout:685: writing smithi06431743-28 from 3362764 to 3546310 tid 3 2024-03-04T06:10:48.318 INFO:tasks.rados.rados.0.smithi064.stdout:686: write (excl) oid 33 current snap is 83 2024-03-04T06:10:48.318 INFO:tasks.rados.rados.0.smithi064.stdout:686: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-03-04T06:10:48.325 INFO:tasks.rados.rados.0.smithi064.stdout:686: writing smithi06431743-33 from 763117 to 1473715 tid 1 2024-03-04T06:10:48.333 INFO:tasks.rados.rados.0.smithi064.stdout:686: writing smithi06431743-33 from 2136128 to 2856229 tid 2 2024-03-04T06:10:48.339 INFO:tasks.rados.rados.0.smithi064.stdout:686: writing smithi06431743-33 from 3390187 to 3563117 tid 3 2024-03-04T06:10:48.339 INFO:tasks.rados.rados.0.smithi064.stdout:687: delete oid 24 current snap is 83 2024-03-04T06:10:48.442 INFO:tasks.rados.rados.0.smithi064.stdout:683: expect (ObjNum 276 snap 74 seq_num 276) 2024-03-04T06:10:48.543 INFO:tasks.rados.rados.0.smithi064.stdout:685: finishing write tid 1 to smithi06431743-28 2024-03-04T06:10:48.543 INFO:tasks.rados.rados.0.smithi064.stdout:684: finishing write tid 1 to smithi06431743-9 2024-03-04T06:10:48.543 INFO:tasks.rados.rados.0.smithi064.stdout:684: finishing write tid 2 to smithi06431743-9 2024-03-04T06:10:48.543 INFO:tasks.rados.rados.0.smithi064.stdout:685: finishing write tid 2 to smithi06431743-28 2024-03-04T06:10:48.543 INFO:tasks.rados.rados.0.smithi064.stdout:684: finishing write tid 3 to smithi06431743-9 2024-03-04T06:10:48.543 INFO:tasks.rados.rados.0.smithi064.stdout:684: finishing write tid 4 to smithi06431743-9 2024-03-04T06:10:48.543 INFO:tasks.rados.rados.0.smithi064.stdout:683: done (4 left) 2024-03-04T06:10:48.543 INFO:tasks.rados.rados.0.smithi064.stdout:687: done (3 left) 2024-03-04T06:10:48.543 INFO:tasks.rados.rados.0.smithi064.stdout:688: write (excl) oid 44 current snap is 83 2024-03-04T06:10:48.543 INFO:tasks.rados.rados.0.smithi064.stdout:684: finishing write tid 5 to smithi06431743-9 2024-03-04T06:10:48.543 INFO:tasks.rados.rados.0.smithi064.stdout:684: oid 9 updating version 0 to 286 2024-03-04T06:10:48.544 INFO:tasks.rados.rados.0.smithi064.stdout:684: oid 9 version 286 is already newer than 284 2024-03-04T06:10:48.544 INFO:tasks.rados.rados.0.smithi064.stdout:684: oid 9 version 286 is already newer than 286 2024-03-04T06:10:48.544 INFO:tasks.rados.rados.0.smithi064.stdout:684: oid 9 updating version 286 to 287 2024-03-04T06:10:48.544 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 287 (ObjNum 329 snap 83 seq_num 329) dirty exists 2024-03-04T06:10:48.544 INFO:tasks.rados.rados.0.smithi064.stdout:684: left oid 9 (ObjNum 329 snap 83 seq_num 329) 2024-03-04T06:10:48.544 INFO:tasks.rados.rados.0.smithi064.stdout:685: finishing write tid 3 to smithi06431743-28 2024-03-04T06:10:48.544 INFO:tasks.rados.rados.0.smithi064.stdout:688: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-03-04T06:10:48.551 INFO:tasks.rados.rados.0.smithi064.stdout:688: writing smithi06431743-44 from 796731 to 1257827 tid 1 2024-03-04T06:10:48.560 INFO:tasks.rados.rados.0.smithi064.stdout:688: writing smithi06431743-44 from 1736739 to 2460509 tid 2 2024-03-04T06:10:48.567 INFO:tasks.rados.rados.0.smithi064.stdout:688: writing smithi06431743-44 from 2928680 to 3596731 tid 3 2024-03-04T06:10:48.567 INFO:tasks.rados.rados.0.smithi064.stdout:684: done (3 left) 2024-03-04T06:10:48.567 INFO:tasks.rados.rados.0.smithi064.stdout:689: delete oid 29 current snap is 83 2024-03-04T06:10:48.567 INFO:tasks.rados.rados.0.smithi064.stdout:685: finishing write tid 4 to smithi06431743-28 2024-03-04T06:10:48.567 INFO:tasks.rados.rados.0.smithi064.stdout:685: finishing write tid 5 to smithi06431743-28 2024-03-04T06:10:48.567 INFO:tasks.rados.rados.0.smithi064.stdout:685: oid 28 updating version 0 to 296 2024-03-04T06:10:48.567 INFO:tasks.rados.rados.0.smithi064.stdout:685: oid 28 version 296 is already newer than 293 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:685: oid 28 version 296 is already newer than 295 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:685: oid 28 version 296 is already newer than 294 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 296 (ObjNum 330 snap 83 seq_num 330) dirty exists 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:685: left oid 28 (ObjNum 330 snap 83 seq_num 330) 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:686: finishing write tid 1 to smithi06431743-33 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:686: finishing write tid 2 to smithi06431743-33 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:686: finishing write tid 3 to smithi06431743-33 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:686: finishing write tid 4 to smithi06431743-33 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:686: finishing write tid 5 to smithi06431743-33 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:686: oid 33 updating version 0 to 291 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:686: oid 33 version 291 is already newer than 291 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:686: oid 33 version 291 is already newer than 291 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:686: oid 33 version 291 is already newer than 288 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 291 (ObjNum 331 snap 83 seq_num 331) dirty exists 2024-03-04T06:10:48.568 INFO:tasks.rados.rados.0.smithi064.stdout:686: left oid 33 (ObjNum 331 snap 83 seq_num 331) 2024-03-04T06:10:48.577 INFO:tasks.rados.rados.0.smithi064.stdout:688: finishing write tid 1 to smithi06431743-44 2024-03-04T06:10:48.584 INFO:tasks.rados.rados.0.smithi064.stdout:685: done (3 left) 2024-03-04T06:10:48.585 INFO:tasks.rados.rados.0.smithi064.stdout:686: done (2 left) 2024-03-04T06:10:48.585 INFO:tasks.rados.rados.0.smithi064.stdout:689: done (1 left) 2024-03-04T06:10:48.585 INFO:tasks.rados.rados.0.smithi064.stdout:690: snap_remove snap 80 2024-03-04T06:10:49.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:49.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:49.199+0000 7f3c5962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:49.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:49.204+0000 7f3c5962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:49.283 INFO:tasks.rados.rados.0.smithi064.stdout:690: done (1 left) 2024-03-04T06:10:49.283 INFO:tasks.rados.rados.0.smithi064.stdout:691: write oid 29 current snap is 83 2024-03-04T06:10:49.283 INFO:tasks.rados.rados.0.smithi064.stdout:688: finishing write tid 2 to smithi06431743-44 2024-03-04T06:10:49.284 INFO:tasks.rados.rados.0.smithi064.stdout:691: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-03-04T06:10:49.289 INFO:tasks.rados.rados.0.smithi064.stdout:691: writing smithi06431743-29 from 430345 to 1158292 tid 1 2024-03-04T06:10:49.295 INFO:tasks.rados.rados.0.smithi064.stdout:691: writing smithi06431743-29 from 1570056 to 2297495 tid 2 2024-03-04T06:10:49.300 INFO:tasks.rados.rados.0.smithi064.stdout:691: writing smithi06431743-29 from 2699879 to 3169768 tid 3 2024-03-04T06:10:49.300 INFO:tasks.rados.rados.0.smithi064.stdout:692: snap_remove snap 77 2024-03-04T06:10:50.287 INFO:tasks.rados.rados.0.smithi064.stdout:692: done (2 left) 2024-03-04T06:10:50.287 INFO:tasks.rados.rados.0.smithi064.stdout:693: snap_create 2024-03-04T06:10:50.287 INFO:tasks.rados.rados.0.smithi064.stdout:688: finishing write tid 3 to smithi06431743-44 2024-03-04T06:10:50.287 INFO:tasks.rados.rados.0.smithi064.stdout:688: finishing write tid 4 to smithi06431743-44 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:688: finishing write tid 5 to smithi06431743-44 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:688: oid 44 updating version 0 to 113 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:688: oid 44 version 113 is already newer than 111 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:688: oid 44 version 113 is already newer than 113 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:688: oid 44 version 113 is already newer than 110 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 113 (ObjNum 333 snap 83 seq_num 333) dirty exists 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:688: left oid 44 (ObjNum 333 snap 83 seq_num 333) 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:691: finishing write tid 1 to smithi06431743-29 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:691: finishing write tid 2 to smithi06431743-29 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:691: finishing write tid 3 to smithi06431743-29 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:691: finishing write tid 4 to smithi06431743-29 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:691: finishing write tid 5 to smithi06431743-29 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:691: oid 29 updating version 0 to 324 2024-03-04T06:10:50.288 INFO:tasks.rados.rados.0.smithi064.stdout:691: oid 29 updating version 324 to 326 2024-03-04T06:10:50.289 INFO:tasks.rados.rados.0.smithi064.stdout:691: oid 29 updating version 326 to 327 2024-03-04T06:10:50.289 INFO:tasks.rados.rados.0.smithi064.stdout:691: oid 29 version 327 is already newer than 326 2024-03-04T06:10:50.289 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 327 (ObjNum 335 snap 83 seq_num 335) dirty exists 2024-03-04T06:10:50.289 INFO:tasks.rados.rados.0.smithi064.stdout:691: left oid 29 (ObjNum 335 snap 83 seq_num 335) 2024-03-04T06:10:51.184 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:10:51.185 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:10:51.185 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:10:51.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:10:51.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:51.289 INFO:tasks.rados.rados.0.smithi064.stdout:688: done (2 left) 2024-03-04T06:10:51.289 INFO:tasks.rados.rados.0.smithi064.stdout:691: done (1 left) 2024-03-04T06:10:51.289 INFO:tasks.rados.rados.0.smithi064.stdout:693: done (0 left) 2024-03-04T06:10:51.290 INFO:tasks.rados.rados.0.smithi064.stdout:694: read oid 19 snap -1 2024-03-04T06:10:51.290 INFO:tasks.rados.rados.0.smithi064.stdout:694: expect deleted 2024-03-04T06:10:51.290 INFO:tasks.rados.rados.0.smithi064.stdout:695: read oid 30 snap -1 2024-03-04T06:10:51.290 INFO:tasks.rados.rados.0.smithi064.stdout:695: expect (ObjNum 288 snap 78 seq_num 288) 2024-03-04T06:10:51.290 INFO:tasks.rados.rados.0.smithi064.stdout:696: delete oid 42 current snap is 84 2024-03-04T06:10:51.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:51.304+0000 7f25f6d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:51.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:51.316+0000 7f25f6d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:51.325 INFO:tasks.rados.rados.0.smithi064.stdout:696: done (2 left) 2024-03-04T06:10:51.325 INFO:tasks.rados.rados.0.smithi064.stdout:697: read oid 23 snap 81 2024-03-04T06:10:51.325 INFO:tasks.rados.rados.0.smithi064.stdout:697: expect (ObjNum 204 snap 44 seq_num 204) 2024-03-04T06:10:51.325 INFO:tasks.rados.rados.0.smithi064.stdout:698: read oid 16 snap -1 2024-03-04T06:10:51.325 INFO:tasks.rados.rados.0.smithi064.stdout:698: expect deleted 2024-03-04T06:10:51.325 INFO:tasks.rados.rados.0.smithi064.stdout:699: read oid 26 snap -1 2024-03-04T06:10:51.325 INFO:tasks.rados.rados.0.smithi064.stdout:699: expect (ObjNum 258 snap 70 seq_num 258) 2024-03-04T06:10:51.326 INFO:tasks.rados.rados.0.smithi064.stdout:700: read oid 47 snap -1 2024-03-04T06:10:51.326 INFO:tasks.rados.rados.0.smithi064.stdout:700: expect deleted 2024-03-04T06:10:51.326 INFO:tasks.rados.rados.0.smithi064.stdout:701: snap_remove snap 82 2024-03-04T06:10:51.338 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:51.338 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:51.338 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:51.339 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:51.339 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:51.339 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:51.339 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:51.339 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:51.339 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:51.339 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:51.339 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:51.339 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:51.342 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:51.342 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:10:51.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:51.407+0000 7f79a615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:51.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:51.411+0000 7f79a615d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:51.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:51.494+0000 7fedd7f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:51.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:51.501+0000 7fedd7f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:51.515 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:51.515 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:51.515 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:51.515 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:51.515 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:51.515 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:51.515 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:51.515 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:51.515 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:51.515 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:51.516 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:51.516 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:51.518 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:51.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:10:51.666 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:51.661+0000 7efc6c8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:51.669 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:51.668+0000 7efc6c8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:51.682 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:51.682 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:51.682 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:51.682 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:51.682 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:51.682 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:51.682 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:51.682 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:51.682 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:51.682 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:51.683 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:51.683 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:51.685 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:51.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:10:51.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:51.787+0000 7f17256ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:51.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:51.794+0000 7f17256ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:10:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:10:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:10:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:10:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:10:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:10:51.808 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:10:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:10:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:10:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:10:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:10:51.809 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:10:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:10:51.811 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:10:52.297 INFO:tasks.rados.rados.0.smithi064.stdout:701: done (6 left) 2024-03-04T06:10:52.297 INFO:tasks.rados.rados.0.smithi064.stdout:702: read oid 32 snap 78 2024-03-04T06:10:52.297 INFO:tasks.rados.rados.0.smithi064.stdout:702: expect (ObjNum 287 snap 78 seq_num 287) 2024-03-04T06:10:52.298 INFO:tasks.rados.rados.0.smithi064.stdout:703: delete oid 46 current snap is 84 2024-03-04T06:10:52.298 INFO:tasks.rados.rados.0.smithi064.stdout:695: expect (ObjNum 288 snap 78 seq_num 288) 2024-03-04T06:10:52.453 INFO:tasks.rados.rados.0.smithi064.stdout:699: expect (ObjNum 258 snap 70 seq_num 258) 2024-03-04T06:10:52.598 INFO:tasks.rados.rados.0.smithi064.stdout:697: expect (ObjNum 204 snap 44 seq_num 204) 2024-03-04T06:10:52.726 INFO:tasks.rados.rados.0.smithi064.stdout:702: expect (ObjNum 287 snap 78 seq_num 287) 2024-03-04T06:10:52.902 INFO:tasks.rados.rados.0.smithi064.stdout:694: done (7 left) 2024-03-04T06:10:52.902 INFO:tasks.rados.rados.0.smithi064.stdout:695: done (6 left) 2024-03-04T06:10:52.902 INFO:tasks.rados.rados.0.smithi064.stdout:697: done (5 left) 2024-03-04T06:10:52.902 INFO:tasks.rados.rados.0.smithi064.stdout:698: done (4 left) 2024-03-04T06:10:52.902 INFO:tasks.rados.rados.0.smithi064.stdout:699: done (3 left) 2024-03-04T06:10:52.902 INFO:tasks.rados.rados.0.smithi064.stdout:700: done (2 left) 2024-03-04T06:10:52.902 INFO:tasks.rados.rados.0.smithi064.stdout:702: done (1 left) 2024-03-04T06:10:52.902 INFO:tasks.rados.rados.0.smithi064.stdout:703: done (0 left) 2024-03-04T06:10:52.902 INFO:tasks.rados.rados.0.smithi064.stdout:704: snap_remove snap 78 2024-03-04T06:10:53.195 INFO:tasks.rados.rados.0.smithi064.stdout:704: done (0 left) 2024-03-04T06:10:53.195 INFO:tasks.rados.rados.0.smithi064.stdout:705: write oid 13 current snap is 84 2024-03-04T06:10:53.195 INFO:tasks.rados.rados.0.smithi064.stdout:705: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-03-04T06:10:53.201 INFO:tasks.rados.rados.0.smithi064.stdout:705: writing smithi06431743-13 from 480766 to 1034460 tid 1 2024-03-04T06:10:53.207 INFO:tasks.rados.rados.0.smithi064.stdout:705: writing smithi06431743-13 from 1629149 to 2220268 tid 2 2024-03-04T06:10:53.213 INFO:tasks.rados.rados.0.smithi064.stdout:705: writing smithi06431743-13 from 2782148 to 3202206 tid 3 2024-03-04T06:10:53.213 INFO:tasks.rados.rados.0.smithi064.stdout:706: read oid 49 snap 81 2024-03-04T06:10:53.213 INFO:tasks.rados.rados.0.smithi064.stdout:706: expect deleted 2024-03-04T06:10:53.213 INFO:tasks.rados.rados.0.smithi064.stdout:707: read oid 16 snap -1 2024-03-04T06:10:53.213 INFO:tasks.rados.rados.0.smithi064.stdout:707: expect deleted 2024-03-04T06:10:53.213 INFO:tasks.rados.rados.0.smithi064.stdout:708: read oid 37 snap 81 2024-03-04T06:10:53.213 INFO:tasks.rados.rados.0.smithi064.stdout:708: expect (ObjNum 316 snap 81 seq_num 316) 2024-03-04T06:10:53.213 INFO:tasks.rados.rados.0.smithi064.stdout:709: read oid 10 snap -1 2024-03-04T06:10:53.213 INFO:tasks.rados.rados.0.smithi064.stdout:709: expect (ObjNum 146 snap 20 seq_num 146) 2024-03-04T06:10:53.214 INFO:tasks.rados.rados.0.smithi064.stdout:710: delete oid 3 current snap is 84 2024-03-04T06:10:53.282 INFO:tasks.rados.rados.0.smithi064.stdout:709: expect (ObjNum 146 snap 20 seq_num 146) 2024-03-04T06:10:53.421 INFO:tasks.rados.rados.0.smithi064.stdout:708: expect (ObjNum 316 snap 81 seq_num 316) 2024-03-04T06:10:53.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:53.548 INFO:tasks.rados.rados.0.smithi064.stdout:705: finishing write tid 1 to smithi06431743-13 2024-03-04T06:10:53.548 INFO:tasks.rados.rados.0.smithi064.stdout:705: finishing write tid 2 to smithi06431743-13 2024-03-04T06:10:53.548 INFO:tasks.rados.rados.0.smithi064.stdout:705: finishing write tid 3 to smithi06431743-13 2024-03-04T06:10:53.548 INFO:tasks.rados.rados.0.smithi064.stdout:705: finishing write tid 4 to smithi06431743-13 2024-03-04T06:10:53.548 INFO:tasks.rados.rados.0.smithi064.stdout:705: finishing write tid 5 to smithi06431743-13 2024-03-04T06:10:53.548 INFO:tasks.rados.rados.0.smithi064.stdout:705: oid 13 updating version 0 to 345 2024-03-04T06:10:53.548 INFO:tasks.rados.rados.0.smithi064.stdout:705: oid 13 version 345 is already newer than 344 2024-03-04T06:10:53.548 INFO:tasks.rados.rados.0.smithi064.stdout:705: oid 13 version 345 is already newer than 344 2024-03-04T06:10:53.548 INFO:tasks.rados.rados.0.smithi064.stdout:705: oid 13 version 345 is already newer than 342 2024-03-04T06:10:53.549 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 345 (ObjNum 338 snap 84 seq_num 338) dirty exists 2024-03-04T06:10:53.549 INFO:tasks.rados.rados.0.smithi064.stdout:705: left oid 13 (ObjNum 338 snap 84 seq_num 338) 2024-03-04T06:10:53.549 INFO:tasks.rados.rados.0.smithi064.stdout:705: done (5 left) 2024-03-04T06:10:53.549 INFO:tasks.rados.rados.0.smithi064.stdout:706: done (4 left) 2024-03-04T06:10:53.549 INFO:tasks.rados.rados.0.smithi064.stdout:707: done (3 left) 2024-03-04T06:10:53.549 INFO:tasks.rados.rados.0.smithi064.stdout:708: done (2 left) 2024-03-04T06:10:53.549 INFO:tasks.rados.rados.0.smithi064.stdout:709: done (1 left) 2024-03-04T06:10:53.549 INFO:tasks.rados.rados.0.smithi064.stdout:710: done (0 left) 2024-03-04T06:10:53.549 INFO:tasks.rados.rados.0.smithi064.stdout:711: write oid 50 current snap is 84 2024-03-04T06:10:53.549 INFO:tasks.rados.rados.0.smithi064.stdout:711: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-03-04T06:10:53.557 INFO:tasks.rados.rados.0.smithi064.stdout:711: writing smithi06431743-50 from 514380 to 1218572 tid 1 2024-03-04T06:10:53.563 INFO:tasks.rados.rados.0.smithi064.stdout:711: writing smithi06431743-50 from 1629760 to 2108195 tid 2 2024-03-04T06:10:53.569 INFO:tasks.rados.rados.0.smithi064.stdout:711: writing smithi06431743-50 from 2604288 to 3046890 tid 3 2024-03-04T06:10:53.570 INFO:tasks.rados.rados.0.smithi064.stdout:712: snap_remove snap 81 2024-03-04T06:10:53.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:53.597+0000 7fa770b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:53.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:53.602+0000 7fa770b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:54.306 INFO:tasks.rados.rados.0.smithi064.stdout:712: done (1 left) 2024-03-04T06:10:54.306 INFO:tasks.rados.rados.0.smithi064.stdout:713: write (excl) oid 1 current snap is 84 2024-03-04T06:10:54.306 INFO:tasks.rados.rados.0.smithi064.stdout:713: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-03-04T06:10:54.313 INFO:tasks.rados.rados.0.smithi064.stdout:713: writing smithi06431743-1 from 531187 to 1310628 tid 1 2024-03-04T06:10:54.321 INFO:tasks.rados.rados.0.smithi064.stdout:713: writing smithi06431743-1 from 2088242 to 2826688 tid 2 2024-03-04T06:10:54.326 INFO:tasks.rados.rados.0.smithi064.stdout:713: writing smithi06431743-1 from 3431711 to 3731187 tid 3 2024-03-04T06:10:54.326 INFO:tasks.rados.rados.0.smithi064.stdout:714: write oid 34 current snap is 84 2024-03-04T06:10:54.326 INFO:tasks.rados.rados.0.smithi064.stdout:714: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-03-04T06:10:54.331 INFO:tasks.rados.rados.0.smithi064.stdout:714: writing smithi06431743-34 from 547994 to 1002684 tid 1 2024-03-04T06:10:54.338 INFO:tasks.rados.rados.0.smithi064.stdout:714: writing smithi06431743-34 from 1746724 to 2228828 tid 2 2024-03-04T06:10:54.343 INFO:tasks.rados.rados.0.smithi064.stdout:714: writing smithi06431743-34 from 2659134 to 3124280 tid 3 2024-03-04T06:10:54.349 INFO:tasks.rados.rados.0.smithi064.stdout:714: writing smithi06431743-34 from 3605976 to 3747994 tid 4 2024-03-04T06:10:54.349 INFO:tasks.rados.rados.0.smithi064.stdout:715: read oid 9 snap 83 2024-03-04T06:10:54.349 INFO:tasks.rados.rados.0.smithi064.stdout:715: expect (ObjNum 329 snap 83 seq_num 329) 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:716: read oid 24 snap -1 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:716: expect deleted 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:717: snap_create 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:711: finishing write tid 1 to smithi06431743-50 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:711: finishing write tid 2 to smithi06431743-50 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:711: finishing write tid 3 to smithi06431743-50 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:711: finishing write tid 4 to smithi06431743-50 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:711: finishing write tid 5 to smithi06431743-50 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:711: oid 50 updating version 0 to 316 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:711: oid 50 version 316 is already newer than 316 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:711: oid 50 version 316 is already newer than 314 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:711: oid 50 version 316 is already newer than 313 2024-03-04T06:10:54.350 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 316 (ObjNum 340 snap 84 seq_num 340) dirty exists 2024-03-04T06:10:54.351 INFO:tasks.rados.rados.0.smithi064.stdout:711: left oid 50 (ObjNum 340 snap 84 seq_num 340) 2024-03-04T06:10:54.414 INFO:tasks.rados.rados.0.smithi064.stdout:713: finishing write tid 1 to smithi06431743-1 2024-03-04T06:10:54.435 INFO:tasks.rados.rados.0.smithi064.stdout:715: expect (ObjNum 329 snap 83 seq_num 329) 2024-03-04T06:10:54.718 INFO:tasks.rados.rados.0.smithi064.stdout:713: finishing write tid 2 to smithi06431743-1 2024-03-04T06:10:54.718 INFO:tasks.rados.rados.0.smithi064.stdout:713: finishing write tid 3 to smithi06431743-1 2024-03-04T06:10:54.718 INFO:tasks.rados.rados.0.smithi064.stdout:713: finishing write tid 4 to smithi06431743-1 2024-03-04T06:10:54.718 INFO:tasks.rados.rados.0.smithi064.stdout:713: finishing write tid 5 to smithi06431743-1 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:713: oid 1 updating version 0 to 333 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:713: oid 1 version 333 is already newer than 331 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:713: oid 1 updating version 333 to 334 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:713: oid 1 version 334 is already newer than 332 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 334 (ObjNum 341 snap 84 seq_num 341) dirty exists 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:713: left oid 1 (ObjNum 341 snap 84 seq_num 341) 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:714: finishing write tid 1 to smithi06431743-34 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:714: finishing write tid 2 to smithi06431743-34 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:714: finishing write tid 3 to smithi06431743-34 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:714: finishing write tid 4 to smithi06431743-34 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:714: finishing write tid 5 to smithi06431743-34 2024-03-04T06:10:54.719 INFO:tasks.rados.rados.0.smithi064.stdout:714: finishing write tid 6 to smithi06431743-34 2024-03-04T06:10:54.720 INFO:tasks.rados.rados.0.smithi064.stdout:714: oid 34 updating version 0 to 338 2024-03-04T06:10:54.720 INFO:tasks.rados.rados.0.smithi064.stdout:714: oid 34 updating version 338 to 339 2024-03-04T06:10:54.720 INFO:tasks.rados.rados.0.smithi064.stdout:714: oid 34 version 339 is already newer than 336 2024-03-04T06:10:54.720 INFO:tasks.rados.rados.0.smithi064.stdout:714: oid 34 updating version 339 to 340 2024-03-04T06:10:54.720 INFO:tasks.rados.rados.0.smithi064.stdout:714: oid 34 version 340 is already newer than 338 2024-03-04T06:10:54.720 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 340 (ObjNum 342 snap 84 seq_num 342) dirty exists 2024-03-04T06:10:54.720 INFO:tasks.rados.rados.0.smithi064.stdout:714: left oid 34 (ObjNum 342 snap 84 seq_num 342) 2024-03-04T06:10:55.309 INFO:tasks.rados.rados.0.smithi064.stdout:711: done (5 left) 2024-03-04T06:10:55.310 INFO:tasks.rados.rados.0.smithi064.stdout:713: done (4 left) 2024-03-04T06:10:55.310 INFO:tasks.rados.rados.0.smithi064.stdout:714: done (3 left) 2024-03-04T06:10:55.310 INFO:tasks.rados.rados.0.smithi064.stdout:715: done (2 left) 2024-03-04T06:10:55.310 INFO:tasks.rados.rados.0.smithi064.stdout:716: done (1 left) 2024-03-04T06:10:55.310 INFO:tasks.rados.rados.0.smithi064.stdout:717: done (0 left) 2024-03-04T06:10:55.310 INFO:tasks.rados.rados.0.smithi064.stdout:718: delete oid 2 current snap is 85 2024-03-04T06:10:55.352 INFO:tasks.rados.rados.0.smithi064.stdout:718: done (0 left) 2024-03-04T06:10:55.352 INFO:tasks.rados.rados.0.smithi064.stdout:719: read oid 14 snap -1 2024-03-04T06:10:55.352 INFO:tasks.rados.rados.0.smithi064.stdout:719: expect deleted 2024-03-04T06:10:55.352 INFO:tasks.rados.rados.0.smithi064.stdout:720: read oid 50 snap 84 2024-03-04T06:10:55.352 INFO:tasks.rados.rados.0.smithi064.stdout:720: expect (ObjNum 340 snap 84 seq_num 340) 2024-03-04T06:10:55.352 INFO:tasks.rados.rados.0.smithi064.stdout:721: read oid 43 snap 84 2024-03-04T06:10:55.352 INFO:tasks.rados.rados.0.smithi064.stdout:721: expect (ObjNum 222 snap 50 seq_num 222) 2024-03-04T06:10:55.353 INFO:tasks.rados.rados.0.smithi064.stdout:722: snap_remove snap 83 2024-03-04T06:10:55.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:10:55.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:55.777+0000 7f437d507640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:55.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:55.781+0000 7f437d507640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:56.310 INFO:tasks.rados.rados.0.smithi064.stdout:722: done (3 left) 2024-03-04T06:10:56.311 INFO:tasks.rados.rados.0.smithi064.stdout:723: snap_create 2024-03-04T06:10:56.311 INFO:tasks.rados.rados.0.smithi064.stdout:721: expect (ObjNum 222 snap 50 seq_num 222) 2024-03-04T06:10:56.460 INFO:tasks.rados.rados.0.smithi064.stdout:720: expect (ObjNum 340 snap 84 seq_num 340) 2024-03-04T06:10:56.811 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:10:56.812 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:10:56.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:10:56.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:56.918+0000 7f92cdf54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:56.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:56.924+0000 7f92cdf54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:57.114 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:10:57.114 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:10:57.125 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-03-04T06:10:56.745282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93500124,"num_objects":252,"num_object_clones":182,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":39,"num_read":1517,"num_read_kb":1040576,"num_write":1316,"num_write_kb":325577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2738,"ondisk_log_size":2738,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":292448,"kb_used_data":163532,"kb_used_omap":0,"kb_used_meta":128064,"kb_avail":377194912,"statfs":{"total":386547056640,"available":386247589888,"internally_reserved":0,"allocated":167456768,"data_stored":167005610,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":131137536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7109394,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":48,"num_read_kb":44973,"num_write":22,"num_write_kb":6406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000803"},"pg_stats":[{"pgid":"3.f","version":"184'123","reported_seq":588,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-04T06:10:55.332157+0000","last_change":"2024-03-04T06:10:54.403168+0000","last_active":"2024-03-04T06:10:55.332157+0000","last_peered":"2024-03-04T06:10:55.332157+0000","last_clean":"2024-03-04T06:10:55.332157+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:10:55.332157+0000","last_undegraded":"2024-03-04T06:10:55.332157+0000","last_fullsized":"2024-03-04T06:10:55.332157+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731154,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":30,"num_read_kb":6459,"num_write":67,"num_write_kb":17547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"184'114","reported_seq":599,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-03-04T06:10:54.361132+0000","last_change":"2024-03-04T06:10:54.378660+0000","last_active":"2024-03-04T06:10:54.361132+0000","last_peered":"2024-03-04T06:10:54.361132+0000","last_clean":"2024-03-04T06:10:54.361132+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:10:54.361132+0000","last_undegraded":"2024-03-04T06:10:54.361132+0000","last_fullsized":"2024-03-04T06:10:54.361132+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311012,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":43,"num_read_kb":28823,"num_write":50,"num_write_kb":13094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"184'171","reported_seq":624,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-04T06:10:55.327631+0000","last_change":"2024-03-04T06:10:54.428792+0000","last_active":"2024-03-04T06:10:55.327631+0000","last_peered":"2024-03-04T06:10:55.327631+0000","last_clean":"2024-03-04T06:10:55.327631+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:10:55.327631+0000","last_undegraded":"2024-03-04T06:10:55.327631+0000","last_fullsized":"2024-03-04T06:10:55.327631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11849001,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":81,"num_read_kb":51880,"num_write":88,"num_write_kb":24286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"178'111","reported_seq":607,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-04T06:10:55.323215+0000","last_change":"2024-03-04T06:10:54.394739+0000","last_active":"2024-03-04T06:10:55.323215+0000","last_peered":"2024-03-04T06:10:55.323215+0000","last_clean":"2024-03-04T06:10:55.323215+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:10:55.323215+0000","last_undegraded":"2024-03-04T06:10:55.323215+0000","last_fullsized":"2024-03-04T06:10:55.323215+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2789995,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":85,"num_read_kb":73883,"num_write":58,"num_write_kb":16172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"184'85","reported_seq":567,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-04T06:10:56.335959+0000","last_change":"2024-03-04T06:10:56.341547+0000","last_active":"2024-03-04T06:10:56.335959+0000","last_peered":"2024-03-04T06:10:56.335959+0000","last_clean":"2024-03-04T06:10:56.335959+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:10:56.335959+0000","last_undegraded":"2024-03-04T06:10:56.335959+0000","last_fullsized":"2024-03-04T06:10:56.335959+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":29,"num_read_kb":17040,"num_write":29,"num_write_kb":7475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"184'119","reported_seq":602,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-04T06:10:55.331454+0000","last_change":"2024-03-04T06:10:54.402628+0000","last_active":"2024-03-04T06:10:55.331454+0000","last_peered":"2024-03-04T06:10:55.331454+0000","last_clean":"2024-03-04T06:10:55.331454+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:10:55.331454+0000","last_undegraded":"2024-03-04T06:10:55.331454+0000","last_fullsized":"2024-03-04T06:10:55.331454+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8638864,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":57283,"num_write":59,"num_write_kb":17872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":516,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-04T06:10:55.328089+0000","last_change":"2024-03-04T06:10:54.403636+0000","last_active":"2024-03-04T06:10:55.328089+0000","last_peered":"2024-03-04T06:10:55.328089+0000","last_clean":"2024-03-04T06:10:55.328089+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:10:55.328089+0000","last_undegraded":"2024-03-04T06:10:55.328089+0000","last_fullsized":"2024-03-04T06:10:55.328089+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"184'129","reported_seq":596,"reported_epoch":187,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:10:56.344268+0000","last_change":"2024-03-04T06:10:56.344268+0000","last_active":"2024-03-04T06:10:56.344268+0000","last_peered":"2024-03-04T06:10:56.344268+0000","last_clean":"2024-03-04T06:10:56.344268+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:10:56.344268+0000","last_undegraded":"2024-03-04T06:10:56.344268+0000","last_fullsized":"2024-03-04T06:10:56.344268+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3680733,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":69,"num_read_kb":69049,"num_write":58,"num_write_kb":16367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":586,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-04T06:10:56.325334+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:10:56.325334+0000","last_peered":"2024-03-04T06:10:56.325334+0000","last_clean":"2024-03-04T06:10:56.325334+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:10:56.325334+0000","last_undegraded":"2024-03-04T06:10:56.325334+0000","last_fullsized":"2024-03-04T06:10:56.325334+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":437,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-04T06:10:55.334132+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:10:55.334132+0000","last_peered":"2024-03-04T06:10:55.334132+0000","last_clean":"2024-03-04T06:10:55.334132+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:10:55.334132+0000","last_undegraded":"2024-03-04T06:10:55.334132+0000","last_fullsized":"2024-03-04T06:10:55.334132+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"184'130","reported_seq":598,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-04T06:10:55.340566+0000","last_change":"2024-03-04T06:10:54.432213+0000","last_active":"2024-03-04T06:10:55.340566+0000","last_peered":"2024-03-04T06:10:55.340566+0000","last_clean":"2024-03-04T06:10:55.340566+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:10:55.340566+0000","last_undegraded":"2024-03-04T06:10:55.340566+0000","last_fullsized":"2024-03-04T06:10:55.340566+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5462308,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":88,"num_read_kb":78153,"num_write":50,"num_write_kb":13051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":440,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-04T06:10:54.397818+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:10:54.397818+0000","last_peered":"2024-03-04T06:10:54.397818+0000","last_clean":"2024-03-04T06:10:54.397818+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:10:54.397818+0000","last_undegraded":"2024-03-04T06:10:54.397818+0000","last_fullsized":"2024-03-04T06:10:54.397818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"184'145","reported_seq":602,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-04T06:10:56.329564+0000","last_change":"2024-03-04T06:10:56.336041+0000","last_active":"2024-03-04T06:10:56.329564+0000","last_peered":"2024-03-04T06:10:56.329564+0000","last_clean":"2024-03-04T06:10:56.329564+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:10:56.329564+0000","last_undegraded":"2024-03-04T06:10:56.329564+0000","last_fullsized":"2024-03-04T06:10:56.329564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":68,"num_read_kb":54041,"num_write":49,"num_write_kb":13998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":439,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-04T06:10:56.342972+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:10:56.342972+0000","last_peered":"2024-03-04T06:10:56.342972+0000","last_clean":"2024-03-04T06:10:56.342972+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:10:56.342972+0000","last_undegraded":"2024-03-04T06:10:56.342972+0000","last_fullsized":"2024-03-04T06:10:56.342972+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"185'340","reported_seq":769,"reported_epoch":187,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:10:56.336807+0000","last_change":"2024-03-04T06:10:56.336807+0000","last_active":"2024-03-04T06:10:56.336807+0000","last_peered":"2024-03-04T06:10:56.336807+0000","last_clean":"2024-03-04T06:10:56.336807+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:10:56.336807+0000","last_undegraded":"2024-03-04T06:10:56.336807+0000","last_fullsized":"2024-03-04T06:10:56.336807+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16588674,"num_objects":34,"num_object_clones":24,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":217,"num_read_kb":160795,"num_write":163,"num_write_kb":46552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":440,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-03-04T06:10:54.390091+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:10:54.390091+0000","last_peered":"2024-03-04T06:10:54.390091+0000","last_clean":"2024-03-04T06:10:54.390091+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:10:54.390091+0000","last_undegraded":"2024-03-04T06:10:54.390091+0000","last_fullsized":"2024-03-04T06:10:54.390091+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"185'347","reported_seq":726,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-03-04T06:10:54.362123+0000","last_change":"2024-03-04T06:10:54.421659+0000","last_active":"2024-03-04T06:10:54.362123+0000","last_peered":"2024-03-04T06:10:54.362123+0000","last_clean":"2024-03-04T06:10:54.362123+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:10:54.362123+0000","last_undegraded":"2024-03-04T06:10:54.362123+0000","last_fullsized":"2024-03-04T06:10:54.362123+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13008750,"num_objects":30,"num_object_clones":22,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":120,"num_read_kb":71194,"num_write":158,"num_write_kb":42625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"184'143","reported_seq":628,"reported_epoch":187,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:10:56.351544+0000","last_change":"2024-03-04T06:10:56.351544+0000","last_active":"2024-03-04T06:10:56.351544+0000","last_peered":"2024-03-04T06:10:56.351544+0000","last_clean":"2024-03-04T06:10:56.351544+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:10:56.351544+0000","last_undegraded":"2024-03-04T06:10:56.351544+0000","last_fullsized":"2024-03-04T06:10:56.351544+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6571570,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":125,"num_read_kb":98834,"num_write":64,"num_write_kb":16590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":463,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-03-04T06:10:54.376251+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:10:54.376251+0000","last_peered":"2024-03-04T06:10:54.376251+0000","last_clean":"2024-03-04T06:10:54.376251+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:10:54.376251+0000","last_undegraded":"2024-03-04T06:10:54.376251+0000","last_fullsized":"2024-03-04T06:10:54.376251+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"185'125","reported_seq":620,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-03-04T06:10:54.369845+0000","last_change":"2024-03-04T06:10:54.415900+0000","last_active":"2024-03-04T06:10:54.369845+0000","last_peered":"2024-03-04T06:10:54.369845+0000","last_clean":"2024-03-04T06:10:54.369845+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:10:54.369845+0000","last_undegraded":"2024-03-04T06:10:54.369845+0000","last_fullsized":"2024-03-04T06:10:54.369845+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":78,"num_read_kb":43420,"num_write":55,"num_write_kb":14221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-04T06:10:56.335292+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:10:56.335292+0000","last_peered":"2024-03-04T06:10:56.335292+0000","last_clean":"2024-03-04T06:10:56.335292+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:10:56.335292+0000","last_undegraded":"2024-03-04T06:10:56.335292+0000","last_fullsized":"2024-03-04T06:10:56.335292+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"185'239","reported_seq":678,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-04T06:10:56.323943+0000","last_change":"2024-03-04T06:10:56.336782+0000","last_active":"2024-03-04T06:10:56.323943+0000","last_peered":"2024-03-04T06:10:56.323943+0000","last_clean":"2024-03-04T06:10:56.323943+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:10:56.323943+0000","last_undegraded":"2024-03-04T06:10:56.323943+0000","last_fullsized":"2024-03-04T06:10:56.323943+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3848770,"num_objects":22,"num_object_clones":17,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":123,"num_read_kb":92829,"num_write":108,"num_write_kb":29658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":463,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-03-04T06:10:54.367124+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:10:54.367124+0000","last_peered":"2024-03-04T06:10:54.367124+0000","last_clean":"2024-03-04T06:10:54.367124+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:10:54.367124+0000","last_undegraded":"2024-03-04T06:10:54.367124+0000","last_fullsized":"2024-03-04T06:10:54.367124+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"185'320","reported_seq":729,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-03-04T06:10:54.433040+0000","last_change":"2024-03-04T06:10:54.441654+0000","last_active":"2024-03-04T06:10:54.433040+0000","last_peered":"2024-03-04T06:10:54.433040+0000","last_clean":"2024-03-04T06:10:54.433040+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:10:54.433040+0000","last_undegraded":"2024-03-04T06:10:54.433040+0000","last_fullsized":"2024-03-04T06:10:54.433040+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13428925,"num_objects":26,"num_object_clones":22,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":181,"num_read_kb":136802,"num_write":123,"num_write_kb":34175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":436,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-04T06:10:55.329665+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:10:55.329665+0000","last_peered":"2024-03-04T06:10:55.329665+0000","last_clean":"2024-03-04T06:10:55.329665+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:10:55.329665+0000","last_undegraded":"2024-03-04T06:10:55.329665+0000","last_fullsized":"2024-03-04T06:10:55.329665+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":92909756,"num_objects":250,"num_object_clones":182,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":39,"num_read":1414,"num_read_kb":1040485,"num_write":1179,"num_write_kb":323683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2641,"ondisk_log_size":2641,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85248,"kb_used_data":50104,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94286592,"statfs":{"total":96636764160,"available":96549470208,"internally_reserved":0,"allocated":51306496,"data_stored":51662128,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77600,"kb_used_data":46616,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94294240,"statfs":{"total":96636764160,"available":96557301760,"internally_reserved":0,"allocated":47734784,"data_stored":47584247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71752,"kb_used_data":39948,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94300088,"statfs":{"total":96636764160,"available":96563290112,"internally_reserved":0,"allocated":40906752,"data_stored":40796946,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":57848,"kb_used_data":26864,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94313992,"statfs":{"total":96636764160,"available":96577527808,"internally_reserved":0,"allocated":27508736,"data_stored":26962289,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:10:57.315 INFO:tasks.rados.rados.0.smithi064.stdout:719: done (3 left) 2024-03-04T06:10:57.315 INFO:tasks.rados.rados.0.smithi064.stdout:720: done (2 left) 2024-03-04T06:10:57.315 INFO:tasks.rados.rados.0.smithi064.stdout:721: done (1 left) 2024-03-04T06:10:57.316 INFO:tasks.rados.rados.0.smithi064.stdout:723: done (0 left) 2024-03-04T06:10:57.316 INFO:tasks.rados.rados.0.smithi064.stdout:724: snap_remove snap 84 2024-03-04T06:10:57.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:10:57.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:57.970+0000 7f5a818dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:57.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:10:57.977+0000 7f5a818dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:10:58.317 INFO:tasks.rados.rados.0.smithi064.stdout:724: done (0 left) 2024-03-04T06:10:58.317 INFO:tasks.rados.rados.0.smithi064.stdout:725: read oid 3 snap -1 2024-03-04T06:10:58.317 INFO:tasks.rados.rados.0.smithi064.stdout:725: expect deleted 2024-03-04T06:10:58.317 INFO:tasks.rados.rados.0.smithi064.stdout:726: write oid 12 current snap is 86 2024-03-04T06:10:58.317 INFO:tasks.rados.rados.0.smithi064.stdout:726: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-03-04T06:10:58.325 INFO:tasks.rados.rados.0.smithi064.stdout:726: writing smithi06431743-12 from 581608 to 1303149 tid 1 2024-03-04T06:10:58.333 INFO:tasks.rados.rados.0.smithi064.stdout:726: writing smithi06431743-12 from 1863688 to 2349461 tid 2 2024-03-04T06:10:58.342 INFO:tasks.rados.rados.0.smithi064.stdout:726: writing smithi06431743-12 from 3113980 to 3718023 tid 3 2024-03-04T06:10:58.342 INFO:tasks.rados.rados.0.smithi064.stdout:727: write oid 43 current snap is 86 2024-03-04T06:10:58.342 INFO:tasks.rados.rados.0.smithi064.stdout:727: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-03-04T06:10:58.351 INFO:tasks.rados.rados.0.smithi064.stdout:727: writing smithi06431743-43 from 598415 to 1395205 tid 1 2024-03-04T06:10:58.359 INFO:tasks.rados.rados.0.smithi064.stdout:727: writing smithi06431743-43 from 1922170 to 2667954 tid 2 2024-03-04T06:10:58.363 INFO:tasks.rados.rados.0.smithi064.stdout:727: writing smithi06431743-43 from 3141403 to 3556718 tid 3 2024-03-04T06:10:58.363 INFO:tasks.rados.rados.0.smithi064.stdout:728: snap_create 2024-03-04T06:10:58.368 INFO:tasks.rados.rados.0.smithi064.stdout:726: finishing write tid 1 to smithi06431743-12 2024-03-04T06:10:58.381 INFO:tasks.rados.rados.0.smithi064.stdout:726: finishing write tid 2 to smithi06431743-12 2024-03-04T06:10:58.388 INFO:tasks.rados.rados.0.smithi064.stdout:727: finishing write tid 1 to smithi06431743-43 2024-03-04T06:10:58.394 INFO:tasks.rados.rados.0.smithi064.stdout:726: finishing write tid 3 to smithi06431743-12 2024-03-04T06:10:58.398 INFO:tasks.rados.rados.0.smithi064.stdout:726: finishing write tid 4 to smithi06431743-12 2024-03-04T06:10:58.400 INFO:tasks.rados.rados.0.smithi064.stdout:726: finishing write tid 5 to smithi06431743-12 2024-03-04T06:10:58.400 INFO:tasks.rados.rados.0.smithi064.stdout:726: oid 12 updating version 0 to 121 2024-03-04T06:10:58.401 INFO:tasks.rados.rados.0.smithi064.stdout:726: oid 12 version 121 is already newer than 121 2024-03-04T06:10:58.401 INFO:tasks.rados.rados.0.smithi064.stdout:726: oid 12 version 121 is already newer than 119 2024-03-04T06:10:58.401 INFO:tasks.rados.rados.0.smithi064.stdout:726: oid 12 version 121 is already newer than 119 2024-03-04T06:10:58.401 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 121 (ObjNum 344 snap 86 seq_num 344) dirty exists 2024-03-04T06:10:58.401 INFO:tasks.rados.rados.0.smithi064.stdout:726: left oid 12 (ObjNum 344 snap 86 seq_num 344) 2024-03-04T06:10:58.404 INFO:tasks.rados.rados.0.smithi064.stdout:727: finishing write tid 2 to smithi06431743-43 2024-03-04T06:10:58.420 INFO:tasks.rados.rados.0.smithi064.stdout:727: finishing write tid 3 to smithi06431743-43 2024-03-04T06:10:58.426 INFO:tasks.rados.rados.0.smithi064.stdout:727: finishing write tid 4 to smithi06431743-43 2024-03-04T06:10:58.427 INFO:tasks.rados.rados.0.smithi064.stdout:727: finishing write tid 5 to smithi06431743-43 2024-03-04T06:10:58.427 INFO:tasks.rados.rados.0.smithi064.stdout:727: oid 43 updating version 0 to 126 2024-03-04T06:10:58.427 INFO:tasks.rados.rados.0.smithi064.stdout:727: oid 43 updating version 126 to 127 2024-03-04T06:10:58.427 INFO:tasks.rados.rados.0.smithi064.stdout:727: oid 43 updating version 127 to 128 2024-03-04T06:10:58.427 INFO:tasks.rados.rados.0.smithi064.stdout:727: oid 43 version 128 is already newer than 125 2024-03-04T06:10:58.427 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 128 (ObjNum 345 snap 86 seq_num 345) dirty exists 2024-03-04T06:10:58.428 INFO:tasks.rados.rados.0.smithi064.stdout:727: left oid 43 (ObjNum 345 snap 86 seq_num 345) 2024-03-04T06:10:59.322 INFO:tasks.rados.rados.0.smithi064.stdout:725: done (3 left) 2024-03-04T06:10:59.323 INFO:tasks.rados.rados.0.smithi064.stdout:726: done (2 left) 2024-03-04T06:10:59.323 INFO:tasks.rados.rados.0.smithi064.stdout:727: done (1 left) 2024-03-04T06:10:59.323 INFO:tasks.rados.rados.0.smithi064.stdout:728: done (0 left) 2024-03-04T06:10:59.323 INFO:tasks.rados.rados.0.smithi064.stdout:729: write (excl) oid 19 current snap is 87 2024-03-04T06:10:59.323 INFO:tasks.rados.rados.0.smithi064.stdout:729: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-03-04T06:10:59.328 INFO:tasks.rados.rados.0.smithi064.stdout:729: writing smithi06431743-19 from 615222 to 1087261 tid 1 2024-03-04T06:10:59.333 INFO:tasks.rados.rados.0.smithi064.stdout:729: writing smithi06431743-19 from 1580652 to 2070094 tid 2 2024-03-04T06:10:59.341 INFO:tasks.rados.rados.0.smithi064.stdout:729: writing smithi06431743-19 from 2768826 to 3395413 tid 3 2024-03-04T06:10:59.341 INFO:tasks.rados.rados.0.smithi064.stdout:730: write oid 9 current snap is 87 2024-03-04T06:10:59.341 INFO:tasks.rados.rados.0.smithi064.stdout:730: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-03-04T06:10:59.347 INFO:tasks.rados.rados.0.smithi064.stdout:730: writing smithi06431743-9 from 632029 to 1179317 tid 1 2024-03-04T06:10:59.354 INFO:tasks.rados.rados.0.smithi064.stdout:730: writing smithi06431743-9 from 1639134 to 2388587 tid 2 2024-03-04T06:10:59.358 INFO:tasks.rados.rados.0.smithi064.stdout:730: writing smithi06431743-9 from 2796249 to 3234108 tid 3 2024-03-04T06:10:59.362 INFO:tasks.rados.rados.0.smithi064.stdout:730: writing smithi06431743-9 from 3653524 to 3832029 tid 4 2024-03-04T06:10:59.362 INFO:tasks.rados.rados.0.smithi064.stdout:731: delete oid 5 current snap is 87 2024-03-04T06:10:59.401 INFO:tasks.rados.rados.0.smithi064.stdout:729: finishing write tid 1 to smithi06431743-19 2024-03-04T06:10:59.425 INFO:tasks.rados.rados.0.smithi064.stdout:729: finishing write tid 2 to smithi06431743-19 2024-03-04T06:10:59.432 INFO:tasks.rados.rados.0.smithi064.stdout:730: finishing write tid 1 to smithi06431743-9 2024-03-04T06:10:59.445 INFO:tasks.rados.rados.0.smithi064.stdout:729: finishing write tid 3 to smithi06431743-19 2024-03-04T06:10:59.447 INFO:tasks.rados.rados.0.smithi064.stdout:729: finishing write tid 4 to smithi06431743-19 2024-03-04T06:10:59.451 INFO:tasks.rados.rados.0.smithi064.stdout:729: finishing write tid 5 to smithi06431743-19 2024-03-04T06:10:59.451 INFO:tasks.rados.rados.0.smithi064.stdout:729: oid 19 updating version 0 to 348 2024-03-04T06:10:59.451 INFO:tasks.rados.rados.0.smithi064.stdout:729: oid 19 updating version 348 to 349 2024-03-04T06:10:59.451 INFO:tasks.rados.rados.0.smithi064.stdout:729: oid 19 updating version 349 to 351 2024-03-04T06:10:59.451 INFO:tasks.rados.rados.0.smithi064.stdout:729: oid 19 version 351 is already newer than 351 2024-03-04T06:10:59.451 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 351 (ObjNum 346 snap 87 seq_num 346) dirty exists 2024-03-04T06:10:59.451 INFO:tasks.rados.rados.0.smithi064.stdout:729: left oid 19 (ObjNum 346 snap 87 seq_num 346) 2024-03-04T06:10:59.455 INFO:tasks.rados.rados.0.smithi064.stdout:730: finishing write tid 2 to smithi06431743-9 2024-03-04T06:10:59.463 INFO:tasks.rados.rados.0.smithi064.stdout:730: finishing write tid 3 to smithi06431743-9 2024-03-04T06:10:59.470 INFO:tasks.rados.rados.0.smithi064.stdout:729: done (2 left) 2024-03-04T06:10:59.470 INFO:tasks.rados.rados.0.smithi064.stdout:731: done (1 left) 2024-03-04T06:10:59.470 INFO:tasks.rados.rados.0.smithi064.stdout:732: snap_remove snap 85 2024-03-04T06:11:00.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:00.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:00.162+0000 7f3d5d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:00.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:00.166+0000 7f3d5d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:00.323 INFO:tasks.rados.rados.0.smithi064.stdout:732: done (1 left) 2024-03-04T06:11:00.323 INFO:tasks.rados.rados.0.smithi064.stdout:733: read oid 10 snap -1 2024-03-04T06:11:00.323 INFO:tasks.rados.rados.0.smithi064.stdout:733: expect (ObjNum 146 snap 20 seq_num 146) 2024-03-04T06:11:00.323 INFO:tasks.rados.rados.0.smithi064.stdout:734: snap_remove snap 86 2024-03-04T06:11:01.332 INFO:tasks.rados.rados.0.smithi064.stdout:734: done (2 left) 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:735: delete oid 24 current snap is 87 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:730: finishing write tid 4 to smithi06431743-9 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:730: finishing write tid 5 to smithi06431743-9 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:730: finishing write tid 6 to smithi06431743-9 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:730: oid 9 updating version 0 to 326 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:730: oid 9 updating version 326 to 328 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:730: oid 9 version 328 is already newer than 326 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:730: oid 9 version 328 is already newer than 327 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:730: oid 9 version 328 is already newer than 324 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 328 (ObjNum 347 snap 87 seq_num 347) dirty exists 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:730: left oid 9 (ObjNum 347 snap 87 seq_num 347) 2024-03-04T06:11:01.333 INFO:tasks.rados.rados.0.smithi064.stdout:733: expect (ObjNum 146 snap 20 seq_num 146) 2024-03-04T06:11:01.470 INFO:tasks.rados.rados.0.smithi064.stdout:730: done (2 left) 2024-03-04T06:11:01.471 INFO:tasks.rados.rados.0.smithi064.stdout:733: done (1 left) 2024-03-04T06:11:01.471 INFO:tasks.rados.rados.0.smithi064.stdout:735: done (0 left) 2024-03-04T06:11:01.471 INFO:tasks.rados.rados.0.smithi064.stdout:736: snap_create 2024-03-04T06:11:02.129 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:11:02.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:11:02.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:02.242+0000 7f377382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:02.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:02.251+0000 7f377382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:02.335 INFO:tasks.rados.rados.0.smithi064.stdout:736: done (0 left) 2024-03-04T06:11:02.335 INFO:tasks.rados.rados.0.smithi064.stdout:737: write oid 12 current snap is 88 2024-03-04T06:11:02.335 INFO:tasks.rados.rados.0.smithi064.stdout:737: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-03-04T06:11:02.341 INFO:tasks.rados.rados.0.smithi064.stdout:737: writing smithi06431743-12 from 682450 to 1171838 tid 1 2024-03-04T06:11:02.347 INFO:tasks.rados.rados.0.smithi064.stdout:737: writing smithi06431743-12 from 1814580 to 2311360 tid 2 2024-03-04T06:11:02.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:02.346+0000 7f788999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:02.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:02.352+0000 7f788999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:02.358 INFO:tasks.rados.rados.0.smithi064.stdout:737: writing smithi06431743-12 from 2878518 to 3666546 tid 3 2024-03-04T06:11:02.358 INFO:tasks.rados.rados.0.smithi064.stdout:738: snap_remove snap 87 2024-03-04T06:11:03.195 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:11:03.197 INFO:tasks.rados.rados.0.smithi064.stdout:738: done (1 left) 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:739: delete oid 31 current snap is 88 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:737: finishing write tid 1 to smithi06431743-12 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:737: finishing write tid 2 to smithi06431743-12 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:737: finishing write tid 3 to smithi06431743-12 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:737: finishing write tid 4 to smithi06431743-12 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:737: finishing write tid 5 to smithi06431743-12 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:737: oid 12 updating version 0 to 126 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:737: oid 12 version 126 is already newer than 125 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:737: oid 12 version 126 is already newer than 125 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:737: oid 12 version 126 is already newer than 123 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 126 (ObjNum 350 snap 88 seq_num 350) dirty exists 2024-03-04T06:11:03.198 INFO:tasks.rados.rados.0.smithi064.stdout:737: left oid 12 (ObjNum 350 snap 88 seq_num 350) 2024-03-04T06:11:03.211 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:11:03.230 INFO:tasks.rados.rados.0.smithi064.stdout:737: done (1 left) 2024-03-04T06:11:03.230 INFO:tasks.rados.rados.0.smithi064.stdout:739: done (0 left) 2024-03-04T06:11:03.231 INFO:tasks.rados.rados.0.smithi064.stdout:740: write (excl) oid 50 current snap is 88 2024-03-04T06:11:03.231 INFO:tasks.rados.rados.0.smithi064.stdout:740: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-03-04T06:11:03.238 INFO:tasks.rados.rados.0.smithi064.stdout:740: writing smithi06431743-50 from 716064 to 1355950 tid 1 2024-03-04T06:11:03.243 INFO:tasks.rados.rados.0.smithi064.stdout:740: writing smithi06431743-50 from 1815191 to 2315640 tid 2 2024-03-04T06:11:03.249 INFO:tasks.rados.rados.0.smithi064.stdout:740: writing smithi06431743-50 from 2817011 to 3343936 tid 3 2024-03-04T06:11:03.249 INFO:tasks.rados.rados.0.smithi064.stdout:741: snap_create 2024-03-04T06:11:03.300 INFO:tasks.rados.rados.0.smithi064.stdout:740: finishing write tid 1 to smithi06431743-50 2024-03-04T06:11:03.313 INFO:tasks.rados.rados.0.smithi064.stdout:740: finishing write tid 2 to smithi06431743-50 2024-03-04T06:11:03.322 INFO:tasks.rados.rados.0.smithi064.stdout:740: finishing write tid 3 to smithi06431743-50 2024-03-04T06:11:03.329 INFO:tasks.rados.rados.0.smithi064.stdout:740: finishing write tid 4 to smithi06431743-50 2024-03-04T06:11:03.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:03.323+0000 7f68cfb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:03.330 INFO:tasks.rados.rados.0.smithi064.stdout:740: finishing write tid 5 to smithi06431743-50 2024-03-04T06:11:03.331 INFO:tasks.rados.rados.0.smithi064.stdout:740: oid 50 updating version 0 to 333 2024-03-04T06:11:03.331 INFO:tasks.rados.rados.0.smithi064.stdout:740: oid 50 version 333 is already newer than 331 2024-03-04T06:11:03.331 INFO:tasks.rados.rados.0.smithi064.stdout:740: oid 50 updating version 333 to 334 2024-03-04T06:11:03.331 INFO:tasks.rados.rados.0.smithi064.stdout:740: oid 50 version 334 is already newer than 332 2024-03-04T06:11:03.331 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 334 (ObjNum 352 snap 88 seq_num 352) dirty exists 2024-03-04T06:11:03.331 INFO:tasks.rados.rados.0.smithi064.stdout:740: left oid 50 (ObjNum 352 snap 88 seq_num 352) 2024-03-04T06:11:03.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:03.333+0000 7f68cfb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:04.196 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:11:04.198 INFO:tasks.rados.rados.0.smithi064.stdout:740: done (1 left) 2024-03-04T06:11:04.198 INFO:tasks.rados.rados.0.smithi064.stdout:741: done (0 left) 2024-03-04T06:11:04.198 INFO:tasks.rados.rados.0.smithi064.stdout:742: read oid 17 snap -1 2024-03-04T06:11:04.198 INFO:tasks.rados.rados.0.smithi064.stdout:742: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:11:04.198 INFO:tasks.rados.rados.0.smithi064.stdout:743: snap_create 2024-03-04T06:11:04.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:11:04.256 INFO:tasks.rados.rados.0.smithi064.stdout:742: expect (ObjNum 219 snap 49 seq_num 219) 2024-03-04T06:11:04.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:04.355+0000 7fc345f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:04.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:04.363+0000 7fc345f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:04.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:04.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:04.576+0000 7f80ee35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:04.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:04.581+0000 7f80ee35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:05.196 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:11:05.199 INFO:tasks.rados.rados.0.smithi064.stdout:742: done (1 left) 2024-03-04T06:11:05.199 INFO:tasks.rados.rados.0.smithi064.stdout:743: done (0 left) 2024-03-04T06:11:05.199 INFO:tasks.rados.rados.0.smithi064.stdout:744: read oid 29 snap -1 2024-03-04T06:11:05.199 INFO:tasks.rados.rados.0.smithi064.stdout:744: expect (ObjNum 335 snap 83 seq_num 335) 2024-03-04T06:11:05.199 INFO:tasks.rados.rados.0.smithi064.stdout:745: write oid 35 current snap is 90 2024-03-04T06:11:05.199 INFO:tasks.rados.rados.0.smithi064.stdout:745: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-03-04T06:11:05.209 INFO:tasks.rados.rados.0.smithi064.stdout:745: writing smithi06431743-35 from 732871 to 1448006 tid 1 2024-03-04T06:11:05.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:11:05.216 INFO:tasks.rados.rados.0.smithi064.stdout:745: writing smithi06431743-35 from 1873673 to 2517780 tid 2 2024-03-04T06:11:05.225 INFO:tasks.rados.rados.0.smithi064.stdout:745: writing smithi06431743-35 from 3244434 to 3932871 tid 3 2024-03-04T06:11:05.225 INFO:tasks.rados.rados.0.smithi064.stdout:746: read oid 46 snap -1 2024-03-04T06:11:05.226 INFO:tasks.rados.rados.0.smithi064.stdout:746: expect deleted 2024-03-04T06:11:05.226 INFO:tasks.rados.rados.0.smithi064.stdout:747: read oid 9 snap -1 2024-03-04T06:11:05.226 INFO:tasks.rados.rados.0.smithi064.stdout:747: expect (ObjNum 347 snap 87 seq_num 347) 2024-03-04T06:11:05.226 INFO:tasks.rados.rados.0.smithi064.stdout:748: delete oid 30 current snap is 90 2024-03-04T06:11:05.273 INFO:tasks.rados.rados.0.smithi064.stdout:744: expect (ObjNum 335 snap 83 seq_num 335) 2024-03-04T06:11:05.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:05.333+0000 7fe47dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:05.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:05.342+0000 7fe47dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:05.434 INFO:tasks.rados.rados.0.smithi064.stdout:747: expect (ObjNum 347 snap 87 seq_num 347) 2024-03-04T06:11:05.787 INFO:tasks.rados.rados.0.smithi064.stdout:745: finishing write tid 1 to smithi06431743-35 2024-03-04T06:11:05.787 INFO:tasks.rados.rados.0.smithi064.stdout:745: finishing write tid 2 to smithi06431743-35 2024-03-04T06:11:05.787 INFO:tasks.rados.rados.0.smithi064.stdout:745: finishing write tid 3 to smithi06431743-35 2024-03-04T06:11:05.787 INFO:tasks.rados.rados.0.smithi064.stdout:745: finishing write tid 4 to smithi06431743-35 2024-03-04T06:11:05.787 INFO:tasks.rados.rados.0.smithi064.stdout:744: done (4 left) 2024-03-04T06:11:05.787 INFO:tasks.rados.rados.0.smithi064.stdout:746: done (3 left) 2024-03-04T06:11:05.788 INFO:tasks.rados.rados.0.smithi064.stdout:747: done (2 left) 2024-03-04T06:11:05.788 INFO:tasks.rados.rados.0.smithi064.stdout:748: done (1 left) 2024-03-04T06:11:05.788 INFO:tasks.rados.rados.0.smithi064.stdout:749: write oid 44 current snap is 90 2024-03-04T06:11:05.788 INFO:tasks.rados.rados.0.smithi064.stdout:745: finishing write tid 5 to smithi06431743-35 2024-03-04T06:11:05.788 INFO:tasks.rados.rados.0.smithi064.stdout:745: oid 35 updating version 0 to 243 2024-03-04T06:11:05.788 INFO:tasks.rados.rados.0.smithi064.stdout:745: oid 35 updating version 243 to 245 2024-03-04T06:11:05.788 INFO:tasks.rados.rados.0.smithi064.stdout:745: oid 35 version 245 is already newer than 244 2024-03-04T06:11:05.788 INFO:tasks.rados.rados.0.smithi064.stdout:745: oid 35 version 245 is already newer than 243 2024-03-04T06:11:05.788 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 245 (ObjNum 353 snap 90 seq_num 353) dirty exists 2024-03-04T06:11:05.788 INFO:tasks.rados.rados.0.smithi064.stdout:745: left oid 35 (ObjNum 353 snap 90 seq_num 353) 2024-03-04T06:11:05.788 INFO:tasks.rados.rados.0.smithi064.stdout:749: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-03-04T06:11:05.793 INFO:tasks.rados.rados.0.smithi064.stdout:749: writing smithi06431743-44 from 766485 to 1232118 tid 1 2024-03-04T06:11:05.801 INFO:tasks.rados.rados.0.smithi064.stdout:749: writing smithi06431743-44 from 1990637 to 2638413 tid 2 2024-03-04T06:11:05.809 INFO:tasks.rados.rados.0.smithi064.stdout:749: writing smithi06431743-44 from 3299280 to 3966485 tid 3 2024-03-04T06:11:05.809 INFO:tasks.rados.rados.0.smithi064.stdout:745: done (1 left) 2024-03-04T06:11:05.809 INFO:tasks.rados.rados.0.smithi064.stdout:750: read oid 5 snap 89 2024-03-04T06:11:05.809 INFO:tasks.rados.rados.0.smithi064.stdout:750: expect deleted 2024-03-04T06:11:05.809 INFO:tasks.rados.rados.0.smithi064.stdout:751: write oid 24 current snap is 90 2024-03-04T06:11:05.809 INFO:tasks.rados.rados.0.smithi064.stdout:751: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-03-04T06:11:05.816 INFO:tasks.rados.rados.0.smithi064.stdout:751: writing smithi06431743-24 from 783292 to 1324174 tid 1 2024-03-04T06:11:05.822 INFO:tasks.rados.rados.0.smithi064.stdout:751: writing smithi06431743-24 from 1932766 to 2440553 tid 2 2024-03-04T06:11:05.831 INFO:tasks.rados.rados.0.smithi064.stdout:751: writing smithi06431743-24 from 3210350 to 3782363 tid 3 2024-03-04T06:11:05.831 INFO:tasks.rados.rados.0.smithi064.stdout:752: write (excl) oid 43 current snap is 90 2024-03-04T06:11:05.831 INFO:tasks.rados.rados.0.smithi064.stdout:752: seq_num 357 ranges {400099=616131,1591248=408851} 2024-03-04T06:11:05.837 INFO:tasks.rados.rados.0.smithi064.stdout:752: writing smithi06431743-43 from 400099 to 1016230 tid 1 2024-03-04T06:11:05.842 INFO:tasks.rados.rados.0.smithi064.stdout:752: writing smithi06431743-43 from 1591248 to 2000099 tid 2 2024-03-04T06:11:05.842 INFO:tasks.rados.rados.0.smithi064.stdout:753: delete oid 21 current snap is 90 2024-03-04T06:11:05.846 INFO:tasks.rados.rados.0.smithi064.stdout:751: finishing write tid 1 to smithi06431743-24 2024-03-04T06:11:05.848 INFO:tasks.rados.rados.0.smithi064.stdout:750: done (4 left) 2024-03-04T06:11:05.848 INFO:tasks.rados.rados.0.smithi064.stdout:753: done (3 left) 2024-03-04T06:11:05.848 INFO:tasks.rados.rados.0.smithi064.stdout:754: delete oid 13 current snap is 90 2024-03-04T06:11:05.870 INFO:tasks.rados.rados.0.smithi064.stdout:751: finishing write tid 2 to smithi06431743-24 2024-03-04T06:11:05.880 INFO:tasks.rados.rados.0.smithi064.stdout:749: finishing write tid 1 to smithi06431743-44 2024-03-04T06:11:05.889 INFO:tasks.rados.rados.0.smithi064.stdout:752: finishing write tid 1 to smithi06431743-43 2024-03-04T06:11:05.897 INFO:tasks.rados.rados.0.smithi064.stdout:751: finishing write tid 3 to smithi06431743-24 2024-03-04T06:11:05.898 INFO:tasks.rados.rados.0.smithi064.stdout:754: done (3 left) 2024-03-04T06:11:05.898 INFO:tasks.rados.rados.0.smithi064.stdout:755: snap_create 2024-03-04T06:11:05.907 INFO:tasks.rados.rados.0.smithi064.stdout:751: finishing write tid 4 to smithi06431743-24 2024-03-04T06:11:05.907 INFO:tasks.rados.rados.0.smithi064.stdout:749: finishing write tid 2 to smithi06431743-44 2024-03-04T06:11:05.909 INFO:tasks.rados.rados.0.smithi064.stdout:751: finishing write tid 5 to smithi06431743-24 2024-03-04T06:11:05.909 INFO:tasks.rados.rados.0.smithi064.stdout:751: oid 24 updating version 0 to 247 2024-03-04T06:11:05.909 INFO:tasks.rados.rados.0.smithi064.stdout:751: oid 24 version 247 is already newer than 246 2024-03-04T06:11:05.910 INFO:tasks.rados.rados.0.smithi064.stdout:751: oid 24 updating version 247 to 249 2024-03-04T06:11:05.910 INFO:tasks.rados.rados.0.smithi064.stdout:751: oid 24 version 249 is already newer than 248 2024-03-04T06:11:05.910 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 249 (ObjNum 356 snap 90 seq_num 356) dirty exists 2024-03-04T06:11:05.910 INFO:tasks.rados.rados.0.smithi064.stdout:751: left oid 24 (ObjNum 356 snap 90 seq_num 356) 2024-03-04T06:11:05.911 INFO:tasks.rados.rados.0.smithi064.stdout:752: finishing write tid 2 to smithi06431743-43 2024-03-04T06:11:05.926 INFO:tasks.rados.rados.0.smithi064.stdout:752: finishing write tid 3 to smithi06431743-43 2024-03-04T06:11:05.926 INFO:tasks.rados.rados.0.smithi064.stdout:749: finishing write tid 3 to smithi06431743-44 2024-03-04T06:11:05.926 INFO:tasks.rados.rados.0.smithi064.stdout:752: finishing write tid 4 to smithi06431743-43 2024-03-04T06:11:05.926 INFO:tasks.rados.rados.0.smithi064.stdout:752: oid 43 updating version 0 to 132 2024-03-04T06:11:05.926 INFO:tasks.rados.rados.0.smithi064.stdout:752: oid 43 updating version 132 to 134 2024-03-04T06:11:05.926 INFO:tasks.rados.rados.0.smithi064.stdout:752: oid 43 version 134 is already newer than 133 2024-03-04T06:11:05.926 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 134 (ObjNum 357 snap 90 seq_num 357) dirty exists 2024-03-04T06:11:05.927 INFO:tasks.rados.rados.0.smithi064.stdout:752: left oid 43 (ObjNum 357 snap 90 seq_num 357) 2024-03-04T06:11:05.928 INFO:tasks.rados.rados.0.smithi064.stdout:749: finishing write tid 4 to smithi06431743-44 2024-03-04T06:11:05.929 INFO:tasks.rados.rados.0.smithi064.stdout:749: finishing write tid 5 to smithi06431743-44 2024-03-04T06:11:05.929 INFO:tasks.rados.rados.0.smithi064.stdout:749: oid 44 updating version 0 to 122 2024-03-04T06:11:05.929 INFO:tasks.rados.rados.0.smithi064.stdout:749: oid 44 version 122 is already newer than 121 2024-03-04T06:11:05.929 INFO:tasks.rados.rados.0.smithi064.stdout:749: oid 44 updating version 122 to 124 2024-03-04T06:11:05.929 INFO:tasks.rados.rados.0.smithi064.stdout:749: oid 44 version 124 is already newer than 124 2024-03-04T06:11:05.929 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 124 (ObjNum 355 snap 90 seq_num 355) dirty exists 2024-03-04T06:11:05.929 INFO:tasks.rados.rados.0.smithi064.stdout:749: left oid 44 (ObjNum 355 snap 90 seq_num 355) 2024-03-04T06:11:06.342 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:11:06.344 INFO:tasks.rados.rados.0.smithi064.stdout:749: done (3 left) 2024-03-04T06:11:06.344 INFO:tasks.rados.rados.0.smithi064.stdout:751: done (2 left) 2024-03-04T06:11:06.344 INFO:tasks.rados.rados.0.smithi064.stdout:752: done (1 left) 2024-03-04T06:11:06.344 INFO:tasks.rados.rados.0.smithi064.stdout:755: done (0 left) 2024-03-04T06:11:06.344 INFO:tasks.rados.rados.0.smithi064.stdout:756: snap_remove snap 88 2024-03-04T06:11:06.357 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:11:06.357 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:11:06.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:06.501+0000 7f91ecf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:06.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:06.507+0000 7f91ecf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:06.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:06.698 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:11:06.698 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:11:06.710 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-03-04T06:11:06.342983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93281633,"num_objects":260,"num_object_clones":188,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":42,"num_read":1556,"num_read_kb":1080253,"num_write":1350,"num_write_kb":336045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2786,"ondisk_log_size":2786,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":301808,"kb_used_data":172368,"kb_used_omap":0,"kb_used_meta":128064,"kb_avail":377185552,"statfs":{"total":386547056640,"available":386238005248,"internally_reserved":0,"allocated":176504832,"data_stored":175755270,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":131137536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2537890,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":21,"num_read_kb":23669,"num_write":11,"num_write_kb":3363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009544"},"pg_stats":[{"pgid":"3.f","version":"191'130","reported_seq":617,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.381767+0000","last_change":"2024-03-04T06:11:01.366349+0000","last_active":"2024-03-04T06:11:02.381767+0000","last_peered":"2024-03-04T06:11:02.381767+0000","last_clean":"2024-03-04T06:11:02.381767+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:11:02.381767+0000","last_undegraded":"2024-03-04T06:11:02.381767+0000","last_fullsized":"2024-03-04T06:11:02.381767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798415,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":37,"num_read_kb":12962,"num_write":72,"num_write_kb":19461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"184'114","reported_seq":631,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.214218+0000","last_change":"2024-03-04T06:11:03.269441+0000","last_active":"2024-03-04T06:11:05.214218+0000","last_peered":"2024-03-04T06:11:05.214218+0000","last_clean":"2024-03-04T06:11:05.214218+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:11:05.214218+0000","last_undegraded":"2024-03-04T06:11:05.214218+0000","last_fullsized":"2024-03-04T06:11:05.214218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311012,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":43,"num_read_kb":28823,"num_write":50,"num_write_kb":13094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"184'171","reported_seq":639,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-04T06:11:00.337592+0000","last_change":"2024-03-04T06:11:00.343064+0000","last_active":"2024-03-04T06:11:00.337592+0000","last_peered":"2024-03-04T06:11:00.337592+0000","last_clean":"2024-03-04T06:11:00.337592+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:11:00.337592+0000","last_undegraded":"2024-03-04T06:11:00.337592+0000","last_fullsized":"2024-03-04T06:11:00.337592+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11849001,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":81,"num_read_kb":51880,"num_write":88,"num_write_kb":24286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"193'126","reported_seq":639,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.452325+0000","last_change":"2024-03-04T06:11:01.357438+0000","last_active":"2024-03-04T06:11:02.452325+0000","last_peered":"2024-03-04T06:11:02.452325+0000","last_clean":"2024-03-04T06:11:02.452325+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:11:02.452325+0000","last_undegraded":"2024-03-04T06:11:02.452325+0000","last_fullsized":"2024-03-04T06:11:02.452325+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3882450,"num_objects":14,"num_object_clones":9,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":88,"num_read_kb":73885,"num_write":69,"num_write_kb":19676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"184'85","reported_seq":580,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-04T06:11:00.402346+0000","last_change":"2024-03-04T06:11:00.402346+0000","last_active":"2024-03-04T06:11:00.402346+0000","last_peered":"2024-03-04T06:11:00.402346+0000","last_clean":"2024-03-04T06:11:00.402346+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:11:00.402346+0000","last_undegraded":"2024-03-04T06:11:00.402346+0000","last_fullsized":"2024-03-04T06:11:00.402346+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":29,"num_read_kb":17040,"num_write":29,"num_write_kb":7475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"184'119","reported_seq":626,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.380583+0000","last_change":"2024-03-04T06:11:01.366648+0000","last_active":"2024-03-04T06:11:02.380583+0000","last_peered":"2024-03-04T06:11:02.380583+0000","last_clean":"2024-03-04T06:11:02.380583+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:11:02.380583+0000","last_undegraded":"2024-03-04T06:11:02.380583+0000","last_fullsized":"2024-03-04T06:11:02.380583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8638864,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":64474,"num_write":59,"num_write_kb":17872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":537,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.382650+0000","last_change":"2024-03-04T06:11:01.366032+0000","last_active":"2024-03-04T06:11:02.382650+0000","last_peered":"2024-03-04T06:11:02.382650+0000","last_clean":"2024-03-04T06:11:02.382650+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:11:02.382650+0000","last_undegraded":"2024-03-04T06:11:02.382650+0000","last_fullsized":"2024-03-04T06:11:02.382650+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"184'129","reported_seq":609,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-04T06:11:00.339389+0000","last_change":"2024-03-04T06:11:00.342606+0000","last_active":"2024-03-04T06:11:00.339389+0000","last_peered":"2024-03-04T06:11:00.339389+0000","last_clean":"2024-03-04T06:11:00.339389+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:11:00.339389+0000","last_undegraded":"2024-03-04T06:11:00.339389+0000","last_fullsized":"2024-03-04T06:11:00.339389+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3680733,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":69,"num_read_kb":69049,"num_write":58,"num_write_kb":16367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":599,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-04T06:11:01.349565+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:11:01.349565+0000","last_peered":"2024-03-04T06:11:01.349565+0000","last_clean":"2024-03-04T06:11:01.349565+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:11:01.349565+0000","last_undegraded":"2024-03-04T06:11:01.349565+0000","last_fullsized":"2024-03-04T06:11:01.349565+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":449,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-04T06:11:00.346167+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:11:00.346167+0000","last_peered":"2024-03-04T06:11:00.346167+0000","last_clean":"2024-03-04T06:11:00.346167+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:11:00.346167+0000","last_undegraded":"2024-03-04T06:11:00.346167+0000","last_fullsized":"2024-03-04T06:11:00.346167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"184'130","reported_seq":613,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-04T06:11:00.343180+0000","last_change":"2024-03-04T06:11:00.348269+0000","last_active":"2024-03-04T06:11:00.343180+0000","last_peered":"2024-03-04T06:11:00.343180+0000","last_clean":"2024-03-04T06:11:00.343180+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:11:00.343180+0000","last_undegraded":"2024-03-04T06:11:00.343180+0000","last_fullsized":"2024-03-04T06:11:00.343180+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5462308,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":88,"num_read_kb":78153,"num_write":50,"num_write_kb":13051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":459,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.379118+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:11:02.379118+0000","last_peered":"2024-03-04T06:11:02.379118+0000","last_clean":"2024-03-04T06:11:02.379118+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:11:02.379118+0000","last_undegraded":"2024-03-04T06:11:02.379118+0000","last_fullsized":"2024-03-04T06:11:02.379118+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"184'145","reported_seq":617,"reported_epoch":192,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:11:01.350651+0000","last_change":"2024-03-04T06:11:01.350651+0000","last_active":"2024-03-04T06:11:01.350651+0000","last_peered":"2024-03-04T06:11:01.350651+0000","last_clean":"2024-03-04T06:11:01.350651+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:11:01.350651+0000","last_undegraded":"2024-03-04T06:11:01.350651+0000","last_fullsized":"2024-03-04T06:11:01.350651+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":68,"num_read_kb":54041,"num_write":49,"num_write_kb":13998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":449,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-04T06:11:00.330860+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:11:00.330860+0000","last_peered":"2024-03-04T06:11:00.330860+0000","last_clean":"2024-03-04T06:11:00.330860+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:11:00.330860+0000","last_undegraded":"2024-03-04T06:11:00.330860+0000","last_fullsized":"2024-03-04T06:11:00.330860+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"187'344","reported_seq":785,"reported_epoch":192,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:11:01.357063+0000","last_change":"2024-03-04T06:11:01.357063+0000","last_active":"2024-03-04T06:11:01.357063+0000","last_peered":"2024-03-04T06:11:01.357063+0000","last_clean":"2024-03-04T06:11:01.357063+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:11:01.357063+0000","last_undegraded":"2024-03-04T06:11:01.357063+0000","last_fullsized":"2024-03-04T06:11:01.357063+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16588674,"num_objects":34,"num_object_clones":24,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":217,"num_read_kb":160795,"num_write":163,"num_write_kb":46552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":459,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.372742+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:11:02.372742+0000","last_peered":"2024-03-04T06:11:02.372742+0000","last_clean":"2024-03-04T06:11:02.372742+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:11:02.372742+0000","last_undegraded":"2024-03-04T06:11:02.372742+0000","last_fullsized":"2024-03-04T06:11:02.372742+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"192'355","reported_seq":768,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.269521+0000","last_change":"2024-03-04T06:11:03.268972+0000","last_active":"2024-03-04T06:11:05.269521+0000","last_peered":"2024-03-04T06:11:05.269521+0000","last_clean":"2024-03-04T06:11:05.269521+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:11:05.269521+0000","last_undegraded":"2024-03-04T06:11:05.269521+0000","last_fullsized":"2024-03-04T06:11:05.269521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11126333,"num_objects":33,"num_object_clones":24,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":128,"num_read_kb":78503,"num_write":165,"num_write_kb":44176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"184'143","reported_seq":641,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-04T06:11:00.354210+0000","last_change":"2024-03-04T06:11:00.358568+0000","last_active":"2024-03-04T06:11:00.354210+0000","last_peered":"2024-03-04T06:11:00.354210+0000","last_clean":"2024-03-04T06:11:00.354210+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:11:00.354210+0000","last_undegraded":"2024-03-04T06:11:00.354210+0000","last_fullsized":"2024-03-04T06:11:00.354210+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6571570,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":125,"num_read_kb":98834,"num_write":64,"num_write_kb":16590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":490,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.224317+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:11:05.224317+0000","last_peered":"2024-03-04T06:11:05.224317+0000","last_clean":"2024-03-04T06:11:05.224317+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:11:05.224317+0000","last_undegraded":"2024-03-04T06:11:05.224317+0000","last_fullsized":"2024-03-04T06:11:05.224317+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"185'125","reported_seq":652,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.227731+0000","last_change":"2024-03-04T06:11:03.273724+0000","last_active":"2024-03-04T06:11:05.227731+0000","last_peered":"2024-03-04T06:11:05.227731+0000","last_clean":"2024-03-04T06:11:05.227731+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:11:05.227731+0000","last_undegraded":"2024-03-04T06:11:05.227731+0000","last_fullsized":"2024-03-04T06:11:05.227731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":78,"num_read_kb":43420,"num_write":55,"num_write_kb":14221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-04T06:11:01.356815+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:11:01.356815+0000","last_peered":"2024-03-04T06:11:01.356815+0000","last_clean":"2024-03-04T06:11:01.356815+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:11:01.356815+0000","last_undegraded":"2024-03-04T06:11:01.356815+0000","last_fullsized":"2024-03-04T06:11:01.356815+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"185'239","reported_seq":693,"reported_epoch":192,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:11:01.344019+0000","last_change":"2024-03-04T06:11:01.344019+0000","last_active":"2024-03-04T06:11:01.344019+0000","last_peered":"2024-03-04T06:11:01.344019+0000","last_clean":"2024-03-04T06:11:01.344019+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:11:01.344019+0000","last_undegraded":"2024-03-04T06:11:01.344019+0000","last_fullsized":"2024-03-04T06:11:01.344019+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3848770,"num_objects":22,"num_object_clones":17,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":123,"num_read_kb":92829,"num_write":108,"num_write_kb":29658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.215811+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:11:05.215811+0000","last_peered":"2024-03-04T06:11:05.215811+0000","last_clean":"2024-03-04T06:11:05.215811+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:11:05.215811+0000","last_undegraded":"2024-03-04T06:11:05.215811+0000","last_fullsized":"2024-03-04T06:11:05.215811+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"194'334","reported_seq":778,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.266261+0000","last_change":"2024-03-04T06:11:03.269270+0000","last_active":"2024-03-04T06:11:05.266261+0000","last_peered":"2024-03-04T06:11:05.266261+0000","last_clean":"2024-03-04T06:11:05.266261+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:11:05.266261+0000","last_undegraded":"2024-03-04T06:11:05.266261+0000","last_fullsized":"2024-03-04T06:11:05.266261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13933135,"num_objects":27,"num_object_clones":23,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":196,"num_read_kb":155474,"num_write":134,"num_write_kb":37674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":449,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-04T06:11:00.346073+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:11:00.346073+0000","last_peered":"2024-03-04T06:11:00.346073+0000","last_clean":"2024-03-04T06:11:00.346073+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:11:00.346073+0000","last_undegraded":"2024-03-04T06:11:00.346073+0000","last_fullsized":"2024-03-04T06:11:00.346073+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":92691265,"num_objects":258,"num_object_clones":188,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":42,"num_read":1453,"num_read_kb":1080162,"num_write":1213,"num_write_kb":334151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2689,"ondisk_log_size":2689,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87872,"kb_used_data":52728,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94283968,"statfs":{"total":96636764160,"available":96546783232,"internally_reserved":0,"allocated":53993472,"data_stored":54277634,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77884,"kb_used_data":46900,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94293956,"statfs":{"total":96636764160,"available":96557010944,"internally_reserved":0,"allocated":48025600,"data_stored":47819774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74764,"kb_used_data":42436,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94297076,"statfs":{"total":96636764160,"available":96560205824,"internally_reserved":0,"allocated":43454464,"data_stored":42914583,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":61288,"kb_used_data":30304,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94310552,"statfs":{"total":96636764160,"available":96574005248,"internally_reserved":0,"allocated":31031296,"data_stored":30743279,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:11:06.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:11:06.756 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:06.755+0000 7ff2b86ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:06.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:06.759+0000 7ff2b86ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:06.821 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:06.817+0000 7f0d93813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:06.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:06.824+0000 7f0d93813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:11:07.014 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:11:07.026 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-03-04T06:11:06.747550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":97349026,"num_objects":262,"num_object_clones":189,"num_object_copies":786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":42,"num_read":1564,"num_read_kb":1090069,"num_write":1360,"num_write_kb":339631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2797,"ondisk_log_size":2797,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":306280,"kb_used_data":174152,"kb_used_omap":0,"kb_used_meta":128064,"kb_avail":377181080,"statfs":{"total":386547056640,"available":386233425920,"internally_reserved":0,"allocated":178331648,"data_stored":176712497,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":131137536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1529503,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":29,"num_read_kb":33485,"num_write":21,"num_write_kb":6949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000793"},"pg_stats":[{"pgid":"3.f","version":"191'130","reported_seq":617,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.381767+0000","last_change":"2024-03-04T06:11:01.366349+0000","last_active":"2024-03-04T06:11:02.381767+0000","last_peered":"2024-03-04T06:11:02.381767+0000","last_clean":"2024-03-04T06:11:02.381767+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:11:02.381767+0000","last_undegraded":"2024-03-04T06:11:02.381767+0000","last_fullsized":"2024-03-04T06:11:02.381767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798415,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":37,"num_read_kb":12962,"num_write":72,"num_write_kb":19461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"184'114","reported_seq":631,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.214218+0000","last_change":"2024-03-04T06:11:03.269441+0000","last_active":"2024-03-04T06:11:05.214218+0000","last_peered":"2024-03-04T06:11:05.214218+0000","last_clean":"2024-03-04T06:11:05.214218+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:11:05.214218+0000","last_undegraded":"2024-03-04T06:11:05.214218+0000","last_fullsized":"2024-03-04T06:11:05.214218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311012,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":43,"num_read_kb":28823,"num_write":50,"num_write_kb":13094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"184'171","reported_seq":654,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.220291+0000","last_change":"2024-03-04T06:11:04.223115+0000","last_active":"2024-03-04T06:11:05.220291+0000","last_peered":"2024-03-04T06:11:05.220291+0000","last_clean":"2024-03-04T06:11:05.220291+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:11:05.220291+0000","last_undegraded":"2024-03-04T06:11:05.220291+0000","last_fullsized":"2024-03-04T06:11:05.220291+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11849001,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":81,"num_read_kb":51880,"num_write":88,"num_write_kb":24286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"193'126","reported_seq":639,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.452325+0000","last_change":"2024-03-04T06:11:01.357438+0000","last_active":"2024-03-04T06:11:02.452325+0000","last_peered":"2024-03-04T06:11:02.452325+0000","last_clean":"2024-03-04T06:11:02.452325+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:11:02.452325+0000","last_undegraded":"2024-03-04T06:11:02.452325+0000","last_fullsized":"2024-03-04T06:11:02.452325+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3882450,"num_objects":14,"num_object_clones":9,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":88,"num_read_kb":73885,"num_write":69,"num_write_kb":19676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"194'86","reported_seq":594,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.242743+0000","last_change":"2024-03-04T06:11:03.225627+0000","last_active":"2024-03-04T06:11:05.242743+0000","last_peered":"2024-03-04T06:11:05.242743+0000","last_clean":"2024-03-04T06:11:05.242743+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:11:05.242743+0000","last_undegraded":"2024-03-04T06:11:05.242743+0000","last_fullsized":"2024-03-04T06:11:05.242743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":29,"num_read_kb":17040,"num_write":29,"num_write_kb":7475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"184'119","reported_seq":626,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.380583+0000","last_change":"2024-03-04T06:11:01.366648+0000","last_active":"2024-03-04T06:11:02.380583+0000","last_peered":"2024-03-04T06:11:02.380583+0000","last_clean":"2024-03-04T06:11:02.380583+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:11:02.380583+0000","last_undegraded":"2024-03-04T06:11:02.380583+0000","last_fullsized":"2024-03-04T06:11:02.380583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8638864,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":83,"num_read_kb":64474,"num_write":59,"num_write_kb":17872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":537,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.382650+0000","last_change":"2024-03-04T06:11:01.366032+0000","last_active":"2024-03-04T06:11:02.382650+0000","last_peered":"2024-03-04T06:11:02.382650+0000","last_clean":"2024-03-04T06:11:02.382650+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:11:02.382650+0000","last_undegraded":"2024-03-04T06:11:02.382650+0000","last_fullsized":"2024-03-04T06:11:02.382650+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"184'129","reported_seq":627,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.221712+0000","last_change":"2024-03-04T06:11:04.223616+0000","last_active":"2024-03-04T06:11:05.221712+0000","last_peered":"2024-03-04T06:11:05.221712+0000","last_clean":"2024-03-04T06:11:05.221712+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:11:05.221712+0000","last_undegraded":"2024-03-04T06:11:05.221712+0000","last_fullsized":"2024-03-04T06:11:05.221712+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3680733,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":75,"num_read_kb":78863,"num_write":58,"num_write_kb":16367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":609,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.232652+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:11:05.232652+0000","last_peered":"2024-03-04T06:11:05.232652+0000","last_clean":"2024-03-04T06:11:05.232652+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:11:05.232652+0000","last_undegraded":"2024-03-04T06:11:05.232652+0000","last_fullsized":"2024-03-04T06:11:05.232652+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":462,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.228574+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:11:05.228574+0000","last_peered":"2024-03-04T06:11:05.228574+0000","last_clean":"2024-03-04T06:11:05.228574+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:11:05.228574+0000","last_undegraded":"2024-03-04T06:11:05.228574+0000","last_fullsized":"2024-03-04T06:11:05.228574+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"184'130","reported_seq":628,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.226295+0000","last_change":"2024-03-04T06:11:04.230892+0000","last_active":"2024-03-04T06:11:05.226295+0000","last_peered":"2024-03-04T06:11:05.226295+0000","last_clean":"2024-03-04T06:11:05.226295+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:11:05.226295+0000","last_undegraded":"2024-03-04T06:11:05.226295+0000","last_fullsized":"2024-03-04T06:11:05.226295+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5462308,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":88,"num_read_kb":78153,"num_write":50,"num_write_kb":13051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":459,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.379118+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:11:02.379118+0000","last_peered":"2024-03-04T06:11:02.379118+0000","last_clean":"2024-03-04T06:11:02.379118+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:11:02.379118+0000","last_undegraded":"2024-03-04T06:11:02.379118+0000","last_fullsized":"2024-03-04T06:11:02.379118+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"184'145","reported_seq":629,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.237765+0000","last_change":"2024-03-04T06:11:03.218848+0000","last_active":"2024-03-04T06:11:05.237765+0000","last_peered":"2024-03-04T06:11:05.237765+0000","last_clean":"2024-03-04T06:11:05.237765+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:11:05.237765+0000","last_undegraded":"2024-03-04T06:11:05.237765+0000","last_fullsized":"2024-03-04T06:11:05.237765+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":68,"num_read_kb":54041,"num_write":49,"num_write_kb":13998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.213380+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:11:05.213380+0000","last_peered":"2024-03-04T06:11:05.213380+0000","last_clean":"2024-03-04T06:11:05.213380+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:11:05.213380+0000","last_undegraded":"2024-03-04T06:11:05.213380+0000","last_fullsized":"2024-03-04T06:11:05.213380+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"187'344","reported_seq":797,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.246468+0000","last_change":"2024-03-04T06:11:03.227067+0000","last_active":"2024-03-04T06:11:05.246468+0000","last_peered":"2024-03-04T06:11:05.246468+0000","last_clean":"2024-03-04T06:11:05.246468+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:11:05.246468+0000","last_undegraded":"2024-03-04T06:11:05.246468+0000","last_fullsized":"2024-03-04T06:11:05.246468+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16588674,"num_objects":34,"num_object_clones":24,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":217,"num_read_kb":160795,"num_write":163,"num_write_kb":46552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":459,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-03-04T06:11:02.372742+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:11:02.372742+0000","last_peered":"2024-03-04T06:11:02.372742+0000","last_clean":"2024-03-04T06:11:02.372742+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:11:02.372742+0000","last_undegraded":"2024-03-04T06:11:02.372742+0000","last_fullsized":"2024-03-04T06:11:02.372742+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"192'355","reported_seq":768,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.269521+0000","last_change":"2024-03-04T06:11:03.268972+0000","last_active":"2024-03-04T06:11:05.269521+0000","last_peered":"2024-03-04T06:11:05.269521+0000","last_clean":"2024-03-04T06:11:05.269521+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:11:05.269521+0000","last_undegraded":"2024-03-04T06:11:05.269521+0000","last_fullsized":"2024-03-04T06:11:05.269521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11126333,"num_objects":33,"num_object_clones":24,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":128,"num_read_kb":78503,"num_write":165,"num_write_kb":44176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"184'143","reported_seq":656,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.227909+0000","last_change":"2024-03-04T06:11:04.231788+0000","last_active":"2024-03-04T06:11:05.227909+0000","last_peered":"2024-03-04T06:11:05.227909+0000","last_clean":"2024-03-04T06:11:05.227909+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:11:05.227909+0000","last_undegraded":"2024-03-04T06:11:05.227909+0000","last_fullsized":"2024-03-04T06:11:05.227909+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6571570,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":125,"num_read_kb":98834,"num_write":64,"num_write_kb":16590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":490,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.224317+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:11:05.224317+0000","last_peered":"2024-03-04T06:11:05.224317+0000","last_clean":"2024-03-04T06:11:05.224317+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:11:05.224317+0000","last_undegraded":"2024-03-04T06:11:05.224317+0000","last_fullsized":"2024-03-04T06:11:05.224317+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"185'125","reported_seq":652,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.227731+0000","last_change":"2024-03-04T06:11:03.273724+0000","last_active":"2024-03-04T06:11:05.227731+0000","last_peered":"2024-03-04T06:11:05.227731+0000","last_clean":"2024-03-04T06:11:05.227731+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:11:05.227731+0000","last_undegraded":"2024-03-04T06:11:05.227731+0000","last_fullsized":"2024-03-04T06:11:05.227731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":78,"num_read_kb":43420,"num_write":55,"num_write_kb":14221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":476,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.259535+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:11:05.259535+0000","last_peered":"2024-03-04T06:11:05.259535+0000","last_clean":"2024-03-04T06:11:05.259535+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:11:05.259535+0000","last_undegraded":"2024-03-04T06:11:05.259535+0000","last_fullsized":"2024-03-04T06:11:05.259535+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"196'249","reported_seq":715,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.895800+0000","last_change":"2024-03-04T06:11:03.219291+0000","last_active":"2024-03-04T06:11:05.895800+0000","last_peered":"2024-03-04T06:11:05.895800+0000","last_clean":"2024-03-04T06:11:05.895800+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:11:05.895800+0000","last_undegraded":"2024-03-04T06:11:05.895800+0000","last_fullsized":"2024-03-04T06:11:05.895800+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7916163,"num_objects":24,"num_object_clones":18,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":125,"num_read_kb":92831,"num_write":118,"num_write_kb":33244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.215811+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:11:05.215811+0000","last_peered":"2024-03-04T06:11:05.215811+0000","last_clean":"2024-03-04T06:11:05.215811+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:11:05.215811+0000","last_undegraded":"2024-03-04T06:11:05.215811+0000","last_fullsized":"2024-03-04T06:11:05.215811+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"194'334","reported_seq":778,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.266261+0000","last_change":"2024-03-04T06:11:03.269270+0000","last_active":"2024-03-04T06:11:05.266261+0000","last_peered":"2024-03-04T06:11:05.266261+0000","last_clean":"2024-03-04T06:11:05.266261+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:11:05.266261+0000","last_undegraded":"2024-03-04T06:11:05.266261+0000","last_fullsized":"2024-03-04T06:11:05.266261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13933135,"num_objects":27,"num_object_clones":23,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":196,"num_read_kb":155474,"num_write":134,"num_write_kb":37674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":462,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-04T06:11:05.225546+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:11:05.225546+0000","last_peered":"2024-03-04T06:11:05.225546+0000","last_clean":"2024-03-04T06:11:05.225546+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:11:05.225546+0000","last_undegraded":"2024-03-04T06:11:05.225546+0000","last_fullsized":"2024-03-04T06:11:05.225546+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":96758658,"num_objects":260,"num_object_clones":189,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":42,"num_read":1461,"num_read_kb":1089978,"num_write":1223,"num_write_kb":337737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2700,"ondisk_log_size":2700,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89932,"kb_used_data":53444,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94281908,"statfs":{"total":96636764160,"available":96544673792,"internally_reserved":0,"allocated":54726656,"data_stored":54738893,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":80296,"kb_used_data":47968,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94291544,"statfs":{"total":96636764160,"available":96554541056,"internally_reserved":0,"allocated":49119232,"data_stored":48315742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74764,"kb_used_data":42436,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94297076,"statfs":{"total":96636764160,"available":96560205824,"internally_reserved":0,"allocated":43454464,"data_stored":42914583,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":61288,"kb_used_data":30304,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94310552,"statfs":{"total":96636764160,"available":96574005248,"internally_reserved":0,"allocated":31031296,"data_stored":30743279,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:11:07.027 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:11:07.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:11:07.028 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:11:07.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:11:07.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:07.130+0000 7f72ecaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:07.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:07.138+0000 7f72ecaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:07.150 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:07.150 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:07.151 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:07.151 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:07.151 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:07.151 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:07.151 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:07.151 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:07.151 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:07.151 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:07.151 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:07.151 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:07.154 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:07.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:11:07.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:07.256+0000 7efce219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:07.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:07.264+0000 7efce219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:07.277 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:07.280 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:07.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:11:07.344 INFO:tasks.rados.rados.0.smithi064.stdout:756: done (0 left) 2024-03-04T06:11:07.344 INFO:tasks.rados.rados.0.smithi064.stdout:757: snap_remove snap 90 2024-03-04T06:11:07.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:07.384+0000 7febd946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:07.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:07.391+0000 7febd946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:07.405 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:07.408 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:07.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:11:07.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:07.511+0000 7f890626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:07.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:07.518+0000 7f890626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:07.531 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:07.532 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:07.532 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:07.532 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:07.532 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:07.532 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:07.532 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:07.532 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:07.532 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:07.532 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:07.532 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:07.532 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:07.534 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:08.350 INFO:tasks.rados.rados.0.smithi064.stdout:757: done (0 left) 2024-03-04T06:11:08.350 INFO:tasks.rados.rados.0.smithi064.stdout:758: write oid 26 current snap is 91 2024-03-04T06:11:08.351 INFO:tasks.rados.rados.0.smithi064.stdout:758: seq_num 360 ranges {450520=558231,1483047=515125} 2024-03-04T06:11:08.357 INFO:tasks.rados.rados.0.smithi064.stdout:758: writing smithi06431743-26 from 450520 to 1008751 tid 1 2024-03-04T06:11:08.365 INFO:tasks.rados.rados.0.smithi064.stdout:758: writing smithi06431743-26 from 1483047 to 1998172 tid 2 2024-03-04T06:11:08.365 INFO:tasks.rados.rados.0.smithi064.stdout:759: read oid 29 snap -1 2024-03-04T06:11:08.365 INFO:tasks.rados.rados.0.smithi064.stdout:759: expect (ObjNum 335 snap 83 seq_num 335) 2024-03-04T06:11:08.365 INFO:tasks.rados.rados.0.smithi064.stdout:760: write (excl) oid 9 current snap is 91 2024-03-04T06:11:08.365 INFO:tasks.rados.rados.0.smithi064.stdout:760: seq_num 361 ranges {467327=633480,1825176=242151} 2024-03-04T06:11:08.372 INFO:tasks.rados.rados.0.smithi064.stdout:760: writing smithi06431743-9 from 467327 to 1100807 tid 1 2024-03-04T06:11:08.377 INFO:tasks.rados.rados.0.smithi064.stdout:760: writing smithi06431743-9 from 1825176 to 2067327 tid 2 2024-03-04T06:11:08.377 INFO:tasks.rados.rados.0.smithi064.stdout:761: snap_remove snap 89 2024-03-04T06:11:08.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:08.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:08.936+0000 7ffbdcf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:08.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:08.940+0000 7ffbdcf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:09.356 INFO:tasks.rados.rados.0.smithi064.stdout:761: done (3 left) 2024-03-04T06:11:09.356 INFO:tasks.rados.rados.0.smithi064.stdout:762: delete oid 5 current snap is 91 2024-03-04T06:11:09.356 INFO:tasks.rados.rados.0.smithi064.stdout:758: finishing write tid 1 to smithi06431743-26 2024-03-04T06:11:09.356 INFO:tasks.rados.rados.0.smithi064.stdout:759: expect (ObjNum 335 snap 83 seq_num 335) 2024-03-04T06:11:09.494 INFO:tasks.rados.rados.0.smithi064.stdout:760: finishing write tid 1 to smithi06431743-9 2024-03-04T06:11:09.494 INFO:tasks.rados.rados.0.smithi064.stdout:758: finishing write tid 2 to smithi06431743-26 2024-03-04T06:11:09.494 INFO:tasks.rados.rados.0.smithi064.stdout:759: done (3 left) 2024-03-04T06:11:09.494 INFO:tasks.rados.rados.0.smithi064.stdout:762: done (2 left) 2024-03-04T06:11:09.494 INFO:tasks.rados.rados.0.smithi064.stdout:763: write (excl) oid 40 current snap is 91 2024-03-04T06:11:09.494 INFO:tasks.rados.rados.0.smithi064.stdout:760: finishing write tid 2 to smithi06431743-9 2024-03-04T06:11:09.494 INFO:tasks.rados.rados.0.smithi064.stdout:763: seq_num 363 ranges {500941=783978,1942140=158801} 2024-03-04T06:11:09.500 INFO:tasks.rados.rados.0.smithi064.stdout:763: writing smithi06431743-40 from 500941 to 1284919 tid 1 2024-03-04T06:11:09.507 INFO:tasks.rados.rados.0.smithi064.stdout:763: writing smithi06431743-40 from 1942140 to 2100941 tid 2 2024-03-04T06:11:09.507 INFO:tasks.rados.rados.0.smithi064.stdout:764: write oid 41 current snap is 91 2024-03-04T06:11:09.507 INFO:tasks.rados.rados.0.smithi064.stdout:764: seq_num 364 ranges {517748=459227,1600622=406110} 2024-03-04T06:11:09.512 INFO:tasks.rados.rados.0.smithi064.stdout:764: writing smithi06431743-41 from 517748 to 976975 tid 1 2024-03-04T06:11:09.520 INFO:tasks.rados.rados.0.smithi064.stdout:764: writing smithi06431743-41 from 1600622 to 2006732 tid 2 2024-03-04T06:11:09.520 INFO:tasks.rados.rados.0.smithi064.stdout:765: write (excl) oid 45 current snap is 91 2024-03-04T06:11:09.520 INFO:tasks.rados.rados.0.smithi064.stdout:765: seq_num 365 ranges {534555=650829,1659104=475451} 2024-03-04T06:11:09.528 INFO:tasks.rados.rados.0.smithi064.stdout:765: writing smithi06431743-45 from 534555 to 1185384 tid 1 2024-03-04T06:11:09.535 INFO:tasks.rados.rados.0.smithi064.stdout:765: writing smithi06431743-45 from 1659104 to 2134555 tid 2 2024-03-04T06:11:09.535 INFO:tasks.rados.rados.0.smithi064.stdout:766: write (excl) oid 17 current snap is 91 2024-03-04T06:11:09.535 INFO:tasks.rados.rados.0.smithi064.stdout:766: seq_num 366 ranges {551362=726078,1717586=409779} 2024-03-04T06:11:09.544 INFO:tasks.rados.rados.0.smithi064.stdout:766: writing smithi06431743-17 from 551362 to 1277440 tid 1 2024-03-04T06:11:09.551 INFO:tasks.rados.rados.0.smithi064.stdout:766: writing smithi06431743-17 from 1717586 to 2127365 tid 2 2024-03-04T06:11:09.552 INFO:tasks.rados.rados.0.smithi064.stdout:767: write (excl) oid 10 current snap is 91 2024-03-04T06:11:09.552 INFO:tasks.rados.rados.0.smithi064.stdout:767: seq_num 367 ranges {568169=401327,1376068=669790} 2024-03-04T06:11:09.557 INFO:tasks.rados.rados.0.smithi064.stdout:767: writing smithi06431743-10 from 568169 to 969496 tid 1 2024-03-04T06:11:09.565 INFO:tasks.rados.rados.0.smithi064.stdout:767: writing smithi06431743-10 from 1376068 to 2045858 tid 2 2024-03-04T06:11:09.565 INFO:tasks.rados.rados.0.smithi064.stdout:768: write (excl) oid 8 current snap is 91 2024-03-04T06:11:09.565 INFO:tasks.rados.rados.0.smithi064.stdout:768: seq_num 368 ranges {584976=476576,1834550=350426} 2024-03-04T06:11:09.572 INFO:tasks.rados.rados.0.smithi064.stdout:768: writing smithi06431743-8 from 584976 to 1061552 tid 1 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:768: writing smithi06431743-8 from 1834550 to 2184976 tid 2 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:758: finishing write tid 3 to smithi06431743-26 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:760: finishing write tid 3 to smithi06431743-9 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:758: finishing write tid 4 to smithi06431743-26 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:758: oid 26 updating version 0 to 348 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:758: oid 26 version 348 is already newer than 346 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:758: oid 26 version 348 is already newer than 348 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 348 (ObjNum 360 snap 91 seq_num 360) dirty exists 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:758: left oid 26 (ObjNum 360 snap 91 seq_num 360) 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:760: finishing write tid 4 to smithi06431743-9 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:760: oid 9 updating version 0 to 338 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:760: oid 9 version 338 is already newer than 336 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:760: oid 9 version 338 is already newer than 338 2024-03-04T06:11:09.580 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 338 (ObjNum 361 snap 91 seq_num 361) dirty exists 2024-03-04T06:11:09.581 INFO:tasks.rados.rados.0.smithi064.stdout:760: left oid 9 (ObjNum 361 snap 91 seq_num 361) 2024-03-04T06:11:09.581 INFO:tasks.rados.rados.0.smithi064.stdout:764: finishing write tid 1 to smithi06431743-41 2024-03-04T06:11:09.581 INFO:tasks.rados.rados.0.smithi064.stdout:764: finishing write tid 2 to smithi06431743-41 2024-03-04T06:11:09.581 INFO:tasks.rados.rados.0.smithi064.stdout:765: finishing write tid 1 to smithi06431743-45 2024-03-04T06:11:09.581 INFO:tasks.rados.rados.0.smithi064.stdout:758: done (7 left) 2024-03-04T06:11:09.581 INFO:tasks.rados.rados.0.smithi064.stdout:760: done (6 left) 2024-03-04T06:11:09.581 INFO:tasks.rados.rados.0.smithi064.stdout:769: write (excl) oid 6 current snap is 91 2024-03-04T06:11:09.581 INFO:tasks.rados.rados.0.smithi064.stdout:769: seq_num 369 ranges {601783=551825,1776679=425104} 2024-03-04T06:11:09.585 INFO:tasks.rados.rados.0.smithi064.stdout:769: writing smithi06431743-6 from 601783 to 1153608 tid 1 2024-03-04T06:11:09.592 INFO:tasks.rados.rados.0.smithi064.stdout:769: writing smithi06431743-6 from 1776679 to 2201783 tid 2 2024-03-04T06:11:09.592 INFO:tasks.rados.rados.0.smithi064.stdout:770: read oid 48 snap -1 2024-03-04T06:11:09.592 INFO:tasks.rados.rados.0.smithi064.stdout:770: expect deleted 2024-03-04T06:11:09.592 INFO:tasks.rados.rados.0.smithi064.stdout:771: write oid 26 current snap is 91 2024-03-04T06:11:09.592 INFO:tasks.rados.rados.0.smithi064.stdout:766: finishing write tid 1 to smithi06431743-17 2024-03-04T06:11:09.592 INFO:tasks.rados.rados.0.smithi064.stdout:771: seq_num 370 ranges {618590=627074,1835161=383429} 2024-03-04T06:11:09.599 INFO:tasks.rados.rados.0.smithi064.stdout:771: writing smithi06431743-26 from 618590 to 1245664 tid 1 2024-03-04T06:11:09.605 INFO:tasks.rados.rados.0.smithi064.stdout:771: writing smithi06431743-26 from 1835161 to 2218590 tid 2 2024-03-04T06:11:09.606 INFO:tasks.rados.rados.0.smithi064.stdout:772: read oid 2 snap -1 2024-03-04T06:11:09.606 INFO:tasks.rados.rados.0.smithi064.stdout:772: expect deleted 2024-03-04T06:11:09.606 INFO:tasks.rados.rados.0.smithi064.stdout:773: read oid 24 snap -1 2024-03-04T06:11:09.606 INFO:tasks.rados.rados.0.smithi064.stdout:773: expect (ObjNum 356 snap 90 seq_num 356) 2024-03-04T06:11:09.606 INFO:tasks.rados.rados.0.smithi064.stdout:774: read oid 29 snap -1 2024-03-04T06:11:09.606 INFO:tasks.rados.rados.0.smithi064.stdout:774: expect (ObjNum 335 snap 83 seq_num 335) 2024-03-04T06:11:09.606 INFO:tasks.rados.rados.0.smithi064.stdout:775: read oid 13 snap -1 2024-03-04T06:11:09.606 INFO:tasks.rados.rados.0.smithi064.stdout:775: expect deleted 2024-03-04T06:11:09.606 INFO:tasks.rados.rados.0.smithi064.stdout:776: write oid 34 current snap is 91 2024-03-04T06:11:09.606 INFO:tasks.rados.rados.0.smithi064.stdout:768: finishing write tid 1 to smithi06431743-8 2024-03-04T06:11:09.606 INFO:tasks.rados.rados.0.smithi064.stdout:776: seq_num 371 ranges {635397=702323,1893643=341754} 2024-03-04T06:11:09.613 INFO:tasks.rados.rados.0.smithi064.stdout:776: writing smithi06431743-34 from 635397 to 1337720 tid 1 2024-03-04T06:11:09.618 INFO:tasks.rados.rados.0.smithi064.stdout:776: writing smithi06431743-34 from 1893643 to 2235397 tid 2 2024-03-04T06:11:09.618 INFO:tasks.rados.rados.0.smithi064.stdout:777: write oid 18 current snap is 91 2024-03-04T06:11:09.618 INFO:tasks.rados.rados.0.smithi064.stdout:777: seq_num 372 ranges {652204=777572,1952125=300079} 2024-03-04T06:11:09.626 INFO:tasks.rados.rados.0.smithi064.stdout:777: writing smithi06431743-18 from 652204 to 1429776 tid 1 2024-03-04T06:11:09.631 INFO:tasks.rados.rados.0.smithi064.stdout:777: writing smithi06431743-18 from 1952125 to 2252204 tid 2 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:778: write (excl) oid 22 current snap is 91 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:764: finishing write tid 3 to smithi06431743-41 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:764: finishing write tid 4 to smithi06431743-41 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:764: oid 41 updating version 0 to 133 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:764: oid 41 version 133 is already newer than 133 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:764: oid 41 version 133 is already newer than 131 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 133 (ObjNum 364 snap 91 seq_num 364) dirty exists 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:764: left oid 41 (ObjNum 364 snap 91 seq_num 364) 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:763: finishing write tid 1 to smithi06431743-40 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:765: finishing write tid 2 to smithi06431743-45 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:766: finishing write tid 2 to smithi06431743-17 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:767: finishing write tid 1 to smithi06431743-10 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:765: finishing write tid 3 to smithi06431743-45 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:768: finishing write tid 2 to smithi06431743-8 2024-03-04T06:11:09.632 INFO:tasks.rados.rados.0.smithi064.stdout:769: finishing write tid 1 to smithi06431743-6 2024-03-04T06:11:09.633 INFO:tasks.rados.rados.0.smithi064.stdout:778: seq_num 373 ranges {669011=569174,2010607=258404} 2024-03-04T06:11:09.638 INFO:tasks.rados.rados.0.smithi064.stdout:778: writing smithi06431743-22 from 669011 to 1238185 tid 1 2024-03-04T06:11:09.644 INFO:tasks.rados.rados.0.smithi064.stdout:778: writing smithi06431743-22 from 2010607 to 2269011 tid 2 2024-03-04T06:11:09.644 INFO:tasks.rados.rados.0.smithi064.stdout:764: done (15 left) 2024-03-04T06:11:09.644 INFO:tasks.rados.rados.0.smithi064.stdout:772: done (14 left) 2024-03-04T06:11:09.644 INFO:tasks.rados.rados.0.smithi064.stdout:779: write (excl) oid 3 current snap is 91 2024-03-04T06:11:09.644 INFO:tasks.rados.rados.0.smithi064.stdout:765: finishing write tid 4 to smithi06431743-45 2024-03-04T06:11:09.644 INFO:tasks.rados.rados.0.smithi064.stdout:765: oid 45 updating version 0 to 174 2024-03-04T06:11:09.645 INFO:tasks.rados.rados.0.smithi064.stdout:765: oid 45 version 174 is already newer than 174 2024-03-04T06:11:09.645 INFO:tasks.rados.rados.0.smithi064.stdout:765: oid 45 version 174 is already newer than 172 2024-03-04T06:11:09.645 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 174 (ObjNum 365 snap 91 seq_num 365) dirty exists 2024-03-04T06:11:09.645 INFO:tasks.rados.rados.0.smithi064.stdout:765: left oid 45 (ObjNum 365 snap 91 seq_num 365) 2024-03-04T06:11:09.645 INFO:tasks.rados.rados.0.smithi064.stdout:779: seq_num 374 ranges {685818=644423,2069089=216729} 2024-03-04T06:11:09.651 INFO:tasks.rados.rados.0.smithi064.stdout:779: writing smithi06431743-3 from 685818 to 1330241 tid 1 2024-03-04T06:11:09.657 INFO:tasks.rados.rados.0.smithi064.stdout:779: writing smithi06431743-3 from 2069089 to 2285818 tid 2 2024-03-04T06:11:09.657 INFO:tasks.rados.rados.0.smithi064.stdout:765: done (14 left) 2024-03-04T06:11:09.657 INFO:tasks.rados.rados.0.smithi064.stdout:780: write oid 2 current snap is 91 2024-03-04T06:11:09.657 INFO:tasks.rados.rados.0.smithi064.stdout:780: seq_num 375 ranges {702625=719672,2127571=175054} 2024-03-04T06:11:09.666 INFO:tasks.rados.rados.0.smithi064.stdout:780: writing smithi06431743-2 from 702625 to 1422297 tid 1 2024-03-04T06:11:09.671 INFO:tasks.rados.rados.0.smithi064.stdout:780: writing smithi06431743-2 from 2127571 to 2302625 tid 2 2024-03-04T06:11:09.671 INFO:tasks.rados.rados.0.smithi064.stdout:781: delete oid 33 current snap is 91 2024-03-04T06:11:09.672 INFO:tasks.rados.rados.0.smithi064.stdout:774: expect (ObjNum 335 snap 83 seq_num 335) 2024-03-04T06:11:09.818 INFO:tasks.rados.rados.0.smithi064.stdout:767: finishing write tid 2 to smithi06431743-10 2024-03-04T06:11:09.818 INFO:tasks.rados.rados.0.smithi064.stdout:768: finishing write tid 3 to smithi06431743-8 2024-03-04T06:11:09.818 INFO:tasks.rados.rados.0.smithi064.stdout:768: finishing write tid 4 to smithi06431743-8 2024-03-04T06:11:09.818 INFO:tasks.rados.rados.0.smithi064.stdout:768: oid 8 updating version 0 to 126 2024-03-04T06:11:09.818 INFO:tasks.rados.rados.0.smithi064.stdout:768: oid 8 updating version 126 to 127 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:768: oid 8 updating version 127 to 128 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 128 (ObjNum 368 snap 91 seq_num 368) dirty exists 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:768: left oid 8 (ObjNum 368 snap 91 seq_num 368) 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:768: done (15 left) 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:774: done (14 left) 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:775: done (13 left) 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:781: done (12 left) 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:782: delete oid 7 current snap is 91 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:766: finishing write tid 3 to smithi06431743-17 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:766: finishing write tid 4 to smithi06431743-17 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:766: oid 17 updating version 0 to 130 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:766: oid 17 updating version 130 to 132 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:766: oid 17 version 132 is already newer than 131 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 132 (ObjNum 366 snap 91 seq_num 366) dirty exists 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:766: left oid 17 (ObjNum 366 snap 91 seq_num 366) 2024-03-04T06:11:09.819 INFO:tasks.rados.rados.0.smithi064.stdout:763: finishing write tid 2 to smithi06431743-40 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:767: finishing write tid 3 to smithi06431743-10 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:767: finishing write tid 4 to smithi06431743-10 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:767: oid 10 updating version 0 to 131 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:767: oid 10 version 131 is already newer than 130 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:767: oid 10 version 131 is already newer than 129 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 131 (ObjNum 367 snap 91 seq_num 367) dirty exists 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:767: left oid 10 (ObjNum 367 snap 91 seq_num 367) 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:769: finishing write tid 2 to smithi06431743-6 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:769: finishing write tid 3 to smithi06431743-6 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:777: finishing write tid 1 to smithi06431743-18 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:769: finishing write tid 4 to smithi06431743-6 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:769: oid 6 updating version 0 to 147 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:769: oid 6 version 147 is already newer than 145 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:769: oid 6 version 147 is already newer than 147 2024-03-04T06:11:09.820 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 147 (ObjNum 369 snap 91 seq_num 369) dirty exists 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:769: left oid 6 (ObjNum 369 snap 91 seq_num 369) 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:778: finishing write tid 1 to smithi06431743-22 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:780: finishing write tid 1 to smithi06431743-2 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:779: finishing write tid 1 to smithi06431743-3 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:763: finishing write tid 3 to smithi06431743-40 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:763: finishing write tid 4 to smithi06431743-40 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:763: oid 40 updating version 0 to 139 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:763: oid 40 version 139 is already newer than 138 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:763: oid 40 version 139 is already newer than 137 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 139 (ObjNum 363 snap 91 seq_num 363) dirty exists 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:763: left oid 40 (ObjNum 363 snap 91 seq_num 363) 2024-03-04T06:11:09.821 INFO:tasks.rados.rados.0.smithi064.stdout:773: expect (ObjNum 356 snap 90 seq_num 356) 2024-03-04T06:11:09.970 INFO:tasks.rados.rados.0.smithi064.stdout:777: finishing write tid 2 to smithi06431743-18 2024-03-04T06:11:09.970 INFO:tasks.rados.rados.0.smithi064.stdout:780: finishing write tid 2 to smithi06431743-2 2024-03-04T06:11:09.970 INFO:tasks.rados.rados.0.smithi064.stdout:778: finishing write tid 2 to smithi06431743-22 2024-03-04T06:11:09.970 INFO:tasks.rados.rados.0.smithi064.stdout:780: finishing write tid 3 to smithi06431743-2 2024-03-04T06:11:09.970 INFO:tasks.rados.rados.0.smithi064.stdout:780: finishing write tid 4 to smithi06431743-2 2024-03-04T06:11:09.970 INFO:tasks.rados.rados.0.smithi064.stdout:780: oid 2 updating version 0 to 131 2024-03-04T06:11:09.970 INFO:tasks.rados.rados.0.smithi064.stdout:780: oid 2 version 131 is already newer than 131 2024-03-04T06:11:09.971 INFO:tasks.rados.rados.0.smithi064.stdout:780: oid 2 version 131 is already newer than 129 2024-03-04T06:11:09.971 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 131 (ObjNum 375 snap 91 seq_num 375) dirty exists 2024-03-04T06:11:09.971 INFO:tasks.rados.rados.0.smithi064.stdout:780: left oid 2 (ObjNum 375 snap 91 seq_num 375) 2024-03-04T06:11:09.971 INFO:tasks.rados.rados.0.smithi064.stdout:777: finishing write tid 3 to smithi06431743-18 2024-03-04T06:11:09.971 INFO:tasks.rados.rados.0.smithi064.stdout:771: finishing write tid 1 to smithi06431743-26 2024-03-04T06:11:09.971 INFO:tasks.rados.rados.0.smithi064.stdout:763: done (12 left) 2024-03-04T06:11:09.971 INFO:tasks.rados.rados.0.smithi064.stdout:766: done (11 left) 2024-03-04T06:11:09.971 INFO:tasks.rados.rados.0.smithi064.stdout:767: done (10 left) 2024-03-04T06:11:09.971 INFO:tasks.rados.rados.0.smithi064.stdout:769: done (9 left) 2024-03-04T06:11:09.971 INFO:tasks.rados.rados.0.smithi064.stdout:770: done (8 left) 2024-03-04T06:11:09.972 INFO:tasks.rados.rados.0.smithi064.stdout:773: done (7 left) 2024-03-04T06:11:09.972 INFO:tasks.rados.rados.0.smithi064.stdout:780: done (6 left) 2024-03-04T06:11:09.972 INFO:tasks.rados.rados.0.smithi064.stdout:782: done (5 left) 2024-03-04T06:11:09.972 INFO:tasks.rados.rados.0.smithi064.stdout:783: delete oid 42 current snap is 91 2024-03-04T06:11:09.972 INFO:tasks.rados.rados.0.smithi064.stdout:777: finishing write tid 4 to smithi06431743-18 2024-03-04T06:11:09.972 INFO:tasks.rados.rados.0.smithi064.stdout:777: oid 18 updating version 0 to 115 2024-03-04T06:11:09.973 INFO:tasks.rados.rados.0.smithi064.stdout:777: oid 18 updating version 115 to 117 2024-03-04T06:11:09.973 INFO:tasks.rados.rados.0.smithi064.stdout:777: oid 18 version 117 is already newer than 116 2024-03-04T06:11:09.973 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 117 (ObjNum 372 snap 91 seq_num 372) dirty exists 2024-03-04T06:11:09.973 INFO:tasks.rados.rados.0.smithi064.stdout:777: left oid 18 (ObjNum 372 snap 91 seq_num 372) 2024-03-04T06:11:09.973 INFO:tasks.rados.rados.0.smithi064.stdout:779: finishing write tid 2 to smithi06431743-3 2024-03-04T06:11:09.973 INFO:tasks.rados.rados.0.smithi064.stdout:778: finishing write tid 3 to smithi06431743-22 2024-03-04T06:11:09.973 INFO:tasks.rados.rados.0.smithi064.stdout:771: finishing write tid 2 to smithi06431743-26 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:778: finishing write tid 4 to smithi06431743-22 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:778: oid 22 updating version 0 to 341 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:778: oid 22 updating version 341 to 342 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:778: oid 22 updating version 342 to 343 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 343 (ObjNum 373 snap 91 seq_num 373) dirty exists 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:778: left oid 22 (ObjNum 373 snap 91 seq_num 373) 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:771: finishing write tid 3 to smithi06431743-26 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:779: finishing write tid 3 to smithi06431743-3 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:779: finishing write tid 4 to smithi06431743-3 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:779: oid 3 updating version 0 to 134 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:779: oid 3 updating version 134 to 136 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:779: oid 3 version 136 is already newer than 135 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 136 (ObjNum 374 snap 91 seq_num 374) dirty exists 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:779: left oid 3 (ObjNum 374 snap 91 seq_num 374) 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:771: finishing write tid 4 to smithi06431743-26 2024-03-04T06:11:09.974 INFO:tasks.rados.rados.0.smithi064.stdout:771: oid 26 updating version 0 to 353 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:771: oid 26 version 353 is already newer than 352 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:771: oid 26 version 353 is already newer than 351 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 353 (ObjNum 370 snap 91 seq_num 370) dirty exists 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:771: left oid 26 (ObjNum 370 snap 91 seq_num 370) 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:776: finishing write tid 1 to smithi06431743-34 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:776: finishing write tid 2 to smithi06431743-34 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:776: finishing write tid 3 to smithi06431743-34 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:776: finishing write tid 4 to smithi06431743-34 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:776: oid 34 updating version 0 to 356 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:776: oid 34 version 356 is already newer than 356 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:776: oid 34 version 356 is already newer than 356 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 356 (ObjNum 371 snap 91 seq_num 371) dirty exists 2024-03-04T06:11:09.975 INFO:tasks.rados.rados.0.smithi064.stdout:776: left oid 34 (ObjNum 371 snap 91 seq_num 371) 2024-03-04T06:11:09.976 INFO:tasks.rados.rados.0.smithi064.stdout:771: done (5 left) 2024-03-04T06:11:09.976 INFO:tasks.rados.rados.0.smithi064.stdout:776: done (4 left) 2024-03-04T06:11:09.976 INFO:tasks.rados.rados.0.smithi064.stdout:777: done (3 left) 2024-03-04T06:11:09.976 INFO:tasks.rados.rados.0.smithi064.stdout:778: done (2 left) 2024-03-04T06:11:09.976 INFO:tasks.rados.rados.0.smithi064.stdout:779: done (1 left) 2024-03-04T06:11:09.976 INFO:tasks.rados.rados.0.smithi064.stdout:783: done (0 left) 2024-03-04T06:11:09.976 INFO:tasks.rados.rados.0.smithi064.stdout:784: read oid 36 snap -1 2024-03-04T06:11:09.976 INFO:tasks.rados.rados.0.smithi064.stdout:784: expect deleted 2024-03-04T06:11:09.977 INFO:tasks.rados.rados.0.smithi064.stdout:785: write oid 24 current snap is 91 2024-03-04T06:11:09.977 INFO:tasks.rados.rados.0.smithi064.stdout:785: seq_num 379 ranges {769853=620668,1845146=524707} 2024-03-04T06:11:09.982 INFO:tasks.rados.rados.0.smithi064.stdout:785: writing smithi06431743-24 from 769853 to 1390521 tid 1 2024-03-04T06:11:09.988 INFO:tasks.rados.rados.0.smithi064.stdout:785: writing smithi06431743-24 from 1845146 to 2369853 tid 2 2024-03-04T06:11:09.988 INFO:tasks.rados.rados.0.smithi064.stdout:786: read oid 28 snap -1 2024-03-04T06:11:09.988 INFO:tasks.rados.rados.0.smithi064.stdout:786: expect (ObjNum 330 snap 83 seq_num 330) 2024-03-04T06:11:09.989 INFO:tasks.rados.rados.0.smithi064.stdout:787: snap_create 2024-03-04T06:11:10.002 INFO:tasks.rados.rados.0.smithi064.stdout:785: finishing write tid 1 to smithi06431743-24 2024-03-04T06:11:10.015 INFO:tasks.rados.rados.0.smithi064.stdout:785: finishing write tid 2 to smithi06431743-24 2024-03-04T06:11:10.022 INFO:tasks.rados.rados.0.smithi064.stdout:786: expect (ObjNum 330 snap 83 seq_num 330) 2024-03-04T06:11:10.211 INFO:tasks.rados.rados.0.smithi064.stdout:785: finishing write tid 3 to smithi06431743-24 2024-03-04T06:11:10.211 INFO:tasks.rados.rados.0.smithi064.stdout:785: finishing write tid 4 to smithi06431743-24 2024-03-04T06:11:10.211 INFO:tasks.rados.rados.0.smithi064.stdout:785: oid 24 updating version 0 to 254 2024-03-04T06:11:10.211 INFO:tasks.rados.rados.0.smithi064.stdout:785: oid 24 updating version 254 to 255 2024-03-04T06:11:10.211 INFO:tasks.rados.rados.0.smithi064.stdout:785: oid 24 updating version 255 to 256 2024-03-04T06:11:10.211 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 256 (ObjNum 379 snap 91 seq_num 379) dirty exists 2024-03-04T06:11:10.211 INFO:tasks.rados.rados.0.smithi064.stdout:785: left oid 24 (ObjNum 379 snap 91 seq_num 379) 2024-03-04T06:11:10.358 INFO:tasks.rados.rados.0.smithi064.stdout:784: done (3 left) 2024-03-04T06:11:10.358 INFO:tasks.rados.rados.0.smithi064.stdout:785: done (2 left) 2024-03-04T06:11:10.358 INFO:tasks.rados.rados.0.smithi064.stdout:786: done (1 left) 2024-03-04T06:11:10.358 INFO:tasks.rados.rados.0.smithi064.stdout:787: done (0 left) 2024-03-04T06:11:10.358 INFO:tasks.rados.rados.0.smithi064.stdout:788: snap_create 2024-03-04T06:11:11.008 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:11.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:11.120+0000 7fb753064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:11.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:11.124+0000 7fb753064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:11.362 INFO:tasks.rados.rados.0.smithi064.stdout:788: done (0 left) 2024-03-04T06:11:11.362 INFO:tasks.rados.rados.0.smithi064.stdout:789: snap_remove snap 91 2024-03-04T06:11:12.365 INFO:tasks.rados.rados.0.smithi064.stdout:789: done (0 left) 2024-03-04T06:11:12.365 INFO:tasks.rados.rados.0.smithi064.stdout:790: write oid 9 current snap is 93 2024-03-04T06:11:12.366 INFO:tasks.rados.rados.0.smithi064.stdout:790: seq_num 380 ranges {786660=695917,1903628=435462} 2024-03-04T06:11:12.373 INFO:tasks.rados.rados.0.smithi064.stdout:790: writing smithi06431743-9 from 786660 to 1482577 tid 1 2024-03-04T06:11:12.378 INFO:tasks.rados.rados.0.smithi064.stdout:790: writing smithi06431743-9 from 1903628 to 2339090 tid 2 2024-03-04T06:11:12.378 INFO:tasks.rados.rados.0.smithi064.stdout:791: read oid 33 snap -1 2024-03-04T06:11:12.378 INFO:tasks.rados.rados.0.smithi064.stdout:791: expect deleted 2024-03-04T06:11:12.379 INFO:tasks.rados.rados.0.smithi064.stdout:792: read oid 31 snap -1 2024-03-04T06:11:12.379 INFO:tasks.rados.rados.0.smithi064.stdout:792: expect deleted 2024-03-04T06:11:12.379 INFO:tasks.rados.rados.0.smithi064.stdout:793: write oid 29 current snap is 93 2024-03-04T06:11:12.379 INFO:tasks.rados.rados.0.smithi064.stdout:793: seq_num 381 ranges {403467=487519,1562110=579120} 2024-03-04T06:11:12.383 INFO:tasks.rados.rados.0.smithi064.stdout:793: writing smithi06431743-29 from 403467 to 890986 tid 1 2024-03-04T06:11:12.390 INFO:tasks.rados.rados.0.smithi064.stdout:793: writing smithi06431743-29 from 1562110 to 2141230 tid 2 2024-03-04T06:11:12.390 INFO:tasks.rados.rados.0.smithi064.stdout:794: delete oid 48 current snap is 93 2024-03-04T06:11:12.432 INFO:tasks.rados.rados.0.smithi064.stdout:792: done (4 left) 2024-03-04T06:11:12.432 INFO:tasks.rados.rados.0.smithi064.stdout:794: done (3 left) 2024-03-04T06:11:12.433 INFO:tasks.rados.rados.0.smithi064.stdout:795: snap_remove snap 92 2024-03-04T06:11:12.535 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:11:12.536 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:11:12.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:11:12.650 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:12.646+0000 7f5d0ec2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:12.653 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:12.652+0000 7f5d0ec2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:12.844 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:11:12.845 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:11:12.856 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-03-04T06:11:12.748804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":90358007,"num_objects":271,"num_object_clones":194,"num_object_copies":813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":43,"num_read":1617,"num_read_kb":1121509,"num_write":1435,"num_write_kb":357777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2889,"ondisk_log_size":2889,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":291796,"kb_used_data":161712,"kb_used_omap":0,"kb_used_meta":128064,"kb_avail":377195564,"statfs":{"total":386547056640,"available":386248257536,"internally_reserved":0,"allocated":165593088,"data_stored":164453954,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":131137536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6991019,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":53,"num_read_kb":31440,"num_write":75,"num_write_kb":18146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000726"},"pg_stats":[{"pgid":"3.f","version":"200'141","reported_seq":646,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.730311+0000","last_change":"2024-03-04T06:11:09.762474+0000","last_active":"2024-03-04T06:11:09.730311+0000","last_peered":"2024-03-04T06:11:09.730311+0000","last_clean":"2024-03-04T06:11:09.730311+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:11:09.730311+0000","last_undegraded":"2024-03-04T06:11:09.730311+0000","last_fullsized":"2024-03-04T06:11:09.730311+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4101040,"num_objects":20,"num_object_clones":13,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":41,"num_read_kb":12964,"num_write":80,"num_write_kb":21385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"200'117","reported_seq":650,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.737574+0000","last_change":"2024-03-04T06:11:09.378707+0000","last_active":"2024-03-04T06:11:09.737574+0000","last_peered":"2024-03-04T06:11:09.737574+0000","last_clean":"2024-03-04T06:11:09.737574+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:11:09.737574+0000","last_undegraded":"2024-03-04T06:11:09.737574+0000","last_fullsized":"2024-03-04T06:11:09.737574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563216,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":44,"num_read_kb":28824,"num_write":54,"num_write_kb":14148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"200'174","reported_seq":676,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.409050+0000","last_change":"2024-03-04T06:11:09.370893+0000","last_active":"2024-03-04T06:11:10.409050+0000","last_peered":"2024-03-04T06:11:10.409050+0000","last_clean":"2024-03-04T06:11:10.409050+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:11:10.409050+0000","last_undegraded":"2024-03-04T06:11:10.409050+0000","last_fullsized":"2024-03-04T06:11:10.409050+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10571702,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":83,"num_read_kb":51881,"num_write":92,"num_write_kb":25387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"200'131","reported_seq":664,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.725704+0000","last_change":"2024-03-04T06:11:09.521451+0000","last_active":"2024-03-04T06:11:09.725704+0000","last_peered":"2024-03-04T06:11:09.725704+0000","last_clean":"2024-03-04T06:11:09.725704+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:11:09.725704+0000","last_undegraded":"2024-03-04T06:11:09.725704+0000","last_fullsized":"2024-03-04T06:11:09.725704+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185075,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":89,"num_read_kb":73886,"num_write":73,"num_write_kb":20550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"194'86","reported_seq":610,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.383377+0000","last_change":"2024-03-04T06:11:09.646673+0000","last_active":"2024-03-04T06:11:10.383377+0000","last_peered":"2024-03-04T06:11:10.383377+0000","last_clean":"2024-03-04T06:11:10.383377+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:11:10.383377+0000","last_undegraded":"2024-03-04T06:11:10.383377+0000","last_fullsized":"2024-03-04T06:11:10.383377+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":29,"num_read_kb":17040,"num_write":29,"num_write_kb":7475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"200'131","reported_seq":656,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.690929+0000","last_change":"2024-03-04T06:11:09.693725+0000","last_active":"2024-03-04T06:11:09.690929+0000","last_peered":"2024-03-04T06:11:09.690929+0000","last_clean":"2024-03-04T06:11:09.690929+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:11:09.690929+0000","last_undegraded":"2024-03-04T06:11:09.690929+0000","last_fullsized":"2024-03-04T06:11:09.690929+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8722965,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":86,"num_read_kb":64476,"num_write":68,"num_write_kb":20659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":558,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.545675+0000","last_change":"2024-03-04T06:11:09.569485+0000","last_active":"2024-03-04T06:11:09.545675+0000","last_peered":"2024-03-04T06:11:09.545675+0000","last_clean":"2024-03-04T06:11:09.545675+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:11:09.545675+0000","last_undegraded":"2024-03-04T06:11:09.545675+0000","last_fullsized":"2024-03-04T06:11:09.545675+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"200'132","reported_seq":649,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.404001+0000","last_change":"2024-03-04T06:11:09.371011+0000","last_active":"2024-03-04T06:11:10.404001+0000","last_peered":"2024-03-04T06:11:10.404001+0000","last_clean":"2024-03-04T06:11:10.404001+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:11:10.404001+0000","last_undegraded":"2024-03-04T06:11:10.404001+0000","last_fullsized":"2024-03-04T06:11:10.404001+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151362,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":77,"num_read_kb":78864,"num_write":62,"num_write_kb":17478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":622,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.369117+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:11:10.369117+0000","last_peered":"2024-03-04T06:11:10.369117+0000","last_clean":"2024-03-04T06:11:10.369117+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:11:10.369117+0000","last_undegraded":"2024-03-04T06:11:10.369117+0000","last_fullsized":"2024-03-04T06:11:10.369117+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":477,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.418330+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:11:10.418330+0000","last_peered":"2024-03-04T06:11:10.418330+0000","last_clean":"2024-03-04T06:11:10.418330+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:11:10.418330+0000","last_undegraded":"2024-03-04T06:11:10.418330+0000","last_fullsized":"2024-03-04T06:11:10.418330+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"200'136","reported_seq":654,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.412866+0000","last_change":"2024-03-04T06:11:09.381364+0000","last_active":"2024-03-04T06:11:10.412866+0000","last_peered":"2024-03-04T06:11:10.412866+0000","last_clean":"2024-03-04T06:11:10.412866+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:11:10.412866+0000","last_undegraded":"2024-03-04T06:11:10.412866+0000","last_fullsized":"2024-03-04T06:11:10.412866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7832194,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":78155,"num_write":58,"num_write_kb":14739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":476,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.522482+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:11:09.522482+0000","last_peered":"2024-03-04T06:11:09.522482+0000","last_clean":"2024-03-04T06:11:09.522482+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:11:09.522482+0000","last_undegraded":"2024-03-04T06:11:09.522482+0000","last_fullsized":"2024-03-04T06:11:09.522482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"200'146","reported_seq":645,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.377267+0000","last_change":"2024-03-04T06:11:09.646875+0000","last_active":"2024-03-04T06:11:10.377267+0000","last_peered":"2024-03-04T06:11:10.377267+0000","last_clean":"2024-03-04T06:11:10.377267+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:11:10.377267+0000","last_undegraded":"2024-03-04T06:11:10.377267+0000","last_fullsized":"2024-03-04T06:11:10.377267+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":68,"num_read_kb":54041,"num_write":49,"num_write_kb":13998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":477,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.407606+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:11:10.407606+0000","last_peered":"2024-03-04T06:11:10.407606+0000","last_clean":"2024-03-04T06:11:10.407606+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:11:10.407606+0000","last_undegraded":"2024-03-04T06:11:10.407606+0000","last_fullsized":"2024-03-04T06:11:10.407606+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"200'356","reported_seq":828,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.384676+0000","last_change":"2024-03-04T06:11:09.809538+0000","last_active":"2024-03-04T06:11:10.384676+0000","last_peered":"2024-03-04T06:11:10.384676+0000","last_clean":"2024-03-04T06:11:10.384676+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:11:10.384676+0000","last_undegraded":"2024-03-04T06:11:10.384676+0000","last_fullsized":"2024-03-04T06:11:10.384676+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14958461,"num_objects":35,"num_object_clones":25,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":226,"num_read_kb":171190,"num_write":175,"num_write_kb":49610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":476,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.514829+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:11:09.514829+0000","last_peered":"2024-03-04T06:11:09.514829+0000","last_clean":"2024-03-04T06:11:09.514829+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:11:09.514829+0000","last_undegraded":"2024-03-04T06:11:09.514829+0000","last_fullsized":"2024-03-04T06:11:09.514829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"200'368","reported_seq":790,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.628570+0000","last_change":"2024-03-04T06:11:09.418997+0000","last_active":"2024-03-04T06:11:09.628570+0000","last_peered":"2024-03-04T06:11:09.628570+0000","last_clean":"2024-03-04T06:11:09.628570+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:11:09.628570+0000","last_undegraded":"2024-03-04T06:11:09.628570+0000","last_fullsized":"2024-03-04T06:11:09.628570+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7445567,"num_objects":34,"num_object_clones":25,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":140,"num_read_kb":94784,"num_write":166,"num_write_kb":44176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"200'147","reported_seq":678,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.417195+0000","last_change":"2024-03-04T06:11:09.377929+0000","last_active":"2024-03-04T06:11:10.417195+0000","last_peered":"2024-03-04T06:11:10.417195+0000","last_clean":"2024-03-04T06:11:10.417195+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:11:10.417195+0000","last_undegraded":"2024-03-04T06:11:10.417195+0000","last_fullsized":"2024-03-04T06:11:10.417195+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5495955,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":127,"num_read_kb":98835,"num_write":68,"num_write_kb":17545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":502,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.387805+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:11:09.387805+0000","last_peered":"2024-03-04T06:11:09.387805+0000","last_clean":"2024-03-04T06:11:09.387805+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:11:09.387805+0000","last_undegraded":"2024-03-04T06:11:09.387805+0000","last_fullsized":"2024-03-04T06:11:09.387805+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"200'128","reported_seq":671,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.622950+0000","last_change":"2024-03-04T06:11:09.393542+0000","last_active":"2024-03-04T06:11:09.622950+0000","last_peered":"2024-03-04T06:11:09.622950+0000","last_clean":"2024-03-04T06:11:09.622950+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:11:09.622950+0000","last_undegraded":"2024-03-04T06:11:09.622950+0000","last_fullsized":"2024-03-04T06:11:09.622950+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2184976,"num_objects":14,"num_object_clones":8,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":80,"num_read_kb":43421,"num_write":59,"num_write_kb":15030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":489,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.383049+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:11:10.383049+0000","last_peered":"2024-03-04T06:11:10.383049+0000","last_clean":"2024-03-04T06:11:10.383049+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:11:10.383049+0000","last_undegraded":"2024-03-04T06:11:10.383049+0000","last_fullsized":"2024-03-04T06:11:10.383049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"200'256","reported_seq":738,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.374067+0000","last_change":"2024-03-04T06:11:09.739718+0000","last_active":"2024-03-04T06:11:10.374067+0000","last_peered":"2024-03-04T06:11:10.374067+0000","last_clean":"2024-03-04T06:11:10.374067+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:11:10.374067+0000","last_undegraded":"2024-03-04T06:11:10.374067+0000","last_fullsized":"2024-03-04T06:11:10.374067+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302724,"num_objects":24,"num_object_clones":18,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":132,"num_read_kb":97581,"num_write":122,"num_write_kb":34364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":502,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.368783+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:11:09.368783+0000","last_peered":"2024-03-04T06:11:09.368783+0000","last_clean":"2024-03-04T06:11:09.368783+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:11:09.368783+0000","last_undegraded":"2024-03-04T06:11:09.368783+0000","last_fullsized":"2024-03-04T06:11:09.368783+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"200'343","reported_seq":802,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-04T06:11:09.769242+0000","last_change":"2024-03-04T06:11:09.417417+0000","last_active":"2024-03-04T06:11:09.769242+0000","last_peered":"2024-03-04T06:11:09.769242+0000","last_clean":"2024-03-04T06:11:09.769242+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:11:09.769242+0000","last_undegraded":"2024-03-04T06:11:09.769242+0000","last_fullsized":"2024-03-04T06:11:09.769242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8252402,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":201,"num_read_kb":155476,"num_write":143,"num_write_kb":39339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":477,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-04T06:11:10.416062+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:11:10.416062+0000","last_peered":"2024-03-04T06:11:10.416062+0000","last_clean":"2024-03-04T06:11:10.416062+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:11:10.416062+0000","last_undegraded":"2024-03-04T06:11:10.416062+0000","last_fullsized":"2024-03-04T06:11:10.416062+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":89767639,"num_objects":269,"num_object_clones":194,"num_object_copies":807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":43,"num_read":1514,"num_read_kb":1121418,"num_write":1298,"num_write_kb":355883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2792,"ondisk_log_size":2792,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81284,"kb_used_data":46140,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94290556,"statfs":{"total":96636764160,"available":96553529344,"internally_reserved":0,"allocated":47247360,"data_stored":46338787,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77300,"kb_used_data":44328,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94294540,"statfs":{"total":96636764160,"available":96557608960,"internally_reserved":0,"allocated":45391872,"data_stored":43969296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70160,"kb_used_data":39176,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94301680,"statfs":{"total":96636764160,"available":96564920320,"internally_reserved":0,"allocated":40116224,"data_stored":40256967,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":63052,"kb_used_data":32068,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94308788,"statfs":{"total":96636764160,"available":96572198912,"internally_reserved":0,"allocated":32837632,"data_stored":33888904,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:11:13.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:13.199 INFO:tasks.rados.rados.0.smithi064.stdout:795: done (3 left) 2024-03-04T06:11:13.199 INFO:tasks.rados.rados.0.smithi064.stdout:796: read oid 42 snap -1 2024-03-04T06:11:13.199 INFO:tasks.rados.rados.0.smithi064.stdout:796: expect deleted 2024-03-04T06:11:13.199 INFO:tasks.rados.rados.0.smithi064.stdout:797: write oid 38 current snap is 93 2024-03-04T06:11:13.199 INFO:tasks.rados.rados.0.smithi064.stdout:790: finishing write tid 1 to smithi06431743-9 2024-03-04T06:11:13.199 INFO:tasks.rados.rados.0.smithi064.stdout:793: finishing write tid 1 to smithi06431743-29 2024-03-04T06:11:13.199 INFO:tasks.rados.rados.0.smithi064.stdout:790: finishing write tid 2 to smithi06431743-9 2024-03-04T06:11:13.199 INFO:tasks.rados.rados.0.smithi064.stdout:793: finishing write tid 2 to smithi06431743-29 2024-03-04T06:11:13.200 INFO:tasks.rados.rados.0.smithi064.stdout:790: finishing write tid 3 to smithi06431743-9 2024-03-04T06:11:13.200 INFO:tasks.rados.rados.0.smithi064.stdout:797: seq_num 383 ranges {437081=638017,1679074=582789} 2024-03-04T06:11:13.207 INFO:tasks.rados.rados.0.smithi064.stdout:797: writing smithi06431743-38 from 437081 to 1075098 tid 1 2024-03-04T06:11:13.215 INFO:tasks.rados.rados.0.smithi064.stdout:797: writing smithi06431743-38 from 1679074 to 2261863 tid 2 2024-03-04T06:11:13.215 INFO:tasks.rados.rados.0.smithi064.stdout:798: write (excl) oid 12 current snap is 93 2024-03-04T06:11:13.215 INFO:tasks.rados.rados.0.smithi064.stdout:790: finishing write tid 4 to smithi06431743-9 2024-03-04T06:11:13.215 INFO:tasks.rados.rados.0.smithi064.stdout:790: oid 9 updating version 0 to 347 2024-03-04T06:11:13.215 INFO:tasks.rados.rados.0.smithi064.stdout:790: oid 9 version 347 is already newer than 347 2024-03-04T06:11:13.215 INFO:tasks.rados.rados.0.smithi064.stdout:790: oid 9 updating version 347 to 348 2024-03-04T06:11:13.215 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 348 (ObjNum 380 snap 93 seq_num 380) dirty exists 2024-03-04T06:11:13.216 INFO:tasks.rados.rados.0.smithi064.stdout:790: left oid 9 (ObjNum 380 snap 93 seq_num 380) 2024-03-04T06:11:13.216 INFO:tasks.rados.rados.0.smithi064.stdout:793: finishing write tid 3 to smithi06431743-29 2024-03-04T06:11:13.216 INFO:tasks.rados.rados.0.smithi064.stdout:793: finishing write tid 4 to smithi06431743-29 2024-03-04T06:11:13.216 INFO:tasks.rados.rados.0.smithi064.stdout:793: oid 29 updating version 0 to 370 2024-03-04T06:11:13.216 INFO:tasks.rados.rados.0.smithi064.stdout:793: oid 29 updating version 370 to 371 2024-03-04T06:11:13.216 INFO:tasks.rados.rados.0.smithi064.stdout:793: oid 29 updating version 371 to 372 2024-03-04T06:11:13.216 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 372 (ObjNum 381 snap 93 seq_num 381) dirty exists 2024-03-04T06:11:13.216 INFO:tasks.rados.rados.0.smithi064.stdout:793: left oid 29 (ObjNum 381 snap 93 seq_num 381) 2024-03-04T06:11:13.216 INFO:tasks.rados.rados.0.smithi064.stdout:798: seq_num 384 ranges {453888=713266,1737556=442800} 2024-03-04T06:11:13.223 INFO:tasks.rados.rados.0.smithi064.stdout:798: writing smithi06431743-12 from 453888 to 1167154 tid 1 2024-03-04T06:11:13.230 INFO:tasks.rados.rados.0.smithi064.stdout:798: writing smithi06431743-12 from 1737556 to 2180356 tid 2 2024-03-04T06:11:13.230 INFO:tasks.rados.rados.0.smithi064.stdout:790: done (5 left) 2024-03-04T06:11:13.230 INFO:tasks.rados.rados.0.smithi064.stdout:791: done (4 left) 2024-03-04T06:11:13.230 INFO:tasks.rados.rados.0.smithi064.stdout:793: done (3 left) 2024-03-04T06:11:13.230 INFO:tasks.rados.rados.0.smithi064.stdout:799: delete oid 44 current snap is 93 2024-03-04T06:11:13.282 INFO:tasks.rados.rados.0.smithi064.stdout:798: finishing write tid 1 to smithi06431743-12 2024-03-04T06:11:13.285 INFO:tasks.rados.rados.0.smithi064.stdout:797: finishing write tid 1 to smithi06431743-38 2024-03-04T06:11:13.297 INFO:tasks.rados.rados.0.smithi064.stdout:798: finishing write tid 2 to smithi06431743-12 2024-03-04T06:11:13.303 INFO:tasks.rados.rados.0.smithi064.stdout:797: finishing write tid 2 to smithi06431743-38 2024-03-04T06:11:13.306 INFO:tasks.rados.rados.0.smithi064.stdout:798: finishing write tid 3 to smithi06431743-12 2024-03-04T06:11:13.306 INFO:tasks.rados.rados.0.smithi064.stdout:798: finishing write tid 4 to smithi06431743-12 2024-03-04T06:11:13.307 INFO:tasks.rados.rados.0.smithi064.stdout:798: oid 12 updating version 0 to 134 2024-03-04T06:11:13.307 INFO:tasks.rados.rados.0.smithi064.stdout:798: oid 12 version 134 is already newer than 133 2024-03-04T06:11:13.307 INFO:tasks.rados.rados.0.smithi064.stdout:798: oid 12 version 134 is already newer than 132 2024-03-04T06:11:13.307 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 134 (ObjNum 384 snap 93 seq_num 384) dirty exists 2024-03-04T06:11:13.307 INFO:tasks.rados.rados.0.smithi064.stdout:798: left oid 12 (ObjNum 384 snap 93 seq_num 384) 2024-03-04T06:11:13.308 INFO:tasks.rados.rados.0.smithi064.stdout:797: finishing write tid 3 to smithi06431743-38 2024-03-04T06:11:13.311 INFO:tasks.rados.rados.0.smithi064.stdout:797: finishing write tid 4 to smithi06431743-38 2024-03-04T06:11:13.311 INFO:tasks.rados.rados.0.smithi064.stdout:797: oid 38 updating version 0 to 132 2024-03-04T06:11:13.311 INFO:tasks.rados.rados.0.smithi064.stdout:797: oid 38 updating version 132 to 134 2024-03-04T06:11:13.311 INFO:tasks.rados.rados.0.smithi064.stdout:797: oid 38 version 134 is already newer than 133 2024-03-04T06:11:13.311 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 134 (ObjNum 383 snap 93 seq_num 383) dirty exists 2024-03-04T06:11:13.311 INFO:tasks.rados.rados.0.smithi064.stdout:797: left oid 38 (ObjNum 383 snap 93 seq_num 383) 2024-03-04T06:11:13.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:13.313+0000 7f187bc17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:13.318 INFO:tasks.rados.rados.0.smithi064.stdout:796: done (3 left) 2024-03-04T06:11:13.318 INFO:tasks.rados.rados.0.smithi064.stdout:797: done (2 left) 2024-03-04T06:11:13.318 INFO:tasks.rados.rados.0.smithi064.stdout:798: done (1 left) 2024-03-04T06:11:13.318 INFO:tasks.rados.rados.0.smithi064.stdout:799: done (0 left) 2024-03-04T06:11:13.318 INFO:tasks.rados.rados.0.smithi064.stdout:800: read oid 37 snap -1 2024-03-04T06:11:13.318 INFO:tasks.rados.rados.0.smithi064.stdout:800: expect (ObjNum 316 snap 81 seq_num 316) 2024-03-04T06:11:13.318 INFO:tasks.rados.rados.0.smithi064.stdout:801: read oid 18 snap -1 2024-03-04T06:11:13.318 INFO:tasks.rados.rados.0.smithi064.stdout:801: expect (ObjNum 372 snap 91 seq_num 372) 2024-03-04T06:11:13.318 INFO:tasks.rados.rados.0.smithi064.stdout:802: snap_create 2024-03-04T06:11:13.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:13.318+0000 7f187bc17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:13.381 INFO:tasks.rados.rados.0.smithi064.stdout:800: expect (ObjNum 316 snap 81 seq_num 316) 2024-03-04T06:11:13.508 INFO:tasks.rados.rados.0.smithi064.stdout:801: expect (ObjNum 372 snap 91 seq_num 372) 2024-03-04T06:11:14.199 INFO:tasks.rados.rados.0.smithi064.stdout:800: done (2 left) 2024-03-04T06:11:14.199 INFO:tasks.rados.rados.0.smithi064.stdout:801: done (1 left) 2024-03-04T06:11:14.199 INFO:tasks.rados.rados.0.smithi064.stdout:802: done (0 left) 2024-03-04T06:11:14.199 INFO:tasks.rados.rados.0.smithi064.stdout:803: snap_remove snap 93 2024-03-04T06:11:15.203 INFO:tasks.rados.rados.0.smithi064.stdout:803: done (0 left) 2024-03-04T06:11:15.203 INFO:tasks.rados.rados.0.smithi064.stdout:804: read oid 29 snap -1 2024-03-04T06:11:15.203 INFO:tasks.rados.rados.0.smithi064.stdout:804: expect (ObjNum 381 snap 93 seq_num 381) 2024-03-04T06:11:15.203 INFO:tasks.rados.rados.0.smithi064.stdout:805: read oid 3 snap -1 2024-03-04T06:11:15.203 INFO:tasks.rados.rados.0.smithi064.stdout:805: expect (ObjNum 374 snap 91 seq_num 374) 2024-03-04T06:11:15.203 INFO:tasks.rados.rados.0.smithi064.stdout:806: snap_create 2024-03-04T06:11:15.242 INFO:tasks.rados.rados.0.smithi064.stdout:805: expect (ObjNum 374 snap 91 seq_num 374) 2024-03-04T06:11:15.331 INFO:tasks.rados.rados.0.smithi064.stdout:804: expect (ObjNum 381 snap 93 seq_num 381) 2024-03-04T06:11:15.418 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:15.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:15.521+0000 7f32a366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:15.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:15.526+0000 7f32a366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:16.203 INFO:tasks.rados.rados.0.smithi064.stdout:804: done (2 left) 2024-03-04T06:11:16.203 INFO:tasks.rados.rados.0.smithi064.stdout:805: done (1 left) 2024-03-04T06:11:16.203 INFO:tasks.rados.rados.0.smithi064.stdout:806: done (0 left) 2024-03-04T06:11:16.204 INFO:tasks.rados.rados.0.smithi064.stdout:807: delete oid 11 current snap is 95 2024-03-04T06:11:16.249 INFO:tasks.rados.rados.0.smithi064.stdout:807: done (0 left) 2024-03-04T06:11:16.249 INFO:tasks.rados.rados.0.smithi064.stdout:808: snap_remove snap 94 2024-03-04T06:11:17.210 INFO:tasks.rados.rados.0.smithi064.stdout:808: done (0 left) 2024-03-04T06:11:17.210 INFO:tasks.rados.rados.0.smithi064.stdout:809: read oid 44 snap -1 2024-03-04T06:11:17.210 INFO:tasks.rados.rados.0.smithi064.stdout:809: expect deleted 2024-03-04T06:11:17.210 INFO:tasks.rados.rados.0.smithi064.stdout:810: write oid 16 current snap is 95 2024-03-04T06:11:17.210 INFO:tasks.rados.rados.0.smithi064.stdout:810: seq_num 387 ranges {504309=539013,1796649=590127} 2024-03-04T06:11:17.215 INFO:tasks.rados.rados.0.smithi064.stdout:810: writing smithi06431743-16 from 504309 to 1043322 tid 1 2024-03-04T06:11:17.223 INFO:tasks.rados.rados.0.smithi064.stdout:810: writing smithi06431743-16 from 1796649 to 2386776 tid 2 2024-03-04T06:11:17.224 INFO:tasks.rados.rados.0.smithi064.stdout:811: read oid 24 snap -1 2024-03-04T06:11:17.224 INFO:tasks.rados.rados.0.smithi064.stdout:811: expect (ObjNum 379 snap 91 seq_num 379) 2024-03-04T06:11:17.224 INFO:tasks.rados.rados.0.smithi064.stdout:812: write oid 19 current snap is 95 2024-03-04T06:11:17.224 INFO:tasks.rados.rados.0.smithi064.stdout:812: seq_num 388 ranges {521116=730615,1971484=549632} 2024-03-04T06:11:17.230 INFO:tasks.rados.rados.0.smithi064.stdout:812: writing smithi06431743-19 from 521116 to 1251731 tid 1 2024-03-04T06:11:17.240 INFO:tasks.rados.rados.0.smithi064.stdout:812: writing smithi06431743-19 from 1971484 to 2521116 tid 2 2024-03-04T06:11:17.240 INFO:tasks.rados.rados.0.smithi064.stdout:813: read oid 42 snap -1 2024-03-04T06:11:17.240 INFO:tasks.rados.rados.0.smithi064.stdout:813: expect deleted 2024-03-04T06:11:17.240 INFO:tasks.rados.rados.0.smithi064.stdout:814: write (excl) oid 11 current snap is 95 2024-03-04T06:11:17.240 INFO:tasks.rados.rados.0.smithi064.stdout:814: seq_num 389 ranges {537923=405864,1513613=593796} 2024-03-04T06:11:17.246 INFO:tasks.rados.rados.0.smithi064.stdout:814: writing smithi06431743-11 from 537923 to 943787 tid 1 2024-03-04T06:11:17.253 INFO:tasks.rados.rados.0.smithi064.stdout:814: writing smithi06431743-11 from 1513613 to 2107409 tid 2 2024-03-04T06:11:17.253 INFO:tasks.rados.rados.0.smithi064.stdout:815: delete oid 13 current snap is 95 2024-03-04T06:11:17.293 INFO:tasks.rados.rados.0.smithi064.stdout:811: expect (ObjNum 379 snap 91 seq_num 379) 2024-03-04T06:11:17.382 INFO:tasks.rados.rados.0.smithi064.stdout:810: finishing write tid 1 to smithi06431743-16 2024-03-04T06:11:17.382 INFO:tasks.rados.rados.0.smithi064.stdout:814: finishing write tid 1 to smithi06431743-11 2024-03-04T06:11:17.382 INFO:tasks.rados.rados.0.smithi064.stdout:812: finishing write tid 1 to smithi06431743-19 2024-03-04T06:11:17.382 INFO:tasks.rados.rados.0.smithi064.stdout:810: finishing write tid 2 to smithi06431743-16 2024-03-04T06:11:17.382 INFO:tasks.rados.rados.0.smithi064.stdout:814: finishing write tid 2 to smithi06431743-11 2024-03-04T06:11:17.382 INFO:tasks.rados.rados.0.smithi064.stdout:812: finishing write tid 2 to smithi06431743-19 2024-03-04T06:11:17.382 INFO:tasks.rados.rados.0.smithi064.stdout:814: finishing write tid 3 to smithi06431743-11 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:812: finishing write tid 3 to smithi06431743-19 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:814: finishing write tid 4 to smithi06431743-11 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:814: oid 11 updating version 0 to 180 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:814: oid 11 version 180 is already newer than 179 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:814: oid 11 updating version 180 to 181 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 181 (ObjNum 389 snap 95 seq_num 389) dirty exists 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:814: left oid 11 (ObjNum 389 snap 95 seq_num 389) 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:812: finishing write tid 4 to smithi06431743-19 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:812: oid 19 updating version 0 to 375 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:812: oid 19 updating version 375 to 377 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:812: oid 19 version 377 is already newer than 377 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 377 (ObjNum 388 snap 95 seq_num 388) dirty exists 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:812: left oid 19 (ObjNum 388 snap 95 seq_num 388) 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:810: finishing write tid 3 to smithi06431743-16 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:809: done (6 left) 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:811: done (5 left) 2024-03-04T06:11:17.383 INFO:tasks.rados.rados.0.smithi064.stdout:812: done (4 left) 2024-03-04T06:11:17.384 INFO:tasks.rados.rados.0.smithi064.stdout:813: done (3 left) 2024-03-04T06:11:17.384 INFO:tasks.rados.rados.0.smithi064.stdout:814: done (2 left) 2024-03-04T06:11:17.384 INFO:tasks.rados.rados.0.smithi064.stdout:815: done (1 left) 2024-03-04T06:11:17.384 INFO:tasks.rados.rados.0.smithi064.stdout:816: delete oid 2 current snap is 95 2024-03-04T06:11:17.384 INFO:tasks.rados.rados.0.smithi064.stdout:810: finishing write tid 4 to smithi06431743-16 2024-03-04T06:11:17.384 INFO:tasks.rados.rados.0.smithi064.stdout:810: oid 16 updating version 0 to 131 2024-03-04T06:11:17.384 INFO:tasks.rados.rados.0.smithi064.stdout:810: oid 16 version 131 is already newer than 130 2024-03-04T06:11:17.384 INFO:tasks.rados.rados.0.smithi064.stdout:810: oid 16 version 131 is already newer than 130 2024-03-04T06:11:17.384 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 131 (ObjNum 387 snap 95 seq_num 387) dirty exists 2024-03-04T06:11:17.384 INFO:tasks.rados.rados.0.smithi064.stdout:810: left oid 16 (ObjNum 387 snap 95 seq_num 387) 2024-03-04T06:11:17.389 INFO:tasks.rados.rados.0.smithi064.stdout:810: done (1 left) 2024-03-04T06:11:17.390 INFO:tasks.rados.rados.0.smithi064.stdout:816: done (0 left) 2024-03-04T06:11:17.390 INFO:tasks.rados.rados.0.smithi064.stdout:817: read oid 45 snap -1 2024-03-04T06:11:17.390 INFO:tasks.rados.rados.0.smithi064.stdout:817: expect (ObjNum 365 snap 91 seq_num 365) 2024-03-04T06:11:17.390 INFO:tasks.rados.rados.0.smithi064.stdout:818: delete oid 25 current snap is 95 2024-03-04T06:11:17.402 INFO:tasks.rados.rados.0.smithi064.stdout:817: expect (ObjNum 365 snap 91 seq_num 365) 2024-03-04T06:11:17.501 INFO:tasks.rados.rados.0.smithi064.stdout:817: done (1 left) 2024-03-04T06:11:17.502 INFO:tasks.rados.rados.0.smithi064.stdout:818: done (0 left) 2024-03-04T06:11:17.502 INFO:tasks.rados.rados.0.smithi064.stdout:819: read oid 26 snap -1 2024-03-04T06:11:17.502 INFO:tasks.rados.rados.0.smithi064.stdout:819: expect (ObjNum 370 snap 91 seq_num 370) 2024-03-04T06:11:17.502 INFO:tasks.rados.rados.0.smithi064.stdout:820: write (excl) oid 16 current snap is 95 2024-03-04T06:11:17.502 INFO:tasks.rados.rados.0.smithi064.stdout:820: seq_num 393 ranges {605151=706860,2031188=573963} 2024-03-04T06:11:17.508 INFO:tasks.rados.rados.0.smithi064.stdout:820: writing smithi06431743-16 from 605151 to 1312011 tid 1 2024-03-04T06:11:17.515 INFO:tasks.rados.rados.0.smithi064.stdout:820: writing smithi06431743-16 from 2031188 to 2605151 tid 2 2024-03-04T06:11:17.515 INFO:tasks.rados.rados.0.smithi064.stdout:821: snap_create 2024-03-04T06:11:17.516 INFO:tasks.rados.rados.0.smithi064.stdout:819: expect (ObjNum 370 snap 91 seq_num 370) 2024-03-04T06:11:17.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:17.645 INFO:tasks.rados.rados.0.smithi064.stdout:820: finishing write tid 1 to smithi06431743-16 2024-03-04T06:11:17.645 INFO:tasks.rados.rados.0.smithi064.stdout:820: finishing write tid 2 to smithi06431743-16 2024-03-04T06:11:17.645 INFO:tasks.rados.rados.0.smithi064.stdout:820: finishing write tid 3 to smithi06431743-16 2024-03-04T06:11:17.645 INFO:tasks.rados.rados.0.smithi064.stdout:820: finishing write tid 4 to smithi06431743-16 2024-03-04T06:11:17.645 INFO:tasks.rados.rados.0.smithi064.stdout:820: oid 16 updating version 0 to 134 2024-03-04T06:11:17.645 INFO:tasks.rados.rados.0.smithi064.stdout:820: oid 16 version 134 is already newer than 133 2024-03-04T06:11:17.645 INFO:tasks.rados.rados.0.smithi064.stdout:820: oid 16 version 134 is already newer than 132 2024-03-04T06:11:17.646 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 134 (ObjNum 393 snap 95 seq_num 393) dirty exists 2024-03-04T06:11:17.646 INFO:tasks.rados.rados.0.smithi064.stdout:820: left oid 16 (ObjNum 393 snap 95 seq_num 393) 2024-03-04T06:11:17.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:17.698+0000 7f5394e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:17.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:17.702+0000 7f5394e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:17.859 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:11:17.859 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:11:17.859 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:11:17.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:11:17.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:17.962+0000 7fd63ed1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:17.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:17.970+0000 7fd63ed1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:17.983 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:17.985 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:17.986 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:17.986 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:11:18.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:18.089+0000 7fa80ab9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:18.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:18.097+0000 7fa80ab9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:18.109 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:18.110 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:18.110 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:18.110 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:18.110 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:18.110 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:18.110 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:18.110 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:18.110 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:18.110 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:18.110 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:18.110 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:18.113 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:18.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:11:18.209 INFO:tasks.rados.rados.0.smithi064.stdout:819: done (2 left) 2024-03-04T06:11:18.210 INFO:tasks.rados.rados.0.smithi064.stdout:820: done (1 left) 2024-03-04T06:11:18.210 INFO:tasks.rados.rados.0.smithi064.stdout:821: done (0 left) 2024-03-04T06:11:18.210 INFO:tasks.rados.rados.0.smithi064.stdout:822: delete oid 23 current snap is 96 2024-03-04T06:11:18.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:18.221+0000 7f2d6091d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:18.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:18.232+0000 7f2d6091d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:18.246 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:18.246 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:18.246 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:18.246 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:18.246 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:18.246 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:18.246 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:18.246 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:18.246 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:18.246 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:18.246 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:18.247 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:18.251 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:18.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:11:18.252 INFO:tasks.rados.rados.0.smithi064.stdout:822: done (0 left) 2024-03-04T06:11:18.252 INFO:tasks.rados.rados.0.smithi064.stdout:823: delete oid 27 current snap is 96 2024-03-04T06:11:18.283 INFO:tasks.rados.rados.0.smithi064.stdout:823: done (0 left) 2024-03-04T06:11:18.283 INFO:tasks.rados.rados.0.smithi064.stdout:824: delete oid 18 current snap is 96 2024-03-04T06:11:18.299 INFO:tasks.rados.rados.0.smithi064.stdout:824: done (0 left) 2024-03-04T06:11:18.299 INFO:tasks.rados.rados.0.smithi064.stdout:825: delete oid 39 current snap is 96 2024-03-04T06:11:18.318 INFO:tasks.rados.rados.0.smithi064.stdout:825: done (0 left) 2024-03-04T06:11:18.318 INFO:tasks.rados.rados.0.smithi064.stdout:826: snap_remove snap 95 2024-03-04T06:11:18.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:18.368+0000 7f3e7d71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:18.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:18.376+0000 7f3e7d71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:18.399 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:18.402 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:19.215 INFO:tasks.rados.rados.0.smithi064.stdout:826: done (0 left) 2024-03-04T06:11:19.215 INFO:tasks.rados.rados.0.smithi064.stdout:827: read oid 6 snap -1 2024-03-04T06:11:19.215 INFO:tasks.rados.rados.0.smithi064.stdout:827: expect (ObjNum 369 snap 91 seq_num 369) 2024-03-04T06:11:19.215 INFO:tasks.rados.rados.0.smithi064.stdout:828: delete oid 47 current snap is 96 2024-03-04T06:11:19.250 INFO:tasks.rados.rados.0.smithi064.stdout:828: done (1 left) 2024-03-04T06:11:19.250 INFO:tasks.rados.rados.0.smithi064.stdout:829: write oid 25 current snap is 96 2024-03-04T06:11:19.250 INFO:tasks.rados.rados.0.smithi064.stdout:829: seq_num 399 ranges {705993=474707,1582080=612141} 2024-03-04T06:11:19.256 INFO:tasks.rados.rados.0.smithi064.stdout:829: writing smithi06431743-25 from 705993 to 1180700 tid 1 2024-03-04T06:11:19.262 INFO:tasks.rados.rados.0.smithi064.stdout:829: writing smithi06431743-25 from 1582080 to 2194221 tid 2 2024-03-04T06:11:19.262 INFO:tasks.rados.rados.0.smithi064.stdout:830: snap_create 2024-03-04T06:11:19.265 INFO:tasks.rados.rados.0.smithi064.stdout:827: expect (ObjNum 369 snap 91 seq_num 369) 2024-03-04T06:11:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:829: finishing write tid 1 to smithi06431743-25 2024-03-04T06:11:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:829: finishing write tid 2 to smithi06431743-25 2024-03-04T06:11:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:829: finishing write tid 3 to smithi06431743-25 2024-03-04T06:11:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:829: finishing write tid 4 to smithi06431743-25 2024-03-04T06:11:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:829: oid 25 updating version 0 to 187 2024-03-04T06:11:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:829: oid 25 updating version 187 to 188 2024-03-04T06:11:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:829: oid 25 updating version 188 to 189 2024-03-04T06:11:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 189 (ObjNum 399 snap 96 seq_num 399) dirty exists 2024-03-04T06:11:19.445 INFO:tasks.rados.rados.0.smithi064.stdout:829: left oid 25 (ObjNum 399 snap 96 seq_num 399) 2024-03-04T06:11:19.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:19.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:19.876+0000 7f7515b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:19.880 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:19.880+0000 7f7515b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:20.218 INFO:tasks.rados.rados.0.smithi064.stdout:827: done (2 left) 2024-03-04T06:11:20.218 INFO:tasks.rados.rados.0.smithi064.stdout:829: done (1 left) 2024-03-04T06:11:20.218 INFO:tasks.rados.rados.0.smithi064.stdout:830: done (0 left) 2024-03-04T06:11:20.218 INFO:tasks.rados.rados.0.smithi064.stdout:831: read oid 3 snap 96 2024-03-04T06:11:20.218 INFO:tasks.rados.rados.0.smithi064.stdout:831: expect (ObjNum 374 snap 91 seq_num 374) 2024-03-04T06:11:20.218 INFO:tasks.rados.rados.0.smithi064.stdout:832: write oid 49 current snap is 97 2024-03-04T06:11:20.218 INFO:tasks.rados.rados.0.smithi064.stdout:832: seq_num 400 ranges {722800=549956,2040562=682238} 2024-03-04T06:11:20.224 INFO:tasks.rados.rados.0.smithi064.stdout:832: writing smithi06431743-49 from 722800 to 1272756 tid 1 2024-03-04T06:11:20.232 INFO:tasks.rados.rados.0.smithi064.stdout:832: writing smithi06431743-49 from 2040562 to 2722800 tid 2 2024-03-04T06:11:20.232 INFO:tasks.rados.rados.0.smithi064.stdout:833: read oid 17 snap -1 2024-03-04T06:11:20.233 INFO:tasks.rados.rados.0.smithi064.stdout:833: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-04T06:11:20.233 INFO:tasks.rados.rados.0.smithi064.stdout:834: snap_create 2024-03-04T06:11:20.261 INFO:tasks.rados.rados.0.smithi064.stdout:831: expect (ObjNum 374 snap 91 seq_num 374) 2024-03-04T06:11:20.362 INFO:tasks.rados.rados.0.smithi064.stdout:832: finishing write tid 1 to smithi06431743-49 2024-03-04T06:11:20.363 INFO:tasks.rados.rados.0.smithi064.stdout:832: finishing write tid 2 to smithi06431743-49 2024-03-04T06:11:20.363 INFO:tasks.rados.rados.0.smithi064.stdout:832: finishing write tid 3 to smithi06431743-49 2024-03-04T06:11:20.363 INFO:tasks.rados.rados.0.smithi064.stdout:832: finishing write tid 4 to smithi06431743-49 2024-03-04T06:11:20.363 INFO:tasks.rados.rados.0.smithi064.stdout:832: oid 49 updating version 0 to 134 2024-03-04T06:11:20.363 INFO:tasks.rados.rados.0.smithi064.stdout:832: oid 49 version 134 is already newer than 134 2024-03-04T06:11:20.363 INFO:tasks.rados.rados.0.smithi064.stdout:832: oid 49 updating version 134 to 135 2024-03-04T06:11:20.363 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 135 (ObjNum 400 snap 97 seq_num 400) dirty exists 2024-03-04T06:11:20.363 INFO:tasks.rados.rados.0.smithi064.stdout:832: left oid 49 (ObjNum 400 snap 97 seq_num 400) 2024-03-04T06:11:20.363 INFO:tasks.rados.rados.0.smithi064.stdout:833: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-04T06:11:21.222 INFO:tasks.rados.rados.0.smithi064.stdout:831: done (3 left) 2024-03-04T06:11:21.222 INFO:tasks.rados.rados.0.smithi064.stdout:832: done (2 left) 2024-03-04T06:11:21.223 INFO:tasks.rados.rados.0.smithi064.stdout:833: done (1 left) 2024-03-04T06:11:21.223 INFO:tasks.rados.rados.0.smithi064.stdout:834: done (0 left) 2024-03-04T06:11:21.223 INFO:tasks.rados.rados.0.smithi064.stdout:835: read oid 1 snap -1 2024-03-04T06:11:21.223 INFO:tasks.rados.rados.0.smithi064.stdout:835: expect (ObjNum 341 snap 84 seq_num 341) 2024-03-04T06:11:21.223 INFO:tasks.rados.rados.0.smithi064.stdout:836: read oid 43 snap -1 2024-03-04T06:11:21.223 INFO:tasks.rados.rados.0.smithi064.stdout:836: expect (ObjNum 357 snap 90 seq_num 357) 2024-03-04T06:11:21.223 INFO:tasks.rados.rados.0.smithi064.stdout:837: read oid 41 snap -1 2024-03-04T06:11:21.223 INFO:tasks.rados.rados.0.smithi064.stdout:837: expect (ObjNum 364 snap 91 seq_num 364) 2024-03-04T06:11:21.223 INFO:tasks.rados.rados.0.smithi064.stdout:838: write (excl) oid 25 current snap is 98 2024-03-04T06:11:21.223 INFO:tasks.rados.rados.0.smithi064.stdout:838: seq_num 401 ranges {739607=625205,1982691=499457} 2024-03-04T06:11:21.230 INFO:tasks.rados.rados.0.smithi064.stdout:838: writing smithi06431743-25 from 739607 to 1364812 tid 1 2024-03-04T06:11:21.236 INFO:tasks.rados.rados.0.smithi064.stdout:838: writing smithi06431743-25 from 1982691 to 2482148 tid 2 2024-03-04T06:11:21.236 INFO:tasks.rados.rados.0.smithi064.stdout:839: delete oid 13 current snap is 98 2024-03-04T06:11:21.274 INFO:tasks.rados.rados.0.smithi064.stdout:837: expect (ObjNum 364 snap 91 seq_num 364) 2024-03-04T06:11:21.432 INFO:tasks.rados.rados.0.smithi064.stdout:835: expect (ObjNum 341 snap 84 seq_num 341) 2024-03-04T06:11:21.569 INFO:tasks.rados.rados.0.smithi064.stdout:836: expect (ObjNum 357 snap 90 seq_num 357) 2024-03-04T06:11:21.688 INFO:tasks.rados.rados.0.smithi064.stdout:838: finishing write tid 1 to smithi06431743-25 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:838: finishing write tid 2 to smithi06431743-25 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:838: finishing write tid 3 to smithi06431743-25 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:838: finishing write tid 4 to smithi06431743-25 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:838: oid 25 updating version 0 to 191 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:838: oid 25 updating version 191 to 192 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:838: oid 25 updating version 192 to 193 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 193 (ObjNum 401 snap 98 seq_num 401) dirty exists 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:838: left oid 25 (ObjNum 401 snap 98 seq_num 401) 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:835: done (4 left) 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:836: done (3 left) 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:837: done (2 left) 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:838: done (1 left) 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:839: done (0 left) 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:840: read oid 22 snap 96 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:840: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-04T06:11:21.689 INFO:tasks.rados.rados.0.smithi064.stdout:841: read oid 6 snap -1 2024-03-04T06:11:21.690 INFO:tasks.rados.rados.0.smithi064.stdout:841: expect (ObjNum 369 snap 91 seq_num 369) 2024-03-04T06:11:21.690 INFO:tasks.rados.rados.0.smithi064.stdout:842: read oid 24 snap 97 2024-03-04T06:11:21.690 INFO:tasks.rados.rados.0.smithi064.stdout:842: expect (ObjNum 379 snap 91 seq_num 379) 2024-03-04T06:11:21.690 INFO:tasks.rados.rados.0.smithi064.stdout:843: delete oid 28 current snap is 98 2024-03-04T06:11:21.699 INFO:tasks.rados.rados.0.smithi064.stdout:841: expect (ObjNum 369 snap 91 seq_num 369) 2024-03-04T06:11:21.879 INFO:tasks.rados.rados.0.smithi064.stdout:841: done (3 left) 2024-03-04T06:11:21.879 INFO:tasks.rados.rados.0.smithi064.stdout:843: done (2 left) 2024-03-04T06:11:21.879 INFO:tasks.rados.rados.0.smithi064.stdout:844: write (excl) oid 8 current snap is 98 2024-03-04T06:11:21.879 INFO:tasks.rados.rados.0.smithi064.stdout:844: seq_num 404 ranges {790028=567305,1874490=763137} 2024-03-04T06:11:21.886 INFO:tasks.rados.rados.0.smithi064.stdout:844: writing smithi06431743-8 from 790028 to 1357333 tid 1 2024-03-04T06:11:21.893 INFO:tasks.rados.rados.0.smithi064.stdout:844: writing smithi06431743-8 from 1874490 to 2637627 tid 2 2024-03-04T06:11:21.893 INFO:tasks.rados.rados.0.smithi064.stdout:845: snap_create 2024-03-04T06:11:21.893 INFO:tasks.rados.rados.0.smithi064.stdout:840: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-04T06:11:21.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:22.025 INFO:tasks.rados.rados.0.smithi064.stdout:842: expect (ObjNum 379 snap 91 seq_num 379) 2024-03-04T06:11:22.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:22.059+0000 7f4d75b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:22.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:22.063+0000 7f4d75b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:22.122 INFO:tasks.rados.rados.0.smithi064.stdout:844: finishing write tid 1 to smithi06431743-8 2024-03-04T06:11:22.122 INFO:tasks.rados.rados.0.smithi064.stdout:844: finishing write tid 2 to smithi06431743-8 2024-03-04T06:11:22.122 INFO:tasks.rados.rados.0.smithi064.stdout:844: finishing write tid 3 to smithi06431743-8 2024-03-04T06:11:22.122 INFO:tasks.rados.rados.0.smithi064.stdout:844: finishing write tid 4 to smithi06431743-8 2024-03-04T06:11:22.122 INFO:tasks.rados.rados.0.smithi064.stdout:844: oid 8 updating version 0 to 137 2024-03-04T06:11:22.122 INFO:tasks.rados.rados.0.smithi064.stdout:844: oid 8 updating version 137 to 138 2024-03-04T06:11:22.122 INFO:tasks.rados.rados.0.smithi064.stdout:844: oid 8 version 138 is already newer than 136 2024-03-04T06:11:22.122 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 138 (ObjNum 404 snap 98 seq_num 404) dirty exists 2024-03-04T06:11:22.122 INFO:tasks.rados.rados.0.smithi064.stdout:844: left oid 8 (ObjNum 404 snap 98 seq_num 404) 2024-03-04T06:11:22.226 INFO:tasks.rados.rados.0.smithi064.stdout:840: done (3 left) 2024-03-04T06:11:22.226 INFO:tasks.rados.rados.0.smithi064.stdout:842: done (2 left) 2024-03-04T06:11:22.227 INFO:tasks.rados.rados.0.smithi064.stdout:844: done (1 left) 2024-03-04T06:11:22.227 INFO:tasks.rados.rados.0.smithi064.stdout:845: done (0 left) 2024-03-04T06:11:22.227 INFO:tasks.rados.rados.0.smithi064.stdout:846: write oid 3 current snap is 99 2024-03-04T06:11:22.227 INFO:tasks.rados.rados.0.smithi064.stdout:846: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-03-04T06:11:22.232 INFO:tasks.rados.rados.0.smithi064.stdout:846: writing smithi06431743-3 from 406835 to 1049389 tid 1 2024-03-04T06:11:22.237 INFO:tasks.rados.rados.0.smithi064.stdout:846: writing smithi06431743-3 from 1532972 to 2039767 tid 2 2024-03-04T06:11:22.242 INFO:tasks.rados.rados.0.smithi064.stdout:846: writing smithi06431743-3 from 2772312 to 2806835 tid 3 2024-03-04T06:11:22.242 INFO:tasks.rados.rados.0.smithi064.stdout:847: read oid 50 snap 96 2024-03-04T06:11:22.242 INFO:tasks.rados.rados.0.smithi064.stdout:847: expect (ObjNum 352 snap 88 seq_num 352) 2024-03-04T06:11:22.242 INFO:tasks.rados.rados.0.smithi064.stdout:848: write (excl) oid 11 current snap is 99 2024-03-04T06:11:22.242 INFO:tasks.rados.rados.0.smithi064.stdout:848: seq_num 406 ranges {423642=717803,1875101=766806} 2024-03-04T06:11:22.248 INFO:tasks.rados.rados.0.smithi064.stdout:848: writing smithi06431743-11 from 423642 to 1141445 tid 1 2024-03-04T06:11:22.257 INFO:tasks.rados.rados.0.smithi064.stdout:848: writing smithi06431743-11 from 1875101 to 2641907 tid 2 2024-03-04T06:11:22.257 INFO:tasks.rados.rados.0.smithi064.stdout:849: read oid 42 snap 98 2024-03-04T06:11:22.257 INFO:tasks.rados.rados.0.smithi064.stdout:849: expect deleted 2024-03-04T06:11:22.258 INFO:tasks.rados.rados.0.smithi064.stdout:850: write (excl) oid 31 current snap is 99 2024-03-04T06:11:22.258 INFO:tasks.rados.rados.0.smithi064.stdout:850: seq_num 407 ranges {440449=793052,1933583=510464} 2024-03-04T06:11:22.265 INFO:tasks.rados.rados.0.smithi064.stdout:850: writing smithi06431743-31 from 440449 to 1233501 tid 1 2024-03-04T06:11:22.272 INFO:tasks.rados.rados.0.smithi064.stdout:850: writing smithi06431743-31 from 1933583 to 2444047 tid 2 2024-03-04T06:11:22.272 INFO:tasks.rados.rados.0.smithi064.stdout:851: delete oid 48 current snap is 99 2024-03-04T06:11:22.279 INFO:tasks.rados.rados.0.smithi064.stdout:846: finishing write tid 1 to smithi06431743-3 2024-03-04T06:11:22.287 INFO:tasks.rados.rados.0.smithi064.stdout:848: finishing write tid 1 to smithi06431743-11 2024-03-04T06:11:22.293 INFO:tasks.rados.rados.0.smithi064.stdout:847: expect (ObjNum 352 snap 88 seq_num 352) 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:846: finishing write tid 2 to smithi06431743-3 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:848: finishing write tid 2 to smithi06431743-11 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:846: finishing write tid 3 to smithi06431743-3 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:847: done (5 left) 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:849: done (4 left) 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:851: done (3 left) 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:852: delete oid 44 current snap is 99 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:848: finishing write tid 3 to smithi06431743-11 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:850: finishing write tid 1 to smithi06431743-31 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:848: finishing write tid 4 to smithi06431743-11 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:848: oid 11 updating version 0 to 195 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:848: oid 11 updating version 195 to 197 2024-03-04T06:11:22.636 INFO:tasks.rados.rados.0.smithi064.stdout:848: oid 11 version 197 is already newer than 196 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 197 (ObjNum 406 snap 99 seq_num 406) dirty exists 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:848: left oid 11 (ObjNum 406 snap 99 seq_num 406) 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:846: finishing write tid 4 to smithi06431743-3 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:846: finishing write tid 5 to smithi06431743-3 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:846: oid 3 updating version 0 to 143 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:846: oid 3 version 143 is already newer than 143 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:846: oid 3 updating version 143 to 145 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:846: oid 3 version 145 is already newer than 144 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 145 (ObjNum 405 snap 99 seq_num 405) dirty exists 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:846: left oid 3 (ObjNum 405 snap 99 seq_num 405) 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:850: finishing write tid 2 to smithi06431743-31 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:850: finishing write tid 3 to smithi06431743-31 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:850: finishing write tid 4 to smithi06431743-31 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:850: oid 31 updating version 0 to 88 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:850: oid 31 version 88 is already newer than 87 2024-03-04T06:11:22.637 INFO:tasks.rados.rados.0.smithi064.stdout:850: oid 31 updating version 88 to 89 2024-03-04T06:11:22.638 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 89 (ObjNum 407 snap 99 seq_num 407) dirty exists 2024-03-04T06:11:22.638 INFO:tasks.rados.rados.0.smithi064.stdout:850: left oid 31 (ObjNum 407 snap 99 seq_num 407) 2024-03-04T06:11:22.640 INFO:tasks.rados.rados.0.smithi064.stdout:846: done (3 left) 2024-03-04T06:11:22.640 INFO:tasks.rados.rados.0.smithi064.stdout:848: done (2 left) 2024-03-04T06:11:22.640 INFO:tasks.rados.rados.0.smithi064.stdout:850: done (1 left) 2024-03-04T06:11:22.640 INFO:tasks.rados.rados.0.smithi064.stdout:852: done (0 left) 2024-03-04T06:11:22.640 INFO:tasks.rados.rados.0.smithi064.stdout:853: delete oid 31 current snap is 99 2024-03-04T06:11:22.650 INFO:tasks.rados.rados.0.smithi064.stdout:853: done (0 left) 2024-03-04T06:11:22.650 INFO:tasks.rados.rados.0.smithi064.stdout:854: read oid 22 snap -1 2024-03-04T06:11:22.650 INFO:tasks.rados.rados.0.smithi064.stdout:854: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-04T06:11:22.650 INFO:tasks.rados.rados.0.smithi064.stdout:855: snap_remove snap 98 2024-03-04T06:11:23.200 INFO:tasks.rados.rados.0.smithi064.stdout:855: done (1 left) 2024-03-04T06:11:23.200 INFO:tasks.rados.rados.0.smithi064.stdout:856: delete oid 3 current snap is 99 2024-03-04T06:11:23.200 INFO:tasks.rados.rados.0.smithi064.stdout:854: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-04T06:11:23.337 INFO:tasks.rados.rados.0.smithi064.stdout:854: done (1 left) 2024-03-04T06:11:23.337 INFO:tasks.rados.rados.0.smithi064.stdout:856: done (0 left) 2024-03-04T06:11:23.337 INFO:tasks.rados.rados.0.smithi064.stdout:857: delete oid 16 current snap is 99 2024-03-04T06:11:23.355 INFO:tasks.rados.rados.0.smithi064.stdout:857: done (0 left) 2024-03-04T06:11:23.355 INFO:tasks.rados.rados.0.smithi064.stdout:858: write oid 4 current snap is 99 2024-03-04T06:11:23.355 INFO:tasks.rados.rados.0.smithi064.stdout:858: seq_num 413 ranges {541291=560899,1884475=521471} 2024-03-04T06:11:23.361 INFO:tasks.rados.rados.0.smithi064.stdout:858: writing smithi06431743-4 from 541291 to 1102190 tid 1 2024-03-04T06:11:23.369 INFO:tasks.rados.rados.0.smithi064.stdout:858: writing smithi06431743-4 from 1884475 to 2405946 tid 2 2024-03-04T06:11:23.369 INFO:tasks.rados.rados.0.smithi064.stdout:859: read oid 11 snap -1 2024-03-04T06:11:23.370 INFO:tasks.rados.rados.0.smithi064.stdout:859: expect (ObjNum 406 snap 99 seq_num 406) 2024-03-04T06:11:23.370 INFO:tasks.rados.rados.0.smithi064.stdout:860: read oid 44 snap 96 2024-03-04T06:11:23.370 INFO:tasks.rados.rados.0.smithi064.stdout:860: expect deleted 2024-03-04T06:11:23.370 INFO:tasks.rados.rados.0.smithi064.stdout:861: delete oid 40 current snap is 99 2024-03-04T06:11:23.380 INFO:tasks.rados.rados.0.smithi064.stdout:858: finishing write tid 1 to smithi06431743-4 2024-03-04T06:11:23.387 INFO:tasks.rados.rados.0.smithi064.stdout:859: expect (ObjNum 406 snap 99 seq_num 406) 2024-03-04T06:11:23.403 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:11:23.403 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:11:23.403 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:11:23.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:11:23.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:23.507+0000 7fcc87ed4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:23.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:23.514+0000 7fcc87ed4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:23.528 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:23.529 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:23.531 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:23.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:11:23.544 INFO:tasks.rados.rados.0.smithi064.stdout:858: finishing write tid 2 to smithi06431743-4 2024-03-04T06:11:23.544 INFO:tasks.rados.rados.0.smithi064.stdout:858: finishing write tid 3 to smithi06431743-4 2024-03-04T06:11:23.544 INFO:tasks.rados.rados.0.smithi064.stdout:858: finishing write tid 4 to smithi06431743-4 2024-03-04T06:11:23.544 INFO:tasks.rados.rados.0.smithi064.stdout:858: oid 4 updating version 0 to 381 2024-03-04T06:11:23.545 INFO:tasks.rados.rados.0.smithi064.stdout:858: oid 4 updating version 381 to 382 2024-03-04T06:11:23.545 INFO:tasks.rados.rados.0.smithi064.stdout:858: oid 4 version 382 is already newer than 380 2024-03-04T06:11:23.545 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 382 (ObjNum 413 snap 99 seq_num 413) dirty exists 2024-03-04T06:11:23.545 INFO:tasks.rados.rados.0.smithi064.stdout:858: left oid 4 (ObjNum 413 snap 99 seq_num 413) 2024-03-04T06:11:23.545 INFO:tasks.rados.rados.0.smithi064.stdout:858: done (3 left) 2024-03-04T06:11:23.545 INFO:tasks.rados.rados.0.smithi064.stdout:859: done (2 left) 2024-03-04T06:11:23.545 INFO:tasks.rados.rados.0.smithi064.stdout:860: done (1 left) 2024-03-04T06:11:23.545 INFO:tasks.rados.rados.0.smithi064.stdout:861: done (0 left) 2024-03-04T06:11:23.545 INFO:tasks.rados.rados.0.smithi064.stdout:862: read oid 29 snap -1 2024-03-04T06:11:23.545 INFO:tasks.rados.rados.0.smithi064.stdout:862: expect (ObjNum 381 snap 93 seq_num 381) 2024-03-04T06:11:23.545 INFO:tasks.rados.rados.0.smithi064.stdout:863: snap_remove snap 96 2024-03-04T06:11:23.639 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:23.634+0000 7f4c4822d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:23.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:23.641+0000 7f4c4822d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:23.654 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:23.655 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:23.655 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:23.655 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:23.655 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:23.655 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:23.655 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:23.655 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:23.655 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:23.655 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:23.655 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:23.655 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:23.657 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:23.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:11:23.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:23.803+0000 7f81b0c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:23.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:23.810+0000 7f81b0c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:23.825 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:23.828 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:23.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:11:23.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:23.930+0000 7ff69d0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:23.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:23.937+0000 7ff69d0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:23.951 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:23.951 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:23.951 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:23.951 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:23.951 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:23.951 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:23.951 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:23.951 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:23.952 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:23.952 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:23.952 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:23.952 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:23.954 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:24.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:24.231 INFO:tasks.rados.rados.0.smithi064.stdout:863: done (1 left) 2024-03-04T06:11:24.231 INFO:tasks.rados.rados.0.smithi064.stdout:864: snap_create 2024-03-04T06:11:24.231 INFO:tasks.rados.rados.0.smithi064.stdout:862: expect (ObjNum 381 snap 93 seq_num 381) 2024-03-04T06:11:24.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:24.245+0000 7fa63c19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:24.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:24.249+0000 7fa63c19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:25.235 INFO:tasks.rados.rados.0.smithi064.stdout:862: done (1 left) 2024-03-04T06:11:25.235 INFO:tasks.rados.rados.0.smithi064.stdout:864: done (0 left) 2024-03-04T06:11:25.235 INFO:tasks.rados.rados.0.smithi064.stdout:865: read oid 7 snap -1 2024-03-04T06:11:25.235 INFO:tasks.rados.rados.0.smithi064.stdout:865: expect deleted 2024-03-04T06:11:25.236 INFO:tasks.rados.rados.0.smithi064.stdout:866: snap_remove snap 99 2024-03-04T06:11:26.236 INFO:tasks.rados.rados.0.smithi064.stdout:866: done (1 left) 2024-03-04T06:11:26.237 INFO:tasks.rados.rados.0.smithi064.stdout:867: delete oid 13 current snap is 100 2024-03-04T06:11:26.284 INFO:tasks.rados.rados.0.smithi064.stdout:865: done (1 left) 2024-03-04T06:11:26.284 INFO:tasks.rados.rados.0.smithi064.stdout:867: done (0 left) 2024-03-04T06:11:26.284 INFO:tasks.rados.rados.0.smithi064.stdout:868: read oid 49 snap -1 2024-03-04T06:11:26.284 INFO:tasks.rados.rados.0.smithi064.stdout:868: expect (ObjNum 400 snap 97 seq_num 400) 2024-03-04T06:11:26.284 INFO:tasks.rados.rados.0.smithi064.stdout:869: delete oid 28 current snap is 100 2024-03-04T06:11:26.315 INFO:tasks.rados.rados.0.smithi064.stdout:869: done (1 left) 2024-03-04T06:11:26.315 INFO:tasks.rados.rados.0.smithi064.stdout:870: read oid 13 snap -1 2024-03-04T06:11:26.315 INFO:tasks.rados.rados.0.smithi064.stdout:870: expect deleted 2024-03-04T06:11:26.315 INFO:tasks.rados.rados.0.smithi064.stdout:871: write oid 34 current snap is 100 2024-03-04T06:11:26.316 INFO:tasks.rados.rados.0.smithi064.stdout:871: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-03-04T06:11:26.321 INFO:tasks.rados.rados.0.smithi064.stdout:871: writing smithi06431743-34 from 608519 to 1070414 tid 1 2024-03-04T06:11:26.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:26.327 INFO:tasks.rados.rados.0.smithi064.stdout:871: writing smithi06431743-34 from 1602050 to 2130859 tid 2 2024-03-04T06:11:26.333 INFO:tasks.rados.rados.0.smithi064.stdout:871: writing smithi06431743-34 from 2585035 to 3008519 tid 3 2024-03-04T06:11:26.333 INFO:tasks.rados.rados.0.smithi064.stdout:872: write (excl) oid 9 current snap is 100 2024-03-04T06:11:26.333 INFO:tasks.rados.rados.0.smithi064.stdout:872: seq_num 418 ranges {625326=537144,1944179=672467} 2024-03-04T06:11:26.340 INFO:tasks.rados.rados.0.smithi064.stdout:872: writing smithi06431743-9 from 625326 to 1162470 tid 1 2024-03-04T06:11:26.349 INFO:tasks.rados.rados.0.smithi064.stdout:872: writing smithi06431743-9 from 1944179 to 2616646 tid 2 2024-03-04T06:11:26.349 INFO:tasks.rados.rados.0.smithi064.stdout:873: write oid 50 current snap is 100 2024-03-04T06:11:26.349 INFO:tasks.rados.rados.0.smithi064.stdout:873: seq_num 419 ranges {642133=728746,2119014=532478} 2024-03-04T06:11:26.357 INFO:tasks.rados.rados.0.smithi064.stdout:873: writing smithi06431743-50 from 642133 to 1370879 tid 1 2024-03-04T06:11:26.364 INFO:tasks.rados.rados.0.smithi064.stdout:873: writing smithi06431743-50 from 2119014 to 2651492 tid 2 2024-03-04T06:11:26.364 INFO:tasks.rados.rados.0.smithi064.stdout:874: write (excl) oid 20 current snap is 100 2024-03-04T06:11:26.364 INFO:tasks.rados.rados.0.smithi064.stdout:874: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-03-04T06:11:26.370 INFO:tasks.rados.rados.0.smithi064.stdout:874: writing smithi06431743-20 from 658940 to 1062935 tid 1 2024-03-04T06:11:26.379 INFO:tasks.rados.rados.0.smithi064.stdout:874: writing smithi06431743-20 from 1777496 to 2453632 tid 2 2024-03-04T06:11:26.383 INFO:tasks.rados.rados.0.smithi064.stdout:874: writing smithi06431743-20 from 2950951 to 3058940 tid 3 2024-03-04T06:11:26.383 INFO:tasks.rados.rados.0.smithi064.stdout:870: done (5 left) 2024-03-04T06:11:26.383 INFO:tasks.rados.rados.0.smithi064.stdout:875: write (excl) oid 45 current snap is 100 2024-03-04T06:11:26.383 INFO:tasks.rados.rados.0.smithi064.stdout:875: seq_num 421 ranges {675747=479244,1835978=536147} 2024-03-04T06:11:26.390 INFO:tasks.rados.rados.0.smithi064.stdout:875: writing smithi06431743-45 from 675747 to 1154991 tid 1 2024-03-04T06:11:26.397 INFO:tasks.rados.rados.0.smithi064.stdout:875: writing smithi06431743-45 from 1835978 to 2372125 tid 2 2024-03-04T06:11:26.398 INFO:tasks.rados.rados.0.smithi064.stdout:876: write oid 28 current snap is 100 2024-03-04T06:11:26.398 INFO:tasks.rados.rados.0.smithi064.stdout:876: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-03-04T06:11:26.404 INFO:tasks.rados.rados.0.smithi064.stdout:876: writing smithi06431743-28 from 692554 to 1247047 tid 1 2024-03-04T06:11:26.411 INFO:tasks.rados.rados.0.smithi064.stdout:876: writing smithi06431743-28 from 1778107 to 2457912 tid 2 2024-03-04T06:11:26.415 INFO:tasks.rados.rados.0.smithi064.stdout:876: writing smithi06431743-28 from 2889444 to 3092554 tid 3 2024-03-04T06:11:26.416 INFO:tasks.rados.rados.0.smithi064.stdout:877: read oid 18 snap -1 2024-03-04T06:11:26.416 INFO:tasks.rados.rados.0.smithi064.stdout:877: expect deleted 2024-03-04T06:11:26.416 INFO:tasks.rados.rados.0.smithi064.stdout:878: write (excl) oid 48 current snap is 100 2024-03-04T06:11:26.416 INFO:tasks.rados.rados.0.smithi064.stdout:871: finishing write tid 1 to smithi06431743-34 2024-03-04T06:11:26.416 INFO:tasks.rados.rados.0.smithi064.stdout:868: expect (ObjNum 400 snap 97 seq_num 400) 2024-03-04T06:11:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:26.458+0000 7feb8a06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:26.464+0000 7feb8a06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:26.566 INFO:tasks.rados.rados.0.smithi064.stdout:872: finishing write tid 1 to smithi06431743-9 2024-03-04T06:11:26.566 INFO:tasks.rados.rados.0.smithi064.stdout:871: finishing write tid 2 to smithi06431743-34 2024-03-04T06:11:26.566 INFO:tasks.rados.rados.0.smithi064.stdout:874: finishing write tid 1 to smithi06431743-20 2024-03-04T06:11:26.566 INFO:tasks.rados.rados.0.smithi064.stdout:878: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-03-04T06:11:26.573 INFO:tasks.rados.rados.0.smithi064.stdout:878: writing smithi06431743-48 from 709361 to 1339103 tid 1 2024-03-04T06:11:26.578 INFO:tasks.rados.rados.0.smithi064.stdout:878: writing smithi06431743-48 from 1836589 to 2376405 tid 2 2024-03-04T06:11:26.585 INFO:tasks.rados.rados.0.smithi064.stdout:878: writing smithi06431743-48 from 3033220 to 3109361 tid 3 2024-03-04T06:11:26.585 INFO:tasks.rados.rados.0.smithi064.stdout:868: done (8 left) 2024-03-04T06:11:26.585 INFO:tasks.rados.rados.0.smithi064.stdout:879: snap_create 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:872: finishing write tid 2 to smithi06431743-9 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:871: finishing write tid 3 to smithi06431743-34 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:875: finishing write tid 1 to smithi06431743-45 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:874: finishing write tid 2 to smithi06431743-20 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:872: finishing write tid 3 to smithi06431743-9 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:872: finishing write tid 4 to smithi06431743-9 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:872: oid 9 updating version 0 to 354 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:872: oid 9 version 354 is already newer than 352 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:872: oid 9 version 354 is already newer than 353 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 354 (ObjNum 418 snap 100 seq_num 418) dirty exists 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:872: left oid 9 (ObjNum 418 snap 100 seq_num 418) 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:875: finishing write tid 2 to smithi06431743-45 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:871: finishing write tid 4 to smithi06431743-34 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:874: finishing write tid 3 to smithi06431743-20 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:871: finishing write tid 5 to smithi06431743-34 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:871: oid 34 updating version 0 to 365 2024-03-04T06:11:26.586 INFO:tasks.rados.rados.0.smithi064.stdout:871: oid 34 version 365 is already newer than 362 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:871: oid 34 version 365 is already newer than 364 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:871: oid 34 version 365 is already newer than 363 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 365 (ObjNum 417 snap 100 seq_num 417) dirty exists 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:871: left oid 34 (ObjNum 417 snap 100 seq_num 417) 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:875: finishing write tid 3 to smithi06431743-45 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:874: finishing write tid 4 to smithi06431743-20 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:873: finishing write tid 1 to smithi06431743-50 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:874: finishing write tid 5 to smithi06431743-20 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:874: oid 20 updating version 0 to 257 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:874: oid 20 updating version 257 to 260 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:874: oid 20 version 260 is already newer than 260 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:874: oid 20 version 260 is already newer than 258 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 260 (ObjNum 420 snap 100 seq_num 420) dirty exists 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:874: left oid 20 (ObjNum 420 snap 100 seq_num 420) 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:875: finishing write tid 4 to smithi06431743-45 2024-03-04T06:11:26.587 INFO:tasks.rados.rados.0.smithi064.stdout:875: oid 45 updating version 0 to 206 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:875: oid 45 updating version 206 to 207 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:875: oid 45 version 207 is already newer than 205 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 207 (ObjNum 421 snap 100 seq_num 421) dirty exists 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:875: left oid 45 (ObjNum 421 snap 100 seq_num 421) 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:873: finishing write tid 2 to smithi06431743-50 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:876: finishing write tid 1 to smithi06431743-28 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:873: finishing write tid 3 to smithi06431743-50 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:873: finishing write tid 4 to smithi06431743-50 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:873: oid 50 updating version 0 to 358 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:873: oid 50 version 358 is already newer than 358 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:873: oid 50 version 358 is already newer than 358 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 358 (ObjNum 419 snap 100 seq_num 419) dirty exists 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:873: left oid 50 (ObjNum 419 snap 100 seq_num 419) 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:876: finishing write tid 2 to smithi06431743-28 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:876: finishing write tid 3 to smithi06431743-28 2024-03-04T06:11:26.588 INFO:tasks.rados.rados.0.smithi064.stdout:876: finishing write tid 4 to smithi06431743-28 2024-03-04T06:11:26.589 INFO:tasks.rados.rados.0.smithi064.stdout:876: finishing write tid 5 to smithi06431743-28 2024-03-04T06:11:26.589 INFO:tasks.rados.rados.0.smithi064.stdout:876: oid 28 updating version 0 to 369 2024-03-04T06:11:26.589 INFO:tasks.rados.rados.0.smithi064.stdout:876: oid 28 version 369 is already newer than 369 2024-03-04T06:11:26.589 INFO:tasks.rados.rados.0.smithi064.stdout:876: oid 28 version 369 is already newer than 369 2024-03-04T06:11:26.589 INFO:tasks.rados.rados.0.smithi064.stdout:876: oid 28 version 369 is already newer than 369 2024-03-04T06:11:26.589 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 369 (ObjNum 422 snap 100 seq_num 422) dirty exists 2024-03-04T06:11:26.589 INFO:tasks.rados.rados.0.smithi064.stdout:876: left oid 28 (ObjNum 422 snap 100 seq_num 422) 2024-03-04T06:11:26.590 INFO:tasks.rados.rados.0.smithi064.stdout:878: finishing write tid 1 to smithi06431743-48 2024-03-04T06:11:26.608 INFO:tasks.rados.rados.0.smithi064.stdout:878: finishing write tid 2 to smithi06431743-48 2024-03-04T06:11:26.621 INFO:tasks.rados.rados.0.smithi064.stdout:878: finishing write tid 3 to smithi06431743-48 2024-03-04T06:11:26.626 INFO:tasks.rados.rados.0.smithi064.stdout:878: finishing write tid 4 to smithi06431743-48 2024-03-04T06:11:26.626 INFO:tasks.rados.rados.0.smithi064.stdout:878: finishing write tid 5 to smithi06431743-48 2024-03-04T06:11:26.626 INFO:tasks.rados.rados.0.smithi064.stdout:878: oid 48 updating version 0 to 152 2024-03-04T06:11:26.626 INFO:tasks.rados.rados.0.smithi064.stdout:878: oid 48 version 152 is already newer than 151 2024-03-04T06:11:26.626 INFO:tasks.rados.rados.0.smithi064.stdout:878: oid 48 version 152 is already newer than 150 2024-03-04T06:11:26.627 INFO:tasks.rados.rados.0.smithi064.stdout:878: oid 48 version 152 is already newer than 149 2024-03-04T06:11:26.627 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 152 (ObjNum 423 snap 100 seq_num 423) dirty exists 2024-03-04T06:11:26.627 INFO:tasks.rados.rados.0.smithi064.stdout:878: left oid 48 (ObjNum 423 snap 100 seq_num 423) 2024-03-04T06:11:27.241 INFO:tasks.rados.rados.0.smithi064.stdout:871: done (8 left) 2024-03-04T06:11:27.242 INFO:tasks.rados.rados.0.smithi064.stdout:872: done (7 left) 2024-03-04T06:11:27.242 INFO:tasks.rados.rados.0.smithi064.stdout:873: done (6 left) 2024-03-04T06:11:27.242 INFO:tasks.rados.rados.0.smithi064.stdout:874: done (5 left) 2024-03-04T06:11:27.242 INFO:tasks.rados.rados.0.smithi064.stdout:875: done (4 left) 2024-03-04T06:11:27.242 INFO:tasks.rados.rados.0.smithi064.stdout:876: done (3 left) 2024-03-04T06:11:27.242 INFO:tasks.rados.rados.0.smithi064.stdout:877: done (2 left) 2024-03-04T06:11:27.242 INFO:tasks.rados.rados.0.smithi064.stdout:878: done (1 left) 2024-03-04T06:11:27.242 INFO:tasks.rados.rados.0.smithi064.stdout:879: done (0 left) 2024-03-04T06:11:27.242 INFO:tasks.rados.rados.0.smithi064.stdout:880: write (excl) oid 48 current snap is 101 2024-03-04T06:11:27.242 INFO:tasks.rados.rados.0.smithi064.stdout:880: seq_num 424 ranges {726168=704991,1895071=683474} 2024-03-04T06:11:27.249 INFO:tasks.rados.rados.0.smithi064.stdout:880: writing smithi06431743-48 from 726168 to 1431159 tid 1 2024-03-04T06:11:27.255 INFO:tasks.rados.rados.0.smithi064.stdout:880: writing smithi06431743-48 from 1895071 to 2578545 tid 2 2024-03-04T06:11:27.255 INFO:tasks.rados.rados.0.smithi064.stdout:881: delete oid 24 current snap is 101 2024-03-04T06:11:27.307 INFO:tasks.rados.rados.0.smithi064.stdout:880: finishing write tid 1 to smithi06431743-48 2024-03-04T06:11:27.309 INFO:tasks.rados.rados.0.smithi064.stdout:881: done (1 left) 2024-03-04T06:11:27.310 INFO:tasks.rados.rados.0.smithi064.stdout:882: snap_remove snap 100 2024-03-04T06:11:28.243 INFO:tasks.rados.rados.0.smithi064.stdout:882: done (1 left) 2024-03-04T06:11:28.258 INFO:tasks.rados.rados.0.smithi064.stdout:883: write (excl) oid 42 current snap is 101 2024-03-04T06:11:28.258 INFO:tasks.rados.rados.0.smithi064.stdout:880: finishing write tid 2 to smithi06431743-48 2024-03-04T06:11:28.258 INFO:tasks.rados.rados.0.smithi064.stdout:880: finishing write tid 3 to smithi06431743-48 2024-03-04T06:11:28.258 INFO:tasks.rados.rados.0.smithi064.stdout:880: finishing write tid 4 to smithi06431743-48 2024-03-04T06:11:28.258 INFO:tasks.rados.rados.0.smithi064.stdout:880: oid 48 updating version 0 to 155 2024-03-04T06:11:28.258 INFO:tasks.rados.rados.0.smithi064.stdout:880: oid 48 updating version 155 to 156 2024-03-04T06:11:28.258 INFO:tasks.rados.rados.0.smithi064.stdout:880: oid 48 version 156 is already newer than 154 2024-03-04T06:11:28.258 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 156 (ObjNum 424 snap 101 seq_num 424) dirty exists 2024-03-04T06:11:28.259 INFO:tasks.rados.rados.0.smithi064.stdout:880: left oid 48 (ObjNum 424 snap 101 seq_num 424) 2024-03-04T06:11:28.259 INFO:tasks.rados.rados.0.smithi064.stdout:883: seq_num 426 ranges {759782=571842,2012035=687143} 2024-03-04T06:11:28.259 INFO:tasks.rados.rados.0.smithi064.stdout:883: writing smithi06431743-42 from 759782 to 1331624 tid 1 2024-03-04T06:11:28.259 INFO:tasks.rados.rados.0.smithi064.stdout:883: writing smithi06431743-42 from 2012035 to 2699178 tid 2 2024-03-04T06:11:28.259 INFO:tasks.rados.rados.0.smithi064.stdout:880: done (1 left) 2024-03-04T06:11:28.259 INFO:tasks.rados.rados.0.smithi064.stdout:884: snap_remove snap 97 2024-03-04T06:11:28.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:28.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:28.737+0000 7f45d619d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:28.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:28.741+0000 7f45d619d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:28.955 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:11:28.955 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:11:28.955 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:11:28.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:11:29.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:29.059+0000 7f500d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:29.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:29.067+0000 7f500d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:29.080 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:29.083 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:29.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:11:29.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:29.186+0000 7fd62c91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:29.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:29.193+0000 7fd62c91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:29.206 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:29.206 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:29.206 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:29.206 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:29.206 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:29.206 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:29.206 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:29.206 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:29.206 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:29.207 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:29.207 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:29.207 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:29.209 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:29.209 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:29.209 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:11:29.251 INFO:tasks.rados.rados.0.smithi064.stdout:884: done (1 left) 2024-03-04T06:11:29.251 INFO:tasks.rados.rados.0.smithi064.stdout:885: write (excl) oid 33 current snap is 101 2024-03-04T06:11:29.251 INFO:tasks.rados.rados.0.smithi064.stdout:883: finishing write tid 1 to smithi06431743-42 2024-03-04T06:11:29.251 INFO:tasks.rados.rados.0.smithi064.stdout:885: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2024-03-04T06:11:29.258 INFO:tasks.rados.rados.0.smithi064.stdout:885: writing smithi06431743-33 from 776589 to 1423680 tid 1 2024-03-04T06:11:29.264 INFO:tasks.rados.rados.0.smithi064.stdout:885: writing smithi06431743-33 from 2070517 to 2501318 tid 2 2024-03-04T06:11:29.269 INFO:tasks.rados.rados.0.smithi064.stdout:885: writing smithi06431743-33 from 3026559 to 3176589 tid 3 2024-03-04T06:11:29.269 INFO:tasks.rados.rados.0.smithi064.stdout:886: delete oid 6 current snap is 101 2024-03-04T06:11:29.269 INFO:tasks.rados.rados.0.smithi064.stdout:883: finishing write tid 2 to smithi06431743-42 2024-03-04T06:11:29.269 INFO:tasks.rados.rados.0.smithi064.stdout:883: finishing write tid 3 to smithi06431743-42 2024-03-04T06:11:29.269 INFO:tasks.rados.rados.0.smithi064.stdout:883: finishing write tid 4 to smithi06431743-42 2024-03-04T06:11:29.269 INFO:tasks.rados.rados.0.smithi064.stdout:883: oid 42 updating version 0 to 149 2024-03-04T06:11:29.269 INFO:tasks.rados.rados.0.smithi064.stdout:883: oid 42 version 149 is already newer than 149 2024-03-04T06:11:29.269 INFO:tasks.rados.rados.0.smithi064.stdout:883: oid 42 version 149 is already newer than 147 2024-03-04T06:11:29.269 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 149 (ObjNum 426 snap 101 seq_num 426) dirty exists 2024-03-04T06:11:29.269 INFO:tasks.rados.rados.0.smithi064.stdout:883: left oid 42 (ObjNum 426 snap 101 seq_num 426) 2024-03-04T06:11:29.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:29.338+0000 7fd332f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:29.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:29.348+0000 7fd332f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:29.368 INFO:tasks.rados.rados.0.smithi064.stdout:883: done (2 left) 2024-03-04T06:11:29.368 INFO:tasks.rados.rados.0.smithi064.stdout:886: done (1 left) 2024-03-04T06:11:29.368 INFO:tasks.rados.rados.0.smithi064.stdout:887: write (excl) oid 15 current snap is 101 2024-03-04T06:11:29.368 INFO:tasks.rados.rados.0.smithi064.stdout:887: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-03-04T06:11:29.369 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:29.369 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:29.369 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:29.369 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:29.369 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:29.369 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:29.370 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:29.370 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:29.370 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:29.370 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:29.370 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:29.370 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:29.372 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:29.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:11:29.374 INFO:tasks.rados.rados.0.smithi064.stdout:887: writing smithi06431743-15 from 410203 to 1207792 tid 1 2024-03-04T06:11:29.380 INFO:tasks.rados.rados.0.smithi064.stdout:887: writing smithi06431743-15 from 1787481 to 2221951 tid 2 2024-03-04T06:11:29.386 INFO:tasks.rados.rados.0.smithi064.stdout:887: writing smithi06431743-15 from 2681405 to 3210203 tid 3 2024-03-04T06:11:29.386 INFO:tasks.rados.rados.0.smithi064.stdout:888: write oid 32 current snap is 101 2024-03-04T06:11:29.386 INFO:tasks.rados.rados.0.smithi064.stdout:888: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-03-04T06:11:29.391 INFO:tasks.rados.rados.0.smithi064.stdout:888: writing smithi06431743-32 from 427010 to 899848 tid 1 2024-03-04T06:11:29.398 INFO:tasks.rados.rados.0.smithi064.stdout:888: writing smithi06431743-32 from 1329610 to 2024091 tid 2 2024-03-04T06:11:29.405 INFO:tasks.rados.rados.0.smithi064.stdout:888: writing smithi06431743-32 from 2708828 to 3227010 tid 3 2024-03-04T06:11:29.406 INFO:tasks.rados.rados.0.smithi064.stdout:889: delete oid 46 current snap is 101 2024-03-04T06:11:29.406 INFO:tasks.rados.rados.0.smithi064.stdout:885: finishing write tid 1 to smithi06431743-33 2024-03-04T06:11:29.406 INFO:tasks.rados.rados.0.smithi064.stdout:887: finishing write tid 1 to smithi06431743-15 2024-03-04T06:11:29.406 INFO:tasks.rados.rados.0.smithi064.stdout:885: finishing write tid 2 to smithi06431743-33 2024-03-04T06:11:29.421 INFO:tasks.rados.rados.0.smithi064.stdout:885: finishing write tid 3 to smithi06431743-33 2024-03-04T06:11:29.442 INFO:tasks.rados.rados.0.smithi064.stdout:885: finishing write tid 4 to smithi06431743-33 2024-03-04T06:11:29.443 INFO:tasks.rados.rados.0.smithi064.stdout:887: finishing write tid 2 to smithi06431743-15 2024-03-04T06:11:29.443 INFO:tasks.rados.rados.0.smithi064.stdout:885: finishing write tid 5 to smithi06431743-33 2024-03-04T06:11:29.443 INFO:tasks.rados.rados.0.smithi064.stdout:885: oid 33 updating version 0 to 365 2024-03-04T06:11:29.443 INFO:tasks.rados.rados.0.smithi064.stdout:885: oid 33 version 365 is already newer than 363 2024-03-04T06:11:29.443 INFO:tasks.rados.rados.0.smithi064.stdout:885: oid 33 version 365 is already newer than 364 2024-03-04T06:11:29.443 INFO:tasks.rados.rados.0.smithi064.stdout:885: oid 33 updating version 365 to 366 2024-03-04T06:11:29.443 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 366 (ObjNum 427 snap 101 seq_num 427) dirty exists 2024-03-04T06:11:29.443 INFO:tasks.rados.rados.0.smithi064.stdout:885: left oid 33 (ObjNum 427 snap 101 seq_num 427) 2024-03-04T06:11:29.480 INFO:tasks.rados.rados.0.smithi064.stdout:887: finishing write tid 3 to smithi06431743-15 2024-03-04T06:11:29.487 INFO:tasks.rados.rados.0.smithi064.stdout:888: finishing write tid 1 to smithi06431743-32 2024-03-04T06:11:29.495 INFO:tasks.rados.rados.0.smithi064.stdout:888: finishing write tid 2 to smithi06431743-32 2024-03-04T06:11:29.495 INFO:tasks.rados.rados.0.smithi064.stdout:887: finishing write tid 4 to smithi06431743-15 2024-03-04T06:11:29.496 INFO:tasks.rados.rados.0.smithi064.stdout:887: finishing write tid 5 to smithi06431743-15 2024-03-04T06:11:29.496 INFO:tasks.rados.rados.0.smithi064.stdout:887: oid 15 updating version 0 to 151 2024-03-04T06:11:29.496 INFO:tasks.rados.rados.0.smithi064.stdout:887: oid 15 version 151 is already newer than 149 2024-03-04T06:11:29.497 INFO:tasks.rados.rados.0.smithi064.stdout:887: oid 15 updating version 151 to 152 2024-03-04T06:11:29.497 INFO:tasks.rados.rados.0.smithi064.stdout:887: oid 15 version 152 is already newer than 150 2024-03-04T06:11:29.497 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 152 (ObjNum 429 snap 101 seq_num 429) dirty exists 2024-03-04T06:11:29.497 INFO:tasks.rados.rados.0.smithi064.stdout:887: left oid 15 (ObjNum 429 snap 101 seq_num 429) 2024-03-04T06:11:29.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:29.493+0000 7f0b6d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:29.502 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:29.501+0000 7f0b6d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:29.510 INFO:tasks.rados.rados.0.smithi064.stdout:888: finishing write tid 3 to smithi06431743-32 2024-03-04T06:11:29.512 INFO:tasks.rados.rados.0.smithi064.stdout:888: finishing write tid 4 to smithi06431743-32 2024-03-04T06:11:29.513 INFO:tasks.rados.rados.0.smithi064.stdout:888: finishing write tid 5 to smithi06431743-32 2024-03-04T06:11:29.514 INFO:tasks.rados.rados.0.smithi064.stdout:888: oid 32 updating version 0 to 377 2024-03-04T06:11:29.514 INFO:tasks.rados.rados.0.smithi064.stdout:888: oid 32 version 377 is already newer than 375 2024-03-04T06:11:29.514 INFO:tasks.rados.rados.0.smithi064.stdout:888: oid 32 version 377 is already newer than 377 2024-03-04T06:11:29.514 INFO:tasks.rados.rados.0.smithi064.stdout:888: oid 32 version 377 is already newer than 374 2024-03-04T06:11:29.514 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 377 (ObjNum 430 snap 101 seq_num 430) dirty exists 2024-03-04T06:11:29.514 INFO:tasks.rados.rados.0.smithi064.stdout:888: left oid 32 (ObjNum 430 snap 101 seq_num 430) 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:29.515 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:29.516 INFO:tasks.rados.rados.0.smithi064.stdout:885: done (3 left) 2024-03-04T06:11:29.516 INFO:tasks.rados.rados.0.smithi064.stdout:887: done (2 left) 2024-03-04T06:11:29.516 INFO:tasks.rados.rados.0.smithi064.stdout:888: done (1 left) 2024-03-04T06:11:29.516 INFO:tasks.rados.rados.0.smithi064.stdout:889: done (0 left) 2024-03-04T06:11:29.516 INFO:tasks.rados.rados.0.smithi064.stdout:890: write oid 16 current snap is 101 2024-03-04T06:11:29.516 INFO:tasks.rados.rados.0.smithi064.stdout:890: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-03-04T06:11:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:29.519 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:29.522 INFO:tasks.rados.rados.0.smithi064.stdout:890: writing smithi06431743-16 from 460624 to 1083960 tid 1 2024-03-04T06:11:29.530 INFO:tasks.rados.rados.0.smithi064.stdout:890: writing smithi06431743-16 from 1846574 to 2544724 tid 2 2024-03-04T06:11:29.534 INFO:tasks.rados.rados.0.smithi064.stdout:890: writing smithi06431743-16 from 3163674 to 3260624 tid 3 2024-03-04T06:11:29.535 INFO:tasks.rados.rados.0.smithi064.stdout:891: write (excl) oid 23 current snap is 101 2024-03-04T06:11:29.535 INFO:tasks.rados.rados.0.smithi064.stdout:891: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-03-04T06:11:29.541 INFO:tasks.rados.rados.0.smithi064.stdout:891: writing smithi06431743-23 from 477431 to 1176016 tid 1 2024-03-04T06:11:29.548 INFO:tasks.rados.rados.0.smithi064.stdout:891: writing smithi06431743-23 from 1905056 to 2346864 tid 2 2024-03-04T06:11:29.554 INFO:tasks.rados.rados.0.smithi064.stdout:891: writing smithi06431743-23 from 3074744 to 3277431 tid 3 2024-03-04T06:11:29.554 INFO:tasks.rados.rados.0.smithi064.stdout:892: read oid 26 snap -1 2024-03-04T06:11:29.554 INFO:tasks.rados.rados.0.smithi064.stdout:892: expect (ObjNum 370 snap 91 seq_num 370) 2024-03-04T06:11:29.554 INFO:tasks.rados.rados.0.smithi064.stdout:893: read oid 15 snap -1 2024-03-04T06:11:29.554 INFO:tasks.rados.rados.0.smithi064.stdout:893: expect (ObjNum 429 snap 101 seq_num 429) 2024-03-04T06:11:29.554 INFO:tasks.rados.rados.0.smithi064.stdout:894: write oid 6 current snap is 101 2024-03-04T06:11:29.554 INFO:tasks.rados.rados.0.smithi064.stdout:890: finishing write tid 1 to smithi06431743-16 2024-03-04T06:11:29.554 INFO:tasks.rados.rados.0.smithi064.stdout:894: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-03-04T06:11:29.559 INFO:tasks.rados.rados.0.smithi064.stdout:894: writing smithi06431743-6 from 494238 to 984425 tid 1 2024-03-04T06:11:29.566 INFO:tasks.rados.rados.0.smithi064.stdout:894: writing smithi06431743-6 from 1563538 to 2265357 tid 2 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:894: writing smithi06431743-6 from 2818520 to 3294238 tid 3 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:895: read oid 34 snap -1 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:895: expect (ObjNum 417 snap 100 seq_num 417) 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:896: snap_create 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:890: finishing write tid 2 to smithi06431743-16 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:890: finishing write tid 3 to smithi06431743-16 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:891: finishing write tid 1 to smithi06431743-23 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:890: finishing write tid 4 to smithi06431743-16 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:890: finishing write tid 5 to smithi06431743-16 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:890: oid 16 updating version 0 to 152 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:890: oid 16 version 152 is already newer than 150 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:890: oid 16 version 152 is already newer than 151 2024-03-04T06:11:29.573 INFO:tasks.rados.rados.0.smithi064.stdout:890: oid 16 version 152 is already newer than 149 2024-03-04T06:11:29.574 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 152 (ObjNum 432 snap 101 seq_num 432) dirty exists 2024-03-04T06:11:29.574 INFO:tasks.rados.rados.0.smithi064.stdout:890: left oid 16 (ObjNum 432 snap 101 seq_num 432) 2024-03-04T06:11:29.574 INFO:tasks.rados.rados.0.smithi064.stdout:892: expect (ObjNum 370 snap 91 seq_num 370) 2024-03-04T06:11:29.711 INFO:tasks.rados.rados.0.smithi064.stdout:891: finishing write tid 2 to smithi06431743-23 2024-03-04T06:11:29.712 INFO:tasks.rados.rados.0.smithi064.stdout:893: expect (ObjNum 429 snap 101 seq_num 429) 2024-03-04T06:11:29.855 INFO:tasks.rados.rados.0.smithi064.stdout:894: finishing write tid 1 to smithi06431743-6 2024-03-04T06:11:29.856 INFO:tasks.rados.rados.0.smithi064.stdout:891: finishing write tid 3 to smithi06431743-23 2024-03-04T06:11:29.856 INFO:tasks.rados.rados.0.smithi064.stdout:895: expect (ObjNum 417 snap 100 seq_num 417) 2024-03-04T06:11:30.064 INFO:tasks.rados.rados.0.smithi064.stdout:891: finishing write tid 4 to smithi06431743-23 2024-03-04T06:11:30.064 INFO:tasks.rados.rados.0.smithi064.stdout:891: finishing write tid 5 to smithi06431743-23 2024-03-04T06:11:30.064 INFO:tasks.rados.rados.0.smithi064.stdout:891: oid 23 updating version 0 to 156 2024-03-04T06:11:30.064 INFO:tasks.rados.rados.0.smithi064.stdout:891: oid 23 version 156 is already newer than 153 2024-03-04T06:11:30.064 INFO:tasks.rados.rados.0.smithi064.stdout:891: oid 23 version 156 is already newer than 156 2024-03-04T06:11:30.064 INFO:tasks.rados.rados.0.smithi064.stdout:891: oid 23 version 156 is already newer than 154 2024-03-04T06:11:30.064 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 156 (ObjNum 433 snap 101 seq_num 433) dirty exists 2024-03-04T06:11:30.064 INFO:tasks.rados.rados.0.smithi064.stdout:891: left oid 23 (ObjNum 433 snap 101 seq_num 433) 2024-03-04T06:11:30.064 INFO:tasks.rados.rados.0.smithi064.stdout:894: finishing write tid 2 to smithi06431743-6 2024-03-04T06:11:30.064 INFO:tasks.rados.rados.0.smithi064.stdout:894: finishing write tid 3 to smithi06431743-6 2024-03-04T06:11:30.064 INFO:tasks.rados.rados.0.smithi064.stdout:894: finishing write tid 4 to smithi06431743-6 2024-03-04T06:11:30.065 INFO:tasks.rados.rados.0.smithi064.stdout:894: finishing write tid 5 to smithi06431743-6 2024-03-04T06:11:30.065 INFO:tasks.rados.rados.0.smithi064.stdout:894: oid 6 updating version 0 to 156 2024-03-04T06:11:30.065 INFO:tasks.rados.rados.0.smithi064.stdout:894: oid 6 version 156 is already newer than 153 2024-03-04T06:11:30.065 INFO:tasks.rados.rados.0.smithi064.stdout:894: oid 6 version 156 is already newer than 154 2024-03-04T06:11:30.065 INFO:tasks.rados.rados.0.smithi064.stdout:894: oid 6 version 156 is already newer than 156 2024-03-04T06:11:30.065 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 156 (ObjNum 434 snap 101 seq_num 434) dirty exists 2024-03-04T06:11:30.065 INFO:tasks.rados.rados.0.smithi064.stdout:894: left oid 6 (ObjNum 434 snap 101 seq_num 434) 2024-03-04T06:11:30.250 INFO:tasks.rados.rados.0.smithi064.stdout:890: done (6 left) 2024-03-04T06:11:30.251 INFO:tasks.rados.rados.0.smithi064.stdout:891: done (5 left) 2024-03-04T06:11:30.251 INFO:tasks.rados.rados.0.smithi064.stdout:892: done (4 left) 2024-03-04T06:11:30.251 INFO:tasks.rados.rados.0.smithi064.stdout:893: done (3 left) 2024-03-04T06:11:30.251 INFO:tasks.rados.rados.0.smithi064.stdout:894: done (2 left) 2024-03-04T06:11:30.251 INFO:tasks.rados.rados.0.smithi064.stdout:895: done (1 left) 2024-03-04T06:11:30.251 INFO:tasks.rados.rados.0.smithi064.stdout:896: done (0 left) 2024-03-04T06:11:30.251 INFO:tasks.rados.rados.0.smithi064.stdout:897: snap_create 2024-03-04T06:11:30.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:30.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:30.917+0000 7f498e99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:30.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:30.921+0000 7f498e99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:31.255 INFO:tasks.rados.rados.0.smithi064.stdout:897: done (0 left) 2024-03-04T06:11:31.256 INFO:tasks.rados.rados.0.smithi064.stdout:898: delete oid 13 current snap is 103 2024-03-04T06:11:31.305 INFO:tasks.rados.rados.0.smithi064.stdout:898: done (0 left) 2024-03-04T06:11:31.306 INFO:tasks.rados.rados.0.smithi064.stdout:899: delete oid 45 current snap is 103 2024-03-04T06:11:31.322 INFO:tasks.rados.rados.0.smithi064.stdout:899: done (0 left) 2024-03-04T06:11:31.322 INFO:tasks.rados.rados.0.smithi064.stdout:900: delete oid 48 current snap is 103 2024-03-04T06:11:31.374 INFO:tasks.rados.rados.0.smithi064.stdout:900: done (0 left) 2024-03-04T06:11:31.374 INFO:tasks.rados.rados.0.smithi064.stdout:901: read oid 11 snap -1 2024-03-04T06:11:31.374 INFO:tasks.rados.rados.0.smithi064.stdout:901: expect (ObjNum 406 snap 99 seq_num 406) 2024-03-04T06:11:31.374 INFO:tasks.rados.rados.0.smithi064.stdout:902: read oid 46 snap -1 2024-03-04T06:11:31.374 INFO:tasks.rados.rados.0.smithi064.stdout:902: expect deleted 2024-03-04T06:11:31.375 INFO:tasks.rados.rados.0.smithi064.stdout:903: read oid 16 snap -1 2024-03-04T06:11:31.375 INFO:tasks.rados.rados.0.smithi064.stdout:903: expect (ObjNum 432 snap 101 seq_num 432) 2024-03-04T06:11:31.375 INFO:tasks.rados.rados.0.smithi064.stdout:904: snap_create 2024-03-04T06:11:31.391 INFO:tasks.rados.rados.0.smithi064.stdout:903: expect (ObjNum 432 snap 101 seq_num 432) 2024-03-04T06:11:31.513 INFO:tasks.rados.rados.0.smithi064.stdout:901: expect (ObjNum 406 snap 99 seq_num 406) 2024-03-04T06:11:32.259 INFO:tasks.rados.rados.0.smithi064.stdout:901: done (3 left) 2024-03-04T06:11:32.259 INFO:tasks.rados.rados.0.smithi064.stdout:902: done (2 left) 2024-03-04T06:11:32.259 INFO:tasks.rados.rados.0.smithi064.stdout:903: done (1 left) 2024-03-04T06:11:32.259 INFO:tasks.rados.rados.0.smithi064.stdout:904: done (0 left) 2024-03-04T06:11:32.259 INFO:tasks.rados.rados.0.smithi064.stdout:905: delete oid 49 current snap is 104 2024-03-04T06:11:32.298 INFO:tasks.rados.rados.0.smithi064.stdout:905: done (0 left) 2024-03-04T06:11:32.299 INFO:tasks.rados.rados.0.smithi064.stdout:906: write oid 8 current snap is 104 2024-03-04T06:11:32.299 INFO:tasks.rados.rados.0.smithi064.stdout:906: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2024-03-04T06:11:32.305 INFO:tasks.rados.rados.0.smithi064.stdout:906: writing smithi06431743-8 from 578273 to 1044705 tid 1 2024-03-04T06:11:32.313 INFO:tasks.rados.rados.0.smithi064.stdout:906: writing smithi06431743-8 from 1739595 to 2192410 tid 2 2024-03-04T06:11:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:906: writing smithi06431743-8 from 2722929 to 3378273 tid 3 2024-03-04T06:11:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:907: write (excl) oid 46 current snap is 104 2024-03-04T06:11:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:907: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-03-04T06:11:32.330 INFO:tasks.rados.rados.0.smithi064.stdout:907: writing smithi06431743-46 from 595080 to 1136761 tid 1 2024-03-04T06:11:32.339 INFO:tasks.rados.rados.0.smithi064.stdout:907: writing smithi06431743-46 from 1798077 to 2394550 tid 2 2024-03-04T06:11:32.345 INFO:tasks.rados.rados.0.smithi064.stdout:907: writing smithi06431743-46 from 3150352 to 3395080 tid 3 2024-03-04T06:11:32.345 INFO:tasks.rados.rados.0.smithi064.stdout:908: snap_remove snap 103 2024-03-04T06:11:32.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:33.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:33.097+0000 7f5eaa22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:33.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:33.101+0000 7f5eaa22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:33.203 INFO:tasks.rados.rados.0.smithi064.stdout:908: done (2 left) 2024-03-04T06:11:33.204 INFO:tasks.rados.rados.0.smithi064.stdout:909: snap_remove snap 102 2024-03-04T06:11:34.203 INFO:tasks.rados.rados.0.smithi064.stdout:909: done (2 left) 2024-03-04T06:11:34.203 INFO:tasks.rados.rados.0.smithi064.stdout:910: read oid 7 snap -1 2024-03-04T06:11:34.203 INFO:tasks.rados.rados.0.smithi064.stdout:910: expect deleted 2024-03-04T06:11:34.203 INFO:tasks.rados.rados.0.smithi064.stdout:911: snap_remove snap 101 2024-03-04T06:11:34.520 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:11:34.520 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:11:34.520 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:11:34.520 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:11:34.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:34.628+0000 7f9ea039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:34.636 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:34.636+0000 7f9ea039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:34.649 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:34.649 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:34.649 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:34.651 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:34.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:11:34.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:34.754+0000 7f25be82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:34.761 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:34.761+0000 7f25be82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:34.773 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:34.779 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:34.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:11:34.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:34.881+0000 7f9469b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:34.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:34.889+0000 7f9469b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:34.905 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:34.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:11:35.013 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:35.008+0000 7ffb3b8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:35.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:35.016+0000 7ffb3b8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:35.029 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:35.032 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:35.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:35.203 INFO:tasks.rados.rados.0.smithi064.stdout:911: done (3 left) 2024-03-04T06:11:35.203 INFO:tasks.rados.rados.0.smithi064.stdout:912: write oid 28 current snap is 104 2024-03-04T06:11:35.203 INFO:tasks.rados.rados.0.smithi064.stdout:907: finishing write tid 1 to smithi06431743-46 2024-03-04T06:11:35.203 INFO:tasks.rados.rados.0.smithi064.stdout:912: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-03-04T06:11:35.210 INFO:tasks.rados.rados.0.smithi064.stdout:912: writing smithi06431743-28 from 611887 to 1345170 tid 1 2024-03-04T06:11:35.216 INFO:tasks.rados.rados.0.smithi064.stdout:912: writing smithi06431743-28 from 1972912 to 2429396 tid 2 2024-03-04T06:11:35.222 INFO:tasks.rados.rados.0.smithi064.stdout:912: writing smithi06431743-28 from 2894128 to 3411887 tid 3 2024-03-04T06:11:35.222 INFO:tasks.rados.rados.0.smithi064.stdout:913: read oid 23 snap -1 2024-03-04T06:11:35.222 INFO:tasks.rados.rados.0.smithi064.stdout:913: expect (ObjNum 433 snap 101 seq_num 433) 2024-03-04T06:11:35.222 INFO:tasks.rados.rados.0.smithi064.stdout:914: delete oid 34 current snap is 104 2024-03-04T06:11:35.222 INFO:tasks.rados.rados.0.smithi064.stdout:906: finishing write tid 1 to smithi06431743-8 2024-03-04T06:11:35.222 INFO:tasks.rados.rados.0.smithi064.stdout:907: finishing write tid 2 to smithi06431743-46 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:906: finishing write tid 2 to smithi06431743-8 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:907: finishing write tid 3 to smithi06431743-46 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:907: finishing write tid 4 to smithi06431743-46 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:906: finishing write tid 3 to smithi06431743-8 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:907: finishing write tid 5 to smithi06431743-46 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:907: oid 46 updating version 0 to 380 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:907: oid 46 updating version 380 to 382 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:907: oid 46 version 382 is already newer than 381 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:907: oid 46 version 382 is already newer than 379 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 382 (ObjNum 440 snap 104 seq_num 440) dirty exists 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:907: left oid 46 (ObjNum 440 snap 104 seq_num 440) 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:906: finishing write tid 4 to smithi06431743-8 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:906: finishing write tid 5 to smithi06431743-8 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:906: oid 8 updating version 0 to 155 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:906: oid 8 updating version 155 to 157 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:906: oid 8 version 157 is already newer than 154 2024-03-04T06:11:35.223 INFO:tasks.rados.rados.0.smithi064.stdout:906: oid 8 version 157 is already newer than 156 2024-03-04T06:11:35.224 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 157 (ObjNum 439 snap 104 seq_num 439) dirty exists 2024-03-04T06:11:35.224 INFO:tasks.rados.rados.0.smithi064.stdout:906: left oid 8 (ObjNum 439 snap 104 seq_num 439) 2024-03-04T06:11:35.281 INFO:tasks.rados.rados.0.smithi064.stdout:912: finishing write tid 1 to smithi06431743-28 2024-03-04T06:11:35.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:35.288+0000 7fb0b322d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:35.292 INFO:tasks.rados.rados.0.smithi064.stdout:913: expect (ObjNum 433 snap 101 seq_num 433) 2024-03-04T06:11:35.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:35.293+0000 7fb0b322d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:35.419 INFO:tasks.rados.rados.0.smithi064.stdout:912: finishing write tid 2 to smithi06431743-28 2024-03-04T06:11:35.419 INFO:tasks.rados.rados.0.smithi064.stdout:912: finishing write tid 3 to smithi06431743-28 2024-03-04T06:11:35.419 INFO:tasks.rados.rados.0.smithi064.stdout:912: finishing write tid 4 to smithi06431743-28 2024-03-04T06:11:35.419 INFO:tasks.rados.rados.0.smithi064.stdout:912: finishing write tid 5 to smithi06431743-28 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:912: oid 28 updating version 0 to 385 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:912: oid 28 updating version 385 to 386 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:912: oid 28 version 386 is already newer than 383 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:912: oid 28 version 386 is already newer than 384 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 386 (ObjNum 441 snap 104 seq_num 441) dirty exists 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:912: left oid 28 (ObjNum 441 snap 104 seq_num 441) 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:906: done (5 left) 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:907: done (4 left) 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:910: done (3 left) 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:912: done (2 left) 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:913: done (1 left) 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:914: done (0 left) 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:915: read oid 22 snap -1 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:915: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-04T06:11:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:916: write oid 21 current snap is 104 2024-03-04T06:11:35.421 INFO:tasks.rados.rados.0.smithi064.stdout:916: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2024-03-04T06:11:35.425 INFO:tasks.rados.rados.0.smithi064.stdout:916: writing smithi06431743-21 from 645501 to 1129282 tid 1 2024-03-04T06:11:35.432 INFO:tasks.rados.rados.0.smithi064.stdout:916: writing smithi06431743-21 from 1573523 to 2033676 tid 2 2024-03-04T06:11:35.439 INFO:tasks.rados.rados.0.smithi064.stdout:916: writing smithi06431743-21 from 2548974 to 3312358 tid 3 2024-03-04T06:11:35.440 INFO:tasks.rados.rados.0.smithi064.stdout:917: read oid 1 snap -1 2024-03-04T06:11:35.440 INFO:tasks.rados.rados.0.smithi064.stdout:917: expect (ObjNum 341 snap 84 seq_num 341) 2024-03-04T06:11:35.440 INFO:tasks.rados.rados.0.smithi064.stdout:918: delete oid 31 current snap is 104 2024-03-04T06:11:35.452 INFO:tasks.rados.rados.0.smithi064.stdout:918: done (3 left) 2024-03-04T06:11:35.452 INFO:tasks.rados.rados.0.smithi064.stdout:919: read oid 28 snap -1 2024-03-04T06:11:35.452 INFO:tasks.rados.rados.0.smithi064.stdout:919: expect (ObjNum 441 snap 104 seq_num 441) 2024-03-04T06:11:35.452 INFO:tasks.rados.rados.0.smithi064.stdout:920: delete oid 27 current snap is 104 2024-03-04T06:11:35.455 INFO:tasks.rados.rados.0.smithi064.stdout:916: finishing write tid 1 to smithi06431743-21 2024-03-04T06:11:35.465 INFO:tasks.rados.rados.0.smithi064.stdout:920: done (4 left) 2024-03-04T06:11:35.465 INFO:tasks.rados.rados.0.smithi064.stdout:921: write (excl) oid 43 current snap is 104 2024-03-04T06:11:35.466 INFO:tasks.rados.rados.0.smithi064.stdout:921: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-03-04T06:11:35.473 INFO:tasks.rados.rados.0.smithi064.stdout:921: writing smithi06431743-43 from 695922 to 1405450 tid 1 2024-03-04T06:11:35.481 INFO:tasks.rados.rados.0.smithi064.stdout:921: writing smithi06431743-43 from 2032616 to 2640096 tid 2 2024-03-04T06:11:35.487 INFO:tasks.rados.rados.0.smithi064.stdout:921: writing smithi06431743-43 from 3198537 to 3495922 tid 3 2024-03-04T06:11:35.487 INFO:tasks.rados.rados.0.smithi064.stdout:922: write (excl) oid 29 current snap is 104 2024-03-04T06:11:35.487 INFO:tasks.rados.rados.0.smithi064.stdout:922: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-03-04T06:11:35.495 INFO:tasks.rados.rados.0.smithi064.stdout:922: writing smithi06431743-29 from 712729 to 1497506 tid 1 2024-03-04T06:11:35.502 INFO:tasks.rados.rados.0.smithi064.stdout:922: writing smithi06431743-29 from 2091098 to 2558589 tid 2 2024-03-04T06:11:35.507 INFO:tasks.rados.rados.0.smithi064.stdout:922: writing smithi06431743-29 from 3342313 to 3512729 tid 3 2024-03-04T06:11:35.507 INFO:tasks.rados.rados.0.smithi064.stdout:923: snap_create 2024-03-04T06:11:35.507 INFO:tasks.rados.rados.0.smithi064.stdout:915: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-04T06:11:35.631 INFO:tasks.rados.rados.0.smithi064.stdout:916: finishing write tid 2 to smithi06431743-21 2024-03-04T06:11:35.631 INFO:tasks.rados.rados.0.smithi064.stdout:917: expect (ObjNum 341 snap 84 seq_num 341) 2024-03-04T06:11:35.769 INFO:tasks.rados.rados.0.smithi064.stdout:916: finishing write tid 3 to smithi06431743-21 2024-03-04T06:11:35.769 INFO:tasks.rados.rados.0.smithi064.stdout:919: expect (ObjNum 441 snap 104 seq_num 441) 2024-03-04T06:11:35.939 INFO:tasks.rados.rados.0.smithi064.stdout:916: finishing write tid 4 to smithi06431743-21 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:916: finishing write tid 5 to smithi06431743-21 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:916: oid 21 updating version 0 to 158 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:916: oid 21 updating version 158 to 160 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:916: oid 21 version 160 is already newer than 159 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:916: oid 21 version 160 is already newer than 157 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 160 (ObjNum 443 snap 104 seq_num 443) dirty exists 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:916: left oid 21 (ObjNum 443 snap 104 seq_num 443) 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:922: finishing write tid 1 to smithi06431743-29 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:921: finishing write tid 1 to smithi06431743-43 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:922: finishing write tid 2 to smithi06431743-29 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:921: finishing write tid 2 to smithi06431743-43 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:922: finishing write tid 3 to smithi06431743-29 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:921: finishing write tid 3 to smithi06431743-43 2024-03-04T06:11:35.940 INFO:tasks.rados.rados.0.smithi064.stdout:922: finishing write tid 4 to smithi06431743-29 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:922: finishing write tid 5 to smithi06431743-29 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:922: oid 29 updating version 0 to 385 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:922: oid 29 updating version 385 to 387 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:922: oid 29 version 387 is already newer than 386 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:922: oid 29 updating version 387 to 388 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 388 (ObjNum 447 snap 104 seq_num 447) dirty exists 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:922: left oid 29 (ObjNum 447 snap 104 seq_num 447) 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:921: finishing write tid 4 to smithi06431743-43 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:921: finishing write tid 5 to smithi06431743-43 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:921: oid 43 updating version 0 to 155 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:921: oid 43 version 155 is already newer than 155 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:921: oid 43 version 155 is already newer than 153 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:921: oid 43 version 155 is already newer than 152 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 155 (ObjNum 446 snap 104 seq_num 446) dirty exists 2024-03-04T06:11:35.941 INFO:tasks.rados.rados.0.smithi064.stdout:921: left oid 43 (ObjNum 446 snap 104 seq_num 446) 2024-03-04T06:11:36.266 INFO:tasks.rados.rados.0.smithi064.stdout:915: done (6 left) 2024-03-04T06:11:36.266 INFO:tasks.rados.rados.0.smithi064.stdout:916: done (5 left) 2024-03-04T06:11:36.267 INFO:tasks.rados.rados.0.smithi064.stdout:917: done (4 left) 2024-03-04T06:11:36.267 INFO:tasks.rados.rados.0.smithi064.stdout:919: done (3 left) 2024-03-04T06:11:36.267 INFO:tasks.rados.rados.0.smithi064.stdout:921: done (2 left) 2024-03-04T06:11:36.267 INFO:tasks.rados.rados.0.smithi064.stdout:922: done (1 left) 2024-03-04T06:11:36.267 INFO:tasks.rados.rados.0.smithi064.stdout:923: done (0 left) 2024-03-04T06:11:36.267 INFO:tasks.rados.rados.0.smithi064.stdout:924: read oid 48 snap -1 2024-03-04T06:11:36.267 INFO:tasks.rados.rados.0.smithi064.stdout:924: expect deleted 2024-03-04T06:11:36.267 INFO:tasks.rados.rados.0.smithi064.stdout:925: delete oid 38 current snap is 105 2024-03-04T06:11:36.305 INFO:tasks.rados.rados.0.smithi064.stdout:924: done (1 left) 2024-03-04T06:11:36.305 INFO:tasks.rados.rados.0.smithi064.stdout:925: done (0 left) 2024-03-04T06:11:36.306 INFO:tasks.rados.rados.0.smithi064.stdout:926: write oid 33 current snap is 105 2024-03-04T06:11:36.306 INFO:tasks.rados.rados.0.smithi064.stdout:926: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2024-03-04T06:11:36.313 INFO:tasks.rados.rados.0.smithi064.stdout:926: writing smithi06431743-33 from 746343 to 1397971 tid 1 2024-03-04T06:11:36.320 INFO:tasks.rados.rados.0.smithi064.stdout:926: writing smithi06431743-33 from 1924415 to 2679222 tid 2 2024-03-04T06:11:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:926: writing smithi06431743-33 from 3397159 to 3546343 tid 3 2024-03-04T06:11:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:927: write oid 26 current snap is 105 2024-03-04T06:11:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:927: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2024-03-04T06:11:36.335 INFO:tasks.rados.rados.0.smithi064.stdout:927: writing smithi06431743-26 from 763150 to 1490027 tid 1 2024-03-04T06:11:36.341 INFO:tasks.rados.rados.0.smithi064.stdout:927: writing smithi06431743-26 from 1982897 to 2597715 tid 2 2024-03-04T06:11:36.347 INFO:tasks.rados.rados.0.smithi064.stdout:927: writing smithi06431743-26 from 3024582 to 3499576 tid 3 2024-03-04T06:11:36.347 INFO:tasks.rados.rados.0.smithi064.stdout:928: write (excl) oid 27 current snap is 105 2024-03-04T06:11:36.347 INFO:tasks.rados.rados.0.smithi064.stdout:928: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-03-04T06:11:36.355 INFO:tasks.rados.rados.0.smithi064.stdout:928: writing smithi06431743-27 from 779957 to 1182083 tid 1 2024-03-04T06:11:36.366 INFO:tasks.rados.rados.0.smithi064.stdout:928: writing smithi06431743-27 from 1925026 to 2683502 tid 2 2024-03-04T06:11:36.371 INFO:tasks.rados.rados.0.smithi064.stdout:928: writing smithi06431743-27 from 3335652 to 3579957 tid 3 2024-03-04T06:11:36.371 INFO:tasks.rados.rados.0.smithi064.stdout:929: write (excl) oid 19 current snap is 105 2024-03-04T06:11:36.372 INFO:tasks.rados.rados.0.smithi064.stdout:926: finishing write tid 1 to smithi06431743-33 2024-03-04T06:11:36.372 INFO:tasks.rados.rados.0.smithi064.stdout:929: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2024-03-04T06:11:36.378 INFO:tasks.rados.rados.0.smithi064.stdout:929: writing smithi06431743-19 from 796764 to 1274139 tid 1 2024-03-04T06:11:36.385 INFO:tasks.rados.rados.0.smithi064.stdout:929: writing smithi06431743-19 from 1983508 to 2601995 tid 2 2024-03-04T06:11:36.391 INFO:tasks.rados.rados.0.smithi064.stdout:929: writing smithi06431743-19 from 3363075 to 3596764 tid 3 2024-03-04T06:11:36.391 INFO:tasks.rados.rados.0.smithi064.stdout:930: delete oid 21 current snap is 105 2024-03-04T06:11:36.391 INFO:tasks.rados.rados.0.smithi064.stdout:926: finishing write tid 2 to smithi06431743-33 2024-03-04T06:11:36.391 INFO:tasks.rados.rados.0.smithi064.stdout:927: finishing write tid 1 to smithi06431743-26 2024-03-04T06:11:36.391 INFO:tasks.rados.rados.0.smithi064.stdout:928: finishing write tid 1 to smithi06431743-27 2024-03-04T06:11:36.400 INFO:tasks.rados.rados.0.smithi064.stdout:926: finishing write tid 3 to smithi06431743-33 2024-03-04T06:11:36.412 INFO:tasks.rados.rados.0.smithi064.stdout:927: finishing write tid 2 to smithi06431743-26 2024-03-04T06:11:36.413 INFO:tasks.rados.rados.0.smithi064.stdout:928: finishing write tid 2 to smithi06431743-27 2024-03-04T06:11:36.417 INFO:tasks.rados.rados.0.smithi064.stdout:928: finishing write tid 3 to smithi06431743-27 2024-03-04T06:11:36.424 INFO:tasks.rados.rados.0.smithi064.stdout:929: finishing write tid 1 to smithi06431743-19 2024-03-04T06:11:36.431 INFO:tasks.rados.rados.0.smithi064.stdout:926: finishing write tid 4 to smithi06431743-33 2024-03-04T06:11:36.436 INFO:tasks.rados.rados.0.smithi064.stdout:928: finishing write tid 4 to smithi06431743-27 2024-03-04T06:11:36.437 INFO:tasks.rados.rados.0.smithi064.stdout:926: finishing write tid 5 to smithi06431743-33 2024-03-04T06:11:36.437 INFO:tasks.rados.rados.0.smithi064.stdout:926: oid 33 updating version 0 to 368 2024-03-04T06:11:36.437 INFO:tasks.rados.rados.0.smithi064.stdout:926: oid 33 updating version 368 to 371 2024-03-04T06:11:36.437 INFO:tasks.rados.rados.0.smithi064.stdout:926: oid 33 version 371 is already newer than 369 2024-03-04T06:11:36.438 INFO:tasks.rados.rados.0.smithi064.stdout:926: oid 33 version 371 is already newer than 370 2024-03-04T06:11:36.438 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 371 (ObjNum 449 snap 105 seq_num 449) dirty exists 2024-03-04T06:11:36.438 INFO:tasks.rados.rados.0.smithi064.stdout:926: left oid 33 (ObjNum 449 snap 105 seq_num 449) 2024-03-04T06:11:36.439 INFO:tasks.rados.rados.0.smithi064.stdout:928: finishing write tid 5 to smithi06431743-27 2024-03-04T06:11:36.439 INFO:tasks.rados.rados.0.smithi064.stdout:928: oid 27 updating version 0 to 124 2024-03-04T06:11:36.439 INFO:tasks.rados.rados.0.smithi064.stdout:928: oid 27 updating version 124 to 126 2024-03-04T06:11:36.439 INFO:tasks.rados.rados.0.smithi064.stdout:928: oid 27 updating version 126 to 127 2024-03-04T06:11:36.439 INFO:tasks.rados.rados.0.smithi064.stdout:928: oid 27 version 127 is already newer than 126 2024-03-04T06:11:36.439 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 127 (ObjNum 451 snap 105 seq_num 451) dirty exists 2024-03-04T06:11:36.439 INFO:tasks.rados.rados.0.smithi064.stdout:928: left oid 27 (ObjNum 451 snap 105 seq_num 451) 2024-03-04T06:11:36.440 INFO:tasks.rados.rados.0.smithi064.stdout:927: finishing write tid 3 to smithi06431743-26 2024-03-04T06:11:36.447 INFO:tasks.rados.rados.0.smithi064.stdout:926: done (4 left) 2024-03-04T06:11:36.447 INFO:tasks.rados.rados.0.smithi064.stdout:928: done (3 left) 2024-03-04T06:11:36.448 INFO:tasks.rados.rados.0.smithi064.stdout:930: done (2 left) 2024-03-04T06:11:36.448 INFO:tasks.rados.rados.0.smithi064.stdout:931: delete oid 16 current snap is 105 2024-03-04T06:11:36.449 INFO:tasks.rados.rados.0.smithi064.stdout:929: finishing write tid 2 to smithi06431743-19 2024-03-04T06:11:36.450 INFO:tasks.rados.rados.0.smithi064.stdout:927: finishing write tid 4 to smithi06431743-26 2024-03-04T06:11:36.452 INFO:tasks.rados.rados.0.smithi064.stdout:927: finishing write tid 5 to smithi06431743-26 2024-03-04T06:11:36.452 INFO:tasks.rados.rados.0.smithi064.stdout:927: oid 26 updating version 0 to 391 2024-03-04T06:11:36.452 INFO:tasks.rados.rados.0.smithi064.stdout:927: oid 26 version 391 is already newer than 389 2024-03-04T06:11:36.452 INFO:tasks.rados.rados.0.smithi064.stdout:927: oid 26 updating version 391 to 392 2024-03-04T06:11:36.452 INFO:tasks.rados.rados.0.smithi064.stdout:927: oid 26 version 392 is already newer than 390 2024-03-04T06:11:36.452 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 392 (ObjNum 450 snap 105 seq_num 450) dirty exists 2024-03-04T06:11:36.452 INFO:tasks.rados.rados.0.smithi064.stdout:927: left oid 26 (ObjNum 450 snap 105 seq_num 450) 2024-03-04T06:11:36.457 INFO:tasks.rados.rados.0.smithi064.stdout:929: finishing write tid 3 to smithi06431743-19 2024-03-04T06:11:36.465 INFO:tasks.rados.rados.0.smithi064.stdout:927: done (2 left) 2024-03-04T06:11:36.465 INFO:tasks.rados.rados.0.smithi064.stdout:931: done (1 left) 2024-03-04T06:11:36.465 INFO:tasks.rados.rados.0.smithi064.stdout:932: snap_create 2024-03-04T06:11:36.468 INFO:tasks.rados.rados.0.smithi064.stdout:929: finishing write tid 4 to smithi06431743-19 2024-03-04T06:11:36.469 INFO:tasks.rados.rados.0.smithi064.stdout:929: finishing write tid 5 to smithi06431743-19 2024-03-04T06:11:36.469 INFO:tasks.rados.rados.0.smithi064.stdout:929: oid 19 updating version 0 to 391 2024-03-04T06:11:36.469 INFO:tasks.rados.rados.0.smithi064.stdout:929: oid 19 version 391 is already newer than 390 2024-03-04T06:11:36.469 INFO:tasks.rados.rados.0.smithi064.stdout:929: oid 19 updating version 391 to 393 2024-03-04T06:11:36.469 INFO:tasks.rados.rados.0.smithi064.stdout:929: oid 19 version 393 is already newer than 392 2024-03-04T06:11:36.469 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 393 (ObjNum 452 snap 105 seq_num 452) dirty exists 2024-03-04T06:11:36.469 INFO:tasks.rados.rados.0.smithi064.stdout:929: left oid 19 (ObjNum 452 snap 105 seq_num 452) 2024-03-04T06:11:37.270 INFO:tasks.rados.rados.0.smithi064.stdout:929: done (1 left) 2024-03-04T06:11:37.270 INFO:tasks.rados.rados.0.smithi064.stdout:932: done (0 left) 2024-03-04T06:11:37.270 INFO:tasks.rados.rados.0.smithi064.stdout:933: read oid 42 snap -1 2024-03-04T06:11:37.270 INFO:tasks.rados.rados.0.smithi064.stdout:933: expect (ObjNum 426 snap 101 seq_num 426) 2024-03-04T06:11:37.271 INFO:tasks.rados.rados.0.smithi064.stdout:934: read oid 13 snap 104 2024-03-04T06:11:37.271 INFO:tasks.rados.rados.0.smithi064.stdout:934: expect deleted 2024-03-04T06:11:37.271 INFO:tasks.rados.rados.0.smithi064.stdout:935: read oid 41 snap 105 2024-03-04T06:11:37.271 INFO:tasks.rados.rados.0.smithi064.stdout:935: expect (ObjNum 364 snap 91 seq_num 364) 2024-03-04T06:11:37.271 INFO:tasks.rados.rados.0.smithi064.stdout:936: write (excl) oid 48 current snap is 106 2024-03-04T06:11:37.271 INFO:tasks.rados.rados.0.smithi064.stdout:936: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-03-04T06:11:37.277 INFO:tasks.rados.rados.0.smithi064.stdout:936: writing smithi06431743-48 from 447185 to 1150307 tid 1 2024-03-04T06:11:37.284 INFO:tasks.rados.rados.0.smithi064.stdout:936: writing smithi06431743-48 from 1642601 to 2408415 tid 2 2024-03-04T06:11:37.290 INFO:tasks.rados.rados.0.smithi064.stdout:936: writing smithi06431743-48 from 2928991 to 3376698 tid 3 2024-03-04T06:11:37.290 INFO:tasks.rados.rados.0.smithi064.stdout:937: write (excl) oid 8 current snap is 106 2024-03-04T06:11:37.291 INFO:tasks.rados.rados.0.smithi064.stdout:937: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2024-03-04T06:11:37.297 INFO:tasks.rados.rados.0.smithi064.stdout:937: writing smithi06431743-8 from 463992 to 1242363 tid 1 2024-03-04T06:11:37.304 INFO:tasks.rados.rados.0.smithi064.stdout:937: writing smithi06431743-8 from 1701083 to 2326908 tid 2 2024-03-04T06:11:37.314 INFO:tasks.rados.rados.0.smithi064.stdout:937: writing smithi06431743-8 from 2956414 to 3615393 tid 3 2024-03-04T06:11:37.314 INFO:tasks.rados.rados.0.smithi064.stdout:938: snap_create 2024-03-04T06:11:37.323 INFO:tasks.rados.rados.0.smithi064.stdout:935: expect (ObjNum 364 snap 91 seq_num 364) 2024-03-04T06:11:37.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:37.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:37.473+0000 7f685786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:37.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:37.477+0000 7f685786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:37.486 INFO:tasks.rados.rados.0.smithi064.stdout:936: finishing write tid 1 to smithi06431743-48 2024-03-04T06:11:37.486 INFO:tasks.rados.rados.0.smithi064.stdout:937: finishing write tid 1 to smithi06431743-8 2024-03-04T06:11:37.486 INFO:tasks.rados.rados.0.smithi064.stdout:933: expect (ObjNum 426 snap 101 seq_num 426) 2024-03-04T06:11:37.605 INFO:tasks.rados.rados.0.smithi064.stdout:937: finishing write tid 2 to smithi06431743-8 2024-03-04T06:11:37.605 INFO:tasks.rados.rados.0.smithi064.stdout:936: finishing write tid 2 to smithi06431743-48 2024-03-04T06:11:37.605 INFO:tasks.rados.rados.0.smithi064.stdout:937: finishing write tid 3 to smithi06431743-8 2024-03-04T06:11:37.605 INFO:tasks.rados.rados.0.smithi064.stdout:936: finishing write tid 3 to smithi06431743-48 2024-03-04T06:11:37.605 INFO:tasks.rados.rados.0.smithi064.stdout:937: finishing write tid 4 to smithi06431743-8 2024-03-04T06:11:37.605 INFO:tasks.rados.rados.0.smithi064.stdout:937: finishing write tid 5 to smithi06431743-8 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:937: oid 8 updating version 0 to 166 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:937: oid 8 updating version 166 to 168 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:937: oid 8 version 168 is already newer than 167 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:937: oid 8 version 168 is already newer than 165 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 168 (ObjNum 456 snap 106 seq_num 456) dirty exists 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:937: left oid 8 (ObjNum 456 snap 106 seq_num 456) 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:936: finishing write tid 4 to smithi06431743-48 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:936: finishing write tid 5 to smithi06431743-48 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:936: oid 48 updating version 0 to 168 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:936: oid 48 version 168 is already newer than 167 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:936: oid 48 version 168 is already newer than 166 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:936: oid 48 version 168 is already newer than 165 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 168 (ObjNum 455 snap 106 seq_num 455) dirty exists 2024-03-04T06:11:37.606 INFO:tasks.rados.rados.0.smithi064.stdout:936: left oid 48 (ObjNum 455 snap 106 seq_num 455) 2024-03-04T06:11:38.272 INFO:tasks.rados.rados.0.smithi064.stdout:933: done (5 left) 2024-03-04T06:11:38.272 INFO:tasks.rados.rados.0.smithi064.stdout:934: done (4 left) 2024-03-04T06:11:38.272 INFO:tasks.rados.rados.0.smithi064.stdout:935: done (3 left) 2024-03-04T06:11:38.273 INFO:tasks.rados.rados.0.smithi064.stdout:936: done (2 left) 2024-03-04T06:11:38.273 INFO:tasks.rados.rados.0.smithi064.stdout:937: done (1 left) 2024-03-04T06:11:38.273 INFO:tasks.rados.rados.0.smithi064.stdout:938: done (0 left) 2024-03-04T06:11:38.273 INFO:tasks.rados.rados.0.smithi064.stdout:939: snap_create 2024-03-04T06:11:39.277 INFO:tasks.rados.rados.0.smithi064.stdout:939: done (0 left) 2024-03-04T06:11:39.277 INFO:tasks.rados.rados.0.smithi064.stdout:940: write (excl) oid 44 current snap is 108 2024-03-04T06:11:39.277 INFO:tasks.rados.rados.0.smithi064.stdout:940: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-03-04T06:11:39.282 INFO:tasks.rados.rados.0.smithi064.stdout:940: writing smithi06431743-44 from 480799 to 1050772 tid 1 2024-03-04T06:11:39.289 INFO:tasks.rados.rados.0.smithi064.stdout:940: writing smithi06431743-44 from 1475918 to 2245401 tid 2 2024-03-04T06:11:39.295 INFO:tasks.rados.rados.0.smithi064.stdout:940: writing smithi06431743-44 from 2700190 to 3286794 tid 3 2024-03-04T06:11:39.295 INFO:tasks.rados.rados.0.smithi064.stdout:941: read oid 11 snap 106 2024-03-04T06:11:39.295 INFO:tasks.rados.rados.0.smithi064.stdout:941: expect (ObjNum 406 snap 99 seq_num 406) 2024-03-04T06:11:39.296 INFO:tasks.rados.rados.0.smithi064.stdout:942: read oid 48 snap -1 2024-03-04T06:11:39.296 INFO:tasks.rados.rados.0.smithi064.stdout:942: expect (ObjNum 455 snap 106 seq_num 455) 2024-03-04T06:11:39.296 INFO:tasks.rados.rados.0.smithi064.stdout:943: snap_remove snap 104 2024-03-04T06:11:39.544 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:39.656 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:39.653+0000 7fa6c475d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:39.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:39.658+0000 7fa6c475d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:40.032 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:11:40.033 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:11:40.033 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:11:40.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:11:40.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:40.137+0000 7f30a506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:40.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:40.144+0000 7f30a506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:40.158 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:40.158 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:40.158 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:40.158 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:40.158 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:40.159 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:40.159 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:40.159 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:40.159 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:40.159 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:40.159 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:40.159 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:40.161 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:40.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:11:40.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:40.264+0000 7f9d0411d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:40.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:40.271+0000 7f9d0411d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:40.285 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:40.285 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:40.285 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:40.285 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:40.285 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:40.285 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:40.285 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:40.285 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:40.285 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:40.286 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:40.286 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:40.286 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:40.286 INFO:tasks.rados.rados.0.smithi064.stdout:943: done (3 left) 2024-03-04T06:11:40.286 INFO:tasks.rados.rados.0.smithi064.stdout:944: snap_remove snap 107 2024-03-04T06:11:40.288 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:40.289 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:40.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:11:40.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:40.395+0000 7f401f59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:40.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:40.402+0000 7f401f59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:40.431 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:40.431 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:40.431 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:40.431 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:40.432 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:40.432 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:40.432 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:40.432 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:40.432 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:40.432 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:40.432 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:40.432 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:40.434 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:40.434 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:40.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:11:40.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:40.535+0000 7fc4cc4dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:40.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:40.543+0000 7fc4cc4dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:40.557 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:40.557 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:40.557 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:40.557 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:40.557 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:40.557 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:40.557 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:40.557 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:40.558 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:40.558 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:40.558 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:40.558 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:40.560 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:41.290 INFO:tasks.rados.rados.0.smithi064.stdout:944: done (3 left) 2024-03-04T06:11:41.291 INFO:tasks.rados.rados.0.smithi064.stdout:945: write oid 19 current snap is 108 2024-03-04T06:11:41.291 INFO:tasks.rados.rados.0.smithi064.stdout:945: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-03-04T06:11:41.297 INFO:tasks.rados.rados.0.smithi064.stdout:945: writing smithi06431743-19 from 497606 to 1142828 tid 1 2024-03-04T06:11:41.305 INFO:tasks.rados.rados.0.smithi064.stdout:945: writing smithi06431743-19 from 1934400 to 2563894 tid 2 2024-03-04T06:11:41.312 INFO:tasks.rados.rados.0.smithi064.stdout:945: writing smithi06431743-19 from 3243966 to 3697606 tid 3 2024-03-04T06:11:41.313 INFO:tasks.rados.rados.0.smithi064.stdout:946: snap_remove snap 105 2024-03-04T06:11:41.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:41.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:41.845+0000 7f7b9af5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:41.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:41.849+0000 7f7b9af5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:42.291 INFO:tasks.rados.rados.0.smithi064.stdout:946: done (4 left) 2024-03-04T06:11:42.291 INFO:tasks.rados.rados.0.smithi064.stdout:947: snap_create 2024-03-04T06:11:42.291 INFO:tasks.rados.rados.0.smithi064.stdout:942: expect (ObjNum 455 snap 106 seq_num 455) 2024-03-04T06:11:42.441 INFO:tasks.rados.rados.0.smithi064.stdout:940: finishing write tid 1 to smithi06431743-44 2024-03-04T06:11:42.441 INFO:tasks.rados.rados.0.smithi064.stdout:941: expect (ObjNum 406 snap 99 seq_num 406) 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:940: finishing write tid 2 to smithi06431743-44 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:940: finishing write tid 3 to smithi06431743-44 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:940: finishing write tid 4 to smithi06431743-44 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:940: finishing write tid 5 to smithi06431743-44 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:940: oid 44 updating version 0 to 141 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:940: oid 44 version 141 is already newer than 140 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:940: oid 44 version 141 is already newer than 139 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:940: oid 44 updating version 141 to 142 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 142 (ObjNum 457 snap 108 seq_num 457) dirty exists 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:940: left oid 44 (ObjNum 457 snap 108 seq_num 457) 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:945: finishing write tid 1 to smithi06431743-19 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:945: finishing write tid 2 to smithi06431743-19 2024-03-04T06:11:42.586 INFO:tasks.rados.rados.0.smithi064.stdout:945: finishing write tid 3 to smithi06431743-19 2024-03-04T06:11:42.587 INFO:tasks.rados.rados.0.smithi064.stdout:945: finishing write tid 4 to smithi06431743-19 2024-03-04T06:11:42.587 INFO:tasks.rados.rados.0.smithi064.stdout:945: finishing write tid 5 to smithi06431743-19 2024-03-04T06:11:42.587 INFO:tasks.rados.rados.0.smithi064.stdout:945: oid 19 updating version 0 to 400 2024-03-04T06:11:42.587 INFO:tasks.rados.rados.0.smithi064.stdout:945: oid 19 version 400 is already newer than 399 2024-03-04T06:11:42.587 INFO:tasks.rados.rados.0.smithi064.stdout:945: oid 19 version 400 is already newer than 398 2024-03-04T06:11:42.587 INFO:tasks.rados.rados.0.smithi064.stdout:945: oid 19 version 400 is already newer than 398 2024-03-04T06:11:42.587 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 400 (ObjNum 458 snap 108 seq_num 458) dirty exists 2024-03-04T06:11:42.587 INFO:tasks.rados.rados.0.smithi064.stdout:945: left oid 19 (ObjNum 458 snap 108 seq_num 458) 2024-03-04T06:11:43.206 INFO:tasks.rados.rados.0.smithi064.stdout:940: done (4 left) 2024-03-04T06:11:43.206 INFO:tasks.rados.rados.0.smithi064.stdout:941: done (3 left) 2024-03-04T06:11:43.206 INFO:tasks.rados.rados.0.smithi064.stdout:942: done (2 left) 2024-03-04T06:11:43.206 INFO:tasks.rados.rados.0.smithi064.stdout:945: done (1 left) 2024-03-04T06:11:43.206 INFO:tasks.rados.rados.0.smithi064.stdout:947: done (0 left) 2024-03-04T06:11:43.206 INFO:tasks.rados.rados.0.smithi064.stdout:948: snap_remove snap 106 2024-03-04T06:11:43.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:44.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:44.037+0000 7fc909c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:44.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:44.042+0000 7fc909c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:44.204 INFO:tasks.rados.rados.0.smithi064.stdout:948: done (0 left) 2024-03-04T06:11:44.204 INFO:tasks.rados.rados.0.smithi064.stdout:949: delete oid 19 current snap is 109 2024-03-04T06:11:44.264 INFO:tasks.rados.rados.0.smithi064.stdout:949: done (0 left) 2024-03-04T06:11:44.264 INFO:tasks.rados.rados.0.smithi064.stdout:950: write oid 48 current snap is 109 2024-03-04T06:11:44.264 INFO:tasks.rados.rados.0.smithi064.stdout:950: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-03-04T06:11:44.271 INFO:tasks.rados.rados.0.smithi064.stdout:950: writing smithi06431743-48 from 531220 to 1326940 tid 1 2024-03-04T06:11:44.277 INFO:tasks.rados.rados.0.smithi064.stdout:950: writing smithi06431743-48 from 1935011 to 2451821 tid 2 2024-03-04T06:11:44.283 INFO:tasks.rados.rados.0.smithi064.stdout:950: writing smithi06431743-48 from 3066106 to 3486526 tid 3 2024-03-04T06:11:44.283 INFO:tasks.rados.rados.0.smithi064.stdout:951: delete oid 50 current snap is 109 2024-03-04T06:11:44.323 INFO:tasks.rados.rados.0.smithi064.stdout:951: done (1 left) 2024-03-04T06:11:44.323 INFO:tasks.rados.rados.0.smithi064.stdout:952: snap_remove snap 108 2024-03-04T06:11:45.310 INFO:tasks.rados.rados.0.smithi064.stdout:952: done (1 left) 2024-03-04T06:11:45.310 INFO:tasks.rados.rados.0.smithi064.stdout:953: write oid 47 current snap is 109 2024-03-04T06:11:45.310 INFO:tasks.rados.rados.0.smithi064.stdout:950: finishing write tid 1 to smithi06431743-48 2024-03-04T06:11:45.310 INFO:tasks.rados.rados.0.smithi064.stdout:953: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-03-04T06:11:45.316 INFO:tasks.rados.rados.0.smithi064.stdout:953: writing smithi06431743-47 from 564834 to 1111052 tid 1 2024-03-04T06:11:45.322 INFO:tasks.rados.rados.0.smithi064.stdout:953: writing smithi06431743-47 from 1651975 to 2172454 tid 2 2024-03-04T06:11:45.328 INFO:tasks.rados.rados.0.smithi064.stdout:953: writing smithi06431743-47 from 2720952 to 3280269 tid 3 2024-03-04T06:11:45.328 INFO:tasks.rados.rados.0.smithi064.stdout:954: write (excl) oid 38 current snap is 109 2024-03-04T06:11:45.328 INFO:tasks.rados.rados.0.smithi064.stdout:950: finishing write tid 2 to smithi06431743-48 2024-03-04T06:11:45.328 INFO:tasks.rados.rados.0.smithi064.stdout:950: finishing write tid 3 to smithi06431743-48 2024-03-04T06:11:45.329 INFO:tasks.rados.rados.0.smithi064.stdout:950: finishing write tid 4 to smithi06431743-48 2024-03-04T06:11:45.329 INFO:tasks.rados.rados.0.smithi064.stdout:950: finishing write tid 5 to smithi06431743-48 2024-03-04T06:11:45.329 INFO:tasks.rados.rados.0.smithi064.stdout:950: oid 48 updating version 0 to 170 2024-03-04T06:11:45.329 INFO:tasks.rados.rados.0.smithi064.stdout:950: oid 48 updating version 170 to 173 2024-03-04T06:11:45.329 INFO:tasks.rados.rados.0.smithi064.stdout:950: oid 48 version 173 is already newer than 172 2024-03-04T06:11:45.329 INFO:tasks.rados.rados.0.smithi064.stdout:950: oid 48 version 173 is already newer than 172 2024-03-04T06:11:45.329 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 173 (ObjNum 460 snap 109 seq_num 460) dirty exists 2024-03-04T06:11:45.329 INFO:tasks.rados.rados.0.smithi064.stdout:950: left oid 48 (ObjNum 460 snap 109 seq_num 460) 2024-03-04T06:11:45.329 INFO:tasks.rados.rados.0.smithi064.stdout:954: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2024-03-04T06:11:45.336 INFO:tasks.rados.rados.0.smithi064.stdout:954: writing smithi06431743-38 from 581641 to 1203108 tid 1 2024-03-04T06:11:45.348 INFO:tasks.rados.rados.0.smithi064.stdout:954: writing smithi06431743-38 from 1994104 to 2774594 tid 2 2024-03-04T06:11:45.355 INFO:tasks.rados.rados.0.smithi064.stdout:954: writing smithi06431743-38 from 3432022 to 3781641 tid 3 2024-03-04T06:11:45.355 INFO:tasks.rados.rados.0.smithi064.stdout:950: done (2 left) 2024-03-04T06:11:45.356 INFO:tasks.rados.rados.0.smithi064.stdout:955: read oid 26 snap -1 2024-03-04T06:11:45.356 INFO:tasks.rados.rados.0.smithi064.stdout:955: expect (ObjNum 450 snap 105 seq_num 450) 2024-03-04T06:11:45.356 INFO:tasks.rados.rados.0.smithi064.stdout:956: read oid 15 snap -1 2024-03-04T06:11:45.356 INFO:tasks.rados.rados.0.smithi064.stdout:956: expect (ObjNum 429 snap 101 seq_num 429) 2024-03-04T06:11:45.356 INFO:tasks.rados.rados.0.smithi064.stdout:957: read oid 49 snap -1 2024-03-04T06:11:45.356 INFO:tasks.rados.rados.0.smithi064.stdout:957: expect deleted 2024-03-04T06:11:45.356 INFO:tasks.rados.rados.0.smithi064.stdout:958: delete oid 46 current snap is 109 2024-03-04T06:11:45.379 INFO:tasks.rados.rados.0.smithi064.stdout:953: finishing write tid 1 to smithi06431743-47 2024-03-04T06:11:45.382 INFO:tasks.rados.rados.0.smithi064.stdout:955: expect (ObjNum 450 snap 105 seq_num 450) 2024-03-04T06:11:45.561 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:11:45.561 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:11:45.561 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:11:45.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:11:45.601 INFO:tasks.rados.rados.0.smithi064.stdout:953: finishing write tid 2 to smithi06431743-47 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:955: done (5 left) 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:957: done (4 left) 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:958: done (3 left) 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:959: snap_create 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:954: finishing write tid 1 to smithi06431743-38 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:953: finishing write tid 3 to smithi06431743-47 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:953: finishing write tid 4 to smithi06431743-47 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:953: finishing write tid 5 to smithi06431743-47 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:953: oid 47 updating version 0 to 138 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:953: oid 47 version 138 is already newer than 137 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:953: oid 47 updating version 138 to 140 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:953: oid 47 version 140 is already newer than 140 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 140 (ObjNum 462 snap 109 seq_num 462) dirty exists 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:953: left oid 47 (ObjNum 462 snap 109 seq_num 462) 2024-03-04T06:11:45.602 INFO:tasks.rados.rados.0.smithi064.stdout:956: expect (ObjNum 429 snap 101 seq_num 429) 2024-03-04T06:11:45.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:45.665+0000 7fa60159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:45.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:45.672+0000 7fa60159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:45.685 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:45.685 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:45.685 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:45.685 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:45.685 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:45.685 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:45.686 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:45.686 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:45.686 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:45.686 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:45.686 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:45.686 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:45.689 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:45.689 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:11:45.746 INFO:tasks.rados.rados.0.smithi064.stdout:954: finishing write tid 2 to smithi06431743-38 2024-03-04T06:11:45.746 INFO:tasks.rados.rados.0.smithi064.stdout:954: finishing write tid 3 to smithi06431743-38 2024-03-04T06:11:45.746 INFO:tasks.rados.rados.0.smithi064.stdout:954: finishing write tid 4 to smithi06431743-38 2024-03-04T06:11:45.746 INFO:tasks.rados.rados.0.smithi064.stdout:954: finishing write tid 5 to smithi06431743-38 2024-03-04T06:11:45.747 INFO:tasks.rados.rados.0.smithi064.stdout:954: oid 38 updating version 0 to 146 2024-03-04T06:11:45.747 INFO:tasks.rados.rados.0.smithi064.stdout:954: oid 38 updating version 146 to 148 2024-03-04T06:11:45.747 INFO:tasks.rados.rados.0.smithi064.stdout:954: oid 38 version 148 is already newer than 147 2024-03-04T06:11:45.747 INFO:tasks.rados.rados.0.smithi064.stdout:954: oid 38 version 148 is already newer than 145 2024-03-04T06:11:45.747 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 148 (ObjNum 463 snap 109 seq_num 463) dirty exists 2024-03-04T06:11:45.747 INFO:tasks.rados.rados.0.smithi064.stdout:954: left oid 38 (ObjNum 463 snap 109 seq_num 463) 2024-03-04T06:11:45.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:45.833+0000 7fac9439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:45.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:45.841+0000 7fac9439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:45.853 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:45.853 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:45.853 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:45.853 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:45.853 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:45.853 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:45.854 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:45.856 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:45.856 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:45.857 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:11:46.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:46.012+0000 7fa9f1b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:46.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:46.019+0000 7fa9f1b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:46.032 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:46.032 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:46.032 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:46.032 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:46.032 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:46.032 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:46.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:46.033 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:46.033 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:46.033 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:46.033 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:46.033 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:46.035 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:46.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:11:46.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:46.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:46.139+0000 7f87ce75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:46.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:46.147+0000 7f87ce75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:46.160 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:46.161 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:46.161 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:46.161 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:46.161 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:46.163 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:46.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:46.218+0000 7f79edd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:46.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:46.223+0000 7f79edd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:46.311 INFO:tasks.rados.rados.0.smithi064.stdout:953: done (3 left) 2024-03-04T06:11:46.311 INFO:tasks.rados.rados.0.smithi064.stdout:954: done (2 left) 2024-03-04T06:11:46.311 INFO:tasks.rados.rados.0.smithi064.stdout:956: done (1 left) 2024-03-04T06:11:46.311 INFO:tasks.rados.rados.0.smithi064.stdout:959: done (0 left) 2024-03-04T06:11:46.311 INFO:tasks.rados.rados.0.smithi064.stdout:960: snap_create 2024-03-04T06:11:47.319 INFO:tasks.rados.rados.0.smithi064.stdout:960: done (0 left) 2024-03-04T06:11:47.319 INFO:tasks.rados.rados.0.smithi064.stdout:961: write (excl) oid 44 current snap is 111 2024-03-04T06:11:47.320 INFO:tasks.rados.rados.0.smithi064.stdout:961: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2024-03-04T06:11:47.325 INFO:tasks.rados.rados.0.smithi064.stdout:961: writing smithi06431743-44 from 615255 to 1103573 tid 1 2024-03-04T06:11:47.334 INFO:tasks.rados.rados.0.smithi064.stdout:961: writing smithi06431743-44 from 1827421 to 2611580 tid 2 2024-03-04T06:11:47.340 INFO:tasks.rados.rados.0.smithi064.stdout:961: writing smithi06431743-44 from 3203221 to 3815255 tid 3 2024-03-04T06:11:47.341 INFO:tasks.rados.rados.0.smithi064.stdout:962: delete oid 38 current snap is 111 2024-03-04T06:11:47.382 INFO:tasks.rados.rados.0.smithi064.stdout:961: finishing write tid 1 to smithi06431743-44 2024-03-04T06:11:47.395 INFO:tasks.rados.rados.0.smithi064.stdout:961: finishing write tid 2 to smithi06431743-44 2024-03-04T06:11:47.409 INFO:tasks.rados.rados.0.smithi064.stdout:961: finishing write tid 3 to smithi06431743-44 2024-03-04T06:11:47.412 INFO:tasks.rados.rados.0.smithi064.stdout:961: finishing write tid 4 to smithi06431743-44 2024-03-04T06:11:47.414 INFO:tasks.rados.rados.0.smithi064.stdout:961: finishing write tid 5 to smithi06431743-44 2024-03-04T06:11:47.414 INFO:tasks.rados.rados.0.smithi064.stdout:961: oid 44 updating version 0 to 153 2024-03-04T06:11:47.414 INFO:tasks.rados.rados.0.smithi064.stdout:961: oid 44 version 153 is already newer than 151 2024-03-04T06:11:47.414 INFO:tasks.rados.rados.0.smithi064.stdout:961: oid 44 version 153 is already newer than 153 2024-03-04T06:11:47.414 INFO:tasks.rados.rados.0.smithi064.stdout:961: oid 44 version 153 is already newer than 150 2024-03-04T06:11:47.414 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 153 (ObjNum 465 snap 111 seq_num 465) dirty exists 2024-03-04T06:11:47.414 INFO:tasks.rados.rados.0.smithi064.stdout:961: left oid 44 (ObjNum 465 snap 111 seq_num 465) 2024-03-04T06:11:47.427 INFO:tasks.rados.rados.0.smithi064.stdout:961: done (1 left) 2024-03-04T06:11:47.427 INFO:tasks.rados.rados.0.smithi064.stdout:962: done (0 left) 2024-03-04T06:11:47.427 INFO:tasks.rados.rados.0.smithi064.stdout:963: write (excl) oid 29 current snap is 111 2024-03-04T06:11:47.427 INFO:tasks.rados.rados.0.smithi064.stdout:963: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2024-03-04T06:11:47.434 INFO:tasks.rados.rados.0.smithi064.stdout:963: writing smithi06431743-29 from 648869 to 1287685 tid 1 2024-03-04T06:11:47.441 INFO:tasks.rados.rados.0.smithi064.stdout:963: writing smithi06431743-29 from 1828032 to 2615860 tid 2 2024-03-04T06:11:47.447 INFO:tasks.rados.rados.0.smithi064.stdout:963: writing smithi06431743-29 from 3141714 to 3673744 tid 3 2024-03-04T06:11:47.447 INFO:tasks.rados.rados.0.smithi064.stdout:964: delete oid 18 current snap is 111 2024-03-04T06:11:47.461 INFO:tasks.rados.rados.0.smithi064.stdout:964: done (1 left) 2024-03-04T06:11:47.462 INFO:tasks.rados.rados.0.smithi064.stdout:965: write oid 3 current snap is 111 2024-03-04T06:11:47.462 INFO:tasks.rados.rados.0.smithi064.stdout:965: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2024-03-04T06:11:47.469 INFO:tasks.rados.rados.0.smithi064.stdout:965: writing smithi06431743-3 from 682483 to 1471797 tid 1 2024-03-04T06:11:47.477 INFO:tasks.rados.rados.0.smithi064.stdout:965: writing smithi06431743-3 from 1944996 to 2736493 tid 2 2024-03-04T06:11:47.483 INFO:tasks.rados.rados.0.smithi064.stdout:965: writing smithi06431743-3 from 3196560 to 3751134 tid 3 2024-03-04T06:11:47.483 INFO:tasks.rados.rados.0.smithi064.stdout:966: read oid 9 snap -1 2024-03-04T06:11:47.484 INFO:tasks.rados.rados.0.smithi064.stdout:966: expect (ObjNum 418 snap 100 seq_num 418) 2024-03-04T06:11:47.484 INFO:tasks.rados.rados.0.smithi064.stdout:967: write (excl) oid 10 current snap is 111 2024-03-04T06:11:47.484 INFO:tasks.rados.rados.0.smithi064.stdout:963: finishing write tid 1 to smithi06431743-29 2024-03-04T06:11:47.484 INFO:tasks.rados.rados.0.smithi064.stdout:967: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2024-03-04T06:11:47.490 INFO:tasks.rados.rados.0.smithi064.stdout:967: writing smithi06431743-10 from 699290 to 1163853 tid 1 2024-03-04T06:11:47.497 INFO:tasks.rados.rados.0.smithi064.stdout:967: writing smithi06431743-10 from 1603478 to 2138633 tid 2 2024-03-04T06:11:47.508 INFO:tasks.rados.rados.0.smithi064.stdout:967: writing smithi06431743-10 from 2823983 to 3589829 tid 3 2024-03-04T06:11:47.508 INFO:tasks.rados.rados.0.smithi064.stdout:968: write oid 20 current snap is 111 2024-03-04T06:11:47.508 INFO:tasks.rados.rados.0.smithi064.stdout:968: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2024-03-04T06:11:47.515 INFO:tasks.rados.rados.0.smithi064.stdout:968: writing smithi06431743-20 from 716097 to 1255909 tid 1 2024-03-04T06:11:47.524 INFO:tasks.rados.rados.0.smithi064.stdout:968: writing smithi06431743-20 from 1945607 to 2740773 tid 2 2024-03-04T06:11:47.531 INFO:tasks.rados.rados.0.smithi064.stdout:968: writing smithi06431743-20 from 3535053 to 3916097 tid 3 2024-03-04T06:11:47.531 INFO:tasks.rados.rados.0.smithi064.stdout:969: snap_create 2024-03-04T06:11:47.532 INFO:tasks.rados.rados.0.smithi064.stdout:963: finishing write tid 2 to smithi06431743-29 2024-03-04T06:11:47.532 INFO:tasks.rados.rados.0.smithi064.stdout:963: finishing write tid 3 to smithi06431743-29 2024-03-04T06:11:47.532 INFO:tasks.rados.rados.0.smithi064.stdout:963: finishing write tid 4 to smithi06431743-29 2024-03-04T06:11:47.532 INFO:tasks.rados.rados.0.smithi064.stdout:963: finishing write tid 5 to smithi06431743-29 2024-03-04T06:11:47.532 INFO:tasks.rados.rados.0.smithi064.stdout:963: oid 29 updating version 0 to 413 2024-03-04T06:11:47.532 INFO:tasks.rados.rados.0.smithi064.stdout:963: oid 29 version 413 is already newer than 413 2024-03-04T06:11:47.532 INFO:tasks.rados.rados.0.smithi064.stdout:963: oid 29 version 413 is already newer than 410 2024-03-04T06:11:47.532 INFO:tasks.rados.rados.0.smithi064.stdout:963: oid 29 version 413 is already newer than 411 2024-03-04T06:11:47.532 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 413 (ObjNum 467 snap 111 seq_num 467) dirty exists 2024-03-04T06:11:47.532 INFO:tasks.rados.rados.0.smithi064.stdout:963: left oid 29 (ObjNum 467 snap 111 seq_num 467) 2024-03-04T06:11:47.532 INFO:tasks.rados.rados.0.smithi064.stdout:966: expect (ObjNum 418 snap 100 seq_num 418) 2024-03-04T06:11:47.861 INFO:tasks.rados.rados.0.smithi064.stdout:967: finishing write tid 1 to smithi06431743-10 2024-03-04T06:11:47.861 INFO:tasks.rados.rados.0.smithi064.stdout:965: finishing write tid 1 to smithi06431743-3 2024-03-04T06:11:47.861 INFO:tasks.rados.rados.0.smithi064.stdout:967: finishing write tid 2 to smithi06431743-10 2024-03-04T06:11:47.861 INFO:tasks.rados.rados.0.smithi064.stdout:967: finishing write tid 3 to smithi06431743-10 2024-03-04T06:11:47.861 INFO:tasks.rados.rados.0.smithi064.stdout:965: finishing write tid 2 to smithi06431743-3 2024-03-04T06:11:47.861 INFO:tasks.rados.rados.0.smithi064.stdout:965: finishing write tid 3 to smithi06431743-3 2024-03-04T06:11:47.861 INFO:tasks.rados.rados.0.smithi064.stdout:967: finishing write tid 4 to smithi06431743-10 2024-03-04T06:11:47.861 INFO:tasks.rados.rados.0.smithi064.stdout:967: finishing write tid 5 to smithi06431743-10 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:967: oid 10 updating version 0 to 159 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:967: oid 10 version 159 is already newer than 158 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:967: oid 10 version 159 is already newer than 157 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:967: oid 10 updating version 159 to 160 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 160 (ObjNum 470 snap 111 seq_num 470) dirty exists 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:967: left oid 10 (ObjNum 470 snap 111 seq_num 470) 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:968: finishing write tid 1 to smithi06431743-20 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:965: finishing write tid 4 to smithi06431743-3 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:965: finishing write tid 5 to smithi06431743-3 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:965: oid 3 updating version 0 to 160 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:965: oid 3 version 160 is already newer than 159 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:965: oid 3 version 160 is already newer than 159 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:965: oid 3 version 160 is already newer than 157 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 160 (ObjNum 469 snap 111 seq_num 469) dirty exists 2024-03-04T06:11:47.862 INFO:tasks.rados.rados.0.smithi064.stdout:965: left oid 3 (ObjNum 469 snap 111 seq_num 469) 2024-03-04T06:11:47.863 INFO:tasks.rados.rados.0.smithi064.stdout:968: finishing write tid 2 to smithi06431743-20 2024-03-04T06:11:47.863 INFO:tasks.rados.rados.0.smithi064.stdout:968: finishing write tid 3 to smithi06431743-20 2024-03-04T06:11:47.863 INFO:tasks.rados.rados.0.smithi064.stdout:968: finishing write tid 4 to smithi06431743-20 2024-03-04T06:11:47.863 INFO:tasks.rados.rados.0.smithi064.stdout:968: finishing write tid 5 to smithi06431743-20 2024-03-04T06:11:47.863 INFO:tasks.rados.rados.0.smithi064.stdout:968: oid 20 updating version 0 to 269 2024-03-04T06:11:47.863 INFO:tasks.rados.rados.0.smithi064.stdout:968: oid 20 version 269 is already newer than 268 2024-03-04T06:11:47.863 INFO:tasks.rados.rados.0.smithi064.stdout:968: oid 20 version 269 is already newer than 268 2024-03-04T06:11:47.863 INFO:tasks.rados.rados.0.smithi064.stdout:968: oid 20 version 269 is already newer than 266 2024-03-04T06:11:47.863 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 269 (ObjNum 471 snap 111 seq_num 471) dirty exists 2024-03-04T06:11:47.863 INFO:tasks.rados.rados.0.smithi064.stdout:968: left oid 20 (ObjNum 471 snap 111 seq_num 471) 2024-03-04T06:11:48.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:963: done (5 left) 2024-03-04T06:11:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:965: done (4 left) 2024-03-04T06:11:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:966: done (3 left) 2024-03-04T06:11:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:967: done (2 left) 2024-03-04T06:11:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:968: done (1 left) 2024-03-04T06:11:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:969: done (0 left) 2024-03-04T06:11:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:970: write oid 46 current snap is 112 2024-03-04T06:11:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:970: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2024-03-04T06:11:48.326 INFO:tasks.rados.rados.0.smithi064.stdout:970: writing smithi06431743-46 from 732904 to 1464318 tid 1 2024-03-04T06:11:48.333 INFO:tasks.rados.rados.0.smithi064.stdout:970: writing smithi06431743-46 from 2120442 to 2659266 tid 2 2024-03-04T06:11:48.341 INFO:tasks.rados.rados.0.smithi064.stdout:970: writing smithi06431743-46 from 3278829 to 3783572 tid 3 2024-03-04T06:11:48.341 INFO:tasks.rados.rados.0.smithi064.stdout:971: read oid 45 snap 110 2024-03-04T06:11:48.341 INFO:tasks.rados.rados.0.smithi064.stdout:971: expect deleted 2024-03-04T06:11:48.341 INFO:tasks.rados.rados.0.smithi064.stdout:972: snap_create 2024-03-04T06:11:48.398 INFO:tasks.rados.rados.0.smithi064.stdout:970: finishing write tid 1 to smithi06431743-46 2024-03-04T06:11:48.411 INFO:tasks.rados.rados.0.smithi064.stdout:970: finishing write tid 2 to smithi06431743-46 2024-03-04T06:11:48.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:48.409+0000 7efe75d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:48.413 INFO:tasks.rados.rados.0.smithi064.stdout:970: finishing write tid 3 to smithi06431743-46 2024-03-04T06:11:48.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:48.414+0000 7efe75d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:48.422 INFO:tasks.rados.rados.0.smithi064.stdout:970: finishing write tid 4 to smithi06431743-46 2024-03-04T06:11:48.423 INFO:tasks.rados.rados.0.smithi064.stdout:970: finishing write tid 5 to smithi06431743-46 2024-03-04T06:11:48.423 INFO:tasks.rados.rados.0.smithi064.stdout:970: oid 46 updating version 0 to 396 2024-03-04T06:11:48.423 INFO:tasks.rados.rados.0.smithi064.stdout:970: oid 46 updating version 396 to 398 2024-03-04T06:11:48.423 INFO:tasks.rados.rados.0.smithi064.stdout:970: oid 46 version 398 is already newer than 398 2024-03-04T06:11:48.423 INFO:tasks.rados.rados.0.smithi064.stdout:970: oid 46 updating version 398 to 399 2024-03-04T06:11:48.423 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 399 (ObjNum 472 snap 112 seq_num 472) dirty exists 2024-03-04T06:11:48.423 INFO:tasks.rados.rados.0.smithi064.stdout:970: left oid 46 (ObjNum 472 snap 112 seq_num 472) 2024-03-04T06:11:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:970: done (2 left) 2024-03-04T06:11:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:971: done (1 left) 2024-03-04T06:11:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:972: done (0 left) 2024-03-04T06:11:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:973: read oid 26 snap 110 2024-03-04T06:11:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:973: expect (ObjNum 450 snap 105 seq_num 450) 2024-03-04T06:11:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:974: snap_remove snap 112 2024-03-04T06:11:50.324 INFO:tasks.rados.rados.0.smithi064.stdout:974: done (1 left) 2024-03-04T06:11:50.324 INFO:tasks.rados.rados.0.smithi064.stdout:975: delete oid 28 current snap is 113 2024-03-04T06:11:50.324 INFO:tasks.rados.rados.0.smithi064.stdout:973: expect (ObjNum 450 snap 105 seq_num 450) 2024-03-04T06:11:50.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:50.546 INFO:tasks.rados.rados.0.smithi064.stdout:973: done (1 left) 2024-03-04T06:11:50.546 INFO:tasks.rados.rados.0.smithi064.stdout:975: done (0 left) 2024-03-04T06:11:50.546 INFO:tasks.rados.rados.0.smithi064.stdout:976: snap_remove snap 111 2024-03-04T06:11:50.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:50.584+0000 7fa171e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:50.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:50.589+0000 7fa171e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:51.163 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:11:51.164 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:11:51.164 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:11:51.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:51.269+0000 7fa612d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:51.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:51.275+0000 7fa612d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:51.328 INFO:tasks.rados.rados.0.smithi064.stdout:976: done (0 left) 2024-03-04T06:11:51.328 INFO:tasks.rados.rados.0.smithi064.stdout:977: delete oid 19 current snap is 113 2024-03-04T06:11:51.362 INFO:tasks.rados.rados.0.smithi064.stdout:977: done (0 left) 2024-03-04T06:11:51.362 INFO:tasks.rados.rados.0.smithi064.stdout:978: snap_remove snap 110 2024-03-04T06:11:51.473 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:11:51.473 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:11:51.485 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-03-04T06:11:51.326422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91635636,"num_objects":322,"num_object_clones":232,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":60,"num_read":1932,"num_read_kb":1352776,"num_write":1695,"num_write_kb":429605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3248,"ondisk_log_size":3248,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":319384,"kb_used_data":187128,"kb_used_omap":0,"kb_used_meta":132224,"kb_avail":377167976,"statfs":{"total":386547056640,"available":386220007424,"internally_reserved":0,"allocated":191619072,"data_stored":192213513,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":135397376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5966551,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":15,"num_read_kb":7258,"num_write":26,"num_write_kb":8782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010631"},"pg_stats":[{"pgid":"3.f","version":"226'155","reported_seq":780,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:50.319653+0000","last_change":"2024-03-04T06:11:45.350448+0000","last_active":"2024-03-04T06:11:50.319653+0000","last_peered":"2024-03-04T06:11:50.319653+0000","last_clean":"2024-03-04T06:11:50.319653+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:11:50.319653+0000","last_undegraded":"2024-03-04T06:11:50.319653+0000","last_fullsized":"2024-03-04T06:11:50.319653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6655704,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":58,"num_read_kb":24580,"num_write":90,"num_write_kb":24194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"238'128","reported_seq":786,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:49.335674+0000","last_change":"2024-03-04T06:11:45.903338+0000","last_active":"2024-03-04T06:11:49.335674+0000","last_peered":"2024-03-04T06:11:49.335674+0000","last_clean":"2024-03-04T06:11:49.335674+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:11:49.335674+0000","last_undegraded":"2024-03-04T06:11:49.335674+0000","last_fullsized":"2024-03-04T06:11:49.335674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6890969,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":58,"num_read_kb":40082,"num_write":60,"num_write_kb":15521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"226'219","reported_seq":819,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.367817+0000","last_change":"2024-03-04T06:11:45.378144+0000","last_active":"2024-03-04T06:11:45.367817+0000","last_peered":"2024-03-04T06:11:45.367817+0000","last_clean":"2024-03-04T06:11:45.367817+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:11:45.367817+0000","last_undegraded":"2024-03-04T06:11:45.367817+0000","last_fullsized":"2024-03-04T06:11:45.367817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563249,"num_objects":24,"num_object_clones":18,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":119,"num_read_kb":75543,"num_write":116,"num_write_kb":30968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"236'140","reported_seq":792,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:50.320102+0000","last_change":"2024-03-04T06:11:45.338789+0000","last_active":"2024-03-04T06:11:50.320102+0000","last_peered":"2024-03-04T06:11:50.320102+0000","last_clean":"2024-03-04T06:11:50.320102+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:11:50.320102+0000","last_undegraded":"2024-03-04T06:11:50.320102+0000","last_fullsized":"2024-03-04T06:11:50.320102+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6218722,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":92,"num_read_kb":73888,"num_write":83,"num_write_kb":23270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"226'91","reported_seq":722,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.330664+0000","last_change":"2024-03-04T06:11:45.337385+0000","last_active":"2024-03-04T06:11:45.330664+0000","last_peered":"2024-03-04T06:11:45.330664+0000","last_clean":"2024-03-04T06:11:45.330664+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:11:45.330664+0000","last_undegraded":"2024-03-04T06:11:45.330664+0000","last_fullsized":"2024-03-04T06:11:45.330664+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":17041,"num_write":34,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"238'160","reported_seq":801,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:50.320669+0000","last_change":"2024-03-04T06:11:45.349838+0000","last_active":"2024-03-04T06:11:50.320669+0000","last_peered":"2024-03-04T06:11:50.320669+0000","last_clean":"2024-03-04T06:11:50.320669+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:11:50.320669+0000","last_undegraded":"2024-03-04T06:11:50.320669+0000","last_fullsized":"2024-03-04T06:11:50.320669+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7714545,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":96,"num_read_kb":64481,"num_write":95,"num_write_kb":29013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":676,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:50.326988+0000","last_change":"2024-03-04T06:11:45.350286+0000","last_active":"2024-03-04T06:11:50.326988+0000","last_peered":"2024-03-04T06:11:50.326988+0000","last_clean":"2024-03-04T06:11:50.326988+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:11:50.326988+0000","last_undegraded":"2024-03-04T06:11:50.326988+0000","last_fullsized":"2024-03-04T06:11:50.326988+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"226'141","reported_seq":767,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.371613+0000","last_change":"2024-03-04T06:11:45.377902+0000","last_active":"2024-03-04T06:11:45.371613+0000","last_peered":"2024-03-04T06:11:45.371613+0000","last_clean":"2024-03-04T06:11:45.371613+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:11:45.371613+0000","last_undegraded":"2024-03-04T06:11:45.371613+0000","last_fullsized":"2024-03-04T06:11:45.371613+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151362,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":91,"num_read_kb":89569,"num_write":67,"num_write_kb":18683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":711,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.324626+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:11:45.324626+0000","last_peered":"2024-03-04T06:11:45.324626+0000","last_clean":"2024-03-04T06:11:45.324626+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:11:45.324626+0000","last_undegraded":"2024-03-04T06:11:45.324626+0000","last_fullsized":"2024-03-04T06:11:45.324626+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":566,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.377685+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:11:45.377685+0000","last_peered":"2024-03-04T06:11:45.377685+0000","last_clean":"2024-03-04T06:11:45.377685+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:11:45.377685+0000","last_undegraded":"2024-03-04T06:11:45.377685+0000","last_fullsized":"2024-03-04T06:11:45.377685+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"220'156","reported_seq":788,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.376433+0000","last_change":"2024-03-04T06:11:45.386249+0000","last_active":"2024-03-04T06:11:45.376433+0000","last_peered":"2024-03-04T06:11:45.376433+0000","last_clean":"2024-03-04T06:11:45.376433+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:11:45.376433+0000","last_undegraded":"2024-03-04T06:11:45.376433+0000","last_fullsized":"2024-03-04T06:11:45.376433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395179,"num_objects":16,"num_object_clones":10,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":125,"num_read_kb":109554,"num_write":70,"num_write_kb":17209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":576,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:50.321091+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:11:50.321091+0000","last_peered":"2024-03-04T06:11:50.321091+0000","last_clean":"2024-03-04T06:11:50.321091+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:11:50.321091+0000","last_undegraded":"2024-03-04T06:11:50.321091+0000","last_fullsized":"2024-03-04T06:11:50.321091+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"236'175","reported_seq":775,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.326051+0000","last_change":"2024-03-04T06:11:45.350422+0000","last_active":"2024-03-04T06:11:45.326051+0000","last_peered":"2024-03-04T06:11:45.326051+0000","last_clean":"2024-03-04T06:11:45.326051+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:11:45.326051+0000","last_undegraded":"2024-03-04T06:11:45.326051+0000","last_fullsized":"2024-03-04T06:11:45.326051+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731220,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":82,"num_read_kb":63041,"num_write":69,"num_write_kb":20140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":566,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.370794+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:11:45.370794+0000","last_peered":"2024-03-04T06:11:45.370794+0000","last_clean":"2024-03-04T06:11:45.370794+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:11:45.370794+0000","last_undegraded":"2024-03-04T06:11:45.370794+0000","last_fullsized":"2024-03-04T06:11:45.370794+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"231'394","reported_seq":990,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.376223+0000","last_change":"2024-03-04T06:11:45.338141+0000","last_active":"2024-03-04T06:11:45.376223+0000","last_peered":"2024-03-04T06:11:45.376223+0000","last_clean":"2024-03-04T06:11:45.376223+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:11:45.376223+0000","last_undegraded":"2024-03-04T06:11:45.376223+0000","last_fullsized":"2024-03-04T06:11:45.376223+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13933234,"num_objects":37,"num_object_clones":28,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":276,"num_read_kb":224531,"num_write":208,"num_write_kb":58841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":576,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:50.319074+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:11:50.319074+0000","last_peered":"2024-03-04T06:11:50.319074+0000","last_clean":"2024-03-04T06:11:50.319074+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:11:50.319074+0000","last_undegraded":"2024-03-04T06:11:50.319074+0000","last_fullsized":"2024-03-04T06:11:50.319074+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"238'413","reported_seq":953,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:49.342691+0000","last_change":"2024-03-04T06:11:45.941999+0000","last_active":"2024-03-04T06:11:49.342691+0000","last_peered":"2024-03-04T06:11:49.342691+0000","last_clean":"2024-03-04T06:11:49.342691+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:11:49.342691+0000","last_undegraded":"2024-03-04T06:11:49.342691+0000","last_fullsized":"2024-03-04T06:11:49.342691+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790160,"num_objects":40,"num_object_clones":30,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":162,"num_read_kb":107095,"num_write":199,"num_write_kb":53823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"232'164","reported_seq":814,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.416624+0000","last_change":"2024-03-04T06:11:45.388551+0000","last_active":"2024-03-04T06:11:45.416624+0000","last_peered":"2024-03-04T06:11:45.416624+0000","last_clean":"2024-03-04T06:11:45.416624+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:11:45.416624+0000","last_undegraded":"2024-03-04T06:11:45.416624+0000","last_fullsized":"2024-03-04T06:11:45.416624+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6504441,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":157,"num_read_kb":122168,"num_write":85,"num_write_kb":22565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":608,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:49.350530+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:11:49.350530+0000","last_peered":"2024-03-04T06:11:49.350530+0000","last_clean":"2024-03-04T06:11:49.350530+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:11:49.350530+0000","last_undegraded":"2024-03-04T06:11:49.350530+0000","last_fullsized":"2024-03-04T06:11:49.350530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"233'174","reported_seq":827,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:49.349166+0000","last_change":"2024-03-04T06:11:45.907333+0000","last_active":"2024-03-04T06:11:49.349166+0000","last_peered":"2024-03-04T06:11:49.349166+0000","last_clean":"2024-03-04T06:11:49.349166+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:11:49.349166+0000","last_undegraded":"2024-03-04T06:11:49.349166+0000","last_fullsized":"2024-03-04T06:11:49.349166+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3663992,"num_objects":21,"num_object_clones":13,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":96,"num_read_kb":49384,"num_write":88,"num_write_kb":23629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":578,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.332495+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:11:45.332495+0000","last_peered":"2024-03-04T06:11:45.332495+0000","last_clean":"2024-03-04T06:11:45.332495+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:11:45.332495+0000","last_undegraded":"2024-03-04T06:11:45.332495+0000","last_fullsized":"2024-03-04T06:11:45.332495+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"220'264","reported_seq":857,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.323460+0000","last_change":"2024-03-04T06:11:45.330239+0000","last_active":"2024-03-04T06:11:45.323460+0000","last_peered":"2024-03-04T06:11:45.323460+0000","last_clean":"2024-03-04T06:11:45.323460+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:11:45.323460+0000","last_undegraded":"2024-03-04T06:11:45.323460+0000","last_fullsized":"2024-03-04T06:11:45.323460+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6991811,"num_objects":26,"num_object_clones":19,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":147,"num_read_kb":105492,"num_write":128,"num_write_kb":35526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":608,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:49.338488+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:11:49.338488+0000","last_peered":"2024-03-04T06:11:49.338488+0000","last_clean":"2024-03-04T06:11:49.338488+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:11:49.338488+0000","last_undegraded":"2024-03-04T06:11:49.338488+0000","last_fullsized":"2024-03-04T06:11:49.338488+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"236'377","reported_seq":964,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-04T06:11:49.339306+0000","last_change":"2024-03-04T06:11:45.938501+0000","last_active":"2024-03-04T06:11:49.339306+0000","last_peered":"2024-03-04T06:11:49.339306+0000","last_clean":"2024-03-04T06:11:49.339306+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:11:49.339306+0000","last_undegraded":"2024-03-04T06:11:49.339306+0000","last_fullsized":"2024-03-04T06:11:49.339306+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8840680,"num_objects":33,"num_object_clones":29,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":239,"num_read_kb":186236,"num_write":166,"num_write_kb":45580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":566,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-03-04T06:11:45.377363+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:11:45.377363+0000","last_peered":"2024-03-04T06:11:45.377363+0000","last_clean":"2024-03-04T06:11:45.377363+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:11:45.377363+0000","last_undegraded":"2024-03-04T06:11:45.377363+0000","last_fullsized":"2024-03-04T06:11:45.377363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91045268,"num_objects":320,"num_object_clones":232,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":60,"num_read":1829,"num_read_kb":1352685,"num_write":1558,"num_write_kb":427711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3151,"ondisk_log_size":3151,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88436,"kb_used_data":53292,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94283404,"statfs":{"total":96636764160,"available":96546205696,"internally_reserved":0,"allocated":54571008,"data_stored":53481880,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77908,"kb_used_data":46924,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94293932,"statfs":{"total":96636764160,"available":96556986368,"internally_reserved":0,"allocated":48050176,"data_stored":46498316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":80752,"kb_used_data":45608,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94291088,"statfs":{"total":96636764160,"available":96554074112,"internally_reserved":0,"allocated":46702592,"data_stored":48113700,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72288,"kb_used_data":41304,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94299552,"statfs":{"total":96636764160,"available":96562741248,"internally_reserved":0,"allocated":42295296,"data_stored":44119617,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:11:52.329 INFO:tasks.rados.rados.0.smithi064.stdout:978: done (0 left) 2024-03-04T06:11:52.329 INFO:tasks.rados.rados.0.smithi064.stdout:979: snap_remove snap 109 2024-03-04T06:11:52.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:52.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:52.775+0000 7fbfbfb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:52.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:52.779+0000 7fbfbfb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:53.205 INFO:tasks.rados.rados.0.smithi064.stdout:979: done (0 left) 2024-03-04T06:11:53.205 INFO:tasks.rados.rados.0.smithi064.stdout:980: delete oid 46 current snap is 113 2024-03-04T06:11:53.273 INFO:tasks.rados.rados.0.smithi064.stdout:980: done (0 left) 2024-03-04T06:11:53.274 INFO:tasks.rados.rados.0.smithi064.stdout:981: write (excl) oid 37 current snap is 113 2024-03-04T06:11:53.274 INFO:tasks.rados.rados.0.smithi064.stdout:981: seq_num 476 ranges {400132=632410,1438017=546162} 2024-03-04T06:11:53.279 INFO:tasks.rados.rados.0.smithi064.stdout:981: writing smithi06431743-37 from 400132 to 1032542 tid 1 2024-03-04T06:11:53.285 INFO:tasks.rados.rados.0.smithi064.stdout:981: writing smithi06431743-37 from 1438017 to 1984179 tid 2 2024-03-04T06:11:53.285 INFO:tasks.rados.rados.0.smithi064.stdout:982: read oid 29 snap -1 2024-03-04T06:11:53.285 INFO:tasks.rados.rados.0.smithi064.stdout:982: expect (ObjNum 467 snap 111 seq_num 467) 2024-03-04T06:11:53.285 INFO:tasks.rados.rados.0.smithi064.stdout:983: read oid 10 snap -1 2024-03-04T06:11:53.285 INFO:tasks.rados.rados.0.smithi064.stdout:983: expect (ObjNum 470 snap 111 seq_num 470) 2024-03-04T06:11:53.285 INFO:tasks.rados.rados.0.smithi064.stdout:984: read oid 14 snap -1 2024-03-04T06:11:53.285 INFO:tasks.rados.rados.0.smithi064.stdout:984: expect deleted 2024-03-04T06:11:53.285 INFO:tasks.rados.rados.0.smithi064.stdout:985: write (excl) oid 34 current snap is 113 2024-03-04T06:11:53.285 INFO:tasks.rados.rados.0.smithi064.stdout:985: seq_num 477 ranges {416939=707659,1896499=120440} 2024-03-04T06:11:53.291 INFO:tasks.rados.rados.0.smithi064.stdout:985: writing smithi06431743-34 from 416939 to 1124598 tid 1 2024-03-04T06:11:53.296 INFO:tasks.rados.rados.0.smithi064.stdout:985: writing smithi06431743-34 from 1896499 to 2016939 tid 2 2024-03-04T06:11:53.296 INFO:tasks.rados.rados.0.smithi064.stdout:986: read oid 47 snap -1 2024-03-04T06:11:53.296 INFO:tasks.rados.rados.0.smithi064.stdout:986: expect (ObjNum 462 snap 109 seq_num 462) 2024-03-04T06:11:53.296 INFO:tasks.rados.rados.0.smithi064.stdout:987: write (excl) oid 20 current snap is 113 2024-03-04T06:11:53.296 INFO:tasks.rados.rados.0.smithi064.stdout:987: seq_num 478 ranges {433746=782908,1954981=78765} 2024-03-04T06:11:53.303 INFO:tasks.rados.rados.0.smithi064.stdout:987: writing smithi06431743-20 from 433746 to 1216654 tid 1 2024-03-04T06:11:53.308 INFO:tasks.rados.rados.0.smithi064.stdout:987: writing smithi06431743-20 from 1954981 to 2033746 tid 2 2024-03-04T06:11:53.308 INFO:tasks.rados.rados.0.smithi064.stdout:988: read oid 8 snap -1 2024-03-04T06:11:53.308 INFO:tasks.rados.rados.0.smithi064.stdout:988: expect (ObjNum 456 snap 106 seq_num 456) 2024-03-04T06:11:53.308 INFO:tasks.rados.rados.0.smithi064.stdout:989: write oid 45 current snap is 113 2024-03-04T06:11:53.308 INFO:tasks.rados.rados.0.smithi064.stdout:989: seq_num 479 ranges {450553=574510,1613463=437090} 2024-03-04T06:11:53.314 INFO:tasks.rados.rados.0.smithi064.stdout:989: writing smithi06431743-45 from 450553 to 1025063 tid 1 2024-03-04T06:11:53.320 INFO:tasks.rados.rados.0.smithi064.stdout:989: writing smithi06431743-45 from 1613463 to 2050553 tid 2 2024-03-04T06:11:53.320 INFO:tasks.rados.rados.0.smithi064.stdout:990: delete oid 21 current snap is 113 2024-03-04T06:11:53.342 INFO:tasks.rados.rados.0.smithi064.stdout:985: finishing write tid 1 to smithi06431743-34 2024-03-04T06:11:53.362 INFO:tasks.rados.rados.0.smithi064.stdout:986: expect (ObjNum 462 snap 109 seq_num 462) 2024-03-04T06:11:53.513 INFO:tasks.rados.rados.0.smithi064.stdout:988: expect (ObjNum 456 snap 106 seq_num 456) 2024-03-04T06:11:53.784 INFO:tasks.rados.rados.0.smithi064.stdout:987: finishing write tid 1 to smithi06431743-20 2024-03-04T06:11:53.784 INFO:tasks.rados.rados.0.smithi064.stdout:984: done (9 left) 2024-03-04T06:11:53.784 INFO:tasks.rados.rados.0.smithi064.stdout:986: done (8 left) 2024-03-04T06:11:53.784 INFO:tasks.rados.rados.0.smithi064.stdout:988: done (7 left) 2024-03-04T06:11:53.784 INFO:tasks.rados.rados.0.smithi064.stdout:990: done (6 left) 2024-03-04T06:11:53.784 INFO:tasks.rados.rados.0.smithi064.stdout:991: write oid 13 current snap is 113 2024-03-04T06:11:53.784 INFO:tasks.rados.rados.0.smithi064.stdout:981: finishing write tid 1 to smithi06431743-37 2024-03-04T06:11:53.784 INFO:tasks.rados.rados.0.smithi064.stdout:982: expect (ObjNum 467 snap 111 seq_num 467) 2024-03-04T06:11:54.026 INFO:tasks.rados.rados.0.smithi064.stdout:983: expect (ObjNum 470 snap 111 seq_num 470) 2024-03-04T06:11:54.280 INFO:tasks.rados.rados.0.smithi064.stdout:985: finishing write tid 2 to smithi06431743-34 2024-03-04T06:11:54.280 INFO:tasks.rados.rados.0.smithi064.stdout:987: finishing write tid 2 to smithi06431743-20 2024-03-04T06:11:54.280 INFO:tasks.rados.rados.0.smithi064.stdout:981: finishing write tid 2 to smithi06431743-37 2024-03-04T06:11:54.280 INFO:tasks.rados.rados.0.smithi064.stdout:985: finishing write tid 3 to smithi06431743-34 2024-03-04T06:11:54.281 INFO:tasks.rados.rados.0.smithi064.stdout:981: finishing write tid 3 to smithi06431743-37 2024-03-04T06:11:54.281 INFO:tasks.rados.rados.0.smithi064.stdout:991: seq_num 481 ranges {484167=725008,1730427=353740} 2024-03-04T06:11:54.287 INFO:tasks.rados.rados.0.smithi064.stdout:991: writing smithi06431743-13 from 484167 to 1209175 tid 1 2024-03-04T06:11:54.293 INFO:tasks.rados.rados.0.smithi064.stdout:991: writing smithi06431743-13 from 1730427 to 2084167 tid 2 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:982: done (6 left) 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:983: done (5 left) 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:992: delete oid 23 current snap is 113 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:985: finishing write tid 4 to smithi06431743-34 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:985: oid 34 updating version 0 to 407 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:985: oid 34 version 407 is already newer than 406 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:985: oid 34 version 407 is already newer than 405 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 407 (ObjNum 477 snap 113 seq_num 477) dirty exists 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:985: left oid 34 (ObjNum 477 snap 113 seq_num 477) 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:981: finishing write tid 4 to smithi06431743-37 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:981: oid 37 updating version 0 to 131 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:981: oid 37 version 131 is already newer than 129 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:981: oid 37 version 131 is already newer than 131 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 131 (ObjNum 476 snap 113 seq_num 476) dirty exists 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:981: left oid 37 (ObjNum 476 snap 113 seq_num 476) 2024-03-04T06:11:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:989: finishing write tid 1 to smithi06431743-45 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:987: finishing write tid 3 to smithi06431743-20 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:987: finishing write tid 4 to smithi06431743-20 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:987: oid 20 updating version 0 to 273 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:987: oid 20 version 273 is already newer than 272 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:987: oid 20 updating version 273 to 274 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 274 (ObjNum 478 snap 113 seq_num 478) dirty exists 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:987: left oid 20 (ObjNum 478 snap 113 seq_num 478) 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:989: finishing write tid 2 to smithi06431743-45 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:989: finishing write tid 3 to smithi06431743-45 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:989: finishing write tid 4 to smithi06431743-45 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:989: oid 45 updating version 0 to 221 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:989: oid 45 updating version 221 to 222 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:989: oid 45 version 222 is already newer than 220 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 222 (ObjNum 479 snap 113 seq_num 479) dirty exists 2024-03-04T06:11:54.295 INFO:tasks.rados.rados.0.smithi064.stdout:989: left oid 45 (ObjNum 479 snap 113 seq_num 479) 2024-03-04T06:11:54.305 INFO:tasks.rados.rados.0.smithi064.stdout:981: done (5 left) 2024-03-04T06:11:54.306 INFO:tasks.rados.rados.0.smithi064.stdout:985: done (4 left) 2024-03-04T06:11:54.306 INFO:tasks.rados.rados.0.smithi064.stdout:987: done (3 left) 2024-03-04T06:11:54.306 INFO:tasks.rados.rados.0.smithi064.stdout:989: done (2 left) 2024-03-04T06:11:54.306 INFO:tasks.rados.rados.0.smithi064.stdout:992: done (1 left) 2024-03-04T06:11:54.306 INFO:tasks.rados.rados.0.smithi064.stdout:993: snap_create 2024-03-04T06:11:54.310 INFO:tasks.rados.rados.0.smithi064.stdout:991: finishing write tid 1 to smithi06431743-13 2024-03-04T06:11:54.319 INFO:tasks.rados.rados.0.smithi064.stdout:991: finishing write tid 2 to smithi06431743-13 2024-03-04T06:11:54.329 INFO:tasks.rados.rados.0.smithi064.stdout:991: finishing write tid 3 to smithi06431743-13 2024-03-04T06:11:54.329 INFO:tasks.rados.rados.0.smithi064.stdout:991: finishing write tid 4 to smithi06431743-13 2024-03-04T06:11:54.329 INFO:tasks.rados.rados.0.smithi064.stdout:991: oid 13 updating version 0 to 419 2024-03-04T06:11:54.329 INFO:tasks.rados.rados.0.smithi064.stdout:991: oid 13 version 419 is already newer than 418 2024-03-04T06:11:54.329 INFO:tasks.rados.rados.0.smithi064.stdout:991: oid 13 version 419 is already newer than 417 2024-03-04T06:11:54.329 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 419 (ObjNum 481 snap 113 seq_num 481) dirty exists 2024-03-04T06:11:54.329 INFO:tasks.rados.rados.0.smithi064.stdout:991: left oid 13 (ObjNum 481 snap 113 seq_num 481) 2024-03-04T06:11:54.339 INFO:tasks.rados.rados.0.smithi064.stdout:991: done (1 left) 2024-03-04T06:11:54.339 INFO:tasks.rados.rados.0.smithi064.stdout:993: done (0 left) 2024-03-04T06:11:54.339 INFO:tasks.rados.rados.0.smithi064.stdout:994: write (excl) oid 41 current snap is 114 2024-03-04T06:11:54.339 INFO:tasks.rados.rados.0.smithi064.stdout:994: seq_num 483 ranges {517781=475506,1731038=386743} 2024-03-04T06:11:54.344 INFO:tasks.rados.rados.0.smithi064.stdout:994: writing smithi06431743-41 from 517781 to 993287 tid 1 2024-03-04T06:11:54.351 INFO:tasks.rados.rados.0.smithi064.stdout:994: writing smithi06431743-41 from 1731038 to 2117781 tid 2 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:995: read oid 17 snap -1 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:995: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:996: read oid 39 snap -1 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:996: expect deleted 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:997: read oid 18 snap -1 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:997: expect deleted 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:998: read oid 6 snap 113 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:998: expect (ObjNum 434 snap 101 seq_num 434) 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:999: read oid 30 snap 113 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:999: expect deleted 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:1000: read oid 1 snap -1 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:1000: expect (ObjNum 341 snap 84 seq_num 341) 2024-03-04T06:11:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:1001: read oid 23 snap -1 2024-03-04T06:11:54.353 INFO:tasks.rados.rados.0.smithi064.stdout:1001: expect deleted 2024-03-04T06:11:54.353 INFO:tasks.rados.rados.0.smithi064.stdout:1002: write (excl) oid 34 current snap is 114 2024-03-04T06:11:54.353 INFO:tasks.rados.rados.0.smithi064.stdout:1002: seq_num 484 ranges {534588=550755,1789520=345068} 2024-03-04T06:11:54.356 INFO:tasks.rados.rados.0.smithi064.stdout:1002: writing smithi06431743-34 from 534588 to 1085343 tid 1 2024-03-04T06:11:54.362 INFO:tasks.rados.rados.0.smithi064.stdout:1002: writing smithi06431743-34 from 1789520 to 2134588 tid 2 2024-03-04T06:11:54.363 INFO:tasks.rados.rados.0.smithi064.stdout:1003: write oid 45 current snap is 114 2024-03-04T06:11:54.363 INFO:tasks.rados.rados.0.smithi064.stdout:1003: seq_num 485 ranges {551395=626004,1848002=303393} 2024-03-04T06:11:54.369 INFO:tasks.rados.rados.0.smithi064.stdout:1003: writing smithi06431743-45 from 551395 to 1177399 tid 1 2024-03-04T06:11:54.376 INFO:tasks.rados.rados.0.smithi064.stdout:1003: writing smithi06431743-45 from 1848002 to 2151395 tid 2 2024-03-04T06:11:54.376 INFO:tasks.rados.rados.0.smithi064.stdout:1004: read oid 4 snap -1 2024-03-04T06:11:54.376 INFO:tasks.rados.rados.0.smithi064.stdout:1004: expect (ObjNum 413 snap 99 seq_num 413) 2024-03-04T06:11:54.376 INFO:tasks.rados.rados.0.smithi064.stdout:1005: read oid 44 snap -1 2024-03-04T06:11:54.376 INFO:tasks.rados.rados.0.smithi064.stdout:1005: expect (ObjNum 465 snap 111 seq_num 465) 2024-03-04T06:11:54.376 INFO:tasks.rados.rados.0.smithi064.stdout:1006: delete oid 36 current snap is 114 2024-03-04T06:11:54.395 INFO:tasks.rados.rados.0.smithi064.stdout:1000: expect (ObjNum 341 snap 84 seq_num 341) 2024-03-04T06:11:54.554 INFO:tasks.rados.rados.0.smithi064.stdout:1004: expect (ObjNum 413 snap 99 seq_num 413) 2024-03-04T06:11:54.665 INFO:tasks.rados.rados.0.smithi064.stdout:998: expect (ObjNum 434 snap 101 seq_num 434) 2024-03-04T06:11:54.786 INFO:tasks.rados.rados.0.smithi064.stdout:994: finishing write tid 1 to smithi06431743-41 2024-03-04T06:11:54.786 INFO:tasks.rados.rados.0.smithi064.stdout:1002: finishing write tid 1 to smithi06431743-34 2024-03-04T06:11:54.786 INFO:tasks.rados.rados.0.smithi064.stdout:994: finishing write tid 2 to smithi06431743-41 2024-03-04T06:11:54.786 INFO:tasks.rados.rados.0.smithi064.stdout:1003: finishing write tid 1 to smithi06431743-45 2024-03-04T06:11:54.786 INFO:tasks.rados.rados.0.smithi064.stdout:995: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-04T06:11:54.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:54.947 INFO:tasks.rados.rados.0.smithi064.stdout:994: finishing write tid 3 to smithi06431743-41 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:995: done (12 left) 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:996: done (11 left) 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:997: done (10 left) 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:998: done (9 left) 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:999: done (8 left) 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:1000: done (7 left) 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:1004: done (6 left) 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:1006: done (5 left) 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:1007: write (excl) oid 11 current snap is 114 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:994: finishing write tid 4 to smithi06431743-41 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:994: oid 41 updating version 0 to 163 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:994: oid 41 updating version 163 to 165 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:994: oid 41 version 165 is already newer than 164 2024-03-04T06:11:54.948 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 165 (ObjNum 483 snap 114 seq_num 483) dirty exists 2024-03-04T06:11:54.949 INFO:tasks.rados.rados.0.smithi064.stdout:994: left oid 41 (ObjNum 483 snap 114 seq_num 483) 2024-03-04T06:11:54.949 INFO:tasks.rados.rados.0.smithi064.stdout:1002: finishing write tid 2 to smithi06431743-34 2024-03-04T06:11:54.949 INFO:tasks.rados.rados.0.smithi064.stdout:1007: seq_num 487 ranges {585009=492855,1564966=620043} 2024-03-04T06:11:54.953 INFO:tasks.rados.rados.0.smithi064.stdout:1007: writing smithi06431743-11 from 585009 to 1077864 tid 1 2024-03-04T06:11:54.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:54.954+0000 7f83c999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:54.960 INFO:tasks.rados.rados.0.smithi064.stdout:1007: writing smithi06431743-11 from 1564966 to 2185009 tid 2 2024-03-04T06:11:54.960 INFO:tasks.rados.rados.0.smithi064.stdout:994: done (5 left) 2024-03-04T06:11:54.960 INFO:tasks.rados.rados.0.smithi064.stdout:1001: done (4 left) 2024-03-04T06:11:54.960 INFO:tasks.rados.rados.0.smithi064.stdout:1008: read oid 15 snap 113 2024-03-04T06:11:54.960 INFO:tasks.rados.rados.0.smithi064.stdout:1008: expect (ObjNum 429 snap 101 seq_num 429) 2024-03-04T06:11:54.960 INFO:tasks.rados.rados.0.smithi064.stdout:1009: delete oid 16 current snap is 114 2024-03-04T06:11:54.960 INFO:tasks.rados.rados.0.smithi064.stdout:1002: finishing write tid 3 to smithi06431743-34 2024-03-04T06:11:54.960 INFO:tasks.rados.rados.0.smithi064.stdout:1002: finishing write tid 4 to smithi06431743-34 2024-03-04T06:11:54.961 INFO:tasks.rados.rados.0.smithi064.stdout:1002: oid 34 updating version 0 to 410 2024-03-04T06:11:54.961 INFO:tasks.rados.rados.0.smithi064.stdout:1002: oid 34 version 410 is already newer than 409 2024-03-04T06:11:54.961 INFO:tasks.rados.rados.0.smithi064.stdout:1002: oid 34 updating version 410 to 411 2024-03-04T06:11:54.961 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 411 (ObjNum 484 snap 114 seq_num 484) dirty exists 2024-03-04T06:11:54.961 INFO:tasks.rados.rados.0.smithi064.stdout:1002: left oid 34 (ObjNum 484 snap 114 seq_num 484) 2024-03-04T06:11:54.961 INFO:tasks.rados.rados.0.smithi064.stdout:1005: expect (ObjNum 465 snap 111 seq_num 465) 2024-03-04T06:11:54.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:54.959+0000 7f83c999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:55.153 INFO:tasks.rados.rados.0.smithi064.stdout:1003: finishing write tid 2 to smithi06431743-45 2024-03-04T06:11:55.153 INFO:tasks.rados.rados.0.smithi064.stdout:1003: finishing write tid 3 to smithi06431743-45 2024-03-04T06:11:55.153 INFO:tasks.rados.rados.0.smithi064.stdout:1003: finishing write tid 4 to smithi06431743-45 2024-03-04T06:11:55.153 INFO:tasks.rados.rados.0.smithi064.stdout:1003: oid 45 updating version 0 to 224 2024-03-04T06:11:55.153 INFO:tasks.rados.rados.0.smithi064.stdout:1003: oid 45 updating version 224 to 226 2024-03-04T06:11:55.153 INFO:tasks.rados.rados.0.smithi064.stdout:1003: oid 45 version 226 is already newer than 226 2024-03-04T06:11:55.153 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 226 (ObjNum 485 snap 114 seq_num 485) dirty exists 2024-03-04T06:11:55.153 INFO:tasks.rados.rados.0.smithi064.stdout:1003: left oid 45 (ObjNum 485 snap 114 seq_num 485) 2024-03-04T06:11:55.153 INFO:tasks.rados.rados.0.smithi064.stdout:1002: done (5 left) 2024-03-04T06:11:55.153 INFO:tasks.rados.rados.0.smithi064.stdout:1003: done (4 left) 2024-03-04T06:11:55.153 INFO:tasks.rados.rados.0.smithi064.stdout:1005: done (3 left) 2024-03-04T06:11:55.154 INFO:tasks.rados.rados.0.smithi064.stdout:1009: done (2 left) 2024-03-04T06:11:55.154 INFO:tasks.rados.rados.0.smithi064.stdout:1010: write (excl) oid 47 current snap is 114 2024-03-04T06:11:55.154 INFO:tasks.rados.rados.0.smithi064.stdout:1007: finishing write tid 1 to smithi06431743-11 2024-03-04T06:11:55.154 INFO:tasks.rados.rados.0.smithi064.stdout:1010: seq_num 489 ranges {618623=643353,1681930=536693} 2024-03-04T06:11:55.159 INFO:tasks.rados.rados.0.smithi064.stdout:1010: writing smithi06431743-47 from 618623 to 1261976 tid 1 2024-03-04T06:11:55.165 INFO:tasks.rados.rados.0.smithi064.stdout:1010: writing smithi06431743-47 from 1681930 to 2218623 tid 2 2024-03-04T06:11:55.165 INFO:tasks.rados.rados.0.smithi064.stdout:1011: read oid 28 snap -1 2024-03-04T06:11:55.165 INFO:tasks.rados.rados.0.smithi064.stdout:1011: expect deleted 2024-03-04T06:11:55.165 INFO:tasks.rados.rados.0.smithi064.stdout:1012: write oid 30 current snap is 114 2024-03-04T06:11:55.165 INFO:tasks.rados.rados.0.smithi064.stdout:1012: seq_num 490 ranges {635430=718602,2140412=95018} 2024-03-04T06:11:55.172 INFO:tasks.rados.rados.0.smithi064.stdout:1012: writing smithi06431743-30 from 635430 to 1354032 tid 1 2024-03-04T06:11:55.177 INFO:tasks.rados.rados.0.smithi064.stdout:1012: writing smithi06431743-30 from 2140412 to 2235430 tid 2 2024-03-04T06:11:55.178 INFO:tasks.rados.rados.0.smithi064.stdout:1013: write oid 49 current snap is 114 2024-03-04T06:11:55.178 INFO:tasks.rados.rados.0.smithi064.stdout:1013: seq_num 491 ranges {652237=793851,2198894=53343} 2024-03-04T06:11:55.185 INFO:tasks.rados.rados.0.smithi064.stdout:1013: writing smithi06431743-49 from 652237 to 1446088 tid 1 2024-03-04T06:11:55.190 INFO:tasks.rados.rados.0.smithi064.stdout:1013: writing smithi06431743-49 from 2198894 to 2252237 tid 2 2024-03-04T06:11:55.190 INFO:tasks.rados.rados.0.smithi064.stdout:1014: write oid 23 current snap is 114 2024-03-04T06:11:55.190 INFO:tasks.rados.rados.0.smithi064.stdout:1014: seq_num 492 ranges {669044=469100,1741023=459161} 2024-03-04T06:11:55.195 INFO:tasks.rados.rados.0.smithi064.stdout:1014: writing smithi06431743-23 from 669044 to 1138144 tid 1 2024-03-04T06:11:55.202 INFO:tasks.rados.rados.0.smithi064.stdout:1014: writing smithi06431743-23 from 1741023 to 2200184 tid 2 2024-03-04T06:11:55.203 INFO:tasks.rados.rados.0.smithi064.stdout:1015: delete oid 26 current snap is 114 2024-03-04T06:11:55.203 INFO:tasks.rados.rados.0.smithi064.stdout:1008: expect (ObjNum 429 snap 101 seq_num 429) 2024-03-04T06:11:55.353 INFO:tasks.rados.rados.0.smithi064.stdout:1007: finishing write tid 2 to smithi06431743-11 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1007: finishing write tid 3 to smithi06431743-11 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1007: finishing write tid 4 to smithi06431743-11 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1007: oid 11 updating version 0 to 230 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1007: oid 11 version 230 is already newer than 229 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1007: oid 11 version 230 is already newer than 228 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 230 (ObjNum 487 snap 114 seq_num 487) dirty exists 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1007: left oid 11 (ObjNum 487 snap 114 seq_num 487) 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1007: done (7 left) 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1008: done (6 left) 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1015: done (5 left) 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1016: delete oid 15 current snap is 114 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1010: finishing write tid 1 to smithi06431743-47 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1012: finishing write tid 1 to smithi06431743-30 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1013: finishing write tid 1 to smithi06431743-49 2024-03-04T06:11:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:1010: finishing write tid 2 to smithi06431743-47 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1012: finishing write tid 2 to smithi06431743-30 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1014: finishing write tid 1 to smithi06431743-23 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1013: finishing write tid 2 to smithi06431743-49 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1012: finishing write tid 3 to smithi06431743-30 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1012: finishing write tid 4 to smithi06431743-30 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1012: oid 30 updating version 0 to 422 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1012: oid 30 version 422 is already newer than 421 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1012: oid 30 version 422 is already newer than 420 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 422 (ObjNum 490 snap 114 seq_num 490) dirty exists 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1012: left oid 30 (ObjNum 490 snap 114 seq_num 490) 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1010: finishing write tid 3 to smithi06431743-47 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1010: finishing write tid 4 to smithi06431743-47 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1010: oid 47 updating version 0 to 143 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1010: oid 47 updating version 143 to 144 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:1010: oid 47 version 144 is already newer than 142 2024-03-04T06:11:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 144 (ObjNum 489 snap 114 seq_num 489) dirty exists 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1010: left oid 47 (ObjNum 489 snap 114 seq_num 489) 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1013: finishing write tid 3 to smithi06431743-49 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1013: finishing write tid 4 to smithi06431743-49 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1013: oid 49 updating version 0 to 142 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1013: oid 49 updating version 142 to 144 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1013: oid 49 version 144 is already newer than 144 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 144 (ObjNum 491 snap 114 seq_num 491) dirty exists 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1013: left oid 49 (ObjNum 491 snap 114 seq_num 491) 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1014: finishing write tid 2 to smithi06431743-23 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1014: finishing write tid 3 to smithi06431743-23 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1014: finishing write tid 4 to smithi06431743-23 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1014: oid 23 updating version 0 to 168 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1014: oid 23 version 168 is already newer than 166 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:1014: oid 23 version 168 is already newer than 167 2024-03-04T06:11:55.356 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 168 (ObjNum 492 snap 114 seq_num 492) dirty exists 2024-03-04T06:11:55.357 INFO:tasks.rados.rados.0.smithi064.stdout:1014: left oid 23 (ObjNum 492 snap 114 seq_num 492) 2024-03-04T06:11:55.370 INFO:tasks.rados.rados.0.smithi064.stdout:1010: done (5 left) 2024-03-04T06:11:55.370 INFO:tasks.rados.rados.0.smithi064.stdout:1011: done (4 left) 2024-03-04T06:11:55.371 INFO:tasks.rados.rados.0.smithi064.stdout:1012: done (3 left) 2024-03-04T06:11:55.371 INFO:tasks.rados.rados.0.smithi064.stdout:1013: done (2 left) 2024-03-04T06:11:55.371 INFO:tasks.rados.rados.0.smithi064.stdout:1014: done (1 left) 2024-03-04T06:11:55.371 INFO:tasks.rados.rados.0.smithi064.stdout:1016: done (0 left) 2024-03-04T06:11:55.371 INFO:tasks.rados.rados.0.smithi064.stdout:1017: delete oid 46 current snap is 114 2024-03-04T06:11:55.375 INFO:tasks.rados.rados.0.smithi064.stdout:1017: done (0 left) 2024-03-04T06:11:55.375 INFO:tasks.rados.rados.0.smithi064.stdout:1018: delete oid 47 current snap is 114 2024-03-04T06:11:55.384 INFO:tasks.rados.rados.0.smithi064.stdout:1018: done (0 left) 2024-03-04T06:11:55.384 INFO:tasks.rados.rados.0.smithi064.stdout:1019: delete oid 44 current snap is 114 2024-03-04T06:11:55.403 INFO:tasks.rados.rados.0.smithi064.stdout:1019: done (0 left) 2024-03-04T06:11:55.403 INFO:tasks.rados.rados.0.smithi064.stdout:1020: write (excl) oid 14 current snap is 114 2024-03-04T06:11:55.403 INFO:tasks.rados.rados.0.smithi064.stdout:1020: seq_num 498 ranges {769886=636947,2091915=277971} 2024-03-04T06:11:55.411 INFO:tasks.rados.rados.0.smithi064.stdout:1020: writing smithi06431743-14 from 769886 to 1406833 tid 1 2024-03-04T06:11:55.416 INFO:tasks.rados.rados.0.smithi064.stdout:1020: writing smithi06431743-14 from 2091915 to 2369886 tid 2 2024-03-04T06:11:55.416 INFO:tasks.rados.rados.0.smithi064.stdout:1021: snap_create 2024-03-04T06:11:55.428 INFO:tasks.rados.rados.0.smithi064.stdout:1020: finishing write tid 1 to smithi06431743-14 2024-03-04T06:11:55.438 INFO:tasks.rados.rados.0.smithi064.stdout:1020: finishing write tid 2 to smithi06431743-14 2024-03-04T06:11:55.443 INFO:tasks.rados.rados.0.smithi064.stdout:1020: finishing write tid 3 to smithi06431743-14 2024-03-04T06:11:55.444 INFO:tasks.rados.rados.0.smithi064.stdout:1020: finishing write tid 4 to smithi06431743-14 2024-03-04T06:11:55.444 INFO:tasks.rados.rados.0.smithi064.stdout:1020: oid 14 updating version 0 to 275 2024-03-04T06:11:55.444 INFO:tasks.rados.rados.0.smithi064.stdout:1020: oid 14 updating version 275 to 276 2024-03-04T06:11:55.444 INFO:tasks.rados.rados.0.smithi064.stdout:1020: oid 14 updating version 276 to 277 2024-03-04T06:11:55.444 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 277 (ObjNum 498 snap 114 seq_num 498) dirty exists 2024-03-04T06:11:55.445 INFO:tasks.rados.rados.0.smithi064.stdout:1020: left oid 14 (ObjNum 498 snap 114 seq_num 498) 2024-03-04T06:11:56.344 INFO:tasks.rados.rados.0.smithi064.stdout:1020: done (1 left) 2024-03-04T06:11:56.344 INFO:tasks.rados.rados.0.smithi064.stdout:1021: done (0 left) 2024-03-04T06:11:56.344 INFO:tasks.rados.rados.0.smithi064.stdout:1022: delete oid 1 current snap is 115 2024-03-04T06:11:56.407 INFO:tasks.rados.rados.0.smithi064.stdout:1022: done (0 left) 2024-03-04T06:11:56.407 INFO:tasks.rados.rados.0.smithi064.stdout:1023: write oid 25 current snap is 115 2024-03-04T06:11:56.407 INFO:tasks.rados.rados.0.smithi064.stdout:1023: seq_num 500 ranges {403500=787445,1692526=473837} 2024-03-04T06:11:56.413 INFO:tasks.rados.rados.0.smithi064.stdout:1023: writing smithi06431743-25 from 403500 to 1190945 tid 1 2024-03-04T06:11:56.419 INFO:tasks.rados.rados.0.smithi064.stdout:1023: writing smithi06431743-25 from 1692526 to 2166363 tid 2 2024-03-04T06:11:56.419 INFO:tasks.rados.rados.0.smithi064.stdout:1024: write oid 37 current snap is 115 2024-03-04T06:11:56.419 INFO:tasks.rados.rados.0.smithi064.stdout:1024: seq_num 501 ranges {420307=462694,1351008=617495} 2024-03-04T06:11:56.424 INFO:tasks.rados.rados.0.smithi064.stdout:1024: writing smithi06431743-37 from 420307 to 883001 tid 1 2024-03-04T06:11:56.431 INFO:tasks.rados.rados.0.smithi064.stdout:1024: writing smithi06431743-37 from 1351008 to 1968503 tid 2 2024-03-04T06:11:56.431 INFO:tasks.rados.rados.0.smithi064.stdout:1025: snap_create 2024-03-04T06:11:56.456 INFO:tasks.rados.rados.0.smithi064.stdout:1024: finishing write tid 1 to smithi06431743-37 2024-03-04T06:11:56.473 INFO:tasks.rados.rados.0.smithi064.stdout:1024: finishing write tid 2 to smithi06431743-37 2024-03-04T06:11:56.475 INFO:tasks.rados.rados.0.smithi064.stdout:1023: finishing write tid 1 to smithi06431743-25 2024-03-04T06:11:56.484 INFO:tasks.rados.rados.0.smithi064.stdout:1024: finishing write tid 3 to smithi06431743-37 2024-03-04T06:11:56.485 INFO:tasks.rados.rados.0.smithi064.stdout:1024: finishing write tid 4 to smithi06431743-37 2024-03-04T06:11:56.485 INFO:tasks.rados.rados.0.smithi064.stdout:1024: oid 37 updating version 0 to 135 2024-03-04T06:11:56.485 INFO:tasks.rados.rados.0.smithi064.stdout:1024: oid 37 version 135 is already newer than 134 2024-03-04T06:11:56.485 INFO:tasks.rados.rados.0.smithi064.stdout:1024: oid 37 version 135 is already newer than 133 2024-03-04T06:11:56.485 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 135 (ObjNum 501 snap 115 seq_num 501) dirty exists 2024-03-04T06:11:56.485 INFO:tasks.rados.rados.0.smithi064.stdout:1024: left oid 37 (ObjNum 501 snap 115 seq_num 501) 2024-03-04T06:11:56.486 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:11:56.486 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:11:56.486 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:11:56.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:11:56.487 INFO:tasks.rados.rados.0.smithi064.stdout:1023: finishing write tid 2 to smithi06431743-25 2024-03-04T06:11:56.495 INFO:tasks.rados.rados.0.smithi064.stdout:1023: finishing write tid 3 to smithi06431743-25 2024-03-04T06:11:56.495 INFO:tasks.rados.rados.0.smithi064.stdout:1023: finishing write tid 4 to smithi06431743-25 2024-03-04T06:11:56.495 INFO:tasks.rados.rados.0.smithi064.stdout:1023: oid 25 updating version 0 to 233 2024-03-04T06:11:56.496 INFO:tasks.rados.rados.0.smithi064.stdout:1023: oid 25 version 233 is already newer than 232 2024-03-04T06:11:56.496 INFO:tasks.rados.rados.0.smithi064.stdout:1023: oid 25 updating version 233 to 234 2024-03-04T06:11:56.496 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 234 (ObjNum 500 snap 115 seq_num 500) dirty exists 2024-03-04T06:11:56.496 INFO:tasks.rados.rados.0.smithi064.stdout:1023: left oid 25 (ObjNum 500 snap 115 seq_num 500) 2024-03-04T06:11:56.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:56.591+0000 7f8c17b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:56.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:56.598+0000 7f8c17b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:56.611 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:56.614 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:56.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:11:56.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:56.758+0000 7f337fc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:56.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:56.765+0000 7f337fc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:56.778 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:56.781 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:56.781 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:11:56.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:56.884+0000 7fb49e82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:56.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:56.891+0000 7fb49e82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:56.904 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:56.908 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:56.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:11:57.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:57.011+0000 7f7c6cb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:57.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:57.018+0000 7f7c6cb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:57.032 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:11:57.032 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:11:57.032 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:11:57.032 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:11:57.032 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:11:57.033 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:11:57.033 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:11:57.033 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:11:57.033 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:11:57.033 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:11:57.033 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:11:57.033 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:11:57.034 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:11:57.035 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:11:57.053 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:11:57.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:57.156+0000 7f4f84b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:57.161 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:57.161+0000 7f4f84b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:57.345 INFO:tasks.rados.rados.0.smithi064.stdout:1023: done (2 left) 2024-03-04T06:11:57.345 INFO:tasks.rados.rados.0.smithi064.stdout:1024: done (1 left) 2024-03-04T06:11:57.345 INFO:tasks.rados.rados.0.smithi064.stdout:1025: done (0 left) 2024-03-04T06:11:57.346 INFO:tasks.rados.rados.0.smithi064.stdout:1026: write oid 23 current snap is 116 2024-03-04T06:11:57.346 INFO:tasks.rados.rados.0.smithi064.stdout:1026: seq_num 502 ranges {437114=654296,1525843=477506} 2024-03-04T06:11:57.351 INFO:tasks.rados.rados.0.smithi064.stdout:1026: writing smithi06431743-23 from 437114 to 1091410 tid 1 2024-03-04T06:11:57.356 INFO:tasks.rados.rados.0.smithi064.stdout:1026: writing smithi06431743-23 from 1525843 to 2003349 tid 2 2024-03-04T06:11:57.357 INFO:tasks.rados.rados.0.smithi064.stdout:1027: snap_create 2024-03-04T06:11:57.394 INFO:tasks.rados.rados.0.smithi064.stdout:1026: finishing write tid 1 to smithi06431743-23 2024-03-04T06:11:57.408 INFO:tasks.rados.rados.0.smithi064.stdout:1026: finishing write tid 2 to smithi06431743-23 2024-03-04T06:11:57.412 INFO:tasks.rados.rados.0.smithi064.stdout:1026: finishing write tid 3 to smithi06431743-23 2024-03-04T06:11:57.416 INFO:tasks.rados.rados.0.smithi064.stdout:1026: finishing write tid 4 to smithi06431743-23 2024-03-04T06:11:57.416 INFO:tasks.rados.rados.0.smithi064.stdout:1026: oid 23 updating version 0 to 171 2024-03-04T06:11:57.416 INFO:tasks.rados.rados.0.smithi064.stdout:1026: oid 23 version 171 is already newer than 170 2024-03-04T06:11:57.416 INFO:tasks.rados.rados.0.smithi064.stdout:1026: oid 23 updating version 171 to 172 2024-03-04T06:11:57.416 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 172 (ObjNum 502 snap 116 seq_num 502) dirty exists 2024-03-04T06:11:57.416 INFO:tasks.rados.rados.0.smithi064.stdout:1026: left oid 23 (ObjNum 502 snap 116 seq_num 502) 2024-03-04T06:11:58.206 INFO:tasks.rados.rados.0.smithi064.stdout:1026: done (1 left) 2024-03-04T06:11:58.207 INFO:tasks.rados.rados.0.smithi064.stdout:1027: done (0 left) 2024-03-04T06:11:58.207 INFO:tasks.rados.rados.0.smithi064.stdout:1028: read oid 17 snap -1 2024-03-04T06:11:58.207 INFO:tasks.rados.rados.0.smithi064.stdout:1028: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-04T06:11:58.207 INFO:tasks.rados.rados.0.smithi064.stdout:1029: read oid 50 snap -1 2024-03-04T06:11:58.207 INFO:tasks.rados.rados.0.smithi064.stdout:1029: expect deleted 2024-03-04T06:11:58.207 INFO:tasks.rados.rados.0.smithi064.stdout:1030: write oid 48 current snap is 117 2024-03-04T06:11:58.207 INFO:tasks.rados.rados.0.smithi064.stdout:1030: seq_num 503 ranges {453921=729545,1584325=621164} 2024-03-04T06:11:58.213 INFO:tasks.rados.rados.0.smithi064.stdout:1030: writing smithi06431743-48 from 453921 to 1183466 tid 1 2024-03-04T06:11:58.218 INFO:tasks.rados.rados.0.smithi064.stdout:1030: writing smithi06431743-48 from 1584325 to 2205489 tid 2 2024-03-04T06:11:58.219 INFO:tasks.rados.rados.0.smithi064.stdout:1031: write oid 30 current snap is 117 2024-03-04T06:11:58.219 INFO:tasks.rados.rados.0.smithi064.stdout:1031: seq_num 504 ranges {470728=404794,1526454=481175} 2024-03-04T06:11:58.223 INFO:tasks.rados.rados.0.smithi064.stdout:1031: writing smithi06431743-30 from 470728 to 875522 tid 1 2024-03-04T06:11:58.229 INFO:tasks.rados.rados.0.smithi064.stdout:1031: writing smithi06431743-30 from 1526454 to 2007629 tid 2 2024-03-04T06:11:58.229 INFO:tasks.rados.rados.0.smithi064.stdout:1032: snap_remove snap 113 2024-03-04T06:11:59.208 INFO:tasks.rados.rados.0.smithi064.stdout:1032: done (4 left) 2024-03-04T06:11:59.208 INFO:tasks.rados.rados.0.smithi064.stdout:1033: read oid 2 snap -1 2024-03-04T06:11:59.208 INFO:tasks.rados.rados.0.smithi064.stdout:1033: expect deleted 2024-03-04T06:11:59.208 INFO:tasks.rados.rados.0.smithi064.stdout:1034: read oid 29 snap -1 2024-03-04T06:11:59.208 INFO:tasks.rados.rados.0.smithi064.stdout:1034: expect (ObjNum 467 snap 111 seq_num 467) 2024-03-04T06:11:59.208 INFO:tasks.rados.rados.0.smithi064.stdout:1035: read oid 32 snap 116 2024-03-04T06:11:59.208 INFO:tasks.rados.rados.0.smithi064.stdout:1035: expect (ObjNum 430 snap 101 seq_num 430) 2024-03-04T06:11:59.209 INFO:tasks.rados.rados.0.smithi064.stdout:1036: delete oid 31 current snap is 117 2024-03-04T06:11:59.209 INFO:tasks.rados.rados.0.smithi064.stdout:1028: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-04T06:11:59.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:11:59.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:59.349+0000 7f0896f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:59.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:11:59.353+0000 7f0896f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1030: finishing write tid 1 to smithi06431743-48 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1030: finishing write tid 2 to smithi06431743-48 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1031: finishing write tid 1 to smithi06431743-30 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1030: finishing write tid 3 to smithi06431743-48 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1030: finishing write tid 4 to smithi06431743-48 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1030: oid 48 updating version 0 to 178 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1030: oid 48 updating version 178 to 179 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1030: oid 48 version 179 is already newer than 178 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 179 (ObjNum 503 snap 117 seq_num 503) dirty exists 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1030: left oid 48 (ObjNum 503 snap 117 seq_num 503) 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1031: finishing write tid 2 to smithi06431743-30 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1031: finishing write tid 3 to smithi06431743-30 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1031: finishing write tid 4 to smithi06431743-30 2024-03-04T06:11:59.382 INFO:tasks.rados.rados.0.smithi064.stdout:1031: oid 30 updating version 0 to 426 2024-03-04T06:11:59.383 INFO:tasks.rados.rados.0.smithi064.stdout:1031: oid 30 version 426 is already newer than 425 2024-03-04T06:11:59.383 INFO:tasks.rados.rados.0.smithi064.stdout:1031: oid 30 version 426 is already newer than 424 2024-03-04T06:11:59.383 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 426 (ObjNum 504 snap 117 seq_num 504) dirty exists 2024-03-04T06:11:59.383 INFO:tasks.rados.rados.0.smithi064.stdout:1031: left oid 30 (ObjNum 504 snap 117 seq_num 504) 2024-03-04T06:11:59.383 INFO:tasks.rados.rados.0.smithi064.stdout:1028: done (7 left) 2024-03-04T06:11:59.383 INFO:tasks.rados.rados.0.smithi064.stdout:1029: done (6 left) 2024-03-04T06:11:59.383 INFO:tasks.rados.rados.0.smithi064.stdout:1030: done (5 left) 2024-03-04T06:11:59.383 INFO:tasks.rados.rados.0.smithi064.stdout:1031: done (4 left) 2024-03-04T06:11:59.383 INFO:tasks.rados.rados.0.smithi064.stdout:1036: done (3 left) 2024-03-04T06:11:59.383 INFO:tasks.rados.rados.0.smithi064.stdout:1037: write oid 16 current snap is 117 2024-03-04T06:11:59.383 INFO:tasks.rados.rados.0.smithi064.stdout:1037: seq_num 506 ranges {504342=555292,1643418=484844} 2024-03-04T06:11:59.389 INFO:tasks.rados.rados.0.smithi064.stdout:1037: writing smithi06431743-16 from 504342 to 1059634 tid 1 2024-03-04T06:11:59.396 INFO:tasks.rados.rados.0.smithi064.stdout:1037: writing smithi06431743-16 from 1643418 to 2128262 tid 2 2024-03-04T06:11:59.396 INFO:tasks.rados.rados.0.smithi064.stdout:1038: read oid 13 snap -1 2024-03-04T06:11:59.397 INFO:tasks.rados.rados.0.smithi064.stdout:1038: expect (ObjNum 481 snap 113 seq_num 481) 2024-03-04T06:11:59.397 INFO:tasks.rados.rados.0.smithi064.stdout:1033: done (4 left) 2024-03-04T06:11:59.397 INFO:tasks.rados.rados.0.smithi064.stdout:1039: snap_remove snap 114 2024-03-04T06:12:00.346 INFO:tasks.rados.rados.0.smithi064.stdout:1039: done (4 left) 2024-03-04T06:12:00.346 INFO:tasks.rados.rados.0.smithi064.stdout:1040: write oid 43 current snap is 117 2024-03-04T06:12:00.346 INFO:tasks.rados.rados.0.smithi064.stdout:1034: expect (ObjNum 467 snap 111 seq_num 467) 2024-03-04T06:12:00.596 INFO:tasks.rados.rados.0.smithi064.stdout:1035: expect (ObjNum 430 snap 101 seq_num 430) 2024-03-04T06:12:00.830 INFO:tasks.rados.rados.0.smithi064.stdout:1037: finishing write tid 1 to smithi06431743-16 2024-03-04T06:12:00.830 INFO:tasks.rados.rados.0.smithi064.stdout:1038: expect (ObjNum 481 snap 113 seq_num 481) 2024-03-04T06:12:00.907 INFO:tasks.rados.rados.0.smithi064.stdout:1037: finishing write tid 2 to smithi06431743-16 2024-03-04T06:12:00.907 INFO:tasks.rados.rados.0.smithi064.stdout:1037: finishing write tid 3 to smithi06431743-16 2024-03-04T06:12:00.907 INFO:tasks.rados.rados.0.smithi064.stdout:1037: finishing write tid 4 to smithi06431743-16 2024-03-04T06:12:00.907 INFO:tasks.rados.rados.0.smithi064.stdout:1037: oid 16 updating version 0 to 179 2024-03-04T06:12:00.907 INFO:tasks.rados.rados.0.smithi064.stdout:1037: oid 16 version 179 is already newer than 178 2024-03-04T06:12:00.908 INFO:tasks.rados.rados.0.smithi064.stdout:1037: oid 16 version 179 is already newer than 177 2024-03-04T06:12:00.908 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 179 (ObjNum 506 snap 117 seq_num 506) dirty exists 2024-03-04T06:12:00.908 INFO:tasks.rados.rados.0.smithi064.stdout:1037: left oid 16 (ObjNum 506 snap 117 seq_num 506) 2024-03-04T06:12:00.908 INFO:tasks.rados.rados.0.smithi064.stdout:1040: seq_num 507 ranges {521149=630541,1701900=628502} 2024-03-04T06:12:00.913 INFO:tasks.rados.rados.0.smithi064.stdout:1040: writing smithi06431743-43 from 521149 to 1151690 tid 1 2024-03-04T06:12:00.920 INFO:tasks.rados.rados.0.smithi064.stdout:1040: writing smithi06431743-43 from 1701900 to 2330402 tid 2 2024-03-04T06:12:00.920 INFO:tasks.rados.rados.0.smithi064.stdout:1034: done (4 left) 2024-03-04T06:12:00.920 INFO:tasks.rados.rados.0.smithi064.stdout:1035: done (3 left) 2024-03-04T06:12:00.920 INFO:tasks.rados.rados.0.smithi064.stdout:1037: done (2 left) 2024-03-04T06:12:00.920 INFO:tasks.rados.rados.0.smithi064.stdout:1038: done (1 left) 2024-03-04T06:12:00.920 INFO:tasks.rados.rados.0.smithi064.stdout:1041: delete oid 23 current snap is 117 2024-03-04T06:12:00.943 INFO:tasks.rados.rados.0.smithi064.stdout:1040: finishing write tid 1 to smithi06431743-43 2024-03-04T06:12:00.972 INFO:tasks.rados.rados.0.smithi064.stdout:1040: finishing write tid 2 to smithi06431743-43 2024-03-04T06:12:00.998 INFO:tasks.rados.rados.0.smithi064.stdout:1040: finishing write tid 3 to smithi06431743-43 2024-03-04T06:12:00.999 INFO:tasks.rados.rados.0.smithi064.stdout:1040: finishing write tid 4 to smithi06431743-43 2024-03-04T06:12:00.999 INFO:tasks.rados.rados.0.smithi064.stdout:1040: oid 43 updating version 0 to 159 2024-03-04T06:12:00.999 INFO:tasks.rados.rados.0.smithi064.stdout:1040: oid 43 version 159 is already newer than 158 2024-03-04T06:12:00.999 INFO:tasks.rados.rados.0.smithi064.stdout:1040: oid 43 version 159 is already newer than 157 2024-03-04T06:12:00.999 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 159 (ObjNum 507 snap 117 seq_num 507) dirty exists 2024-03-04T06:12:00.999 INFO:tasks.rados.rados.0.smithi064.stdout:1040: left oid 43 (ObjNum 507 snap 117 seq_num 507) 2024-03-04T06:12:01.004 INFO:tasks.rados.rados.0.smithi064.stdout:1040: done (1 left) 2024-03-04T06:12:01.004 INFO:tasks.rados.rados.0.smithi064.stdout:1041: done (0 left) 2024-03-04T06:12:01.004 INFO:tasks.rados.rados.0.smithi064.stdout:1042: write oid 47 current snap is 117 2024-03-04T06:12:01.004 INFO:tasks.rados.rados.0.smithi064.stdout:1042: seq_num 509 ranges {554763=781039,2102511=452252} 2024-03-04T06:12:01.011 INFO:tasks.rados.rados.0.smithi064.stdout:1042: writing smithi06431743-47 from 554763 to 1335802 tid 1 2024-03-04T06:12:01.018 INFO:tasks.rados.rados.0.smithi064.stdout:1042: writing smithi06431743-47 from 2102511 to 2554763 tid 2 2024-03-04T06:12:01.018 INFO:tasks.rados.rados.0.smithi064.stdout:1043: write (excl) oid 35 current snap is 117 2024-03-04T06:12:01.018 INFO:tasks.rados.rados.0.smithi064.stdout:1043: seq_num 510 ranges {571570=572641,1877346=694224} 2024-03-04T06:12:01.024 INFO:tasks.rados.rados.0.smithi064.stdout:1043: writing smithi06431743-35 from 571570 to 1144211 tid 1 2024-03-04T06:12:01.033 INFO:tasks.rados.rados.0.smithi064.stdout:1043: writing smithi06431743-35 from 1877346 to 2571570 tid 2 2024-03-04T06:12:01.033 INFO:tasks.rados.rados.0.smithi064.stdout:1044: read oid 2 snap 115 2024-03-04T06:12:01.033 INFO:tasks.rados.rados.0.smithi064.stdout:1044: expect deleted 2024-03-04T06:12:01.033 INFO:tasks.rados.rados.0.smithi064.stdout:1045: delete oid 8 current snap is 117 2024-03-04T06:12:01.033 INFO:tasks.rados.rados.0.smithi064.stdout:1042: finishing write tid 1 to smithi06431743-47 2024-03-04T06:12:01.057 INFO:tasks.rados.rados.0.smithi064.stdout:1042: finishing write tid 2 to smithi06431743-47 2024-03-04T06:12:01.075 INFO:tasks.rados.rados.0.smithi064.stdout:1042: finishing write tid 3 to smithi06431743-47 2024-03-04T06:12:01.077 INFO:tasks.rados.rados.0.smithi064.stdout:1042: finishing write tid 4 to smithi06431743-47 2024-03-04T06:12:01.077 INFO:tasks.rados.rados.0.smithi064.stdout:1042: oid 47 updating version 0 to 150 2024-03-04T06:12:01.077 INFO:tasks.rados.rados.0.smithi064.stdout:1042: oid 47 version 150 is already newer than 149 2024-03-04T06:12:01.077 INFO:tasks.rados.rados.0.smithi064.stdout:1042: oid 47 version 150 is already newer than 148 2024-03-04T06:12:01.077 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 150 (ObjNum 509 snap 117 seq_num 509) dirty exists 2024-03-04T06:12:01.077 INFO:tasks.rados.rados.0.smithi064.stdout:1042: left oid 47 (ObjNum 509 snap 117 seq_num 509) 2024-03-04T06:12:01.116 INFO:tasks.rados.rados.0.smithi064.stdout:1042: done (3 left) 2024-03-04T06:12:01.116 INFO:tasks.rados.rados.0.smithi064.stdout:1044: done (2 left) 2024-03-04T06:12:01.116 INFO:tasks.rados.rados.0.smithi064.stdout:1045: done (1 left) 2024-03-04T06:12:01.116 INFO:tasks.rados.rados.0.smithi064.stdout:1046: snap_create 2024-03-04T06:12:01.117 INFO:tasks.rados.rados.0.smithi064.stdout:1043: finishing write tid 1 to smithi06431743-35 2024-03-04T06:12:01.131 INFO:tasks.rados.rados.0.smithi064.stdout:1043: finishing write tid 2 to smithi06431743-35 2024-03-04T06:12:01.138 INFO:tasks.rados.rados.0.smithi064.stdout:1043: finishing write tid 3 to smithi06431743-35 2024-03-04T06:12:01.139 INFO:tasks.rados.rados.0.smithi064.stdout:1043: finishing write tid 4 to smithi06431743-35 2024-03-04T06:12:01.139 INFO:tasks.rados.rados.0.smithi064.stdout:1043: oid 35 updating version 0 to 280 2024-03-04T06:12:01.139 INFO:tasks.rados.rados.0.smithi064.stdout:1043: oid 35 updating version 280 to 281 2024-03-04T06:12:01.139 INFO:tasks.rados.rados.0.smithi064.stdout:1043: oid 35 version 281 is already newer than 279 2024-03-04T06:12:01.139 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 281 (ObjNum 510 snap 117 seq_num 510) dirty exists 2024-03-04T06:12:01.139 INFO:tasks.rados.rados.0.smithi064.stdout:1043: left oid 35 (ObjNum 510 snap 117 seq_num 510) 2024-03-04T06:12:01.349 INFO:tasks.rados.rados.0.smithi064.stdout:1043: done (1 left) 2024-03-04T06:12:01.349 INFO:tasks.rados.rados.0.smithi064.stdout:1046: done (0 left) 2024-03-04T06:12:01.349 INFO:tasks.rados.rados.0.smithi064.stdout:1047: read oid 48 snap -1 2024-03-04T06:12:01.349 INFO:tasks.rados.rados.0.smithi064.stdout:1047: expect (ObjNum 503 snap 117 seq_num 503) 2024-03-04T06:12:01.349 INFO:tasks.rados.rados.0.smithi064.stdout:1048: write (excl) oid 45 current snap is 118 2024-03-04T06:12:01.349 INFO:tasks.rados.rados.0.smithi064.stdout:1048: seq_num 512 ranges {605184=723139,1877957=727227} 2024-03-04T06:12:01.355 INFO:tasks.rados.rados.0.smithi064.stdout:1048: writing smithi06431743-45 from 605184 to 1328323 tid 1 2024-03-04T06:12:01.364 INFO:tasks.rados.rados.0.smithi064.stdout:1048: writing smithi06431743-45 from 1877957 to 2605184 tid 2 2024-03-04T06:12:01.364 INFO:tasks.rados.rados.0.smithi064.stdout:1049: delete oid 9 current snap is 118 2024-03-04T06:12:01.414 INFO:tasks.rados.rados.0.smithi064.stdout:1048: finishing write tid 1 to smithi06431743-45 2024-03-04T06:12:01.416 INFO:tasks.rados.rados.0.smithi064.stdout:1047: expect (ObjNum 503 snap 117 seq_num 503) 2024-03-04T06:12:01.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:01.550 INFO:tasks.rados.rados.0.smithi064.stdout:1048: finishing write tid 2 to smithi06431743-45 2024-03-04T06:12:01.550 INFO:tasks.rados.rados.0.smithi064.stdout:1048: finishing write tid 3 to smithi06431743-45 2024-03-04T06:12:01.550 INFO:tasks.rados.rados.0.smithi064.stdout:1048: finishing write tid 4 to smithi06431743-45 2024-03-04T06:12:01.550 INFO:tasks.rados.rados.0.smithi064.stdout:1048: oid 45 updating version 0 to 240 2024-03-04T06:12:01.550 INFO:tasks.rados.rados.0.smithi064.stdout:1048: oid 45 updating version 240 to 241 2024-03-04T06:12:01.550 INFO:tasks.rados.rados.0.smithi064.stdout:1048: oid 45 updating version 241 to 242 2024-03-04T06:12:01.551 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 242 (ObjNum 512 snap 118 seq_num 512) dirty exists 2024-03-04T06:12:01.551 INFO:tasks.rados.rados.0.smithi064.stdout:1048: left oid 45 (ObjNum 512 snap 118 seq_num 512) 2024-03-04T06:12:01.551 INFO:tasks.rados.rados.0.smithi064.stdout:1047: done (2 left) 2024-03-04T06:12:01.551 INFO:tasks.rados.rados.0.smithi064.stdout:1048: done (1 left) 2024-03-04T06:12:01.551 INFO:tasks.rados.rados.0.smithi064.stdout:1049: done (0 left) 2024-03-04T06:12:01.551 INFO:tasks.rados.rados.0.smithi064.stdout:1050: read oid 22 snap -1 2024-03-04T06:12:01.551 INFO:tasks.rados.rados.0.smithi064.stdout:1050: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-04T06:12:01.551 INFO:tasks.rados.rados.0.smithi064.stdout:1051: write oid 16 current snap is 118 2024-03-04T06:12:01.551 INFO:tasks.rados.rados.0.smithi064.stdout:1051: seq_num 514 ranges {638798=473637,1594921=783167} 2024-03-04T06:12:01.555 INFO:tasks.rados.rados.0.smithi064.stdout:1051: writing smithi06431743-16 from 638798 to 1112435 tid 1 2024-03-04T06:12:01.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:01.556+0000 7f7af9c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:01.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:01.560+0000 7f7af9c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:01.564 INFO:tasks.rados.rados.0.smithi064.stdout:1051: writing smithi06431743-16 from 1594921 to 2378088 tid 2 2024-03-04T06:12:01.564 INFO:tasks.rados.rados.0.smithi064.stdout:1052: read oid 7 snap 116 2024-03-04T06:12:01.564 INFO:tasks.rados.rados.0.smithi064.stdout:1052: expect deleted 2024-03-04T06:12:01.564 INFO:tasks.rados.rados.0.smithi064.stdout:1053: delete oid 10 current snap is 118 2024-03-04T06:12:01.564 INFO:tasks.rados.rados.0.smithi064.stdout:1050: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-04T06:12:01.697 INFO:tasks.rados.rados.0.smithi064.stdout:1051: finishing write tid 1 to smithi06431743-16 2024-03-04T06:12:01.698 INFO:tasks.rados.rados.0.smithi064.stdout:1051: finishing write tid 2 to smithi06431743-16 2024-03-04T06:12:01.698 INFO:tasks.rados.rados.0.smithi064.stdout:1051: finishing write tid 3 to smithi06431743-16 2024-03-04T06:12:01.698 INFO:tasks.rados.rados.0.smithi064.stdout:1051: finishing write tid 4 to smithi06431743-16 2024-03-04T06:12:01.698 INFO:tasks.rados.rados.0.smithi064.stdout:1051: oid 16 updating version 0 to 185 2024-03-04T06:12:01.698 INFO:tasks.rados.rados.0.smithi064.stdout:1051: oid 16 version 185 is already newer than 183 2024-03-04T06:12:01.698 INFO:tasks.rados.rados.0.smithi064.stdout:1051: oid 16 version 185 is already newer than 184 2024-03-04T06:12:01.698 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 185 (ObjNum 514 snap 118 seq_num 514) dirty exists 2024-03-04T06:12:01.698 INFO:tasks.rados.rados.0.smithi064.stdout:1051: left oid 16 (ObjNum 514 snap 118 seq_num 514) 2024-03-04T06:12:01.698 INFO:tasks.rados.rados.0.smithi064.stdout:1050: done (3 left) 2024-03-04T06:12:01.698 INFO:tasks.rados.rados.0.smithi064.stdout:1051: done (2 left) 2024-03-04T06:12:01.698 INFO:tasks.rados.rados.0.smithi064.stdout:1052: done (1 left) 2024-03-04T06:12:01.699 INFO:tasks.rados.rados.0.smithi064.stdout:1053: done (0 left) 2024-03-04T06:12:01.699 INFO:tasks.rados.rados.0.smithi064.stdout:1054: snap_create 2024-03-04T06:12:02.035 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:12:02.036 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:12:02.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:12:02.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:02.140+0000 7ff173add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:02.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:02.146+0000 7ff173add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:12:02.339 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:12:02.351 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-03-04T06:12:01.347273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":90157247,"num_objects":342,"num_object_clones":244,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":65,"num_read":2040,"num_read_kb":1457281,"num_write":1785,"num_write_kb":451187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3352,"ondisk_log_size":3352,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":329932,"kb_used_data":187420,"kb_used_omap":0,"kb_used_meta":136384,"kb_avail":377157428,"statfs":{"total":386547056640,"available":386209206272,"internally_reserved":0,"allocated":191918080,"data_stored":193941870,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":139657216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2874063,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":15,"num_read_kb":15265,"num_write":14,"num_write_kb":2939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.003794"},"pg_stats":[{"pgid":"3.f","version":"226'155","reported_seq":810,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.236766+0000","last_change":"2024-03-04T06:11:59.244069+0000","last_active":"2024-03-04T06:11:59.236766+0000","last_peered":"2024-03-04T06:11:59.236766+0000","last_clean":"2024-03-04T06:11:59.236766+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:11:59.236766+0000","last_undegraded":"2024-03-04T06:11:59.236766+0000","last_fullsized":"2024-03-04T06:11:59.236766+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6655704,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":58,"num_read_kb":24580,"num_write":90,"num_write_kb":24194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"249'137","reported_seq":822,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.224260+0000","last_change":"2024-03-04T06:11:59.255387+0000","last_active":"2024-03-04T06:11:59.224260+0000","last_peered":"2024-03-04T06:11:59.224260+0000","last_clean":"2024-03-04T06:11:59.224260+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:11:59.224260+0000","last_undegraded":"2024-03-04T06:11:59.224260+0000","last_fullsized":"2024-03-04T06:11:59.224260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6000264,"num_objects":16,"num_object_clones":10,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":61,"num_read_kb":40084,"num_write":68,"num_write_kb":17729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"245'230","reported_seq":858,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-03-04T06:11:55.000927+0000","last_change":"2024-03-04T06:11:53.375646+0000","last_active":"2024-03-04T06:11:55.000927+0000","last_peered":"2024-03-04T06:11:55.000927+0000","last_clean":"2024-03-04T06:11:55.000927+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:11:55.000927+0000","last_undegraded":"2024-03-04T06:11:55.000927+0000","last_fullsized":"2024-03-04T06:11:55.000927+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7076011,"num_objects":27,"num_object_clones":20,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":123,"num_read_kb":75546,"num_write":128,"num_write_kb":33954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"249'147","reported_seq":830,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.219416+0000","last_change":"2024-03-04T06:11:59.264653+0000","last_active":"2024-03-04T06:11:59.219416+0000","last_peered":"2024-03-04T06:11:59.219416+0000","last_clean":"2024-03-04T06:11:59.219416+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:11:59.219416+0000","last_undegraded":"2024-03-04T06:11:59.219416+0000","last_fullsized":"2024-03-04T06:11:59.219416+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2453888,"num_objects":16,"num_object_clones":10,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":100,"num_read_kb":80742,"num_write":88,"num_write_kb":24424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"226'91","reported_seq":749,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-04T06:11:56.365112+0000","last_change":"2024-03-04T06:11:53.256963+0000","last_active":"2024-03-04T06:11:56.365112+0000","last_peered":"2024-03-04T06:11:56.365112+0000","last_clean":"2024-03-04T06:11:56.365112+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:11:56.365112+0000","last_undegraded":"2024-03-04T06:11:56.365112+0000","last_fullsized":"2024-03-04T06:11:56.365112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":17041,"num_write":34,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"249'170","reported_seq":838,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.235253+0000","last_change":"2024-03-04T06:11:59.276745+0000","last_active":"2024-03-04T06:11:59.235253+0000","last_peered":"2024-03-04T06:11:59.235253+0000","last_clean":"2024-03-04T06:11:59.235253+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:11:59.235253+0000","last_undegraded":"2024-03-04T06:11:59.235253+0000","last_fullsized":"2024-03-04T06:11:59.235253+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3899290,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":108,"num_read_kb":83332,"num_write":96,"num_write_kb":29013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":706,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.232982+0000","last_change":"2024-03-04T06:11:59.243460+0000","last_active":"2024-03-04T06:11:59.232982+0000","last_peered":"2024-03-04T06:11:59.232982+0000","last_clean":"2024-03-04T06:11:59.232982+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:11:59.232982+0000","last_undegraded":"2024-03-04T06:11:59.232982+0000","last_fullsized":"2024-03-04T06:11:59.232982+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"245'144","reported_seq":801,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-03-04T06:11:55.214980+0000","last_change":"2024-03-04T06:11:53.376397+0000","last_active":"2024-03-04T06:11:55.214980+0000","last_peered":"2024-03-04T06:11:55.214980+0000","last_clean":"2024-03-04T06:11:55.214980+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:11:55.214980+0000","last_undegraded":"2024-03-04T06:11:55.214980+0000","last_fullsized":"2024-03-04T06:11:55.214980+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4403599,"num_objects":16,"num_object_clones":10,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":98,"num_read_kb":95873,"num_write":71,"num_write_kb":19512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":734,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-04T06:11:56.360222+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:11:56.360222+0000","last_peered":"2024-03-04T06:11:56.360222+0000","last_clean":"2024-03-04T06:11:56.360222+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:11:56.360222+0000","last_undegraded":"2024-03-04T06:11:56.360222+0000","last_fullsized":"2024-03-04T06:11:56.360222+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":589,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-03-04T06:11:54.366244+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:11:54.366244+0000","last_peered":"2024-03-04T06:11:54.366244+0000","last_clean":"2024-03-04T06:11:54.366244+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:11:54.366244+0000","last_undegraded":"2024-03-04T06:11:54.366244+0000","last_fullsized":"2024-03-04T06:11:54.366244+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"245'168","reported_seq":829,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-03-04T06:11:55.260021+0000","last_change":"2024-03-04T06:11:53.368468+0000","last_active":"2024-03-04T06:11:55.260021+0000","last_peered":"2024-03-04T06:11:55.260021+0000","last_clean":"2024-03-04T06:11:55.260021+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:11:55.260021+0000","last_undegraded":"2024-03-04T06:11:55.260021+0000","last_fullsized":"2024-03-04T06:11:55.260021+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269308,"num_objects":18,"num_object_clones":11,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":130,"num_read_kb":109557,"num_write":84,"num_write_kb":21046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":601,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.228320+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:11:59.228320+0000","last_peered":"2024-03-04T06:11:59.228320+0000","last_clean":"2024-03-04T06:11:59.228320+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:11:59.228320+0000","last_undegraded":"2024-03-04T06:11:59.228320+0000","last_fullsized":"2024-03-04T06:11:59.228320+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"236'175","reported_seq":803,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-04T06:11:56.358196+0000","last_change":"2024-03-04T06:11:53.257861+0000","last_active":"2024-03-04T06:11:56.358196+0000","last_peered":"2024-03-04T06:11:56.358196+0000","last_clean":"2024-03-04T06:11:56.358196+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:11:56.358196+0000","last_undegraded":"2024-03-04T06:11:56.358196+0000","last_fullsized":"2024-03-04T06:11:56.358196+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731220,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":82,"num_read_kb":63041,"num_write":69,"num_write_kb":20140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":589,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-03-04T06:11:54.353543+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:11:54.353543+0000","last_peered":"2024-03-04T06:11:54.353543+0000","last_clean":"2024-03-04T06:11:54.353543+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:11:54.353543+0000","last_undegraded":"2024-03-04T06:11:54.353543+0000","last_fullsized":"2024-03-04T06:11:54.353543+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"245'414","reported_seq":1040,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-04T06:11:56.368422+0000","last_change":"2024-03-04T06:11:54.365096+0000","last_active":"2024-03-04T06:11:56.368422+0000","last_peered":"2024-03-04T06:11:56.368422+0000","last_clean":"2024-03-04T06:11:56.368422+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:11:56.368422+0000","last_undegraded":"2024-03-04T06:11:56.368422+0000","last_fullsized":"2024-03-04T06:11:56.368422+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9092785,"num_objects":41,"num_object_clones":31,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":294,"num_read_kb":244037,"num_write":224,"num_write_kb":62261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":601,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.225000+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:11:59.225000+0000","last_peered":"2024-03-04T06:11:59.225000+0000","last_clean":"2024-03-04T06:11:59.225000+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:11:59.225000+0000","last_undegraded":"2024-03-04T06:11:59.225000+0000","last_fullsized":"2024-03-04T06:11:59.225000+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"248'426","reported_seq":1005,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.415492+0000","last_change":"2024-03-04T06:11:59.229756+0000","last_active":"2024-03-04T06:11:59.415492+0000","last_peered":"2024-03-04T06:11:59.415492+0000","last_clean":"2024-03-04T06:11:59.415492+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:11:59.415492+0000","last_undegraded":"2024-03-04T06:11:59.415492+0000","last_fullsized":"2024-03-04T06:11:59.415492+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11345055,"num_objects":43,"num_object_clones":31,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":189,"num_read_kb":136811,"num_write":211,"num_write_kb":56539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"244'165","reported_seq":848,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-03-04T06:11:54.987882+0000","last_change":"2024-03-04T06:11:53.369186+0000","last_active":"2024-03-04T06:11:54.987882+0000","last_peered":"2024-03-04T06:11:54.987882+0000","last_clean":"2024-03-04T06:11:54.987882+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:11:54.987882+0000","last_undegraded":"2024-03-04T06:11:54.987882+0000","last_fullsized":"2024-03-04T06:11:54.987882+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294238,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":169,"num_read_kb":135613,"num_write":86,"num_write_kb":22565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":631,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.239062+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:11:59.239062+0000","last_peered":"2024-03-04T06:11:59.239062+0000","last_clean":"2024-03-04T06:11:59.239062+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:11:59.239062+0000","last_undegraded":"2024-03-04T06:11:59.239062+0000","last_fullsized":"2024-03-04T06:11:59.239062+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"249'179","reported_seq":862,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.424921+0000","last_change":"2024-03-04T06:11:59.244339+0000","last_active":"2024-03-04T06:11:59.424921+0000","last_peered":"2024-03-04T06:11:59.424921+0000","last_clean":"2024-03-04T06:11:59.424921+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:11:59.424921+0000","last_undegraded":"2024-03-04T06:11:59.424921+0000","last_fullsized":"2024-03-04T06:11:59.424921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168334,"num_objects":22,"num_object_clones":13,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":103,"num_read_kb":59202,"num_write":92,"num_write_kb":24646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":602,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-04T06:11:56.365843+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:11:56.365843+0000","last_peered":"2024-03-04T06:11:56.365843+0000","last_clean":"2024-03-04T06:11:56.365843+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:11:56.365843+0000","last_undegraded":"2024-03-04T06:11:56.365843+0000","last_fullsized":"2024-03-04T06:11:56.365843+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"245'277","reported_seq":896,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-04T06:11:55.444284+0000","last_change":"2024-03-04T06:11:54.360335+0000","last_active":"2024-03-04T06:11:55.444284+0000","last_peered":"2024-03-04T06:11:55.444284+0000","last_clean":"2024-03-04T06:11:55.444284+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:11:55.444284+0000","last_undegraded":"2024-03-04T06:11:55.444284+0000","last_fullsized":"2024-03-04T06:11:55.444284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8336503,"num_objects":28,"num_object_clones":20,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":152,"num_read_kb":105495,"num_write":141,"num_write_kb":38941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":631,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.220513+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:11:59.220513+0000","last_peered":"2024-03-04T06:11:59.220513+0000","last_clean":"2024-03-04T06:11:59.220513+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:11:59.220513+0000","last_undegraded":"2024-03-04T06:11:59.220513+0000","last_fullsized":"2024-03-04T06:11:59.220513+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"236'377","reported_seq":992,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-04T06:11:59.222513+0000","last_change":"2024-03-04T06:11:59.232022+0000","last_active":"2024-03-04T06:11:59.222513+0000","last_peered":"2024-03-04T06:11:59.222513+0000","last_clean":"2024-03-04T06:11:59.222513+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:11:59.222513+0000","last_undegraded":"2024-03-04T06:11:59.222513+0000","last_fullsized":"2024-03-04T06:11:59.222513+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8840680,"num_objects":33,"num_object_clones":29,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":239,"num_read_kb":186236,"num_write":166,"num_write_kb":45580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":589,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-03-04T06:11:54.362585+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:11:54.362585+0000","last_peered":"2024-03-04T06:11:54.362585+0000","last_clean":"2024-03-04T06:11:54.362585+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:11:54.362585+0000","last_undegraded":"2024-03-04T06:11:54.362585+0000","last_fullsized":"2024-03-04T06:11:54.362585+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":89566879,"num_objects":340,"num_object_clones":244,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":65,"num_read":1937,"num_read_kb":1457190,"num_write":1648,"num_write_kb":449293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3255,"ondisk_log_size":3255,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96220,"kb_used_data":58028,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94275620,"statfs":{"total":96636764160,"available":96538234880,"internally_reserved":0,"allocated":59420672,"data_stored":59739789,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89228,"kb_used_data":51036,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94282612,"statfs":{"total":96636764160,"available":96545394688,"internally_reserved":0,"allocated":52260864,"data_stored":52092900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71660,"kb_used_data":36516,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94300180,"statfs":{"total":96636764160,"available":96563384320,"internally_reserved":0,"allocated":37392384,"data_stored":37200052,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":72824,"kb_used_data":41840,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94299016,"statfs":{"total":96636764160,"available":96562192384,"internally_reserved":0,"allocated":42844160,"data_stored":44909129,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:12:02.353 INFO:tasks.rados.rados.0.smithi064.stdout:1054: done (0 left) 2024-03-04T06:12:02.353 INFO:tasks.rados.rados.0.smithi064.stdout:1055: write (excl) oid 38 current snap is 119 2024-03-04T06:12:02.353 INFO:tasks.rados.rados.0.smithi064.stdout:1055: seq_num 516 ranges {672412=624135,1995532=676880} 2024-03-04T06:12:02.359 INFO:tasks.rados.rados.0.smithi064.stdout:1055: writing smithi06431743-38 from 672412 to 1296547 tid 1 2024-03-04T06:12:02.368 INFO:tasks.rados.rados.0.smithi064.stdout:1055: writing smithi06431743-38 from 1995532 to 2672412 tid 2 2024-03-04T06:12:02.369 INFO:tasks.rados.rados.0.smithi064.stdout:1056: read oid 25 snap -1 2024-03-04T06:12:02.369 INFO:tasks.rados.rados.0.smithi064.stdout:1056: expect (ObjNum 500 snap 115 seq_num 500) 2024-03-04T06:12:02.369 INFO:tasks.rados.rados.0.smithi064.stdout:1057: read oid 17 snap -1 2024-03-04T06:12:02.369 INFO:tasks.rados.rados.0.smithi064.stdout:1057: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-04T06:12:02.369 INFO:tasks.rados.rados.0.smithi064.stdout:1058: read oid 43 snap -1 2024-03-04T06:12:02.369 INFO:tasks.rados.rados.0.smithi064.stdout:1058: expect (ObjNum 507 snap 117 seq_num 507) 2024-03-04T06:12:02.369 INFO:tasks.rados.rados.0.smithi064.stdout:1059: read oid 18 snap 117 2024-03-04T06:12:02.369 INFO:tasks.rados.rados.0.smithi064.stdout:1059: expect deleted 2024-03-04T06:12:02.369 INFO:tasks.rados.rados.0.smithi064.stdout:1060: snap_create 2024-03-04T06:12:02.413 INFO:tasks.rados.rados.0.smithi064.stdout:1058: expect (ObjNum 507 snap 117 seq_num 507) 2024-03-04T06:12:02.597 INFO:tasks.rados.rados.0.smithi064.stdout:1055: finishing write tid 1 to smithi06431743-38 2024-03-04T06:12:02.598 INFO:tasks.rados.rados.0.smithi064.stdout:1056: expect (ObjNum 500 snap 115 seq_num 500) 2024-03-04T06:12:02.747 INFO:tasks.rados.rados.0.smithi064.stdout:1057: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-04T06:12:02.908 INFO:tasks.rados.rados.0.smithi064.stdout:1055: finishing write tid 2 to smithi06431743-38 2024-03-04T06:12:02.909 INFO:tasks.rados.rados.0.smithi064.stdout:1055: finishing write tid 3 to smithi06431743-38 2024-03-04T06:12:02.909 INFO:tasks.rados.rados.0.smithi064.stdout:1055: finishing write tid 4 to smithi06431743-38 2024-03-04T06:12:02.909 INFO:tasks.rados.rados.0.smithi064.stdout:1055: oid 38 updating version 0 to 174 2024-03-04T06:12:02.909 INFO:tasks.rados.rados.0.smithi064.stdout:1055: oid 38 version 174 is already newer than 173 2024-03-04T06:12:02.909 INFO:tasks.rados.rados.0.smithi064.stdout:1055: oid 38 updating version 174 to 175 2024-03-04T06:12:02.909 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 175 (ObjNum 516 snap 119 seq_num 516) dirty exists 2024-03-04T06:12:02.909 INFO:tasks.rados.rados.0.smithi064.stdout:1055: left oid 38 (ObjNum 516 snap 119 seq_num 516) 2024-03-04T06:12:03.356 INFO:tasks.rados.rados.0.smithi064.stdout:1055: done (5 left) 2024-03-04T06:12:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:1056: done (4 left) 2024-03-04T06:12:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:1057: done (3 left) 2024-03-04T06:12:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:1058: done (2 left) 2024-03-04T06:12:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:1059: done (1 left) 2024-03-04T06:12:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:1060: done (0 left) 2024-03-04T06:12:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:1061: snap_create 2024-03-04T06:12:03.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:03.771 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:03.769+0000 7fa12339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:03.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:03.773+0000 7fa12339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:04.358 INFO:tasks.rados.rados.0.smithi064.stdout:1061: done (0 left) 2024-03-04T06:12:04.358 INFO:tasks.rados.rados.0.smithi064.stdout:1062: read oid 24 snap 116 2024-03-04T06:12:04.358 INFO:tasks.rados.rados.0.smithi064.stdout:1062: expect deleted 2024-03-04T06:12:04.358 INFO:tasks.rados.rados.0.smithi064.stdout:1063: snap_create 2024-03-04T06:12:05.361 INFO:tasks.rados.rados.0.smithi064.stdout:1062: done (1 left) 2024-03-04T06:12:05.361 INFO:tasks.rados.rados.0.smithi064.stdout:1063: done (0 left) 2024-03-04T06:12:05.361 INFO:tasks.rados.rados.0.smithi064.stdout:1064: write (excl) oid 37 current snap is 122 2024-03-04T06:12:05.361 INFO:tasks.rados.rados.0.smithi064.stdout:1064: seq_num 517 ranges {689219=415737,1770367=646847} 2024-03-04T06:12:05.367 INFO:tasks.rados.rados.0.smithi064.stdout:1064: writing smithi06431743-37 from 689219 to 1104956 tid 1 2024-03-04T06:12:05.374 INFO:tasks.rados.rados.0.smithi064.stdout:1064: writing smithi06431743-37 from 1770367 to 2417214 tid 2 2024-03-04T06:12:05.374 INFO:tasks.rados.rados.0.smithi064.stdout:1065: read oid 22 snap -1 2024-03-04T06:12:05.375 INFO:tasks.rados.rados.0.smithi064.stdout:1065: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-04T06:12:05.375 INFO:tasks.rados.rados.0.smithi064.stdout:1066: snap_remove snap 121 2024-03-04T06:12:05.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:05.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:05.961+0000 7fcf3066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:05.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:05.966+0000 7fcf3066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:06.376 INFO:tasks.rados.rados.0.smithi064.stdout:1066: done (2 left) 2024-03-04T06:12:06.377 INFO:tasks.rados.rados.0.smithi064.stdout:1067: read oid 15 snap -1 2024-03-04T06:12:06.377 INFO:tasks.rados.rados.0.smithi064.stdout:1067: expect deleted 2024-03-04T06:12:06.377 INFO:tasks.rados.rados.0.smithi064.stdout:1068: write (excl) oid 16 current snap is 122 2024-03-04T06:12:06.377 INFO:tasks.rados.rados.0.smithi064.stdout:1068: seq_num 518 ranges {706026=490986,1828849=790505} 2024-03-04T06:12:06.382 INFO:tasks.rados.rados.0.smithi064.stdout:1068: writing smithi06431743-16 from 706026 to 1197012 tid 1 2024-03-04T06:12:06.391 INFO:tasks.rados.rados.0.smithi064.stdout:1068: writing smithi06431743-16 from 1828849 to 2619354 tid 2 2024-03-04T06:12:06.391 INFO:tasks.rados.rados.0.smithi064.stdout:1069: read oid 6 snap -1 2024-03-04T06:12:06.391 INFO:tasks.rados.rados.0.smithi064.stdout:1069: expect (ObjNum 434 snap 101 seq_num 434) 2024-03-04T06:12:06.391 INFO:tasks.rados.rados.0.smithi064.stdout:1070: write (excl) oid 30 current snap is 122 2024-03-04T06:12:06.391 INFO:tasks.rados.rados.0.smithi064.stdout:1070: seq_num 519 ranges {722833=566235,1887331=650516} 2024-03-04T06:12:06.397 INFO:tasks.rados.rados.0.smithi064.stdout:1070: writing smithi06431743-30 from 722833 to 1289068 tid 1 2024-03-04T06:12:06.404 INFO:tasks.rados.rados.0.smithi064.stdout:1070: writing smithi06431743-30 from 1887331 to 2537847 tid 2 2024-03-04T06:12:06.404 INFO:tasks.rados.rados.0.smithi064.stdout:1071: write oid 23 current snap is 122 2024-03-04T06:12:06.404 INFO:tasks.rados.rados.0.smithi064.stdout:1071: seq_num 520 ranges {739640=641484,1829460=794174} 2024-03-04T06:12:06.412 INFO:tasks.rados.rados.0.smithi064.stdout:1071: writing smithi06431743-23 from 739640 to 1381124 tid 1 2024-03-04T06:12:06.419 INFO:tasks.rados.rados.0.smithi064.stdout:1071: writing smithi06431743-23 from 1829460 to 2623634 tid 2 2024-03-04T06:12:06.419 INFO:tasks.rados.rados.0.smithi064.stdout:1072: write oid 43 current snap is 122 2024-03-04T06:12:06.419 INFO:tasks.rados.rados.0.smithi064.stdout:1065: expect (ObjNum 373 snap 91 seq_num 373) 2024-03-04T06:12:06.556 INFO:tasks.rados.rados.0.smithi064.stdout:1064: finishing write tid 1 to smithi06431743-37 2024-03-04T06:12:06.556 INFO:tasks.rados.rados.0.smithi064.stdout:1064: finishing write tid 2 to smithi06431743-37 2024-03-04T06:12:06.556 INFO:tasks.rados.rados.0.smithi064.stdout:1064: finishing write tid 3 to smithi06431743-37 2024-03-04T06:12:06.556 INFO:tasks.rados.rados.0.smithi064.stdout:1064: finishing write tid 4 to smithi06431743-37 2024-03-04T06:12:06.556 INFO:tasks.rados.rados.0.smithi064.stdout:1064: oid 37 updating version 0 to 141 2024-03-04T06:12:06.556 INFO:tasks.rados.rados.0.smithi064.stdout:1064: oid 37 updating version 141 to 143 2024-03-04T06:12:06.556 INFO:tasks.rados.rados.0.smithi064.stdout:1064: oid 37 version 143 is already newer than 143 2024-03-04T06:12:06.556 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 143 (ObjNum 517 snap 122 seq_num 517) dirty exists 2024-03-04T06:12:06.556 INFO:tasks.rados.rados.0.smithi064.stdout:1064: left oid 37 (ObjNum 517 snap 122 seq_num 517) 2024-03-04T06:12:06.556 INFO:tasks.rados.rados.0.smithi064.stdout:1072: seq_num 521 ranges {756447=716733,1887942=537832} 2024-03-04T06:12:06.564 INFO:tasks.rados.rados.0.smithi064.stdout:1072: writing smithi06431743-43 from 756447 to 1473180 tid 1 2024-03-04T06:12:06.569 INFO:tasks.rados.rados.0.smithi064.stdout:1072: writing smithi06431743-43 from 1887942 to 2425774 tid 2 2024-03-04T06:12:06.569 INFO:tasks.rados.rados.0.smithi064.stdout:1064: done (7 left) 2024-03-04T06:12:06.569 INFO:tasks.rados.rados.0.smithi064.stdout:1065: done (6 left) 2024-03-04T06:12:06.569 INFO:tasks.rados.rados.0.smithi064.stdout:1073: write oid 47 current snap is 122 2024-03-04T06:12:06.569 INFO:tasks.rados.rados.0.smithi064.stdout:1068: finishing write tid 1 to smithi06431743-16 2024-03-04T06:12:06.570 INFO:tasks.rados.rados.0.smithi064.stdout:1073: seq_num 522 ranges {773254=791982,2346424=426830} 2024-03-04T06:12:06.578 INFO:tasks.rados.rados.0.smithi064.stdout:1073: writing smithi06431743-47 from 773254 to 1565236 tid 1 2024-03-04T06:12:06.584 INFO:tasks.rados.rados.0.smithi064.stdout:1073: writing smithi06431743-47 from 2346424 to 2773254 tid 2 2024-03-04T06:12:06.585 INFO:tasks.rados.rados.0.smithi064.stdout:1074: snap_remove snap 118 2024-03-04T06:12:07.354 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:12:07.354 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:12:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:1074: done (7 left) 2024-03-04T06:12:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:1075: write oid 9 current snap is 122 2024-03-04T06:12:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:1075: seq_num 523 ranges {790061=467231,2004906=541501} 2024-03-04T06:12:07.393 INFO:tasks.rados.rados.0.smithi064.stdout:1075: writing smithi06431743-9 from 790061 to 1257292 tid 1 2024-03-04T06:12:07.400 INFO:tasks.rados.rados.0.smithi064.stdout:1075: writing smithi06431743-9 from 2004906 to 2546407 tid 2 2024-03-04T06:12:07.400 INFO:tasks.rados.rados.0.smithi064.stdout:1076: write (excl) oid 35 current snap is 122 2024-03-04T06:12:07.400 INFO:tasks.rados.rados.0.smithi064.stdout:1076: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-03-04T06:12:07.405 INFO:tasks.rados.rados.0.smithi064.stdout:1076: writing smithi06431743-35 from 406868 to 949348 tid 1 2024-03-04T06:12:07.411 INFO:tasks.rados.rados.0.smithi064.stdout:1076: writing smithi06431743-35 from 1547035 to 1948547 tid 2 2024-03-04T06:12:07.416 INFO:tasks.rados.rados.0.smithi064.stdout:1076: writing smithi06431743-35 from 2690354 to 2806868 tid 3 2024-03-04T06:12:07.416 INFO:tasks.rados.rados.0.smithi064.stdout:1077: read oid 17 snap -1 2024-03-04T06:12:07.416 INFO:tasks.rados.rados.0.smithi064.stdout:1077: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-04T06:12:07.416 INFO:tasks.rados.rados.0.smithi064.stdout:1078: read oid 34 snap -1 2024-03-04T06:12:07.416 INFO:tasks.rados.rados.0.smithi064.stdout:1078: expect (ObjNum 484 snap 114 seq_num 484) 2024-03-04T06:12:07.416 INFO:tasks.rados.rados.0.smithi064.stdout:1079: delete oid 3 current snap is 122 2024-03-04T06:12:07.416 INFO:tasks.rados.rados.0.smithi064.stdout:1070: finishing write tid 1 to smithi06431743-30 2024-03-04T06:12:07.416 INFO:tasks.rados.rados.0.smithi064.stdout:1069: expect (ObjNum 434 snap 101 seq_num 434) 2024-03-04T06:12:07.511 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:07.504+0000 7fed21e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:07.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:07.515+0000 7fed21e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1068: finishing write tid 2 to smithi06431743-16 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1071: finishing write tid 1 to smithi06431743-23 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1071: finishing write tid 2 to smithi06431743-23 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1068: finishing write tid 3 to smithi06431743-16 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1068: finishing write tid 4 to smithi06431743-16 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1068: oid 16 updating version 0 to 189 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1068: oid 16 version 189 is already newer than 189 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1068: oid 16 version 189 is already newer than 187 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 189 (ObjNum 518 snap 122 seq_num 518) dirty exists 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1068: left oid 16 (ObjNum 518 snap 122 seq_num 518) 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1070: finishing write tid 2 to smithi06431743-30 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1070: finishing write tid 3 to smithi06431743-30 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1070: finishing write tid 4 to smithi06431743-30 2024-03-04T06:12:07.560 INFO:tasks.rados.rados.0.smithi064.stdout:1070: oid 30 updating version 0 to 431 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1070: oid 30 updating version 431 to 432 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1070: oid 30 version 432 is already newer than 430 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 432 (ObjNum 519 snap 122 seq_num 519) dirty exists 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1070: left oid 30 (ObjNum 519 snap 122 seq_num 519) 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1071: finishing write tid 3 to smithi06431743-23 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1071: finishing write tid 4 to smithi06431743-23 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1071: oid 23 updating version 0 to 180 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1071: oid 23 updating version 180 to 181 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1071: oid 23 version 181 is already newer than 180 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 181 (ObjNum 520 snap 122 seq_num 520) dirty exists 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1071: left oid 23 (ObjNum 520 snap 122 seq_num 520) 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1067: done (11 left) 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1068: done (10 left) 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1069: done (9 left) 2024-03-04T06:12:07.561 INFO:tasks.rados.rados.0.smithi064.stdout:1070: done (8 left) 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1071: done (7 left) 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1079: done (6 left) 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1080: snap_create 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1072: finishing write tid 1 to smithi06431743-43 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1073: finishing write tid 1 to smithi06431743-47 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1073: finishing write tid 2 to smithi06431743-47 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1072: finishing write tid 2 to smithi06431743-43 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1073: finishing write tid 3 to smithi06431743-47 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1073: finishing write tid 4 to smithi06431743-47 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1073: oid 47 updating version 0 to 153 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1073: oid 47 updating version 153 to 154 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1073: oid 47 version 154 is already newer than 152 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 154 (ObjNum 522 snap 122 seq_num 522) dirty exists 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1073: left oid 47 (ObjNum 522 snap 122 seq_num 522) 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1072: finishing write tid 3 to smithi06431743-43 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1072: finishing write tid 4 to smithi06431743-43 2024-03-04T06:12:07.562 INFO:tasks.rados.rados.0.smithi064.stdout:1072: oid 43 updating version 0 to 162 2024-03-04T06:12:07.563 INFO:tasks.rados.rados.0.smithi064.stdout:1072: oid 43 updating version 162 to 163 2024-03-04T06:12:07.563 INFO:tasks.rados.rados.0.smithi064.stdout:1072: oid 43 version 163 is already newer than 161 2024-03-04T06:12:07.563 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 163 (ObjNum 521 snap 122 seq_num 521) dirty exists 2024-03-04T06:12:07.563 INFO:tasks.rados.rados.0.smithi064.stdout:1072: left oid 43 (ObjNum 521 snap 122 seq_num 521) 2024-03-04T06:12:07.563 INFO:tasks.rados.rados.0.smithi064.stdout:1076: finishing write tid 1 to smithi06431743-35 2024-03-04T06:12:07.563 INFO:tasks.rados.rados.0.smithi064.stdout:1078: expect (ObjNum 484 snap 114 seq_num 484) 2024-03-04T06:12:07.655 INFO:tasks.rados.rados.0.smithi064.stdout:1075: finishing write tid 1 to smithi06431743-9 2024-03-04T06:12:07.656 INFO:tasks.rados.rados.0.smithi064.stdout:1075: finishing write tid 2 to smithi06431743-9 2024-03-04T06:12:07.656 INFO:tasks.rados.rados.0.smithi064.stdout:1076: finishing write tid 2 to smithi06431743-35 2024-03-04T06:12:07.656 INFO:tasks.rados.rados.0.smithi064.stdout:1075: finishing write tid 3 to smithi06431743-9 2024-03-04T06:12:07.656 INFO:tasks.rados.rados.0.smithi064.stdout:1075: finishing write tid 4 to smithi06431743-9 2024-03-04T06:12:07.656 INFO:tasks.rados.rados.0.smithi064.stdout:1075: oid 9 updating version 0 to 380 2024-03-04T06:12:07.656 INFO:tasks.rados.rados.0.smithi064.stdout:1075: oid 9 updating version 380 to 381 2024-03-04T06:12:07.656 INFO:tasks.rados.rados.0.smithi064.stdout:1075: oid 9 updating version 381 to 382 2024-03-04T06:12:07.656 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 382 (ObjNum 523 snap 122 seq_num 523) dirty exists 2024-03-04T06:12:07.656 INFO:tasks.rados.rados.0.smithi064.stdout:1075: left oid 9 (ObjNum 523 snap 122 seq_num 523) 2024-03-04T06:12:07.656 INFO:tasks.rados.rados.0.smithi064.stdout:1077: expect (ObjNum 366 snap 91 seq_num 366) 2024-03-04T06:12:07.817 INFO:tasks.rados.rados.0.smithi064.stdout:1076: finishing write tid 3 to smithi06431743-35 2024-03-04T06:12:07.817 INFO:tasks.rados.rados.0.smithi064.stdout:1076: finishing write tid 4 to smithi06431743-35 2024-03-04T06:12:07.817 INFO:tasks.rados.rados.0.smithi064.stdout:1076: finishing write tid 5 to smithi06431743-35 2024-03-04T06:12:07.817 INFO:tasks.rados.rados.0.smithi064.stdout:1076: oid 35 updating version 0 to 284 2024-03-04T06:12:07.817 INFO:tasks.rados.rados.0.smithi064.stdout:1076: oid 35 updating version 284 to 286 2024-03-04T06:12:07.817 INFO:tasks.rados.rados.0.smithi064.stdout:1076: oid 35 version 286 is already newer than 283 2024-03-04T06:12:07.817 INFO:tasks.rados.rados.0.smithi064.stdout:1076: oid 35 version 286 is already newer than 286 2024-03-04T06:12:07.817 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 286 (ObjNum 524 snap 122 seq_num 524) dirty exists 2024-03-04T06:12:07.817 INFO:tasks.rados.rados.0.smithi064.stdout:1076: left oid 35 (ObjNum 524 snap 122 seq_num 524) 2024-03-04T06:12:08.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:08.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:08.146+0000 7fa2ac864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:08.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:08.151+0000 7fa2ac864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:08.205 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:12:08.211 INFO:tasks.rados.rados.0.smithi064.stdout:1072: done (6 left) 2024-03-04T06:12:08.211 INFO:tasks.rados.rados.0.smithi064.stdout:1073: done (5 left) 2024-03-04T06:12:08.211 INFO:tasks.rados.rados.0.smithi064.stdout:1075: done (4 left) 2024-03-04T06:12:08.211 INFO:tasks.rados.rados.0.smithi064.stdout:1076: done (3 left) 2024-03-04T06:12:08.211 INFO:tasks.rados.rados.0.smithi064.stdout:1077: done (2 left) 2024-03-04T06:12:08.211 INFO:tasks.rados.rados.0.smithi064.stdout:1078: done (1 left) 2024-03-04T06:12:08.211 INFO:tasks.rados.rados.0.smithi064.stdout:1080: done (0 left) 2024-03-04T06:12:08.211 INFO:tasks.rados.rados.0.smithi064.stdout:1081: delete oid 17 current snap is 123 2024-03-04T06:12:08.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:12:08.257 INFO:tasks.rados.rados.0.smithi064.stdout:1081: done (0 left) 2024-03-04T06:12:08.257 INFO:tasks.rados.rados.0.smithi064.stdout:1082: write oid 1 current snap is 123 2024-03-04T06:12:08.257 INFO:tasks.rados.rados.0.smithi064.stdout:1082: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-03-04T06:12:08.264 INFO:tasks.rados.rados.0.smithi064.stdout:1082: writing smithi06431743-1 from 457289 to 941869 tid 1 2024-03-04T06:12:08.272 INFO:tasks.rados.rados.0.smithi064.stdout:1082: writing smithi06431743-1 from 1438834 to 1987673 tid 2 2024-03-04T06:12:08.277 INFO:tasks.rados.rados.0.smithi064.stdout:1082: writing smithi06431743-1 from 2772623 to 2857289 tid 3 2024-03-04T06:12:08.277 INFO:tasks.rados.rados.0.smithi064.stdout:1083: snap_remove snap 122 2024-03-04T06:12:08.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:08.345+0000 7f5c0d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:08.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:08.354+0000 7f5c0d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:09.215 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:12:09.217 INFO:tasks.rados.rados.0.smithi064.stdout:1083: done (1 left) 2024-03-04T06:12:09.218 INFO:tasks.rados.rados.0.smithi064.stdout:1084: read oid 7 snap 117 2024-03-04T06:12:09.218 INFO:tasks.rados.rados.0.smithi064.stdout:1084: expect deleted 2024-03-04T06:12:09.218 INFO:tasks.rados.rados.0.smithi064.stdout:1085: write (excl) oid 9 current snap is 123 2024-03-04T06:12:09.218 INFO:tasks.rados.rados.0.smithi064.stdout:1082: finishing write tid 1 to smithi06431743-1 2024-03-04T06:12:09.218 INFO:tasks.rados.rados.0.smithi064.stdout:1082: finishing write tid 2 to smithi06431743-1 2024-03-04T06:12:09.218 INFO:tasks.rados.rados.0.smithi064.stdout:1082: finishing write tid 3 to smithi06431743-1 2024-03-04T06:12:09.218 INFO:tasks.rados.rados.0.smithi064.stdout:1085: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2024-03-04T06:12:09.223 INFO:tasks.rados.rados.0.smithi064.stdout:1085: writing smithi06431743-9 from 474096 to 1033925 tid 1 2024-03-04T06:12:09.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:12:09.231 INFO:tasks.rados.rados.0.smithi064.stdout:1085: writing smithi06431743-9 from 1780963 to 2189813 tid 2 2024-03-04T06:12:09.236 INFO:tasks.rados.rados.0.smithi064.stdout:1085: writing smithi06431743-9 from 2800046 to 2874096 tid 3 2024-03-04T06:12:09.236 INFO:tasks.rados.rados.0.smithi064.stdout:1086: read oid 47 snap -1 2024-03-04T06:12:09.236 INFO:tasks.rados.rados.0.smithi064.stdout:1086: expect (ObjNum 522 snap 122 seq_num 522) 2024-03-04T06:12:09.236 INFO:tasks.rados.rados.0.smithi064.stdout:1087: read oid 13 snap -1 2024-03-04T06:12:09.236 INFO:tasks.rados.rados.0.smithi064.stdout:1087: expect (ObjNum 481 snap 113 seq_num 481) 2024-03-04T06:12:09.236 INFO:tasks.rados.rados.0.smithi064.stdout:1088: read oid 20 snap -1 2024-03-04T06:12:09.236 INFO:tasks.rados.rados.0.smithi064.stdout:1088: expect (ObjNum 478 snap 113 seq_num 478) 2024-03-04T06:12:09.236 INFO:tasks.rados.rados.0.smithi064.stdout:1089: read oid 14 snap -1 2024-03-04T06:12:09.236 INFO:tasks.rados.rados.0.smithi064.stdout:1089: expect (ObjNum 498 snap 114 seq_num 498) 2024-03-04T06:12:09.236 INFO:tasks.rados.rados.0.smithi064.stdout:1090: write oid 8 current snap is 123 2024-03-04T06:12:09.237 INFO:tasks.rados.rados.0.smithi064.stdout:1082: finishing write tid 4 to smithi06431743-1 2024-03-04T06:12:09.237 INFO:tasks.rados.rados.0.smithi064.stdout:1082: finishing write tid 5 to smithi06431743-1 2024-03-04T06:12:09.237 INFO:tasks.rados.rados.0.smithi064.stdout:1082: oid 1 updating version 0 to 428 2024-03-04T06:12:09.237 INFO:tasks.rados.rados.0.smithi064.stdout:1082: oid 1 version 428 is already newer than 427 2024-03-04T06:12:09.237 INFO:tasks.rados.rados.0.smithi064.stdout:1082: oid 1 version 428 is already newer than 425 2024-03-04T06:12:09.237 INFO:tasks.rados.rados.0.smithi064.stdout:1082: oid 1 version 428 is already newer than 427 2024-03-04T06:12:09.237 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 428 (ObjNum 527 snap 123 seq_num 527) dirty exists 2024-03-04T06:12:09.237 INFO:tasks.rados.rados.0.smithi064.stdout:1082: left oid 1 (ObjNum 527 snap 123 seq_num 527) 2024-03-04T06:12:09.237 INFO:tasks.rados.rados.0.smithi064.stdout:1090: seq_num 529 ranges {490903=635078,1839445=552508} 2024-03-04T06:12:09.242 INFO:tasks.rados.rados.0.smithi064.stdout:1090: writing smithi06431743-8 from 490903 to 1125981 tid 1 2024-03-04T06:12:09.250 INFO:tasks.rados.rados.0.smithi064.stdout:1090: writing smithi06431743-8 from 1839445 to 2391953 tid 2 2024-03-04T06:12:09.250 INFO:tasks.rados.rados.0.smithi064.stdout:1082: done (7 left) 2024-03-04T06:12:09.250 INFO:tasks.rados.rados.0.smithi064.stdout:1091: read oid 31 snap -1 2024-03-04T06:12:09.251 INFO:tasks.rados.rados.0.smithi064.stdout:1091: expect deleted 2024-03-04T06:12:09.251 INFO:tasks.rados.rados.0.smithi064.stdout:1092: read oid 26 snap -1 2024-03-04T06:12:09.251 INFO:tasks.rados.rados.0.smithi064.stdout:1092: expect deleted 2024-03-04T06:12:09.251 INFO:tasks.rados.rados.0.smithi064.stdout:1093: snap_remove snap 116 2024-03-04T06:12:09.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:09.356+0000 7f6324d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:09.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:09.364+0000 7f6324d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:10.215 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:12:10.218 INFO:tasks.rados.rados.0.smithi064.stdout:1093: done (9 left) 2024-03-04T06:12:10.218 INFO:tasks.rados.rados.0.smithi064.stdout:1094: read oid 45 snap -1 2024-03-04T06:12:10.218 INFO:tasks.rados.rados.0.smithi064.stdout:1094: expect (ObjNum 512 snap 118 seq_num 512) 2024-03-04T06:12:10.218 INFO:tasks.rados.rados.0.smithi064.stdout:1095: write (excl) oid 41 current snap is 123 2024-03-04T06:12:10.218 INFO:tasks.rados.rados.0.smithi064.stdout:1095: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2024-03-04T06:12:10.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:10.226 INFO:tasks.rados.rados.0.smithi064.stdout:1095: writing smithi06431743-41 from 507710 to 1218037 tid 1 2024-03-04T06:12:10.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:12:10.231 INFO:tasks.rados.rados.0.smithi064.stdout:1095: writing smithi06431743-41 from 1897927 to 2310446 tid 2 2024-03-04T06:12:10.234 INFO:tasks.rados.rados.0.smithi064.stdout:1095: writing smithi06431743-41 from 2854892 to 2907710 tid 3 2024-03-04T06:12:10.234 INFO:tasks.rados.rados.0.smithi064.stdout:1096: delete oid 39 current snap is 123 2024-03-04T06:12:10.235 INFO:tasks.rados.rados.0.smithi064.stdout:1088: expect (ObjNum 478 snap 113 seq_num 478) 2024-03-04T06:12:10.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:10.359+0000 7f485a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:10.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:10.364+0000 7f485a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:10.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:10.378+0000 7f4e0419d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:10.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:10.389+0000 7f4e0419d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:10.410 INFO:tasks.rados.rados.0.smithi064.stdout:1087: expect (ObjNum 481 snap 113 seq_num 481) 2024-03-04T06:12:10.488 INFO:tasks.rados.rados.0.smithi064.stdout:1086: expect (ObjNum 522 snap 122 seq_num 522) 2024-03-04T06:12:10.624 INFO:tasks.rados.rados.0.smithi064.stdout:1085: finishing write tid 1 to smithi06431743-9 2024-03-04T06:12:10.624 INFO:tasks.rados.rados.0.smithi064.stdout:1090: finishing write tid 1 to smithi06431743-8 2024-03-04T06:12:10.625 INFO:tasks.rados.rados.0.smithi064.stdout:1089: expect (ObjNum 498 snap 114 seq_num 498) 2024-03-04T06:12:10.714 INFO:tasks.rados.rados.0.smithi064.stdout:1085: finishing write tid 2 to smithi06431743-9 2024-03-04T06:12:10.714 INFO:tasks.rados.rados.0.smithi064.stdout:1090: finishing write tid 2 to smithi06431743-8 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1085: finishing write tid 3 to smithi06431743-9 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1085: finishing write tid 4 to smithi06431743-9 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1085: finishing write tid 5 to smithi06431743-9 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1085: oid 9 updating version 0 to 386 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1085: oid 9 version 386 is already newer than 383 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1085: oid 9 version 386 is already newer than 386 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1085: oid 9 version 386 is already newer than 384 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 386 (ObjNum 528 snap 123 seq_num 528) dirty exists 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1085: left oid 9 (ObjNum 528 snap 123 seq_num 528) 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1090: finishing write tid 3 to smithi06431743-8 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1084: done (11 left) 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1085: done (10 left) 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1086: done (9 left) 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1087: done (8 left) 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1088: done (7 left) 2024-03-04T06:12:10.715 INFO:tasks.rados.rados.0.smithi064.stdout:1089: done (6 left) 2024-03-04T06:12:10.716 INFO:tasks.rados.rados.0.smithi064.stdout:1091: done (5 left) 2024-03-04T06:12:10.716 INFO:tasks.rados.rados.0.smithi064.stdout:1092: done (4 left) 2024-03-04T06:12:10.716 INFO:tasks.rados.rados.0.smithi064.stdout:1096: done (3 left) 2024-03-04T06:12:10.716 INFO:tasks.rados.rados.0.smithi064.stdout:1097: write oid 47 current snap is 123 2024-03-04T06:12:10.716 INFO:tasks.rados.rados.0.smithi064.stdout:1090: finishing write tid 4 to smithi06431743-8 2024-03-04T06:12:10.716 INFO:tasks.rados.rados.0.smithi064.stdout:1090: oid 8 updating version 0 to 193 2024-03-04T06:12:10.716 INFO:tasks.rados.rados.0.smithi064.stdout:1090: oid 8 version 193 is already newer than 192 2024-03-04T06:12:10.716 INFO:tasks.rados.rados.0.smithi064.stdout:1090: oid 8 updating version 193 to 194 2024-03-04T06:12:10.716 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 194 (ObjNum 529 snap 123 seq_num 529) dirty exists 2024-03-04T06:12:10.716 INFO:tasks.rados.rados.0.smithi064.stdout:1090: left oid 8 (ObjNum 529 snap 123 seq_num 529) 2024-03-04T06:12:10.716 INFO:tasks.rados.rados.0.smithi064.stdout:1097: seq_num 532 ranges {541324=460825,1498538=416188,2393385=547939} 2024-03-04T06:12:10.719 INFO:tasks.rados.rados.0.smithi064.stdout:1097: writing smithi06431743-47 from 541324 to 1002149 tid 1 2024-03-04T06:12:10.724 INFO:tasks.rados.rados.0.smithi064.stdout:1097: writing smithi06431743-47 from 1498538 to 1914726 tid 2 2024-03-04T06:12:10.731 INFO:tasks.rados.rados.0.smithi064.stdout:1097: writing smithi06431743-47 from 2393385 to 2941324 tid 3 2024-03-04T06:12:10.731 INFO:tasks.rados.rados.0.smithi064.stdout:1090: done (3 left) 2024-03-04T06:12:10.731 INFO:tasks.rados.rados.0.smithi064.stdout:1098: write oid 44 current snap is 123 2024-03-04T06:12:10.731 INFO:tasks.rados.rados.0.smithi064.stdout:1098: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2024-03-04T06:12:10.739 INFO:tasks.rados.rados.0.smithi064.stdout:1098: writing smithi06431743-44 from 558131 to 1210558 tid 1 2024-03-04T06:12:10.746 INFO:tasks.rados.rados.0.smithi064.stdout:1098: writing smithi06431743-44 from 1673373 to 2233219 tid 2 2024-03-04T06:12:10.751 INFO:tasks.rados.rados.0.smithi064.stdout:1098: writing smithi06431743-44 from 2820808 to 2958131 tid 3 2024-03-04T06:12:10.751 INFO:tasks.rados.rados.0.smithi064.stdout:1099: write (excl) oid 48 current snap is 123 2024-03-04T06:12:10.751 INFO:tasks.rados.rados.0.smithi064.stdout:1099: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2024-03-04T06:12:10.758 INFO:tasks.rados.rados.0.smithi064.stdout:1099: writing smithi06431743-48 from 574938 to 1302614 tid 1 2024-03-04T06:12:10.766 INFO:tasks.rados.rados.0.smithi064.stdout:1099: writing smithi06431743-48 from 1731855 to 2435359 tid 2 2024-03-04T06:12:10.769 INFO:tasks.rados.rados.0.smithi064.stdout:1099: writing smithi06431743-48 from 2848231 to 2974938 tid 3 2024-03-04T06:12:10.769 INFO:tasks.rados.rados.0.smithi064.stdout:1100: snap_create 2024-03-04T06:12:10.769 INFO:tasks.rados.rados.0.smithi064.stdout:1094: expect (ObjNum 512 snap 118 seq_num 512) 2024-03-04T06:12:10.936 INFO:tasks.rados.rados.0.smithi064.stdout:1095: finishing write tid 1 to smithi06431743-41 2024-03-04T06:12:10.936 INFO:tasks.rados.rados.0.smithi064.stdout:1095: finishing write tid 2 to smithi06431743-41 2024-03-04T06:12:10.936 INFO:tasks.rados.rados.0.smithi064.stdout:1095: finishing write tid 3 to smithi06431743-41 2024-03-04T06:12:10.936 INFO:tasks.rados.rados.0.smithi064.stdout:1095: finishing write tid 4 to smithi06431743-41 2024-03-04T06:12:10.936 INFO:tasks.rados.rados.0.smithi064.stdout:1095: finishing write tid 5 to smithi06431743-41 2024-03-04T06:12:10.936 INFO:tasks.rados.rados.0.smithi064.stdout:1095: oid 41 updating version 0 to 192 2024-03-04T06:12:10.936 INFO:tasks.rados.rados.0.smithi064.stdout:1095: oid 41 version 192 is already newer than 189 2024-03-04T06:12:10.936 INFO:tasks.rados.rados.0.smithi064.stdout:1095: oid 41 version 192 is already newer than 191 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1095: oid 41 version 192 is already newer than 191 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 192 (ObjNum 530 snap 123 seq_num 530) dirty exists 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1095: left oid 41 (ObjNum 530 snap 123 seq_num 530) 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1097: finishing write tid 1 to smithi06431743-47 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1098: finishing write tid 1 to smithi06431743-44 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1098: finishing write tid 2 to smithi06431743-44 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1098: finishing write tid 3 to smithi06431743-44 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1097: finishing write tid 2 to smithi06431743-47 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1098: finishing write tid 4 to smithi06431743-44 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1098: finishing write tid 5 to smithi06431743-44 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1098: oid 44 updating version 0 to 181 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1098: oid 44 version 181 is already newer than 180 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1098: oid 44 version 181 is already newer than 180 2024-03-04T06:12:10.937 INFO:tasks.rados.rados.0.smithi064.stdout:1098: oid 44 version 181 is already newer than 178 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 181 (ObjNum 533 snap 123 seq_num 533) dirty exists 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1098: left oid 44 (ObjNum 533 snap 123 seq_num 533) 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1097: finishing write tid 3 to smithi06431743-47 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1097: finishing write tid 4 to smithi06431743-47 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1097: finishing write tid 5 to smithi06431743-47 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1097: oid 47 updating version 0 to 159 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1097: oid 47 updating version 159 to 160 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1097: oid 47 version 160 is already newer than 157 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1097: oid 47 version 160 is already newer than 159 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 160 (ObjNum 532 snap 123 seq_num 532) dirty exists 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1097: left oid 47 (ObjNum 532 snap 123 seq_num 532) 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1099: finishing write tid 1 to smithi06431743-48 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1099: finishing write tid 2 to smithi06431743-48 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1099: finishing write tid 3 to smithi06431743-48 2024-03-04T06:12:10.938 INFO:tasks.rados.rados.0.smithi064.stdout:1099: finishing write tid 4 to smithi06431743-48 2024-03-04T06:12:10.939 INFO:tasks.rados.rados.0.smithi064.stdout:1099: finishing write tid 5 to smithi06431743-48 2024-03-04T06:12:10.939 INFO:tasks.rados.rados.0.smithi064.stdout:1099: oid 48 updating version 0 to 187 2024-03-04T06:12:10.939 INFO:tasks.rados.rados.0.smithi064.stdout:1099: oid 48 updating version 187 to 189 2024-03-04T06:12:10.939 INFO:tasks.rados.rados.0.smithi064.stdout:1099: oid 48 updating version 189 to 190 2024-03-04T06:12:10.939 INFO:tasks.rados.rados.0.smithi064.stdout:1099: oid 48 version 190 is already newer than 188 2024-03-04T06:12:10.939 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 190 (ObjNum 534 snap 123 seq_num 534) dirty exists 2024-03-04T06:12:10.939 INFO:tasks.rados.rados.0.smithi064.stdout:1099: left oid 48 (ObjNum 534 snap 123 seq_num 534) 2024-03-04T06:12:11.389 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:12:11.392 INFO:tasks.rados.rados.0.smithi064.stdout:1094: done (5 left) 2024-03-04T06:12:11.392 INFO:tasks.rados.rados.0.smithi064.stdout:1095: done (4 left) 2024-03-04T06:12:11.392 INFO:tasks.rados.rados.0.smithi064.stdout:1097: done (3 left) 2024-03-04T06:12:11.392 INFO:tasks.rados.rados.0.smithi064.stdout:1098: done (2 left) 2024-03-04T06:12:11.392 INFO:tasks.rados.rados.0.smithi064.stdout:1099: done (1 left) 2024-03-04T06:12:11.392 INFO:tasks.rados.rados.0.smithi064.stdout:1100: done (0 left) 2024-03-04T06:12:11.392 INFO:tasks.rados.rados.0.smithi064.stdout:1101: write oid 2 current snap is 124 2024-03-04T06:12:11.392 INFO:tasks.rados.rados.0.smithi064.stdout:1101: seq_num 535 ranges {591745=402925,1790337=563515} 2024-03-04T06:12:11.397 INFO:tasks.rados.rados.0.smithi064.stdout:1101: writing smithi06431743-2 from 591745 to 994670 tid 1 2024-03-04T06:12:11.406 INFO:tasks.rados.rados.0.smithi064.stdout:1101: writing smithi06431743-2 from 1790337 to 2353852 tid 2 2024-03-04T06:12:11.406 INFO:tasks.rados.rados.0.smithi064.stdout:1102: read oid 6 snap -1 2024-03-04T06:12:11.406 INFO:tasks.rados.rados.0.smithi064.stdout:1102: expect (ObjNum 434 snap 101 seq_num 434) 2024-03-04T06:12:11.406 INFO:tasks.rados.rados.0.smithi064.stdout:1103: delete oid 13 current snap is 124 2024-03-04T06:12:11.410 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:12:11.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:12:11.441 INFO:tasks.rados.rados.0.smithi064.stdout:1102: expect (ObjNum 434 snap 101 seq_num 434) 2024-03-04T06:12:11.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:11.520+0000 7fab8126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:11.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:11.527+0000 7fab8126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:11.565 INFO:tasks.rados.rados.0.smithi064.stdout:1101: finishing write tid 1 to smithi06431743-2 2024-03-04T06:12:11.565 INFO:tasks.rados.rados.0.smithi064.stdout:1101: finishing write tid 2 to smithi06431743-2 2024-03-04T06:12:11.565 INFO:tasks.rados.rados.0.smithi064.stdout:1101: finishing write tid 3 to smithi06431743-2 2024-03-04T06:12:11.565 INFO:tasks.rados.rados.0.smithi064.stdout:1101: finishing write tid 4 to smithi06431743-2 2024-03-04T06:12:11.565 INFO:tasks.rados.rados.0.smithi064.stdout:1101: oid 2 updating version 0 to 162 2024-03-04T06:12:11.565 INFO:tasks.rados.rados.0.smithi064.stdout:1101: oid 2 version 162 is already newer than 161 2024-03-04T06:12:11.565 INFO:tasks.rados.rados.0.smithi064.stdout:1101: oid 2 updating version 162 to 163 2024-03-04T06:12:11.566 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 163 (ObjNum 535 snap 124 seq_num 535) dirty exists 2024-03-04T06:12:11.566 INFO:tasks.rados.rados.0.smithi064.stdout:1101: left oid 2 (ObjNum 535 snap 124 seq_num 535) 2024-03-04T06:12:11.566 INFO:tasks.rados.rados.0.smithi064.stdout:1101: done (2 left) 2024-03-04T06:12:11.566 INFO:tasks.rados.rados.0.smithi064.stdout:1102: done (1 left) 2024-03-04T06:12:11.566 INFO:tasks.rados.rados.0.smithi064.stdout:1103: done (0 left) 2024-03-04T06:12:11.566 INFO:tasks.rados.rados.0.smithi064.stdout:1104: snap_remove snap 120 2024-03-04T06:12:11.720 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:12:11.721 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:12:11.731 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-03-04T06:12:11.390140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83350709,"num_objects":367,"num_object_clones":266,"num_object_copies":1101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2172,"num_read_kb":1547153,"num_write":1885,"num_write_kb":477704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3510,"ondisk_log_size":3510,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":360740,"kb_used_data":223064,"kb_used_omap":0,"kb_used_meta":136384,"kb_avail":377126620,"statfs":{"total":386547056640,"available":386177658880,"internally_reserved":0,"allocated":228417536,"data_stored":245060303,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":139657216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7916361,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":75,"num_read_kb":45800,"num_write":59,"num_write_kb":15268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.006155"},"pg_stats":[{"pgid":"3.f","version":"258'165","reported_seq":851,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.267898+0000","last_change":"2024-03-04T06:12:09.272246+0000","last_active":"2024-03-04T06:12:09.267898+0000","last_peered":"2024-03-04T06:12:09.267898+0000","last_clean":"2024-03-04T06:12:09.267898+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:12:09.267898+0000","last_undegraded":"2024-03-04T06:12:09.267898+0000","last_fullsized":"2024-03-04T06:12:09.267898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916229,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":66,"num_read_kb":30351,"num_write":98,"num_write_kb":26650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"257'147","reported_seq":858,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.240306+0000","last_change":"2024-03-04T06:12:09.260623+0000","last_active":"2024-03-04T06:12:09.240306+0000","last_peered":"2024-03-04T06:12:09.240306+0000","last_clean":"2024-03-04T06:12:09.240306+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:12:09.240306+0000","last_undegraded":"2024-03-04T06:12:09.240306+0000","last_fullsized":"2024-03-04T06:12:09.240306+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269176,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":63,"num_read_kb":40085,"num_write":72,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"260'247","reported_seq":917,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.313939+0000","last_change":"2024-03-04T06:12:10.335756+0000","last_active":"2024-03-04T06:12:10.313939+0000","last_peered":"2024-03-04T06:12:10.313939+0000","last_clean":"2024-03-04T06:12:10.313939+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:12:10.313939+0000","last_undegraded":"2024-03-04T06:12:10.313939+0000","last_fullsized":"2024-03-04T06:12:10.313939+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193693,"num_objects":29,"num_object_clones":22,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":138,"num_read_kb":88901,"num_write":136,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"258'156","reported_seq":871,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.290612+0000","last_change":"2024-03-04T06:12:09.263036+0000","last_active":"2024-03-04T06:12:09.290612+0000","last_peered":"2024-03-04T06:12:09.290612+0000","last_clean":"2024-03-04T06:12:09.290612+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:12:09.290612+0000","last_undegraded":"2024-03-04T06:12:09.290612+0000","last_fullsized":"2024-03-04T06:12:09.290612+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227142,"num_objects":18,"num_object_clones":11,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":108,"num_read_kb":86737,"num_write":96,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"249'92","reported_seq":792,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.792309+0000","last_change":"2024-03-04T06:12:10.805776+0000","last_active":"2024-03-04T06:12:10.792309+0000","last_peered":"2024-03-04T06:12:10.792309+0000","last_clean":"2024-03-04T06:12:10.792309+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:12:10.792309+0000","last_undegraded":"2024-03-04T06:12:10.792309+0000","last_fullsized":"2024-03-04T06:12:10.792309+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":17041,"num_write":34,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"252'175","reported_seq":873,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.266416+0000","last_change":"2024-03-04T06:12:09.272094+0000","last_active":"2024-03-04T06:12:09.266416+0000","last_peered":"2024-03-04T06:12:09.266416+0000","last_clean":"2024-03-04T06:12:09.266416+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:12:09.266416+0000","last_undegraded":"2024-03-04T06:12:09.266416+0000","last_fullsized":"2024-03-04T06:12:09.266416+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672412,"num_objects":19,"num_object_clones":14,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":110,"num_read_kb":83333,"num_write":101,"num_write_kb":30285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":736,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.265076+0000","last_change":"2024-03-04T06:12:09.271716+0000","last_active":"2024-03-04T06:12:09.265076+0000","last_peered":"2024-03-04T06:12:09.265076+0000","last_clean":"2024-03-04T06:12:09.265076+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:12:09.265076+0000","last_undegraded":"2024-03-04T06:12:09.265076+0000","last_fullsized":"2024-03-04T06:12:09.265076+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"260'150","reported_seq":856,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.235193+0000","last_change":"2024-03-04T06:12:10.285726+0000","last_active":"2024-03-04T06:12:10.235193+0000","last_peered":"2024-03-04T06:12:10.235193+0000","last_clean":"2024-03-04T06:12:10.235193+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:12:10.235193+0000","last_undegraded":"2024-03-04T06:12:10.235193+0000","last_fullsized":"2024-03-04T06:12:10.235193+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252237,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":116,"num_read_kb":111954,"num_write":72,"num_write_kb":19512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":771,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.787578+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:12:10.787578+0000","last_peered":"2024-03-04T06:12:10.787578+0000","last_clean":"2024-03-04T06:12:10.787578+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:12:10.787578+0000","last_undegraded":"2024-03-04T06:12:10.787578+0000","last_fullsized":"2024-03-04T06:12:10.787578+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":628,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.251648+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:12:10.251648+0000","last_peered":"2024-03-04T06:12:10.251648+0000","last_clean":"2024-03-04T06:12:10.251648+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:12:10.251648+0000","last_undegraded":"2024-03-04T06:12:10.251648+0000","last_fullsized":"2024-03-04T06:12:10.251648+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"260'192","reported_seq":889,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.352724+0000","last_change":"2024-03-04T06:12:10.360372+0000","last_active":"2024-03-04T06:12:10.352724+0000","last_peered":"2024-03-04T06:12:10.352724+0000","last_clean":"2024-03-04T06:12:10.352724+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:12:10.352724+0000","last_undegraded":"2024-03-04T06:12:10.352724+0000","last_fullsized":"2024-03-04T06:12:10.352724+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647350,"num_objects":22,"num_object_clones":15,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":136,"num_read_kb":109560,"num_write":99,"num_write_kb":24704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":627,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.269883+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:12:09.269883+0000","last_peered":"2024-03-04T06:12:09.269883+0000","last_clean":"2024-03-04T06:12:09.269883+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:12:09.269883+0000","last_undegraded":"2024-03-04T06:12:09.269883+0000","last_fullsized":"2024-03-04T06:12:09.269883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"260'190","reported_seq":857,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.919526+0000","last_change":"2024-03-04T06:12:10.921210+0000","last_active":"2024-03-04T06:12:10.919526+0000","last_peered":"2024-03-04T06:12:10.919526+0000","last_clean":"2024-03-04T06:12:10.919526+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:12:10.919526+0000","last_undegraded":"2024-03-04T06:12:10.919526+0000","last_fullsized":"2024-03-04T06:12:10.919526+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2974938,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":91,"num_read_kb":69544,"num_write":78,"num_write_kb":22983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":628,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.234217+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:12:10.234217+0000","last_peered":"2024-03-04T06:12:10.234217+0000","last_clean":"2024-03-04T06:12:10.234217+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:12:10.234217+0000","last_undegraded":"2024-03-04T06:12:10.234217+0000","last_fullsized":"2024-03-04T06:12:10.234217+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"258'428","reported_seq":1094,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.794821+0000","last_change":"2024-03-04T06:12:10.806193+0000","last_active":"2024-03-04T06:12:10.794821+0000","last_peered":"2024-03-04T06:12:10.794821+0000","last_clean":"2024-03-04T06:12:10.794821+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:12:10.794821+0000","last_undegraded":"2024-03-04T06:12:10.794821+0000","last_fullsized":"2024-03-04T06:12:10.794821+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8218887,"num_objects":43,"num_object_clones":32,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":308,"num_read_kb":258279,"num_write":230,"num_write_kb":63354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":627,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.257517+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:12:09.257517+0000","last_peered":"2024-03-04T06:12:09.257517+0000","last_clean":"2024-03-04T06:12:09.257517+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:12:09.257517+0000","last_undegraded":"2024-03-04T06:12:09.257517+0000","last_fullsized":"2024-03-04T06:12:09.257517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"257'434","reported_seq":1044,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.288588+0000","last_change":"2024-03-04T06:12:09.261291+0000","last_active":"2024-03-04T06:12:09.288588+0000","last_peered":"2024-03-04T06:12:09.288588+0000","last_clean":"2024-03-04T06:12:09.288588+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:12:09.288588+0000","last_undegraded":"2024-03-04T06:12:09.288588+0000","last_fullsized":"2024-03-04T06:12:09.288588+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11597160,"num_objects":44,"num_object_clones":32,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":197,"num_read_kb":139974,"num_write":215,"num_write_kb":57728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"250'169","reported_seq":896,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.250306+0000","last_change":"2024-03-04T06:12:10.255232+0000","last_active":"2024-03-04T06:12:10.250306+0000","last_peered":"2024-03-04T06:12:10.250306+0000","last_clean":"2024-03-04T06:12:10.250306+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:12:10.250306+0000","last_undegraded":"2024-03-04T06:12:10.250306+0000","last_fullsized":"2024-03-04T06:12:10.250306+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294238,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":175,"num_read_kb":140500,"num_write":86,"num_write_kb":22565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":659,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.263694+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:12:09.263694+0000","last_peered":"2024-03-04T06:12:09.263694+0000","last_clean":"2024-03-04T06:12:09.263694+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:12:09.263694+0000","last_undegraded":"2024-03-04T06:12:09.263694+0000","last_fullsized":"2024-03-04T06:12:09.263694+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"259'194","reported_seq":907,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.326640+0000","last_change":"2024-03-04T06:12:09.268719+0000","last_active":"2024-03-04T06:12:09.326640+0000","last_peered":"2024-03-04T06:12:09.326640+0000","last_clean":"2024-03-04T06:12:09.326640+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:12:09.326640+0000","last_undegraded":"2024-03-04T06:12:09.326640+0000","last_fullsized":"2024-03-04T06:12:09.326640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596929,"num_objects":25,"num_object_clones":16,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":108,"num_read_kb":59205,"num_write":105,"num_write_kb":28287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":638,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.798352+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:12:10.798352+0000","last_peered":"2024-03-04T06:12:10.798352+0000","last_clean":"2024-03-04T06:12:10.798352+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:12:10.798352+0000","last_undegraded":"2024-03-04T06:12:10.798352+0000","last_fullsized":"2024-03-04T06:12:10.798352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"260'288","reported_seq":955,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.786265+0000","last_change":"2024-03-04T06:12:10.821208+0000","last_active":"2024-03-04T06:12:10.786265+0000","last_peered":"2024-03-04T06:12:10.786265+0000","last_clean":"2024-03-04T06:12:10.786265+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:12:10.786265+0000","last_undegraded":"2024-03-04T06:12:10.786265+0000","last_fullsized":"2024-03-04T06:12:10.786265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210500,"num_objects":30,"num_object_clones":22,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":168,"num_read_kb":115116,"num_write":150,"num_write_kb":41216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":659,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.238218+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:12:09.238218+0000","last_peered":"2024-03-04T06:12:09.238218+0000","last_clean":"2024-03-04T06:12:09.238218+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:12:09.238218+0000","last_undegraded":"2024-03-04T06:12:09.238218+0000","last_fullsized":"2024-03-04T06:12:09.238218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"259'386","reported_seq":1040,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.322660+0000","last_change":"2024-03-04T06:12:09.260165+0000","last_active":"2024-03-04T06:12:09.322660+0000","last_peered":"2024-03-04T06:12:09.322660+0000","last_clean":"2024-03-04T06:12:09.322660+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:12:09.322660+0000","last_undegraded":"2024-03-04T06:12:09.322660+0000","last_fullsized":"2024-03-04T06:12:09.322660+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8689450,"num_objects":34,"num_object_clones":30,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":254,"num_read_kb":196482,"num_write":176,"num_write_kb":47586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":628,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.244284+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:12:10.244284+0000","last_peered":"2024-03-04T06:12:10.244284+0000","last_clean":"2024-03-04T06:12:10.244284+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:12:10.244284+0000","last_undegraded":"2024-03-04T06:12:10.244284+0000","last_fullsized":"2024-03-04T06:12:10.244284+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":82760341,"num_objects":365,"num_object_clones":266,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2069,"num_read_kb":1547062,"num_write":1748,"num_write_kb":475810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3413,"ondisk_log_size":3413,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105536,"kb_used_data":69972,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94266304,"statfs":{"total":96636764160,"available":96528695296,"internally_reserved":0,"allocated":71651328,"data_stored":77773194,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90336,"kb_used_data":54772,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94281504,"statfs":{"total":96636764160,"available":96544260096,"internally_reserved":0,"allocated":56086528,"data_stored":59493682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79776,"kb_used_data":44212,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94292064,"statfs":{"total":96636764160,"available":96555073536,"internally_reserved":0,"allocated":45273088,"data_stored":47152927,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85092,"kb_used_data":54108,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94286748,"statfs":{"total":96636764160,"available":96549629952,"internally_reserved":0,"allocated":55406592,"data_stored":60640500,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:12:11.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:12:11.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:11.874+0000 7fdada46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:11.881 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:11.880+0000 7fdada46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:12.070 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:12:12.070 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:12:12.082 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-03-04T06:12:11.390140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83350709,"num_objects":367,"num_object_clones":266,"num_object_copies":1101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2172,"num_read_kb":1547153,"num_write":1885,"num_write_kb":477704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3510,"ondisk_log_size":3510,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":360740,"kb_used_data":223064,"kb_used_omap":0,"kb_used_meta":136384,"kb_avail":377126620,"statfs":{"total":386547056640,"available":386177658880,"internally_reserved":0,"allocated":228417536,"data_stored":245060303,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":139657216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7916361,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":75,"num_read_kb":45800,"num_write":59,"num_write_kb":15268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.006155"},"pg_stats":[{"pgid":"3.f","version":"258'165","reported_seq":851,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.267898+0000","last_change":"2024-03-04T06:12:09.272246+0000","last_active":"2024-03-04T06:12:09.267898+0000","last_peered":"2024-03-04T06:12:09.267898+0000","last_clean":"2024-03-04T06:12:09.267898+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:12:09.267898+0000","last_undegraded":"2024-03-04T06:12:09.267898+0000","last_fullsized":"2024-03-04T06:12:09.267898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916229,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":66,"num_read_kb":30351,"num_write":98,"num_write_kb":26650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"257'147","reported_seq":858,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.240306+0000","last_change":"2024-03-04T06:12:09.260623+0000","last_active":"2024-03-04T06:12:09.240306+0000","last_peered":"2024-03-04T06:12:09.240306+0000","last_clean":"2024-03-04T06:12:09.240306+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:12:09.240306+0000","last_undegraded":"2024-03-04T06:12:09.240306+0000","last_fullsized":"2024-03-04T06:12:09.240306+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269176,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":63,"num_read_kb":40085,"num_write":72,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"260'247","reported_seq":917,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.313939+0000","last_change":"2024-03-04T06:12:10.335756+0000","last_active":"2024-03-04T06:12:10.313939+0000","last_peered":"2024-03-04T06:12:10.313939+0000","last_clean":"2024-03-04T06:12:10.313939+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:12:10.313939+0000","last_undegraded":"2024-03-04T06:12:10.313939+0000","last_fullsized":"2024-03-04T06:12:10.313939+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193693,"num_objects":29,"num_object_clones":22,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":138,"num_read_kb":88901,"num_write":136,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"258'156","reported_seq":871,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.290612+0000","last_change":"2024-03-04T06:12:09.263036+0000","last_active":"2024-03-04T06:12:09.290612+0000","last_peered":"2024-03-04T06:12:09.290612+0000","last_clean":"2024-03-04T06:12:09.290612+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:12:09.290612+0000","last_undegraded":"2024-03-04T06:12:09.290612+0000","last_fullsized":"2024-03-04T06:12:09.290612+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227142,"num_objects":18,"num_object_clones":11,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":108,"num_read_kb":86737,"num_write":96,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"249'92","reported_seq":792,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.792309+0000","last_change":"2024-03-04T06:12:10.805776+0000","last_active":"2024-03-04T06:12:10.792309+0000","last_peered":"2024-03-04T06:12:10.792309+0000","last_clean":"2024-03-04T06:12:10.792309+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:12:10.792309+0000","last_undegraded":"2024-03-04T06:12:10.792309+0000","last_fullsized":"2024-03-04T06:12:10.792309+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":17041,"num_write":34,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"252'175","reported_seq":873,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.266416+0000","last_change":"2024-03-04T06:12:09.272094+0000","last_active":"2024-03-04T06:12:09.266416+0000","last_peered":"2024-03-04T06:12:09.266416+0000","last_clean":"2024-03-04T06:12:09.266416+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:12:09.266416+0000","last_undegraded":"2024-03-04T06:12:09.266416+0000","last_fullsized":"2024-03-04T06:12:09.266416+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672412,"num_objects":19,"num_object_clones":14,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":110,"num_read_kb":83333,"num_write":101,"num_write_kb":30285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":736,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.265076+0000","last_change":"2024-03-04T06:12:09.271716+0000","last_active":"2024-03-04T06:12:09.265076+0000","last_peered":"2024-03-04T06:12:09.265076+0000","last_clean":"2024-03-04T06:12:09.265076+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:12:09.265076+0000","last_undegraded":"2024-03-04T06:12:09.265076+0000","last_fullsized":"2024-03-04T06:12:09.265076+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"260'150","reported_seq":856,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.235193+0000","last_change":"2024-03-04T06:12:10.285726+0000","last_active":"2024-03-04T06:12:10.235193+0000","last_peered":"2024-03-04T06:12:10.235193+0000","last_clean":"2024-03-04T06:12:10.235193+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:12:10.235193+0000","last_undegraded":"2024-03-04T06:12:10.235193+0000","last_fullsized":"2024-03-04T06:12:10.235193+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252237,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":116,"num_read_kb":111954,"num_write":72,"num_write_kb":19512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":771,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.787578+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:12:10.787578+0000","last_peered":"2024-03-04T06:12:10.787578+0000","last_clean":"2024-03-04T06:12:10.787578+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:12:10.787578+0000","last_undegraded":"2024-03-04T06:12:10.787578+0000","last_fullsized":"2024-03-04T06:12:10.787578+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":628,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.251648+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:12:10.251648+0000","last_peered":"2024-03-04T06:12:10.251648+0000","last_clean":"2024-03-04T06:12:10.251648+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:12:10.251648+0000","last_undegraded":"2024-03-04T06:12:10.251648+0000","last_fullsized":"2024-03-04T06:12:10.251648+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"260'192","reported_seq":889,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.352724+0000","last_change":"2024-03-04T06:12:10.360372+0000","last_active":"2024-03-04T06:12:10.352724+0000","last_peered":"2024-03-04T06:12:10.352724+0000","last_clean":"2024-03-04T06:12:10.352724+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:12:10.352724+0000","last_undegraded":"2024-03-04T06:12:10.352724+0000","last_fullsized":"2024-03-04T06:12:10.352724+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647350,"num_objects":22,"num_object_clones":15,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":136,"num_read_kb":109560,"num_write":99,"num_write_kb":24704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":627,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.269883+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:12:09.269883+0000","last_peered":"2024-03-04T06:12:09.269883+0000","last_clean":"2024-03-04T06:12:09.269883+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:12:09.269883+0000","last_undegraded":"2024-03-04T06:12:09.269883+0000","last_fullsized":"2024-03-04T06:12:09.269883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"260'190","reported_seq":857,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.919526+0000","last_change":"2024-03-04T06:12:10.921210+0000","last_active":"2024-03-04T06:12:10.919526+0000","last_peered":"2024-03-04T06:12:10.919526+0000","last_clean":"2024-03-04T06:12:10.919526+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:12:10.919526+0000","last_undegraded":"2024-03-04T06:12:10.919526+0000","last_fullsized":"2024-03-04T06:12:10.919526+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2974938,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":91,"num_read_kb":69544,"num_write":78,"num_write_kb":22983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":628,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.234217+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:12:10.234217+0000","last_peered":"2024-03-04T06:12:10.234217+0000","last_clean":"2024-03-04T06:12:10.234217+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:12:10.234217+0000","last_undegraded":"2024-03-04T06:12:10.234217+0000","last_fullsized":"2024-03-04T06:12:10.234217+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"258'428","reported_seq":1094,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.794821+0000","last_change":"2024-03-04T06:12:10.806193+0000","last_active":"2024-03-04T06:12:10.794821+0000","last_peered":"2024-03-04T06:12:10.794821+0000","last_clean":"2024-03-04T06:12:10.794821+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:12:10.794821+0000","last_undegraded":"2024-03-04T06:12:10.794821+0000","last_fullsized":"2024-03-04T06:12:10.794821+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8218887,"num_objects":43,"num_object_clones":32,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":308,"num_read_kb":258279,"num_write":230,"num_write_kb":63354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":627,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.257517+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:12:09.257517+0000","last_peered":"2024-03-04T06:12:09.257517+0000","last_clean":"2024-03-04T06:12:09.257517+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:12:09.257517+0000","last_undegraded":"2024-03-04T06:12:09.257517+0000","last_fullsized":"2024-03-04T06:12:09.257517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"257'434","reported_seq":1044,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.288588+0000","last_change":"2024-03-04T06:12:09.261291+0000","last_active":"2024-03-04T06:12:09.288588+0000","last_peered":"2024-03-04T06:12:09.288588+0000","last_clean":"2024-03-04T06:12:09.288588+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:12:09.288588+0000","last_undegraded":"2024-03-04T06:12:09.288588+0000","last_fullsized":"2024-03-04T06:12:09.288588+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11597160,"num_objects":44,"num_object_clones":32,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":197,"num_read_kb":139974,"num_write":215,"num_write_kb":57728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"250'169","reported_seq":896,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.250306+0000","last_change":"2024-03-04T06:12:10.255232+0000","last_active":"2024-03-04T06:12:10.250306+0000","last_peered":"2024-03-04T06:12:10.250306+0000","last_clean":"2024-03-04T06:12:10.250306+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:12:10.250306+0000","last_undegraded":"2024-03-04T06:12:10.250306+0000","last_fullsized":"2024-03-04T06:12:10.250306+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294238,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":175,"num_read_kb":140500,"num_write":86,"num_write_kb":22565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":659,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.263694+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:12:09.263694+0000","last_peered":"2024-03-04T06:12:09.263694+0000","last_clean":"2024-03-04T06:12:09.263694+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:12:09.263694+0000","last_undegraded":"2024-03-04T06:12:09.263694+0000","last_fullsized":"2024-03-04T06:12:09.263694+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"259'194","reported_seq":907,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.326640+0000","last_change":"2024-03-04T06:12:09.268719+0000","last_active":"2024-03-04T06:12:09.326640+0000","last_peered":"2024-03-04T06:12:09.326640+0000","last_clean":"2024-03-04T06:12:09.326640+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:12:09.326640+0000","last_undegraded":"2024-03-04T06:12:09.326640+0000","last_fullsized":"2024-03-04T06:12:09.326640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596929,"num_objects":25,"num_object_clones":16,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":108,"num_read_kb":59205,"num_write":105,"num_write_kb":28287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":638,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.798352+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:12:10.798352+0000","last_peered":"2024-03-04T06:12:10.798352+0000","last_clean":"2024-03-04T06:12:10.798352+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:12:10.798352+0000","last_undegraded":"2024-03-04T06:12:10.798352+0000","last_fullsized":"2024-03-04T06:12:10.798352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"260'288","reported_seq":955,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.786265+0000","last_change":"2024-03-04T06:12:10.821208+0000","last_active":"2024-03-04T06:12:10.786265+0000","last_peered":"2024-03-04T06:12:10.786265+0000","last_clean":"2024-03-04T06:12:10.786265+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:12:10.786265+0000","last_undegraded":"2024-03-04T06:12:10.786265+0000","last_fullsized":"2024-03-04T06:12:10.786265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210500,"num_objects":30,"num_object_clones":22,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":168,"num_read_kb":115116,"num_write":150,"num_write_kb":41216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":659,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.238218+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:12:09.238218+0000","last_peered":"2024-03-04T06:12:09.238218+0000","last_clean":"2024-03-04T06:12:09.238218+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:12:09.238218+0000","last_undegraded":"2024-03-04T06:12:09.238218+0000","last_fullsized":"2024-03-04T06:12:09.238218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"259'386","reported_seq":1040,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.322660+0000","last_change":"2024-03-04T06:12:09.260165+0000","last_active":"2024-03-04T06:12:09.322660+0000","last_peered":"2024-03-04T06:12:09.322660+0000","last_clean":"2024-03-04T06:12:09.322660+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:12:09.322660+0000","last_undegraded":"2024-03-04T06:12:09.322660+0000","last_fullsized":"2024-03-04T06:12:09.322660+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8689450,"num_objects":34,"num_object_clones":30,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":254,"num_read_kb":196482,"num_write":176,"num_write_kb":47586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":628,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.244284+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:12:10.244284+0000","last_peered":"2024-03-04T06:12:10.244284+0000","last_clean":"2024-03-04T06:12:10.244284+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:12:10.244284+0000","last_undegraded":"2024-03-04T06:12:10.244284+0000","last_fullsized":"2024-03-04T06:12:10.244284+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":82760341,"num_objects":365,"num_object_clones":266,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2069,"num_read_kb":1547062,"num_write":1748,"num_write_kb":475810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3413,"ondisk_log_size":3413,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105536,"kb_used_data":69972,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94266304,"statfs":{"total":96636764160,"available":96528695296,"internally_reserved":0,"allocated":71651328,"data_stored":77773194,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90336,"kb_used_data":54772,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94281504,"statfs":{"total":96636764160,"available":96544260096,"internally_reserved":0,"allocated":56086528,"data_stored":59493682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79776,"kb_used_data":44212,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94292064,"statfs":{"total":96636764160,"available":96555073536,"internally_reserved":0,"allocated":45273088,"data_stored":47152927,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85092,"kb_used_data":54108,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94286748,"statfs":{"total":96636764160,"available":96549629952,"internally_reserved":0,"allocated":55406592,"data_stored":60640500,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:12:12.083 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:12:12.084 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:12:12.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:12:12.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:12.186+0000 7fd194e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:12.193 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:12.193+0000 7fd194e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:12.383 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:12:12.383 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:12:12.393 INFO:tasks.rados.rados.0.smithi064.stdout:1104: done (0 left) 2024-03-04T06:12:12.394 INFO:tasks.rados.rados.0.smithi064.stdout:1105: snap_remove snap 117 2024-03-04T06:12:12.396 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-03-04T06:12:11.390140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83350709,"num_objects":367,"num_object_clones":266,"num_object_copies":1101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2172,"num_read_kb":1547153,"num_write":1885,"num_write_kb":477704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3510,"ondisk_log_size":3510,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":360740,"kb_used_data":223064,"kb_used_omap":0,"kb_used_meta":136384,"kb_avail":377126620,"statfs":{"total":386547056640,"available":386177658880,"internally_reserved":0,"allocated":228417536,"data_stored":245060303,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":139657216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7916361,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":75,"num_read_kb":45800,"num_write":59,"num_write_kb":15268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.006155"},"pg_stats":[{"pgid":"3.f","version":"258'165","reported_seq":851,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.267898+0000","last_change":"2024-03-04T06:12:09.272246+0000","last_active":"2024-03-04T06:12:09.267898+0000","last_peered":"2024-03-04T06:12:09.267898+0000","last_clean":"2024-03-04T06:12:09.267898+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:12:09.267898+0000","last_undegraded":"2024-03-04T06:12:09.267898+0000","last_fullsized":"2024-03-04T06:12:09.267898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916229,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":66,"num_read_kb":30351,"num_write":98,"num_write_kb":26650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"257'147","reported_seq":858,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.240306+0000","last_change":"2024-03-04T06:12:09.260623+0000","last_active":"2024-03-04T06:12:09.240306+0000","last_peered":"2024-03-04T06:12:09.240306+0000","last_clean":"2024-03-04T06:12:09.240306+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:12:09.240306+0000","last_undegraded":"2024-03-04T06:12:09.240306+0000","last_fullsized":"2024-03-04T06:12:09.240306+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269176,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":63,"num_read_kb":40085,"num_write":72,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"260'247","reported_seq":917,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.313939+0000","last_change":"2024-03-04T06:12:10.335756+0000","last_active":"2024-03-04T06:12:10.313939+0000","last_peered":"2024-03-04T06:12:10.313939+0000","last_clean":"2024-03-04T06:12:10.313939+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:12:10.313939+0000","last_undegraded":"2024-03-04T06:12:10.313939+0000","last_fullsized":"2024-03-04T06:12:10.313939+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193693,"num_objects":29,"num_object_clones":22,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":138,"num_read_kb":88901,"num_write":136,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"258'156","reported_seq":871,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.290612+0000","last_change":"2024-03-04T06:12:09.263036+0000","last_active":"2024-03-04T06:12:09.290612+0000","last_peered":"2024-03-04T06:12:09.290612+0000","last_clean":"2024-03-04T06:12:09.290612+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:12:09.290612+0000","last_undegraded":"2024-03-04T06:12:09.290612+0000","last_fullsized":"2024-03-04T06:12:09.290612+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227142,"num_objects":18,"num_object_clones":11,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":108,"num_read_kb":86737,"num_write":96,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"249'92","reported_seq":792,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.792309+0000","last_change":"2024-03-04T06:12:10.805776+0000","last_active":"2024-03-04T06:12:10.792309+0000","last_peered":"2024-03-04T06:12:10.792309+0000","last_clean":"2024-03-04T06:12:10.792309+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:12:10.792309+0000","last_undegraded":"2024-03-04T06:12:10.792309+0000","last_fullsized":"2024-03-04T06:12:10.792309+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":17041,"num_write":34,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"252'175","reported_seq":873,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.266416+0000","last_change":"2024-03-04T06:12:09.272094+0000","last_active":"2024-03-04T06:12:09.266416+0000","last_peered":"2024-03-04T06:12:09.266416+0000","last_clean":"2024-03-04T06:12:09.266416+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:12:09.266416+0000","last_undegraded":"2024-03-04T06:12:09.266416+0000","last_fullsized":"2024-03-04T06:12:09.266416+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672412,"num_objects":19,"num_object_clones":14,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":110,"num_read_kb":83333,"num_write":101,"num_write_kb":30285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":736,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.265076+0000","last_change":"2024-03-04T06:12:09.271716+0000","last_active":"2024-03-04T06:12:09.265076+0000","last_peered":"2024-03-04T06:12:09.265076+0000","last_clean":"2024-03-04T06:12:09.265076+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:12:09.265076+0000","last_undegraded":"2024-03-04T06:12:09.265076+0000","last_fullsized":"2024-03-04T06:12:09.265076+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"260'150","reported_seq":856,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.235193+0000","last_change":"2024-03-04T06:12:10.285726+0000","last_active":"2024-03-04T06:12:10.235193+0000","last_peered":"2024-03-04T06:12:10.235193+0000","last_clean":"2024-03-04T06:12:10.235193+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:12:10.235193+0000","last_undegraded":"2024-03-04T06:12:10.235193+0000","last_fullsized":"2024-03-04T06:12:10.235193+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252237,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":116,"num_read_kb":111954,"num_write":72,"num_write_kb":19512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":771,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.787578+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:12:10.787578+0000","last_peered":"2024-03-04T06:12:10.787578+0000","last_clean":"2024-03-04T06:12:10.787578+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:12:10.787578+0000","last_undegraded":"2024-03-04T06:12:10.787578+0000","last_fullsized":"2024-03-04T06:12:10.787578+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":628,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.251648+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:12:10.251648+0000","last_peered":"2024-03-04T06:12:10.251648+0000","last_clean":"2024-03-04T06:12:10.251648+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:12:10.251648+0000","last_undegraded":"2024-03-04T06:12:10.251648+0000","last_fullsized":"2024-03-04T06:12:10.251648+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"260'192","reported_seq":889,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.352724+0000","last_change":"2024-03-04T06:12:10.360372+0000","last_active":"2024-03-04T06:12:10.352724+0000","last_peered":"2024-03-04T06:12:10.352724+0000","last_clean":"2024-03-04T06:12:10.352724+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:12:10.352724+0000","last_undegraded":"2024-03-04T06:12:10.352724+0000","last_fullsized":"2024-03-04T06:12:10.352724+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647350,"num_objects":22,"num_object_clones":15,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":136,"num_read_kb":109560,"num_write":99,"num_write_kb":24704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":627,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.269883+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:12:09.269883+0000","last_peered":"2024-03-04T06:12:09.269883+0000","last_clean":"2024-03-04T06:12:09.269883+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:12:09.269883+0000","last_undegraded":"2024-03-04T06:12:09.269883+0000","last_fullsized":"2024-03-04T06:12:09.269883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"260'190","reported_seq":857,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.919526+0000","last_change":"2024-03-04T06:12:10.921210+0000","last_active":"2024-03-04T06:12:10.919526+0000","last_peered":"2024-03-04T06:12:10.919526+0000","last_clean":"2024-03-04T06:12:10.919526+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:12:10.919526+0000","last_undegraded":"2024-03-04T06:12:10.919526+0000","last_fullsized":"2024-03-04T06:12:10.919526+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2974938,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":91,"num_read_kb":69544,"num_write":78,"num_write_kb":22983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":628,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.234217+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:12:10.234217+0000","last_peered":"2024-03-04T06:12:10.234217+0000","last_clean":"2024-03-04T06:12:10.234217+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:12:10.234217+0000","last_undegraded":"2024-03-04T06:12:10.234217+0000","last_fullsized":"2024-03-04T06:12:10.234217+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"258'428","reported_seq":1094,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.794821+0000","last_change":"2024-03-04T06:12:10.806193+0000","last_active":"2024-03-04T06:12:10.794821+0000","last_peered":"2024-03-04T06:12:10.794821+0000","last_clean":"2024-03-04T06:12:10.794821+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:12:10.794821+0000","last_undegraded":"2024-03-04T06:12:10.794821+0000","last_fullsized":"2024-03-04T06:12:10.794821+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8218887,"num_objects":43,"num_object_clones":32,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":308,"num_read_kb":258279,"num_write":230,"num_write_kb":63354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":627,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.257517+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:12:09.257517+0000","last_peered":"2024-03-04T06:12:09.257517+0000","last_clean":"2024-03-04T06:12:09.257517+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:12:09.257517+0000","last_undegraded":"2024-03-04T06:12:09.257517+0000","last_fullsized":"2024-03-04T06:12:09.257517+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"257'434","reported_seq":1044,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.288588+0000","last_change":"2024-03-04T06:12:09.261291+0000","last_active":"2024-03-04T06:12:09.288588+0000","last_peered":"2024-03-04T06:12:09.288588+0000","last_clean":"2024-03-04T06:12:09.288588+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:12:09.288588+0000","last_undegraded":"2024-03-04T06:12:09.288588+0000","last_fullsized":"2024-03-04T06:12:09.288588+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11597160,"num_objects":44,"num_object_clones":32,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":197,"num_read_kb":139974,"num_write":215,"num_write_kb":57728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"250'169","reported_seq":896,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.250306+0000","last_change":"2024-03-04T06:12:10.255232+0000","last_active":"2024-03-04T06:12:10.250306+0000","last_peered":"2024-03-04T06:12:10.250306+0000","last_clean":"2024-03-04T06:12:10.250306+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:12:10.250306+0000","last_undegraded":"2024-03-04T06:12:10.250306+0000","last_fullsized":"2024-03-04T06:12:10.250306+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294238,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":175,"num_read_kb":140500,"num_write":86,"num_write_kb":22565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":659,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.263694+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:12:09.263694+0000","last_peered":"2024-03-04T06:12:09.263694+0000","last_clean":"2024-03-04T06:12:09.263694+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:12:09.263694+0000","last_undegraded":"2024-03-04T06:12:09.263694+0000","last_fullsized":"2024-03-04T06:12:09.263694+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"259'194","reported_seq":907,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.326640+0000","last_change":"2024-03-04T06:12:09.268719+0000","last_active":"2024-03-04T06:12:09.326640+0000","last_peered":"2024-03-04T06:12:09.326640+0000","last_clean":"2024-03-04T06:12:09.326640+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:12:09.326640+0000","last_undegraded":"2024-03-04T06:12:09.326640+0000","last_fullsized":"2024-03-04T06:12:09.326640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596929,"num_objects":25,"num_object_clones":16,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":108,"num_read_kb":59205,"num_write":105,"num_write_kb":28287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":638,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.798352+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:12:10.798352+0000","last_peered":"2024-03-04T06:12:10.798352+0000","last_clean":"2024-03-04T06:12:10.798352+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:12:10.798352+0000","last_undegraded":"2024-03-04T06:12:10.798352+0000","last_fullsized":"2024-03-04T06:12:10.798352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"260'288","reported_seq":955,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.786265+0000","last_change":"2024-03-04T06:12:10.821208+0000","last_active":"2024-03-04T06:12:10.786265+0000","last_peered":"2024-03-04T06:12:10.786265+0000","last_clean":"2024-03-04T06:12:10.786265+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:12:10.786265+0000","last_undegraded":"2024-03-04T06:12:10.786265+0000","last_fullsized":"2024-03-04T06:12:10.786265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210500,"num_objects":30,"num_object_clones":22,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":168,"num_read_kb":115116,"num_write":150,"num_write_kb":41216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":659,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.238218+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:12:09.238218+0000","last_peered":"2024-03-04T06:12:09.238218+0000","last_clean":"2024-03-04T06:12:09.238218+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:12:09.238218+0000","last_undegraded":"2024-03-04T06:12:09.238218+0000","last_fullsized":"2024-03-04T06:12:09.238218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"259'386","reported_seq":1040,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-04T06:12:09.322660+0000","last_change":"2024-03-04T06:12:09.260165+0000","last_active":"2024-03-04T06:12:09.322660+0000","last_peered":"2024-03-04T06:12:09.322660+0000","last_clean":"2024-03-04T06:12:09.322660+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:12:09.322660+0000","last_undegraded":"2024-03-04T06:12:09.322660+0000","last_fullsized":"2024-03-04T06:12:09.322660+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8689450,"num_objects":34,"num_object_clones":30,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":254,"num_read_kb":196482,"num_write":176,"num_write_kb":47586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":628,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-04T06:12:10.244284+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:12:10.244284+0000","last_peered":"2024-03-04T06:12:10.244284+0000","last_clean":"2024-03-04T06:12:10.244284+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:12:10.244284+0000","last_undegraded":"2024-03-04T06:12:10.244284+0000","last_fullsized":"2024-03-04T06:12:10.244284+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":82760341,"num_objects":365,"num_object_clones":266,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2069,"num_read_kb":1547062,"num_write":1748,"num_write_kb":475810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3413,"ondisk_log_size":3413,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105536,"kb_used_data":69972,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94266304,"statfs":{"total":96636764160,"available":96528695296,"internally_reserved":0,"allocated":71651328,"data_stored":77773194,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90336,"kb_used_data":54772,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94281504,"statfs":{"total":96636764160,"available":96544260096,"internally_reserved":0,"allocated":56086528,"data_stored":59493682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79776,"kb_used_data":44212,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94292064,"statfs":{"total":96636764160,"available":96555073536,"internally_reserved":0,"allocated":45273088,"data_stored":47152927,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":85092,"kb_used_data":54108,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94286748,"statfs":{"total":96636764160,"available":96549629952,"internally_reserved":0,"allocated":55406592,"data_stored":60640500,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:12:12.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:12.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:12.545+0000 7f322e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:12.549+0000 7f322e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:13.399 INFO:tasks.rados.rados.0.smithi064.stdout:1105: done (0 left) 2024-03-04T06:12:13.399 INFO:tasks.rados.rados.0.smithi064.stdout:1106: snap_create 2024-03-04T06:12:14.402 INFO:tasks.rados.rados.0.smithi064.stdout:1106: done (0 left) 2024-03-04T06:12:14.402 INFO:tasks.rados.rados.0.smithi064.stdout:1107: write (excl) oid 40 current snap is 125 2024-03-04T06:12:14.402 INFO:tasks.rados.rados.0.smithi064.stdout:1107: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2024-03-04T06:12:14.408 INFO:tasks.rados.rados.0.smithi064.stdout:1107: writing smithi06431743-40 from 625359 to 1178782 tid 1 2024-03-04T06:12:14.415 INFO:tasks.rados.rados.0.smithi064.stdout:1107: writing smithi06431743-40 from 1790948 to 2358132 tid 2 2024-03-04T06:12:14.419 INFO:tasks.rados.rados.0.smithi064.stdout:1107: writing smithi06431743-40 from 2930500 to 3025359 tid 3 2024-03-04T06:12:14.420 INFO:tasks.rados.rados.0.smithi064.stdout:1108: write (excl) oid 48 current snap is 125 2024-03-04T06:12:14.420 INFO:tasks.rados.rados.0.smithi064.stdout:1108: seq_num 538 ranges {642166=628672,1849430=710842} 2024-03-04T06:12:14.429 INFO:tasks.rados.rados.0.smithi064.stdout:1108: writing smithi06431743-48 from 642166 to 1270838 tid 1 2024-03-04T06:12:14.439 INFO:tasks.rados.rados.0.smithi064.stdout:1108: writing smithi06431743-48 from 1849430 to 2560272 tid 2 2024-03-04T06:12:14.439 INFO:tasks.rados.rados.0.smithi064.stdout:1109: write (excl) oid 13 current snap is 125 2024-03-04T06:12:14.439 INFO:tasks.rados.rados.0.smithi064.stdout:1109: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2024-03-04T06:12:14.448 INFO:tasks.rados.rados.0.smithi064.stdout:1109: writing smithi06431743-13 from 658973 to 1362894 tid 1 2024-03-04T06:12:14.454 INFO:tasks.rados.rados.0.smithi064.stdout:1109: writing smithi06431743-13 from 1907912 to 2478765 tid 2 2024-03-04T06:12:14.457 INFO:tasks.rados.rados.0.smithi064.stdout:1109: writing smithi06431743-13 from 2985346 to 3058973 tid 3 2024-03-04T06:12:14.458 INFO:tasks.rados.rados.0.smithi064.stdout:1110: write (excl) oid 43 current snap is 125 2024-03-04T06:12:14.458 INFO:tasks.rados.rados.0.smithi064.stdout:1110: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2024-03-04T06:12:14.465 INFO:tasks.rados.rados.0.smithi064.stdout:1110: writing smithi06431743-43 from 675780 to 1171303 tid 1 2024-03-04T06:12:14.472 INFO:tasks.rados.rados.0.smithi064.stdout:1110: writing smithi06431743-43 from 1682747 to 2397258 tid 2 2024-03-04T06:12:14.477 INFO:tasks.rados.rados.0.smithi064.stdout:1110: writing smithi06431743-43 from 3012769 to 3075780 tid 3 2024-03-04T06:12:14.477 INFO:tasks.rados.rados.0.smithi064.stdout:1111: snap_remove snap 115 2024-03-04T06:12:14.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:14.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:14.720+0000 7fd6abb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:14.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:14.724+0000 7fd6abb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:15.416 INFO:tasks.rados.rados.0.smithi064.stdout:1111: done (4 left) 2024-03-04T06:12:15.416 INFO:tasks.rados.rados.0.smithi064.stdout:1112: write oid 3 current snap is 125 2024-03-04T06:12:15.416 INFO:tasks.rados.rados.0.smithi064.stdout:1107: finishing write tid 1 to smithi06431743-40 2024-03-04T06:12:15.416 INFO:tasks.rados.rados.0.smithi064.stdout:1112: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2024-03-04T06:12:15.422 INFO:tasks.rados.rados.0.smithi064.stdout:1112: writing smithi06431743-3 from 692587 to 1263359 tid 1 2024-03-04T06:12:15.431 INFO:tasks.rados.rados.0.smithi064.stdout:1112: writing smithi06431743-3 from 2024876 to 2599398 tid 2 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1112: writing smithi06431743-3 from 3040192 to 3092587 tid 3 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1113: snap_create 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1109: finishing write tid 1 to smithi06431743-13 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1108: finishing write tid 1 to smithi06431743-48 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1107: finishing write tid 2 to smithi06431743-40 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1109: finishing write tid 2 to smithi06431743-13 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1108: finishing write tid 2 to smithi06431743-48 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1108: finishing write tid 3 to smithi06431743-48 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1108: finishing write tid 4 to smithi06431743-48 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1108: oid 48 updating version 0 to 196 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1108: oid 48 version 196 is already newer than 194 2024-03-04T06:12:15.435 INFO:tasks.rados.rados.0.smithi064.stdout:1108: oid 48 version 196 is already newer than 195 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 196 (ObjNum 538 snap 125 seq_num 538) dirty exists 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:1108: left oid 48 (ObjNum 538 snap 125 seq_num 538) 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:1107: finishing write tid 3 to smithi06431743-40 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:1109: finishing write tid 3 to smithi06431743-13 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:1109: finishing write tid 4 to smithi06431743-13 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:1107: finishing write tid 4 to smithi06431743-40 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:1109: finishing write tid 5 to smithi06431743-13 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:1109: oid 13 updating version 0 to 447 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:1109: oid 13 updating version 447 to 449 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:1109: oid 13 version 449 is already newer than 448 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:1109: oid 13 updating version 449 to 450 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 450 (ObjNum 539 snap 125 seq_num 539) dirty exists 2024-03-04T06:12:15.436 INFO:tasks.rados.rados.0.smithi064.stdout:1109: left oid 13 (ObjNum 539 snap 125 seq_num 539) 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1107: finishing write tid 5 to smithi06431743-40 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1107: oid 40 updating version 0 to 172 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1107: oid 40 version 172 is already newer than 171 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1107: oid 40 updating version 172 to 173 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1107: oid 40 version 173 is already newer than 170 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 173 (ObjNum 537 snap 125 seq_num 537) dirty exists 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1107: left oid 40 (ObjNum 537 snap 125 seq_num 537) 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1110: finishing write tid 1 to smithi06431743-43 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1110: finishing write tid 2 to smithi06431743-43 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1110: finishing write tid 3 to smithi06431743-43 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1110: finishing write tid 4 to smithi06431743-43 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1110: finishing write tid 5 to smithi06431743-43 2024-03-04T06:12:15.437 INFO:tasks.rados.rados.0.smithi064.stdout:1110: oid 43 updating version 0 to 178 2024-03-04T06:12:15.438 INFO:tasks.rados.rados.0.smithi064.stdout:1110: oid 43 version 178 is already newer than 178 2024-03-04T06:12:15.438 INFO:tasks.rados.rados.0.smithi064.stdout:1110: oid 43 version 178 is already newer than 178 2024-03-04T06:12:15.438 INFO:tasks.rados.rados.0.smithi064.stdout:1110: oid 43 version 178 is already newer than 175 2024-03-04T06:12:15.438 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 178 (ObjNum 540 snap 125 seq_num 540) dirty exists 2024-03-04T06:12:15.438 INFO:tasks.rados.rados.0.smithi064.stdout:1110: left oid 43 (ObjNum 540 snap 125 seq_num 540) 2024-03-04T06:12:15.492 INFO:tasks.rados.rados.0.smithi064.stdout:1112: finishing write tid 1 to smithi06431743-3 2024-03-04T06:12:15.499 INFO:tasks.rados.rados.0.smithi064.stdout:1112: finishing write tid 2 to smithi06431743-3 2024-03-04T06:12:15.506 INFO:tasks.rados.rados.0.smithi064.stdout:1112: finishing write tid 3 to smithi06431743-3 2024-03-04T06:12:15.514 INFO:tasks.rados.rados.0.smithi064.stdout:1112: finishing write tid 4 to smithi06431743-3 2024-03-04T06:12:15.515 INFO:tasks.rados.rados.0.smithi064.stdout:1112: finishing write tid 5 to smithi06431743-3 2024-03-04T06:12:15.515 INFO:tasks.rados.rados.0.smithi064.stdout:1112: oid 3 updating version 0 to 201 2024-03-04T06:12:15.515 INFO:tasks.rados.rados.0.smithi064.stdout:1112: oid 3 updating version 201 to 203 2024-03-04T06:12:15.515 INFO:tasks.rados.rados.0.smithi064.stdout:1112: oid 3 updating version 203 to 204 2024-03-04T06:12:15.515 INFO:tasks.rados.rados.0.smithi064.stdout:1112: oid 3 version 204 is already newer than 202 2024-03-04T06:12:15.515 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 204 (ObjNum 541 snap 125 seq_num 541) dirty exists 2024-03-04T06:12:15.515 INFO:tasks.rados.rados.0.smithi064.stdout:1112: left oid 3 (ObjNum 541 snap 125 seq_num 541) 2024-03-04T06:12:16.417 INFO:tasks.rados.rados.0.smithi064.stdout:1107: done (5 left) 2024-03-04T06:12:16.418 INFO:tasks.rados.rados.0.smithi064.stdout:1108: done (4 left) 2024-03-04T06:12:16.418 INFO:tasks.rados.rados.0.smithi064.stdout:1109: done (3 left) 2024-03-04T06:12:16.418 INFO:tasks.rados.rados.0.smithi064.stdout:1110: done (2 left) 2024-03-04T06:12:16.418 INFO:tasks.rados.rados.0.smithi064.stdout:1112: done (1 left) 2024-03-04T06:12:16.418 INFO:tasks.rados.rados.0.smithi064.stdout:1113: done (0 left) 2024-03-04T06:12:16.418 INFO:tasks.rados.rados.0.smithi064.stdout:1114: snap_create 2024-03-04T06:12:16.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:16.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:16.906+0000 7f8ab619d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:16.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:16.911+0000 7f8ab619d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:17.398 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:12:17.398 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:12:17.398 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:12:17.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:12:17.421 INFO:tasks.rados.rados.0.smithi064.stdout:1114: done (0 left) 2024-03-04T06:12:17.421 INFO:tasks.rados.rados.0.smithi064.stdout:1115: read oid 17 snap -1 2024-03-04T06:12:17.421 INFO:tasks.rados.rados.0.smithi064.stdout:1115: expect deleted 2024-03-04T06:12:17.421 INFO:tasks.rados.rados.0.smithi064.stdout:1116: write oid 12 current snap is 127 2024-03-04T06:12:17.422 INFO:tasks.rados.rados.0.smithi064.stdout:1116: seq_num 542 ranges {709394=646021,2083358=718180} 2024-03-04T06:12:17.429 INFO:tasks.rados.rados.0.smithi064.stdout:1116: writing smithi06431743-12 from 709394 to 1355415 tid 1 2024-03-04T06:12:17.437 INFO:tasks.rados.rados.0.smithi064.stdout:1116: writing smithi06431743-12 from 2083358 to 2801538 tid 2 2024-03-04T06:12:17.437 INFO:tasks.rados.rados.0.smithi064.stdout:1117: read oid 7 snap -1 2024-03-04T06:12:17.438 INFO:tasks.rados.rados.0.smithi064.stdout:1117: expect deleted 2024-03-04T06:12:17.438 INFO:tasks.rados.rados.0.smithi064.stdout:1118: snap_remove snap 123 2024-03-04T06:12:17.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:17.517+0000 7f0ea499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:17.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:17.525+0000 7f0ea499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:17.539 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:17.539 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:17.539 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:17.539 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:17.540 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:17.540 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:17.540 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:17.540 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:17.540 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:17.540 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:17.540 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:17.540 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:17.543 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:17.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:12:17.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:17.685+0000 7f5837e2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:17.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:17.693+0000 7f5837e2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:17.705 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:17.705 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:17.705 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:17.705 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:17.705 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:17.705 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:17.705 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:17.706 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:17.706 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:17.706 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:17.706 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:17.706 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:17.708 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:17.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:12:17.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:17.811+0000 7fdaee66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:17.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:17.818+0000 7fdaee66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:17.832 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:17.832 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:17.832 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:17.832 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:17.832 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:17.832 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:17.832 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:17.832 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:17.833 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:17.833 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:17.833 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:17.833 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:17.834 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:17.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:12:17.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:17.951+0000 7f276c017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:17.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:17.961+0000 7f276c017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:17.976 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:18.212 INFO:tasks.rados.rados.0.smithi064.stdout:1118: done (3 left) 2024-03-04T06:12:18.212 INFO:tasks.rados.rados.0.smithi064.stdout:1119: snap_create 2024-03-04T06:12:18.212 INFO:tasks.rados.rados.0.smithi064.stdout:1116: finishing write tid 1 to smithi06431743-12 2024-03-04T06:12:18.212 INFO:tasks.rados.rados.0.smithi064.stdout:1116: finishing write tid 2 to smithi06431743-12 2024-03-04T06:12:18.212 INFO:tasks.rados.rados.0.smithi064.stdout:1116: finishing write tid 3 to smithi06431743-12 2024-03-04T06:12:18.212 INFO:tasks.rados.rados.0.smithi064.stdout:1116: finishing write tid 4 to smithi06431743-12 2024-03-04T06:12:18.213 INFO:tasks.rados.rados.0.smithi064.stdout:1116: oid 12 updating version 0 to 168 2024-03-04T06:12:18.213 INFO:tasks.rados.rados.0.smithi064.stdout:1116: oid 12 updating version 168 to 169 2024-03-04T06:12:18.213 INFO:tasks.rados.rados.0.smithi064.stdout:1116: oid 12 version 169 is already newer than 167 2024-03-04T06:12:18.213 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 169 (ObjNum 542 snap 127 seq_num 542) dirty exists 2024-03-04T06:12:18.213 INFO:tasks.rados.rados.0.smithi064.stdout:1116: left oid 12 (ObjNum 542 snap 127 seq_num 542) 2024-03-04T06:12:18.983 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:19.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:19.093+0000 7f1d64c17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:19.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:19.097+0000 7f1d64c17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:19.210 INFO:tasks.rados.rados.0.smithi064.stdout:1115: done (3 left) 2024-03-04T06:12:19.210 INFO:tasks.rados.rados.0.smithi064.stdout:1116: done (2 left) 2024-03-04T06:12:19.210 INFO:tasks.rados.rados.0.smithi064.stdout:1117: done (1 left) 2024-03-04T06:12:19.210 INFO:tasks.rados.rados.0.smithi064.stdout:1119: done (0 left) 2024-03-04T06:12:19.210 INFO:tasks.rados.rados.0.smithi064.stdout:1120: write oid 1 current snap is 128 2024-03-04T06:12:19.210 INFO:tasks.rados.rados.0.smithi064.stdout:1120: seq_num 543 ranges {726201=721270,2141840=578191} 2024-03-04T06:12:19.217 INFO:tasks.rados.rados.0.smithi064.stdout:1120: writing smithi06431743-1 from 726201 to 1447471 tid 1 2024-03-04T06:12:19.224 INFO:tasks.rados.rados.0.smithi064.stdout:1120: writing smithi06431743-1 from 2141840 to 2720031 tid 2 2024-03-04T06:12:19.224 INFO:tasks.rados.rados.0.smithi064.stdout:1121: delete oid 30 current snap is 128 2024-03-04T06:12:19.271 INFO:tasks.rados.rados.0.smithi064.stdout:1121: done (1 left) 2024-03-04T06:12:19.271 INFO:tasks.rados.rados.0.smithi064.stdout:1122: read oid 24 snap -1 2024-03-04T06:12:19.271 INFO:tasks.rados.rados.0.smithi064.stdout:1122: expect deleted 2024-03-04T06:12:19.271 INFO:tasks.rados.rados.0.smithi064.stdout:1123: snap_remove snap 125 2024-03-04T06:12:20.210 INFO:tasks.rados.rados.0.smithi064.stdout:1123: done (2 left) 2024-03-04T06:12:20.211 INFO:tasks.rados.rados.0.smithi064.stdout:1124: snap_create 2024-03-04T06:12:20.211 INFO:tasks.rados.rados.0.smithi064.stdout:1120: finishing write tid 1 to smithi06431743-1 2024-03-04T06:12:20.211 INFO:tasks.rados.rados.0.smithi064.stdout:1120: finishing write tid 2 to smithi06431743-1 2024-03-04T06:12:20.211 INFO:tasks.rados.rados.0.smithi064.stdout:1120: finishing write tid 3 to smithi06431743-1 2024-03-04T06:12:20.211 INFO:tasks.rados.rados.0.smithi064.stdout:1120: finishing write tid 4 to smithi06431743-1 2024-03-04T06:12:20.211 INFO:tasks.rados.rados.0.smithi064.stdout:1120: oid 1 updating version 0 to 432 2024-03-04T06:12:20.211 INFO:tasks.rados.rados.0.smithi064.stdout:1120: oid 1 version 432 is already newer than 431 2024-03-04T06:12:20.211 INFO:tasks.rados.rados.0.smithi064.stdout:1120: oid 1 version 432 is already newer than 431 2024-03-04T06:12:20.211 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 432 (ObjNum 543 snap 128 seq_num 543) dirty exists 2024-03-04T06:12:20.211 INFO:tasks.rados.rados.0.smithi064.stdout:1120: left oid 1 (ObjNum 543 snap 128 seq_num 543) 2024-03-04T06:12:21.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:21.218 INFO:tasks.rados.rados.0.smithi064.stdout:1120: done (2 left) 2024-03-04T06:12:21.219 INFO:tasks.rados.rados.0.smithi064.stdout:1122: done (1 left) 2024-03-04T06:12:21.219 INFO:tasks.rados.rados.0.smithi064.stdout:1124: done (0 left) 2024-03-04T06:12:21.219 INFO:tasks.rados.rados.0.smithi064.stdout:1125: write oid 48 current snap is 129 2024-03-04T06:12:21.219 INFO:tasks.rados.rados.0.smithi064.stdout:1125: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2024-03-04T06:12:21.225 INFO:tasks.rados.rados.0.smithi064.stdout:1125: writing smithi06431743-48 from 759815 to 1231583 tid 1 2024-03-04T06:12:21.231 INFO:tasks.rados.rados.0.smithi064.stdout:1125: writing smithi06431743-48 from 1742451 to 2207958 tid 2 2024-03-04T06:12:21.236 INFO:tasks.rados.rados.0.smithi064.stdout:1125: writing smithi06431743-48 from 2917178 to 3159815 tid 3 2024-03-04T06:12:21.236 INFO:tasks.rados.rados.0.smithi064.stdout:1126: snap_remove snap 124 2024-03-04T06:12:21.314 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:21.310+0000 7fc9c435d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:21.317 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:21.317+0000 7fc9c435d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:22.220 INFO:tasks.rados.rados.0.smithi064.stdout:1126: done (1 left) 2024-03-04T06:12:22.221 INFO:tasks.rados.rados.0.smithi064.stdout:1127: read oid 12 snap 126 2024-03-04T06:12:22.221 INFO:tasks.rados.rados.0.smithi064.stdout:1127: expect (ObjNum 384 snap 93 seq_num 384) 2024-03-04T06:12:22.221 INFO:tasks.rados.rados.0.smithi064.stdout:1128: write oid 38 current snap is 129 2024-03-04T06:12:22.221 INFO:tasks.rados.rados.0.smithi064.stdout:1128: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2024-03-04T06:12:22.227 INFO:tasks.rados.rados.0.smithi064.stdout:1128: writing smithi06431743-38 from 776622 to 1323639 tid 1 2024-03-04T06:12:22.234 INFO:tasks.rados.rados.0.smithi064.stdout:1128: writing smithi06431743-38 from 1800933 to 2526451 tid 2 2024-03-04T06:12:22.238 INFO:tasks.rados.rados.0.smithi064.stdout:1128: writing smithi06431743-38 from 2944601 to 3176622 tid 3 2024-03-04T06:12:22.239 INFO:tasks.rados.rados.0.smithi064.stdout:1129: delete oid 22 current snap is 129 2024-03-04T06:12:22.239 INFO:tasks.rados.rados.0.smithi064.stdout:1125: finishing write tid 1 to smithi06431743-48 2024-03-04T06:12:22.239 INFO:tasks.rados.rados.0.smithi064.stdout:1125: finishing write tid 2 to smithi06431743-48 2024-03-04T06:12:22.239 INFO:tasks.rados.rados.0.smithi064.stdout:1125: finishing write tid 3 to smithi06431743-48 2024-03-04T06:12:22.239 INFO:tasks.rados.rados.0.smithi064.stdout:1125: finishing write tid 4 to smithi06431743-48 2024-03-04T06:12:22.239 INFO:tasks.rados.rados.0.smithi064.stdout:1125: finishing write tid 5 to smithi06431743-48 2024-03-04T06:12:22.239 INFO:tasks.rados.rados.0.smithi064.stdout:1125: oid 48 updating version 0 to 205 2024-03-04T06:12:22.239 INFO:tasks.rados.rados.0.smithi064.stdout:1125: oid 48 version 205 is already newer than 205 2024-03-04T06:12:22.239 INFO:tasks.rados.rados.0.smithi064.stdout:1125: oid 48 version 205 is already newer than 203 2024-03-04T06:12:22.240 INFO:tasks.rados.rados.0.smithi064.stdout:1125: oid 48 version 205 is already newer than 202 2024-03-04T06:12:22.240 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 205 (ObjNum 545 snap 129 seq_num 545) dirty exists 2024-03-04T06:12:22.240 INFO:tasks.rados.rados.0.smithi064.stdout:1125: left oid 48 (ObjNum 545 snap 129 seq_num 545) 2024-03-04T06:12:22.278 INFO:tasks.rados.rados.0.smithi064.stdout:1127: expect (ObjNum 384 snap 93 seq_num 384) 2024-03-04T06:12:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:1128: finishing write tid 1 to smithi06431743-38 2024-03-04T06:12:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:1128: finishing write tid 2 to smithi06431743-38 2024-03-04T06:12:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:1128: finishing write tid 3 to smithi06431743-38 2024-03-04T06:12:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:1125: done (3 left) 2024-03-04T06:12:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:1127: done (2 left) 2024-03-04T06:12:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:1129: done (1 left) 2024-03-04T06:12:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:1130: snap_remove snap 128 2024-03-04T06:12:22.977 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:12:22.977 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:12:22.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:12:23.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:23.084+0000 7f35d579d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:23.091 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:23.090+0000 7f35d579d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:23.228 INFO:tasks.rados.rados.0.smithi064.stdout:1130: done (1 left) 2024-03-04T06:12:23.228 INFO:tasks.rados.rados.0.smithi064.stdout:1131: write (excl) oid 23 current snap is 129 2024-03-04T06:12:23.228 INFO:tasks.rados.rados.0.smithi064.stdout:1128: finishing write tid 4 to smithi06431743-38 2024-03-04T06:12:23.228 INFO:tasks.rados.rados.0.smithi064.stdout:1131: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2024-03-04T06:12:23.232 INFO:tasks.rados.rados.0.smithi064.stdout:1131: writing smithi06431743-23 from 410236 to 824104 tid 1 2024-03-04T06:12:23.239 INFO:tasks.rados.rados.0.smithi064.stdout:1131: writing smithi06431743-23 from 1234250 to 1963437 tid 2 2024-03-04T06:12:23.246 INFO:tasks.rados.rados.0.smithi064.stdout:1131: writing smithi06431743-23 from 2715800 to 3138863 tid 3 2024-03-04T06:12:23.246 INFO:tasks.rados.rados.0.smithi064.stdout:1128: finishing write tid 5 to smithi06431743-38 2024-03-04T06:12:23.246 INFO:tasks.rados.rados.0.smithi064.stdout:1128: oid 38 updating version 0 to 188 2024-03-04T06:12:23.246 INFO:tasks.rados.rados.0.smithi064.stdout:1128: oid 38 updating version 188 to 189 2024-03-04T06:12:23.246 INFO:tasks.rados.rados.0.smithi064.stdout:1128: oid 38 version 189 is already newer than 187 2024-03-04T06:12:23.246 INFO:tasks.rados.rados.0.smithi064.stdout:1128: oid 38 updating version 189 to 190 2024-03-04T06:12:23.246 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 190 (ObjNum 546 snap 129 seq_num 546) dirty exists 2024-03-04T06:12:23.246 INFO:tasks.rados.rados.0.smithi064.stdout:1128: left oid 38 (ObjNum 546 snap 129 seq_num 546) 2024-03-04T06:12:23.247 INFO:tasks.rados.rados.0.smithi064.stdout:1132: read oid 21 snap -1 2024-03-04T06:12:23.247 INFO:tasks.rados.rados.0.smithi064.stdout:1132: expect deleted 2024-03-04T06:12:23.247 INFO:tasks.rados.rados.0.smithi064.stdout:1128: done (2 left) 2024-03-04T06:12:23.247 INFO:tasks.rados.rados.0.smithi064.stdout:1133: delete oid 15 current snap is 129 2024-03-04T06:12:23.278 INFO:tasks.rados.rados.0.smithi064.stdout:1132: done (2 left) 2024-03-04T06:12:23.278 INFO:tasks.rados.rados.0.smithi064.stdout:1133: done (1 left) 2024-03-04T06:12:23.278 INFO:tasks.rados.rados.0.smithi064.stdout:1134: snap_create 2024-03-04T06:12:23.283 INFO:tasks.rados.rados.0.smithi064.stdout:1131: finishing write tid 1 to smithi06431743-23 2024-03-04T06:12:23.301 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:12:23.302 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:12:23.309 INFO:tasks.rados.rados.0.smithi064.stdout:1131: finishing write tid 2 to smithi06431743-23 2024-03-04T06:12:23.316 INFO:tasks.rados.rados.0.smithi064.stdout:1131: finishing write tid 3 to smithi06431743-23 2024-03-04T06:12:23.318 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-03-04T06:12:23.225022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95267202,"num_objects":377,"num_object_clones":273,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2194,"num_read_kb":1555229,"num_write":1938,"num_write_kb":491350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3641,"ondisk_log_size":3641,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":356896,"kb_used_data":216320,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377130464,"statfs":{"total":386547056640,"available":386181595136,"internally_reserved":0,"allocated":221511680,"data_stored":239681369,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1680766,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":4,"num_read_kb":3,"num_write":14,"num_write_kb":3756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016370"},"pg_stats":[{"pgid":"3.f","version":"265'180","reported_seq":889,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.327168+0000","last_change":"2024-03-04T06:12:15.488091+0000","last_active":"2024-03-04T06:12:20.327168+0000","last_peered":"2024-03-04T06:12:20.327168+0000","last_clean":"2024-03-04T06:12:20.327168+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:12:20.327168+0000","last_undegraded":"2024-03-04T06:12:20.327168+0000","last_fullsized":"2024-03-04T06:12:20.327168+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9260921,"num_objects":26,"num_object_clones":17,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":70,"num_read_kb":30353,"num_write":108,"num_write_kb":29082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"267'155","reported_seq":890,"reported_epoch":270,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:20.324659+0000","last_change":"2024-03-04T06:12:20.324659+0000","last_active":"2024-03-04T06:12:20.324659+0000","last_peered":"2024-03-04T06:12:20.324659+0000","last_clean":"2024-03-04T06:12:20.324659+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:12:20.324659+0000","last_undegraded":"2024-03-04T06:12:20.324659+0000","last_fullsized":"2024-03-04T06:12:20.324659+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269176,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":63,"num_read_kb":40085,"num_write":72,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"265'251","reported_seq":944,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.239497+0000","last_change":"2024-03-04T06:12:20.242495+0000","last_active":"2024-03-04T06:12:20.239497+0000","last_peered":"2024-03-04T06:12:20.239497+0000","last_clean":"2024-03-04T06:12:20.239497+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:12:20.239497+0000","last_undegraded":"2024-03-04T06:12:20.239497+0000","last_fullsized":"2024-03-04T06:12:20.239497+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193693,"num_objects":29,"num_object_clones":22,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":138,"num_read_kb":88901,"num_write":136,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"267'169","reported_seq":915,"reported_epoch":270,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:20.320461+0000","last_change":"2024-03-04T06:12:20.320461+0000","last_active":"2024-03-04T06:12:20.320461+0000","last_peered":"2024-03-04T06:12:20.320461+0000","last_clean":"2024-03-04T06:12:20.320461+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:12:20.320461+0000","last_undegraded":"2024-03-04T06:12:20.320461+0000","last_fullsized":"2024-03-04T06:12:20.320461+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9042463,"num_objects":20,"num_object_clones":12,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":111,"num_read_kb":86740,"num_write":109,"num_write_kb":30492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"249'92","reported_seq":822,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-04T06:12:21.249074+0000","last_change":"2024-03-04T06:12:20.350498+0000","last_active":"2024-03-04T06:12:21.249074+0000","last_peered":"2024-03-04T06:12:21.249074+0000","last_clean":"2024-03-04T06:12:21.249074+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:12:21.249074+0000","last_undegraded":"2024-03-04T06:12:21.249074+0000","last_fullsized":"2024-03-04T06:12:21.249074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":17041,"num_write":34,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"265'185","reported_seq":906,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.324805+0000","last_change":"2024-03-04T06:12:15.486886+0000","last_active":"2024-03-04T06:12:20.324805+0000","last_peered":"2024-03-04T06:12:20.324805+0000","last_clean":"2024-03-04T06:12:20.324805+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:12:20.324805+0000","last_undegraded":"2024-03-04T06:12:20.324805+0000","last_fullsized":"2024-03-04T06:12:20.324805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5630543,"num_objects":20,"num_object_clones":14,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":111,"num_read_kb":83334,"num_write":106,"num_write_kb":31605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":764,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.322743+0000","last_change":"2024-03-04T06:12:15.463902+0000","last_active":"2024-03-04T06:12:20.322743+0000","last_peered":"2024-03-04T06:12:20.322743+0000","last_clean":"2024-03-04T06:12:20.322743+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:12:20.322743+0000","last_undegraded":"2024-03-04T06:12:20.322743+0000","last_fullsized":"2024-03-04T06:12:20.322743+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"265'154","reported_seq":883,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.239352+0000","last_change":"2024-03-04T06:12:20.242693+0000","last_active":"2024-03-04T06:12:20.239352+0000","last_peered":"2024-03-04T06:12:20.239352+0000","last_clean":"2024-03-04T06:12:20.239352+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:12:20.239352+0000","last_undegraded":"2024-03-04T06:12:20.239352+0000","last_fullsized":"2024-03-04T06:12:20.239352+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252237,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":116,"num_read_kb":111954,"num_write":72,"num_write_kb":19512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":796,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-04T06:12:21.246075+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:12:21.246075+0000","last_peered":"2024-03-04T06:12:21.246075+0000","last_clean":"2024-03-04T06:12:21.246075+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:12:21.246075+0000","last_undegraded":"2024-03-04T06:12:21.246075+0000","last_fullsized":"2024-03-04T06:12:21.246075+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":650,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.241569+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:12:20.241569+0000","last_peered":"2024-03-04T06:12:20.241569+0000","last_clean":"2024-03-04T06:12:20.241569+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:12:20.241569+0000","last_undegraded":"2024-03-04T06:12:20.241569+0000","last_fullsized":"2024-03-04T06:12:20.241569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"265'206","reported_seq":921,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.246817+0000","last_change":"2024-03-04T06:12:20.248457+0000","last_active":"2024-03-04T06:12:20.246817+0000","last_peered":"2024-03-04T06:12:20.246817+0000","last_clean":"2024-03-04T06:12:20.246817+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:12:20.246817+0000","last_undegraded":"2024-03-04T06:12:20.246817+0000","last_fullsized":"2024-03-04T06:12:20.246817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8739937,"num_objects":22,"num_object_clones":15,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":137,"num_read_kb":109561,"num_write":104,"num_write_kb":25876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":651,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.324018+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:12:20.324018+0000","last_peered":"2024-03-04T06:12:20.324018+0000","last_clean":"2024-03-04T06:12:20.324018+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:12:20.324018+0000","last_undegraded":"2024-03-04T06:12:20.324018+0000","last_fullsized":"2024-03-04T06:12:20.324018+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"271'205","reported_seq":896,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-04T06:12:21.324129+0000","last_change":"2024-03-04T06:12:20.339245+0000","last_active":"2024-03-04T06:12:21.324129+0000","last_peered":"2024-03-04T06:12:21.324129+0000","last_clean":"2024-03-04T06:12:21.324129+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:12:21.324129+0000","last_undegraded":"2024-03-04T06:12:21.324129+0000","last_fullsized":"2024-03-04T06:12:21.324129+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3159815,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":94,"num_read_kb":69546,"num_write":87,"num_write_kb":25445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":650,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.231156+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:12:20.231156+0000","last_peered":"2024-03-04T06:12:20.231156+0000","last_clean":"2024-03-04T06:12:20.231156+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:12:20.231156+0000","last_undegraded":"2024-03-04T06:12:20.231156+0000","last_fullsized":"2024-03-04T06:12:20.231156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"270'434","reported_seq":1128,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-04T06:12:21.248182+0000","last_change":"2024-03-04T06:12:20.362457+0000","last_active":"2024-03-04T06:12:21.248182+0000","last_peered":"2024-03-04T06:12:21.248182+0000","last_clean":"2024-03-04T06:12:21.248182+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:12:21.248182+0000","last_undegraded":"2024-03-04T06:12:21.248182+0000","last_fullsized":"2024-03-04T06:12:21.248182+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8487799,"num_objects":44,"num_object_clones":33,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":309,"num_read_kb":258280,"num_write":234,"num_write_kb":64624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":654,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.328415+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:12:20.328415+0000","last_peered":"2024-03-04T06:12:20.328415+0000","last_clean":"2024-03-04T06:12:20.328415+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:12:20.328415+0000","last_undegraded":"2024-03-04T06:12:20.328415+0000","last_fullsized":"2024-03-04T06:12:20.328415+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"269'458","reported_seq":1083,"reported_epoch":270,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:20.318510+0000","last_change":"2024-03-04T06:12:20.318510+0000","last_active":"2024-03-04T06:12:20.318510+0000","last_peered":"2024-03-04T06:12:20.318510+0000","last_clean":"2024-03-04T06:12:20.318510+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:12:20.318510+0000","last_undegraded":"2024-03-04T06:12:20.318510+0000","last_fullsized":"2024-03-04T06:12:20.318510+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9849133,"num_objects":46,"num_object_clones":34,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":200,"num_read_kb":139975,"num_write":222,"num_write_kb":59046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"250'169","reported_seq":926,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.247551+0000","last_change":"2024-03-04T06:12:20.249560+0000","last_active":"2024-03-04T06:12:20.247551+0000","last_peered":"2024-03-04T06:12:20.247551+0000","last_clean":"2024-03-04T06:12:20.247551+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:12:20.247551+0000","last_undegraded":"2024-03-04T06:12:20.247551+0000","last_fullsized":"2024-03-04T06:12:20.247551+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294238,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":181,"num_read_kb":148565,"num_write":86,"num_write_kb":22565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":686,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.338573+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:12:20.338573+0000","last_peered":"2024-03-04T06:12:20.338573+0000","last_clean":"2024-03-04T06:12:20.338573+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:12:20.338573+0000","last_undegraded":"2024-03-04T06:12:20.338573+0000","last_fullsized":"2024-03-04T06:12:20.338573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"267'202","reported_seq":939,"reported_epoch":270,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:20.340932+0000","last_change":"2024-03-04T06:12:20.340932+0000","last_active":"2024-03-04T06:12:20.340932+0000","last_peered":"2024-03-04T06:12:20.340932+0000","last_clean":"2024-03-04T06:12:20.340932+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:12:20.340932+0000","last_undegraded":"2024-03-04T06:12:20.340932+0000","last_fullsized":"2024-03-04T06:12:20.340932+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5596929,"num_objects":25,"num_object_clones":16,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":108,"num_read_kb":59205,"num_write":105,"num_write_kb":28287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":663,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-04T06:12:21.252485+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:12:21.252485+0000","last_peered":"2024-03-04T06:12:21.252485+0000","last_clean":"2024-03-04T06:12:21.252485+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:12:21.252485+0000","last_undegraded":"2024-03-04T06:12:21.252485+0000","last_fullsized":"2024-03-04T06:12:21.252485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"267'292","reported_seq":985,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-03-04T06:12:21.245281+0000","last_change":"2024-03-04T06:12:20.339096+0000","last_active":"2024-03-04T06:12:21.245281+0000","last_peered":"2024-03-04T06:12:21.245281+0000","last_clean":"2024-03-04T06:12:21.245281+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:12:21.245281+0000","last_undegraded":"2024-03-04T06:12:21.245281+0000","last_fullsized":"2024-03-04T06:12:21.245281+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210500,"num_objects":30,"num_object_clones":22,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":168,"num_read_kb":115116,"num_write":150,"num_write_kb":41216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":686,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.321505+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:12:20.321505+0000","last_peered":"2024-03-04T06:12:20.321505+0000","last_clean":"2024-03-04T06:12:20.321505+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:12:20.321505+0000","last_undegraded":"2024-03-04T06:12:20.321505+0000","last_fullsized":"2024-03-04T06:12:20.321505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"267'392","reported_seq":1072,"reported_epoch":270,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:20.322342+0000","last_change":"2024-03-04T06:12:20.322342+0000","last_active":"2024-03-04T06:12:20.322342+0000","last_peered":"2024-03-04T06:12:20.322342+0000","last_clean":"2024-03-04T06:12:20.322342+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:12:20.322342+0000","last_undegraded":"2024-03-04T06:12:20.322342+0000","last_fullsized":"2024-03-04T06:12:20.322342+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8689450,"num_objects":34,"num_object_clones":30,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":254,"num_read_kb":196482,"num_write":176,"num_write_kb":47586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":650,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T06:12:20.240150+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:12:20.240150+0000","last_peered":"2024-03-04T06:12:20.240150+0000","last_clean":"2024-03-04T06:12:20.240150+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:12:20.240150+0000","last_undegraded":"2024-03-04T06:12:20.240150+0000","last_fullsized":"2024-03-04T06:12:20.240150+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94676834,"num_objects":375,"num_object_clones":273,"num_object_copies":1125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2091,"num_read_kb":1555138,"num_write":1801,"num_write_kb":489456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3544,"ondisk_log_size":3544,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":103932,"kb_used_data":68788,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94267908,"statfs":{"total":96636764160,"available":96530337792,"internally_reserved":0,"allocated":70438912,"data_stored":76925486,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87084,"kb_used_data":51940,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94284756,"statfs":{"total":96636764160,"available":96547590144,"internally_reserved":0,"allocated":53186560,"data_stored":57307627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77964,"kb_used_data":42820,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94293876,"statfs":{"total":96636764160,"available":96556929024,"internally_reserved":0,"allocated":43847680,"data_stored":46318887,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87916,"kb_used_data":52772,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94283924,"statfs":{"total":96636764160,"available":96546738176,"internally_reserved":0,"allocated":54038528,"data_stored":59129369,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:12:23.319 INFO:tasks.rados.rados.0.smithi064.stdout:1131: finishing write tid 4 to smithi06431743-23 2024-03-04T06:12:23.320 INFO:tasks.rados.rados.0.smithi064.stdout:1131: finishing write tid 5 to smithi06431743-23 2024-03-04T06:12:23.320 INFO:tasks.rados.rados.0.smithi064.stdout:1131: oid 23 updating version 0 to 211 2024-03-04T06:12:23.320 INFO:tasks.rados.rados.0.smithi064.stdout:1131: oid 23 version 211 is already newer than 211 2024-03-04T06:12:23.320 INFO:tasks.rados.rados.0.smithi064.stdout:1131: oid 23 version 211 is already newer than 208 2024-03-04T06:12:23.320 INFO:tasks.rados.rados.0.smithi064.stdout:1131: oid 23 version 211 is already newer than 209 2024-03-04T06:12:23.320 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 211 (ObjNum 548 snap 129 seq_num 548) dirty exists 2024-03-04T06:12:23.321 INFO:tasks.rados.rados.0.smithi064.stdout:1131: left oid 23 (ObjNum 548 snap 129 seq_num 548) 2024-03-04T06:12:23.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:23.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:23.493+0000 7fba50b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:23.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:23.497+0000 7fba50b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:24.228 INFO:tasks.rados.rados.0.smithi064.stdout:1131: done (1 left) 2024-03-04T06:12:24.228 INFO:tasks.rados.rados.0.smithi064.stdout:1134: done (0 left) 2024-03-04T06:12:24.228 INFO:tasks.rados.rados.0.smithi064.stdout:1135: write oid 36 current snap is 130 2024-03-04T06:12:24.228 INFO:tasks.rados.rados.0.smithi064.stdout:1135: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2024-03-04T06:12:24.233 INFO:tasks.rados.rados.0.smithi064.stdout:1135: writing smithi06431743-36 from 443850 to 1008216 tid 1 2024-03-04T06:12:24.241 INFO:tasks.rados.rados.0.smithi064.stdout:1135: writing smithi06431743-36 from 1634861 to 2367717 tid 2 2024-03-04T06:12:24.246 INFO:tasks.rados.rados.0.smithi064.stdout:1135: writing smithi06431743-36 from 2770646 to 3243850 tid 3 2024-03-04T06:12:24.246 INFO:tasks.rados.rados.0.smithi064.stdout:1136: write oid 5 current snap is 130 2024-03-04T06:12:24.246 INFO:tasks.rados.rados.0.smithi064.stdout:1136: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-03-04T06:12:24.252 INFO:tasks.rados.rados.0.smithi064.stdout:1136: writing smithi06431743-5 from 460657 to 1100272 tid 1 2024-03-04T06:12:24.258 INFO:tasks.rados.rados.0.smithi064.stdout:1136: writing smithi06431743-5 from 1693343 to 2169857 tid 2 2024-03-04T06:12:24.264 INFO:tasks.rados.rados.0.smithi064.stdout:1136: writing smithi06431743-5 from 2681716 to 3260657 tid 3 2024-03-04T06:12:24.264 INFO:tasks.rados.rados.0.smithi064.stdout:1137: snap_create 2024-03-04T06:12:24.273 INFO:tasks.rados.rados.0.smithi064.stdout:1136: finishing write tid 1 to smithi06431743-5 2024-03-04T06:12:24.281 INFO:tasks.rados.rados.0.smithi064.stdout:1135: finishing write tid 1 to smithi06431743-36 2024-03-04T06:12:24.291 INFO:tasks.rados.rados.0.smithi064.stdout:1136: finishing write tid 2 to smithi06431743-5 2024-03-04T06:12:24.300 INFO:tasks.rados.rados.0.smithi064.stdout:1135: finishing write tid 2 to smithi06431743-36 2024-03-04T06:12:24.310 INFO:tasks.rados.rados.0.smithi064.stdout:1136: finishing write tid 3 to smithi06431743-5 2024-03-04T06:12:24.314 INFO:tasks.rados.rados.0.smithi064.stdout:1136: finishing write tid 4 to smithi06431743-5 2024-03-04T06:12:24.314 INFO:tasks.rados.rados.0.smithi064.stdout:1135: finishing write tid 3 to smithi06431743-36 2024-03-04T06:12:24.315 INFO:tasks.rados.rados.0.smithi064.stdout:1136: finishing write tid 5 to smithi06431743-5 2024-03-04T06:12:24.315 INFO:tasks.rados.rados.0.smithi064.stdout:1136: oid 5 updating version 0 to 466 2024-03-04T06:12:24.315 INFO:tasks.rados.rados.0.smithi064.stdout:1136: oid 5 version 466 is already newer than 466 2024-03-04T06:12:24.315 INFO:tasks.rados.rados.0.smithi064.stdout:1136: oid 5 version 466 is already newer than 463 2024-03-04T06:12:24.316 INFO:tasks.rados.rados.0.smithi064.stdout:1136: oid 5 version 466 is already newer than 464 2024-03-04T06:12:24.316 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 466 (ObjNum 551 snap 130 seq_num 551) dirty exists 2024-03-04T06:12:24.316 INFO:tasks.rados.rados.0.smithi064.stdout:1136: left oid 5 (ObjNum 551 snap 130 seq_num 551) 2024-03-04T06:12:24.319 INFO:tasks.rados.rados.0.smithi064.stdout:1135: finishing write tid 4 to smithi06431743-36 2024-03-04T06:12:24.320 INFO:tasks.rados.rados.0.smithi064.stdout:1135: finishing write tid 5 to smithi06431743-36 2024-03-04T06:12:24.320 INFO:tasks.rados.rados.0.smithi064.stdout:1135: oid 36 updating version 0 to 205 2024-03-04T06:12:24.320 INFO:tasks.rados.rados.0.smithi064.stdout:1135: oid 36 updating version 205 to 206 2024-03-04T06:12:24.320 INFO:tasks.rados.rados.0.smithi064.stdout:1135: oid 36 version 206 is already newer than 204 2024-03-04T06:12:24.320 INFO:tasks.rados.rados.0.smithi064.stdout:1135: oid 36 version 206 is already newer than 203 2024-03-04T06:12:24.320 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 206 (ObjNum 550 snap 130 seq_num 550) dirty exists 2024-03-04T06:12:24.320 INFO:tasks.rados.rados.0.smithi064.stdout:1135: left oid 36 (ObjNum 550 snap 130 seq_num 550) 2024-03-04T06:12:25.239 INFO:tasks.rados.rados.0.smithi064.stdout:1135: done (2 left) 2024-03-04T06:12:25.240 INFO:tasks.rados.rados.0.smithi064.stdout:1136: done (1 left) 2024-03-04T06:12:25.240 INFO:tasks.rados.rados.0.smithi064.stdout:1137: done (0 left) 2024-03-04T06:12:25.240 INFO:tasks.rados.rados.0.smithi064.stdout:1138: write oid 34 current snap is 131 2024-03-04T06:12:25.240 INFO:tasks.rados.rados.0.smithi064.stdout:1138: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2024-03-04T06:12:25.246 INFO:tasks.rados.rados.0.smithi064.stdout:1138: writing smithi06431743-34 from 477464 to 1192328 tid 1 2024-03-04T06:12:25.253 INFO:tasks.rados.rados.0.smithi064.stdout:1138: writing smithi06431743-34 from 1751825 to 2488350 tid 2 2024-03-04T06:12:25.258 INFO:tasks.rados.rados.0.smithi064.stdout:1138: writing smithi06431743-34 from 3225492 to 3277464 tid 3 2024-03-04T06:12:25.258 INFO:tasks.rados.rados.0.smithi064.stdout:1139: snap_create 2024-03-04T06:12:25.327 INFO:tasks.rados.rados.0.smithi064.stdout:1138: finishing write tid 1 to smithi06431743-34 2024-03-04T06:12:25.334 INFO:tasks.rados.rados.0.smithi064.stdout:1138: finishing write tid 2 to smithi06431743-34 2024-03-04T06:12:25.342 INFO:tasks.rados.rados.0.smithi064.stdout:1138: finishing write tid 3 to smithi06431743-34 2024-03-04T06:12:25.350 INFO:tasks.rados.rados.0.smithi064.stdout:1138: finishing write tid 4 to smithi06431743-34 2024-03-04T06:12:25.351 INFO:tasks.rados.rados.0.smithi064.stdout:1138: finishing write tid 5 to smithi06431743-34 2024-03-04T06:12:25.351 INFO:tasks.rados.rados.0.smithi064.stdout:1138: oid 34 updating version 0 to 439 2024-03-04T06:12:25.351 INFO:tasks.rados.rados.0.smithi064.stdout:1138: oid 34 updating version 439 to 440 2024-03-04T06:12:25.351 INFO:tasks.rados.rados.0.smithi064.stdout:1138: oid 34 updating version 440 to 441 2024-03-04T06:12:25.351 INFO:tasks.rados.rados.0.smithi064.stdout:1138: oid 34 version 441 is already newer than 439 2024-03-04T06:12:25.351 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 441 (ObjNum 552 snap 131 seq_num 552) dirty exists 2024-03-04T06:12:25.352 INFO:tasks.rados.rados.0.smithi064.stdout:1138: left oid 34 (ObjNum 552 snap 131 seq_num 552) 2024-03-04T06:12:25.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:25.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:25.679+0000 7fe13bb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:25.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:25.684+0000 7fe13bb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:26.244 INFO:tasks.rados.rados.0.smithi064.stdout:1138: done (1 left) 2024-03-04T06:12:26.245 INFO:tasks.rados.rados.0.smithi064.stdout:1139: done (0 left) 2024-03-04T06:12:26.245 INFO:tasks.rados.rados.0.smithi064.stdout:1140: snap_remove snap 126 2024-03-04T06:12:27.249 INFO:tasks.rados.rados.0.smithi064.stdout:1140: done (0 left) 2024-03-04T06:12:27.250 INFO:tasks.rados.rados.0.smithi064.stdout:1141: read oid 44 snap -1 2024-03-04T06:12:27.250 INFO:tasks.rados.rados.0.smithi064.stdout:1141: expect (ObjNum 533 snap 123 seq_num 533) 2024-03-04T06:12:27.250 INFO:tasks.rados.rados.0.smithi064.stdout:1142: write oid 12 current snap is 132 2024-03-04T06:12:27.250 INFO:tasks.rados.rados.0.smithi064.stdout:1142: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2024-03-04T06:12:27.257 INFO:tasks.rados.rados.0.smithi064.stdout:1142: writing smithi06431743-12 from 494271 to 1284384 tid 1 2024-03-04T06:12:27.262 INFO:tasks.rados.rados.0.smithi064.stdout:1142: writing smithi06431743-12 from 1693954 to 2174137 tid 2 2024-03-04T06:12:27.268 INFO:tasks.rados.rados.0.smithi064.stdout:1142: writing smithi06431743-12 from 2620209 to 3132338 tid 3 2024-03-04T06:12:27.268 INFO:tasks.rados.rados.0.smithi064.stdout:1143: read oid 25 snap -1 2024-03-04T06:12:27.268 INFO:tasks.rados.rados.0.smithi064.stdout:1143: expect (ObjNum 500 snap 115 seq_num 500) 2024-03-04T06:12:27.268 INFO:tasks.rados.rados.0.smithi064.stdout:1144: read oid 36 snap -1 2024-03-04T06:12:27.268 INFO:tasks.rados.rados.0.smithi064.stdout:1144: expect (ObjNum 550 snap 130 seq_num 550) 2024-03-04T06:12:27.268 INFO:tasks.rados.rados.0.smithi064.stdout:1145: write oid 9 current snap is 132 2024-03-04T06:12:27.269 INFO:tasks.rados.rados.0.smithi064.stdout:1145: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2024-03-04T06:12:27.274 INFO:tasks.rados.rados.0.smithi064.stdout:1145: writing smithi06431743-9 from 511078 to 976440 tid 1 2024-03-04T06:12:27.285 INFO:tasks.rados.rados.0.smithi064.stdout:1145: writing smithi06431743-9 from 1752436 to 2492630 tid 2 2024-03-04T06:12:27.292 INFO:tasks.rados.rados.0.smithi064.stdout:1145: writing smithi06431743-9 from 3163985 to 3311078 tid 3 2024-03-04T06:12:27.292 INFO:tasks.rados.rados.0.smithi064.stdout:1146: read oid 22 snap 130 2024-03-04T06:12:27.292 INFO:tasks.rados.rados.0.smithi064.stdout:1146: expect deleted 2024-03-04T06:12:27.292 INFO:tasks.rados.rados.0.smithi064.stdout:1147: write (excl) oid 15 current snap is 132 2024-03-04T06:12:27.292 INFO:tasks.rados.rados.0.smithi064.stdout:1147: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2024-03-04T06:12:27.299 INFO:tasks.rados.rados.0.smithi064.stdout:1147: writing smithi06431743-15 from 527885 to 1184849 tid 1 2024-03-04T06:12:27.307 INFO:tasks.rados.rados.0.smithi064.stdout:1147: writing smithi06431743-15 from 1927271 to 2411123 tid 2 2024-03-04T06:12:27.313 INFO:tasks.rados.rados.0.smithi064.stdout:1147: writing smithi06431743-15 from 3191408 to 3327885 tid 3 2024-03-04T06:12:27.313 INFO:tasks.rados.rados.0.smithi064.stdout:1148: read oid 48 snap -1 2024-03-04T06:12:27.313 INFO:tasks.rados.rados.0.smithi064.stdout:1148: expect (ObjNum 545 snap 129 seq_num 545) 2024-03-04T06:12:27.313 INFO:tasks.rados.rados.0.smithi064.stdout:1149: write oid 35 current snap is 132 2024-03-04T06:12:27.313 INFO:tasks.rados.rados.0.smithi064.stdout:1149: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2024-03-04T06:12:27.320 INFO:tasks.rados.rados.0.smithi064.stdout:1149: writing smithi06431743-35 from 544692 to 1276905 tid 1 2024-03-04T06:12:27.329 INFO:tasks.rados.rados.0.smithi064.stdout:1149: writing smithi06431743-35 from 1985753 to 2729616 tid 2 2024-03-04T06:12:27.334 INFO:tasks.rados.rados.0.smithi064.stdout:1149: writing smithi06431743-35 from 3335184 to 3344692 tid 3 2024-03-04T06:12:27.334 INFO:tasks.rados.rados.0.smithi064.stdout:1150: write oid 20 current snap is 132 2024-03-04T06:12:27.334 INFO:tasks.rados.rados.0.smithi064.stdout:1144: expect (ObjNum 550 snap 130 seq_num 550) 2024-03-04T06:12:27.468 INFO:tasks.rados.rados.0.smithi064.stdout:1141: expect (ObjNum 533 snap 123 seq_num 533) 2024-03-04T06:12:27.577 INFO:tasks.rados.rados.0.smithi064.stdout:1145: finishing write tid 1 to smithi06431743-9 2024-03-04T06:12:27.577 INFO:tasks.rados.rados.0.smithi064.stdout:1143: expect (ObjNum 500 snap 115 seq_num 500) 2024-03-04T06:12:27.727 INFO:tasks.rados.rados.0.smithi064.stdout:1147: finishing write tid 1 to smithi06431743-15 2024-03-04T06:12:27.727 INFO:tasks.rados.rados.0.smithi064.stdout:1142: finishing write tid 1 to smithi06431743-12 2024-03-04T06:12:27.727 INFO:tasks.rados.rados.0.smithi064.stdout:1145: finishing write tid 2 to smithi06431743-9 2024-03-04T06:12:27.727 INFO:tasks.rados.rados.0.smithi064.stdout:1145: finishing write tid 3 to smithi06431743-9 2024-03-04T06:12:27.727 INFO:tasks.rados.rados.0.smithi064.stdout:1147: finishing write tid 2 to smithi06431743-15 2024-03-04T06:12:27.727 INFO:tasks.rados.rados.0.smithi064.stdout:1147: finishing write tid 3 to smithi06431743-15 2024-03-04T06:12:27.727 INFO:tasks.rados.rados.0.smithi064.stdout:1142: finishing write tid 2 to smithi06431743-12 2024-03-04T06:12:27.727 INFO:tasks.rados.rados.0.smithi064.stdout:1150: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2024-03-04T06:12:27.732 INFO:tasks.rados.rados.0.smithi064.stdout:1150: writing smithi06431743-20 from 561499 to 968961 tid 1 2024-03-04T06:12:27.737 INFO:tasks.rados.rados.0.smithi064.stdout:1150: writing smithi06431743-20 from 1527882 to 2015403 tid 2 2024-03-04T06:12:27.748 INFO:tasks.rados.rados.0.smithi064.stdout:1150: writing smithi06431743-20 from 2729901 to 3287118 tid 3 2024-03-04T06:12:27.748 INFO:tasks.rados.rados.0.smithi064.stdout:1141: done (9 left) 2024-03-04T06:12:27.748 INFO:tasks.rados.rados.0.smithi064.stdout:1143: done (8 left) 2024-03-04T06:12:27.748 INFO:tasks.rados.rados.0.smithi064.stdout:1144: done (7 left) 2024-03-04T06:12:27.748 INFO:tasks.rados.rados.0.smithi064.stdout:1151: delete oid 7 current snap is 132 2024-03-04T06:12:27.748 INFO:tasks.rados.rados.0.smithi064.stdout:1145: finishing write tid 4 to smithi06431743-9 2024-03-04T06:12:27.748 INFO:tasks.rados.rados.0.smithi064.stdout:1145: finishing write tid 5 to smithi06431743-9 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1145: oid 9 updating version 0 to 402 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1145: oid 9 version 402 is already newer than 400 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1145: oid 9 version 402 is already newer than 402 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1145: oid 9 updating version 402 to 403 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 403 (ObjNum 554 snap 132 seq_num 554) dirty exists 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1145: left oid 9 (ObjNum 554 snap 132 seq_num 554) 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1149: finishing write tid 1 to smithi06431743-35 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1147: finishing write tid 4 to smithi06431743-15 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1147: finishing write tid 5 to smithi06431743-15 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1147: oid 15 updating version 0 to 173 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1147: oid 15 version 173 is already newer than 173 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1147: oid 15 version 173 is already newer than 171 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:1147: oid 15 updating version 173 to 174 2024-03-04T06:12:27.749 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 174 (ObjNum 555 snap 132 seq_num 555) dirty exists 2024-03-04T06:12:27.750 INFO:tasks.rados.rados.0.smithi064.stdout:1147: left oid 15 (ObjNum 555 snap 132 seq_num 555) 2024-03-04T06:12:27.750 INFO:tasks.rados.rados.0.smithi064.stdout:1142: finishing write tid 3 to smithi06431743-12 2024-03-04T06:12:27.750 INFO:tasks.rados.rados.0.smithi064.stdout:1149: finishing write tid 2 to smithi06431743-35 2024-03-04T06:12:27.750 INFO:tasks.rados.rados.0.smithi064.stdout:1148: expect (ObjNum 545 snap 129 seq_num 545) 2024-03-04T06:12:27.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:27.864 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:27.862+0000 7f79b291d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:27.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:27.866+0000 7f79b291d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:28.023 INFO:tasks.rados.rados.0.smithi064.stdout:1149: finishing write tid 3 to smithi06431743-35 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1142: finishing write tid 4 to smithi06431743-12 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1149: finishing write tid 4 to smithi06431743-35 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1149: finishing write tid 5 to smithi06431743-35 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1149: oid 35 updating version 0 to 297 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1149: oid 35 version 297 is already newer than 295 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1149: oid 35 version 297 is already newer than 295 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1149: oid 35 version 297 is already newer than 297 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 297 (ObjNum 556 snap 132 seq_num 556) dirty exists 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1149: left oid 35 (ObjNum 556 snap 132 seq_num 556) 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1142: finishing write tid 5 to smithi06431743-12 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1142: oid 12 updating version 0 to 179 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1142: oid 12 version 179 is already newer than 178 2024-03-04T06:12:28.024 INFO:tasks.rados.rados.0.smithi064.stdout:1142: oid 12 version 179 is already newer than 177 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1142: oid 12 updating version 179 to 180 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 180 (ObjNum 553 snap 132 seq_num 553) dirty exists 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1142: left oid 12 (ObjNum 553 snap 132 seq_num 553) 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1150: finishing write tid 1 to smithi06431743-20 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1150: finishing write tid 2 to smithi06431743-20 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1150: finishing write tid 3 to smithi06431743-20 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1150: finishing write tid 4 to smithi06431743-20 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1150: finishing write tid 5 to smithi06431743-20 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1150: oid 20 updating version 0 to 300 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1150: oid 20 updating version 300 to 302 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1150: oid 20 version 302 is already newer than 299 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1150: oid 20 version 302 is already newer than 302 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 302 (ObjNum 557 snap 132 seq_num 557) dirty exists 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1150: left oid 20 (ObjNum 557 snap 132 seq_num 557) 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1142: done (7 left) 2024-03-04T06:12:28.025 INFO:tasks.rados.rados.0.smithi064.stdout:1145: done (6 left) 2024-03-04T06:12:28.026 INFO:tasks.rados.rados.0.smithi064.stdout:1146: done (5 left) 2024-03-04T06:12:28.026 INFO:tasks.rados.rados.0.smithi064.stdout:1147: done (4 left) 2024-03-04T06:12:28.026 INFO:tasks.rados.rados.0.smithi064.stdout:1148: done (3 left) 2024-03-04T06:12:28.026 INFO:tasks.rados.rados.0.smithi064.stdout:1149: done (2 left) 2024-03-04T06:12:28.026 INFO:tasks.rados.rados.0.smithi064.stdout:1150: done (1 left) 2024-03-04T06:12:28.026 INFO:tasks.rados.rados.0.smithi064.stdout:1151: done (0 left) 2024-03-04T06:12:28.026 INFO:tasks.rados.rados.0.smithi064.stdout:1152: snap_remove snap 130 2024-03-04T06:12:28.211 INFO:tasks.rados.rados.0.smithi064.stdout:1152: done (0 left) 2024-03-04T06:12:28.211 INFO:tasks.rados.rados.0.smithi064.stdout:1153: read oid 21 snap 127 2024-03-04T06:12:28.211 INFO:tasks.rados.rados.0.smithi064.stdout:1153: expect deleted 2024-03-04T06:12:28.211 INFO:tasks.rados.rados.0.smithi064.stdout:1154: write oid 28 current snap is 132 2024-03-04T06:12:28.212 INFO:tasks.rados.rados.0.smithi064.stdout:1154: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2024-03-04T06:12:28.218 INFO:tasks.rados.rados.0.smithi064.stdout:1154: writing smithi06431743-28 from 595113 to 1153073 tid 1 2024-03-04T06:12:28.225 INFO:tasks.rados.rados.0.smithi064.stdout:1154: writing smithi06431743-28 from 1644846 to 2136036 tid 2 2024-03-04T06:12:28.234 INFO:tasks.rados.rados.0.smithi064.stdout:1154: writing smithi06431743-28 from 2784747 to 3395113 tid 3 2024-03-04T06:12:28.234 INFO:tasks.rados.rados.0.smithi064.stdout:1155: read oid 26 snap -1 2024-03-04T06:12:28.234 INFO:tasks.rados.rados.0.smithi064.stdout:1155: expect deleted 2024-03-04T06:12:28.234 INFO:tasks.rados.rados.0.smithi064.stdout:1156: write (excl) oid 41 current snap is 132 2024-03-04T06:12:28.234 INFO:tasks.rados.rados.0.smithi064.stdout:1156: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2024-03-04T06:12:28.242 INFO:tasks.rados.rados.0.smithi064.stdout:1156: writing smithi06431743-41 from 611920 to 1245129 tid 1 2024-03-04T06:12:28.250 INFO:tasks.rados.rados.0.smithi064.stdout:1156: writing smithi06431743-41 from 1703328 to 2338176 tid 2 2024-03-04T06:12:28.257 INFO:tasks.rados.rados.0.smithi064.stdout:1156: writing smithi06431743-41 from 2812170 to 3319556 tid 3 2024-03-04T06:12:28.257 INFO:tasks.rados.rados.0.smithi064.stdout:1157: read oid 33 snap -1 2024-03-04T06:12:28.257 INFO:tasks.rados.rados.0.smithi064.stdout:1157: expect (ObjNum 449 snap 105 seq_num 449) 2024-03-04T06:12:28.257 INFO:tasks.rados.rados.0.smithi064.stdout:1158: write oid 40 current snap is 132 2024-03-04T06:12:28.257 INFO:tasks.rados.rados.0.smithi064.stdout:1158: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2024-03-04T06:12:28.266 INFO:tasks.rados.rados.0.smithi064.stdout:1158: writing smithi06431743-40 from 628727 to 1337185 tid 1 2024-03-04T06:12:28.275 INFO:tasks.rados.rados.0.smithi064.stdout:1158: writing smithi06431743-40 from 2045457 to 2540316 tid 2 2024-03-04T06:12:28.282 INFO:tasks.rados.rados.0.smithi064.stdout:1158: writing smithi06431743-40 from 3123240 to 3428727 tid 3 2024-03-04T06:12:28.282 INFO:tasks.rados.rados.0.smithi064.stdout:1159: read oid 43 snap 129 2024-03-04T06:12:28.282 INFO:tasks.rados.rados.0.smithi064.stdout:1159: expect (ObjNum 540 snap 125 seq_num 540) 2024-03-04T06:12:28.282 INFO:tasks.rados.rados.0.smithi064.stdout:1160: snap_remove snap 131 2024-03-04T06:12:28.321 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:12:28.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:12:28.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:28.427+0000 7f2e93e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:28.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:28.435+0000 7f2e93e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:29.252 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:12:29.255 INFO:tasks.rados.rados.0.smithi064.stdout:1160: done (7 left) 2024-03-04T06:12:29.255 INFO:tasks.rados.rados.0.smithi064.stdout:1161: write oid 22 current snap is 132 2024-03-04T06:12:29.255 INFO:tasks.rados.rados.0.smithi064.stdout:1161: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2024-03-04T06:12:29.263 INFO:tasks.rados.rados.0.smithi064.stdout:1161: writing smithi06431743-22 from 645534 to 1429241 tid 1 2024-03-04T06:12:29.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:12:29.273 INFO:tasks.rados.rados.0.smithi064.stdout:1161: writing smithi06431743-22 from 2103939 to 2742456 tid 2 2024-03-04T06:12:29.277 INFO:tasks.rados.rados.0.smithi064.stdout:1161: writing smithi06431743-22 from 3150663 to 3445534 tid 3 2024-03-04T06:12:29.277 INFO:tasks.rados.rados.0.smithi064.stdout:1162: read oid 24 snap -1 2024-03-04T06:12:29.277 INFO:tasks.rados.rados.0.smithi064.stdout:1162: expect deleted 2024-03-04T06:12:29.277 INFO:tasks.rados.rados.0.smithi064.stdout:1163: delete oid 35 current snap is 132 2024-03-04T06:12:29.277 INFO:tasks.rados.rados.0.smithi064.stdout:1154: finishing write tid 1 to smithi06431743-28 2024-03-04T06:12:29.277 INFO:tasks.rados.rados.0.smithi064.stdout:1156: finishing write tid 1 to smithi06431743-41 2024-03-04T06:12:29.278 INFO:tasks.rados.rados.0.smithi064.stdout:1157: expect (ObjNum 449 snap 105 seq_num 449) 2024-03-04T06:12:29.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:29.396+0000 7f77efadd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:29.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:29.404+0000 7f77efadd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:29.481 INFO:tasks.rados.rados.0.smithi064.stdout:1158: finishing write tid 1 to smithi06431743-40 2024-03-04T06:12:29.481 INFO:tasks.rados.rados.0.smithi064.stdout:1156: finishing write tid 2 to smithi06431743-41 2024-03-04T06:12:29.481 INFO:tasks.rados.rados.0.smithi064.stdout:1154: finishing write tid 2 to smithi06431743-28 2024-03-04T06:12:29.481 INFO:tasks.rados.rados.0.smithi064.stdout:1153: done (9 left) 2024-03-04T06:12:29.481 INFO:tasks.rados.rados.0.smithi064.stdout:1157: done (8 left) 2024-03-04T06:12:29.481 INFO:tasks.rados.rados.0.smithi064.stdout:1163: done (7 left) 2024-03-04T06:12:29.481 INFO:tasks.rados.rados.0.smithi064.stdout:1164: read oid 31 snap -1 2024-03-04T06:12:29.481 INFO:tasks.rados.rados.0.smithi064.stdout:1164: expect deleted 2024-03-04T06:12:29.481 INFO:tasks.rados.rados.0.smithi064.stdout:1165: snap_remove snap 119 2024-03-04T06:12:29.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:30.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:30.038+0000 7f0a3319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:30.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:30.041+0000 7f0a3319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:30.252 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:12:30.256 INFO:tasks.rados.rados.0.smithi064.stdout:1165: done (8 left) 2024-03-04T06:12:30.256 INFO:tasks.rados.rados.0.smithi064.stdout:1166: snap_create 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1158: finishing write tid 2 to smithi06431743-40 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1154: finishing write tid 3 to smithi06431743-28 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1156: finishing write tid 3 to smithi06431743-41 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1154: finishing write tid 4 to smithi06431743-28 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1154: finishing write tid 5 to smithi06431743-28 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1154: oid 28 updating version 0 to 449 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1154: oid 28 version 449 is already newer than 448 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1154: oid 28 updating version 449 to 451 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1154: oid 28 version 451 is already newer than 451 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 451 (ObjNum 559 snap 132 seq_num 559) dirty exists 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1154: left oid 28 (ObjNum 559 snap 132 seq_num 559) 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1158: finishing write tid 3 to smithi06431743-40 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1156: finishing write tid 4 to smithi06431743-41 2024-03-04T06:12:30.257 INFO:tasks.rados.rados.0.smithi064.stdout:1158: finishing write tid 4 to smithi06431743-40 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1156: finishing write tid 5 to smithi06431743-41 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1156: oid 41 updating version 0 to 218 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1156: oid 41 version 218 is already newer than 217 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1156: oid 41 version 218 is already newer than 217 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1156: oid 41 version 218 is already newer than 215 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 218 (ObjNum 560 snap 132 seq_num 560) dirty exists 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1156: left oid 41 (ObjNum 560 snap 132 seq_num 560) 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1158: finishing write tid 5 to smithi06431743-40 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1158: oid 40 updating version 0 to 189 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1158: oid 40 version 189 is already newer than 187 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1158: oid 40 version 189 is already newer than 186 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1158: oid 40 version 189 is already newer than 189 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 189 (ObjNum 561 snap 132 seq_num 561) dirty exists 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1158: left oid 40 (ObjNum 561 snap 132 seq_num 561) 2024-03-04T06:12:30.258 INFO:tasks.rados.rados.0.smithi064.stdout:1159: expect (ObjNum 540 snap 125 seq_num 540) 2024-03-04T06:12:30.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:12:30.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:30.381+0000 7f4d490dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:30.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:30.391+0000 7f4d490dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:30.569 INFO:tasks.rados.rados.0.smithi064.stdout:1161: finishing write tid 1 to smithi06431743-22 2024-03-04T06:12:30.569 INFO:tasks.rados.rados.0.smithi064.stdout:1161: finishing write tid 2 to smithi06431743-22 2024-03-04T06:12:30.569 INFO:tasks.rados.rados.0.smithi064.stdout:1161: finishing write tid 3 to smithi06431743-22 2024-03-04T06:12:30.570 INFO:tasks.rados.rados.0.smithi064.stdout:1161: finishing write tid 4 to smithi06431743-22 2024-03-04T06:12:30.570 INFO:tasks.rados.rados.0.smithi064.stdout:1161: finishing write tid 5 to smithi06431743-22 2024-03-04T06:12:30.570 INFO:tasks.rados.rados.0.smithi064.stdout:1161: oid 22 updating version 0 to 409 2024-03-04T06:12:30.570 INFO:tasks.rados.rados.0.smithi064.stdout:1161: oid 22 version 409 is already newer than 408 2024-03-04T06:12:30.570 INFO:tasks.rados.rados.0.smithi064.stdout:1161: oid 22 updating version 409 to 410 2024-03-04T06:12:30.570 INFO:tasks.rados.rados.0.smithi064.stdout:1161: oid 22 updating version 410 to 411 2024-03-04T06:12:30.570 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 411 (ObjNum 562 snap 132 seq_num 562) dirty exists 2024-03-04T06:12:30.570 INFO:tasks.rados.rados.0.smithi064.stdout:1161: left oid 22 (ObjNum 562 snap 132 seq_num 562) 2024-03-04T06:12:31.253 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:12:31.256 INFO:tasks.rados.rados.0.smithi064.stdout:1154: done (8 left) 2024-03-04T06:12:31.256 INFO:tasks.rados.rados.0.smithi064.stdout:1155: done (7 left) 2024-03-04T06:12:31.256 INFO:tasks.rados.rados.0.smithi064.stdout:1156: done (6 left) 2024-03-04T06:12:31.256 INFO:tasks.rados.rados.0.smithi064.stdout:1158: done (5 left) 2024-03-04T06:12:31.256 INFO:tasks.rados.rados.0.smithi064.stdout:1159: done (4 left) 2024-03-04T06:12:31.256 INFO:tasks.rados.rados.0.smithi064.stdout:1161: done (3 left) 2024-03-04T06:12:31.256 INFO:tasks.rados.rados.0.smithi064.stdout:1162: done (2 left) 2024-03-04T06:12:31.256 INFO:tasks.rados.rados.0.smithi064.stdout:1164: done (1 left) 2024-03-04T06:12:31.256 INFO:tasks.rados.rados.0.smithi064.stdout:1166: done (0 left) 2024-03-04T06:12:31.256 INFO:tasks.rados.rados.0.smithi064.stdout:1167: write oid 35 current snap is 133 2024-03-04T06:12:31.257 INFO:tasks.rados.rados.0.smithi064.stdout:1167: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2024-03-04T06:12:31.265 INFO:tasks.rados.rados.0.smithi064.stdout:1167: writing smithi06431743-35 from 679148 to 1329706 tid 1 2024-03-04T06:12:31.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:12:31.274 INFO:tasks.rados.rados.0.smithi064.stdout:1167: writing smithi06431743-35 from 1937256 to 2579442 tid 2 2024-03-04T06:12:31.280 INFO:tasks.rados.rados.0.smithi064.stdout:1167: writing smithi06431743-35 from 3321862 to 3479148 tid 3 2024-03-04T06:12:31.280 INFO:tasks.rados.rados.0.smithi064.stdout:1168: read oid 33 snap 127 2024-03-04T06:12:31.280 INFO:tasks.rados.rados.0.smithi064.stdout:1168: expect (ObjNum 449 snap 105 seq_num 449) 2024-03-04T06:12:31.280 INFO:tasks.rados.rados.0.smithi064.stdout:1169: delete oid 26 current snap is 133 2024-03-04T06:12:31.312 INFO:tasks.rados.rados.0.smithi064.stdout:1169: done (2 left) 2024-03-04T06:12:31.312 INFO:tasks.rados.rados.0.smithi064.stdout:1170: read oid 27 snap -1 2024-03-04T06:12:31.312 INFO:tasks.rados.rados.0.smithi064.stdout:1170: expect (ObjNum 451 snap 105 seq_num 451) 2024-03-04T06:12:31.312 INFO:tasks.rados.rados.0.smithi064.stdout:1171: write oid 15 current snap is 133 2024-03-04T06:12:31.312 INFO:tasks.rados.rados.0.smithi064.stdout:1171: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2024-03-04T06:12:31.318 INFO:tasks.rados.rados.0.smithi064.stdout:1171: writing smithi06431743-15 from 712762 to 1113818 tid 1 2024-03-04T06:12:31.325 INFO:tasks.rados.rados.0.smithi064.stdout:1171: writing smithi06431743-15 from 1537867 to 2183722 tid 2 2024-03-04T06:12:31.334 INFO:tasks.rados.rados.0.smithi064.stdout:1171: writing smithi06431743-15 from 2860355 to 3512762 tid 3 2024-03-04T06:12:31.334 INFO:tasks.rados.rados.0.smithi064.stdout:1172: write (excl) oid 45 current snap is 133 2024-03-04T06:12:31.334 INFO:tasks.rados.rados.0.smithi064.stdout:1167: finishing write tid 1 to smithi06431743-35 2024-03-04T06:12:31.334 INFO:tasks.rados.rados.0.smithi064.stdout:1172: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-03-04T06:12:31.340 INFO:tasks.rados.rados.0.smithi064.stdout:1172: writing smithi06431743-45 from 729569 to 1205874 tid 1 2024-03-04T06:12:31.347 INFO:tasks.rados.rados.0.smithi064.stdout:1172: writing smithi06431743-45 from 1996349 to 2502215 tid 2 2024-03-04T06:12:31.353 INFO:tasks.rados.rados.0.smithi064.stdout:1172: writing smithi06431743-45 from 3004131 to 3529569 tid 3 2024-03-04T06:12:31.353 INFO:tasks.rados.rados.0.smithi064.stdout:1173: snap_remove snap 132 2024-03-04T06:12:31.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:31.421+0000 7fb57de6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:31.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:31.430+0000 7fb57de6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:32.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:32.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:32.221+0000 7f170ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:32.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:32.225+0000 7f170ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:32.260 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:12:32.262 INFO:tasks.rados.rados.0.smithi064.stdout:1173: done (5 left) 2024-03-04T06:12:32.263 INFO:tasks.rados.rados.0.smithi064.stdout:1174: delete oid 19 current snap is 133 2024-03-04T06:12:32.263 INFO:tasks.rados.rados.0.smithi064.stdout:1168: expect (ObjNum 449 snap 105 seq_num 449) 2024-03-04T06:12:32.273 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:12:32.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:12:32.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:32.392+0000 7faec579d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:32.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:32.399+0000 7faec579d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:32.461 INFO:tasks.rados.rados.0.smithi064.stdout:1170: expect (ObjNum 451 snap 105 seq_num 451) 2024-03-04T06:12:32.589 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:12:32.590 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:12:32.598 INFO:tasks.rados.rados.0.smithi064.stdout:1167: finishing write tid 2 to smithi06431743-35 2024-03-04T06:12:32.598 INFO:tasks.rados.rados.0.smithi064.stdout:1167: finishing write tid 3 to smithi06431743-35 2024-03-04T06:12:32.598 INFO:tasks.rados.rados.0.smithi064.stdout:1167: finishing write tid 4 to smithi06431743-35 2024-03-04T06:12:32.598 INFO:tasks.rados.rados.0.smithi064.stdout:1171: finishing write tid 1 to smithi06431743-15 2024-03-04T06:12:32.598 INFO:tasks.rados.rados.0.smithi064.stdout:1167: finishing write tid 5 to smithi06431743-35 2024-03-04T06:12:32.598 INFO:tasks.rados.rados.0.smithi064.stdout:1167: oid 35 updating version 0 to 319 2024-03-04T06:12:32.598 INFO:tasks.rados.rados.0.smithi064.stdout:1167: oid 35 version 319 is already newer than 319 2024-03-04T06:12:32.598 INFO:tasks.rados.rados.0.smithi064.stdout:1167: oid 35 version 319 is already newer than 317 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1167: oid 35 version 319 is already newer than 316 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 319 (ObjNum 564 snap 133 seq_num 564) dirty exists 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1167: left oid 35 (ObjNum 564 snap 133 seq_num 564) 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1172: finishing write tid 1 to smithi06431743-45 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1171: finishing write tid 2 to smithi06431743-15 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1172: finishing write tid 2 to smithi06431743-45 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1171: finishing write tid 3 to smithi06431743-15 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1171: finishing write tid 4 to smithi06431743-15 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1171: finishing write tid 5 to smithi06431743-15 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1171: oid 15 updating version 0 to 176 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1171: oid 15 updating version 176 to 179 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1171: oid 15 version 179 is already newer than 179 2024-03-04T06:12:32.599 INFO:tasks.rados.rados.0.smithi064.stdout:1171: oid 15 version 179 is already newer than 177 2024-03-04T06:12:32.600 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 179 (ObjNum 566 snap 133 seq_num 566) dirty exists 2024-03-04T06:12:32.600 INFO:tasks.rados.rados.0.smithi064.stdout:1171: left oid 15 (ObjNum 566 snap 133 seq_num 566) 2024-03-04T06:12:32.600 INFO:tasks.rados.rados.0.smithi064.stdout:1172: finishing write tid 3 to smithi06431743-45 2024-03-04T06:12:32.600 INFO:tasks.rados.rados.0.smithi064.stdout:1172: finishing write tid 4 to smithi06431743-45 2024-03-04T06:12:32.600 INFO:tasks.rados.rados.0.smithi064.stdout:1172: finishing write tid 5 to smithi06431743-45 2024-03-04T06:12:32.600 INFO:tasks.rados.rados.0.smithi064.stdout:1172: oid 45 updating version 0 to 256 2024-03-04T06:12:32.600 INFO:tasks.rados.rados.0.smithi064.stdout:1172: oid 45 version 256 is already newer than 253 2024-03-04T06:12:32.600 INFO:tasks.rados.rados.0.smithi064.stdout:1172: oid 45 version 256 is already newer than 255 2024-03-04T06:12:32.600 INFO:tasks.rados.rados.0.smithi064.stdout:1172: oid 45 version 256 is already newer than 254 2024-03-04T06:12:32.600 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 256 (ObjNum 567 snap 133 seq_num 567) dirty exists 2024-03-04T06:12:32.600 INFO:tasks.rados.rados.0.smithi064.stdout:1172: left oid 45 (ObjNum 567 snap 133 seq_num 567) 2024-03-04T06:12:32.601 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-03-04T06:12:32.261050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115318382,"num_objects":391,"num_object_clones":283,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2253,"num_read_kb":1602128,"num_write":2015,"num_write_kb":514325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3794,"ondisk_log_size":3794,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":404172,"kb_used_data":259020,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377083188,"statfs":{"total":386547056640,"available":386133184512,"internally_reserved":0,"allocated":265236480,"data_stored":285763967,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13698002,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":47,"num_read_kb":41643,"num_write":51,"num_write_kb":15114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051146"},"pg_stats":[{"pgid":"3.f","version":"280'193","reported_seq":934,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.282121+0000","last_change":"2024-03-04T06:12:30.313838+0000","last_active":"2024-03-04T06:12:30.282121+0000","last_peered":"2024-03-04T06:12:30.282121+0000","last_clean":"2024-03-04T06:12:30.282121+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:12:30.282121+0000","last_undegraded":"2024-03-04T06:12:30.282121+0000","last_fullsized":"2024-03-04T06:12:30.282121+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9664289,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":77,"num_read_kb":39366,"num_write":113,"num_write_kb":30557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"267'155","reported_seq":921,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.284106+0000","last_change":"2024-03-04T06:12:29.298501+0000","last_active":"2024-03-04T06:12:29.284106+0000","last_peered":"2024-03-04T06:12:29.284106+0000","last_clean":"2024-03-04T06:12:29.284106+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:12:29.284106+0000","last_undegraded":"2024-03-04T06:12:29.284106+0000","last_fullsized":"2024-03-04T06:12:29.284106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269176,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":63,"num_read_kb":40085,"num_write":72,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"265'251","reported_seq":980,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.343900+0000","last_change":"2024-03-04T06:12:30.423437+0000","last_active":"2024-03-04T06:12:31.343900+0000","last_peered":"2024-03-04T06:12:31.343900+0000","last_clean":"2024-03-04T06:12:31.343900+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:12:31.343900+0000","last_undegraded":"2024-03-04T06:12:31.343900+0000","last_fullsized":"2024-03-04T06:12:31.343900+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193693,"num_objects":29,"num_object_clones":22,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":144,"num_read_kb":95338,"num_write":136,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"280'188","reported_seq":957,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.272279+0000","last_change":"2024-03-04T06:12:30.300305+0000","last_active":"2024-03-04T06:12:30.272279+0000","last_peered":"2024-03-04T06:12:30.272279+0000","last_clean":"2024-03-04T06:12:30.272279+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:12:30.272279+0000","last_undegraded":"2024-03-04T06:12:30.272279+0000","last_fullsized":"2024-03-04T06:12:30.272279+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9227340,"num_objects":21,"num_object_clones":13,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":118,"num_read_kb":91992,"num_write":114,"num_write_kb":32234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"249'92","reported_seq":854,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.291569+0000","last_change":"2024-03-04T06:12:30.429717+0000","last_active":"2024-03-04T06:12:31.291569+0000","last_peered":"2024-03-04T06:12:31.291569+0000","last_clean":"2024-03-04T06:12:31.291569+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:12:31.291569+0000","last_undegraded":"2024-03-04T06:12:31.291569+0000","last_fullsized":"2024-03-04T06:12:31.291569+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":17041,"num_write":34,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"280'196","reported_seq":951,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.279168+0000","last_change":"2024-03-04T06:12:30.302450+0000","last_active":"2024-03-04T06:12:30.279168+0000","last_peered":"2024-03-04T06:12:30.279168+0000","last_clean":"2024-03-04T06:12:30.279168+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:12:30.279168+0000","last_undegraded":"2024-03-04T06:12:30.279168+0000","last_fullsized":"2024-03-04T06:12:30.279168+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134753,"num_objects":21,"num_object_clones":15,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":118,"num_read_kb":87289,"num_write":111,"num_write_kb":33076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":801,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.281031+0000","last_change":"2024-03-04T06:12:30.294096+0000","last_active":"2024-03-04T06:12:30.281031+0000","last_peered":"2024-03-04T06:12:30.281031+0000","last_clean":"2024-03-04T06:12:30.281031+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:12:30.281031+0000","last_undegraded":"2024-03-04T06:12:30.281031+0000","last_fullsized":"2024-03-04T06:12:30.281031+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"280'156","reported_seq":914,"reported_epoch":281,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:31.348885+0000","last_change":"2024-03-04T06:12:30.395347+0000","last_active":"2024-03-04T06:12:31.348885+0000","last_peered":"2024-03-04T06:12:31.348885+0000","last_clean":"2024-03-04T06:12:31.348885+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:12:31.348885+0000","last_undegraded":"2024-03-04T06:12:31.348885+0000","last_fullsized":"2024-03-04T06:12:31.348885+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252237,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":116,"num_read_kb":111954,"num_write":72,"num_write_kb":19512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":822,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.278592+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:12:31.278592+0000","last_peered":"2024-03-04T06:12:31.278592+0000","last_clean":"2024-03-04T06:12:31.278592+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:12:31.278592+0000","last_undegraded":"2024-03-04T06:12:31.278592+0000","last_fullsized":"2024-03-04T06:12:31.278592+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":677,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.349496+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:12:31.349496+0000","last_peered":"2024-03-04T06:12:31.349496+0000","last_clean":"2024-03-04T06:12:31.349496+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:12:31.349496+0000","last_undegraded":"2024-03-04T06:12:31.349496+0000","last_fullsized":"2024-03-04T06:12:31.349496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"280'222","reported_seq":962,"reported_epoch":281,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:31.348774+0000","last_change":"2024-03-04T06:12:30.405593+0000","last_active":"2024-03-04T06:12:31.348774+0000","last_peered":"2024-03-04T06:12:31.348774+0000","last_clean":"2024-03-04T06:12:31.348774+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:12:31.348774+0000","last_undegraded":"2024-03-04T06:12:31.348774+0000","last_fullsized":"2024-03-04T06:12:31.348774+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9714743,"num_objects":24,"num_object_clones":17,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":141,"num_read_kb":109563,"num_write":114,"num_write_kb":29143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":680,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.280946+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:12:30.280946+0000","last_peered":"2024-03-04T06:12:30.280946+0000","last_clean":"2024-03-04T06:12:30.280946+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:12:30.280946+0000","last_undegraded":"2024-03-04T06:12:30.280946+0000","last_fullsized":"2024-03-04T06:12:30.280946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"280'214","reported_seq":931,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.282359+0000","last_change":"2024-03-04T06:12:30.481562+0000","last_active":"2024-03-04T06:12:31.282359+0000","last_peered":"2024-03-04T06:12:31.282359+0000","last_clean":"2024-03-04T06:12:31.282359+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:12:31.282359+0000","last_undegraded":"2024-03-04T06:12:31.282359+0000","last_fullsized":"2024-03-04T06:12:31.282359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3159815,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":100,"num_read_kb":76866,"num_write":87,"num_write_kb":25445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":677,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.341539+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:12:31.341539+0000","last_peered":"2024-03-04T06:12:31.341539+0000","last_clean":"2024-03-04T06:12:31.341539+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:12:31.341539+0000","last_undegraded":"2024-03-04T06:12:31.341539+0000","last_fullsized":"2024-03-04T06:12:31.341539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"280'453","reported_seq":1170,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.289856+0000","last_change":"2024-03-04T06:12:30.482057+0000","last_active":"2024-03-04T06:12:31.289856+0000","last_peered":"2024-03-04T06:12:31.289856+0000","last_clean":"2024-03-04T06:12:31.289856+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:12:31.289856+0000","last_undegraded":"2024-03-04T06:12:31.289856+0000","last_fullsized":"2024-03-04T06:12:31.289856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13025788,"num_objects":46,"num_object_clones":34,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":311,"num_read_kb":258282,"num_write":244,"num_write_kb":67716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":680,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.270724+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:12:30.270724+0000","last_peered":"2024-03-04T06:12:30.270724+0000","last_clean":"2024-03-04T06:12:30.270724+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:12:30.270724+0000","last_undegraded":"2024-03-04T06:12:30.270724+0000","last_fullsized":"2024-03-04T06:12:30.270724+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"277'468","reported_seq":1119,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.287104+0000","last_change":"2024-03-04T06:12:29.297776+0000","last_active":"2024-03-04T06:12:29.287104+0000","last_peered":"2024-03-04T06:12:29.287104+0000","last_clean":"2024-03-04T06:12:29.287104+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:12:29.287104+0000","last_undegraded":"2024-03-04T06:12:29.287104+0000","last_fullsized":"2024-03-04T06:12:29.287104+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13109790,"num_objects":47,"num_object_clones":34,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":201,"num_read_kb":139976,"num_write":227,"num_write_kb":60703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"277'174","reported_seq":964,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.350819+0000","last_change":"2024-03-04T06:12:30.412156+0000","last_active":"2024-03-04T06:12:31.350819+0000","last_peered":"2024-03-04T06:12:31.350819+0000","last_clean":"2024-03-04T06:12:31.350819+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:12:31.350819+0000","last_undegraded":"2024-03-04T06:12:31.350819+0000","last_fullsized":"2024-03-04T06:12:31.350819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6622123,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":183,"num_read_kb":148566,"num_write":91,"num_write_kb":23814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":711,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.291009+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:12:29.291009+0000","last_peered":"2024-03-04T06:12:29.291009+0000","last_clean":"2024-03-04T06:12:29.291009+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:12:29.291009+0000","last_undegraded":"2024-03-04T06:12:29.291009+0000","last_fullsized":"2024-03-04T06:12:29.291009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"274'206","reported_seq":978,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.295593+0000","last_change":"2024-03-04T06:12:29.299943+0000","last_active":"2024-03-04T06:12:29.295593+0000","last_peered":"2024-03-04T06:12:29.295593+0000","last_clean":"2024-03-04T06:12:29.295593+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:12:29.295593+0000","last_undegraded":"2024-03-04T06:12:29.295593+0000","last_fullsized":"2024-03-04T06:12:29.295593+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8840779,"num_objects":26,"num_object_clones":16,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":115,"num_read_kb":67271,"num_write":110,"num_write_kb":30018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":689,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.292629+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:12:31.292629+0000","last_peered":"2024-03-04T06:12:31.292629+0000","last_clean":"2024-03-04T06:12:31.292629+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:12:31.292629+0000","last_undegraded":"2024-03-04T06:12:31.292629+0000","last_fullsized":"2024-03-04T06:12:31.292629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"281'318","reported_seq":1032,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.351717+0000","last_change":"2024-03-04T06:12:30.489692+0000","last_active":"2024-03-04T06:12:31.351717+0000","last_peered":"2024-03-04T06:12:31.351717+0000","last_clean":"2024-03-04T06:12:31.351717+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:12:31.351717+0000","last_undegraded":"2024-03-04T06:12:31.351717+0000","last_fullsized":"2024-03-04T06:12:31.351717+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9210533,"num_objects":32,"num_object_clones":24,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":170,"num_read_kb":115118,"num_write":166,"num_write_kb":45507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":711,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.275941+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:12:29.275941+0000","last_peered":"2024-03-04T06:12:29.275941+0000","last_clean":"2024-03-04T06:12:29.275941+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:12:29.275941+0000","last_undegraded":"2024-03-04T06:12:29.275941+0000","last_fullsized":"2024-03-04T06:12:29.275941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"279'411","reported_seq":1117,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.372269+0000","last_change":"2024-03-04T06:12:29.373824+0000","last_active":"2024-03-04T06:12:29.372269+0000","last_peered":"2024-03-04T06:12:29.372269+0000","last_clean":"2024-03-04T06:12:29.372269+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:12:29.372269+0000","last_undegraded":"2024-03-04T06:12:29.372269+0000","last_fullsized":"2024-03-04T06:12:29.372269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10302955,"num_objects":36,"num_object_clones":32,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":262,"num_read_kb":203330,"num_write":187,"num_write_kb":50586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":677,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.352658+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:12:31.352658+0000","last_peered":"2024-03-04T06:12:31.352658+0000","last_clean":"2024-03-04T06:12:31.352658+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:12:31.352658+0000","last_undegraded":"2024-03-04T06:12:31.352658+0000","last_fullsized":"2024-03-04T06:12:31.352658+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":114728014,"num_objects":389,"num_object_clones":283,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2150,"num_read_kb":1602037,"num_write":1878,"num_write_kb":512431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3697,"ondisk_log_size":3697,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112872,"kb_used_data":74568,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94258968,"statfs":{"total":96636764160,"available":96521183232,"internally_reserved":0,"allocated":76357632,"data_stored":80811543,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99280,"kb_used_data":62720,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94272560,"statfs":{"total":96636764160,"available":96535101440,"internally_reserved":0,"allocated":64225280,"data_stored":69348782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705948,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":92088,"kb_used_data":56944,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94279752,"statfs":{"total":96636764160,"available":96542466048,"internally_reserved":0,"allocated":58310656,"data_stored":61931851,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99932,"kb_used_data":64788,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94271908,"statfs":{"total":96636764160,"available":96534433792,"internally_reserved":0,"allocated":66342912,"data_stored":73671791,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:12:32.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:12:32.605 INFO:tasks.rados.rados.0.smithi064.stdout:1167: done (5 left) 2024-03-04T06:12:32.605 INFO:tasks.rados.rados.0.smithi064.stdout:1168: done (4 left) 2024-03-04T06:12:32.605 INFO:tasks.rados.rados.0.smithi064.stdout:1170: done (3 left) 2024-03-04T06:12:32.605 INFO:tasks.rados.rados.0.smithi064.stdout:1171: done (2 left) 2024-03-04T06:12:32.606 INFO:tasks.rados.rados.0.smithi064.stdout:1172: done (1 left) 2024-03-04T06:12:32.606 INFO:tasks.rados.rados.0.smithi064.stdout:1174: done (0 left) 2024-03-04T06:12:32.606 INFO:tasks.rados.rados.0.smithi064.stdout:1175: snap_create 2024-03-04T06:12:32.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:32.705+0000 7ff96d8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:32.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:32.711+0000 7ff96d8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:32.901 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:12:32.901 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:12:32.913 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-03-04T06:12:32.765569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115318382,"num_objects":391,"num_object_clones":283,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2253,"num_read_kb":1602128,"num_write":2015,"num_write_kb":514325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3794,"ondisk_log_size":3794,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":404172,"kb_used_data":259020,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377083188,"statfs":{"total":386547056640,"available":386133184512,"internally_reserved":0,"allocated":265236480,"data_stored":285763967,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13698002,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":47,"num_read_kb":41643,"num_write":51,"num_write_kb":15114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000911"},"pg_stats":[{"pgid":"3.f","version":"280'193","reported_seq":934,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.282121+0000","last_change":"2024-03-04T06:12:30.313838+0000","last_active":"2024-03-04T06:12:30.282121+0000","last_peered":"2024-03-04T06:12:30.282121+0000","last_clean":"2024-03-04T06:12:30.282121+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:12:30.282121+0000","last_undegraded":"2024-03-04T06:12:30.282121+0000","last_fullsized":"2024-03-04T06:12:30.282121+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9664289,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":77,"num_read_kb":39366,"num_write":113,"num_write_kb":30557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"267'155","reported_seq":921,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.284106+0000","last_change":"2024-03-04T06:12:29.298501+0000","last_active":"2024-03-04T06:12:29.284106+0000","last_peered":"2024-03-04T06:12:29.284106+0000","last_clean":"2024-03-04T06:12:29.284106+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:12:29.284106+0000","last_undegraded":"2024-03-04T06:12:29.284106+0000","last_fullsized":"2024-03-04T06:12:29.284106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269176,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":63,"num_read_kb":40085,"num_write":72,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"265'251","reported_seq":980,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.343900+0000","last_change":"2024-03-04T06:12:30.423437+0000","last_active":"2024-03-04T06:12:31.343900+0000","last_peered":"2024-03-04T06:12:31.343900+0000","last_clean":"2024-03-04T06:12:31.343900+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:12:31.343900+0000","last_undegraded":"2024-03-04T06:12:31.343900+0000","last_fullsized":"2024-03-04T06:12:31.343900+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193693,"num_objects":29,"num_object_clones":22,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":144,"num_read_kb":95338,"num_write":136,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"280'188","reported_seq":957,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.272279+0000","last_change":"2024-03-04T06:12:30.300305+0000","last_active":"2024-03-04T06:12:30.272279+0000","last_peered":"2024-03-04T06:12:30.272279+0000","last_clean":"2024-03-04T06:12:30.272279+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:12:30.272279+0000","last_undegraded":"2024-03-04T06:12:30.272279+0000","last_fullsized":"2024-03-04T06:12:30.272279+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9227340,"num_objects":21,"num_object_clones":13,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":118,"num_read_kb":91992,"num_write":114,"num_write_kb":32234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"249'92","reported_seq":854,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.291569+0000","last_change":"2024-03-04T06:12:30.429717+0000","last_active":"2024-03-04T06:12:31.291569+0000","last_peered":"2024-03-04T06:12:31.291569+0000","last_clean":"2024-03-04T06:12:31.291569+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:12:31.291569+0000","last_undegraded":"2024-03-04T06:12:31.291569+0000","last_fullsized":"2024-03-04T06:12:31.291569+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":17041,"num_write":34,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"280'196","reported_seq":951,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.279168+0000","last_change":"2024-03-04T06:12:30.302450+0000","last_active":"2024-03-04T06:12:30.279168+0000","last_peered":"2024-03-04T06:12:30.279168+0000","last_clean":"2024-03-04T06:12:30.279168+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:12:30.279168+0000","last_undegraded":"2024-03-04T06:12:30.279168+0000","last_fullsized":"2024-03-04T06:12:30.279168+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134753,"num_objects":21,"num_object_clones":15,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":118,"num_read_kb":87289,"num_write":111,"num_write_kb":33076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":801,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.281031+0000","last_change":"2024-03-04T06:12:30.294096+0000","last_active":"2024-03-04T06:12:30.281031+0000","last_peered":"2024-03-04T06:12:30.281031+0000","last_clean":"2024-03-04T06:12:30.281031+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:12:30.281031+0000","last_undegraded":"2024-03-04T06:12:30.281031+0000","last_fullsized":"2024-03-04T06:12:30.281031+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"280'156","reported_seq":914,"reported_epoch":281,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:31.348885+0000","last_change":"2024-03-04T06:12:30.395347+0000","last_active":"2024-03-04T06:12:31.348885+0000","last_peered":"2024-03-04T06:12:31.348885+0000","last_clean":"2024-03-04T06:12:31.348885+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:12:31.348885+0000","last_undegraded":"2024-03-04T06:12:31.348885+0000","last_fullsized":"2024-03-04T06:12:31.348885+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252237,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":116,"num_read_kb":111954,"num_write":72,"num_write_kb":19512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":822,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.278592+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:12:31.278592+0000","last_peered":"2024-03-04T06:12:31.278592+0000","last_clean":"2024-03-04T06:12:31.278592+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:12:31.278592+0000","last_undegraded":"2024-03-04T06:12:31.278592+0000","last_fullsized":"2024-03-04T06:12:31.278592+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":677,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.349496+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:12:31.349496+0000","last_peered":"2024-03-04T06:12:31.349496+0000","last_clean":"2024-03-04T06:12:31.349496+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:12:31.349496+0000","last_undegraded":"2024-03-04T06:12:31.349496+0000","last_fullsized":"2024-03-04T06:12:31.349496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"280'222","reported_seq":962,"reported_epoch":281,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:31.348774+0000","last_change":"2024-03-04T06:12:30.405593+0000","last_active":"2024-03-04T06:12:31.348774+0000","last_peered":"2024-03-04T06:12:31.348774+0000","last_clean":"2024-03-04T06:12:31.348774+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:12:31.348774+0000","last_undegraded":"2024-03-04T06:12:31.348774+0000","last_fullsized":"2024-03-04T06:12:31.348774+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9714743,"num_objects":24,"num_object_clones":17,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":141,"num_read_kb":109563,"num_write":114,"num_write_kb":29143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":680,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.280946+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:12:30.280946+0000","last_peered":"2024-03-04T06:12:30.280946+0000","last_clean":"2024-03-04T06:12:30.280946+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:12:30.280946+0000","last_undegraded":"2024-03-04T06:12:30.280946+0000","last_fullsized":"2024-03-04T06:12:30.280946+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"280'214","reported_seq":931,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.282359+0000","last_change":"2024-03-04T06:12:30.481562+0000","last_active":"2024-03-04T06:12:31.282359+0000","last_peered":"2024-03-04T06:12:31.282359+0000","last_clean":"2024-03-04T06:12:31.282359+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:12:31.282359+0000","last_undegraded":"2024-03-04T06:12:31.282359+0000","last_fullsized":"2024-03-04T06:12:31.282359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3159815,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":100,"num_read_kb":76866,"num_write":87,"num_write_kb":25445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":677,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.341539+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:12:31.341539+0000","last_peered":"2024-03-04T06:12:31.341539+0000","last_clean":"2024-03-04T06:12:31.341539+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:12:31.341539+0000","last_undegraded":"2024-03-04T06:12:31.341539+0000","last_fullsized":"2024-03-04T06:12:31.341539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"280'453","reported_seq":1170,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.289856+0000","last_change":"2024-03-04T06:12:30.482057+0000","last_active":"2024-03-04T06:12:31.289856+0000","last_peered":"2024-03-04T06:12:31.289856+0000","last_clean":"2024-03-04T06:12:31.289856+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:12:31.289856+0000","last_undegraded":"2024-03-04T06:12:31.289856+0000","last_fullsized":"2024-03-04T06:12:31.289856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13025788,"num_objects":46,"num_object_clones":34,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":311,"num_read_kb":258282,"num_write":244,"num_write_kb":67716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":680,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-03-04T06:12:30.270724+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:12:30.270724+0000","last_peered":"2024-03-04T06:12:30.270724+0000","last_clean":"2024-03-04T06:12:30.270724+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:12:30.270724+0000","last_undegraded":"2024-03-04T06:12:30.270724+0000","last_fullsized":"2024-03-04T06:12:30.270724+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"277'468","reported_seq":1119,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.287104+0000","last_change":"2024-03-04T06:12:29.297776+0000","last_active":"2024-03-04T06:12:29.287104+0000","last_peered":"2024-03-04T06:12:29.287104+0000","last_clean":"2024-03-04T06:12:29.287104+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:12:29.287104+0000","last_undegraded":"2024-03-04T06:12:29.287104+0000","last_fullsized":"2024-03-04T06:12:29.287104+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13109790,"num_objects":47,"num_object_clones":34,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":201,"num_read_kb":139976,"num_write":227,"num_write_kb":60703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"277'174","reported_seq":964,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.350819+0000","last_change":"2024-03-04T06:12:30.412156+0000","last_active":"2024-03-04T06:12:31.350819+0000","last_peered":"2024-03-04T06:12:31.350819+0000","last_clean":"2024-03-04T06:12:31.350819+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:12:31.350819+0000","last_undegraded":"2024-03-04T06:12:31.350819+0000","last_fullsized":"2024-03-04T06:12:31.350819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6622123,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":183,"num_read_kb":148566,"num_write":91,"num_write_kb":23814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":711,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.291009+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:12:29.291009+0000","last_peered":"2024-03-04T06:12:29.291009+0000","last_clean":"2024-03-04T06:12:29.291009+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:12:29.291009+0000","last_undegraded":"2024-03-04T06:12:29.291009+0000","last_fullsized":"2024-03-04T06:12:29.291009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"274'206","reported_seq":978,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.295593+0000","last_change":"2024-03-04T06:12:29.299943+0000","last_active":"2024-03-04T06:12:29.295593+0000","last_peered":"2024-03-04T06:12:29.295593+0000","last_clean":"2024-03-04T06:12:29.295593+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:12:29.295593+0000","last_undegraded":"2024-03-04T06:12:29.295593+0000","last_fullsized":"2024-03-04T06:12:29.295593+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8840779,"num_objects":26,"num_object_clones":16,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":115,"num_read_kb":67271,"num_write":110,"num_write_kb":30018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":689,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.292629+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:12:31.292629+0000","last_peered":"2024-03-04T06:12:31.292629+0000","last_clean":"2024-03-04T06:12:31.292629+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:12:31.292629+0000","last_undegraded":"2024-03-04T06:12:31.292629+0000","last_fullsized":"2024-03-04T06:12:31.292629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"281'318","reported_seq":1032,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.351717+0000","last_change":"2024-03-04T06:12:30.489692+0000","last_active":"2024-03-04T06:12:31.351717+0000","last_peered":"2024-03-04T06:12:31.351717+0000","last_clean":"2024-03-04T06:12:31.351717+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:12:31.351717+0000","last_undegraded":"2024-03-04T06:12:31.351717+0000","last_fullsized":"2024-03-04T06:12:31.351717+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9210533,"num_objects":32,"num_object_clones":24,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":170,"num_read_kb":115118,"num_write":166,"num_write_kb":45507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":711,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.275941+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:12:29.275941+0000","last_peered":"2024-03-04T06:12:29.275941+0000","last_clean":"2024-03-04T06:12:29.275941+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:12:29.275941+0000","last_undegraded":"2024-03-04T06:12:29.275941+0000","last_fullsized":"2024-03-04T06:12:29.275941+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"279'411","reported_seq":1117,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-03-04T06:12:29.372269+0000","last_change":"2024-03-04T06:12:29.373824+0000","last_active":"2024-03-04T06:12:29.372269+0000","last_peered":"2024-03-04T06:12:29.372269+0000","last_clean":"2024-03-04T06:12:29.372269+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:12:29.372269+0000","last_undegraded":"2024-03-04T06:12:29.372269+0000","last_fullsized":"2024-03-04T06:12:29.372269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10302955,"num_objects":36,"num_object_clones":32,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":262,"num_read_kb":203330,"num_write":187,"num_write_kb":50586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":677,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-03-04T06:12:31.352658+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:12:31.352658+0000","last_peered":"2024-03-04T06:12:31.352658+0000","last_clean":"2024-03-04T06:12:31.352658+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:12:31.352658+0000","last_undegraded":"2024-03-04T06:12:31.352658+0000","last_fullsized":"2024-03-04T06:12:31.352658+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":114728014,"num_objects":389,"num_object_clones":283,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2150,"num_read_kb":1602037,"num_write":1878,"num_write_kb":512431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3697,"ondisk_log_size":3697,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112872,"kb_used_data":74568,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94258968,"statfs":{"total":96636764160,"available":96521183232,"internally_reserved":0,"allocated":76357632,"data_stored":80811543,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99280,"kb_used_data":62720,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94272560,"statfs":{"total":96636764160,"available":96535101440,"internally_reserved":0,"allocated":64225280,"data_stored":69348782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705948,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":92088,"kb_used_data":56944,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94279752,"statfs":{"total":96636764160,"available":96542466048,"internally_reserved":0,"allocated":58310656,"data_stored":61931851,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99932,"kb_used_data":64788,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94271908,"statfs":{"total":96636764160,"available":96534433792,"internally_reserved":0,"allocated":66342912,"data_stored":73671791,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:12:32.914 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:12:32.914 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:12:32.914 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:12:32.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:12:33.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:33.017+0000 7f918091d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:33.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:33.024+0000 7f918091d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:33.037 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:33.037 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:33.037 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:33.037 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:33.037 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:33.038 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:33.038 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:33.038 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:33.038 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:33.038 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:33.038 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:33.038 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:33.040 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:33.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:12:33.147 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:33.142+0000 7ffa5986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:33.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:33.150+0000 7ffa5986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:33.162 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:33.162 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:33.162 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:33.162 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:33.163 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:33.163 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:33.163 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:33.163 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:33.163 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:33.163 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:33.163 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:33.163 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:33.165 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:33.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:12:33.268 INFO:tasks.rados.rados.0.smithi064.stdout:1175: done (0 left) 2024-03-04T06:12:33.268 INFO:tasks.rados.rados.0.smithi064.stdout:1176: write (excl) oid 9 current snap is 134 2024-03-04T06:12:33.268 INFO:tasks.rados.rados.0.smithi064.stdout:1176: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2024-03-04T06:12:33.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:33.268+0000 7f3877a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:33.276 INFO:tasks.rados.rados.0.smithi064.stdout:1176: writing smithi06431743-9 from 763183 to 1389986 tid 1 2024-03-04T06:12:33.276 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:33.276+0000 7f3877a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:33.284 INFO:tasks.rados.rados.0.smithi064.stdout:1176: writing smithi06431743-9 from 1996960 to 2790142 tid 2 2024-03-04T06:12:33.288 INFO:tasks.rados.rados.0.smithi064.stdout:1176: writing smithi06431743-9 from 3226271 to 3563183 tid 3 2024-03-04T06:12:33.288 INFO:tasks.rados.rados.0.smithi064.stdout:1177: delete oid 39 current snap is 134 2024-03-04T06:12:33.298 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:33.298 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:33.298 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:33.298 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:33.298 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:33.298 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:33.299 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:33.299 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:33.299 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:33.299 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:33.299 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:33.299 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:33.301 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:33.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:12:33.313 INFO:tasks.rados.rados.0.smithi064.stdout:1177: done (1 left) 2024-03-04T06:12:33.314 INFO:tasks.rados.rados.0.smithi064.stdout:1178: read oid 19 snap -1 2024-03-04T06:12:33.314 INFO:tasks.rados.rados.0.smithi064.stdout:1178: expect deleted 2024-03-04T06:12:33.314 INFO:tasks.rados.rados.0.smithi064.stdout:1179: snap_remove snap 127 2024-03-04T06:12:33.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:33.412+0000 7fd9d799d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:33.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:33.419+0000 7fd9d799d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:33.432 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:33.435 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:34.276 INFO:tasks.rados.rados.0.smithi064.stdout:1179: done (2 left) 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:1180: snap_create 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:1176: finishing write tid 1 to smithi06431743-9 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:1176: finishing write tid 2 to smithi06431743-9 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:1176: finishing write tid 3 to smithi06431743-9 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:1176: finishing write tid 4 to smithi06431743-9 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:1176: finishing write tid 5 to smithi06431743-9 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:1176: oid 9 updating version 0 to 415 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:1176: oid 9 updating version 415 to 418 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:1176: oid 9 version 418 is already newer than 416 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:1176: oid 9 version 418 is already newer than 417 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 418 (ObjNum 569 snap 134 seq_num 569) dirty exists 2024-03-04T06:12:34.277 INFO:tasks.rados.rados.0.smithi064.stdout:1176: left oid 9 (ObjNum 569 snap 134 seq_num 569) 2024-03-04T06:12:34.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:34.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:34.424+0000 7f1989c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:34.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:34.429+0000 7f1989c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:35.283 INFO:tasks.rados.rados.0.smithi064.stdout:1176: done (2 left) 2024-03-04T06:12:35.283 INFO:tasks.rados.rados.0.smithi064.stdout:1178: done (1 left) 2024-03-04T06:12:35.284 INFO:tasks.rados.rados.0.smithi064.stdout:1180: done (0 left) 2024-03-04T06:12:35.284 INFO:tasks.rados.rados.0.smithi064.stdout:1181: snap_create 2024-03-04T06:12:36.285 INFO:tasks.rados.rados.0.smithi064.stdout:1181: done (0 left) 2024-03-04T06:12:36.285 INFO:tasks.rados.rados.0.smithi064.stdout:1182: write oid 23 current snap is 136 2024-03-04T06:12:36.285 INFO:tasks.rados.rados.0.smithi064.stdout:1182: seq_num 571 ranges {796797=493654,1830277=796851,3397470=199327} 2024-03-04T06:12:36.291 INFO:tasks.rados.rados.0.smithi064.stdout:1182: writing smithi06431743-23 from 796797 to 1290451 tid 1 2024-03-04T06:12:36.300 INFO:tasks.rados.rados.0.smithi064.stdout:1182: writing smithi06431743-23 from 1830277 to 2627128 tid 2 2024-03-04T06:12:36.305 INFO:tasks.rados.rados.0.smithi064.stdout:1182: writing smithi06431743-23 from 3397470 to 3596797 tid 3 2024-03-04T06:12:36.306 INFO:tasks.rados.rados.0.smithi064.stdout:1183: snap_create 2024-03-04T06:12:36.402 INFO:tasks.rados.rados.0.smithi064.stdout:1182: finishing write tid 1 to smithi06431743-23 2024-03-04T06:12:36.414 INFO:tasks.rados.rados.0.smithi064.stdout:1182: finishing write tid 2 to smithi06431743-23 2024-03-04T06:12:36.423 INFO:tasks.rados.rados.0.smithi064.stdout:1182: finishing write tid 3 to smithi06431743-23 2024-03-04T06:12:36.426 INFO:tasks.rados.rados.0.smithi064.stdout:1182: finishing write tid 4 to smithi06431743-23 2024-03-04T06:12:36.427 INFO:tasks.rados.rados.0.smithi064.stdout:1182: finishing write tid 5 to smithi06431743-23 2024-03-04T06:12:36.427 INFO:tasks.rados.rados.0.smithi064.stdout:1182: oid 23 updating version 0 to 233 2024-03-04T06:12:36.427 INFO:tasks.rados.rados.0.smithi064.stdout:1182: oid 23 version 233 is already newer than 230 2024-03-04T06:12:36.427 INFO:tasks.rados.rados.0.smithi064.stdout:1182: oid 23 version 233 is already newer than 233 2024-03-04T06:12:36.427 INFO:tasks.rados.rados.0.smithi064.stdout:1182: oid 23 version 233 is already newer than 231 2024-03-04T06:12:36.427 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 233 (ObjNum 571 snap 136 seq_num 571) dirty exists 2024-03-04T06:12:36.427 INFO:tasks.rados.rados.0.smithi064.stdout:1182: left oid 23 (ObjNum 571 snap 136 seq_num 571) 2024-03-04T06:12:36.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:36.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:36.602+0000 7f55b646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:36.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:36.606+0000 7f55b646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:37.294 INFO:tasks.rados.rados.0.smithi064.stdout:1182: done (1 left) 2024-03-04T06:12:37.294 INFO:tasks.rados.rados.0.smithi064.stdout:1183: done (0 left) 2024-03-04T06:12:37.294 INFO:tasks.rados.rados.0.smithi064.stdout:1184: snap_remove snap 135 2024-03-04T06:12:38.214 INFO:tasks.rados.rados.0.smithi064.stdout:1184: done (0 left) 2024-03-04T06:12:38.214 INFO:tasks.rados.rados.0.smithi064.stdout:1185: read oid 15 snap 134 2024-03-04T06:12:38.214 INFO:tasks.rados.rados.0.smithi064.stdout:1185: expect (ObjNum 566 snap 133 seq_num 566) 2024-03-04T06:12:38.214 INFO:tasks.rados.rados.0.smithi064.stdout:1186: read oid 4 snap 134 2024-03-04T06:12:38.214 INFO:tasks.rados.rados.0.smithi064.stdout:1186: expect (ObjNum 413 snap 99 seq_num 413) 2024-03-04T06:12:38.214 INFO:tasks.rados.rados.0.smithi064.stdout:1187: write oid 39 current snap is 137 2024-03-04T06:12:38.214 INFO:tasks.rados.rados.0.smithi064.stdout:1187: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2024-03-04T06:12:38.220 INFO:tasks.rados.rados.0.smithi064.stdout:1187: writing smithi06431743-39 from 413604 to 982507 tid 1 2024-03-04T06:12:38.228 INFO:tasks.rados.rados.0.smithi064.stdout:1187: writing smithi06431743-39 from 1488759 to 2145621 tid 2 2024-03-04T06:12:38.236 INFO:tasks.rados.rados.0.smithi064.stdout:1187: writing smithi06431743-39 from 2624893 to 3216602 tid 3 2024-03-04T06:12:38.236 INFO:tasks.rados.rados.0.smithi064.stdout:1188: snap_create 2024-03-04T06:12:38.282 INFO:tasks.rados.rados.0.smithi064.stdout:1187: finishing write tid 1 to smithi06431743-39 2024-03-04T06:12:38.283 INFO:tasks.rados.rados.0.smithi064.stdout:1186: expect (ObjNum 413 snap 99 seq_num 413) 2024-03-04T06:12:38.402 INFO:tasks.rados.rados.0.smithi064.stdout:1187: finishing write tid 2 to smithi06431743-39 2024-03-04T06:12:38.402 INFO:tasks.rados.rados.0.smithi064.stdout:1187: finishing write tid 3 to smithi06431743-39 2024-03-04T06:12:38.402 INFO:tasks.rados.rados.0.smithi064.stdout:1185: expect (ObjNum 566 snap 133 seq_num 566) 2024-03-04T06:12:38.435 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:12:38.436 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:12:38.436 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:12:38.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:12:38.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:38.543+0000 7efd1679d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:38.551 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:38.550+0000 7efd1679d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:38.564 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:38.567 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:38.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:12:38.572 INFO:tasks.rados.rados.0.smithi064.stdout:1187: finishing write tid 4 to smithi06431743-39 2024-03-04T06:12:38.572 INFO:tasks.rados.rados.0.smithi064.stdout:1187: finishing write tid 5 to smithi06431743-39 2024-03-04T06:12:38.572 INFO:tasks.rados.rados.0.smithi064.stdout:1187: oid 39 updating version 0 to 265 2024-03-04T06:12:38.572 INFO:tasks.rados.rados.0.smithi064.stdout:1187: oid 39 version 265 is already newer than 264 2024-03-04T06:12:38.572 INFO:tasks.rados.rados.0.smithi064.stdout:1187: oid 39 version 265 is already newer than 263 2024-03-04T06:12:38.572 INFO:tasks.rados.rados.0.smithi064.stdout:1187: oid 39 version 265 is already newer than 263 2024-03-04T06:12:38.572 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 265 (ObjNum 572 snap 137 seq_num 572) dirty exists 2024-03-04T06:12:38.572 INFO:tasks.rados.rados.0.smithi064.stdout:1187: left oid 39 (ObjNum 572 snap 137 seq_num 572) 2024-03-04T06:12:38.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:38.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:38.670+0000 7f4b9a11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:38.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:38.677+0000 7f4b9a11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:38.691 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:38.691 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:38.691 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:38.691 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:38.691 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:38.691 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:38.692 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:38.692 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:38.692 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:38.692 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:38.692 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:38.692 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:38.693 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:38.694 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:38.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:12:38.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:38.778+0000 7f3ca4a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:38.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:38.783+0000 7f3ca4a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:38.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:38.841+0000 7f2ff5006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:38.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:38.849+0000 7f2ff5006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:38.863 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:38.863 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:38.863 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:38.864 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:38.864 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:38.864 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:38.864 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:38.864 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:38.864 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:38.864 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:38.864 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:38.864 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:38.866 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:38.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:12:39.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:39.011+0000 7fb3bb624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:39.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:39.018+0000 7fb3bb624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:39.032 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:39.032 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:39.032 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:39.032 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:39.032 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:39.032 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:39.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:39.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:39.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:39.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:39.033 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:39.033 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:39.035 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:39.216 INFO:tasks.rados.rados.0.smithi064.stdout:1185: done (3 left) 2024-03-04T06:12:39.216 INFO:tasks.rados.rados.0.smithi064.stdout:1186: done (2 left) 2024-03-04T06:12:39.216 INFO:tasks.rados.rados.0.smithi064.stdout:1187: done (1 left) 2024-03-04T06:12:39.216 INFO:tasks.rados.rados.0.smithi064.stdout:1188: done (0 left) 2024-03-04T06:12:39.216 INFO:tasks.rados.rados.0.smithi064.stdout:1189: snap_remove snap 136 2024-03-04T06:12:40.215 INFO:tasks.rados.rados.0.smithi064.stdout:1189: done (0 left) 2024-03-04T06:12:40.215 INFO:tasks.rados.rados.0.smithi064.stdout:1190: write (excl) oid 32 current snap is 138 2024-03-04T06:12:40.215 INFO:tasks.rados.rados.0.smithi064.stdout:1190: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2024-03-04T06:12:40.220 INFO:tasks.rados.rados.0.smithi064.stdout:1190: writing smithi06431743-32 from 430411 to 1074563 tid 1 2024-03-04T06:12:40.226 INFO:tasks.rados.rados.0.smithi064.stdout:1190: writing smithi06431743-32 from 1830888 to 2231408 tid 2 2024-03-04T06:12:40.232 INFO:tasks.rados.rados.0.smithi064.stdout:1190: writing smithi06431743-32 from 2935963 to 3338944 tid 3 2024-03-04T06:12:40.232 INFO:tasks.rados.rados.0.smithi064.stdout:1191: snap_remove snap 134 2024-03-04T06:12:40.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:40.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:40.973+0000 7fc80135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:40.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:40.977+0000 7fc80135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:41.216 INFO:tasks.rados.rados.0.smithi064.stdout:1191: done (1 left) 2024-03-04T06:12:41.216 INFO:tasks.rados.rados.0.smithi064.stdout:1192: snap_remove snap 129 2024-03-04T06:12:42.218 INFO:tasks.rados.rados.0.smithi064.stdout:1192: done (1 left) 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:1193: delete oid 37 current snap is 138 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:1190: finishing write tid 1 to smithi06431743-32 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:1190: finishing write tid 2 to smithi06431743-32 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:1190: finishing write tid 3 to smithi06431743-32 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:1190: finishing write tid 4 to smithi06431743-32 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:1190: finishing write tid 5 to smithi06431743-32 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:1190: oid 32 updating version 0 to 463 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:1190: oid 32 version 463 is already newer than 463 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:1190: oid 32 version 463 is already newer than 461 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:1190: oid 32 version 463 is already newer than 460 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 463 (ObjNum 573 snap 138 seq_num 573) dirty exists 2024-03-04T06:12:42.219 INFO:tasks.rados.rados.0.smithi064.stdout:1190: left oid 32 (ObjNum 573 snap 138 seq_num 573) 2024-03-04T06:12:42.266 INFO:tasks.rados.rados.0.smithi064.stdout:1190: done (1 left) 2024-03-04T06:12:42.266 INFO:tasks.rados.rados.0.smithi064.stdout:1193: done (0 left) 2024-03-04T06:12:42.266 INFO:tasks.rados.rados.0.smithi064.stdout:1194: write oid 25 current snap is 138 2024-03-04T06:12:42.266 INFO:tasks.rados.rados.0.smithi064.stdout:1194: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2024-03-04T06:12:42.272 INFO:tasks.rados.rados.0.smithi064.stdout:1194: writing smithi06431743-25 from 464025 to 1258675 tid 1 2024-03-04T06:12:42.278 INFO:tasks.rados.rados.0.smithi064.stdout:1194: writing smithi06431743-25 from 1947852 to 2352041 tid 2 2024-03-04T06:12:42.284 INFO:tasks.rados.rados.0.smithi064.stdout:1194: writing smithi06431743-25 from 2990809 to 3416334 tid 3 2024-03-04T06:12:42.285 INFO:tasks.rados.rados.0.smithi064.stdout:1195: read oid 48 snap -1 2024-03-04T06:12:42.285 INFO:tasks.rados.rados.0.smithi064.stdout:1195: expect (ObjNum 545 snap 129 seq_num 545) 2024-03-04T06:12:42.285 INFO:tasks.rados.rados.0.smithi064.stdout:1196: delete oid 7 current snap is 138 2024-03-04T06:12:42.367 INFO:tasks.rados.rados.0.smithi064.stdout:1194: finishing write tid 1 to smithi06431743-25 2024-03-04T06:12:42.381 INFO:tasks.rados.rados.0.smithi064.stdout:1195: expect (ObjNum 545 snap 129 seq_num 545) 2024-03-04T06:12:42.653 INFO:tasks.rados.rados.0.smithi064.stdout:1194: finishing write tid 2 to smithi06431743-25 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1194: finishing write tid 3 to smithi06431743-25 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1194: finishing write tid 4 to smithi06431743-25 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1194: finishing write tid 5 to smithi06431743-25 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1194: oid 25 updating version 0 to 269 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1194: oid 25 updating version 269 to 270 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1194: oid 25 updating version 270 to 272 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1194: oid 25 version 272 is already newer than 272 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 272 (ObjNum 575 snap 138 seq_num 575) dirty exists 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1194: left oid 25 (ObjNum 575 snap 138 seq_num 575) 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1194: done (2 left) 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1195: done (1 left) 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1196: done (0 left) 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1197: write (excl) oid 6 current snap is 138 2024-03-04T06:12:42.654 INFO:tasks.rados.rados.0.smithi064.stdout:1197: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-03-04T06:12:42.658 INFO:tasks.rados.rados.0.smithi064.stdout:1197: writing smithi06431743-6 from 497639 to 1042787 tid 1 2024-03-04T06:12:42.665 INFO:tasks.rados.rados.0.smithi064.stdout:1197: writing smithi06431743-6 from 1548463 to 1956321 tid 2 2024-03-04T06:12:42.671 INFO:tasks.rados.rados.0.smithi064.stdout:1197: writing smithi06431743-6 from 2529302 to 3093724 tid 3 2024-03-04T06:12:42.674 INFO:tasks.rados.rados.0.smithi064.stdout:1197: writing smithi06431743-6 from 3528143 to 3697639 tid 4 2024-03-04T06:12:42.674 INFO:tasks.rados.rados.0.smithi064.stdout:1198: snap_create 2024-03-04T06:12:42.687 INFO:tasks.rados.rados.0.smithi064.stdout:1197: finishing write tid 1 to smithi06431743-6 2024-03-04T06:12:42.700 INFO:tasks.rados.rados.0.smithi064.stdout:1197: finishing write tid 2 to smithi06431743-6 2024-03-04T06:12:42.718 INFO:tasks.rados.rados.0.smithi064.stdout:1197: finishing write tid 3 to smithi06431743-6 2024-03-04T06:12:42.724 INFO:tasks.rados.rados.0.smithi064.stdout:1197: finishing write tid 4 to smithi06431743-6 2024-03-04T06:12:42.726 INFO:tasks.rados.rados.0.smithi064.stdout:1197: finishing write tid 5 to smithi06431743-6 2024-03-04T06:12:42.727 INFO:tasks.rados.rados.0.smithi064.stdout:1197: finishing write tid 6 to smithi06431743-6 2024-03-04T06:12:42.727 INFO:tasks.rados.rados.0.smithi064.stdout:1197: oid 6 updating version 0 to 184 2024-03-04T06:12:42.727 INFO:tasks.rados.rados.0.smithi064.stdout:1197: oid 6 version 184 is already newer than 183 2024-03-04T06:12:42.727 INFO:tasks.rados.rados.0.smithi064.stdout:1197: oid 6 updating version 184 to 187 2024-03-04T06:12:42.728 INFO:tasks.rados.rados.0.smithi064.stdout:1197: oid 6 version 187 is already newer than 185 2024-03-04T06:12:42.728 INFO:tasks.rados.rados.0.smithi064.stdout:1197: oid 6 version 187 is already newer than 187 2024-03-04T06:12:42.728 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 187 (ObjNum 577 snap 138 seq_num 577) dirty exists 2024-03-04T06:12:42.728 INFO:tasks.rados.rados.0.smithi064.stdout:1197: left oid 6 (ObjNum 577 snap 138 seq_num 577) 2024-03-04T06:12:43.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:43.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:43.155+0000 7fd665264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:43.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:43.159+0000 7fd665264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:43.224 INFO:tasks.rados.rados.0.smithi064.stdout:1197: done (1 left) 2024-03-04T06:12:43.225 INFO:tasks.rados.rados.0.smithi064.stdout:1198: done (0 left) 2024-03-04T06:12:43.225 INFO:tasks.rados.rados.0.smithi064.stdout:1199: read oid 31 snap 133 2024-03-04T06:12:43.225 INFO:tasks.rados.rados.0.smithi064.stdout:1199: expect deleted 2024-03-04T06:12:43.225 INFO:tasks.rados.rados.0.smithi064.stdout:1200: write (excl) oid 45 current snap is 139 2024-03-04T06:12:43.225 INFO:tasks.rados.rados.0.smithi064.stdout:1200: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2024-03-04T06:12:43.231 INFO:tasks.rados.rados.0.smithi064.stdout:1200: writing smithi06431743-45 from 514446 to 1251196 tid 1 2024-03-04T06:12:43.239 INFO:tasks.rados.rados.0.smithi064.stdout:1200: writing smithi06431743-45 from 1723298 to 2391167 tid 2 2024-03-04T06:12:43.246 INFO:tasks.rados.rados.0.smithi064.stdout:1200: writing smithi06431743-45 from 3073078 to 3714446 tid 3 2024-03-04T06:12:43.247 INFO:tasks.rados.rados.0.smithi064.stdout:1201: write (excl) oid 17 current snap is 139 2024-03-04T06:12:43.247 INFO:tasks.rados.rados.0.smithi064.stdout:1201: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2024-03-04T06:12:43.253 INFO:tasks.rados.rados.0.smithi064.stdout:1201: writing smithi06431743-17 from 531253 to 943252 tid 1 2024-03-04T06:12:43.258 INFO:tasks.rados.rados.0.smithi064.stdout:1201: writing smithi06431743-17 from 1381780 to 1793307 tid 2 2024-03-04T06:12:43.264 INFO:tasks.rados.rados.0.smithi064.stdout:1201: writing smithi06431743-17 from 2300501 to 2887467 tid 3 2024-03-04T06:12:43.269 INFO:tasks.rados.rados.0.smithi064.stdout:1201: writing smithi06431743-17 from 3376974 to 3731253 tid 4 2024-03-04T06:12:43.269 INFO:tasks.rados.rados.0.smithi064.stdout:1202: write oid 2 current snap is 139 2024-03-04T06:12:43.269 INFO:tasks.rados.rados.0.smithi064.stdout:1202: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2024-03-04T06:12:43.276 INFO:tasks.rados.rados.0.smithi064.stdout:1202: writing smithi06431743-2 from 548060 to 1035308 tid 1 2024-03-04T06:12:43.283 INFO:tasks.rados.rados.0.smithi064.stdout:1202: writing smithi06431743-2 from 1440262 to 2111800 tid 2 2024-03-04T06:12:43.293 INFO:tasks.rados.rados.0.smithi064.stdout:1202: writing smithi06431743-2 from 2844277 to 3642515 tid 3 2024-03-04T06:12:43.294 INFO:tasks.rados.rados.0.smithi064.stdout:1199: done (3 left) 2024-03-04T06:12:43.294 INFO:tasks.rados.rados.0.smithi064.stdout:1203: write (excl) oid 14 current snap is 139 2024-03-04T06:12:43.294 INFO:tasks.rados.rados.0.smithi064.stdout:1203: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2024-03-04T06:12:43.299 INFO:tasks.rados.rados.0.smithi064.stdout:1203: writing smithi06431743-14 from 564867 to 1127364 tid 1 2024-03-04T06:12:43.306 INFO:tasks.rados.rados.0.smithi064.stdout:1203: writing smithi06431743-14 from 1898744 to 2313940 tid 2 2024-03-04T06:12:43.314 INFO:tasks.rados.rados.0.smithi064.stdout:1203: writing smithi06431743-14 from 2755347 to 3481210 tid 3 2024-03-04T06:12:43.314 INFO:tasks.rados.rados.0.smithi064.stdout:1201: finishing write tid 1 to smithi06431743-17 2024-03-04T06:12:43.314 INFO:tasks.rados.rados.0.smithi064.stdout:1201: finishing write tid 2 to smithi06431743-17 2024-03-04T06:12:43.314 INFO:tasks.rados.rados.0.smithi064.stdout:1204: snap_remove snap 137 2024-03-04T06:12:44.036 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:12:44.036 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:12:44.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:12:44.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:44.148+0000 7fc390754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:44.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:44.154+0000 7fc390754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:44.232 INFO:tasks.rados.rados.0.smithi064.stdout:1204: done (4 left) 2024-03-04T06:12:44.232 INFO:tasks.rados.rados.0.smithi064.stdout:1205: read oid 25 snap -1 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1205: expect (ObjNum 575 snap 138 seq_num 575) 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1206: read oid 10 snap -1 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1206: expect deleted 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1207: delete oid 30 current snap is 139 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1200: finishing write tid 1 to smithi06431743-45 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1203: finishing write tid 1 to smithi06431743-14 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1202: finishing write tid 1 to smithi06431743-2 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1201: finishing write tid 3 to smithi06431743-17 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1200: finishing write tid 2 to smithi06431743-45 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1203: finishing write tid 2 to smithi06431743-14 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1200: finishing write tid 3 to smithi06431743-45 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1202: finishing write tid 2 to smithi06431743-2 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1201: finishing write tid 4 to smithi06431743-17 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1200: finishing write tid 4 to smithi06431743-45 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1200: finishing write tid 5 to smithi06431743-45 2024-03-04T06:12:44.233 INFO:tasks.rados.rados.0.smithi064.stdout:1200: oid 45 updating version 0 to 274 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1200: oid 45 updating version 274 to 276 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1200: oid 45 version 276 is already newer than 276 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1200: oid 45 updating version 276 to 277 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 277 (ObjNum 578 snap 139 seq_num 578) dirty exists 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1200: left oid 45 (ObjNum 578 snap 139 seq_num 578) 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1203: finishing write tid 3 to smithi06431743-14 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1201: finishing write tid 5 to smithi06431743-17 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1201: finishing write tid 6 to smithi06431743-17 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1201: oid 17 updating version 0 to 158 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1201: oid 17 updating version 158 to 160 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1201: oid 17 updating version 160 to 161 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1201: oid 17 version 161 is already newer than 159 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1201: oid 17 version 161 is already newer than 157 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 161 (ObjNum 579 snap 139 seq_num 579) dirty exists 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1201: left oid 17 (ObjNum 579 snap 139 seq_num 579) 2024-03-04T06:12:44.234 INFO:tasks.rados.rados.0.smithi064.stdout:1202: finishing write tid 3 to smithi06431743-2 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1203: finishing write tid 4 to smithi06431743-14 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1203: finishing write tid 5 to smithi06431743-14 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1203: oid 14 updating version 0 to 330 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1203: oid 14 updating version 330 to 332 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1203: oid 14 version 332 is already newer than 332 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1203: oid 14 version 332 is already newer than 329 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 332 (ObjNum 581 snap 139 seq_num 581) dirty exists 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1203: left oid 14 (ObjNum 581 snap 139 seq_num 581) 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1202: finishing write tid 4 to smithi06431743-2 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1202: finishing write tid 5 to smithi06431743-2 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1202: oid 2 updating version 0 to 196 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1202: oid 2 version 196 is already newer than 195 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1202: oid 2 version 196 is already newer than 194 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:1202: oid 2 updating version 196 to 197 2024-03-04T06:12:44.235 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 197 (ObjNum 580 snap 139 seq_num 580) dirty exists 2024-03-04T06:12:44.236 INFO:tasks.rados.rados.0.smithi064.stdout:1202: left oid 2 (ObjNum 580 snap 139 seq_num 580) 2024-03-04T06:12:44.283 INFO:tasks.rados.rados.0.smithi064.stdout:1205: expect (ObjNum 575 snap 138 seq_num 575) 2024-03-04T06:12:44.365 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:12:44.366 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:12:44.379 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-03-04T06:12:44.230598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121083315,"num_objects":397,"num_object_clones":288,"num_object_copies":1191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2287,"num_read_kb":1629205,"num_write":2045,"num_write_kb":523828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3873,"ondisk_log_size":3873,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":387852,"kb_used_data":247276,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":377099508,"statfs":{"total":386547056640,"available":386149896192,"internally_reserved":0,"allocated":253210624,"data_stored":265746649,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4403566,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":13986,"num_write":15,"num_write_kb":4649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017222"},"pg_stats":[{"pgid":"3.f","version":"285'195","reported_seq":957,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-04T06:12:38.366600+0000","last_change":"2024-03-04T06:12:38.378224+0000","last_active":"2024-03-04T06:12:38.366600+0000","last_peered":"2024-03-04T06:12:38.366600+0000","last_clean":"2024-03-04T06:12:38.366600+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:12:38.366600+0000","last_undegraded":"2024-03-04T06:12:38.366600+0000","last_fullsized":"2024-03-04T06:12:38.366600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9664289,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":77,"num_read_kb":39366,"num_write":113,"num_write_kb":30557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"280'157","reported_seq":956,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.235651+0000","last_change":"2024-03-04T06:12:40.245657+0000","last_active":"2024-03-04T06:12:40.235651+0000","last_peered":"2024-03-04T06:12:40.235651+0000","last_clean":"2024-03-04T06:12:40.235651+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:12:40.235651+0000","last_undegraded":"2024-03-04T06:12:40.235651+0000","last_fullsized":"2024-03-04T06:12:40.235651+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269176,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":69,"num_read_kb":46326,"num_write":72,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"288'265","reported_seq":1019,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-04T06:12:41.238323+0000","last_change":"2024-03-04T06:12:41.240973+0000","last_active":"2024-03-04T06:12:41.238323+0000","last_peered":"2024-03-04T06:12:41.238323+0000","last_clean":"2024-03-04T06:12:41.238323+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:12:41.238323+0000","last_undegraded":"2024-03-04T06:12:41.238323+0000","last_fullsized":"2024-03-04T06:12:41.238323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11731682,"num_objects":31,"num_object_clones":23,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":147,"num_read_kb":95340,"num_write":146,"num_write_kb":39855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"285'190","reported_seq":980,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-04T06:12:38.360950+0000","last_change":"2024-03-04T06:12:38.364934+0000","last_active":"2024-03-04T06:12:38.360950+0000","last_peered":"2024-03-04T06:12:38.360950+0000","last_clean":"2024-03-04T06:12:38.360950+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:12:38.360950+0000","last_undegraded":"2024-03-04T06:12:38.360950+0000","last_fullsized":"2024-03-04T06:12:38.360950+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9227340,"num_objects":21,"num_object_clones":13,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":118,"num_read_kb":91992,"num_write":114,"num_write_kb":32234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"249'92","reported_seq":877,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.264484+0000","last_change":"2024-03-04T06:12:40.267416+0000","last_active":"2024-03-04T06:12:40.264484+0000","last_peered":"2024-03-04T06:12:40.264484+0000","last_clean":"2024-03-04T06:12:40.264484+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:12:40.264484+0000","last_undegraded":"2024-03-04T06:12:40.264484+0000","last_fullsized":"2024-03-04T06:12:40.264484+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":17041,"num_write":34,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"285'198","reported_seq":974,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-04T06:12:38.370289+0000","last_change":"2024-03-04T06:12:38.378611+0000","last_active":"2024-03-04T06:12:38.370289+0000","last_peered":"2024-03-04T06:12:38.370289+0000","last_clean":"2024-03-04T06:12:38.370289+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:12:38.370289+0000","last_undegraded":"2024-03-04T06:12:38.370289+0000","last_fullsized":"2024-03-04T06:12:38.370289+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6134753,"num_objects":21,"num_object_clones":15,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":118,"num_read_kb":87289,"num_write":111,"num_write_kb":33076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":824,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-04T06:12:38.369511+0000","last_change":"2024-03-04T06:12:38.378724+0000","last_active":"2024-03-04T06:12:38.369511+0000","last_peered":"2024-03-04T06:12:38.369511+0000","last_clean":"2024-03-04T06:12:38.369511+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:12:38.369511+0000","last_undegraded":"2024-03-04T06:12:38.369511+0000","last_fullsized":"2024-03-04T06:12:38.369511+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"280'156","reported_seq":945,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-04T06:12:41.238477+0000","last_change":"2024-03-04T06:12:41.241381+0000","last_active":"2024-03-04T06:12:41.238477+0000","last_peered":"2024-03-04T06:12:41.238477+0000","last_clean":"2024-03-04T06:12:41.238477+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:12:41.238477+0000","last_undegraded":"2024-03-04T06:12:41.238477+0000","last_fullsized":"2024-03-04T06:12:41.238477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252237,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":116,"num_read_kb":111954,"num_write":72,"num_write_kb":19512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":841,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.245713+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:12:40.245713+0000","last_peered":"2024-03-04T06:12:40.245713+0000","last_clean":"2024-03-04T06:12:40.245713+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:12:40.245713+0000","last_undegraded":"2024-03-04T06:12:40.245713+0000","last_fullsized":"2024-03-04T06:12:40.245713+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-04T06:12:41.243404+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:12:41.243404+0000","last_peered":"2024-03-04T06:12:41.243404+0000","last_clean":"2024-03-04T06:12:41.243404+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:12:41.243404+0000","last_undegraded":"2024-03-04T06:12:41.243404+0000","last_fullsized":"2024-03-04T06:12:41.243404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"288'235","reported_seq":996,"reported_epoch":291,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:41.239506+0000","last_change":"2024-03-04T06:12:41.239506+0000","last_active":"2024-03-04T06:12:41.239506+0000","last_peered":"2024-03-04T06:12:41.239506+0000","last_clean":"2024-03-04T06:12:41.239506+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:12:41.239506+0000","last_undegraded":"2024-03-04T06:12:41.239506+0000","last_fullsized":"2024-03-04T06:12:41.239506+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10101304,"num_objects":25,"num_object_clones":18,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":142,"num_read_kb":109564,"num_write":119,"num_write_kb":30600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":700,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-04T06:12:38.365579+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:12:38.365579+0000","last_peered":"2024-03-04T06:12:38.365579+0000","last_clean":"2024-03-04T06:12:38.365579+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:12:38.365579+0000","last_undegraded":"2024-03-04T06:12:38.365579+0000","last_fullsized":"2024-03-04T06:12:38.365579+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"286'216","reported_seq":954,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.250645+0000","last_change":"2024-03-04T06:12:40.257391+0000","last_active":"2024-03-04T06:12:40.250645+0000","last_peered":"2024-03-04T06:12:40.250645+0000","last_clean":"2024-03-04T06:12:40.250645+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:12:40.250645+0000","last_undegraded":"2024-03-04T06:12:40.250645+0000","last_fullsized":"2024-03-04T06:12:40.250645+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3159815,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":100,"num_read_kb":76866,"num_write":87,"num_write_kb":25445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-04T06:12:41.228812+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:12:41.228812+0000","last_peered":"2024-03-04T06:12:41.228812+0000","last_clean":"2024-03-04T06:12:41.228812+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:12:41.228812+0000","last_undegraded":"2024-03-04T06:12:41.228812+0000","last_fullsized":"2024-03-04T06:12:41.228812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"290'463","reported_seq":1198,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.307943+0000","last_change":"2024-03-04T06:12:40.267034+0000","last_active":"2024-03-04T06:12:40.307943+0000","last_peered":"2024-03-04T06:12:40.307943+0000","last_clean":"2024-03-04T06:12:40.307943+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:12:40.307943+0000","last_undegraded":"2024-03-04T06:12:40.307943+0000","last_fullsized":"2024-03-04T06:12:40.307943+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13429189,"num_objects":47,"num_object_clones":35,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":313,"num_read_kb":258283,"num_write":249,"num_write_kb":69132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":700,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-04T06:12:38.357281+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:12:38.357281+0000","last_peered":"2024-03-04T06:12:38.357281+0000","last_clean":"2024-03-04T06:12:38.357281+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:12:38.357281+0000","last_undegraded":"2024-03-04T06:12:38.357281+0000","last_fullsized":"2024-03-04T06:12:38.357281+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"284'475","reported_seq":1154,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.238549+0000","last_change":"2024-03-04T06:12:40.245136+0000","last_active":"2024-03-04T06:12:40.238549+0000","last_peered":"2024-03-04T06:12:40.238549+0000","last_clean":"2024-03-04T06:12:40.238549+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:12:40.238549+0000","last_undegraded":"2024-03-04T06:12:40.238549+0000","last_fullsized":"2024-03-04T06:12:40.238549+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13109790,"num_objects":47,"num_object_clones":34,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":207,"num_read_kb":144965,"num_write":227,"num_write_kb":60703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"282'181","reported_seq":1001,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-04T06:12:41.246030+0000","last_change":"2024-03-04T06:12:41.247461+0000","last_active":"2024-03-04T06:12:41.246030+0000","last_peered":"2024-03-04T06:12:41.246030+0000","last_clean":"2024-03-04T06:12:41.246030+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:12:41.246030+0000","last_undegraded":"2024-03-04T06:12:41.246030+0000","last_fullsized":"2024-03-04T06:12:41.246030+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6807000,"num_objects":19,"num_object_clones":14,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":190,"num_read_kb":157561,"num_write":96,"num_write_kb":25475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":738,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.244635+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:12:40.244635+0000","last_peered":"2024-03-04T06:12:40.244635+0000","last_clean":"2024-03-04T06:12:40.244635+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:12:40.244635+0000","last_undegraded":"2024-03-04T06:12:40.244635+0000","last_fullsized":"2024-03-04T06:12:40.244635+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"280'208","reported_seq":1010,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.251681+0000","last_change":"2024-03-04T06:12:40.258252+0000","last_active":"2024-03-04T06:12:40.251681+0000","last_peered":"2024-03-04T06:12:40.251681+0000","last_clean":"2024-03-04T06:12:40.251681+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:12:40.251681+0000","last_undegraded":"2024-03-04T06:12:40.251681+0000","last_fullsized":"2024-03-04T06:12:40.251681+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8840779,"num_objects":26,"num_object_clones":16,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":115,"num_read_kb":67271,"num_write":110,"num_write_kb":30018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":708,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.251441+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:12:40.251441+0000","last_peered":"2024-03-04T06:12:40.251441+0000","last_clean":"2024-03-04T06:12:40.251441+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:12:40.251441+0000","last_undegraded":"2024-03-04T06:12:40.251441+0000","last_fullsized":"2024-03-04T06:12:40.251441+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"286'323","reported_seq":1056,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.246692+0000","last_change":"2024-03-04T06:12:40.257636+0000","last_active":"2024-03-04T06:12:40.246692+0000","last_peered":"2024-03-04T06:12:40.246692+0000","last_clean":"2024-03-04T06:12:40.246692+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:12:40.246692+0000","last_undegraded":"2024-03-04T06:12:40.246692+0000","last_fullsized":"2024-03-04T06:12:40.246692+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9210533,"num_objects":32,"num_object_clones":24,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":171,"num_read_kb":115119,"num_write":166,"num_write_kb":45507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":738,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.232510+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:12:40.232510+0000","last_peered":"2024-03-04T06:12:40.232510+0000","last_clean":"2024-03-04T06:12:40.232510+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:12:40.232510+0000","last_undegraded":"2024-03-04T06:12:40.232510+0000","last_fullsized":"2024-03-04T06:12:40.232510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"284'422","reported_seq":1157,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-04T06:12:40.230762+0000","last_change":"2024-03-04T06:12:40.244797+0000","last_active":"2024-03-04T06:12:40.230762+0000","last_peered":"2024-03-04T06:12:40.230762+0000","last_clean":"2024-03-04T06:12:40.230762+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:12:40.230762+0000","last_undegraded":"2024-03-04T06:12:40.230762+0000","last_fullsized":"2024-03-04T06:12:40.230762+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10555060,"num_objects":37,"num_object_clones":33,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":270,"num_read_kb":210177,"num_write":192,"num_write_kb":52304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":701,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-04T06:12:41.240318+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:12:41.240318+0000","last_peered":"2024-03-04T06:12:41.240318+0000","last_clean":"2024-03-04T06:12:41.240318+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:12:41.240318+0000","last_undegraded":"2024-03-04T06:12:41.240318+0000","last_fullsized":"2024-03-04T06:12:41.240318+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":120492947,"num_objects":395,"num_object_clones":288,"num_object_copies":1185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":69,"num_read":2184,"num_read_kb":1629114,"num_write":1908,"num_write_kb":521934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3776,"ondisk_log_size":3776,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114216,"kb_used_data":79072,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94257624,"statfs":{"total":96636764160,"available":96519806976,"internally_reserved":0,"allocated":80969728,"data_stored":86108774,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99112,"kb_used_data":63968,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94272728,"statfs":{"total":96636764160,"available":96535273472,"internally_reserved":0,"allocated":65503232,"data_stored":69654186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83620,"kb_used_data":48476,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288220,"statfs":{"total":96636764160,"available":96551137280,"internally_reserved":0,"allocated":49639424,"data_stored":50135807,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90904,"kb_used_data":55760,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94280936,"statfs":{"total":96636764160,"available":96543678464,"internally_reserved":0,"allocated":57098240,"data_stored":59847882,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:12:44.539 INFO:tasks.rados.rados.0.smithi064.stdout:1200: done (6 left) 2024-03-04T06:12:44.539 INFO:tasks.rados.rados.0.smithi064.stdout:1201: done (5 left) 2024-03-04T06:12:44.539 INFO:tasks.rados.rados.0.smithi064.stdout:1202: done (4 left) 2024-03-04T06:12:44.539 INFO:tasks.rados.rados.0.smithi064.stdout:1203: done (3 left) 2024-03-04T06:12:44.539 INFO:tasks.rados.rados.0.smithi064.stdout:1205: done (2 left) 2024-03-04T06:12:44.540 INFO:tasks.rados.rados.0.smithi064.stdout:1206: done (1 left) 2024-03-04T06:12:44.540 INFO:tasks.rados.rados.0.smithi064.stdout:1207: done (0 left) 2024-03-04T06:12:44.540 INFO:tasks.rados.rados.0.smithi064.stdout:1208: read oid 28 snap -1 2024-03-04T06:12:44.540 INFO:tasks.rados.rados.0.smithi064.stdout:1208: expect (ObjNum 559 snap 132 seq_num 559) 2024-03-04T06:12:44.540 INFO:tasks.rados.rados.0.smithi064.stdout:1209: snap_create 2024-03-04T06:12:44.559 INFO:tasks.rados.rados.0.smithi064.stdout:1208: expect (ObjNum 559 snap 132 seq_num 559) 2024-03-04T06:12:45.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:45.238 INFO:tasks.rados.rados.0.smithi064.stdout:1208: done (1 left) 2024-03-04T06:12:45.238 INFO:tasks.rados.rados.0.smithi064.stdout:1209: done (0 left) 2024-03-04T06:12:45.238 INFO:tasks.rados.rados.0.smithi064.stdout:1210: write (excl) oid 7 current snap is 140 2024-03-04T06:12:45.238 INFO:tasks.rados.rados.0.smithi064.stdout:1210: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2024-03-04T06:12:45.245 INFO:tasks.rados.rados.0.smithi064.stdout:1210: writing smithi06431743-7 from 598481 to 1311476 tid 1 2024-03-04T06:12:45.250 INFO:tasks.rados.rados.0.smithi064.stdout:1210: writing smithi06431743-7 from 1899355 to 2318220 tid 2 2024-03-04T06:12:45.258 INFO:tasks.rados.rados.0.smithi064.stdout:1210: writing smithi06431743-7 from 3093840 to 3798481 tid 3 2024-03-04T06:12:45.259 INFO:tasks.rados.rados.0.smithi064.stdout:1211: delete oid 36 current snap is 140 2024-03-04T06:12:45.288 INFO:tasks.rados.rados.0.smithi064.stdout:1210: finishing write tid 1 to smithi06431743-7 2024-03-04T06:12:45.302 INFO:tasks.rados.rados.0.smithi064.stdout:1210: finishing write tid 2 to smithi06431743-7 2024-03-04T06:12:45.317 INFO:tasks.rados.rados.0.smithi064.stdout:1211: done (1 left) 2024-03-04T06:12:45.317 INFO:tasks.rados.rados.0.smithi064.stdout:1212: write (excl) oid 8 current snap is 140 2024-03-04T06:12:45.317 INFO:tasks.rados.rados.0.smithi064.stdout:1212: seq_num 585 ranges {632095=463493,1616319=422534,2748686=487304} 2024-03-04T06:12:45.323 INFO:tasks.rados.rados.0.smithi064.stdout:1212: writing smithi06431743-8 from 632095 to 1095588 tid 1 2024-03-04T06:12:45.328 INFO:tasks.rados.rados.0.smithi064.stdout:1212: writing smithi06431743-8 from 1616319 to 2038853 tid 2 2024-03-04T06:12:45.335 INFO:tasks.rados.rados.0.smithi064.stdout:1212: writing smithi06431743-8 from 2748686 to 3235990 tid 3 2024-03-04T06:12:45.335 INFO:tasks.rados.rados.0.smithi064.stdout:1213: delete oid 14 current snap is 140 2024-03-04T06:12:45.335 INFO:tasks.rados.rados.0.smithi064.stdout:1210: finishing write tid 3 to smithi06431743-7 2024-03-04T06:12:45.344 INFO:tasks.rados.rados.0.smithi064.stdout:1210: finishing write tid 4 to smithi06431743-7 2024-03-04T06:12:45.348 INFO:tasks.rados.rados.0.smithi064.stdout:1210: finishing write tid 5 to smithi06431743-7 2024-03-04T06:12:45.348 INFO:tasks.rados.rados.0.smithi064.stdout:1210: oid 7 updating version 0 to 221 2024-03-04T06:12:45.348 INFO:tasks.rados.rados.0.smithi064.stdout:1210: oid 7 version 221 is already newer than 219 2024-03-04T06:12:45.348 INFO:tasks.rados.rados.0.smithi064.stdout:1210: oid 7 version 221 is already newer than 218 2024-03-04T06:12:45.348 INFO:tasks.rados.rados.0.smithi064.stdout:1210: oid 7 version 221 is already newer than 220 2024-03-04T06:12:45.348 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 221 (ObjNum 583 snap 140 seq_num 583) dirty exists 2024-03-04T06:12:45.348 INFO:tasks.rados.rados.0.smithi064.stdout:1210: left oid 7 (ObjNum 583 snap 140 seq_num 583) 2024-03-04T06:12:45.352 INFO:tasks.rados.rados.0.smithi064.stdout:1212: finishing write tid 1 to smithi06431743-8 2024-03-04T06:12:45.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:45.356+0000 7fcb3f19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:45.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:45.362+0000 7fcb3f19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:45.364 INFO:tasks.rados.rados.0.smithi064.stdout:1212: finishing write tid 2 to smithi06431743-8 2024-03-04T06:12:45.367 INFO:tasks.rados.rados.0.smithi064.stdout:1210: done (2 left) 2024-03-04T06:12:45.367 INFO:tasks.rados.rados.0.smithi064.stdout:1213: done (1 left) 2024-03-04T06:12:45.367 INFO:tasks.rados.rados.0.smithi064.stdout:1214: write oid 40 current snap is 140 2024-03-04T06:12:45.367 INFO:tasks.rados.rados.0.smithi064.stdout:1214: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2024-03-04T06:12:45.375 INFO:tasks.rados.rados.0.smithi064.stdout:1214: writing smithi06431743-40 from 665709 to 1396053 tid 1 2024-03-04T06:12:45.382 INFO:tasks.rados.rados.0.smithi064.stdout:1214: writing smithi06431743-40 from 2133283 to 2559486 tid 2 2024-03-04T06:12:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:1214: writing smithi06431743-40 from 3203532 to 3713380 tid 3 2024-03-04T06:12:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:1215: read oid 42 snap -1 2024-03-04T06:12:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:1215: expect (ObjNum 426 snap 101 seq_num 426) 2024-03-04T06:12:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:1216: read oid 20 snap -1 2024-03-04T06:12:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:1216: expect (ObjNum 557 snap 132 seq_num 557) 2024-03-04T06:12:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:1217: write oid 10 current snap is 140 2024-03-04T06:12:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:1212: finishing write tid 3 to smithi06431743-8 2024-03-04T06:12:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:1217: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2024-03-04T06:12:45.394 INFO:tasks.rados.rados.0.smithi064.stdout:1217: writing smithi06431743-10 from 682516 to 1088109 tid 1 2024-03-04T06:12:45.402 INFO:tasks.rados.rados.0.smithi064.stdout:1217: writing smithi06431743-10 from 1791765 to 2361626 tid 2 2024-03-04T06:12:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1217: writing smithi06431743-10 from 2830955 to 3552075 tid 3 2024-03-04T06:12:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1218: read oid 2 snap 138 2024-03-04T06:12:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1218: expect (ObjNum 535 snap 124 seq_num 535) 2024-03-04T06:12:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1219: snap_create 2024-03-04T06:12:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1212: finishing write tid 4 to smithi06431743-8 2024-03-04T06:12:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1212: finishing write tid 5 to smithi06431743-8 2024-03-04T06:12:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1212: oid 8 updating version 0 to 213 2024-03-04T06:12:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1212: oid 8 updating version 213 to 214 2024-03-04T06:12:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1212: oid 8 updating version 214 to 215 2024-03-04T06:12:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1212: oid 8 version 215 is already newer than 212 2024-03-04T06:12:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 215 (ObjNum 585 snap 140 seq_num 585) dirty exists 2024-03-04T06:12:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1212: left oid 8 (ObjNum 585 snap 140 seq_num 585) 2024-03-04T06:12:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1216: expect (ObjNum 557 snap 132 seq_num 557) 2024-03-04T06:12:45.613 INFO:tasks.rados.rados.0.smithi064.stdout:1218: expect (ObjNum 535 snap 124 seq_num 535) 2024-03-04T06:12:45.728 INFO:tasks.rados.rados.0.smithi064.stdout:1217: finishing write tid 1 to smithi06431743-10 2024-03-04T06:12:45.728 INFO:tasks.rados.rados.0.smithi064.stdout:1214: finishing write tid 1 to smithi06431743-40 2024-03-04T06:12:45.728 INFO:tasks.rados.rados.0.smithi064.stdout:1217: finishing write tid 2 to smithi06431743-10 2024-03-04T06:12:45.728 INFO:tasks.rados.rados.0.smithi064.stdout:1214: finishing write tid 2 to smithi06431743-40 2024-03-04T06:12:45.728 INFO:tasks.rados.rados.0.smithi064.stdout:1217: finishing write tid 3 to smithi06431743-10 2024-03-04T06:12:45.728 INFO:tasks.rados.rados.0.smithi064.stdout:1214: finishing write tid 3 to smithi06431743-40 2024-03-04T06:12:45.728 INFO:tasks.rados.rados.0.smithi064.stdout:1214: finishing write tid 4 to smithi06431743-40 2024-03-04T06:12:45.728 INFO:tasks.rados.rados.0.smithi064.stdout:1217: finishing write tid 4 to smithi06431743-10 2024-03-04T06:12:45.728 INFO:tasks.rados.rados.0.smithi064.stdout:1214: finishing write tid 5 to smithi06431743-40 2024-03-04T06:12:45.728 INFO:tasks.rados.rados.0.smithi064.stdout:1214: oid 40 updating version 0 to 199 2024-03-04T06:12:45.728 INFO:tasks.rados.rados.0.smithi064.stdout:1214: oid 40 updating version 199 to 200 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:1214: oid 40 updating version 200 to 202 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:1214: oid 40 version 202 is already newer than 202 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 202 (ObjNum 587 snap 140 seq_num 587) dirty exists 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:1214: left oid 40 (ObjNum 587 snap 140 seq_num 587) 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:1217: finishing write tid 5 to smithi06431743-10 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:1217: oid 10 updating version 0 to 201 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:1217: oid 10 version 201 is already newer than 199 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:1217: oid 10 updating version 201 to 202 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:1217: oid 10 version 202 is already newer than 200 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 202 (ObjNum 588 snap 140 seq_num 588) dirty exists 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:1217: left oid 10 (ObjNum 588 snap 140 seq_num 588) 2024-03-04T06:12:45.729 INFO:tasks.rados.rados.0.smithi064.stdout:1215: expect (ObjNum 426 snap 101 seq_num 426) 2024-03-04T06:12:46.237 INFO:tasks.rados.rados.0.smithi064.stdout:1212: done (6 left) 2024-03-04T06:12:46.237 INFO:tasks.rados.rados.0.smithi064.stdout:1214: done (5 left) 2024-03-04T06:12:46.237 INFO:tasks.rados.rados.0.smithi064.stdout:1215: done (4 left) 2024-03-04T06:12:46.237 INFO:tasks.rados.rados.0.smithi064.stdout:1216: done (3 left) 2024-03-04T06:12:46.237 INFO:tasks.rados.rados.0.smithi064.stdout:1217: done (2 left) 2024-03-04T06:12:46.237 INFO:tasks.rados.rados.0.smithi064.stdout:1218: done (1 left) 2024-03-04T06:12:46.237 INFO:tasks.rados.rados.0.smithi064.stdout:1219: done (0 left) 2024-03-04T06:12:46.237 INFO:tasks.rados.rados.0.smithi064.stdout:1220: delete oid 41 current snap is 141 2024-03-04T06:12:46.279 INFO:tasks.rados.rados.0.smithi064.stdout:1220: done (0 left) 2024-03-04T06:12:46.279 INFO:tasks.rados.rados.0.smithi064.stdout:1221: read oid 43 snap -1 2024-03-04T06:12:46.279 INFO:tasks.rados.rados.0.smithi064.stdout:1221: expect (ObjNum 540 snap 125 seq_num 540) 2024-03-04T06:12:46.279 INFO:tasks.rados.rados.0.smithi064.stdout:1222: delete oid 18 current snap is 141 2024-03-04T06:12:46.312 INFO:tasks.rados.rados.0.smithi064.stdout:1222: done (1 left) 2024-03-04T06:12:46.312 INFO:tasks.rados.rados.0.smithi064.stdout:1223: read oid 17 snap 138 2024-03-04T06:12:46.312 INFO:tasks.rados.rados.0.smithi064.stdout:1223: expect deleted 2024-03-04T06:12:46.312 INFO:tasks.rados.rados.0.smithi064.stdout:1224: write oid 10 current snap is 141 2024-03-04T06:12:46.312 INFO:tasks.rados.rados.0.smithi064.stdout:1224: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2024-03-04T06:12:46.319 INFO:tasks.rados.rados.0.smithi064.stdout:1224: writing smithi06431743-10 from 732937 to 1364277 tid 1 2024-03-04T06:12:46.325 INFO:tasks.rados.rados.0.smithi064.stdout:1224: writing smithi06431743-10 from 1850858 to 2284399 tid 2 2024-03-04T06:12:46.334 INFO:tasks.rados.rados.0.smithi064.stdout:1224: writing smithi06431743-10 from 2796871 to 3468160 tid 3 2024-03-04T06:12:46.337 INFO:tasks.rados.rados.0.smithi064.stdout:1224: writing smithi06431743-10 from 3888195 to 3932937 tid 4 2024-03-04T06:12:46.338 INFO:tasks.rados.rados.0.smithi064.stdout:1225: snap_create 2024-03-04T06:12:46.338 INFO:tasks.rados.rados.0.smithi064.stdout:1221: expect (ObjNum 540 snap 125 seq_num 540) 2024-03-04T06:12:46.630 INFO:tasks.rados.rados.0.smithi064.stdout:1224: finishing write tid 1 to smithi06431743-10 2024-03-04T06:12:46.630 INFO:tasks.rados.rados.0.smithi064.stdout:1224: finishing write tid 2 to smithi06431743-10 2024-03-04T06:12:46.630 INFO:tasks.rados.rados.0.smithi064.stdout:1224: finishing write tid 3 to smithi06431743-10 2024-03-04T06:12:46.630 INFO:tasks.rados.rados.0.smithi064.stdout:1224: finishing write tid 4 to smithi06431743-10 2024-03-04T06:12:46.630 INFO:tasks.rados.rados.0.smithi064.stdout:1224: finishing write tid 5 to smithi06431743-10 2024-03-04T06:12:46.630 INFO:tasks.rados.rados.0.smithi064.stdout:1224: finishing write tid 6 to smithi06431743-10 2024-03-04T06:12:46.630 INFO:tasks.rados.rados.0.smithi064.stdout:1224: oid 10 updating version 0 to 208 2024-03-04T06:12:46.630 INFO:tasks.rados.rados.0.smithi064.stdout:1224: oid 10 version 208 is already newer than 204 2024-03-04T06:12:46.630 INFO:tasks.rados.rados.0.smithi064.stdout:1224: oid 10 version 208 is already newer than 207 2024-03-04T06:12:46.630 INFO:tasks.rados.rados.0.smithi064.stdout:1224: oid 10 version 208 is already newer than 205 2024-03-04T06:12:46.630 INFO:tasks.rados.rados.0.smithi064.stdout:1224: oid 10 version 208 is already newer than 206 2024-03-04T06:12:46.631 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 208 (ObjNum 591 snap 141 seq_num 591) dirty exists 2024-03-04T06:12:46.631 INFO:tasks.rados.rados.0.smithi064.stdout:1224: left oid 10 (ObjNum 591 snap 141 seq_num 591) 2024-03-04T06:12:47.250 INFO:tasks.rados.rados.0.smithi064.stdout:1221: done (3 left) 2024-03-04T06:12:47.250 INFO:tasks.rados.rados.0.smithi064.stdout:1223: done (2 left) 2024-03-04T06:12:47.251 INFO:tasks.rados.rados.0.smithi064.stdout:1224: done (1 left) 2024-03-04T06:12:47.251 INFO:tasks.rados.rados.0.smithi064.stdout:1225: done (0 left) 2024-03-04T06:12:47.251 INFO:tasks.rados.rados.0.smithi064.stdout:1226: read oid 28 snap 139 2024-03-04T06:12:47.251 INFO:tasks.rados.rados.0.smithi064.stdout:1226: expect (ObjNum 559 snap 132 seq_num 559) 2024-03-04T06:12:47.251 INFO:tasks.rados.rados.0.smithi064.stdout:1227: snap_remove snap 141 2024-03-04T06:12:47.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:47.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:47.567+0000 7f5b7786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:47.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:47.572+0000 7f5b7786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:48.217 INFO:tasks.rados.rados.0.smithi064.stdout:1227: done (1 left) 2024-03-04T06:12:48.218 INFO:tasks.rados.rados.0.smithi064.stdout:1228: read oid 46 snap -1 2024-03-04T06:12:48.218 INFO:tasks.rados.rados.0.smithi064.stdout:1228: expect deleted 2024-03-04T06:12:48.218 INFO:tasks.rados.rados.0.smithi064.stdout:1229: read oid 14 snap -1 2024-03-04T06:12:48.218 INFO:tasks.rados.rados.0.smithi064.stdout:1229: expect deleted 2024-03-04T06:12:48.218 INFO:tasks.rados.rados.0.smithi064.stdout:1230: read oid 29 snap 140 2024-03-04T06:12:48.218 INFO:tasks.rados.rados.0.smithi064.stdout:1230: expect (ObjNum 467 snap 111 seq_num 467) 2024-03-04T06:12:48.218 INFO:tasks.rados.rados.0.smithi064.stdout:1231: delete oid 1 current snap is 142 2024-03-04T06:12:48.218 INFO:tasks.rados.rados.0.smithi064.stdout:1226: expect (ObjNum 559 snap 132 seq_num 559) 2024-03-04T06:12:48.359 INFO:tasks.rados.rados.0.smithi064.stdout:1230: expect (ObjNum 467 snap 111 seq_num 467) 2024-03-04T06:12:48.597 INFO:tasks.rados.rados.0.smithi064.stdout:1226: done (4 left) 2024-03-04T06:12:48.597 INFO:tasks.rados.rados.0.smithi064.stdout:1228: done (3 left) 2024-03-04T06:12:48.598 INFO:tasks.rados.rados.0.smithi064.stdout:1229: done (2 left) 2024-03-04T06:12:48.598 INFO:tasks.rados.rados.0.smithi064.stdout:1230: done (1 left) 2024-03-04T06:12:48.598 INFO:tasks.rados.rados.0.smithi064.stdout:1231: done (0 left) 2024-03-04T06:12:48.598 INFO:tasks.rados.rados.0.smithi064.stdout:1232: write oid 22 current snap is 142 2024-03-04T06:12:48.598 INFO:tasks.rados.rados.0.smithi064.stdout:1232: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2024-03-04T06:12:48.604 INFO:tasks.rados.rados.0.smithi064.stdout:1232: writing smithi06431743-22 from 766551 to 1264742 tid 1 2024-03-04T06:12:48.610 INFO:tasks.rados.rados.0.smithi064.stdout:1232: writing smithi06431743-22 from 1684175 to 2405032 tid 2 2024-03-04T06:12:48.618 INFO:tasks.rados.rados.0.smithi064.stdout:1232: writing smithi06431743-22 from 2851717 to 3545550 tid 3 2024-03-04T06:12:48.618 INFO:tasks.rados.rados.0.smithi064.stdout:1233: write oid 21 current snap is 142 2024-03-04T06:12:48.618 INFO:tasks.rados.rados.0.smithi064.stdout:1233: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2024-03-04T06:12:48.626 INFO:tasks.rados.rados.0.smithi064.stdout:1233: writing smithi06431743-21 from 783358 to 1356798 tid 1 2024-03-04T06:12:48.634 INFO:tasks.rados.rados.0.smithi064.stdout:1233: writing smithi06431743-21 from 2026304 to 2607172 tid 2 2024-03-04T06:12:48.642 INFO:tasks.rados.rados.0.smithi064.stdout:1233: writing smithi06431743-21 from 3279140 to 3900598 tid 3 2024-03-04T06:12:48.642 INFO:tasks.rados.rados.0.smithi064.stdout:1234: snap_create 2024-03-04T06:12:48.642 INFO:tasks.rados.rados.0.smithi064.stdout:1232: finishing write tid 1 to smithi06431743-22 2024-03-04T06:12:48.660 INFO:tasks.rados.rados.0.smithi064.stdout:1232: finishing write tid 2 to smithi06431743-22 2024-03-04T06:12:48.670 INFO:tasks.rados.rados.0.smithi064.stdout:1232: finishing write tid 3 to smithi06431743-22 2024-03-04T06:12:48.682 INFO:tasks.rados.rados.0.smithi064.stdout:1232: finishing write tid 4 to smithi06431743-22 2024-03-04T06:12:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1232: finishing write tid 5 to smithi06431743-22 2024-03-04T06:12:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1232: oid 22 updating version 0 to 426 2024-03-04T06:12:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1232: oid 22 updating version 426 to 428 2024-03-04T06:12:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1232: oid 22 version 428 is already newer than 428 2024-03-04T06:12:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1232: oid 22 updating version 428 to 429 2024-03-04T06:12:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 429 (ObjNum 593 snap 142 seq_num 593) dirty exists 2024-03-04T06:12:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1232: left oid 22 (ObjNum 593 snap 142 seq_num 593) 2024-03-04T06:12:48.686 INFO:tasks.rados.rados.0.smithi064.stdout:1233: finishing write tid 1 to smithi06431743-21 2024-03-04T06:12:48.701 INFO:tasks.rados.rados.0.smithi064.stdout:1233: finishing write tid 2 to smithi06431743-21 2024-03-04T06:12:48.711 INFO:tasks.rados.rados.0.smithi064.stdout:1233: finishing write tid 3 to smithi06431743-21 2024-03-04T06:12:48.722 INFO:tasks.rados.rados.0.smithi064.stdout:1233: finishing write tid 4 to smithi06431743-21 2024-03-04T06:12:48.722 INFO:tasks.rados.rados.0.smithi064.stdout:1233: finishing write tid 5 to smithi06431743-21 2024-03-04T06:12:48.723 INFO:tasks.rados.rados.0.smithi064.stdout:1233: oid 21 updating version 0 to 191 2024-03-04T06:12:48.723 INFO:tasks.rados.rados.0.smithi064.stdout:1233: oid 21 updating version 191 to 193 2024-03-04T06:12:48.723 INFO:tasks.rados.rados.0.smithi064.stdout:1233: oid 21 version 193 is already newer than 190 2024-03-04T06:12:48.723 INFO:tasks.rados.rados.0.smithi064.stdout:1233: oid 21 version 193 is already newer than 192 2024-03-04T06:12:48.723 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 193 (ObjNum 594 snap 142 seq_num 594) dirty exists 2024-03-04T06:12:48.723 INFO:tasks.rados.rados.0.smithi064.stdout:1233: left oid 21 (ObjNum 594 snap 142 seq_num 594) 2024-03-04T06:12:49.250 INFO:tasks.rados.rados.0.smithi064.stdout:1232: done (2 left) 2024-03-04T06:12:49.250 INFO:tasks.rados.rados.0.smithi064.stdout:1233: done (1 left) 2024-03-04T06:12:49.251 INFO:tasks.rados.rados.0.smithi064.stdout:1234: done (0 left) 2024-03-04T06:12:49.251 INFO:tasks.rados.rados.0.smithi064.stdout:1235: write oid 32 current snap is 143 2024-03-04T06:12:49.251 INFO:tasks.rados.rados.0.smithi064.stdout:1235: seq_num 595 ranges {400165=648689,1684786=315379} 2024-03-04T06:12:49.256 INFO:tasks.rados.rados.0.smithi064.stdout:1235: writing smithi06431743-32 from 400165 to 1048854 tid 1 2024-03-04T06:12:49.261 INFO:tasks.rados.rados.0.smithi064.stdout:1235: writing smithi06431743-32 from 1684786 to 2000165 tid 2 2024-03-04T06:12:49.261 INFO:tasks.rados.rados.0.smithi064.stdout:1236: delete oid 34 current snap is 143 2024-03-04T06:12:49.308 INFO:tasks.rados.rados.0.smithi064.stdout:1235: finishing write tid 1 to smithi06431743-32 2024-03-04T06:12:49.318 INFO:tasks.rados.rados.0.smithi064.stdout:1235: finishing write tid 2 to smithi06431743-32 2024-03-04T06:12:49.327 INFO:tasks.rados.rados.0.smithi064.stdout:1235: finishing write tid 3 to smithi06431743-32 2024-03-04T06:12:49.330 INFO:tasks.rados.rados.0.smithi064.stdout:1235: finishing write tid 4 to smithi06431743-32 2024-03-04T06:12:49.330 INFO:tasks.rados.rados.0.smithi064.stdout:1235: oid 32 updating version 0 to 477 2024-03-04T06:12:49.330 INFO:tasks.rados.rados.0.smithi064.stdout:1235: oid 32 version 477 is already newer than 476 2024-03-04T06:12:49.330 INFO:tasks.rados.rados.0.smithi064.stdout:1235: oid 32 version 477 is already newer than 475 2024-03-04T06:12:49.330 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 477 (ObjNum 595 snap 143 seq_num 595) dirty exists 2024-03-04T06:12:49.331 INFO:tasks.rados.rados.0.smithi064.stdout:1235: left oid 32 (ObjNum 595 snap 143 seq_num 595) 2024-03-04T06:12:49.344 INFO:tasks.rados.rados.0.smithi064.stdout:1235: done (1 left) 2024-03-04T06:12:49.344 INFO:tasks.rados.rados.0.smithi064.stdout:1236: done (0 left) 2024-03-04T06:12:49.344 INFO:tasks.rados.rados.0.smithi064.stdout:1237: read oid 37 snap -1 2024-03-04T06:12:49.344 INFO:tasks.rados.rados.0.smithi064.stdout:1237: expect deleted 2024-03-04T06:12:49.344 INFO:tasks.rados.rados.0.smithi064.stdout:1238: delete oid 19 current snap is 143 2024-03-04T06:12:49.377 INFO:tasks.rados.rados.0.smithi064.stdout:1237: done (1 left) 2024-03-04T06:12:49.377 INFO:tasks.rados.rados.0.smithi064.stdout:1238: done (0 left) 2024-03-04T06:12:49.377 INFO:tasks.rados.rados.0.smithi064.stdout:1239: snap_create 2024-03-04T06:12:49.381 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:12:49.381 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:12:49.381 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:12:49.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:12:49.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:49.487+0000 7f9cd7b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:49.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:49.494+0000 7f9cd7b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:49.507 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:49.510 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:49.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:12:49.618 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:49.613+0000 7fd8f0107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:49.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:49.620+0000 7fd8f0107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:49.632 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:49.633 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:49.635 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:49.636 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:49.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:12:49.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:49.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:49.740+0000 7f85e866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:49.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:49.747+0000 7f85e866d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:49.761 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:49.761 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:49.761 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:49.761 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:49.761 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:49.761 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:49.761 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:49.762 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:49.762 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:49.762 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:49.762 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:49.762 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:49.764 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:49.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:12:49.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:49.785+0000 7f81f5806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:49.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:49.789+0000 7f81f5806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:49.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:49.876+0000 7fbb91e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:49.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:49.886+0000 7fbb91e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:49.903 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:12:49.903 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:12:49.903 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:12:49.903 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:12:49.903 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:12:49.903 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:12:49.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:12:49.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:12:49.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:12:49.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:12:49.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:12:49.904 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:12:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:12:49.907 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:12:50.254 INFO:tasks.rados.rados.0.smithi064.stdout:1239: done (0 left) 2024-03-04T06:12:50.254 INFO:tasks.rados.rados.0.smithi064.stdout:1240: delete oid 15 current snap is 144 2024-03-04T06:12:50.297 INFO:tasks.rados.rados.0.smithi064.stdout:1240: done (0 left) 2024-03-04T06:12:50.297 INFO:tasks.rados.rados.0.smithi064.stdout:1241: read oid 32 snap -1 2024-03-04T06:12:50.298 INFO:tasks.rados.rados.0.smithi064.stdout:1241: expect (ObjNum 595 snap 143 seq_num 595) 2024-03-04T06:12:50.298 INFO:tasks.rados.rados.0.smithi064.stdout:1242: read oid 49 snap -1 2024-03-04T06:12:50.298 INFO:tasks.rados.rados.0.smithi064.stdout:1242: expect (ObjNum 491 snap 114 seq_num 491) 2024-03-04T06:12:50.298 INFO:tasks.rados.rados.0.smithi064.stdout:1243: read oid 37 snap 138 2024-03-04T06:12:50.298 INFO:tasks.rados.rados.0.smithi064.stdout:1243: expect deleted 2024-03-04T06:12:50.298 INFO:tasks.rados.rados.0.smithi064.stdout:1244: write (excl) oid 28 current snap is 144 2024-03-04T06:12:50.298 INFO:tasks.rados.rados.0.smithi064.stdout:1244: seq_num 599 ranges {467393=549685,1802361=265032} 2024-03-04T06:12:50.303 INFO:tasks.rados.rados.0.smithi064.stdout:1244: writing smithi06431743-28 from 467393 to 1017078 tid 1 2024-03-04T06:12:50.309 INFO:tasks.rados.rados.0.smithi064.stdout:1244: writing smithi06431743-28 from 1802361 to 2067393 tid 2 2024-03-04T06:12:50.309 INFO:tasks.rados.rados.0.smithi064.stdout:1245: read oid 34 snap -1 2024-03-04T06:12:50.309 INFO:tasks.rados.rados.0.smithi064.stdout:1245: expect deleted 2024-03-04T06:12:50.309 INFO:tasks.rados.rados.0.smithi064.stdout:1246: write oid 40 current snap is 144 2024-03-04T06:12:50.309 INFO:tasks.rados.rados.0.smithi064.stdout:1246: seq_num 600 ranges {484200=624934,1860843=223357} 2024-03-04T06:12:50.315 INFO:tasks.rados.rados.0.smithi064.stdout:1246: writing smithi06431743-40 from 484200 to 1109134 tid 1 2024-03-04T06:12:50.321 INFO:tasks.rados.rados.0.smithi064.stdout:1246: writing smithi06431743-40 from 1860843 to 2084200 tid 2 2024-03-04T06:12:50.321 INFO:tasks.rados.rados.0.smithi064.stdout:1247: write (excl) oid 48 current snap is 144 2024-03-04T06:12:50.321 INFO:tasks.rados.rados.0.smithi064.stdout:1247: seq_num 601 ranges {501007=416536,1635678=465329} 2024-03-04T06:12:50.328 INFO:tasks.rados.rados.0.smithi064.stdout:1247: writing smithi06431743-48 from 501007 to 917543 tid 1 2024-03-04T06:12:50.334 INFO:tasks.rados.rados.0.smithi064.stdout:1247: writing smithi06431743-48 from 1635678 to 2101007 tid 2 2024-03-04T06:12:50.334 INFO:tasks.rados.rados.0.smithi064.stdout:1248: snap_remove snap 143 2024-03-04T06:12:51.263 INFO:tasks.rados.rados.0.smithi064.stdout:1248: done (7 left) 2024-03-04T06:12:51.263 INFO:tasks.rados.rados.0.smithi064.stdout:1249: read oid 15 snap -1 2024-03-04T06:12:51.263 INFO:tasks.rados.rados.0.smithi064.stdout:1249: expect deleted 2024-03-04T06:12:51.263 INFO:tasks.rados.rados.0.smithi064.stdout:1250: write oid 27 current snap is 144 2024-03-04T06:12:51.263 INFO:tasks.rados.rados.0.smithi064.stdout:1242: expect (ObjNum 491 snap 114 seq_num 491) 2024-03-04T06:12:51.348 INFO:tasks.rados.rados.0.smithi064.stdout:1247: finishing write tid 1 to smithi06431743-48 2024-03-04T06:12:51.348 INFO:tasks.rados.rados.0.smithi064.stdout:1250: seq_num 602 ranges {517814=491785,1577807=540007} 2024-03-04T06:12:51.354 INFO:tasks.rados.rados.0.smithi064.stdout:1250: writing smithi06431743-27 from 517814 to 1009599 tid 1 2024-03-04T06:12:51.360 INFO:tasks.rados.rados.0.smithi064.stdout:1250: writing smithi06431743-27 from 1577807 to 2117814 tid 2 2024-03-04T06:12:51.360 INFO:tasks.rados.rados.0.smithi064.stdout:1242: done (8 left) 2024-03-04T06:12:51.361 INFO:tasks.rados.rados.0.smithi064.stdout:1243: done (7 left) 2024-03-04T06:12:51.361 INFO:tasks.rados.rados.0.smithi064.stdout:1251: read oid 50 snap -1 2024-03-04T06:12:51.361 INFO:tasks.rados.rados.0.smithi064.stdout:1251: expect deleted 2024-03-04T06:12:51.361 INFO:tasks.rados.rados.0.smithi064.stdout:1252: read oid 25 snap -1 2024-03-04T06:12:51.361 INFO:tasks.rados.rados.0.smithi064.stdout:1252: expect (ObjNum 575 snap 138 seq_num 575) 2024-03-04T06:12:51.361 INFO:tasks.rados.rados.0.smithi064.stdout:1253: snap_remove snap 142 2024-03-04T06:12:51.864 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:51.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:51.969+0000 7fbb57add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:51.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:51.973+0000 7fbb57add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:52.259 INFO:tasks.rados.rados.0.smithi064.stdout:1253: done (9 left) 2024-03-04T06:12:52.259 INFO:tasks.rados.rados.0.smithi064.stdout:1254: read oid 20 snap -1 2024-03-04T06:12:52.259 INFO:tasks.rados.rados.0.smithi064.stdout:1254: expect (ObjNum 557 snap 132 seq_num 557) 2024-03-04T06:12:52.259 INFO:tasks.rados.rados.0.smithi064.stdout:1255: snap_remove snap 138 2024-03-04T06:12:53.265 INFO:tasks.rados.rados.0.smithi064.stdout:1255: done (10 left) 2024-03-04T06:12:53.265 INFO:tasks.rados.rados.0.smithi064.stdout:1256: snap_create 2024-03-04T06:12:53.265 INFO:tasks.rados.rados.0.smithi064.stdout:1246: finishing write tid 1 to smithi06431743-40 2024-03-04T06:12:53.265 INFO:tasks.rados.rados.0.smithi064.stdout:1241: expect (ObjNum 595 snap 143 seq_num 595) 2024-03-04T06:12:53.424 INFO:tasks.rados.rados.0.smithi064.stdout:1247: finishing write tid 2 to smithi06431743-48 2024-03-04T06:12:53.424 INFO:tasks.rados.rados.0.smithi064.stdout:1246: finishing write tid 2 to smithi06431743-40 2024-03-04T06:12:53.424 INFO:tasks.rados.rados.0.smithi064.stdout:1244: finishing write tid 1 to smithi06431743-28 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1247: finishing write tid 3 to smithi06431743-48 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1247: finishing write tid 4 to smithi06431743-48 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1247: oid 48 updating version 0 to 224 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1247: oid 48 updating version 224 to 225 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1247: oid 48 version 225 is already newer than 223 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 225 (ObjNum 601 snap 144 seq_num 601) dirty exists 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1247: left oid 48 (ObjNum 601 snap 144 seq_num 601) 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1246: finishing write tid 3 to smithi06431743-40 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1246: finishing write tid 4 to smithi06431743-40 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1246: oid 40 updating version 0 to 204 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1246: oid 40 updating version 204 to 206 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1246: oid 40 version 206 is already newer than 205 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 206 (ObjNum 600 snap 144 seq_num 600) dirty exists 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1246: left oid 40 (ObjNum 600 snap 144 seq_num 600) 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1244: finishing write tid 2 to smithi06431743-28 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1244: finishing write tid 3 to smithi06431743-28 2024-03-04T06:12:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1244: finishing write tid 4 to smithi06431743-28 2024-03-04T06:12:53.426 INFO:tasks.rados.rados.0.smithi064.stdout:1244: oid 28 updating version 0 to 483 2024-03-04T06:12:53.426 INFO:tasks.rados.rados.0.smithi064.stdout:1244: oid 28 version 483 is already newer than 482 2024-03-04T06:12:53.426 INFO:tasks.rados.rados.0.smithi064.stdout:1244: oid 28 version 483 is already newer than 481 2024-03-04T06:12:53.426 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 483 (ObjNum 599 snap 144 seq_num 599) dirty exists 2024-03-04T06:12:53.426 INFO:tasks.rados.rados.0.smithi064.stdout:1244: left oid 28 (ObjNum 599 snap 144 seq_num 599) 2024-03-04T06:12:53.426 INFO:tasks.rados.rados.0.smithi064.stdout:1252: expect (ObjNum 575 snap 138 seq_num 575) 2024-03-04T06:12:53.679 INFO:tasks.rados.rados.0.smithi064.stdout:1250: finishing write tid 1 to smithi06431743-27 2024-03-04T06:12:53.680 INFO:tasks.rados.rados.0.smithi064.stdout:1250: finishing write tid 2 to smithi06431743-27 2024-03-04T06:12:53.680 INFO:tasks.rados.rados.0.smithi064.stdout:1250: finishing write tid 3 to smithi06431743-27 2024-03-04T06:12:53.680 INFO:tasks.rados.rados.0.smithi064.stdout:1250: finishing write tid 4 to smithi06431743-27 2024-03-04T06:12:53.680 INFO:tasks.rados.rados.0.smithi064.stdout:1250: oid 27 updating version 0 to 166 2024-03-04T06:12:53.680 INFO:tasks.rados.rados.0.smithi064.stdout:1250: oid 27 version 166 is already newer than 165 2024-03-04T06:12:53.680 INFO:tasks.rados.rados.0.smithi064.stdout:1250: oid 27 version 166 is already newer than 164 2024-03-04T06:12:53.680 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 166 (ObjNum 602 snap 144 seq_num 602) dirty exists 2024-03-04T06:12:53.680 INFO:tasks.rados.rados.0.smithi064.stdout:1250: left oid 27 (ObjNum 602 snap 144 seq_num 602) 2024-03-04T06:12:53.680 INFO:tasks.rados.rados.0.smithi064.stdout:1254: expect (ObjNum 557 snap 132 seq_num 557) 2024-03-04T06:12:54.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:54.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:54.148+0000 7fb9e8c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:54.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:54.152+0000 7fb9e8c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:54.265 INFO:tasks.rados.rados.0.smithi064.stdout:1241: done (10 left) 2024-03-04T06:12:54.265 INFO:tasks.rados.rados.0.smithi064.stdout:1244: done (9 left) 2024-03-04T06:12:54.265 INFO:tasks.rados.rados.0.smithi064.stdout:1245: done (8 left) 2024-03-04T06:12:54.265 INFO:tasks.rados.rados.0.smithi064.stdout:1246: done (7 left) 2024-03-04T06:12:54.265 INFO:tasks.rados.rados.0.smithi064.stdout:1247: done (6 left) 2024-03-04T06:12:54.265 INFO:tasks.rados.rados.0.smithi064.stdout:1249: done (5 left) 2024-03-04T06:12:54.265 INFO:tasks.rados.rados.0.smithi064.stdout:1250: done (4 left) 2024-03-04T06:12:54.266 INFO:tasks.rados.rados.0.smithi064.stdout:1251: done (3 left) 2024-03-04T06:12:54.266 INFO:tasks.rados.rados.0.smithi064.stdout:1252: done (2 left) 2024-03-04T06:12:54.266 INFO:tasks.rados.rados.0.smithi064.stdout:1254: done (1 left) 2024-03-04T06:12:54.266 INFO:tasks.rados.rados.0.smithi064.stdout:1256: done (0 left) 2024-03-04T06:12:54.266 INFO:tasks.rados.rados.0.smithi064.stdout:1257: snap_remove snap 144 2024-03-04T06:12:54.908 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:12:54.908 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:12:54.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:12:55.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:55.013+0000 7fd60575d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:55.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:55.019+0000 7fd60575d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:55.209 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:12:55.209 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:12:55.220 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-03-04T06:12:54.770331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115385577,"num_objects":421,"num_object_clones":308,"num_object_copies":1263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":76,"num_read":2381,"num_read_kb":1714971,"num_write":2133,"num_write_kb":549350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4007,"ondisk_log_size":4007,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":440884,"kb_used_data":296148,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377046476,"statfs":{"total":386547056640,"available":386095591424,"internally_reserved":0,"allocated":303255552,"data_stored":323655827,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9949843,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":25,"num_read_kb":20651,"num_write":20,"num_write_kb":4335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000915"},"pg_stats":[{"pgid":"3.f","version":"295'202","reported_seq":1002,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.324119+0000","last_change":"2024-03-04T06:12:48.261040+0000","last_active":"2024-03-04T06:12:50.324119+0000","last_peered":"2024-03-04T06:12:50.324119+0000","last_clean":"2024-03-04T06:12:50.324119+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:12:50.324119+0000","last_undegraded":"2024-03-04T06:12:50.324119+0000","last_fullsized":"2024-03-04T06:12:50.324119+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10101271,"num_objects":29,"num_object_clones":20,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":90,"num_read_kb":55731,"num_write":119,"num_write_kb":32797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"296'162","reported_seq":987,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.321323+0000","last_change":"2024-03-04T06:12:48.255182+0000","last_active":"2024-03-04T06:12:50.321323+0000","last_peered":"2024-03-04T06:12:50.321323+0000","last_clean":"2024-03-04T06:12:50.321323+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:12:50.321323+0000","last_undegraded":"2024-03-04T06:12:50.321323+0000","last_fullsized":"2024-03-04T06:12:50.321323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3579957,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":69,"num_read_kb":46326,"num_write":73,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"294'281","reported_seq":1062,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-03-04T06:12:51.268166+0000","last_change":"2024-03-04T06:12:51.276984+0000","last_active":"2024-03-04T06:12:51.268166+0000","last_peered":"2024-03-04T06:12:51.268166+0000","last_clean":"2024-03-04T06:12:51.268166+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:12:51.268166+0000","last_undegraded":"2024-03-04T06:12:51.268166+0000","last_fullsized":"2024-03-04T06:12:51.268166+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13177084,"num_objects":33,"num_object_clones":25,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":156,"num_read_kb":104657,"num_write":156,"num_write_kb":43443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"294'199","reported_seq":1021,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.313371+0000","last_change":"2024-03-04T06:12:48.252364+0000","last_active":"2024-03-04T06:12:50.313371+0000","last_peered":"2024-03-04T06:12:50.313371+0000","last_clean":"2024-03-04T06:12:50.313371+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:12:50.313371+0000","last_undegraded":"2024-03-04T06:12:50.313371+0000","last_fullsized":"2024-03-04T06:12:50.313371+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983655,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":125,"num_read_kb":96803,"num_write":119,"num_write_kb":34146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"249'92","reported_seq":911,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.329470+0000","last_change":"2024-03-04T06:12:48.242647+0000","last_active":"2024-03-04T06:12:50.329470+0000","last_peered":"2024-03-04T06:12:50.329470+0000","last_clean":"2024-03-04T06:12:50.329470+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:12:50.329470+0000","last_undegraded":"2024-03-04T06:12:50.329470+0000","last_fullsized":"2024-03-04T06:12:50.329470+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":17041,"num_write":34,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"296'208","reported_seq":1018,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.326583+0000","last_change":"2024-03-04T06:12:48.261452+0000","last_active":"2024-03-04T06:12:50.326583+0000","last_peered":"2024-03-04T06:12:50.326583+0000","last_clean":"2024-03-04T06:12:50.326583+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:12:50.326583+0000","last_undegraded":"2024-03-04T06:12:50.326583+0000","last_fullsized":"2024-03-04T06:12:50.326583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10067690,"num_objects":23,"num_object_clones":16,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":120,"num_read_kb":87291,"num_write":122,"num_write_kb":36476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":857,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.325842+0000","last_change":"2024-03-04T06:12:48.261549+0000","last_active":"2024-03-04T06:12:50.325842+0000","last_peered":"2024-03-04T06:12:50.325842+0000","last_clean":"2024-03-04T06:12:50.325842+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:12:50.325842+0000","last_undegraded":"2024-03-04T06:12:50.325842+0000","last_fullsized":"2024-03-04T06:12:50.325842+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"293'161","reported_seq":984,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-03-04T06:12:51.265361+0000","last_change":"2024-03-04T06:12:51.276706+0000","last_active":"2024-03-04T06:12:51.265361+0000","last_peered":"2024-03-04T06:12:51.265361+0000","last_clean":"2024-03-04T06:12:51.265361+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:12:51.265361+0000","last_undegraded":"2024-03-04T06:12:51.265361+0000","last_fullsized":"2024-03-04T06:12:51.265361+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5983490,"num_objects":18,"num_object_clones":11,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":124,"num_read_kb":117183,"num_write":78,"num_write_kb":21237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":871,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.310977+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:12:50.310977+0000","last_peered":"2024-03-04T06:12:50.310977+0000","last_clean":"2024-03-04T06:12:50.310977+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:12:50.310977+0000","last_undegraded":"2024-03-04T06:12:50.310977+0000","last_fullsized":"2024-03-04T06:12:50.310977+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":727,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-03-04T06:12:51.279470+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:12:51.279470+0000","last_peered":"2024-03-04T06:12:51.279470+0000","last_clean":"2024-03-04T06:12:51.279470+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:12:51.279470+0000","last_undegraded":"2024-03-04T06:12:51.279470+0000","last_fullsized":"2024-03-04T06:12:51.279470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"296'243","reported_seq":1029,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-03-04T06:12:51.279040+0000","last_change":"2024-03-04T06:12:51.281412+0000","last_active":"2024-03-04T06:12:51.279040+0000","last_peered":"2024-03-04T06:12:51.279040+0000","last_clean":"2024-03-04T06:12:51.279040+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:12:51.279040+0000","last_undegraded":"2024-03-04T06:12:51.279040+0000","last_fullsized":"2024-03-04T06:12:51.279040+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6689384,"num_objects":26,"num_object_clones":19,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":143,"num_read_kb":109564,"num_write":120,"num_write_kb":30600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":728,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.327133+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:12:50.327133+0000","last_peered":"2024-03-04T06:12:50.327133+0000","last_clean":"2024-03-04T06:12:50.327133+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:12:50.327133+0000","last_undegraded":"2024-03-04T06:12:50.327133+0000","last_fullsized":"2024-03-04T06:12:50.327133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"300'225","reported_seq":1000,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.397683+0000","last_change":"2024-03-04T06:12:48.248163+0000","last_active":"2024-03-04T06:12:50.397683+0000","last_peered":"2024-03-04T06:12:50.397683+0000","last_clean":"2024-03-04T06:12:50.397683+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:12:50.397683+0000","last_undegraded":"2024-03-04T06:12:50.397683+0000","last_fullsized":"2024-03-04T06:12:50.397683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899488,"num_objects":23,"num_object_clones":18,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":110,"num_read_kb":86126,"num_write":96,"num_write_kb":28103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":727,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-03-04T06:12:51.271441+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:12:51.271441+0000","last_peered":"2024-03-04T06:12:51.271441+0000","last_clean":"2024-03-04T06:12:51.271441+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:12:51.271441+0000","last_undegraded":"2024-03-04T06:12:51.271441+0000","last_fullsized":"2024-03-04T06:12:51.271441+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"300'483","reported_seq":1251,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.421013+0000","last_change":"2024-03-04T06:12:48.254198+0000","last_active":"2024-03-04T06:12:50.421013+0000","last_peered":"2024-03-04T06:12:50.421013+0000","last_clean":"2024-03-04T06:12:50.421013+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:12:50.421013+0000","last_undegraded":"2024-03-04T06:12:50.421013+0000","last_fullsized":"2024-03-04T06:12:50.421013+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4067558,"num_objects":51,"num_object_clones":39,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":335,"num_read_kb":281957,"num_write":259,"num_write_kb":70870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":728,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.318963+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:12:50.318963+0000","last_peered":"2024-03-04T06:12:50.318963+0000","last_clean":"2024-03-04T06:12:50.318963+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:12:50.318963+0000","last_undegraded":"2024-03-04T06:12:50.318963+0000","last_fullsized":"2024-03-04T06:12:50.318963+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"299'477","reported_seq":1185,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:49.359149+0000","last_change":"2024-03-04T06:12:48.255028+0000","last_active":"2024-03-04T06:12:49.359149+0000","last_peered":"2024-03-04T06:12:49.359149+0000","last_clean":"2024-03-04T06:12:49.359149+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:12:49.359149+0000","last_undegraded":"2024-03-04T06:12:49.359149+0000","last_fullsized":"2024-03-04T06:12:49.359149+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13109790,"num_objects":47,"num_object_clones":34,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":213,"num_read_kb":155101,"num_write":227,"num_write_kb":60703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"301'197","reported_seq":1043,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-03-04T06:12:51.278785+0000","last_change":"2024-03-04T06:12:51.297622+0000","last_active":"2024-03-04T06:12:51.278785+0000","last_peered":"2024-03-04T06:12:51.278785+0000","last_clean":"2024-03-04T06:12:51.278785+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:12:51.278785+0000","last_undegraded":"2024-03-04T06:12:51.278785+0000","last_fullsized":"2024-03-04T06:12:51.278785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680997,"num_objects":22,"num_object_clones":16,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":193,"num_read_kb":157563,"num_write":108,"num_write_kb":28860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":764,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.328260+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:12:50.328260+0000","last_peered":"2024-03-04T06:12:50.328260+0000","last_clean":"2024-03-04T06:12:50.328260+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:12:50.328260+0000","last_undegraded":"2024-03-04T06:12:50.328260+0000","last_fullsized":"2024-03-04T06:12:50.328260+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"295'215","reported_seq":1046,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.323495+0000","last_change":"2024-03-04T06:12:48.270036+0000","last_active":"2024-03-04T06:12:50.323495+0000","last_peered":"2024-03-04T06:12:50.323495+0000","last_clean":"2024-03-04T06:12:50.323495+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:12:50.323495+0000","last_undegraded":"2024-03-04T06:12:50.323495+0000","last_fullsized":"2024-03-04T06:12:50.323495+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6538121,"num_objects":28,"num_object_clones":18,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":118,"num_read_kb":67272,"num_write":116,"num_write_kb":31360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":738,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.322030+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:12:50.322030+0000","last_peered":"2024-03-04T06:12:50.322030+0000","last_clean":"2024-03-04T06:12:50.322030+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:12:50.322030+0000","last_undegraded":"2024-03-04T06:12:50.322030+0000","last_fullsized":"2024-03-04T06:12:50.322030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"295'336","reported_seq":1099,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.315916+0000","last_change":"2024-03-04T06:12:48.247743+0000","last_active":"2024-03-04T06:12:50.315916+0000","last_peered":"2024-03-04T06:12:50.315916+0000","last_clean":"2024-03-04T06:12:50.315916+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:12:50.315916+0000","last_undegraded":"2024-03-04T06:12:50.315916+0000","last_fullsized":"2024-03-04T06:12:50.315916+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840647,"num_objects":34,"num_object_clones":26,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":180,"num_read_kb":122087,"num_write":172,"num_write_kb":47172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":764,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.317536+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:12:50.317536+0000","last_peered":"2024-03-04T06:12:50.317536+0000","last_clean":"2024-03-04T06:12:50.317536+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:12:50.317536+0000","last_undegraded":"2024-03-04T06:12:50.317536+0000","last_fullsized":"2024-03-04T06:12:50.317536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"298'429","reported_seq":1192,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-03-04T06:12:50.323931+0000","last_change":"2024-03-04T06:12:48.255590+0000","last_active":"2024-03-04T06:12:50.323931+0000","last_peered":"2024-03-04T06:12:50.323931+0000","last_clean":"2024-03-04T06:12:50.323931+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:12:50.323931+0000","last_undegraded":"2024-03-04T06:12:50.323931+0000","last_fullsized":"2024-03-04T06:12:50.323931+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11076077,"num_objects":38,"num_object_clones":34,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":271,"num_read_kb":210178,"num_write":197,"num_write_kb":54173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":727,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-03-04T06:12:51.275581+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:12:51.275581+0000","last_peered":"2024-03-04T06:12:51.275581+0000","last_clean":"2024-03-04T06:12:51.275581+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:12:51.275581+0000","last_undegraded":"2024-03-04T06:12:51.275581+0000","last_fullsized":"2024-03-04T06:12:51.275581+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":114795209,"num_objects":419,"num_object_clones":308,"num_object_copies":1257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":76,"num_read":2278,"num_read_kb":1714880,"num_write":1996,"num_write_kb":547456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3910,"ondisk_log_size":3910,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":131564,"kb_used_data":92260,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94240276,"statfs":{"total":96636764160,"available":96502042624,"internally_reserved":0,"allocated":94474240,"data_stored":101913856,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":113836,"kb_used_data":78692,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94258004,"statfs":{"total":96636764160,"available":96520196096,"internally_reserved":0,"allocated":80580608,"data_stored":87079520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96304,"kb_used_data":61160,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94275536,"statfs":{"total":96636764160,"available":96538148864,"internally_reserved":0,"allocated":62627840,"data_stored":66171874,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99180,"kb_used_data":64036,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94272660,"statfs":{"total":96636764160,"available":96535203840,"internally_reserved":0,"allocated":65572864,"data_stored":68490577,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:12:55.270 INFO:tasks.rados.rados.0.smithi064.stdout:1257: done (0 left) 2024-03-04T06:12:55.270 INFO:tasks.rados.rados.0.smithi064.stdout:1258: read oid 22 snap -1 2024-03-04T06:12:55.270 INFO:tasks.rados.rados.0.smithi064.stdout:1258: expect (ObjNum 593 snap 142 seq_num 593) 2024-03-04T06:12:55.270 INFO:tasks.rados.rados.0.smithi064.stdout:1259: read oid 27 snap -1 2024-03-04T06:12:55.270 INFO:tasks.rados.rados.0.smithi064.stdout:1259: expect (ObjNum 602 snap 144 seq_num 602) 2024-03-04T06:12:55.270 INFO:tasks.rados.rados.0.smithi064.stdout:1260: write oid 34 current snap is 145 2024-03-04T06:12:55.270 INFO:tasks.rados.rados.0.smithi064.stdout:1260: seq_num 603 ranges {534621=567034,1636289=498332} 2024-03-04T06:12:55.276 INFO:tasks.rados.rados.0.smithi064.stdout:1260: writing smithi06431743-34 from 534621 to 1101655 tid 1 2024-03-04T06:12:55.282 INFO:tasks.rados.rados.0.smithi064.stdout:1260: writing smithi06431743-34 from 1636289 to 2134621 tid 2 2024-03-04T06:12:55.282 INFO:tasks.rados.rados.0.smithi064.stdout:1261: read oid 43 snap 133 2024-03-04T06:12:55.282 INFO:tasks.rados.rados.0.smithi064.stdout:1261: expect (ObjNum 540 snap 125 seq_num 540) 2024-03-04T06:12:55.282 INFO:tasks.rados.rados.0.smithi064.stdout:1262: read oid 7 snap -1 2024-03-04T06:12:55.282 INFO:tasks.rados.rados.0.smithi064.stdout:1262: expect (ObjNum 583 snap 140 seq_num 583) 2024-03-04T06:12:55.282 INFO:tasks.rados.rados.0.smithi064.stdout:1263: delete oid 24 current snap is 145 2024-03-04T06:12:55.320 INFO:tasks.rados.rados.0.smithi064.stdout:1261: expect (ObjNum 540 snap 125 seq_num 540) 2024-03-04T06:12:55.613 INFO:tasks.rados.rados.0.smithi064.stdout:1259: expect (ObjNum 602 snap 144 seq_num 602) 2024-03-04T06:12:55.709 INFO:tasks.rados.rados.0.smithi064.stdout:1258: expect (ObjNum 593 snap 142 seq_num 593) 2024-03-04T06:12:55.908 INFO:tasks.rados.rados.0.smithi064.stdout:1260: finishing write tid 1 to smithi06431743-34 2024-03-04T06:12:55.908 INFO:tasks.rados.rados.0.smithi064.stdout:1262: expect (ObjNum 583 snap 140 seq_num 583) 2024-03-04T06:12:56.049 INFO:tasks.rados.rados.0.smithi064.stdout:1260: finishing write tid 2 to smithi06431743-34 2024-03-04T06:12:56.049 INFO:tasks.rados.rados.0.smithi064.stdout:1260: finishing write tid 3 to smithi06431743-34 2024-03-04T06:12:56.049 INFO:tasks.rados.rados.0.smithi064.stdout:1260: finishing write tid 4 to smithi06431743-34 2024-03-04T06:12:56.049 INFO:tasks.rados.rados.0.smithi064.stdout:1260: oid 34 updating version 0 to 496 2024-03-04T06:12:56.049 INFO:tasks.rados.rados.0.smithi064.stdout:1260: oid 34 version 496 is already newer than 494 2024-03-04T06:12:56.049 INFO:tasks.rados.rados.0.smithi064.stdout:1260: oid 34 version 496 is already newer than 496 2024-03-04T06:12:56.050 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 496 (ObjNum 603 snap 145 seq_num 603) dirty exists 2024-03-04T06:12:56.050 INFO:tasks.rados.rados.0.smithi064.stdout:1260: left oid 34 (ObjNum 603 snap 145 seq_num 603) 2024-03-04T06:12:56.050 INFO:tasks.rados.rados.0.smithi064.stdout:1258: done (5 left) 2024-03-04T06:12:56.050 INFO:tasks.rados.rados.0.smithi064.stdout:1259: done (4 left) 2024-03-04T06:12:56.050 INFO:tasks.rados.rados.0.smithi064.stdout:1260: done (3 left) 2024-03-04T06:12:56.050 INFO:tasks.rados.rados.0.smithi064.stdout:1261: done (2 left) 2024-03-04T06:12:56.050 INFO:tasks.rados.rados.0.smithi064.stdout:1262: done (1 left) 2024-03-04T06:12:56.050 INFO:tasks.rados.rados.0.smithi064.stdout:1263: done (0 left) 2024-03-04T06:12:56.050 INFO:tasks.rados.rados.0.smithi064.stdout:1264: read oid 26 snap -1 2024-03-04T06:12:56.050 INFO:tasks.rados.rados.0.smithi064.stdout:1264: expect deleted 2024-03-04T06:12:56.050 INFO:tasks.rados.rados.0.smithi064.stdout:1265: snap_create 2024-03-04T06:12:56.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:12:56.272 INFO:tasks.rados.rados.0.smithi064.stdout:1264: done (1 left) 2024-03-04T06:12:56.272 INFO:tasks.rados.rados.0.smithi064.stdout:1265: done (0 left) 2024-03-04T06:12:56.272 INFO:tasks.rados.rados.0.smithi064.stdout:1266: write (excl) oid 40 current snap is 146 2024-03-04T06:12:56.272 INFO:tasks.rados.rados.0.smithi064.stdout:1266: seq_num 605 ranges {568235=717532,1753253=414982} 2024-03-04T06:12:56.278 INFO:tasks.rados.rados.0.smithi064.stdout:1266: writing smithi06431743-40 from 568235 to 1285767 tid 1 2024-03-04T06:12:56.284 INFO:tasks.rados.rados.0.smithi064.stdout:1266: writing smithi06431743-40 from 1753253 to 2168235 tid 2 2024-03-04T06:12:56.284 INFO:tasks.rados.rados.0.smithi064.stdout:1267: write (excl) oid 4 current snap is 146 2024-03-04T06:12:56.284 INFO:tasks.rados.rados.0.smithi064.stdout:1267: seq_num 606 ranges {585042=792781,2095382=89660} 2024-03-04T06:12:56.291 INFO:tasks.rados.rados.0.smithi064.stdout:1267: writing smithi06431743-4 from 585042 to 1377823 tid 1 2024-03-04T06:12:56.296 INFO:tasks.rados.rados.0.smithi064.stdout:1267: writing smithi06431743-4 from 2095382 to 2185042 tid 2 2024-03-04T06:12:56.296 INFO:tasks.rados.rados.0.smithi064.stdout:1268: write oid 30 current snap is 146 2024-03-04T06:12:56.296 INFO:tasks.rados.rados.0.smithi064.stdout:1268: seq_num 607 ranges {601849=468030,1753864=447985} 2024-03-04T06:12:56.302 INFO:tasks.rados.rados.0.smithi064.stdout:1268: writing smithi06431743-30 from 601849 to 1069879 tid 1 2024-03-04T06:12:56.309 INFO:tasks.rados.rados.0.smithi064.stdout:1268: writing smithi06431743-30 from 1753864 to 2201849 tid 2 2024-03-04T06:12:56.309 INFO:tasks.rados.rados.0.smithi064.stdout:1269: read oid 44 snap -1 2024-03-04T06:12:56.309 INFO:tasks.rados.rados.0.smithi064.stdout:1269: expect (ObjNum 533 snap 123 seq_num 533) 2024-03-04T06:12:56.309 INFO:tasks.rados.rados.0.smithi064.stdout:1270: write oid 41 current snap is 146 2024-03-04T06:12:56.310 INFO:tasks.rados.rados.0.smithi064.stdout:1270: seq_num 608 ranges {618656=543279,1812346=406310} 2024-03-04T06:12:56.317 INFO:tasks.rados.rados.0.smithi064.stdout:1270: writing smithi06431743-41 from 618656 to 1161935 tid 1 2024-03-04T06:12:56.324 INFO:tasks.rados.rados.0.smithi064.stdout:1270: writing smithi06431743-41 from 1812346 to 2218656 tid 2 2024-03-04T06:12:56.324 INFO:tasks.rados.rados.0.smithi064.stdout:1271: write oid 29 current snap is 146 2024-03-04T06:12:56.324 INFO:tasks.rados.rados.0.smithi064.stdout:1271: seq_num 609 ranges {635463=734881,1987181=248282} 2024-03-04T06:12:56.332 INFO:tasks.rados.rados.0.smithi064.stdout:1271: writing smithi06431743-29 from 635463 to 1370344 tid 1 2024-03-04T06:12:56.337 INFO:tasks.rados.rados.0.smithi064.stdout:1271: writing smithi06431743-29 from 1987181 to 2235463 tid 2 2024-03-04T06:12:56.337 INFO:tasks.rados.rados.0.smithi064.stdout:1272: write (excl) oid 1 current snap is 146 2024-03-04T06:12:56.338 INFO:tasks.rados.rados.0.smithi064.stdout:1272: seq_num 610 ranges {652270=410130,1529310=493867} 2024-03-04T06:12:56.343 INFO:tasks.rados.rados.0.smithi064.stdout:1272: writing smithi06431743-1 from 652270 to 1062400 tid 1 2024-03-04T06:12:56.350 INFO:tasks.rados.rados.0.smithi064.stdout:1272: writing smithi06431743-1 from 1529310 to 2023177 tid 2 2024-03-04T06:12:56.350 INFO:tasks.rados.rados.0.smithi064.stdout:1273: snap_remove snap 139 2024-03-04T06:12:56.356 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:56.353+0000 7fe41062e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:56.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:56.360+0000 7fe41062e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:57.277 INFO:tasks.rados.rados.0.smithi064.stdout:1273: done (7 left) 2024-03-04T06:12:57.277 INFO:tasks.rados.rados.0.smithi064.stdout:1274: write oid 26 current snap is 146 2024-03-04T06:12:57.277 INFO:tasks.rados.rados.0.smithi064.stdout:1274: seq_num 611 ranges {669077=485379,1587792=681285} 2024-03-04T06:12:57.283 INFO:tasks.rados.rados.0.smithi064.stdout:1274: writing smithi06431743-26 from 669077 to 1154456 tid 1 2024-03-04T06:12:57.290 INFO:tasks.rados.rados.0.smithi064.stdout:1274: writing smithi06431743-26 from 1587792 to 2269077 tid 2 2024-03-04T06:12:57.290 INFO:tasks.rados.rados.0.smithi064.stdout:1275: write (excl) oid 31 current snap is 146 2024-03-04T06:12:57.290 INFO:tasks.rados.rados.0.smithi064.stdout:1275: seq_num 612 ranges {685884=560628,2046274=239610} 2024-03-04T06:12:57.297 INFO:tasks.rados.rados.0.smithi064.stdout:1275: writing smithi06431743-31 from 685884 to 1246512 tid 1 2024-03-04T06:12:57.303 INFO:tasks.rados.rados.0.smithi064.stdout:1275: writing smithi06431743-31 from 2046274 to 2285884 tid 2 2024-03-04T06:12:57.303 INFO:tasks.rados.rados.0.smithi064.stdout:1276: read oid 39 snap -1 2024-03-04T06:12:57.303 INFO:tasks.rados.rados.0.smithi064.stdout:1276: expect (ObjNum 572 snap 137 seq_num 572) 2024-03-04T06:12:57.303 INFO:tasks.rados.rados.0.smithi064.stdout:1277: write oid 27 current snap is 146 2024-03-04T06:12:57.303 INFO:tasks.rados.rados.0.smithi064.stdout:1267: finishing write tid 1 to smithi06431743-4 2024-03-04T06:12:57.303 INFO:tasks.rados.rados.0.smithi064.stdout:1269: expect (ObjNum 533 snap 123 seq_num 533) 2024-03-04T06:12:57.422 INFO:tasks.rados.rados.0.smithi064.stdout:1267: finishing write tid 2 to smithi06431743-4 2024-03-04T06:12:57.422 INFO:tasks.rados.rados.0.smithi064.stdout:1266: finishing write tid 1 to smithi06431743-40 2024-03-04T06:12:57.422 INFO:tasks.rados.rados.0.smithi064.stdout:1267: finishing write tid 3 to smithi06431743-4 2024-03-04T06:12:57.422 INFO:tasks.rados.rados.0.smithi064.stdout:1267: finishing write tid 4 to smithi06431743-4 2024-03-04T06:12:57.422 INFO:tasks.rados.rados.0.smithi064.stdout:1267: oid 4 updating version 0 to 479 2024-03-04T06:12:57.422 INFO:tasks.rados.rados.0.smithi064.stdout:1267: oid 4 updating version 479 to 480 2024-03-04T06:12:57.422 INFO:tasks.rados.rados.0.smithi064.stdout:1267: oid 4 updating version 480 to 481 2024-03-04T06:12:57.422 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 481 (ObjNum 606 snap 146 seq_num 606) dirty exists 2024-03-04T06:12:57.423 INFO:tasks.rados.rados.0.smithi064.stdout:1267: left oid 4 (ObjNum 606 snap 146 seq_num 606) 2024-03-04T06:12:57.423 INFO:tasks.rados.rados.0.smithi064.stdout:1272: finishing write tid 1 to smithi06431743-1 2024-03-04T06:12:57.423 INFO:tasks.rados.rados.0.smithi064.stdout:1266: finishing write tid 2 to smithi06431743-40 2024-03-04T06:12:57.423 INFO:tasks.rados.rados.0.smithi064.stdout:1270: finishing write tid 1 to smithi06431743-41 2024-03-04T06:12:57.423 INFO:tasks.rados.rados.0.smithi064.stdout:1268: finishing write tid 1 to smithi06431743-30 2024-03-04T06:12:57.423 INFO:tasks.rados.rados.0.smithi064.stdout:1268: finishing write tid 2 to smithi06431743-30 2024-03-04T06:12:57.423 INFO:tasks.rados.rados.0.smithi064.stdout:1272: finishing write tid 2 to smithi06431743-1 2024-03-04T06:12:57.423 INFO:tasks.rados.rados.0.smithi064.stdout:1277: seq_num 613 ranges {702691=635877,2104756=197935} 2024-03-04T06:12:57.429 INFO:tasks.rados.rados.0.smithi064.stdout:1277: writing smithi06431743-27 from 702691 to 1338568 tid 1 2024-03-04T06:12:57.434 INFO:tasks.rados.rados.0.smithi064.stdout:1277: writing smithi06431743-27 from 2104756 to 2302691 tid 2 2024-03-04T06:12:57.435 INFO:tasks.rados.rados.0.smithi064.stdout:1267: done (10 left) 2024-03-04T06:12:57.435 INFO:tasks.rados.rados.0.smithi064.stdout:1269: done (9 left) 2024-03-04T06:12:57.435 INFO:tasks.rados.rados.0.smithi064.stdout:1278: snap_remove snap 133 2024-03-04T06:12:58.279 INFO:tasks.rados.rados.0.smithi064.stdout:1278: done (9 left) 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1279: delete oid 6 current snap is 146 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1268: finishing write tid 3 to smithi06431743-30 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1266: finishing write tid 3 to smithi06431743-40 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1272: finishing write tid 3 to smithi06431743-1 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1266: finishing write tid 4 to smithi06431743-40 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1266: oid 40 updating version 0 to 213 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1266: oid 40 version 213 is already newer than 212 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1266: oid 40 updating version 213 to 214 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 214 (ObjNum 605 snap 146 seq_num 605) dirty exists 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1266: left oid 40 (ObjNum 605 snap 146 seq_num 605) 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1270: finishing write tid 2 to smithi06431743-41 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1272: finishing write tid 4 to smithi06431743-1 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1272: oid 1 updating version 0 to 499 2024-03-04T06:12:58.280 INFO:tasks.rados.rados.0.smithi064.stdout:1272: oid 1 updating version 499 to 501 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1272: oid 1 version 501 is already newer than 501 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 501 (ObjNum 610 snap 146 seq_num 610) dirty exists 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1272: left oid 1 (ObjNum 610 snap 146 seq_num 610) 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1270: finishing write tid 3 to smithi06431743-41 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1270: finishing write tid 4 to smithi06431743-41 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1270: oid 41 updating version 0 to 247 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1270: oid 41 version 247 is already newer than 246 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1270: oid 41 updating version 247 to 248 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 248 (ObjNum 608 snap 146 seq_num 608) dirty exists 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1270: left oid 41 (ObjNum 608 snap 146 seq_num 608) 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1268: finishing write tid 4 to smithi06431743-30 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1268: oid 30 updating version 0 to 484 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1268: oid 30 version 484 is already newer than 484 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1268: oid 30 version 484 is already newer than 484 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 484 (ObjNum 607 snap 146 seq_num 607) dirty exists 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1268: left oid 30 (ObjNum 607 snap 146 seq_num 607) 2024-03-04T06:12:58.281 INFO:tasks.rados.rados.0.smithi064.stdout:1271: finishing write tid 1 to smithi06431743-29 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1271: finishing write tid 2 to smithi06431743-29 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1271: finishing write tid 3 to smithi06431743-29 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1271: finishing write tid 4 to smithi06431743-29 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1271: oid 29 updating version 0 to 488 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1271: oid 29 version 488 is already newer than 488 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1271: oid 29 version 488 is already newer than 488 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 488 (ObjNum 609 snap 146 seq_num 609) dirty exists 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1271: left oid 29 (ObjNum 609 snap 146 seq_num 609) 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1275: finishing write tid 1 to smithi06431743-31 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1274: finishing write tid 1 to smithi06431743-26 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1275: finishing write tid 2 to smithi06431743-31 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1275: finishing write tid 3 to smithi06431743-31 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1275: finishing write tid 4 to smithi06431743-31 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1275: oid 31 updating version 0 to 95 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1275: oid 31 version 95 is already newer than 94 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:1275: oid 31 version 95 is already newer than 93 2024-03-04T06:12:58.282 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 95 (ObjNum 612 snap 146 seq_num 612) dirty exists 2024-03-04T06:12:58.283 INFO:tasks.rados.rados.0.smithi064.stdout:1275: left oid 31 (ObjNum 612 snap 146 seq_num 612) 2024-03-04T06:12:58.283 INFO:tasks.rados.rados.0.smithi064.stdout:1274: finishing write tid 2 to smithi06431743-26 2024-03-04T06:12:58.283 INFO:tasks.rados.rados.0.smithi064.stdout:1276: expect (ObjNum 572 snap 137 seq_num 572) 2024-03-04T06:12:58.444 INFO:tasks.rados.rados.0.smithi064.stdout:1274: finishing write tid 3 to smithi06431743-26 2024-03-04T06:12:58.444 INFO:tasks.rados.rados.0.smithi064.stdout:1274: finishing write tid 4 to smithi06431743-26 2024-03-04T06:12:58.444 INFO:tasks.rados.rados.0.smithi064.stdout:1274: oid 26 updating version 0 to 508 2024-03-04T06:12:58.444 INFO:tasks.rados.rados.0.smithi064.stdout:1274: oid 26 version 508 is already newer than 506 2024-03-04T06:12:58.444 INFO:tasks.rados.rados.0.smithi064.stdout:1274: oid 26 version 508 is already newer than 507 2024-03-04T06:12:58.444 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 508 (ObjNum 611 snap 146 seq_num 611) dirty exists 2024-03-04T06:12:58.444 INFO:tasks.rados.rados.0.smithi064.stdout:1274: left oid 26 (ObjNum 611 snap 146 seq_num 611) 2024-03-04T06:12:58.444 INFO:tasks.rados.rados.0.smithi064.stdout:1277: finishing write tid 1 to smithi06431743-27 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1277: finishing write tid 2 to smithi06431743-27 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1266: done (9 left) 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1268: done (8 left) 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1270: done (7 left) 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1271: done (6 left) 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1272: done (5 left) 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1274: done (4 left) 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1275: done (3 left) 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1276: done (2 left) 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1279: done (1 left) 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1280: snap_create 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1277: finishing write tid 3 to smithi06431743-27 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1277: finishing write tid 4 to smithi06431743-27 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1277: oid 27 updating version 0 to 175 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1277: oid 27 updating version 175 to 176 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:1277: oid 27 version 176 is already newer than 174 2024-03-04T06:12:58.445 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 176 (ObjNum 613 snap 146 seq_num 613) dirty exists 2024-03-04T06:12:58.446 INFO:tasks.rados.rados.0.smithi064.stdout:1277: left oid 27 (ObjNum 613 snap 146 seq_num 613) 2024-03-04T06:12:58.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:12:58.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:58.569+0000 7fe10ce2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:58.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:12:58.573+0000 7fe10ce2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:12:59.292 INFO:tasks.rados.rados.0.smithi064.stdout:1277: done (1 left) 2024-03-04T06:12:59.292 INFO:tasks.rados.rados.0.smithi064.stdout:1280: done (0 left) 2024-03-04T06:12:59.292 INFO:tasks.rados.rados.0.smithi064.stdout:1281: snap_remove snap 146 2024-03-04T06:13:00.222 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:13:00.222 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:13:00.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:13:00.293 INFO:tasks.rados.rados.0.smithi064.stdout:1281: done (0 left) 2024-03-04T06:13:00.294 INFO:tasks.rados.rados.0.smithi064.stdout:1282: write (excl) oid 43 current snap is 147 2024-03-04T06:13:00.294 INFO:tasks.rados.rados.0.smithi064.stdout:1282: seq_num 615 ranges {736305=786375,2105367=230938} 2024-03-04T06:13:00.301 INFO:tasks.rados.rados.0.smithi064.stdout:1282: writing smithi06431743-43 from 736305 to 1522680 tid 1 2024-03-04T06:13:00.306 INFO:tasks.rados.rados.0.smithi064.stdout:1282: writing smithi06431743-43 from 2105367 to 2336305 tid 2 2024-03-04T06:13:00.306 INFO:tasks.rados.rados.0.smithi064.stdout:1283: snap_remove snap 140 2024-03-04T06:13:00.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:00.348+0000 7f201db1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:00.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:00.356+0000 7f201db1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:00.556 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:13:00.556 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:13:00.568 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-03-04T06:13:00.291224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114276546,"num_objects":422,"num_object_clones":309,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":75,"num_read":2416,"num_read_kb":1754491,"num_write":2144,"num_write_kb":551619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4059,"ondisk_log_size":4059,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":448576,"kb_used_data":303312,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377038784,"statfs":{"total":386547056640,"available":386087714816,"internally_reserved":0,"allocated":310591488,"data_stored":332975137,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2134621,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":19,"num_read_kb":25516,"num_write":4,"num_write_kb":1041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019906"},"pg_stats":[{"pgid":"3.f","version":"304'210","reported_seq":1023,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.317442+0000","last_change":"2024-03-04T06:12:55.309481+0000","last_active":"2024-03-04T06:12:55.317442+0000","last_peered":"2024-03-04T06:12:55.317442+0000","last_clean":"2024-03-04T06:12:55.317442+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:12:55.317442+0000","last_undegraded":"2024-03-04T06:12:55.317442+0000","last_fullsized":"2024-03-04T06:12:55.317442+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8319762,"num_objects":29,"num_object_clones":20,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":97,"num_read_kb":63791,"num_write":122,"num_write_kb":33016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"303'170","reported_seq":1011,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.296663+0000","last_change":"2024-03-04T06:12:55.296663+0000","last_active":"2024-03-04T06:12:55.296663+0000","last_peered":"2024-03-04T06:12:55.296663+0000","last_clean":"2024-03-04T06:12:55.296663+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:12:55.296663+0000","last_undegraded":"2024-03-04T06:12:55.296663+0000","last_fullsized":"2024-03-04T06:12:55.296663+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2117814,"num_objects":19,"num_object_clones":13,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":74,"num_read_kb":48396,"num_write":77,"num_write_kb":19776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"304'283","reported_seq":1076,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-03-04T06:12:56.348508+0000","last_change":"2024-03-04T06:12:56.353617+0000","last_active":"2024-03-04T06:12:56.348508+0000","last_peered":"2024-03-04T06:12:56.348508+0000","last_clean":"2024-03-04T06:12:56.348508+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:12:56.348508+0000","last_undegraded":"2024-03-04T06:12:56.348508+0000","last_fullsized":"2024-03-04T06:12:56.348508+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13177084,"num_objects":33,"num_object_clones":25,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":162,"num_read_kb":112550,"num_write":156,"num_write_kb":43443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"304'201","reported_seq":1036,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.293226+0000","last_change":"2024-03-04T06:12:55.301248+0000","last_active":"2024-03-04T06:12:55.293226+0000","last_peered":"2024-03-04T06:12:55.293226+0000","last_clean":"2024-03-04T06:12:55.293226+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:12:55.293226+0000","last_undegraded":"2024-03-04T06:12:55.293226+0000","last_fullsized":"2024-03-04T06:12:55.293226+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983655,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":125,"num_read_kb":96803,"num_write":119,"num_write_kb":34146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"249'92","reported_seq":929,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-03-04T06:12:56.355045+0000","last_change":"2024-03-04T06:12:55.317492+0000","last_active":"2024-03-04T06:12:56.355045+0000","last_peered":"2024-03-04T06:12:56.355045+0000","last_clean":"2024-03-04T06:12:56.355045+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:12:56.355045+0000","last_undegraded":"2024-03-04T06:12:56.355045+0000","last_fullsized":"2024-03-04T06:12:56.355045+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":31,"num_read_kb":17041,"num_write":34,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"296'208","reported_seq":1033,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.304938+0000","last_change":"2024-03-04T06:12:55.309386+0000","last_active":"2024-03-04T06:12:55.304938+0000","last_peered":"2024-03-04T06:12:55.304938+0000","last_clean":"2024-03-04T06:12:55.304938+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:12:55.304938+0000","last_undegraded":"2024-03-04T06:12:55.304938+0000","last_fullsized":"2024-03-04T06:12:55.304938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10067690,"num_objects":23,"num_object_clones":16,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":120,"num_read_kb":87291,"num_write":122,"num_write_kb":36476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":872,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.302089+0000","last_change":"2024-03-04T06:12:55.308874+0000","last_active":"2024-03-04T06:12:55.302089+0000","last_peered":"2024-03-04T06:12:55.302089+0000","last_clean":"2024-03-04T06:12:55.302089+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:12:55.302089+0000","last_undegraded":"2024-03-04T06:12:55.302089+0000","last_fullsized":"2024-03-04T06:12:55.302089+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"293'161","reported_seq":996,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-03-04T06:12:56.339121+0000","last_change":"2024-03-04T06:12:56.348694+0000","last_active":"2024-03-04T06:12:56.339121+0000","last_peered":"2024-03-04T06:12:56.339121+0000","last_clean":"2024-03-04T06:12:56.339121+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:12:56.339121+0000","last_undegraded":"2024-03-04T06:12:56.339121+0000","last_fullsized":"2024-03-04T06:12:56.339121+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5983490,"num_objects":18,"num_object_clones":11,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":124,"num_read_kb":117183,"num_write":78,"num_write_kb":21237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":885,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-03-04T06:12:56.346819+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:12:56.346819+0000","last_peered":"2024-03-04T06:12:56.346819+0000","last_clean":"2024-03-04T06:12:56.346819+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:12:56.346819+0000","last_undegraded":"2024-03-04T06:12:56.346819+0000","last_fullsized":"2024-03-04T06:12:56.346819+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":736,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-03-04T06:12:56.356290+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:12:56.356290+0000","last_peered":"2024-03-04T06:12:56.356290+0000","last_clean":"2024-03-04T06:12:56.356290+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:12:56.356290+0000","last_undegraded":"2024-03-04T06:12:56.356290+0000","last_fullsized":"2024-03-04T06:12:56.356290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"304'245","reported_seq":1039,"reported_epoch":306,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:56.353308+0000","last_change":"2024-03-04T06:12:54.415043+0000","last_active":"2024-03-04T06:12:56.353308+0000","last_peered":"2024-03-04T06:12:56.353308+0000","last_clean":"2024-03-04T06:12:56.353308+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:12:56.353308+0000","last_undegraded":"2024-03-04T06:12:56.353308+0000","last_fullsized":"2024-03-04T06:12:56.353308+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6689384,"num_objects":26,"num_object_clones":19,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":143,"num_read_kb":109564,"num_write":120,"num_write_kb":30600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":739,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.307641+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:12:55.307641+0000","last_peered":"2024-03-04T06:12:55.307641+0000","last_clean":"2024-03-04T06:12:55.307641+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:12:55.307641+0000","last_undegraded":"2024-03-04T06:12:55.307641+0000","last_fullsized":"2024-03-04T06:12:55.307641+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"304'229","reported_seq":1021,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-03-04T06:12:56.348601+0000","last_change":"2024-03-04T06:12:55.307885+0000","last_active":"2024-03-04T06:12:56.348601+0000","last_peered":"2024-03-04T06:12:56.348601+0000","last_clean":"2024-03-04T06:12:56.348601+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:12:56.348601+0000","last_undegraded":"2024-03-04T06:12:56.348601+0000","last_fullsized":"2024-03-04T06:12:56.348601+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899488,"num_objects":23,"num_object_clones":18,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":116,"num_read_kb":95340,"num_write":96,"num_write_kb":28103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":736,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-03-04T06:12:56.340383+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:12:56.340383+0000","last_peered":"2024-03-04T06:12:56.340383+0000","last_clean":"2024-03-04T06:12:56.340383+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:12:56.340383+0000","last_undegraded":"2024-03-04T06:12:56.340383+0000","last_fullsized":"2024-03-04T06:12:56.340383+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"305'498","reported_seq":1272,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-03-04T06:12:56.361441+0000","last_change":"2024-03-04T06:12:55.319791+0000","last_active":"2024-03-04T06:12:56.361441+0000","last_peered":"2024-03-04T06:12:56.361441+0000","last_clean":"2024-03-04T06:12:56.361441+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:12:56.361441+0000","last_undegraded":"2024-03-04T06:12:56.361441+0000","last_fullsized":"2024-03-04T06:12:56.361441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6202179,"num_objects":51,"num_object_clones":39,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":336,"num_read_kb":281958,"num_write":263,"num_write_kb":71911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":739,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.296355+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:12:55.296355+0000","last_peered":"2024-03-04T06:12:55.296355+0000","last_clean":"2024-03-04T06:12:55.296355+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:12:55.296355+0000","last_undegraded":"2024-03-04T06:12:55.296355+0000","last_fullsized":"2024-03-04T06:12:55.296355+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"299'477","reported_seq":1206,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.289062+0000","last_change":"2024-03-04T06:12:55.295956+0000","last_active":"2024-03-04T06:12:55.289062+0000","last_peered":"2024-03-04T06:12:55.289062+0000","last_clean":"2024-03-04T06:12:55.289062+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:12:55.289062+0000","last_undegraded":"2024-03-04T06:12:55.289062+0000","last_fullsized":"2024-03-04T06:12:55.289062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13109790,"num_objects":47,"num_object_clones":34,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":213,"num_read_kb":155101,"num_write":227,"num_write_kb":60703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"304'199","reported_seq":1053,"reported_epoch":306,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:12:56.353563+0000","last_change":"2024-03-04T06:12:54.422522+0000","last_active":"2024-03-04T06:12:56.353563+0000","last_peered":"2024-03-04T06:12:56.353563+0000","last_clean":"2024-03-04T06:12:56.353563+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:12:56.353563+0000","last_undegraded":"2024-03-04T06:12:56.353563+0000","last_fullsized":"2024-03-04T06:12:56.353563+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680997,"num_objects":22,"num_object_clones":16,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":193,"num_read_kb":157563,"num_write":108,"num_write_kb":28860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":779,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.293399+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:12:55.293399+0000","last_peered":"2024-03-04T06:12:55.293399+0000","last_clean":"2024-03-04T06:12:55.293399+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:12:55.293399+0000","last_undegraded":"2024-03-04T06:12:55.293399+0000","last_fullsized":"2024-03-04T06:12:55.293399+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"303'219","reported_seq":1065,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.295612+0000","last_change":"2024-03-04T06:12:55.300165+0000","last_active":"2024-03-04T06:12:55.295612+0000","last_peered":"2024-03-04T06:12:55.295612+0000","last_clean":"2024-03-04T06:12:55.295612+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:12:55.295612+0000","last_undegraded":"2024-03-04T06:12:55.295612+0000","last_fullsized":"2024-03-04T06:12:55.295612+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6538121,"num_objects":28,"num_object_clones":18,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":118,"num_read_kb":67272,"num_write":116,"num_write_kb":31360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":752,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-03-04T06:12:56.358843+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:12:56.358843+0000","last_peered":"2024-03-04T06:12:56.358843+0000","last_clean":"2024-03-04T06:12:56.358843+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:12:56.358843+0000","last_undegraded":"2024-03-04T06:12:56.358843+0000","last_fullsized":"2024-03-04T06:12:56.358843+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"305'339","reported_seq":1120,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-03-04T06:12:56.354428+0000","last_change":"2024-03-04T06:12:55.308088+0000","last_active":"2024-03-04T06:12:56.354428+0000","last_peered":"2024-03-04T06:12:56.354428+0000","last_clean":"2024-03-04T06:12:56.354428+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:12:56.354428+0000","last_undegraded":"2024-03-04T06:12:56.354428+0000","last_fullsized":"2024-03-04T06:12:56.354428+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840647,"num_objects":34,"num_object_clones":26,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":186,"num_read_kb":130495,"num_write":172,"num_write_kb":47172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":779,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.288011+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:12:55.288011+0000","last_peered":"2024-03-04T06:12:55.288011+0000","last_clean":"2024-03-04T06:12:55.288011+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:12:55.288011+0000","last_undegraded":"2024-03-04T06:12:55.288011+0000","last_fullsized":"2024-03-04T06:12:55.288011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"303'431","reported_seq":1212,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-03-04T06:12:55.295622+0000","last_change":"2024-03-04T06:12:55.295622+0000","last_active":"2024-03-04T06:12:55.295622+0000","last_peered":"2024-03-04T06:12:55.295622+0000","last_clean":"2024-03-04T06:12:55.295622+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:12:55.295622+0000","last_undegraded":"2024-03-04T06:12:55.295622+0000","last_fullsized":"2024-03-04T06:12:55.295622+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11076077,"num_objects":38,"num_object_clones":34,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":275,"num_read_kb":214052,"num_write":197,"num_write_kb":54173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":736,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-03-04T06:12:56.351618+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:12:56.351618+0000","last_peered":"2024-03-04T06:12:56.351618+0000","last_clean":"2024-03-04T06:12:56.351618+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:12:56.351618+0000","last_undegraded":"2024-03-04T06:12:56.351618+0000","last_fullsized":"2024-03-04T06:12:56.351618+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":113686178,"num_objects":420,"num_object_clones":309,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":75,"num_read":2313,"num_read_kb":1754400,"num_write":2007,"num_write_kb":549725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3962,"ondisk_log_size":3962,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":133668,"kb_used_data":94148,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94238172,"statfs":{"total":96636764160,"available":96499888128,"internally_reserved":0,"allocated":96407552,"data_stored":104125075,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":115932,"kb_used_data":80692,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94255908,"statfs":{"total":96636764160,"available":96518049792,"internally_reserved":0,"allocated":82628608,"data_stored":89340426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":98048,"kb_used_data":62792,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94273792,"statfs":{"total":96636764160,"available":96536363008,"internally_reserved":0,"allocated":64299008,"data_stored":68422221,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100928,"kb_used_data":65680,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94270912,"statfs":{"total":96636764160,"available":96533413888,"internally_reserved":0,"allocated":67256320,"data_stored":71087415,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:13:00.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:00.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:00.745+0000 7f0dd5a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:00.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:00.749+0000 7f0dd5a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:01.304 INFO:tasks.rados.rados.0.smithi064.stdout:1283: done (1 left) 2024-03-04T06:13:01.304 INFO:tasks.rados.rados.0.smithi064.stdout:1284: delete oid 28 current snap is 147 2024-03-04T06:13:01.304 INFO:tasks.rados.rados.0.smithi064.stdout:1282: finishing write tid 1 to smithi06431743-43 2024-03-04T06:13:01.304 INFO:tasks.rados.rados.0.smithi064.stdout:1282: finishing write tid 2 to smithi06431743-43 2024-03-04T06:13:01.304 INFO:tasks.rados.rados.0.smithi064.stdout:1282: finishing write tid 3 to smithi06431743-43 2024-03-04T06:13:01.304 INFO:tasks.rados.rados.0.smithi064.stdout:1282: finishing write tid 4 to smithi06431743-43 2024-03-04T06:13:01.304 INFO:tasks.rados.rados.0.smithi064.stdout:1282: oid 43 updating version 0 to 222 2024-03-04T06:13:01.304 INFO:tasks.rados.rados.0.smithi064.stdout:1282: oid 43 version 222 is already newer than 220 2024-03-04T06:13:01.304 INFO:tasks.rados.rados.0.smithi064.stdout:1282: oid 43 version 222 is already newer than 222 2024-03-04T06:13:01.304 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 222 (ObjNum 615 snap 147 seq_num 615) dirty exists 2024-03-04T06:13:01.305 INFO:tasks.rados.rados.0.smithi064.stdout:1282: left oid 43 (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:13:01.383 INFO:tasks.rados.rados.0.smithi064.stdout:1282: done (1 left) 2024-03-04T06:13:01.383 INFO:tasks.rados.rados.0.smithi064.stdout:1284: done (0 left) 2024-03-04T06:13:01.383 INFO:tasks.rados.rados.0.smithi064.stdout:1285: delete oid 45 current snap is 147 2024-03-04T06:13:01.456 INFO:tasks.rados.rados.0.smithi064.stdout:1285: done (0 left) 2024-03-04T06:13:01.456 INFO:tasks.rados.rados.0.smithi064.stdout:1286: read oid 27 snap -1 2024-03-04T06:13:01.457 INFO:tasks.rados.rados.0.smithi064.stdout:1286: expect (ObjNum 613 snap 146 seq_num 613) 2024-03-04T06:13:01.457 INFO:tasks.rados.rados.0.smithi064.stdout:1287: write (excl) oid 30 current snap is 147 2024-03-04T06:13:01.457 INFO:tasks.rados.rados.0.smithi064.stdout:1287: seq_num 618 ranges {786726=728475,2280813=105913} 2024-03-04T06:13:01.464 INFO:tasks.rados.rados.0.smithi064.stdout:1287: writing smithi06431743-30 from 786726 to 1515201 tid 1 2024-03-04T06:13:01.469 INFO:tasks.rados.rados.0.smithi064.stdout:1287: writing smithi06431743-30 from 2280813 to 2386726 tid 2 2024-03-04T06:13:01.469 INFO:tasks.rados.rados.0.smithi064.stdout:1288: delete oid 18 current snap is 147 2024-03-04T06:13:01.537 INFO:tasks.rados.rados.0.smithi064.stdout:1287: finishing write tid 1 to smithi06431743-30 2024-03-04T06:13:01.543 INFO:tasks.rados.rados.0.smithi064.stdout:1286: expect (ObjNum 613 snap 146 seq_num 613) 2024-03-04T06:13:01.659 INFO:tasks.rados.rados.0.smithi064.stdout:1287: finishing write tid 2 to smithi06431743-30 2024-03-04T06:13:01.660 INFO:tasks.rados.rados.0.smithi064.stdout:1287: finishing write tid 3 to smithi06431743-30 2024-03-04T06:13:01.660 INFO:tasks.rados.rados.0.smithi064.stdout:1287: finishing write tid 4 to smithi06431743-30 2024-03-04T06:13:01.660 INFO:tasks.rados.rados.0.smithi064.stdout:1287: oid 30 updating version 0 to 501 2024-03-04T06:13:01.660 INFO:tasks.rados.rados.0.smithi064.stdout:1287: oid 30 updating version 501 to 502 2024-03-04T06:13:01.660 INFO:tasks.rados.rados.0.smithi064.stdout:1287: oid 30 updating version 502 to 503 2024-03-04T06:13:01.660 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 503 (ObjNum 618 snap 147 seq_num 618) dirty exists 2024-03-04T06:13:01.660 INFO:tasks.rados.rados.0.smithi064.stdout:1287: left oid 30 (ObjNum 618 snap 147 seq_num 618) 2024-03-04T06:13:01.660 INFO:tasks.rados.rados.0.smithi064.stdout:1286: done (2 left) 2024-03-04T06:13:01.660 INFO:tasks.rados.rados.0.smithi064.stdout:1287: done (1 left) 2024-03-04T06:13:01.660 INFO:tasks.rados.rados.0.smithi064.stdout:1288: done (0 left) 2024-03-04T06:13:01.660 INFO:tasks.rados.rados.0.smithi064.stdout:1289: snap_remove snap 145 2024-03-04T06:13:02.303 INFO:tasks.rados.rados.0.smithi064.stdout:1289: done (0 left) 2024-03-04T06:13:02.303 INFO:tasks.rados.rados.0.smithi064.stdout:1290: write (excl) oid 31 current snap is 147 2024-03-04T06:13:02.303 INFO:tasks.rados.rados.0.smithi064.stdout:1290: seq_num 620 ranges {420340=478973,1597777=512212} 2024-03-04T06:13:02.308 INFO:tasks.rados.rados.0.smithi064.stdout:1290: writing smithi06431743-31 from 420340 to 899313 tid 1 2024-03-04T06:13:02.315 INFO:tasks.rados.rados.0.smithi064.stdout:1290: writing smithi06431743-31 from 1597777 to 2109989 tid 2 2024-03-04T06:13:02.315 INFO:tasks.rados.rados.0.smithi064.stdout:1291: snap_create 2024-03-04T06:13:02.399 INFO:tasks.rados.rados.0.smithi064.stdout:1290: finishing write tid 1 to smithi06431743-31 2024-03-04T06:13:02.412 INFO:tasks.rados.rados.0.smithi064.stdout:1290: finishing write tid 2 to smithi06431743-31 2024-03-04T06:13:02.415 INFO:tasks.rados.rados.0.smithi064.stdout:1290: finishing write tid 3 to smithi06431743-31 2024-03-04T06:13:02.416 INFO:tasks.rados.rados.0.smithi064.stdout:1290: finishing write tid 4 to smithi06431743-31 2024-03-04T06:13:02.417 INFO:tasks.rados.rados.0.smithi064.stdout:1290: oid 31 updating version 0 to 96 2024-03-04T06:13:02.417 INFO:tasks.rados.rados.0.smithi064.stdout:1290: oid 31 updating version 96 to 98 2024-03-04T06:13:02.417 INFO:tasks.rados.rados.0.smithi064.stdout:1290: oid 31 version 98 is already newer than 97 2024-03-04T06:13:02.417 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 98 (ObjNum 620 snap 147 seq_num 620) dirty exists 2024-03-04T06:13:02.417 INFO:tasks.rados.rados.0.smithi064.stdout:1290: left oid 31 (ObjNum 620 snap 147 seq_num 620) 2024-03-04T06:13:02.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:02.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:02.922+0000 7f6cb9e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:02.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:02.927+0000 7f6cb9e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:03.310 INFO:tasks.rados.rados.0.smithi064.stdout:1290: done (1 left) 2024-03-04T06:13:03.310 INFO:tasks.rados.rados.0.smithi064.stdout:1291: done (0 left) 2024-03-04T06:13:03.310 INFO:tasks.rados.rados.0.smithi064.stdout:1292: delete oid 40 current snap is 148 2024-03-04T06:13:03.352 INFO:tasks.rados.rados.0.smithi064.stdout:1292: done (0 left) 2024-03-04T06:13:03.352 INFO:tasks.rados.rados.0.smithi064.stdout:1293: write (excl) oid 27 current snap is 148 2024-03-04T06:13:03.352 INFO:tasks.rados.rados.0.smithi064.stdout:1293: seq_num 622 ranges {453954=629471,1598388=515881} 2024-03-04T06:13:03.355 INFO:tasks.rados.rados.0.smithi064.stdout:1293: writing smithi06431743-27 from 453954 to 1083425 tid 1 2024-03-04T06:13:03.361 INFO:tasks.rados.rados.0.smithi064.stdout:1293: writing smithi06431743-27 from 1598388 to 2114269 tid 2 2024-03-04T06:13:03.361 INFO:tasks.rados.rados.0.smithi064.stdout:1294: delete oid 49 current snap is 148 2024-03-04T06:13:03.411 INFO:tasks.rados.rados.0.smithi064.stdout:1294: done (1 left) 2024-03-04T06:13:03.411 INFO:tasks.rados.rados.0.smithi064.stdout:1295: delete oid 21 current snap is 148 2024-03-04T06:13:03.414 INFO:tasks.rados.rados.0.smithi064.stdout:1293: finishing write tid 1 to smithi06431743-27 2024-03-04T06:13:03.434 INFO:tasks.rados.rados.0.smithi064.stdout:1295: done (1 left) 2024-03-04T06:13:03.434 INFO:tasks.rados.rados.0.smithi064.stdout:1296: snap_create 2024-03-04T06:13:03.441 INFO:tasks.rados.rados.0.smithi064.stdout:1293: finishing write tid 2 to smithi06431743-27 2024-03-04T06:13:03.447 INFO:tasks.rados.rados.0.smithi064.stdout:1293: finishing write tid 3 to smithi06431743-27 2024-03-04T06:13:03.449 INFO:tasks.rados.rados.0.smithi064.stdout:1293: finishing write tid 4 to smithi06431743-27 2024-03-04T06:13:03.449 INFO:tasks.rados.rados.0.smithi064.stdout:1293: oid 27 updating version 0 to 188 2024-03-04T06:13:03.449 INFO:tasks.rados.rados.0.smithi064.stdout:1293: oid 27 version 188 is already newer than 187 2024-03-04T06:13:03.449 INFO:tasks.rados.rados.0.smithi064.stdout:1293: oid 27 updating version 188 to 189 2024-03-04T06:13:03.449 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 189 (ObjNum 622 snap 148 seq_num 622) dirty exists 2024-03-04T06:13:03.449 INFO:tasks.rados.rados.0.smithi064.stdout:1293: left oid 27 (ObjNum 622 snap 148 seq_num 622) 2024-03-04T06:13:04.314 INFO:tasks.rados.rados.0.smithi064.stdout:1293: done (1 left) 2024-03-04T06:13:04.314 INFO:tasks.rados.rados.0.smithi064.stdout:1296: done (0 left) 2024-03-04T06:13:04.314 INFO:tasks.rados.rados.0.smithi064.stdout:1297: snap_create 2024-03-04T06:13:04.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:05.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:05.114+0000 7f73a5617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:05.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:05.119+0000 7f73a5617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:05.318 INFO:tasks.rados.rados.0.smithi064.stdout:1297: done (0 left) 2024-03-04T06:13:05.318 INFO:tasks.rados.rados.0.smithi064.stdout:1298: read oid 20 snap -1 2024-03-04T06:13:05.318 INFO:tasks.rados.rados.0.smithi064.stdout:1298: expect (ObjNum 557 snap 132 seq_num 557) 2024-03-04T06:13:05.318 INFO:tasks.rados.rados.0.smithi064.stdout:1299: snap_remove snap 148 2024-03-04T06:13:05.570 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:13:05.570 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:13:05.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:13:05.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:05.674+0000 7ff37d19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:05.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:05.680+0000 7ff37d19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:13:05.873 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:13:05.887 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-03-04T06:13:05.316398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117638507,"num_objects":431,"num_object_clones":315,"num_object_copies":1293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":75,"num_read":2446,"num_read_kb":1780561,"num_write":2182,"num_write_kb":559996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4166,"ondisk_log_size":4166,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":428216,"kb_used_data":282944,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377059144,"statfs":{"total":386547056640,"available":386108563456,"internally_reserved":0,"allocated":289734656,"data_stored":301686520,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3260855,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":7083,"num_write":18,"num_write_kb":3735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015395"},"pg_stats":[{"pgid":"3.f","version":"308'218","reported_seq":1039,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-03-04T06:12:58.347524+0000","last_change":"2024-03-04T06:12:58.347524+0000","last_active":"2024-03-04T06:12:58.347524+0000","last_peered":"2024-03-04T06:12:58.347524+0000","last_clean":"2024-03-04T06:12:58.347524+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:12:58.347524+0000","last_undegraded":"2024-03-04T06:12:58.347524+0000","last_fullsized":"2024-03-04T06:12:58.347524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":218,"log_dups_size":0,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8403797,"num_objects":30,"num_object_clones":21,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":99,"num_read_kb":63792,"num_write":126,"num_write_kb":34123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"309'180","reported_seq":1029,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-04T06:12:59.305884+0000","last_change":"2024-03-04T06:12:59.368958+0000","last_active":"2024-03-04T06:12:59.305884+0000","last_peered":"2024-03-04T06:12:59.305884+0000","last_clean":"2024-03-04T06:12:59.305884+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:12:59.305884+0000","last_undegraded":"2024-03-04T06:12:59.305884+0000","last_fullsized":"2024-03-04T06:12:59.305884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2302691,"num_objects":20,"num_object_clones":14,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":77,"num_read_kb":52535,"num_write":81,"num_write_kb":20591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"308'287","reported_seq":1088,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T06:12:58.308692+0000","last_change":"2024-03-04T06:12:58.387252+0000","last_active":"2024-03-04T06:12:58.308692+0000","last_peered":"2024-03-04T06:12:58.308692+0000","last_clean":"2024-03-04T06:12:58.308692+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:12:58.308692+0000","last_undegraded":"2024-03-04T06:12:58.308692+0000","last_fullsized":"2024-03-04T06:12:58.308692+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13177084,"num_objects":33,"num_object_clones":25,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":168,"num_read_kb":119629,"num_write":156,"num_write_kb":43443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"308'203","reported_seq":1052,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-03-04T06:13:00.329587+0000","last_change":"2024-03-04T06:13:00.334453+0000","last_active":"2024-03-04T06:13:00.329587+0000","last_peered":"2024-03-04T06:13:00.329587+0000","last_clean":"2024-03-04T06:13:00.329587+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:13:00.329587+0000","last_undegraded":"2024-03-04T06:13:00.329587+0000","last_fullsized":"2024-03-04T06:13:00.329587+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983655,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":125,"num_read_kb":96803,"num_write":119,"num_write_kb":34146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"307'95","reported_seq":948,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-04T06:13:01.336520+0000","last_change":"2024-03-04T06:13:01.340362+0000","last_active":"2024-03-04T06:13:01.336520+0000","last_peered":"2024-03-04T06:13:01.336520+0000","last_clean":"2024-03-04T06:13:01.336520+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:13:01.336520+0000","last_undegraded":"2024-03-04T06:13:01.336520+0000","last_fullsized":"2024-03-04T06:13:01.336520+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2285884,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":33,"num_read_kb":17042,"num_write":38,"num_write_kb":9531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"296'208","reported_seq":1048,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-03-04T06:12:58.308784+0000","last_change":"2024-03-04T06:12:58.308784+0000","last_active":"2024-03-04T06:12:58.308784+0000","last_peered":"2024-03-04T06:12:58.308784+0000","last_clean":"2024-03-04T06:12:58.308784+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:12:58.308784+0000","last_undegraded":"2024-03-04T06:12:58.308784+0000","last_fullsized":"2024-03-04T06:12:58.308784+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10067690,"num_objects":23,"num_object_clones":16,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":126,"num_read_kb":94387,"num_write":122,"num_write_kb":36476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":887,"reported_epoch":310,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:13:00.336989+0000","last_change":"2024-03-04T06:13:00.336989+0000","last_active":"2024-03-04T06:13:00.336989+0000","last_peered":"2024-03-04T06:13:00.336989+0000","last_clean":"2024-03-04T06:13:00.336989+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:13:00.336989+0000","last_undegraded":"2024-03-04T06:13:00.336989+0000","last_fullsized":"2024-03-04T06:13:00.336989+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"293'161","reported_seq":1004,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T06:12:58.310455+0000","last_change":"2024-03-04T06:12:58.319933+0000","last_active":"2024-03-04T06:12:58.310455+0000","last_peered":"2024-03-04T06:12:58.310455+0000","last_clean":"2024-03-04T06:12:58.310455+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:12:58.310455+0000","last_undegraded":"2024-03-04T06:12:58.310455+0000","last_fullsized":"2024-03-04T06:12:58.310455+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5983490,"num_objects":18,"num_object_clones":11,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":124,"num_read_kb":117183,"num_write":78,"num_write_kb":21237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":896,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-04T06:13:01.321971+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:13:01.321971+0000","last_peered":"2024-03-04T06:13:01.321971+0000","last_clean":"2024-03-04T06:13:01.321971+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:13:01.321971+0000","last_undegraded":"2024-03-04T06:13:01.321971+0000","last_fullsized":"2024-03-04T06:13:01.321971+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":742,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T06:12:58.331483+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:12:58.331483+0000","last_peered":"2024-03-04T06:12:58.331483+0000","last_clean":"2024-03-04T06:12:58.331483+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:12:58.331483+0000","last_undegraded":"2024-03-04T06:12:58.331483+0000","last_fullsized":"2024-03-04T06:12:58.331483+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"308'254","reported_seq":1053,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T06:12:58.317861+0000","last_change":"2024-03-04T06:12:58.380833+0000","last_active":"2024-03-04T06:12:58.317861+0000","last_peered":"2024-03-04T06:12:58.317861+0000","last_clean":"2024-03-04T06:12:58.317861+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:12:58.317861+0000","last_undegraded":"2024-03-04T06:12:58.317861+0000","last_fullsized":"2024-03-04T06:12:58.317861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8908040,"num_objects":26,"num_object_clones":19,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":144,"num_read_kb":109565,"num_write":124,"num_write_kb":31528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":749,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-03-04T06:12:58.312586+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:12:58.312586+0000","last_peered":"2024-03-04T06:12:58.312586+0000","last_clean":"2024-03-04T06:12:58.312586+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:12:58.312586+0000","last_undegraded":"2024-03-04T06:12:58.312586+0000","last_fullsized":"2024-03-04T06:12:58.312586+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"310'233","reported_seq":1033,"reported_epoch":311,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:13:01.320723+0000","last_change":"2024-03-04T06:13:00.405145+0000","last_active":"2024-03-04T06:13:01.320723+0000","last_peered":"2024-03-04T06:13:01.320723+0000","last_clean":"2024-03-04T06:13:01.320723+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:13:01.320723+0000","last_undegraded":"2024-03-04T06:13:01.320723+0000","last_fullsized":"2024-03-04T06:13:01.320723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899488,"num_objects":23,"num_object_clones":18,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":116,"num_read_kb":95340,"num_write":96,"num_write_kb":28103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":742,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T06:12:58.312236+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:12:58.312236+0000","last_peered":"2024-03-04T06:12:58.312236+0000","last_clean":"2024-03-04T06:12:58.312236+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:12:58.312236+0000","last_undegraded":"2024-03-04T06:12:58.312236+0000","last_fullsized":"2024-03-04T06:12:58.312236+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"311'520","reported_seq":1294,"reported_epoch":311,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:13:01.335171+0000","last_change":"2024-03-04T06:13:00.422246+0000","last_active":"2024-03-04T06:13:01.335171+0000","last_peered":"2024-03-04T06:13:01.335171+0000","last_clean":"2024-03-04T06:13:01.335171+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:13:01.335171+0000","last_undegraded":"2024-03-04T06:13:01.335171+0000","last_fullsized":"2024-03-04T06:13:01.335171+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8656133,"num_objects":53,"num_object_clones":40,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":340,"num_read_kb":281960,"num_write":272,"num_write_kb":73936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":752,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-03-04T06:13:00.327672+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:13:00.327672+0000","last_peered":"2024-03-04T06:13:00.327672+0000","last_clean":"2024-03-04T06:13:00.327672+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:13:00.327672+0000","last_undegraded":"2024-03-04T06:13:00.327672+0000","last_fullsized":"2024-03-04T06:13:00.327672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"309'496","reported_seq":1230,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-04T06:12:59.308243+0000","last_change":"2024-03-04T06:12:59.366296+0000","last_active":"2024-03-04T06:12:59.308243+0000","last_peered":"2024-03-04T06:12:59.308243+0000","last_clean":"2024-03-04T06:12:59.308243+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:12:59.308243+0000","last_undegraded":"2024-03-04T06:12:59.308243+0000","last_fullsized":"2024-03-04T06:12:59.308243+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12941984,"num_objects":50,"num_object_clones":36,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":217,"num_read_kb":155104,"num_write":239,"num_write_kb":63423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"308'207","reported_seq":1064,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T06:12:58.359277+0000","last_change":"2024-03-04T06:12:58.404200+0000","last_active":"2024-03-04T06:12:58.359277+0000","last_peered":"2024-03-04T06:12:58.359277+0000","last_clean":"2024-03-04T06:12:58.359277+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:12:58.359277+0000","last_undegraded":"2024-03-04T06:12:58.359277+0000","last_fullsized":"2024-03-04T06:12:58.359277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3983358,"num_objects":23,"num_object_clones":17,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":193,"num_read_kb":157563,"num_write":109,"num_write_kb":28860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":789,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-04T06:12:59.328951+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:12:59.328951+0000","last_peered":"2024-03-04T06:12:59.328951+0000","last_clean":"2024-03-04T06:12:59.328951+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:12:59.328951+0000","last_undegraded":"2024-03-04T06:12:59.328951+0000","last_fullsized":"2024-03-04T06:12:59.328951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"309'227","reported_seq":1077,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-04T06:12:59.329986+0000","last_change":"2024-03-04T06:12:59.374862+0000","last_active":"2024-03-04T06:12:59.329986+0000","last_peered":"2024-03-04T06:12:59.329986+0000","last_clean":"2024-03-04T06:12:59.329986+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:12:59.329986+0000","last_undegraded":"2024-03-04T06:12:59.329986+0000","last_fullsized":"2024-03-04T06:12:59.329986+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6538121,"num_objects":28,"num_object_clones":18,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":118,"num_read_kb":67272,"num_write":116,"num_write_kb":31360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":763,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-04T06:13:01.332571+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:13:01.332571+0000","last_peered":"2024-03-04T06:13:01.332571+0000","last_clean":"2024-03-04T06:13:01.332571+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:13:01.332571+0000","last_undegraded":"2024-03-04T06:13:01.332571+0000","last_fullsized":"2024-03-04T06:13:01.332571+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"310'343","reported_seq":1134,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-04T06:13:01.328220+0000","last_change":"2024-03-04T06:13:01.336132+0000","last_active":"2024-03-04T06:13:01.328220+0000","last_peered":"2024-03-04T06:13:01.328220+0000","last_clean":"2024-03-04T06:13:01.328220+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:13:01.328220+0000","last_undegraded":"2024-03-04T06:13:01.328220+0000","last_fullsized":"2024-03-04T06:13:01.328220+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840647,"num_objects":34,"num_object_clones":26,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":186,"num_read_kb":130495,"num_write":172,"num_write_kb":47172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":789,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-04T06:12:59.311849+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:12:59.311849+0000","last_peered":"2024-03-04T06:12:59.311849+0000","last_clean":"2024-03-04T06:12:59.311849+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:12:59.311849+0000","last_undegraded":"2024-03-04T06:12:59.311849+0000","last_fullsized":"2024-03-04T06:12:59.311849+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"309'437","reported_seq":1226,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-04T06:12:59.310959+0000","last_change":"2024-03-04T06:12:59.375171+0000","last_active":"2024-03-04T06:12:59.310959+0000","last_peered":"2024-03-04T06:12:59.310959+0000","last_clean":"2024-03-04T06:12:59.310959+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:12:59.310959+0000","last_undegraded":"2024-03-04T06:12:59.310959+0000","last_fullsized":"2024-03-04T06:12:59.310959+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11076077,"num_objects":38,"num_object_clones":34,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":277,"num_read_kb":221800,"num_write":197,"num_write_kb":54173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":742,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T06:12:58.320366+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:12:58.320366+0000","last_peered":"2024-03-04T06:12:58.320366+0000","last_clean":"2024-03-04T06:12:58.320366+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:12:58.320366+0000","last_undegraded":"2024-03-04T06:12:58.320366+0000","last_fullsized":"2024-03-04T06:12:58.320366+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":117048139,"num_objects":429,"num_object_clones":315,"num_object_copies":1287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":75,"num_read":2343,"num_read_kb":1780470,"num_write":2045,"num_write_kb":558102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4069,"ondisk_log_size":4069,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126192,"kb_used_data":86792,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94245648,"statfs":{"total":96636764160,"available":96507543552,"internally_reserved":0,"allocated":88875008,"data_stored":92817695,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":108020,"kb_used_data":72876,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94263820,"statfs":{"total":96636764160,"available":96526151680,"internally_reserved":0,"allocated":74625024,"data_stored":77330823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":98112,"kb_used_data":62752,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94273728,"statfs":{"total":96636764160,"available":96536297472,"internally_reserved":0,"allocated":64258048,"data_stored":68380944,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":95892,"kb_used_data":60524,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94275948,"statfs":{"total":96636764160,"available":96538570752,"internally_reserved":0,"allocated":61976576,"data_stored":63157058,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:13:06.319 INFO:tasks.rados.rados.0.smithi064.stdout:1299: done (1 left) 2024-03-04T06:13:06.319 INFO:tasks.rados.rados.0.smithi064.stdout:1300: snap_create 2024-03-04T06:13:06.319 INFO:tasks.rados.rados.0.smithi064.stdout:1298: expect (ObjNum 557 snap 132 seq_num 557) 2024-03-04T06:13:07.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:07.302 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:07.300+0000 7fa4dcad4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:07.305 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:07.305+0000 7fa4dcad4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:07.319 INFO:tasks.rados.rados.0.smithi064.stdout:1298: done (1 left) 2024-03-04T06:13:07.319 INFO:tasks.rados.rados.0.smithi064.stdout:1300: done (0 left) 2024-03-04T06:13:07.319 INFO:tasks.rados.rados.0.smithi064.stdout:1301: delete oid 50 current snap is 151 2024-03-04T06:13:07.379 INFO:tasks.rados.rados.0.smithi064.stdout:1301: done (0 left) 2024-03-04T06:13:07.379 INFO:tasks.rados.rados.0.smithi064.stdout:1302: write (excl) oid 1 current snap is 151 2024-03-04T06:13:07.379 INFO:tasks.rados.rados.0.smithi064.stdout:1302: seq_num 626 ranges {521182=646820,1948669=523219} 2024-03-04T06:13:07.385 INFO:tasks.rados.rados.0.smithi064.stdout:1302: writing smithi06431743-1 from 521182 to 1168002 tid 1 2024-03-04T06:13:07.392 INFO:tasks.rados.rados.0.smithi064.stdout:1302: writing smithi06431743-1 from 1948669 to 2471888 tid 2 2024-03-04T06:13:07.392 INFO:tasks.rados.rados.0.smithi064.stdout:1303: snap_create 2024-03-04T06:13:07.416 INFO:tasks.rados.rados.0.smithi064.stdout:1302: finishing write tid 1 to smithi06431743-1 2024-03-04T06:13:07.432 INFO:tasks.rados.rados.0.smithi064.stdout:1302: finishing write tid 2 to smithi06431743-1 2024-03-04T06:13:07.435 INFO:tasks.rados.rados.0.smithi064.stdout:1302: finishing write tid 3 to smithi06431743-1 2024-03-04T06:13:07.435 INFO:tasks.rados.rados.0.smithi064.stdout:1302: finishing write tid 4 to smithi06431743-1 2024-03-04T06:13:07.435 INFO:tasks.rados.rados.0.smithi064.stdout:1302: oid 1 updating version 0 to 532 2024-03-04T06:13:07.435 INFO:tasks.rados.rados.0.smithi064.stdout:1302: oid 1 version 532 is already newer than 530 2024-03-04T06:13:07.435 INFO:tasks.rados.rados.0.smithi064.stdout:1302: oid 1 version 532 is already newer than 532 2024-03-04T06:13:07.436 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 532 (ObjNum 626 snap 151 seq_num 626) dirty exists 2024-03-04T06:13:07.436 INFO:tasks.rados.rados.0.smithi064.stdout:1302: left oid 1 (ObjNum 626 snap 151 seq_num 626) 2024-03-04T06:13:08.327 INFO:tasks.rados.rados.0.smithi064.stdout:1302: done (1 left) 2024-03-04T06:13:08.327 INFO:tasks.rados.rados.0.smithi064.stdout:1303: done (0 left) 2024-03-04T06:13:08.327 INFO:tasks.rados.rados.0.smithi064.stdout:1304: write (excl) oid 33 current snap is 152 2024-03-04T06:13:08.327 INFO:tasks.rados.rados.0.smithi064.stdout:1304: seq_num 627 ranges {537989=722069,1890798=647191} 2024-03-04T06:13:08.335 INFO:tasks.rados.rados.0.smithi064.stdout:1304: writing smithi06431743-33 from 537989 to 1260058 tid 1 2024-03-04T06:13:08.344 INFO:tasks.rados.rados.0.smithi064.stdout:1304: writing smithi06431743-33 from 1890798 to 2537989 tid 2 2024-03-04T06:13:08.344 INFO:tasks.rados.rados.0.smithi064.stdout:1305: snap_remove snap 147 2024-03-04T06:13:09.329 INFO:tasks.rados.rados.0.smithi064.stdout:1305: done (1 left) 2024-03-04T06:13:09.329 INFO:tasks.rados.rados.0.smithi064.stdout:1306: delete oid 41 current snap is 152 2024-03-04T06:13:09.330 INFO:tasks.rados.rados.0.smithi064.stdout:1304: finishing write tid 1 to smithi06431743-33 2024-03-04T06:13:09.330 INFO:tasks.rados.rados.0.smithi064.stdout:1304: finishing write tid 2 to smithi06431743-33 2024-03-04T06:13:09.330 INFO:tasks.rados.rados.0.smithi064.stdout:1304: finishing write tid 3 to smithi06431743-33 2024-03-04T06:13:09.330 INFO:tasks.rados.rados.0.smithi064.stdout:1304: finishing write tid 4 to smithi06431743-33 2024-03-04T06:13:09.330 INFO:tasks.rados.rados.0.smithi064.stdout:1304: oid 33 updating version 0 to 442 2024-03-04T06:13:09.330 INFO:tasks.rados.rados.0.smithi064.stdout:1304: oid 33 updating version 442 to 444 2024-03-04T06:13:09.330 INFO:tasks.rados.rados.0.smithi064.stdout:1304: oid 33 version 444 is already newer than 443 2024-03-04T06:13:09.330 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 444 (ObjNum 627 snap 152 seq_num 627) dirty exists 2024-03-04T06:13:09.330 INFO:tasks.rados.rados.0.smithi064.stdout:1304: left oid 33 (ObjNum 627 snap 152 seq_num 627) 2024-03-04T06:13:09.382 INFO:tasks.rados.rados.0.smithi064.stdout:1304: done (1 left) 2024-03-04T06:13:09.382 INFO:tasks.rados.rados.0.smithi064.stdout:1306: done (0 left) 2024-03-04T06:13:09.383 INFO:tasks.rados.rados.0.smithi064.stdout:1307: snap_create 2024-03-04T06:13:09.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:09.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:09.516+0000 7f7c25edd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:09.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:09.520+0000 7f7c25edd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:10.330 INFO:tasks.rados.rados.0.smithi064.stdout:1307: done (0 left) 2024-03-04T06:13:10.330 INFO:tasks.rados.rados.0.smithi064.stdout:1308: delete oid 46 current snap is 153 2024-03-04T06:13:10.368 INFO:tasks.rados.rados.0.smithi064.stdout:1308: done (0 left) 2024-03-04T06:13:10.368 INFO:tasks.rados.rados.0.smithi064.stdout:1309: delete oid 16 current snap is 153 2024-03-04T06:13:10.436 INFO:tasks.rados.rados.0.smithi064.stdout:1309: done (0 left) 2024-03-04T06:13:10.436 INFO:tasks.rados.rados.0.smithi064.stdout:1310: write (excl) oid 24 current snap is 153 2024-03-04T06:13:10.436 INFO:tasks.rados.rados.0.smithi064.stdout:1310: seq_num 631 ranges {605217=739418,2124726=480491} 2024-03-04T06:13:10.443 INFO:tasks.rados.rados.0.smithi064.stdout:1310: writing smithi06431743-24 from 605217 to 1344635 tid 1 2024-03-04T06:13:10.451 INFO:tasks.rados.rados.0.smithi064.stdout:1310: writing smithi06431743-24 from 2124726 to 2605217 tid 2 2024-03-04T06:13:10.451 INFO:tasks.rados.rados.0.smithi064.stdout:1311: read oid 40 snap -1 2024-03-04T06:13:10.451 INFO:tasks.rados.rados.0.smithi064.stdout:1311: expect deleted 2024-03-04T06:13:10.451 INFO:tasks.rados.rados.0.smithi064.stdout:1312: write oid 30 current snap is 153 2024-03-04T06:13:10.451 INFO:tasks.rados.rados.0.smithi064.stdout:1312: seq_num 632 ranges {622024=414667,1783208=417873} 2024-03-04T06:13:10.458 INFO:tasks.rados.rados.0.smithi064.stdout:1312: writing smithi06431743-30 from 622024 to 1036691 tid 1 2024-03-04T06:13:10.465 INFO:tasks.rados.rados.0.smithi064.stdout:1312: writing smithi06431743-30 from 1783208 to 2201081 tid 2 2024-03-04T06:13:10.465 INFO:tasks.rados.rados.0.smithi064.stdout:1313: write (excl) oid 23 current snap is 153 2024-03-04T06:13:10.465 INFO:tasks.rados.rados.0.smithi064.stdout:1313: seq_num 633 ranges {638831=489916,1841690=677884} 2024-03-04T06:13:10.473 INFO:tasks.rados.rados.0.smithi064.stdout:1313: writing smithi06431743-23 from 638831 to 1128747 tid 1 2024-03-04T06:13:10.481 INFO:tasks.rados.rados.0.smithi064.stdout:1313: writing smithi06431743-23 from 1841690 to 2519574 tid 2 2024-03-04T06:13:10.481 INFO:tasks.rados.rados.0.smithi064.stdout:1314: snap_create 2024-03-04T06:13:10.482 INFO:tasks.rados.rados.0.smithi064.stdout:1310: finishing write tid 1 to smithi06431743-24 2024-03-04T06:13:10.482 INFO:tasks.rados.rados.0.smithi064.stdout:1312: finishing write tid 1 to smithi06431743-30 2024-03-04T06:13:10.486 INFO:tasks.rados.rados.0.smithi064.stdout:1310: finishing write tid 2 to smithi06431743-24 2024-03-04T06:13:10.498 INFO:tasks.rados.rados.0.smithi064.stdout:1312: finishing write tid 2 to smithi06431743-30 2024-03-04T06:13:10.509 INFO:tasks.rados.rados.0.smithi064.stdout:1310: finishing write tid 3 to smithi06431743-24 2024-03-04T06:13:10.511 INFO:tasks.rados.rados.0.smithi064.stdout:1310: finishing write tid 4 to smithi06431743-24 2024-03-04T06:13:10.511 INFO:tasks.rados.rados.0.smithi064.stdout:1310: oid 24 updating version 0 to 344 2024-03-04T06:13:10.511 INFO:tasks.rados.rados.0.smithi064.stdout:1310: oid 24 updating version 344 to 345 2024-03-04T06:13:10.511 INFO:tasks.rados.rados.0.smithi064.stdout:1310: oid 24 updating version 345 to 346 2024-03-04T06:13:10.512 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 346 (ObjNum 631 snap 153 seq_num 631) dirty exists 2024-03-04T06:13:10.512 INFO:tasks.rados.rados.0.smithi064.stdout:1310: left oid 24 (ObjNum 631 snap 153 seq_num 631) 2024-03-04T06:13:10.512 INFO:tasks.rados.rados.0.smithi064.stdout:1312: finishing write tid 3 to smithi06431743-30 2024-03-04T06:13:10.513 INFO:tasks.rados.rados.0.smithi064.stdout:1312: finishing write tid 4 to smithi06431743-30 2024-03-04T06:13:10.513 INFO:tasks.rados.rados.0.smithi064.stdout:1312: oid 30 updating version 0 to 510 2024-03-04T06:13:10.513 INFO:tasks.rados.rados.0.smithi064.stdout:1312: oid 30 version 510 is already newer than 509 2024-03-04T06:13:10.513 INFO:tasks.rados.rados.0.smithi064.stdout:1312: oid 30 updating version 510 to 511 2024-03-04T06:13:10.513 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 511 (ObjNum 632 snap 153 seq_num 632) dirty exists 2024-03-04T06:13:10.513 INFO:tasks.rados.rados.0.smithi064.stdout:1312: left oid 30 (ObjNum 632 snap 153 seq_num 632) 2024-03-04T06:13:10.515 INFO:tasks.rados.rados.0.smithi064.stdout:1313: finishing write tid 1 to smithi06431743-23 2024-03-04T06:13:10.525 INFO:tasks.rados.rados.0.smithi064.stdout:1313: finishing write tid 2 to smithi06431743-23 2024-03-04T06:13:10.533 INFO:tasks.rados.rados.0.smithi064.stdout:1313: finishing write tid 3 to smithi06431743-23 2024-03-04T06:13:10.533 INFO:tasks.rados.rados.0.smithi064.stdout:1313: finishing write tid 4 to smithi06431743-23 2024-03-04T06:13:10.534 INFO:tasks.rados.rados.0.smithi064.stdout:1313: oid 23 updating version 0 to 262 2024-03-04T06:13:10.534 INFO:tasks.rados.rados.0.smithi064.stdout:1313: oid 23 version 262 is already newer than 260 2024-03-04T06:13:10.534 INFO:tasks.rados.rados.0.smithi064.stdout:1313: oid 23 version 262 is already newer than 261 2024-03-04T06:13:10.534 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 262 (ObjNum 633 snap 153 seq_num 633) dirty exists 2024-03-04T06:13:10.534 INFO:tasks.rados.rados.0.smithi064.stdout:1313: left oid 23 (ObjNum 633 snap 153 seq_num 633) 2024-03-04T06:13:10.890 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:13:10.890 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:13:10.890 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:13:10.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:13:10.999 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:10.994+0000 7f5294c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:11.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:11.001+0000 7f5294c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:11.014 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:11.015 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:11.015 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:11.015 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:11.015 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:11.015 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:11.015 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:11.015 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:11.015 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:11.015 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:11.015 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:11.015 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:11.017 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:11.018 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:11.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:13:11.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:11.121+0000 7fcfb335d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:11.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:11.129+0000 7fcfb335d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:11.142 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:11.144 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:11.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:13:11.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:11.248+0000 7f23da39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:11.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:11.255+0000 7f23da39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:11.269 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:11.271 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:11.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:13:11.337 INFO:tasks.rados.rados.0.smithi064.stdout:1310: done (4 left) 2024-03-04T06:13:11.338 INFO:tasks.rados.rados.0.smithi064.stdout:1311: done (3 left) 2024-03-04T06:13:11.338 INFO:tasks.rados.rados.0.smithi064.stdout:1312: done (2 left) 2024-03-04T06:13:11.338 INFO:tasks.rados.rados.0.smithi064.stdout:1313: done (1 left) 2024-03-04T06:13:11.338 INFO:tasks.rados.rados.0.smithi064.stdout:1314: done (0 left) 2024-03-04T06:13:11.338 INFO:tasks.rados.rados.0.smithi064.stdout:1315: read oid 1 snap 150 2024-03-04T06:13:11.338 INFO:tasks.rados.rados.0.smithi064.stdout:1315: expect (ObjNum 610 snap 146 seq_num 610) 2024-03-04T06:13:11.338 INFO:tasks.rados.rados.0.smithi064.stdout:1316: snap_remove snap 149 2024-03-04T06:13:11.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:11.380+0000 7eff9c424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:11.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:11.388+0000 7eff9c424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:11.400 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:11.401 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:11.401 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:11.401 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:11.401 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:11.401 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:11.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:11.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:11.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:11.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:11.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:11.401 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:11.403 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:11.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:11.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:11.691+0000 7ff11111d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:11.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:11.695+0000 7ff11111d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:12.338 INFO:tasks.rados.rados.0.smithi064.stdout:1316: done (1 left) 2024-03-04T06:13:12.338 INFO:tasks.rados.rados.0.smithi064.stdout:1317: write oid 39 current snap is 154 2024-03-04T06:13:12.338 INFO:tasks.rados.rados.0.smithi064.stdout:1317: seq_num 634 ranges {655638=565165,1900172=421542} 2024-03-04T06:13:12.344 INFO:tasks.rados.rados.0.smithi064.stdout:1317: writing smithi06431743-39 from 655638 to 1220803 tid 1 2024-03-04T06:13:12.351 INFO:tasks.rados.rados.0.smithi064.stdout:1317: writing smithi06431743-39 from 1900172 to 2321714 tid 2 2024-03-04T06:13:12.351 INFO:tasks.rados.rados.0.smithi064.stdout:1318: delete oid 34 current snap is 154 2024-03-04T06:13:12.351 INFO:tasks.rados.rados.0.smithi064.stdout:1315: expect (ObjNum 610 snap 146 seq_num 610) 2024-03-04T06:13:12.448 INFO:tasks.rados.rados.0.smithi064.stdout:1315: done (2 left) 2024-03-04T06:13:12.448 INFO:tasks.rados.rados.0.smithi064.stdout:1318: done (1 left) 2024-03-04T06:13:12.448 INFO:tasks.rados.rados.0.smithi064.stdout:1319: write (excl) oid 10 current snap is 154 2024-03-04T06:13:12.448 INFO:tasks.rados.rados.0.smithi064.stdout:1319: seq_num 636 ranges {689252=715663,1900783=425211} 2024-03-04T06:13:12.456 INFO:tasks.rados.rados.0.smithi064.stdout:1319: writing smithi06431743-10 from 689252 to 1404915 tid 1 2024-03-04T06:13:12.462 INFO:tasks.rados.rados.0.smithi064.stdout:1319: writing smithi06431743-10 from 1900783 to 2325994 tid 2 2024-03-04T06:13:12.462 INFO:tasks.rados.rados.0.smithi064.stdout:1320: read oid 28 snap -1 2024-03-04T06:13:12.462 INFO:tasks.rados.rados.0.smithi064.stdout:1320: expect deleted 2024-03-04T06:13:12.462 INFO:tasks.rados.rados.0.smithi064.stdout:1321: write (excl) oid 3 current snap is 154 2024-03-04T06:13:12.462 INFO:tasks.rados.rados.0.smithi064.stdout:1317: finishing write tid 1 to smithi06431743-39 2024-03-04T06:13:12.462 INFO:tasks.rados.rados.0.smithi064.stdout:1317: finishing write tid 2 to smithi06431743-39 2024-03-04T06:13:12.462 INFO:tasks.rados.rados.0.smithi064.stdout:1321: seq_num 637 ranges {706059=790912,1959265=685222} 2024-03-04T06:13:12.470 INFO:tasks.rados.rados.0.smithi064.stdout:1321: writing smithi06431743-3 from 706059 to 1496971 tid 1 2024-03-04T06:13:12.478 INFO:tasks.rados.rados.0.smithi064.stdout:1321: writing smithi06431743-3 from 1959265 to 2644487 tid 2 2024-03-04T06:13:12.478 INFO:tasks.rados.rados.0.smithi064.stdout:1322: write oid 36 current snap is 154 2024-03-04T06:13:12.478 INFO:tasks.rados.rados.0.smithi064.stdout:1317: finishing write tid 3 to smithi06431743-39 2024-03-04T06:13:12.478 INFO:tasks.rados.rados.0.smithi064.stdout:1322: seq_num 638 ranges {722866=466161,1617747=428880,2587634=135232} 2024-03-04T06:13:12.486 INFO:tasks.rados.rados.0.smithi064.stdout:1322: writing smithi06431743-36 from 722866 to 1189027 tid 1 2024-03-04T06:13:12.491 INFO:tasks.rados.rados.0.smithi064.stdout:1322: writing smithi06431743-36 from 1617747 to 2046627 tid 2 2024-03-04T06:13:12.495 INFO:tasks.rados.rados.0.smithi064.stdout:1322: writing smithi06431743-36 from 2587634 to 2722866 tid 3 2024-03-04T06:13:12.495 INFO:tasks.rados.rados.0.smithi064.stdout:1323: read oid 33 snap -1 2024-03-04T06:13:12.495 INFO:tasks.rados.rados.0.smithi064.stdout:1323: expect (ObjNum 627 snap 152 seq_num 627) 2024-03-04T06:13:12.495 INFO:tasks.rados.rados.0.smithi064.stdout:1324: snap_remove snap 153 2024-03-04T06:13:13.344 INFO:tasks.rados.rados.0.smithi064.stdout:1324: done (6 left) 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1325: snap_create 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1317: finishing write tid 4 to smithi06431743-39 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1317: oid 39 updating version 0 to 293 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1317: oid 39 updating version 293 to 295 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1317: oid 39 version 295 is already newer than 294 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 295 (ObjNum 634 snap 154 seq_num 634) dirty exists 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1317: left oid 39 (ObjNum 634 snap 154 seq_num 634) 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1319: finishing write tid 1 to smithi06431743-10 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1321: finishing write tid 1 to smithi06431743-3 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1319: finishing write tid 2 to smithi06431743-10 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1321: finishing write tid 2 to smithi06431743-3 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1321: finishing write tid 3 to smithi06431743-3 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1321: finishing write tid 4 to smithi06431743-3 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1321: oid 3 updating version 0 to 270 2024-03-04T06:13:13.345 INFO:tasks.rados.rados.0.smithi064.stdout:1321: oid 3 version 270 is already newer than 268 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:1321: oid 3 version 270 is already newer than 269 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 270 (ObjNum 637 snap 154 seq_num 637) dirty exists 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:1321: left oid 3 (ObjNum 637 snap 154 seq_num 637) 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:1319: finishing write tid 3 to smithi06431743-10 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:1319: finishing write tid 4 to smithi06431743-10 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:1319: oid 10 updating version 0 to 212 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:1319: oid 10 updating version 212 to 213 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:1319: oid 10 updating version 213 to 214 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 214 (ObjNum 636 snap 154 seq_num 636) dirty exists 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:1319: left oid 10 (ObjNum 636 snap 154 seq_num 636) 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:1322: finishing write tid 1 to smithi06431743-36 2024-03-04T06:13:13.346 INFO:tasks.rados.rados.0.smithi064.stdout:1323: expect (ObjNum 627 snap 152 seq_num 627) 2024-03-04T06:13:13.482 INFO:tasks.rados.rados.0.smithi064.stdout:1322: finishing write tid 2 to smithi06431743-36 2024-03-04T06:13:13.482 INFO:tasks.rados.rados.0.smithi064.stdout:1322: finishing write tid 3 to smithi06431743-36 2024-03-04T06:13:13.482 INFO:tasks.rados.rados.0.smithi064.stdout:1322: finishing write tid 4 to smithi06431743-36 2024-03-04T06:13:13.482 INFO:tasks.rados.rados.0.smithi064.stdout:1322: finishing write tid 5 to smithi06431743-36 2024-03-04T06:13:13.482 INFO:tasks.rados.rados.0.smithi064.stdout:1322: oid 36 updating version 0 to 233 2024-03-04T06:13:13.482 INFO:tasks.rados.rados.0.smithi064.stdout:1322: oid 36 updating version 233 to 235 2024-03-04T06:13:13.482 INFO:tasks.rados.rados.0.smithi064.stdout:1322: oid 36 version 235 is already newer than 233 2024-03-04T06:13:13.482 INFO:tasks.rados.rados.0.smithi064.stdout:1322: oid 36 version 235 is already newer than 235 2024-03-04T06:13:13.482 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 235 (ObjNum 638 snap 154 seq_num 638) dirty exists 2024-03-04T06:13:13.482 INFO:tasks.rados.rados.0.smithi064.stdout:1322: left oid 36 (ObjNum 638 snap 154 seq_num 638) 2024-03-04T06:13:13.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:13.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:13.867+0000 7ff31806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:13.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:13.872+0000 7ff31806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:14.345 INFO:tasks.rados.rados.0.smithi064.stdout:1317: done (6 left) 2024-03-04T06:13:14.345 INFO:tasks.rados.rados.0.smithi064.stdout:1319: done (5 left) 2024-03-04T06:13:14.345 INFO:tasks.rados.rados.0.smithi064.stdout:1320: done (4 left) 2024-03-04T06:13:14.345 INFO:tasks.rados.rados.0.smithi064.stdout:1321: done (3 left) 2024-03-04T06:13:14.345 INFO:tasks.rados.rados.0.smithi064.stdout:1322: done (2 left) 2024-03-04T06:13:14.345 INFO:tasks.rados.rados.0.smithi064.stdout:1323: done (1 left) 2024-03-04T06:13:14.346 INFO:tasks.rados.rados.0.smithi064.stdout:1325: done (0 left) 2024-03-04T06:13:14.346 INFO:tasks.rados.rados.0.smithi064.stdout:1326: read oid 22 snap -1 2024-03-04T06:13:14.346 INFO:tasks.rados.rados.0.smithi064.stdout:1326: expect (ObjNum 593 snap 142 seq_num 593) 2024-03-04T06:13:14.346 INFO:tasks.rados.rados.0.smithi064.stdout:1327: read oid 50 snap 154 2024-03-04T06:13:14.346 INFO:tasks.rados.rados.0.smithi064.stdout:1327: expect deleted 2024-03-04T06:13:14.346 INFO:tasks.rados.rados.0.smithi064.stdout:1328: write oid 30 current snap is 155 2024-03-04T06:13:14.346 INFO:tasks.rados.rados.0.smithi064.stdout:1328: seq_num 639 ranges {739673=657763,2076229=663444} 2024-03-04T06:13:14.352 INFO:tasks.rados.rados.0.smithi064.stdout:1328: writing smithi06431743-30 from 739673 to 1397436 tid 1 2024-03-04T06:13:14.360 INFO:tasks.rados.rados.0.smithi064.stdout:1328: writing smithi06431743-30 from 2076229 to 2739673 tid 2 2024-03-04T06:13:14.360 INFO:tasks.rados.rados.0.smithi064.stdout:1329: snap_remove snap 151 2024-03-04T06:13:15.350 INFO:tasks.rados.rados.0.smithi064.stdout:1329: done (3 left) 2024-03-04T06:13:15.351 INFO:tasks.rados.rados.0.smithi064.stdout:1330: read oid 43 snap -1 2024-03-04T06:13:15.351 INFO:tasks.rados.rados.0.smithi064.stdout:1330: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:13:15.351 INFO:tasks.rados.rados.0.smithi064.stdout:1331: snap_create 2024-03-04T06:13:15.351 INFO:tasks.rados.rados.0.smithi064.stdout:1326: expect (ObjNum 593 snap 142 seq_num 593) 2024-03-04T06:13:15.551 INFO:tasks.rados.rados.0.smithi064.stdout:1328: finishing write tid 1 to smithi06431743-30 2024-03-04T06:13:15.551 INFO:tasks.rados.rados.0.smithi064.stdout:1328: finishing write tid 2 to smithi06431743-30 2024-03-04T06:13:15.551 INFO:tasks.rados.rados.0.smithi064.stdout:1328: finishing write tid 3 to smithi06431743-30 2024-03-04T06:13:15.551 INFO:tasks.rados.rados.0.smithi064.stdout:1328: finishing write tid 4 to smithi06431743-30 2024-03-04T06:13:15.551 INFO:tasks.rados.rados.0.smithi064.stdout:1328: oid 30 updating version 0 to 517 2024-03-04T06:13:15.551 INFO:tasks.rados.rados.0.smithi064.stdout:1328: oid 30 version 517 is already newer than 515 2024-03-04T06:13:15.551 INFO:tasks.rados.rados.0.smithi064.stdout:1328: oid 30 version 517 is already newer than 517 2024-03-04T06:13:15.551 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 517 (ObjNum 639 snap 155 seq_num 639) dirty exists 2024-03-04T06:13:15.551 INFO:tasks.rados.rados.0.smithi064.stdout:1328: left oid 30 (ObjNum 639 snap 155 seq_num 639) 2024-03-04T06:13:15.551 INFO:tasks.rados.rados.0.smithi064.stdout:1330: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:13:15.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:16.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:16.097+0000 7fef5075d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:16.102 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:16.101+0000 7fef5075d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:16.351 INFO:tasks.rados.rados.0.smithi064.stdout:1326: done (4 left) 2024-03-04T06:13:16.351 INFO:tasks.rados.rados.0.smithi064.stdout:1327: done (3 left) 2024-03-04T06:13:16.351 INFO:tasks.rados.rados.0.smithi064.stdout:1328: done (2 left) 2024-03-04T06:13:16.351 INFO:tasks.rados.rados.0.smithi064.stdout:1330: done (1 left) 2024-03-04T06:13:16.351 INFO:tasks.rados.rados.0.smithi064.stdout:1331: done (0 left) 2024-03-04T06:13:16.351 INFO:tasks.rados.rados.0.smithi064.stdout:1332: read oid 45 snap -1 2024-03-04T06:13:16.351 INFO:tasks.rados.rados.0.smithi064.stdout:1332: expect deleted 2024-03-04T06:13:16.351 INFO:tasks.rados.rados.0.smithi064.stdout:1333: delete oid 15 current snap is 156 2024-03-04T06:13:16.397 INFO:tasks.rados.rados.0.smithi064.stdout:1332: done (1 left) 2024-03-04T06:13:16.398 INFO:tasks.rados.rados.0.smithi064.stdout:1333: done (0 left) 2024-03-04T06:13:16.398 INFO:tasks.rados.rados.0.smithi064.stdout:1334: delete oid 30 current snap is 156 2024-03-04T06:13:16.404 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:13:16.404 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:13:16.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:13:16.459 INFO:tasks.rados.rados.0.smithi064.stdout:1334: done (0 left) 2024-03-04T06:13:16.459 INFO:tasks.rados.rados.0.smithi064.stdout:1335: read oid 36 snap -1 2024-03-04T06:13:16.459 INFO:tasks.rados.rados.0.smithi064.stdout:1335: expect (ObjNum 638 snap 154 seq_num 638) 2024-03-04T06:13:16.459 INFO:tasks.rados.rados.0.smithi064.stdout:1336: read oid 9 snap 154 2024-03-04T06:13:16.459 INFO:tasks.rados.rados.0.smithi064.stdout:1336: expect (ObjNum 569 snap 134 seq_num 569) 2024-03-04T06:13:16.459 INFO:tasks.rados.rados.0.smithi064.stdout:1337: read oid 43 snap 152 2024-03-04T06:13:16.459 INFO:tasks.rados.rados.0.smithi064.stdout:1337: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:13:16.459 INFO:tasks.rados.rados.0.smithi064.stdout:1338: read oid 15 snap -1 2024-03-04T06:13:16.460 INFO:tasks.rados.rados.0.smithi064.stdout:1338: expect deleted 2024-03-04T06:13:16.460 INFO:tasks.rados.rados.0.smithi064.stdout:1339: delete oid 27 current snap is 156 2024-03-04T06:13:16.487 INFO:tasks.rados.rados.0.smithi064.stdout:1335: expect (ObjNum 638 snap 154 seq_num 638) 2024-03-04T06:13:16.527 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:16.523+0000 7f78aa86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:16.530 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:16.529+0000 7f78aa86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:16.598 INFO:tasks.rados.rados.0.smithi064.stdout:1336: expect (ObjNum 569 snap 134 seq_num 569) 2024-03-04T06:13:16.724 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:13:16.725 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:13:16.737 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-03-04T06:13:16.349360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103066607,"num_objects":447,"num_object_clones":329,"num_object_copies":1341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":81,"num_read":2497,"num_read_kb":1803533,"num_write":2237,"num_write_kb":572938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4290,"ondisk_log_size":4290,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":348592,"kb_used_data":202408,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377138768,"statfs":{"total":386547056640,"available":386190098432,"internally_reserved":0,"allocated":207265792,"data_stored":211773149,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-873898,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":18,"num_read_kb":12908,"num_write":18,"num_write_kb":4228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012805"},"pg_stats":[{"pgid":"3.f","version":"320'234","reported_seq":1087,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-04T06:13:13.899531+0000","last_change":"2024-03-04T06:13:13.907282+0000","last_active":"2024-03-04T06:13:13.899531+0000","last_peered":"2024-03-04T06:13:13.899531+0000","last_clean":"2024-03-04T06:13:13.899531+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:13:13.899531+0000","last_undegraded":"2024-03-04T06:13:13.899531+0000","last_fullsized":"2024-03-04T06:13:13.899531+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496087,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":102,"num_read_kb":63793,"num_write":131,"num_write_kb":35117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"320'191","reported_seq":1078,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-04T06:13:14.362417+0000","last_change":"2024-03-04T06:13:13.903780+0000","last_active":"2024-03-04T06:13:14.362417+0000","last_peered":"2024-03-04T06:13:14.362417+0000","last_clean":"2024-03-04T06:13:14.362417+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:13:14.362417+0000","last_undegraded":"2024-03-04T06:13:14.362417+0000","last_fullsized":"2024-03-04T06:13:14.362417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2453954,"num_objects":21,"num_object_clones":15,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":85,"num_read_kb":57066,"num_write":85,"num_write_kb":21710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"312'291","reported_seq":1124,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-04T06:13:10.389413+0000","last_change":"2024-03-04T06:13:09.353120+0000","last_active":"2024-03-04T06:13:10.389413+0000","last_peered":"2024-03-04T06:13:10.389413+0000","last_clean":"2024-03-04T06:13:10.389413+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:13:10.389413+0000","last_undegraded":"2024-03-04T06:13:10.389413+0000","last_fullsized":"2024-03-04T06:13:10.389413+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9462638,"num_objects":34,"num_object_clones":26,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":169,"num_read_kb":119629,"num_write":157,"num_write_kb":43443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"308'203","reported_seq":1091,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-04T06:13:13.893153+0000","last_change":"2024-03-04T06:13:13.909496+0000","last_active":"2024-03-04T06:13:13.893153+0000","last_peered":"2024-03-04T06:13:13.893153+0000","last_clean":"2024-03-04T06:13:13.893153+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:13:13.893153+0000","last_undegraded":"2024-03-04T06:13:13.893153+0000","last_fullsized":"2024-03-04T06:13:13.893153+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983655,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":125,"num_read_kb":96803,"num_write":119,"num_write_kb":34146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"312'98","reported_seq":981,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-04T06:13:11.364370+0000","last_change":"2024-03-04T06:13:09.365296+0000","last_active":"2024-03-04T06:13:11.364370+0000","last_peered":"2024-03-04T06:13:11.364370+0000","last_clean":"2024-03-04T06:13:11.364370+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:13:11.364370+0000","last_undegraded":"2024-03-04T06:13:11.364370+0000","last_fullsized":"2024-03-04T06:13:11.364370+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2420340,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":17043,"num_write":42,"num_write_kb":10500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"323'216","reported_seq":1095,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-04T06:13:13.905069+0000","last_change":"2024-03-04T06:13:13.933600+0000","last_active":"2024-03-04T06:13:13.905069+0000","last_peered":"2024-03-04T06:13:13.905069+0000","last_clean":"2024-03-04T06:13:13.905069+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:13:13.905069+0000","last_undegraded":"2024-03-04T06:13:13.905069+0000","last_fullsized":"2024-03-04T06:13:13.905069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8824005,"num_objects":24,"num_object_clones":17,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":128,"num_read_kb":94388,"num_write":126,"num_write_kb":37591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":927,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-04T06:13:13.897402+0000","last_change":"2024-03-04T06:13:13.907976+0000","last_active":"2024-03-04T06:13:13.897402+0000","last_peered":"2024-03-04T06:13:13.897402+0000","last_clean":"2024-03-04T06:13:13.897402+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:13:13.897402+0000","last_undegraded":"2024-03-04T06:13:13.897402+0000","last_fullsized":"2024-03-04T06:13:13.897402+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"319'165","reported_seq":1040,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-04T06:13:10.388654+0000","last_change":"2024-03-04T06:13:09.365690+0000","last_active":"2024-03-04T06:13:10.388654+0000","last_peered":"2024-03-04T06:13:10.388654+0000","last_clean":"2024-03-04T06:13:10.388654+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:13:10.388654+0000","last_undegraded":"2024-03-04T06:13:10.388654+0000","last_fullsized":"2024-03-04T06:13:10.388654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731253,"num_objects":19,"num_object_clones":12,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":125,"num_read_kb":117183,"num_write":79,"num_write_kb":21237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":922,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-04T06:13:11.354557+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:13:11.354557+0000","last_peered":"2024-03-04T06:13:11.354557+0000","last_clean":"2024-03-04T06:13:11.354557+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:13:11.354557+0000","last_undegraded":"2024-03-04T06:13:11.354557+0000","last_fullsized":"2024-03-04T06:13:11.354557+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":772,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-04T06:13:10.396146+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:13:10.396146+0000","last_peered":"2024-03-04T06:13:10.396146+0000","last_clean":"2024-03-04T06:13:10.396146+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:13:10.396146+0000","last_undegraded":"2024-03-04T06:13:10.396146+0000","last_fullsized":"2024-03-04T06:13:10.396146+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"320'262","reported_seq":1093,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-04T06:13:10.526118+0000","last_change":"2024-03-04T06:13:09.358948+0000","last_active":"2024-03-04T06:13:10.526118+0000","last_peered":"2024-03-04T06:13:10.526118+0000","last_clean":"2024-03-04T06:13:10.526118+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:13:10.526118+0000","last_undegraded":"2024-03-04T06:13:10.526118+0000","last_fullsized":"2024-03-04T06:13:10.526118+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5731418,"num_objects":28,"num_object_clones":21,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":147,"num_read_kb":109566,"num_write":129,"num_write_kb":32669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":785,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-04T06:13:13.897591+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:13:13.897591+0000","last_peered":"2024-03-04T06:13:13.897591+0000","last_clean":"2024-03-04T06:13:13.897591+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:13:13.897591+0000","last_undegraded":"2024-03-04T06:13:13.897591+0000","last_fullsized":"2024-03-04T06:13:13.897591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"311'235","reported_seq":1065,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-04T06:13:11.349983+0000","last_change":"2024-03-04T06:13:09.359013+0000","last_active":"2024-03-04T06:13:11.349983+0000","last_peered":"2024-03-04T06:13:11.349983+0000","last_clean":"2024-03-04T06:13:11.349983+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:13:11.349983+0000","last_undegraded":"2024-03-04T06:13:11.349983+0000","last_fullsized":"2024-03-04T06:13:11.349983+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899488,"num_objects":23,"num_object_clones":18,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":116,"num_read_kb":95340,"num_write":96,"num_write_kb":28103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":772,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-04T06:13:10.390376+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:13:10.390376+0000","last_peered":"2024-03-04T06:13:10.390376+0000","last_clean":"2024-03-04T06:13:10.390376+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:13:10.390376+0000","last_undegraded":"2024-03-04T06:13:10.390376+0000","last_fullsized":"2024-03-04T06:13:10.390376+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"320'535","reported_seq":1330,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-04T06:13:11.365682+0000","last_change":"2024-03-04T06:13:09.384064+0000","last_active":"2024-03-04T06:13:11.365682+0000","last_peered":"2024-03-04T06:13:11.365682+0000","last_clean":"2024-03-04T06:13:11.365682+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:13:11.365682+0000","last_undegraded":"2024-03-04T06:13:11.365682+0000","last_fullsized":"2024-03-04T06:13:11.365682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8925045,"num_objects":54,"num_object_clones":41,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":342,"num_read_kb":281961,"num_write":276,"num_write_kb":75079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":785,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-04T06:13:13.889395+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:13:13.889395+0000","last_peered":"2024-03-04T06:13:13.889395+0000","last_clean":"2024-03-04T06:13:13.889395+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:13:13.889395+0000","last_undegraded":"2024-03-04T06:13:13.889395+0000","last_fullsized":"2024-03-04T06:13:13.889395+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"324'517","reported_seq":1284,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-04T06:13:14.419698+0000","last_change":"2024-03-04T06:13:13.903334+0000","last_active":"2024-03-04T06:13:14.419698+0000","last_peered":"2024-03-04T06:13:14.419698+0000","last_clean":"2024-03-04T06:13:14.419698+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:13:14.419698+0000","last_undegraded":"2024-03-04T06:13:14.419698+0000","last_fullsized":"2024-03-04T06:13:14.419698+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13479808,"num_objects":52,"num_object_clones":38,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":221,"num_read_kb":155107,"num_write":251,"num_write_kb":66344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"319'217","reported_seq":1100,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-04T06:13:10.395502+0000","last_change":"2024-03-04T06:13:09.382641+0000","last_active":"2024-03-04T06:13:10.395502+0000","last_peered":"2024-03-04T06:13:10.395502+0000","last_clean":"2024-03-04T06:13:10.395502+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:13:10.395502+0000","last_undegraded":"2024-03-04T06:13:10.395502+0000","last_fullsized":"2024-03-04T06:13:10.395502+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":24,"num_object_clones":18,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":194,"num_read_kb":157563,"num_write":110,"num_write_kb":28860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":824,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-04T06:13:14.376391+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:13:14.376391+0000","last_peered":"2024-03-04T06:13:14.376391+0000","last_clean":"2024-03-04T06:13:14.376391+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:13:14.376391+0000","last_undegraded":"2024-03-04T06:13:14.376391+0000","last_fullsized":"2024-03-04T06:13:14.376391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"322'235","reported_seq":1125,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-04T06:13:14.374561+0000","last_change":"2024-03-04T06:13:13.914939+0000","last_active":"2024-03-04T06:13:14.374561+0000","last_peered":"2024-03-04T06:13:14.374561+0000","last_clean":"2024-03-04T06:13:14.374561+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:13:14.374561+0000","last_undegraded":"2024-03-04T06:13:14.374561+0000","last_fullsized":"2024-03-04T06:13:14.374561+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554961,"num_objects":30,"num_object_clones":19,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":120,"num_read_kb":67273,"num_write":122,"num_write_kb":32368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":789,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-04T06:13:11.364093+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:13:11.364093+0000","last_peered":"2024-03-04T06:13:11.364093+0000","last_clean":"2024-03-04T06:13:11.364093+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:13:11.364093+0000","last_undegraded":"2024-03-04T06:13:11.364093+0000","last_fullsized":"2024-03-04T06:13:11.364093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"320'346","reported_seq":1171,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-03-04T06:13:11.355750+0000","last_change":"2024-03-04T06:13:09.357116+0000","last_active":"2024-03-04T06:13:11.355750+0000","last_peered":"2024-03-04T06:13:11.355750+0000","last_clean":"2024-03-04T06:13:11.355750+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:13:11.355750+0000","last_undegraded":"2024-03-04T06:13:11.355750+0000","last_fullsized":"2024-03-04T06:13:11.355750+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9445864,"num_objects":35,"num_object_clones":26,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":194,"num_read_kb":136022,"num_write":176,"num_write_kb":48365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":824,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-04T06:13:14.367016+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:13:14.367016+0000","last_peered":"2024-03-04T06:13:14.367016+0000","last_clean":"2024-03-04T06:13:14.367016+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:13:14.367016+0000","last_undegraded":"2024-03-04T06:13:14.367016+0000","last_fullsized":"2024-03-04T06:13:14.367016+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"322'448","reported_seq":1278,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-04T06:13:14.402633+0000","last_change":"2024-03-04T06:13:13.903138+0000","last_active":"2024-03-04T06:13:14.402633+0000","last_peered":"2024-03-04T06:13:14.402633+0000","last_clean":"2024-03-04T06:13:14.402633+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:13:14.402633+0000","last_undegraded":"2024-03-04T06:13:14.402633+0000","last_fullsized":"2024-03-04T06:13:14.402633+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10067723,"num_objects":39,"num_object_clones":35,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":291,"num_read_kb":234705,"num_write":201,"num_write_kb":55512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":772,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-04T06:13:10.395051+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:13:10.395051+0000","last_peered":"2024-03-04T06:13:10.395051+0000","last_clean":"2024-03-04T06:13:10.395051+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:13:10.395051+0000","last_undegraded":"2024-03-04T06:13:10.395051+0000","last_fullsized":"2024-03-04T06:13:10.395051+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102476239,"num_objects":445,"num_object_clones":329,"num_object_copies":1335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":81,"num_read":2394,"num_read_kb":1803442,"num_write":2100,"num_write_kb":571044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4193,"ondisk_log_size":4193,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":98464,"kb_used_data":58824,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94273376,"statfs":{"total":96636764160,"available":96535937024,"internally_reserved":0,"allocated":60235776,"data_stored":61151851,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78948,"kb_used_data":43244,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94292892,"statfs":{"total":96636764160,"available":96555921408,"internally_reserved":0,"allocated":44281856,"data_stored":43738040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81876,"kb_used_data":46404,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94289964,"statfs":{"total":96636764160,"available":96552923136,"internally_reserved":0,"allocated":47517696,"data_stored":49045051,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89304,"kb_used_data":53936,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94282536,"statfs":{"total":96636764160,"available":96545316864,"internally_reserved":0,"allocated":55230464,"data_stored":57838207,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:13:16.869 INFO:tasks.rados.rados.0.smithi064.stdout:1337: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:13:17.047 INFO:tasks.rados.rados.0.smithi064.stdout:1335: done (4 left) 2024-03-04T06:13:17.047 INFO:tasks.rados.rados.0.smithi064.stdout:1336: done (3 left) 2024-03-04T06:13:17.047 INFO:tasks.rados.rados.0.smithi064.stdout:1337: done (2 left) 2024-03-04T06:13:17.047 INFO:tasks.rados.rados.0.smithi064.stdout:1338: done (1 left) 2024-03-04T06:13:17.047 INFO:tasks.rados.rados.0.smithi064.stdout:1339: done (0 left) 2024-03-04T06:13:17.047 INFO:tasks.rados.rados.0.smithi064.stdout:1340: delete oid 49 current snap is 156 2024-03-04T06:13:17.051 INFO:tasks.rados.rados.0.smithi064.stdout:1340: done (0 left) 2024-03-04T06:13:17.051 INFO:tasks.rados.rados.0.smithi064.stdout:1341: read oid 48 snap -1 2024-03-04T06:13:17.051 INFO:tasks.rados.rados.0.smithi064.stdout:1341: expect (ObjNum 601 snap 144 seq_num 601) 2024-03-04T06:13:17.051 INFO:tasks.rados.rados.0.smithi064.stdout:1342: write (excl) oid 2 current snap is 156 2024-03-04T06:13:17.051 INFO:tasks.rados.rados.0.smithi064.stdout:1342: seq_num 644 ranges {423708=634008,1852286=439887,2752172=71536} 2024-03-04T06:13:17.056 INFO:tasks.rados.rados.0.smithi064.stdout:1342: writing smithi06431743-2 from 423708 to 1057716 tid 1 2024-03-04T06:13:17.063 INFO:tasks.rados.rados.0.smithi064.stdout:1342: writing smithi06431743-2 from 1852286 to 2292173 tid 2 2024-03-04T06:13:17.066 INFO:tasks.rados.rados.0.smithi064.stdout:1342: writing smithi06431743-2 from 2752172 to 2823708 tid 3 2024-03-04T06:13:17.066 INFO:tasks.rados.rados.0.smithi064.stdout:1343: snap_create 2024-03-04T06:13:17.072 INFO:tasks.rados.rados.0.smithi064.stdout:1341: expect (ObjNum 601 snap 144 seq_num 601) 2024-03-04T06:13:17.264 INFO:tasks.rados.rados.0.smithi064.stdout:1342: finishing write tid 1 to smithi06431743-2 2024-03-04T06:13:17.265 INFO:tasks.rados.rados.0.smithi064.stdout:1342: finishing write tid 2 to smithi06431743-2 2024-03-04T06:13:17.265 INFO:tasks.rados.rados.0.smithi064.stdout:1342: finishing write tid 3 to smithi06431743-2 2024-03-04T06:13:17.265 INFO:tasks.rados.rados.0.smithi064.stdout:1342: finishing write tid 4 to smithi06431743-2 2024-03-04T06:13:17.265 INFO:tasks.rados.rados.0.smithi064.stdout:1342: finishing write tid 5 to smithi06431743-2 2024-03-04T06:13:17.265 INFO:tasks.rados.rados.0.smithi064.stdout:1342: oid 2 updating version 0 to 205 2024-03-04T06:13:17.265 INFO:tasks.rados.rados.0.smithi064.stdout:1342: oid 2 updating version 205 to 208 2024-03-04T06:13:17.265 INFO:tasks.rados.rados.0.smithi064.stdout:1342: oid 2 version 208 is already newer than 207 2024-03-04T06:13:17.265 INFO:tasks.rados.rados.0.smithi064.stdout:1342: oid 2 version 208 is already newer than 207 2024-03-04T06:13:17.265 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 208 (ObjNum 644 snap 156 seq_num 644) dirty exists 2024-03-04T06:13:17.265 INFO:tasks.rados.rados.0.smithi064.stdout:1342: left oid 2 (ObjNum 644 snap 156 seq_num 644) 2024-03-04T06:13:17.356 INFO:tasks.rados.rados.0.smithi064.stdout:1341: done (2 left) 2024-03-04T06:13:17.356 INFO:tasks.rados.rados.0.smithi064.stdout:1342: done (1 left) 2024-03-04T06:13:17.356 INFO:tasks.rados.rados.0.smithi064.stdout:1343: done (0 left) 2024-03-04T06:13:17.357 INFO:tasks.rados.rados.0.smithi064.stdout:1344: write oid 23 current snap is 157 2024-03-04T06:13:17.357 INFO:tasks.rados.rados.0.smithi064.stdout:1344: seq_num 645 ranges {440515=709257,1910768=583545} 2024-03-04T06:13:17.362 INFO:tasks.rados.rados.0.smithi064.stdout:1344: writing smithi06431743-23 from 440515 to 1149772 tid 1 2024-03-04T06:13:17.370 INFO:tasks.rados.rados.0.smithi064.stdout:1344: writing smithi06431743-23 from 1910768 to 2494313 tid 2 2024-03-04T06:13:17.370 INFO:tasks.rados.rados.0.smithi064.stdout:1345: snap_remove snap 155 2024-03-04T06:13:18.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:18.287 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:18.285+0000 7fbd4bd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:18.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:18.288+0000 7fbd4bd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:18.360 INFO:tasks.rados.rados.0.smithi064.stdout:1345: done (1 left) 2024-03-04T06:13:18.360 INFO:tasks.rados.rados.0.smithi064.stdout:1346: delete oid 21 current snap is 157 2024-03-04T06:13:18.360 INFO:tasks.rados.rados.0.smithi064.stdout:1344: finishing write tid 1 to smithi06431743-23 2024-03-04T06:13:18.360 INFO:tasks.rados.rados.0.smithi064.stdout:1344: finishing write tid 2 to smithi06431743-23 2024-03-04T06:13:18.360 INFO:tasks.rados.rados.0.smithi064.stdout:1344: finishing write tid 3 to smithi06431743-23 2024-03-04T06:13:18.360 INFO:tasks.rados.rados.0.smithi064.stdout:1344: finishing write tid 4 to smithi06431743-23 2024-03-04T06:13:18.360 INFO:tasks.rados.rados.0.smithi064.stdout:1344: oid 23 updating version 0 to 280 2024-03-04T06:13:18.360 INFO:tasks.rados.rados.0.smithi064.stdout:1344: oid 23 updating version 280 to 281 2024-03-04T06:13:18.360 INFO:tasks.rados.rados.0.smithi064.stdout:1344: oid 23 updating version 281 to 282 2024-03-04T06:13:18.360 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 282 (ObjNum 645 snap 157 seq_num 645) dirty exists 2024-03-04T06:13:18.360 INFO:tasks.rados.rados.0.smithi064.stdout:1344: left oid 23 (ObjNum 645 snap 157 seq_num 645) 2024-03-04T06:13:18.396 INFO:tasks.rados.rados.0.smithi064.stdout:1344: done (1 left) 2024-03-04T06:13:18.396 INFO:tasks.rados.rados.0.smithi064.stdout:1346: done (0 left) 2024-03-04T06:13:18.396 INFO:tasks.rados.rados.0.smithi064.stdout:1347: write oid 46 current snap is 157 2024-03-04T06:13:18.396 INFO:tasks.rados.rados.0.smithi064.stdout:1347: seq_num 647 ranges {474129=576108,1627732=587214,2718088=156041} 2024-03-04T06:13:18.402 INFO:tasks.rados.rados.0.smithi064.stdout:1347: writing smithi06431743-46 from 474129 to 1050237 tid 1 2024-03-04T06:13:18.410 INFO:tasks.rados.rados.0.smithi064.stdout:1347: writing smithi06431743-46 from 1627732 to 2214946 tid 2 2024-03-04T06:13:18.414 INFO:tasks.rados.rados.0.smithi064.stdout:1347: writing smithi06431743-46 from 2718088 to 2874129 tid 3 2024-03-04T06:13:18.414 INFO:tasks.rados.rados.0.smithi064.stdout:1348: write oid 25 current snap is 157 2024-03-04T06:13:18.415 INFO:tasks.rados.rados.0.smithi064.stdout:1348: seq_num 648 ranges {490936=651357,1686214=447225,2861864=29072} 2024-03-04T06:13:18.422 INFO:tasks.rados.rados.0.smithi064.stdout:1348: writing smithi06431743-25 from 490936 to 1142293 tid 1 2024-03-04T06:13:18.429 INFO:tasks.rados.rados.0.smithi064.stdout:1348: writing smithi06431743-25 from 1686214 to 2133439 tid 2 2024-03-04T06:13:18.434 INFO:tasks.rados.rados.0.smithi064.stdout:1348: writing smithi06431743-25 from 2861864 to 2890936 tid 3 2024-03-04T06:13:18.434 INFO:tasks.rados.rados.0.smithi064.stdout:1349: write (excl) oid 15 current snap is 157 2024-03-04T06:13:18.434 INFO:tasks.rados.rados.0.smithi064.stdout:1349: seq_num 649 ranges {507743=726606,1744696=590883,2772934=134809} 2024-03-04T06:13:18.441 INFO:tasks.rados.rados.0.smithi064.stdout:1349: writing smithi06431743-15 from 507743 to 1234349 tid 1 2024-03-04T06:13:18.447 INFO:tasks.rados.rados.0.smithi064.stdout:1349: writing smithi06431743-15 from 1744696 to 2335579 tid 2 2024-03-04T06:13:18.451 INFO:tasks.rados.rados.0.smithi064.stdout:1349: writing smithi06431743-15 from 2772934 to 2907743 tid 3 2024-03-04T06:13:18.451 INFO:tasks.rados.rados.0.smithi064.stdout:1350: snap_remove snap 150 2024-03-04T06:13:19.369 INFO:tasks.rados.rados.0.smithi064.stdout:1350: done (3 left) 2024-03-04T06:13:19.369 INFO:tasks.rados.rados.0.smithi064.stdout:1351: snap_remove snap 156 2024-03-04T06:13:20.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:20.370 INFO:tasks.rados.rados.0.smithi064.stdout:1351: done (3 left) 2024-03-04T06:13:20.370 INFO:tasks.rados.rados.0.smithi064.stdout:1352: snap_remove snap 152 2024-03-04T06:13:20.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:20.478+0000 7f31c3213640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:20.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:20.483+0000 7f31c3213640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:21.374 INFO:tasks.rados.rados.0.smithi064.stdout:1352: done (3 left) 2024-03-04T06:13:21.375 INFO:tasks.rados.rados.0.smithi064.stdout:1353: delete oid 24 current snap is 157 2024-03-04T06:13:21.375 INFO:tasks.rados.rados.0.smithi064.stdout:1348: finishing write tid 1 to smithi06431743-25 2024-03-04T06:13:21.376 INFO:tasks.rados.rados.0.smithi064.stdout:1347: finishing write tid 1 to smithi06431743-46 2024-03-04T06:13:21.376 INFO:tasks.rados.rados.0.smithi064.stdout:1347: finishing write tid 2 to smithi06431743-46 2024-03-04T06:13:21.376 INFO:tasks.rados.rados.0.smithi064.stdout:1348: finishing write tid 2 to smithi06431743-25 2024-03-04T06:13:21.376 INFO:tasks.rados.rados.0.smithi064.stdout:1347: finishing write tid 3 to smithi06431743-46 2024-03-04T06:13:21.376 INFO:tasks.rados.rados.0.smithi064.stdout:1349: finishing write tid 1 to smithi06431743-15 2024-03-04T06:13:21.376 INFO:tasks.rados.rados.0.smithi064.stdout:1348: finishing write tid 3 to smithi06431743-25 2024-03-04T06:13:21.376 INFO:tasks.rados.rados.0.smithi064.stdout:1347: finishing write tid 4 to smithi06431743-46 2024-03-04T06:13:21.376 INFO:tasks.rados.rados.0.smithi064.stdout:1348: finishing write tid 4 to smithi06431743-25 2024-03-04T06:13:21.376 INFO:tasks.rados.rados.0.smithi064.stdout:1349: finishing write tid 2 to smithi06431743-15 2024-03-04T06:13:21.376 INFO:tasks.rados.rados.0.smithi064.stdout:1347: finishing write tid 5 to smithi06431743-46 2024-03-04T06:13:21.377 INFO:tasks.rados.rados.0.smithi064.stdout:1347: oid 46 updating version 0 to 544 2024-03-04T06:13:21.377 INFO:tasks.rados.rados.0.smithi064.stdout:1347: oid 46 updating version 544 to 546 2024-03-04T06:13:21.377 INFO:tasks.rados.rados.0.smithi064.stdout:1347: oid 46 version 546 is already newer than 546 2024-03-04T06:13:21.377 INFO:tasks.rados.rados.0.smithi064.stdout:1347: oid 46 updating version 546 to 547 2024-03-04T06:13:21.377 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 547 (ObjNum 647 snap 157 seq_num 647) dirty exists 2024-03-04T06:13:21.377 INFO:tasks.rados.rados.0.smithi064.stdout:1347: left oid 46 (ObjNum 647 snap 157 seq_num 647) 2024-03-04T06:13:21.377 INFO:tasks.rados.rados.0.smithi064.stdout:1348: finishing write tid 5 to smithi06431743-25 2024-03-04T06:13:21.377 INFO:tasks.rados.rados.0.smithi064.stdout:1348: oid 25 updating version 0 to 304 2024-03-04T06:13:21.377 INFO:tasks.rados.rados.0.smithi064.stdout:1348: oid 25 version 304 is already newer than 304 2024-03-04T06:13:21.377 INFO:tasks.rados.rados.0.smithi064.stdout:1348: oid 25 version 304 is already newer than 301 2024-03-04T06:13:21.378 INFO:tasks.rados.rados.0.smithi064.stdout:1348: oid 25 version 304 is already newer than 302 2024-03-04T06:13:21.378 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 304 (ObjNum 648 snap 157 seq_num 648) dirty exists 2024-03-04T06:13:21.378 INFO:tasks.rados.rados.0.smithi064.stdout:1348: left oid 25 (ObjNum 648 snap 157 seq_num 648) 2024-03-04T06:13:21.378 INFO:tasks.rados.rados.0.smithi064.stdout:1349: finishing write tid 3 to smithi06431743-15 2024-03-04T06:13:21.378 INFO:tasks.rados.rados.0.smithi064.stdout:1349: finishing write tid 4 to smithi06431743-15 2024-03-04T06:13:21.378 INFO:tasks.rados.rados.0.smithi064.stdout:1349: finishing write tid 5 to smithi06431743-15 2024-03-04T06:13:21.378 INFO:tasks.rados.rados.0.smithi064.stdout:1349: oid 15 updating version 0 to 221 2024-03-04T06:13:21.378 INFO:tasks.rados.rados.0.smithi064.stdout:1349: oid 15 updating version 221 to 222 2024-03-04T06:13:21.378 INFO:tasks.rados.rados.0.smithi064.stdout:1349: oid 15 version 222 is already newer than 220 2024-03-04T06:13:21.378 INFO:tasks.rados.rados.0.smithi064.stdout:1349: oid 15 updating version 222 to 223 2024-03-04T06:13:21.379 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 223 (ObjNum 649 snap 157 seq_num 649) dirty exists 2024-03-04T06:13:21.379 INFO:tasks.rados.rados.0.smithi064.stdout:1349: left oid 15 (ObjNum 649 snap 157 seq_num 649) 2024-03-04T06:13:21.458 INFO:tasks.rados.rados.0.smithi064.stdout:1347: done (3 left) 2024-03-04T06:13:21.458 INFO:tasks.rados.rados.0.smithi064.stdout:1348: done (2 left) 2024-03-04T06:13:21.458 INFO:tasks.rados.rados.0.smithi064.stdout:1349: done (1 left) 2024-03-04T06:13:21.458 INFO:tasks.rados.rados.0.smithi064.stdout:1353: done (0 left) 2024-03-04T06:13:21.458 INFO:tasks.rados.rados.0.smithi064.stdout:1354: read oid 29 snap 154 2024-03-04T06:13:21.458 INFO:tasks.rados.rados.0.smithi064.stdout:1354: expect (ObjNum 609 snap 146 seq_num 609) 2024-03-04T06:13:21.458 INFO:tasks.rados.rados.0.smithi064.stdout:1355: read oid 27 snap -1 2024-03-04T06:13:21.458 INFO:tasks.rados.rados.0.smithi064.stdout:1355: expect deleted 2024-03-04T06:13:21.458 INFO:tasks.rados.rados.0.smithi064.stdout:1356: read oid 12 snap -1 2024-03-04T06:13:21.458 INFO:tasks.rados.rados.0.smithi064.stdout:1356: expect (ObjNum 553 snap 132 seq_num 553) 2024-03-04T06:13:21.459 INFO:tasks.rados.rados.0.smithi064.stdout:1357: read oid 8 snap -1 2024-03-04T06:13:21.459 INFO:tasks.rados.rados.0.smithi064.stdout:1357: expect (ObjNum 585 snap 140 seq_num 585) 2024-03-04T06:13:21.459 INFO:tasks.rados.rados.0.smithi064.stdout:1358: delete oid 23 current snap is 157 2024-03-04T06:13:21.485 INFO:tasks.rados.rados.0.smithi064.stdout:1358: done (4 left) 2024-03-04T06:13:21.485 INFO:tasks.rados.rados.0.smithi064.stdout:1359: write oid 44 current snap is 157 2024-03-04T06:13:21.485 INFO:tasks.rados.rados.0.smithi064.stdout:1359: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2024-03-04T06:13:21.492 INFO:tasks.rados.rados.0.smithi064.stdout:1359: writing smithi06431743-44 from 558164 to 1110517 tid 1 2024-03-04T06:13:21.500 INFO:tasks.rados.rados.0.smithi064.stdout:1359: writing smithi06431743-44 from 1803789 to 2258352 tid 2 2024-03-04T06:13:21.504 INFO:tasks.rados.rados.0.smithi064.stdout:1359: writing smithi06431743-44 from 2855203 to 2958164 tid 3 2024-03-04T06:13:21.504 INFO:tasks.rados.rados.0.smithi064.stdout:1360: snap_create 2024-03-04T06:13:21.528 INFO:tasks.rados.rados.0.smithi064.stdout:1357: expect (ObjNum 585 snap 140 seq_num 585) 2024-03-04T06:13:21.725 INFO:tasks.rados.rados.0.smithi064.stdout:1354: expect (ObjNum 609 snap 146 seq_num 609) 2024-03-04T06:13:21.740 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:13:21.740 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:13:21.740 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:13:21.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:13:21.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:21.844+0000 7f3170d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:21.852 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:21.851+0000 7f3170d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:21.864 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:21.864 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:21.864 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:21.864 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:21.864 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:21.865 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:21.865 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:21.865 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:21.865 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:21.865 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:21.865 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:21.865 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:21.867 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:21.867 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:21.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:13:21.888 INFO:tasks.rados.rados.0.smithi064.stdout:1356: expect (ObjNum 553 snap 132 seq_num 553) 2024-03-04T06:13:21.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:21.985+0000 7fa2bd19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:21.993 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:21.992+0000 7fa2bd19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:22.005 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:22.005 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:22.005 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:22.005 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:22.005 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:22.005 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:22.005 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:22.005 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:22.005 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:22.005 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:22.006 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:22.006 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:22.008 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:22.008 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:22.008 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:13:22.074 INFO:tasks.rados.rados.0.smithi064.stdout:1359: finishing write tid 1 to smithi06431743-44 2024-03-04T06:13:22.074 INFO:tasks.rados.rados.0.smithi064.stdout:1359: finishing write tid 2 to smithi06431743-44 2024-03-04T06:13:22.074 INFO:tasks.rados.rados.0.smithi064.stdout:1359: finishing write tid 3 to smithi06431743-44 2024-03-04T06:13:22.074 INFO:tasks.rados.rados.0.smithi064.stdout:1359: finishing write tid 4 to smithi06431743-44 2024-03-04T06:13:22.075 INFO:tasks.rados.rados.0.smithi064.stdout:1359: finishing write tid 5 to smithi06431743-44 2024-03-04T06:13:22.075 INFO:tasks.rados.rados.0.smithi064.stdout:1359: oid 44 updating version 0 to 226 2024-03-04T06:13:22.075 INFO:tasks.rados.rados.0.smithi064.stdout:1359: oid 44 updating version 226 to 227 2024-03-04T06:13:22.075 INFO:tasks.rados.rados.0.smithi064.stdout:1359: oid 44 version 227 is already newer than 224 2024-03-04T06:13:22.075 INFO:tasks.rados.rados.0.smithi064.stdout:1359: oid 44 version 227 is already newer than 225 2024-03-04T06:13:22.075 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 227 (ObjNum 652 snap 157 seq_num 652) dirty exists 2024-03-04T06:13:22.075 INFO:tasks.rados.rados.0.smithi064.stdout:1359: left oid 44 (ObjNum 652 snap 157 seq_num 652) 2024-03-04T06:13:22.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:22.152+0000 7ff65b66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:22.160 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:22.159+0000 7ff65b66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:22.172 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:22.172 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:22.172 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:22.172 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:22.172 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:22.172 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:22.172 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:22.173 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:22.173 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:22.173 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:22.173 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:22.173 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:22.175 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:22.175 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:13:22.324 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:22.319+0000 7f07af71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:22.327 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:22.326+0000 7f07af71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:22.339 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:22.342 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:22.385 INFO:tasks.rados.rados.0.smithi064.stdout:1354: done (5 left) 2024-03-04T06:13:22.385 INFO:tasks.rados.rados.0.smithi064.stdout:1355: done (4 left) 2024-03-04T06:13:22.385 INFO:tasks.rados.rados.0.smithi064.stdout:1356: done (3 left) 2024-03-04T06:13:22.385 INFO:tasks.rados.rados.0.smithi064.stdout:1357: done (2 left) 2024-03-04T06:13:22.385 INFO:tasks.rados.rados.0.smithi064.stdout:1359: done (1 left) 2024-03-04T06:13:22.385 INFO:tasks.rados.rados.0.smithi064.stdout:1360: done (0 left) 2024-03-04T06:13:22.385 INFO:tasks.rados.rados.0.smithi064.stdout:1361: delete oid 16 current snap is 158 2024-03-04T06:13:22.421 INFO:tasks.rados.rados.0.smithi064.stdout:1361: done (0 left) 2024-03-04T06:13:22.421 INFO:tasks.rados.rados.0.smithi064.stdout:1362: snap_create 2024-03-04T06:13:22.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:22.659 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:22.657+0000 7f8f0f22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:22.662 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:22.661+0000 7f8f0f22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:23.399 INFO:tasks.rados.rados.0.smithi064.stdout:1362: done (0 left) 2024-03-04T06:13:23.399 INFO:tasks.rados.rados.0.smithi064.stdout:1363: write (excl) oid 50 current snap is 159 2024-03-04T06:13:23.400 INFO:tasks.rados.rados.0.smithi064.stdout:1363: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2024-03-04T06:13:23.404 INFO:tasks.rados.rados.0.smithi064.stdout:1363: writing smithi06431743-50 from 591778 to 1010982 tid 1 2024-03-04T06:13:23.412 INFO:tasks.rados.rados.0.smithi064.stdout:1363: writing smithi06431743-50 from 1637106 to 2378985 tid 2 2024-03-04T06:13:23.416 INFO:tasks.rados.rados.0.smithi064.stdout:1363: writing smithi06431743-50 from 2910049 to 2991778 tid 3 2024-03-04T06:13:23.416 INFO:tasks.rados.rados.0.smithi064.stdout:1364: read oid 37 snap 157 2024-03-04T06:13:23.417 INFO:tasks.rados.rados.0.smithi064.stdout:1364: expect deleted 2024-03-04T06:13:23.417 INFO:tasks.rados.rados.0.smithi064.stdout:1365: read oid 36 snap 154 2024-03-04T06:13:23.417 INFO:tasks.rados.rados.0.smithi064.stdout:1365: expect (ObjNum 638 snap 154 seq_num 638) 2024-03-04T06:13:23.417 INFO:tasks.rados.rados.0.smithi064.stdout:1366: read oid 2 snap 157 2024-03-04T06:13:23.417 INFO:tasks.rados.rados.0.smithi064.stdout:1366: expect (ObjNum 644 snap 156 seq_num 644) 2024-03-04T06:13:23.417 INFO:tasks.rados.rados.0.smithi064.stdout:1367: read oid 15 snap -1 2024-03-04T06:13:23.417 INFO:tasks.rados.rados.0.smithi064.stdout:1367: expect (ObjNum 649 snap 157 seq_num 649) 2024-03-04T06:13:23.417 INFO:tasks.rados.rados.0.smithi064.stdout:1368: read oid 9 snap -1 2024-03-04T06:13:23.417 INFO:tasks.rados.rados.0.smithi064.stdout:1368: expect (ObjNum 569 snap 134 seq_num 569) 2024-03-04T06:13:23.417 INFO:tasks.rados.rados.0.smithi064.stdout:1369: snap_create 2024-03-04T06:13:23.460 INFO:tasks.rados.rados.0.smithi064.stdout:1363: finishing write tid 1 to smithi06431743-50 2024-03-04T06:13:23.463 INFO:tasks.rados.rados.0.smithi064.stdout:1365: expect (ObjNum 638 snap 154 seq_num 638) 2024-03-04T06:13:23.569 INFO:tasks.rados.rados.0.smithi064.stdout:1367: expect (ObjNum 649 snap 157 seq_num 649) 2024-03-04T06:13:23.676 INFO:tasks.rados.rados.0.smithi064.stdout:1366: expect (ObjNum 644 snap 156 seq_num 644) 2024-03-04T06:13:23.846 INFO:tasks.rados.rados.0.smithi064.stdout:1363: finishing write tid 2 to smithi06431743-50 2024-03-04T06:13:23.846 INFO:tasks.rados.rados.0.smithi064.stdout:1363: finishing write tid 3 to smithi06431743-50 2024-03-04T06:13:23.846 INFO:tasks.rados.rados.0.smithi064.stdout:1363: finishing write tid 4 to smithi06431743-50 2024-03-04T06:13:23.846 INFO:tasks.rados.rados.0.smithi064.stdout:1363: finishing write tid 5 to smithi06431743-50 2024-03-04T06:13:23.846 INFO:tasks.rados.rados.0.smithi064.stdout:1363: oid 50 updating version 0 to 453 2024-03-04T06:13:23.846 INFO:tasks.rados.rados.0.smithi064.stdout:1363: oid 50 updating version 453 to 456 2024-03-04T06:13:23.846 INFO:tasks.rados.rados.0.smithi064.stdout:1363: oid 50 version 456 is already newer than 454 2024-03-04T06:13:23.846 INFO:tasks.rados.rados.0.smithi064.stdout:1363: oid 50 version 456 is already newer than 456 2024-03-04T06:13:23.846 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 456 (ObjNum 654 snap 159 seq_num 654) dirty exists 2024-03-04T06:13:23.846 INFO:tasks.rados.rados.0.smithi064.stdout:1363: left oid 50 (ObjNum 654 snap 159 seq_num 654) 2024-03-04T06:13:23.846 INFO:tasks.rados.rados.0.smithi064.stdout:1368: expect (ObjNum 569 snap 134 seq_num 569) 2024-03-04T06:13:24.398 INFO:tasks.rados.rados.0.smithi064.stdout:1363: done (6 left) 2024-03-04T06:13:24.398 INFO:tasks.rados.rados.0.smithi064.stdout:1364: done (5 left) 2024-03-04T06:13:24.398 INFO:tasks.rados.rados.0.smithi064.stdout:1365: done (4 left) 2024-03-04T06:13:24.398 INFO:tasks.rados.rados.0.smithi064.stdout:1366: done (3 left) 2024-03-04T06:13:24.399 INFO:tasks.rados.rados.0.smithi064.stdout:1367: done (2 left) 2024-03-04T06:13:24.399 INFO:tasks.rados.rados.0.smithi064.stdout:1368: done (1 left) 2024-03-04T06:13:24.399 INFO:tasks.rados.rados.0.smithi064.stdout:1369: done (0 left) 2024-03-04T06:13:24.399 INFO:tasks.rados.rados.0.smithi064.stdout:1370: snap_remove snap 159 2024-03-04T06:13:24.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:24.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:24.837+0000 7f0d7a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:24.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:24.841+0000 7f0d7a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:25.402 INFO:tasks.rados.rados.0.smithi064.stdout:1370: done (0 left) 2024-03-04T06:13:25.402 INFO:tasks.rados.rados.0.smithi064.stdout:1371: read oid 40 snap -1 2024-03-04T06:13:25.402 INFO:tasks.rados.rados.0.smithi064.stdout:1371: expect deleted 2024-03-04T06:13:25.402 INFO:tasks.rados.rados.0.smithi064.stdout:1372: snap_create 2024-03-04T06:13:26.403 INFO:tasks.rados.rados.0.smithi064.stdout:1371: done (1 left) 2024-03-04T06:13:26.403 INFO:tasks.rados.rados.0.smithi064.stdout:1372: done (0 left) 2024-03-04T06:13:26.403 INFO:tasks.rados.rados.0.smithi064.stdout:1373: snap_remove snap 157 2024-03-04T06:13:26.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:27.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:27.022+0000 7fd28a95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:27.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:27.026+0000 7fd28a95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:27.343 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:13:27.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:13:27.403 INFO:tasks.rados.rados.0.smithi064.stdout:1373: done (0 left) 2024-03-04T06:13:27.403 INFO:tasks.rados.rados.0.smithi064.stdout:1374: snap_create 2024-03-04T06:13:27.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:27.455+0000 7f80ff154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:27.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:27.463+0000 7f80ff154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:28.402 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:13:28.407 INFO:tasks.rados.rados.0.smithi064.stdout:1374: done (0 left) 2024-03-04T06:13:28.407 INFO:tasks.rados.rados.0.smithi064.stdout:1375: write (excl) oid 17 current snap is 162 2024-03-04T06:13:28.408 INFO:tasks.rados.rados.0.smithi064.stdout:1375: seq_num 655 ranges {608585=494453,1579235=601890,2821119=187466} 2024-03-04T06:13:28.414 INFO:tasks.rados.rados.0.smithi064.stdout:1375: writing smithi06431743-17 from 608585 to 1103038 tid 1 2024-03-04T06:13:28.420 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:13:28.423 INFO:tasks.rados.rados.0.smithi064.stdout:1375: writing smithi06431743-17 from 1579235 to 2181125 tid 2 2024-03-04T06:13:28.429 INFO:tasks.rados.rados.0.smithi064.stdout:1375: writing smithi06431743-17 from 2821119 to 3008585 tid 3 2024-03-04T06:13:28.429 INFO:tasks.rados.rados.0.smithi064.stdout:1376: snap_create 2024-03-04T06:13:28.475 INFO:tasks.rados.rados.0.smithi064.stdout:1375: finishing write tid 1 to smithi06431743-17 2024-03-04T06:13:28.489 INFO:tasks.rados.rados.0.smithi064.stdout:1375: finishing write tid 2 to smithi06431743-17 2024-03-04T06:13:28.502 INFO:tasks.rados.rados.0.smithi064.stdout:1375: finishing write tid 3 to smithi06431743-17 2024-03-04T06:13:28.508 INFO:tasks.rados.rados.0.smithi064.stdout:1375: finishing write tid 4 to smithi06431743-17 2024-03-04T06:13:28.509 INFO:tasks.rados.rados.0.smithi064.stdout:1375: finishing write tid 5 to smithi06431743-17 2024-03-04T06:13:28.509 INFO:tasks.rados.rados.0.smithi064.stdout:1375: oid 17 updating version 0 to 168 2024-03-04T06:13:28.509 INFO:tasks.rados.rados.0.smithi064.stdout:1375: oid 17 updating version 168 to 170 2024-03-04T06:13:28.509 INFO:tasks.rados.rados.0.smithi064.stdout:1375: oid 17 updating version 170 to 171 2024-03-04T06:13:28.509 INFO:tasks.rados.rados.0.smithi064.stdout:1375: oid 17 version 171 is already newer than 169 2024-03-04T06:13:28.509 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 171 (ObjNum 655 snap 162 seq_num 655) dirty exists 2024-03-04T06:13:28.509 INFO:tasks.rados.rados.0.smithi064.stdout:1375: left oid 17 (ObjNum 655 snap 162 seq_num 655) 2024-03-04T06:13:28.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:28.533+0000 7f76db15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:28.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:28.541+0000 7f76db15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:29.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:29.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:29.201+0000 7fa6fd79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:29.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:29.206+0000 7fa6fd79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:29.413 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:13:29.417 INFO:tasks.rados.rados.0.smithi064.stdout:1375: done (1 left) 2024-03-04T06:13:29.417 INFO:tasks.rados.rados.0.smithi064.stdout:1376: done (0 left) 2024-03-04T06:13:29.417 INFO:tasks.rados.rados.0.smithi064.stdout:1377: delete oid 13 current snap is 163 2024-03-04T06:13:29.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:13:29.500 INFO:tasks.rados.rados.0.smithi064.stdout:1377: done (0 left) 2024-03-04T06:13:29.500 INFO:tasks.rados.rados.0.smithi064.stdout:1378: snap_create 2024-03-04T06:13:29.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:29.529+0000 7f00dfe17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:29.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:29.538+0000 7f00dfe17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:30.414 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:13:30.418 INFO:tasks.rados.rados.0.smithi064.stdout:1378: done (0 left) 2024-03-04T06:13:30.419 INFO:tasks.rados.rados.0.smithi064.stdout:1379: snap_remove snap 154 2024-03-04T06:13:30.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:13:30.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:30.534+0000 7f06dc42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:30.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:30.542+0000 7f06dc42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:31.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:31.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:31.382+0000 7fbcf3d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:31.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:31.387+0000 7fbcf3d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:31.415 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:13:31.418 INFO:tasks.rados.rados.0.smithi064.stdout:1379: done (0 left) 2024-03-04T06:13:31.418 INFO:tasks.rados.rados.0.smithi064.stdout:1380: delete oid 1 current snap is 164 2024-03-04T06:13:31.430 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:13:31.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:13:31.481 INFO:tasks.rados.rados.0.smithi064.stdout:1380: done (0 left) 2024-03-04T06:13:31.481 INFO:tasks.rados.rados.0.smithi064.stdout:1381: delete oid 38 current snap is 164 2024-03-04T06:13:31.499 INFO:tasks.rados.rados.0.smithi064.stdout:1381: done (0 left) 2024-03-04T06:13:31.499 INFO:tasks.rados.rados.0.smithi064.stdout:1382: write (excl) oid 33 current snap is 164 2024-03-04T06:13:31.499 INFO:tasks.rados.rados.0.smithi064.stdout:1382: seq_num 659 ranges {675813=795449,2096810=609228} 2024-03-04T06:13:31.507 INFO:tasks.rados.rados.0.smithi064.stdout:1382: writing smithi06431743-33 from 675813 to 1471262 tid 1 2024-03-04T06:13:31.514 INFO:tasks.rados.rados.0.smithi064.stdout:1382: writing smithi06431743-33 from 2096810 to 2706038 tid 2 2024-03-04T06:13:31.514 INFO:tasks.rados.rados.0.smithi064.stdout:1383: snap_create 2024-03-04T06:13:31.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:31.547+0000 7ffb390ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:31.553+0000 7ffb390ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:31.587 INFO:tasks.rados.rados.0.smithi064.stdout:1382: finishing write tid 1 to smithi06431743-33 2024-03-04T06:13:31.598 INFO:tasks.rados.rados.0.smithi064.stdout:1382: finishing write tid 2 to smithi06431743-33 2024-03-04T06:13:31.607 INFO:tasks.rados.rados.0.smithi064.stdout:1382: finishing write tid 3 to smithi06431743-33 2024-03-04T06:13:31.608 INFO:tasks.rados.rados.0.smithi064.stdout:1382: finishing write tid 4 to smithi06431743-33 2024-03-04T06:13:31.608 INFO:tasks.rados.rados.0.smithi064.stdout:1382: oid 33 updating version 0 to 460 2024-03-04T06:13:31.608 INFO:tasks.rados.rados.0.smithi064.stdout:1382: oid 33 version 460 is already newer than 458 2024-03-04T06:13:31.608 INFO:tasks.rados.rados.0.smithi064.stdout:1382: oid 33 version 460 is already newer than 459 2024-03-04T06:13:31.608 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 460 (ObjNum 659 snap 164 seq_num 659) dirty exists 2024-03-04T06:13:31.608 INFO:tasks.rados.rados.0.smithi064.stdout:1382: left oid 33 (ObjNum 659 snap 164 seq_num 659) 2024-03-04T06:13:31.744 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:13:31.744 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:13:31.756 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-03-04T06:13:31.416237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92444418,"num_objects":462,"num_object_clones":341,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":87,"num_read":2594,"num_read_kb":1891713,"num_write":2290,"num_write_kb":585096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4435,"ondisk_log_size":4435,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":341824,"kb_used_data":197088,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377145536,"statfs":{"total":386547056640,"available":386197028864,"internally_reserved":0,"allocated":201818112,"data_stored":206782655,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3781641,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014071"},"pg_stats":[{"pgid":"3.f","version":"320'234","reported_seq":1138,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.890938+0000","last_change":"2024-03-04T06:13:28.449431+0000","last_active":"2024-03-04T06:13:29.890938+0000","last_peered":"2024-03-04T06:13:29.890938+0000","last_clean":"2024-03-04T06:13:29.890938+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:13:29.890938+0000","last_undegraded":"2024-03-04T06:13:29.890938+0000","last_fullsized":"2024-03-04T06:13:29.890938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496087,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":114,"num_read_kb":76210,"num_write":131,"num_write_kb":35117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"331'199","reported_seq":1121,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.448407+0000","last_change":"2024-03-04T06:13:29.471419+0000","last_active":"2024-03-04T06:13:29.448407+0000","last_peered":"2024-03-04T06:13:29.448407+0000","last_clean":"2024-03-04T06:13:29.448407+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:13:29.448407+0000","last_undegraded":"2024-03-04T06:13:29.448407+0000","last_fullsized":"2024-03-04T06:13:29.448407+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":16,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":85,"num_read_kb":57066,"num_write":86,"num_write_kb":21710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"331'314","reported_seq":1191,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.891830+0000","last_change":"2024-03-04T06:13:27.418862+0000","last_active":"2024-03-04T06:13:29.891830+0000","last_peered":"2024-03-04T06:13:29.891830+0000","last_clean":"2024-03-04T06:13:29.891830+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:13:29.891830+0000","last_undegraded":"2024-03-04T06:13:29.891830+0000","last_fullsized":"2024-03-04T06:13:29.891830+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7731583,"num_objects":36,"num_object_clones":28,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":171,"num_read_kb":119631,"num_write":166,"num_write_kb":45510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"331'214","reported_seq":1147,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.884957+0000","last_change":"2024-03-04T06:13:28.438358+0000","last_active":"2024-03-04T06:13:29.884957+0000","last_peered":"2024-03-04T06:13:29.884957+0000","last_clean":"2024-03-04T06:13:29.884957+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:13:29.884957+0000","last_undegraded":"2024-03-04T06:13:29.884957+0000","last_fullsized":"2024-03-04T06:13:29.884957+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9059303,"num_objects":23,"num_object_clones":15,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":139,"num_read_kb":112624,"num_write":124,"num_write_kb":35266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"312'98","reported_seq":1033,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.440314+0000","last_change":"2024-03-04T06:13:29.476735+0000","last_active":"2024-03-04T06:13:30.440314+0000","last_peered":"2024-03-04T06:13:30.440314+0000","last_clean":"2024-03-04T06:13:30.440314+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:13:30.440314+0000","last_undegraded":"2024-03-04T06:13:30.440314+0000","last_fullsized":"2024-03-04T06:13:30.440314+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2420340,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":17043,"num_write":42,"num_write_kb":10500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"331'227","reported_seq":1145,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.894326+0000","last_change":"2024-03-04T06:13:28.449042+0000","last_active":"2024-03-04T06:13:29.894326+0000","last_peered":"2024-03-04T06:13:29.894326+0000","last_clean":"2024-03-04T06:13:29.894326+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:13:29.894326+0000","last_undegraded":"2024-03-04T06:13:29.894326+0000","last_fullsized":"2024-03-04T06:13:29.894326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8824038,"num_objects":25,"num_object_clones":18,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":129,"num_read_kb":94389,"num_write":131,"num_write_kb":38676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":972,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.895785+0000","last_change":"2024-03-04T06:13:28.449704+0000","last_active":"2024-03-04T06:13:29.895785+0000","last_peered":"2024-03-04T06:13:29.895785+0000","last_clean":"2024-03-04T06:13:29.895785+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:13:29.895785+0000","last_undegraded":"2024-03-04T06:13:29.895785+0000","last_fullsized":"2024-03-04T06:13:29.895785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"338'171","reported_seq":1103,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.890624+0000","last_change":"2024-03-04T06:13:27.419129+0000","last_active":"2024-03-04T06:13:29.890624+0000","last_peered":"2024-03-04T06:13:29.890624+0000","last_clean":"2024-03-04T06:13:29.890624+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:13:29.890624+0000","last_undegraded":"2024-03-04T06:13:29.890624+0000","last_fullsized":"2024-03-04T06:13:29.890624+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3008585,"num_objects":20,"num_object_clones":13,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":127,"num_read_kb":117184,"num_write":84,"num_write_kb":22492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":965,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.433027+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:13:30.433027+0000","last_peered":"2024-03-04T06:13:30.433027+0000","last_clean":"2024-03-04T06:13:30.433027+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:13:30.433027+0000","last_undegraded":"2024-03-04T06:13:30.433027+0000","last_fullsized":"2024-03-04T06:13:30.433027+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":821,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.898103+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:13:29.898103+0000","last_peered":"2024-03-04T06:13:29.898103+0000","last_clean":"2024-03-04T06:13:29.898103+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:13:29.898103+0000","last_undegraded":"2024-03-04T06:13:29.898103+0000","last_fullsized":"2024-03-04T06:13:29.898103+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"331'297","reported_seq":1160,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.897520+0000","last_change":"2024-03-04T06:13:27.429065+0000","last_active":"2024-03-04T06:13:29.897520+0000","last_peered":"2024-03-04T06:13:29.897520+0000","last_clean":"2024-03-04T06:13:29.897520+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:13:29.897520+0000","last_undegraded":"2024-03-04T06:13:29.897520+0000","last_fullsized":"2024-03-04T06:13:29.897520+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706059,"num_objects":30,"num_object_clones":23,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":151,"num_read_kb":109568,"num_write":138,"num_write_kb":35375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":823,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.904353+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:13:29.904353+0000","last_peered":"2024-03-04T06:13:29.904353+0000","last_clean":"2024-03-04T06:13:29.904353+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:13:29.904353+0000","last_undegraded":"2024-03-04T06:13:29.904353+0000","last_fullsized":"2024-03-04T06:13:29.904353+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"311'235","reported_seq":1120,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.434223+0000","last_change":"2024-03-04T06:13:29.454613+0000","last_active":"2024-03-04T06:13:30.434223+0000","last_peered":"2024-03-04T06:13:30.434223+0000","last_clean":"2024-03-04T06:13:30.434223+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:13:30.434223+0000","last_undegraded":"2024-03-04T06:13:30.434223+0000","last_fullsized":"2024-03-04T06:13:30.434223+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899488,"num_objects":23,"num_object_clones":18,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":122,"num_read_kb":100808,"num_write":96,"num_write_kb":28103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":821,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.889276+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:13:29.889276+0000","last_peered":"2024-03-04T06:13:29.889276+0000","last_clean":"2024-03-04T06:13:29.889276+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:13:29.889276+0000","last_undegraded":"2024-03-04T06:13:29.889276+0000","last_fullsized":"2024-03-04T06:13:29.889276+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"331'553","reported_seq":1391,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.439336+0000","last_change":"2024-03-04T06:13:29.476213+0000","last_active":"2024-03-04T06:13:30.439336+0000","last_peered":"2024-03-04T06:13:30.439336+0000","last_clean":"2024-03-04T06:13:30.439336+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:13:30.439336+0000","last_undegraded":"2024-03-04T06:13:30.439336+0000","last_fullsized":"2024-03-04T06:13:30.439336+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9664553,"num_objects":56,"num_object_clones":42,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":350,"num_read_kb":285928,"num_write":282,"num_write_kb":76369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":823,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.888023+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:13:29.888023+0000","last_peered":"2024-03-04T06:13:29.888023+0000","last_clean":"2024-03-04T06:13:29.888023+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:13:29.888023+0000","last_undegraded":"2024-03-04T06:13:29.888023+0000","last_fullsized":"2024-03-04T06:13:29.888023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"331'527","reported_seq":1331,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.470778+0000","last_change":"2024-03-04T06:13:29.470778+0000","last_active":"2024-03-04T06:13:29.470778+0000","last_peered":"2024-03-04T06:13:29.470778+0000","last_clean":"2024-03-04T06:13:29.470778+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:13:29.470778+0000","last_undegraded":"2024-03-04T06:13:29.470778+0000","last_fullsized":"2024-03-04T06:13:29.470778+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7681162,"num_objects":54,"num_object_clones":40,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":228,"num_read_kb":160285,"num_write":253,"num_write_kb":66344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"328'223","reported_seq":1166,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.897303+0000","last_change":"2024-03-04T06:13:27.429062+0000","last_active":"2024-03-04T06:13:29.897303+0000","last_peered":"2024-03-04T06:13:29.897303+0000","last_clean":"2024-03-04T06:13:29.897303+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:13:29.897303+0000","last_undegraded":"2024-03-04T06:13:29.897303+0000","last_fullsized":"2024-03-04T06:13:29.897303+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2907743,"num_objects":25,"num_object_clones":18,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":202,"num_read_kb":164663,"num_write":115,"num_write_kb":30280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":859,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.478239+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:13:29.478239+0000","last_peered":"2024-03-04T06:13:29.478239+0000","last_clean":"2024-03-04T06:13:29.478239+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:13:29.478239+0000","last_undegraded":"2024-03-04T06:13:29.478239+0000","last_fullsized":"2024-03-04T06:13:29.478239+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"332'242","reported_seq":1176,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.479785+0000","last_change":"2024-03-04T06:13:29.484085+0000","last_active":"2024-03-04T06:13:29.479785+0000","last_peered":"2024-03-04T06:13:29.479785+0000","last_clean":"2024-03-04T06:13:29.479785+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:13:29.479785+0000","last_undegraded":"2024-03-04T06:13:29.479785+0000","last_fullsized":"2024-03-04T06:13:29.479785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554961,"num_objects":30,"num_object_clones":19,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":138,"num_read_kb":85725,"num_write":122,"num_write_kb":32368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":832,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.439790+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:13:30.439790+0000","last_peered":"2024-03-04T06:13:30.439790+0000","last_clean":"2024-03-04T06:13:30.439790+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:13:30.439790+0000","last_undegraded":"2024-03-04T06:13:30.439790+0000","last_fullsized":"2024-03-04T06:13:30.439790+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"331'348","reported_seq":1224,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.431113+0000","last_change":"2024-03-04T06:13:29.455004+0000","last_active":"2024-03-04T06:13:30.431113+0000","last_peered":"2024-03-04T06:13:30.431113+0000","last_clean":"2024-03-04T06:13:30.431113+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:13:30.431113+0000","last_undegraded":"2024-03-04T06:13:30.431113+0000","last_fullsized":"2024-03-04T06:13:30.431113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840647,"num_objects":36,"num_object_clones":27,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":195,"num_read_kb":136022,"num_write":177,"num_write_kb":48365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":859,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.449557+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:13:29.449557+0000","last_peered":"2024-03-04T06:13:29.449557+0000","last_clean":"2024-03-04T06:13:29.449557+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:13:29.449557+0000","last_undegraded":"2024-03-04T06:13:29.449557+0000","last_fullsized":"2024-03-04T06:13:29.449557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"333'456","reported_seq":1331,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.464677+0000","last_change":"2024-03-04T06:13:29.471568+0000","last_active":"2024-03-04T06:13:29.464677+0000","last_peered":"2024-03-04T06:13:29.464677+0000","last_clean":"2024-03-04T06:13:29.464677+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:13:29.464677+0000","last_undegraded":"2024-03-04T06:13:29.464677+0000","last_fullsized":"2024-03-04T06:13:29.464677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13059501,"num_objects":40,"num_object_clones":35,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":305,"num_read_kb":254476,"num_write":206,"num_write_kb":56727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":821,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.899402+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:13:29.899402+0000","last_peered":"2024-03-04T06:13:29.899402+0000","last_clean":"2024-03-04T06:13:29.899402+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:13:29.899402+0000","last_undegraded":"2024-03-04T06:13:29.899402+0000","last_fullsized":"2024-03-04T06:13:29.899402+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91854050,"num_objects":460,"num_object_clones":341,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":87,"num_read":2491,"num_read_kb":1891622,"num_write":2153,"num_write_kb":583202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4338,"ondisk_log_size":4338,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99404,"kb_used_data":60100,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94272436,"statfs":{"total":96636764160,"available":96534974464,"internally_reserved":0,"allocated":61542400,"data_stored":63089018,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81132,"kb_used_data":45988,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94290708,"statfs":{"total":96636764160,"available":96553684992,"internally_reserved":0,"allocated":47091712,"data_stored":48038234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79876,"kb_used_data":44732,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94291964,"statfs":{"total":96636764160,"available":96554971136,"internally_reserved":0,"allocated":45805568,"data_stored":47121895,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81412,"kb_used_data":46268,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94290428,"statfs":{"total":96636764160,"available":96553398272,"internally_reserved":0,"allocated":47378432,"data_stored":48533508,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:13:31.757 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:13:31.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:31.898+0000 7fe567f10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:31.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:31.904+0000 7fe567f10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:32.092 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:13:32.092 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:13:32.104 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-03-04T06:13:31.416237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92444418,"num_objects":462,"num_object_clones":341,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":87,"num_read":2594,"num_read_kb":1891713,"num_write":2290,"num_write_kb":585096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4435,"ondisk_log_size":4435,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":341824,"kb_used_data":197088,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377145536,"statfs":{"total":386547056640,"available":386197028864,"internally_reserved":0,"allocated":201818112,"data_stored":206782655,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3781641,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014071"},"pg_stats":[{"pgid":"3.f","version":"320'234","reported_seq":1138,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.890938+0000","last_change":"2024-03-04T06:13:28.449431+0000","last_active":"2024-03-04T06:13:29.890938+0000","last_peered":"2024-03-04T06:13:29.890938+0000","last_clean":"2024-03-04T06:13:29.890938+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:13:29.890938+0000","last_undegraded":"2024-03-04T06:13:29.890938+0000","last_fullsized":"2024-03-04T06:13:29.890938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496087,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":114,"num_read_kb":76210,"num_write":131,"num_write_kb":35117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"331'199","reported_seq":1121,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.448407+0000","last_change":"2024-03-04T06:13:29.471419+0000","last_active":"2024-03-04T06:13:29.448407+0000","last_peered":"2024-03-04T06:13:29.448407+0000","last_clean":"2024-03-04T06:13:29.448407+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:13:29.448407+0000","last_undegraded":"2024-03-04T06:13:29.448407+0000","last_fullsized":"2024-03-04T06:13:29.448407+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":16,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":85,"num_read_kb":57066,"num_write":86,"num_write_kb":21710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"331'314","reported_seq":1191,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.891830+0000","last_change":"2024-03-04T06:13:27.418862+0000","last_active":"2024-03-04T06:13:29.891830+0000","last_peered":"2024-03-04T06:13:29.891830+0000","last_clean":"2024-03-04T06:13:29.891830+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:13:29.891830+0000","last_undegraded":"2024-03-04T06:13:29.891830+0000","last_fullsized":"2024-03-04T06:13:29.891830+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7731583,"num_objects":36,"num_object_clones":28,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":171,"num_read_kb":119631,"num_write":166,"num_write_kb":45510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"331'214","reported_seq":1147,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.884957+0000","last_change":"2024-03-04T06:13:28.438358+0000","last_active":"2024-03-04T06:13:29.884957+0000","last_peered":"2024-03-04T06:13:29.884957+0000","last_clean":"2024-03-04T06:13:29.884957+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:13:29.884957+0000","last_undegraded":"2024-03-04T06:13:29.884957+0000","last_fullsized":"2024-03-04T06:13:29.884957+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9059303,"num_objects":23,"num_object_clones":15,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":139,"num_read_kb":112624,"num_write":124,"num_write_kb":35266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"312'98","reported_seq":1033,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.440314+0000","last_change":"2024-03-04T06:13:29.476735+0000","last_active":"2024-03-04T06:13:30.440314+0000","last_peered":"2024-03-04T06:13:30.440314+0000","last_clean":"2024-03-04T06:13:30.440314+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:13:30.440314+0000","last_undegraded":"2024-03-04T06:13:30.440314+0000","last_fullsized":"2024-03-04T06:13:30.440314+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2420340,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":17043,"num_write":42,"num_write_kb":10500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"331'227","reported_seq":1145,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.894326+0000","last_change":"2024-03-04T06:13:28.449042+0000","last_active":"2024-03-04T06:13:29.894326+0000","last_peered":"2024-03-04T06:13:29.894326+0000","last_clean":"2024-03-04T06:13:29.894326+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:13:29.894326+0000","last_undegraded":"2024-03-04T06:13:29.894326+0000","last_fullsized":"2024-03-04T06:13:29.894326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8824038,"num_objects":25,"num_object_clones":18,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":129,"num_read_kb":94389,"num_write":131,"num_write_kb":38676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":972,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.895785+0000","last_change":"2024-03-04T06:13:28.449704+0000","last_active":"2024-03-04T06:13:29.895785+0000","last_peered":"2024-03-04T06:13:29.895785+0000","last_clean":"2024-03-04T06:13:29.895785+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:13:29.895785+0000","last_undegraded":"2024-03-04T06:13:29.895785+0000","last_fullsized":"2024-03-04T06:13:29.895785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"338'171","reported_seq":1103,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.890624+0000","last_change":"2024-03-04T06:13:27.419129+0000","last_active":"2024-03-04T06:13:29.890624+0000","last_peered":"2024-03-04T06:13:29.890624+0000","last_clean":"2024-03-04T06:13:29.890624+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:13:29.890624+0000","last_undegraded":"2024-03-04T06:13:29.890624+0000","last_fullsized":"2024-03-04T06:13:29.890624+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3008585,"num_objects":20,"num_object_clones":13,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":127,"num_read_kb":117184,"num_write":84,"num_write_kb":22492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":965,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.433027+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:13:30.433027+0000","last_peered":"2024-03-04T06:13:30.433027+0000","last_clean":"2024-03-04T06:13:30.433027+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:13:30.433027+0000","last_undegraded":"2024-03-04T06:13:30.433027+0000","last_fullsized":"2024-03-04T06:13:30.433027+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":821,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.898103+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:13:29.898103+0000","last_peered":"2024-03-04T06:13:29.898103+0000","last_clean":"2024-03-04T06:13:29.898103+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:13:29.898103+0000","last_undegraded":"2024-03-04T06:13:29.898103+0000","last_fullsized":"2024-03-04T06:13:29.898103+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"331'297","reported_seq":1160,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.897520+0000","last_change":"2024-03-04T06:13:27.429065+0000","last_active":"2024-03-04T06:13:29.897520+0000","last_peered":"2024-03-04T06:13:29.897520+0000","last_clean":"2024-03-04T06:13:29.897520+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:13:29.897520+0000","last_undegraded":"2024-03-04T06:13:29.897520+0000","last_fullsized":"2024-03-04T06:13:29.897520+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706059,"num_objects":30,"num_object_clones":23,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":151,"num_read_kb":109568,"num_write":138,"num_write_kb":35375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":823,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.904353+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:13:29.904353+0000","last_peered":"2024-03-04T06:13:29.904353+0000","last_clean":"2024-03-04T06:13:29.904353+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:13:29.904353+0000","last_undegraded":"2024-03-04T06:13:29.904353+0000","last_fullsized":"2024-03-04T06:13:29.904353+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"311'235","reported_seq":1120,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.434223+0000","last_change":"2024-03-04T06:13:29.454613+0000","last_active":"2024-03-04T06:13:30.434223+0000","last_peered":"2024-03-04T06:13:30.434223+0000","last_clean":"2024-03-04T06:13:30.434223+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:13:30.434223+0000","last_undegraded":"2024-03-04T06:13:30.434223+0000","last_fullsized":"2024-03-04T06:13:30.434223+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899488,"num_objects":23,"num_object_clones":18,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":122,"num_read_kb":100808,"num_write":96,"num_write_kb":28103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":821,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.889276+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:13:29.889276+0000","last_peered":"2024-03-04T06:13:29.889276+0000","last_clean":"2024-03-04T06:13:29.889276+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:13:29.889276+0000","last_undegraded":"2024-03-04T06:13:29.889276+0000","last_fullsized":"2024-03-04T06:13:29.889276+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"331'553","reported_seq":1391,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.439336+0000","last_change":"2024-03-04T06:13:29.476213+0000","last_active":"2024-03-04T06:13:30.439336+0000","last_peered":"2024-03-04T06:13:30.439336+0000","last_clean":"2024-03-04T06:13:30.439336+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:13:30.439336+0000","last_undegraded":"2024-03-04T06:13:30.439336+0000","last_fullsized":"2024-03-04T06:13:30.439336+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9664553,"num_objects":56,"num_object_clones":42,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":350,"num_read_kb":285928,"num_write":282,"num_write_kb":76369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":823,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.888023+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:13:29.888023+0000","last_peered":"2024-03-04T06:13:29.888023+0000","last_clean":"2024-03-04T06:13:29.888023+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:13:29.888023+0000","last_undegraded":"2024-03-04T06:13:29.888023+0000","last_fullsized":"2024-03-04T06:13:29.888023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"331'527","reported_seq":1331,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.470778+0000","last_change":"2024-03-04T06:13:29.470778+0000","last_active":"2024-03-04T06:13:29.470778+0000","last_peered":"2024-03-04T06:13:29.470778+0000","last_clean":"2024-03-04T06:13:29.470778+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:13:29.470778+0000","last_undegraded":"2024-03-04T06:13:29.470778+0000","last_fullsized":"2024-03-04T06:13:29.470778+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7681162,"num_objects":54,"num_object_clones":40,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":228,"num_read_kb":160285,"num_write":253,"num_write_kb":66344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"328'223","reported_seq":1166,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.897303+0000","last_change":"2024-03-04T06:13:27.429062+0000","last_active":"2024-03-04T06:13:29.897303+0000","last_peered":"2024-03-04T06:13:29.897303+0000","last_clean":"2024-03-04T06:13:29.897303+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:13:29.897303+0000","last_undegraded":"2024-03-04T06:13:29.897303+0000","last_fullsized":"2024-03-04T06:13:29.897303+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2907743,"num_objects":25,"num_object_clones":18,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":202,"num_read_kb":164663,"num_write":115,"num_write_kb":30280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":859,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.478239+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:13:29.478239+0000","last_peered":"2024-03-04T06:13:29.478239+0000","last_clean":"2024-03-04T06:13:29.478239+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:13:29.478239+0000","last_undegraded":"2024-03-04T06:13:29.478239+0000","last_fullsized":"2024-03-04T06:13:29.478239+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"332'242","reported_seq":1176,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.479785+0000","last_change":"2024-03-04T06:13:29.484085+0000","last_active":"2024-03-04T06:13:29.479785+0000","last_peered":"2024-03-04T06:13:29.479785+0000","last_clean":"2024-03-04T06:13:29.479785+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:13:29.479785+0000","last_undegraded":"2024-03-04T06:13:29.479785+0000","last_fullsized":"2024-03-04T06:13:29.479785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554961,"num_objects":30,"num_object_clones":19,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":138,"num_read_kb":85725,"num_write":122,"num_write_kb":32368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":832,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.439790+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:13:30.439790+0000","last_peered":"2024-03-04T06:13:30.439790+0000","last_clean":"2024-03-04T06:13:30.439790+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:13:30.439790+0000","last_undegraded":"2024-03-04T06:13:30.439790+0000","last_fullsized":"2024-03-04T06:13:30.439790+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"331'348","reported_seq":1224,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.431113+0000","last_change":"2024-03-04T06:13:29.455004+0000","last_active":"2024-03-04T06:13:30.431113+0000","last_peered":"2024-03-04T06:13:30.431113+0000","last_clean":"2024-03-04T06:13:30.431113+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:13:30.431113+0000","last_undegraded":"2024-03-04T06:13:30.431113+0000","last_fullsized":"2024-03-04T06:13:30.431113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840647,"num_objects":36,"num_object_clones":27,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":195,"num_read_kb":136022,"num_write":177,"num_write_kb":48365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":859,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.449557+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:13:29.449557+0000","last_peered":"2024-03-04T06:13:29.449557+0000","last_clean":"2024-03-04T06:13:29.449557+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:13:29.449557+0000","last_undegraded":"2024-03-04T06:13:29.449557+0000","last_fullsized":"2024-03-04T06:13:29.449557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"333'456","reported_seq":1331,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.464677+0000","last_change":"2024-03-04T06:13:29.471568+0000","last_active":"2024-03-04T06:13:29.464677+0000","last_peered":"2024-03-04T06:13:29.464677+0000","last_clean":"2024-03-04T06:13:29.464677+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:13:29.464677+0000","last_undegraded":"2024-03-04T06:13:29.464677+0000","last_fullsized":"2024-03-04T06:13:29.464677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13059501,"num_objects":40,"num_object_clones":35,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":305,"num_read_kb":254476,"num_write":206,"num_write_kb":56727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":821,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.899402+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:13:29.899402+0000","last_peered":"2024-03-04T06:13:29.899402+0000","last_clean":"2024-03-04T06:13:29.899402+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:13:29.899402+0000","last_undegraded":"2024-03-04T06:13:29.899402+0000","last_fullsized":"2024-03-04T06:13:29.899402+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91854050,"num_objects":460,"num_object_clones":341,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":87,"num_read":2491,"num_read_kb":1891622,"num_write":2153,"num_write_kb":583202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4338,"ondisk_log_size":4338,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99404,"kb_used_data":60100,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94272436,"statfs":{"total":96636764160,"available":96534974464,"internally_reserved":0,"allocated":61542400,"data_stored":63089018,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81132,"kb_used_data":45988,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94290708,"statfs":{"total":96636764160,"available":96553684992,"internally_reserved":0,"allocated":47091712,"data_stored":48038234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79876,"kb_used_data":44732,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94291964,"statfs":{"total":96636764160,"available":96554971136,"internally_reserved":0,"allocated":45805568,"data_stored":47121895,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81412,"kb_used_data":46268,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94290428,"statfs":{"total":96636764160,"available":96553398272,"internally_reserved":0,"allocated":47378432,"data_stored":48533508,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:13:32.105 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:13:32.105 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:13:32.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:13:32.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:32.209+0000 7f7ab6624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:32.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:32.215+0000 7f7ab6624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:32.425 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:13:32.425 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:13:32.427 INFO:tasks.rados.rados.0.smithi064.stdout:1382: done (1 left) 2024-03-04T06:13:32.427 INFO:tasks.rados.rados.0.smithi064.stdout:1383: done (0 left) 2024-03-04T06:13:32.427 INFO:tasks.rados.rados.0.smithi064.stdout:1384: read oid 17 snap -1 2024-03-04T06:13:32.427 INFO:tasks.rados.rados.0.smithi064.stdout:1384: expect (ObjNum 655 snap 162 seq_num 655) 2024-03-04T06:13:32.427 INFO:tasks.rados.rados.0.smithi064.stdout:1385: delete oid 23 current snap is 165 2024-03-04T06:13:32.438 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-03-04T06:13:31.416237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92444418,"num_objects":462,"num_object_clones":341,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":87,"num_read":2594,"num_read_kb":1891713,"num_write":2290,"num_write_kb":585096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4435,"ondisk_log_size":4435,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":341824,"kb_used_data":197088,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377145536,"statfs":{"total":386547056640,"available":386197028864,"internally_reserved":0,"allocated":201818112,"data_stored":206782655,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3781641,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014071"},"pg_stats":[{"pgid":"3.f","version":"320'234","reported_seq":1138,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.890938+0000","last_change":"2024-03-04T06:13:28.449431+0000","last_active":"2024-03-04T06:13:29.890938+0000","last_peered":"2024-03-04T06:13:29.890938+0000","last_clean":"2024-03-04T06:13:29.890938+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:13:29.890938+0000","last_undegraded":"2024-03-04T06:13:29.890938+0000","last_fullsized":"2024-03-04T06:13:29.890938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496087,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":114,"num_read_kb":76210,"num_write":131,"num_write_kb":35117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"331'199","reported_seq":1121,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.448407+0000","last_change":"2024-03-04T06:13:29.471419+0000","last_active":"2024-03-04T06:13:29.448407+0000","last_peered":"2024-03-04T06:13:29.448407+0000","last_clean":"2024-03-04T06:13:29.448407+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:13:29.448407+0000","last_undegraded":"2024-03-04T06:13:29.448407+0000","last_fullsized":"2024-03-04T06:13:29.448407+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":16,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":85,"num_read_kb":57066,"num_write":86,"num_write_kb":21710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"331'314","reported_seq":1191,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.891830+0000","last_change":"2024-03-04T06:13:27.418862+0000","last_active":"2024-03-04T06:13:29.891830+0000","last_peered":"2024-03-04T06:13:29.891830+0000","last_clean":"2024-03-04T06:13:29.891830+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:13:29.891830+0000","last_undegraded":"2024-03-04T06:13:29.891830+0000","last_fullsized":"2024-03-04T06:13:29.891830+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7731583,"num_objects":36,"num_object_clones":28,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":171,"num_read_kb":119631,"num_write":166,"num_write_kb":45510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"331'214","reported_seq":1147,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.884957+0000","last_change":"2024-03-04T06:13:28.438358+0000","last_active":"2024-03-04T06:13:29.884957+0000","last_peered":"2024-03-04T06:13:29.884957+0000","last_clean":"2024-03-04T06:13:29.884957+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:13:29.884957+0000","last_undegraded":"2024-03-04T06:13:29.884957+0000","last_fullsized":"2024-03-04T06:13:29.884957+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9059303,"num_objects":23,"num_object_clones":15,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":139,"num_read_kb":112624,"num_write":124,"num_write_kb":35266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"312'98","reported_seq":1033,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.440314+0000","last_change":"2024-03-04T06:13:29.476735+0000","last_active":"2024-03-04T06:13:30.440314+0000","last_peered":"2024-03-04T06:13:30.440314+0000","last_clean":"2024-03-04T06:13:30.440314+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:13:30.440314+0000","last_undegraded":"2024-03-04T06:13:30.440314+0000","last_fullsized":"2024-03-04T06:13:30.440314+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2420340,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":35,"num_read_kb":17043,"num_write":42,"num_write_kb":10500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"331'227","reported_seq":1145,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.894326+0000","last_change":"2024-03-04T06:13:28.449042+0000","last_active":"2024-03-04T06:13:29.894326+0000","last_peered":"2024-03-04T06:13:29.894326+0000","last_clean":"2024-03-04T06:13:29.894326+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:13:29.894326+0000","last_undegraded":"2024-03-04T06:13:29.894326+0000","last_fullsized":"2024-03-04T06:13:29.894326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8824038,"num_objects":25,"num_object_clones":18,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":129,"num_read_kb":94389,"num_write":131,"num_write_kb":38676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":972,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.895785+0000","last_change":"2024-03-04T06:13:28.449704+0000","last_active":"2024-03-04T06:13:29.895785+0000","last_peered":"2024-03-04T06:13:29.895785+0000","last_clean":"2024-03-04T06:13:29.895785+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:13:29.895785+0000","last_undegraded":"2024-03-04T06:13:29.895785+0000","last_fullsized":"2024-03-04T06:13:29.895785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"338'171","reported_seq":1103,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.890624+0000","last_change":"2024-03-04T06:13:27.419129+0000","last_active":"2024-03-04T06:13:29.890624+0000","last_peered":"2024-03-04T06:13:29.890624+0000","last_clean":"2024-03-04T06:13:29.890624+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:13:29.890624+0000","last_undegraded":"2024-03-04T06:13:29.890624+0000","last_fullsized":"2024-03-04T06:13:29.890624+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3008585,"num_objects":20,"num_object_clones":13,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":127,"num_read_kb":117184,"num_write":84,"num_write_kb":22492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":965,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.433027+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:13:30.433027+0000","last_peered":"2024-03-04T06:13:30.433027+0000","last_clean":"2024-03-04T06:13:30.433027+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:13:30.433027+0000","last_undegraded":"2024-03-04T06:13:30.433027+0000","last_fullsized":"2024-03-04T06:13:30.433027+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":821,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.898103+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:13:29.898103+0000","last_peered":"2024-03-04T06:13:29.898103+0000","last_clean":"2024-03-04T06:13:29.898103+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:13:29.898103+0000","last_undegraded":"2024-03-04T06:13:29.898103+0000","last_fullsized":"2024-03-04T06:13:29.898103+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"331'297","reported_seq":1160,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.897520+0000","last_change":"2024-03-04T06:13:27.429065+0000","last_active":"2024-03-04T06:13:29.897520+0000","last_peered":"2024-03-04T06:13:29.897520+0000","last_clean":"2024-03-04T06:13:29.897520+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:13:29.897520+0000","last_undegraded":"2024-03-04T06:13:29.897520+0000","last_fullsized":"2024-03-04T06:13:29.897520+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706059,"num_objects":30,"num_object_clones":23,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":151,"num_read_kb":109568,"num_write":138,"num_write_kb":35375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":823,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.904353+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:13:29.904353+0000","last_peered":"2024-03-04T06:13:29.904353+0000","last_clean":"2024-03-04T06:13:29.904353+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:13:29.904353+0000","last_undegraded":"2024-03-04T06:13:29.904353+0000","last_fullsized":"2024-03-04T06:13:29.904353+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"311'235","reported_seq":1120,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.434223+0000","last_change":"2024-03-04T06:13:29.454613+0000","last_active":"2024-03-04T06:13:30.434223+0000","last_peered":"2024-03-04T06:13:30.434223+0000","last_clean":"2024-03-04T06:13:30.434223+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:13:30.434223+0000","last_undegraded":"2024-03-04T06:13:30.434223+0000","last_fullsized":"2024-03-04T06:13:30.434223+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899488,"num_objects":23,"num_object_clones":18,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":122,"num_read_kb":100808,"num_write":96,"num_write_kb":28103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":821,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.889276+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:13:29.889276+0000","last_peered":"2024-03-04T06:13:29.889276+0000","last_clean":"2024-03-04T06:13:29.889276+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:13:29.889276+0000","last_undegraded":"2024-03-04T06:13:29.889276+0000","last_fullsized":"2024-03-04T06:13:29.889276+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"331'553","reported_seq":1391,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.439336+0000","last_change":"2024-03-04T06:13:29.476213+0000","last_active":"2024-03-04T06:13:30.439336+0000","last_peered":"2024-03-04T06:13:30.439336+0000","last_clean":"2024-03-04T06:13:30.439336+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:13:30.439336+0000","last_undegraded":"2024-03-04T06:13:30.439336+0000","last_fullsized":"2024-03-04T06:13:30.439336+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9664553,"num_objects":56,"num_object_clones":42,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":350,"num_read_kb":285928,"num_write":282,"num_write_kb":76369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":823,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.888023+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:13:29.888023+0000","last_peered":"2024-03-04T06:13:29.888023+0000","last_clean":"2024-03-04T06:13:29.888023+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:13:29.888023+0000","last_undegraded":"2024-03-04T06:13:29.888023+0000","last_fullsized":"2024-03-04T06:13:29.888023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"331'527","reported_seq":1331,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.470778+0000","last_change":"2024-03-04T06:13:29.470778+0000","last_active":"2024-03-04T06:13:29.470778+0000","last_peered":"2024-03-04T06:13:29.470778+0000","last_clean":"2024-03-04T06:13:29.470778+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:13:29.470778+0000","last_undegraded":"2024-03-04T06:13:29.470778+0000","last_fullsized":"2024-03-04T06:13:29.470778+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7681162,"num_objects":54,"num_object_clones":40,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":228,"num_read_kb":160285,"num_write":253,"num_write_kb":66344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"328'223","reported_seq":1166,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.897303+0000","last_change":"2024-03-04T06:13:27.429062+0000","last_active":"2024-03-04T06:13:29.897303+0000","last_peered":"2024-03-04T06:13:29.897303+0000","last_clean":"2024-03-04T06:13:29.897303+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:13:29.897303+0000","last_undegraded":"2024-03-04T06:13:29.897303+0000","last_fullsized":"2024-03-04T06:13:29.897303+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2907743,"num_objects":25,"num_object_clones":18,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":202,"num_read_kb":164663,"num_write":115,"num_write_kb":30280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":859,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.478239+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:13:29.478239+0000","last_peered":"2024-03-04T06:13:29.478239+0000","last_clean":"2024-03-04T06:13:29.478239+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:13:29.478239+0000","last_undegraded":"2024-03-04T06:13:29.478239+0000","last_fullsized":"2024-03-04T06:13:29.478239+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"332'242","reported_seq":1176,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.479785+0000","last_change":"2024-03-04T06:13:29.484085+0000","last_active":"2024-03-04T06:13:29.479785+0000","last_peered":"2024-03-04T06:13:29.479785+0000","last_clean":"2024-03-04T06:13:29.479785+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:13:29.479785+0000","last_undegraded":"2024-03-04T06:13:29.479785+0000","last_fullsized":"2024-03-04T06:13:29.479785+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6554961,"num_objects":30,"num_object_clones":19,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":138,"num_read_kb":85725,"num_write":122,"num_write_kb":32368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":832,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.439790+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:13:30.439790+0000","last_peered":"2024-03-04T06:13:30.439790+0000","last_clean":"2024-03-04T06:13:30.439790+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:13:30.439790+0000","last_undegraded":"2024-03-04T06:13:30.439790+0000","last_fullsized":"2024-03-04T06:13:30.439790+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"331'348","reported_seq":1224,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-04T06:13:30.431113+0000","last_change":"2024-03-04T06:13:29.455004+0000","last_active":"2024-03-04T06:13:30.431113+0000","last_peered":"2024-03-04T06:13:30.431113+0000","last_clean":"2024-03-04T06:13:30.431113+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:13:30.431113+0000","last_undegraded":"2024-03-04T06:13:30.431113+0000","last_fullsized":"2024-03-04T06:13:30.431113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840647,"num_objects":36,"num_object_clones":27,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":195,"num_read_kb":136022,"num_write":177,"num_write_kb":48365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":859,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.449557+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:13:29.449557+0000","last_peered":"2024-03-04T06:13:29.449557+0000","last_clean":"2024-03-04T06:13:29.449557+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:13:29.449557+0000","last_undegraded":"2024-03-04T06:13:29.449557+0000","last_fullsized":"2024-03-04T06:13:29.449557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"333'456","reported_seq":1331,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.464677+0000","last_change":"2024-03-04T06:13:29.471568+0000","last_active":"2024-03-04T06:13:29.464677+0000","last_peered":"2024-03-04T06:13:29.464677+0000","last_clean":"2024-03-04T06:13:29.464677+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:13:29.464677+0000","last_undegraded":"2024-03-04T06:13:29.464677+0000","last_fullsized":"2024-03-04T06:13:29.464677+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13059501,"num_objects":40,"num_object_clones":35,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":305,"num_read_kb":254476,"num_write":206,"num_write_kb":56727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":821,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-04T06:13:29.899402+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:13:29.899402+0000","last_peered":"2024-03-04T06:13:29.899402+0000","last_clean":"2024-03-04T06:13:29.899402+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:13:29.899402+0000","last_undegraded":"2024-03-04T06:13:29.899402+0000","last_fullsized":"2024-03-04T06:13:29.899402+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91854050,"num_objects":460,"num_object_clones":341,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":87,"num_read":2491,"num_read_kb":1891622,"num_write":2153,"num_write_kb":583202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4338,"ondisk_log_size":4338,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99404,"kb_used_data":60100,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94272436,"statfs":{"total":96636764160,"available":96534974464,"internally_reserved":0,"allocated":61542400,"data_stored":63089018,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81132,"kb_used_data":45988,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94290708,"statfs":{"total":96636764160,"available":96553684992,"internally_reserved":0,"allocated":47091712,"data_stored":48038234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79876,"kb_used_data":44732,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94291964,"statfs":{"total":96636764160,"available":96554971136,"internally_reserved":0,"allocated":45805568,"data_stored":47121895,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81412,"kb_used_data":46268,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94290428,"statfs":{"total":96636764160,"available":96553398272,"internally_reserved":0,"allocated":47378432,"data_stored":48533508,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:13:32.466 INFO:tasks.rados.rados.0.smithi064.stdout:1385: done (1 left) 2024-03-04T06:13:32.466 INFO:tasks.rados.rados.0.smithi064.stdout:1386: write oid 4 current snap is 165 2024-03-04T06:13:32.466 INFO:tasks.rados.rados.0.smithi064.stdout:1386: seq_num 661 ranges {709427=545947,1813774=612897,2985657=123770} 2024-03-04T06:13:32.473 INFO:tasks.rados.rados.0.smithi064.stdout:1386: writing smithi06431743-4 from 709427 to 1255374 tid 1 2024-03-04T06:13:32.479 INFO:tasks.rados.rados.0.smithi064.stdout:1386: writing smithi06431743-4 from 1813774 to 2426671 tid 2 2024-03-04T06:13:32.484 INFO:tasks.rados.rados.0.smithi064.stdout:1386: writing smithi06431743-4 from 2985657 to 3109427 tid 3 2024-03-04T06:13:32.484 INFO:tasks.rados.rados.0.smithi064.stdout:1387: delete oid 22 current snap is 165 2024-03-04T06:13:32.484 INFO:tasks.rados.rados.0.smithi064.stdout:1384: expect (ObjNum 655 snap 162 seq_num 655) 2024-03-04T06:13:32.635 INFO:tasks.rados.rados.0.smithi064.stdout:1386: finishing write tid 1 to smithi06431743-4 2024-03-04T06:13:32.635 INFO:tasks.rados.rados.0.smithi064.stdout:1386: finishing write tid 2 to smithi06431743-4 2024-03-04T06:13:32.635 INFO:tasks.rados.rados.0.smithi064.stdout:1386: finishing write tid 3 to smithi06431743-4 2024-03-04T06:13:32.635 INFO:tasks.rados.rados.0.smithi064.stdout:1386: finishing write tid 4 to smithi06431743-4 2024-03-04T06:13:32.635 INFO:tasks.rados.rados.0.smithi064.stdout:1386: finishing write tid 5 to smithi06431743-4 2024-03-04T06:13:32.635 INFO:tasks.rados.rados.0.smithi064.stdout:1386: oid 4 updating version 0 to 535 2024-03-04T06:13:32.635 INFO:tasks.rados.rados.0.smithi064.stdout:1386: oid 4 updating version 535 to 538 2024-03-04T06:13:32.635 INFO:tasks.rados.rados.0.smithi064.stdout:1386: oid 4 version 538 is already newer than 537 2024-03-04T06:13:32.635 INFO:tasks.rados.rados.0.smithi064.stdout:1386: oid 4 version 538 is already newer than 536 2024-03-04T06:13:32.635 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 538 (ObjNum 661 snap 165 seq_num 661) dirty exists 2024-03-04T06:13:32.636 INFO:tasks.rados.rados.0.smithi064.stdout:1386: left oid 4 (ObjNum 661 snap 165 seq_num 661) 2024-03-04T06:13:32.636 INFO:tasks.rados.rados.0.smithi064.stdout:1384: done (2 left) 2024-03-04T06:13:32.636 INFO:tasks.rados.rados.0.smithi064.stdout:1386: done (1 left) 2024-03-04T06:13:32.636 INFO:tasks.rados.rados.0.smithi064.stdout:1387: done (0 left) 2024-03-04T06:13:32.636 INFO:tasks.rados.rados.0.smithi064.stdout:1388: write oid 8 current snap is 165 2024-03-04T06:13:32.636 INFO:tasks.rados.rados.0.smithi064.stdout:1388: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2024-03-04T06:13:32.641 INFO:tasks.rados.rados.0.smithi064.stdout:1388: writing smithi06431743-8 from 743041 to 1155839 tid 1 2024-03-04T06:13:32.648 INFO:tasks.rados.rados.0.smithi064.stdout:1388: writing smithi06431743-8 from 1930738 to 2547304 tid 2 2024-03-04T06:13:32.652 INFO:tasks.rados.rados.0.smithi064.stdout:1388: writing smithi06431743-8 from 3040503 to 3143041 tid 3 2024-03-04T06:13:32.653 INFO:tasks.rados.rados.0.smithi064.stdout:1389: snap_remove snap 160 2024-03-04T06:13:33.437 INFO:tasks.rados.rados.0.smithi064.stdout:1389: done (1 left) 2024-03-04T06:13:33.437 INFO:tasks.rados.rados.0.smithi064.stdout:1390: read oid 15 snap -1 2024-03-04T06:13:33.437 INFO:tasks.rados.rados.0.smithi064.stdout:1390: expect (ObjNum 649 snap 157 seq_num 649) 2024-03-04T06:13:33.437 INFO:tasks.rados.rados.0.smithi064.stdout:1391: read oid 3 snap -1 2024-03-04T06:13:33.437 INFO:tasks.rados.rados.0.smithi064.stdout:1391: expect (ObjNum 637 snap 154 seq_num 637) 2024-03-04T06:13:33.437 INFO:tasks.rados.rados.0.smithi064.stdout:1392: snap_remove snap 158 2024-03-04T06:13:33.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:33.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:33.600+0000 7f68e735d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:33.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:33.604+0000 7f68e735d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:34.437 INFO:tasks.rados.rados.0.smithi064.stdout:1392: done (3 left) 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:1393: snap_create 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:1388: finishing write tid 1 to smithi06431743-8 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:1388: finishing write tid 2 to smithi06431743-8 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:1388: finishing write tid 3 to smithi06431743-8 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:1388: finishing write tid 4 to smithi06431743-8 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:1388: finishing write tid 5 to smithi06431743-8 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:1388: oid 8 updating version 0 to 246 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:1388: oid 8 version 246 is already newer than 244 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:1388: oid 8 version 246 is already newer than 245 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:1388: oid 8 updating version 246 to 247 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 247 (ObjNum 663 snap 165 seq_num 663) dirty exists 2024-03-04T06:13:34.438 INFO:tasks.rados.rados.0.smithi064.stdout:1388: left oid 8 (ObjNum 663 snap 165 seq_num 663) 2024-03-04T06:13:34.439 INFO:tasks.rados.rados.0.smithi064.stdout:1390: expect (ObjNum 649 snap 157 seq_num 649) 2024-03-04T06:13:34.557 INFO:tasks.rados.rados.0.smithi064.stdout:1391: expect (ObjNum 637 snap 154 seq_num 637) 2024-03-04T06:13:35.437 INFO:tasks.rados.rados.0.smithi064.stdout:1388: done (3 left) 2024-03-04T06:13:35.437 INFO:tasks.rados.rados.0.smithi064.stdout:1390: done (2 left) 2024-03-04T06:13:35.437 INFO:tasks.rados.rados.0.smithi064.stdout:1391: done (1 left) 2024-03-04T06:13:35.437 INFO:tasks.rados.rados.0.smithi064.stdout:1393: done (0 left) 2024-03-04T06:13:35.437 INFO:tasks.rados.rados.0.smithi064.stdout:1394: read oid 43 snap -1 2024-03-04T06:13:35.437 INFO:tasks.rados.rados.0.smithi064.stdout:1394: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:13:35.437 INFO:tasks.rados.rados.0.smithi064.stdout:1395: read oid 5 snap -1 2024-03-04T06:13:35.437 INFO:tasks.rados.rados.0.smithi064.stdout:1395: expect (ObjNum 551 snap 130 seq_num 551) 2024-03-04T06:13:35.437 INFO:tasks.rados.rados.0.smithi064.stdout:1396: read oid 48 snap -1 2024-03-04T06:13:35.437 INFO:tasks.rados.rados.0.smithi064.stdout:1396: expect (ObjNum 601 snap 144 seq_num 601) 2024-03-04T06:13:35.438 INFO:tasks.rados.rados.0.smithi064.stdout:1397: delete oid 16 current snap is 166 2024-03-04T06:13:35.478 INFO:tasks.rados.rados.0.smithi064.stdout:1394: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:13:35.658 INFO:tasks.rados.rados.0.smithi064.stdout:1395: expect (ObjNum 551 snap 130 seq_num 551) 2024-03-04T06:13:35.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:35.777 INFO:tasks.rados.rados.0.smithi064.stdout:1396: expect (ObjNum 601 snap 144 seq_num 601) 2024-03-04T06:13:35.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:35.777+0000 7f8bf4b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:35.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:35.781+0000 7f8bf4b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:35.969 INFO:tasks.rados.rados.0.smithi064.stdout:1394: done (3 left) 2024-03-04T06:13:35.969 INFO:tasks.rados.rados.0.smithi064.stdout:1395: done (2 left) 2024-03-04T06:13:35.969 INFO:tasks.rados.rados.0.smithi064.stdout:1396: done (1 left) 2024-03-04T06:13:35.969 INFO:tasks.rados.rados.0.smithi064.stdout:1397: done (0 left) 2024-03-04T06:13:35.969 INFO:tasks.rados.rados.0.smithi064.stdout:1398: snap_remove snap 164 2024-03-04T06:13:36.447 INFO:tasks.rados.rados.0.smithi064.stdout:1398: done (0 left) 2024-03-04T06:13:36.447 INFO:tasks.rados.rados.0.smithi064.stdout:1399: read oid 16 snap -1 2024-03-04T06:13:36.447 INFO:tasks.rados.rados.0.smithi064.stdout:1399: expect deleted 2024-03-04T06:13:36.447 INFO:tasks.rados.rados.0.smithi064.stdout:1400: read oid 3 snap -1 2024-03-04T06:13:36.447 INFO:tasks.rados.rados.0.smithi064.stdout:1400: expect (ObjNum 637 snap 154 seq_num 637) 2024-03-04T06:13:36.447 INFO:tasks.rados.rados.0.smithi064.stdout:1401: write (excl) oid 46 current snap is 166 2024-03-04T06:13:36.447 INFO:tasks.rados.rados.0.smithi064.stdout:1401: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2024-03-04T06:13:36.454 INFO:tasks.rados.rados.0.smithi064.stdout:1401: writing smithi06431743-46 from 776655 to 1339951 tid 1 2024-03-04T06:13:36.462 INFO:tasks.rados.rados.0.smithi064.stdout:1401: writing smithi06431743-46 from 2047702 to 2667937 tid 2 2024-03-04T06:13:36.465 INFO:tasks.rados.rados.0.smithi064.stdout:1401: writing smithi06431743-46 from 3095349 to 3176655 tid 3 2024-03-04T06:13:36.465 INFO:tasks.rados.rados.0.smithi064.stdout:1402: write (excl) oid 41 current snap is 166 2024-03-04T06:13:36.465 INFO:tasks.rados.rados.0.smithi064.stdout:1402: seq_num 666 ranges {793462=638545,2106184=763893} 2024-03-04T06:13:36.473 INFO:tasks.rados.rados.0.smithi064.stdout:1402: writing smithi06431743-41 from 793462 to 1432007 tid 1 2024-03-04T06:13:36.484 INFO:tasks.rados.rados.0.smithi064.stdout:1402: writing smithi06431743-41 from 2106184 to 2870077 tid 2 2024-03-04T06:13:36.484 INFO:tasks.rados.rados.0.smithi064.stdout:1403: read oid 35 snap 161 2024-03-04T06:13:36.484 INFO:tasks.rados.rados.0.smithi064.stdout:1403: expect (ObjNum 564 snap 133 seq_num 564) 2024-03-04T06:13:36.484 INFO:tasks.rados.rados.0.smithi064.stdout:1404: read oid 34 snap 165 2024-03-04T06:13:36.484 INFO:tasks.rados.rados.0.smithi064.stdout:1404: expect deleted 2024-03-04T06:13:36.484 INFO:tasks.rados.rados.0.smithi064.stdout:1405: write oid 9 current snap is 166 2024-03-04T06:13:36.484 INFO:tasks.rados.rados.0.smithi064.stdout:1405: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-03-04T06:13:36.491 INFO:tasks.rados.rados.0.smithi064.stdout:1405: writing smithi06431743-9 from 410269 to 1124063 tid 1 2024-03-04T06:13:36.498 INFO:tasks.rados.rados.0.smithi064.stdout:1405: writing smithi06431743-9 from 1648313 to 2155864 tid 2 2024-03-04T06:13:36.505 INFO:tasks.rados.rados.0.smithi064.stdout:1405: writing smithi06431743-9 from 2917489 to 3210269 tid 3 2024-03-04T06:13:36.505 INFO:tasks.rados.rados.0.smithi064.stdout:1406: read oid 15 snap -1 2024-03-04T06:13:36.506 INFO:tasks.rados.rados.0.smithi064.stdout:1406: expect (ObjNum 649 snap 157 seq_num 649) 2024-03-04T06:13:36.506 INFO:tasks.rados.rados.0.smithi064.stdout:1407: write (excl) oid 14 current snap is 166 2024-03-04T06:13:36.506 INFO:tasks.rados.rados.0.smithi064.stdout:1407: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2024-03-04T06:13:36.514 INFO:tasks.rados.rados.0.smithi064.stdout:1407: writing smithi06431743-14 from 427076 to 1216119 tid 1 2024-03-04T06:13:36.522 INFO:tasks.rados.rados.0.smithi064.stdout:1407: writing smithi06431743-14 from 1706795 to 2474357 tid 2 2024-03-04T06:13:36.526 INFO:tasks.rados.rados.0.smithi064.stdout:1407: writing smithi06431743-14 from 2944912 to 3227076 tid 3 2024-03-04T06:13:36.526 INFO:tasks.rados.rados.0.smithi064.stdout:1408: write oid 24 current snap is 166 2024-03-04T06:13:36.526 INFO:tasks.rados.rados.0.smithi064.stdout:1408: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2024-03-04T06:13:36.532 INFO:tasks.rados.rados.0.smithi064.stdout:1408: writing smithi06431743-24 from 443883 to 908175 tid 1 2024-03-04T06:13:36.538 INFO:tasks.rados.rados.0.smithi064.stdout:1408: writing smithi06431743-24 from 1365277 to 1876497 tid 2 2024-03-04T06:13:36.547 INFO:tasks.rados.rados.0.smithi064.stdout:1408: writing smithi06431743-24 from 2572335 to 3243883 tid 3 2024-03-04T06:13:36.547 INFO:tasks.rados.rados.0.smithi064.stdout:1409: write (excl) oid 44 current snap is 166 2024-03-04T06:13:36.547 INFO:tasks.rados.rados.0.smithi064.stdout:1409: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2024-03-04T06:13:36.554 INFO:tasks.rados.rados.0.smithi064.stdout:1409: writing smithi06431743-44 from 460690 to 1116584 tid 1 2024-03-04T06:13:36.562 INFO:tasks.rados.rados.0.smithi064.stdout:1409: writing smithi06431743-44 from 1540112 to 2311343 tid 2 2024-03-04T06:13:36.567 INFO:tasks.rados.rados.0.smithi064.stdout:1409: writing smithi06431743-44 from 2716111 to 3255889 tid 3 2024-03-04T06:13:36.567 INFO:tasks.rados.rados.0.smithi064.stdout:1410: write oid 1 current snap is 166 2024-03-04T06:13:36.567 INFO:tasks.rados.rados.0.smithi064.stdout:1410: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-03-04T06:13:36.576 INFO:tasks.rados.rados.0.smithi064.stdout:1410: writing smithi06431743-1 from 477497 to 1208640 tid 1 2024-03-04T06:13:36.585 INFO:tasks.rados.rados.0.smithi064.stdout:1410: writing smithi06431743-1 from 1998594 to 2513483 tid 2 2024-03-04T06:13:36.590 INFO:tasks.rados.rados.0.smithi064.stdout:1410: writing smithi06431743-1 from 3143534 to 3277497 tid 3 2024-03-04T06:13:36.590 INFO:tasks.rados.rados.0.smithi064.stdout:1399: done (11 left) 2024-03-04T06:13:36.590 INFO:tasks.rados.rados.0.smithi064.stdout:1411: read oid 20 snap -1 2024-03-04T06:13:36.590 INFO:tasks.rados.rados.0.smithi064.stdout:1411: expect (ObjNum 557 snap 132 seq_num 557) 2024-03-04T06:13:36.590 INFO:tasks.rados.rados.0.smithi064.stdout:1412: write (excl) oid 25 current snap is 166 2024-03-04T06:13:36.590 INFO:tasks.rados.rados.0.smithi064.stdout:1412: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2024-03-04T06:13:36.595 INFO:tasks.rados.rados.0.smithi064.stdout:1412: writing smithi06431743-25 from 494304 to 900696 tid 1 2024-03-04T06:13:36.605 INFO:tasks.rados.rados.0.smithi064.stdout:1412: writing smithi06431743-25 from 1540723 to 2315623 tid 2 2024-03-04T06:13:36.611 INFO:tasks.rados.rados.0.smithi064.stdout:1412: writing smithi06431743-25 from 2770957 to 3294304 tid 3 2024-03-04T06:13:36.611 INFO:tasks.rados.rados.0.smithi064.stdout:1413: write oid 13 current snap is 166 2024-03-04T06:13:36.611 INFO:tasks.rados.rados.0.smithi064.stdout:1413: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2024-03-04T06:13:36.617 INFO:tasks.rados.rados.0.smithi064.stdout:1413: writing smithi06431743-13 from 511111 to 992752 tid 1 2024-03-04T06:13:36.623 INFO:tasks.rados.rados.0.smithi064.stdout:1413: writing smithi06431743-13 from 1599205 to 2117763 tid 2 2024-03-04T06:13:36.630 INFO:tasks.rados.rados.0.smithi064.stdout:1413: writing smithi06431743-13 from 2682027 to 3171974 tid 3 2024-03-04T06:13:36.630 INFO:tasks.rados.rados.0.smithi064.stdout:1414: snap_remove snap 162 2024-03-04T06:13:37.440 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:13:37.440 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:13:37.440 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:13:37.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:13:37.451 INFO:tasks.rados.rados.0.smithi064.stdout:1414: done (14 left) 2024-03-04T06:13:37.451 INFO:tasks.rados.rados.0.smithi064.stdout:1415: snap_create 2024-03-04T06:13:37.451 INFO:tasks.rados.rados.0.smithi064.stdout:1406: expect (ObjNum 649 snap 157 seq_num 649) 2024-03-04T06:13:37.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:37.554+0000 7f0e4746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:37.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:37.561+0000 7f0e4746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:37.570 INFO:tasks.rados.rados.0.smithi064.stdout:1400: expect (ObjNum 637 snap 154 seq_num 637) 2024-03-04T06:13:37.574 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:37.574 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:37.574 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:37.574 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:37.574 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:37.574 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:37.574 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:37.574 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:37.575 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:37.575 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:37.575 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:37.575 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:37.577 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:37.578 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:37.578 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:13:37.680 INFO:tasks.rados.rados.0.smithi064.stdout:1401: finishing write tid 1 to smithi06431743-46 2024-03-04T06:13:37.680 INFO:tasks.rados.rados.0.smithi064.stdout:1405: finishing write tid 1 to smithi06431743-9 2024-03-04T06:13:37.680 INFO:tasks.rados.rados.0.smithi064.stdout:1403: expect (ObjNum 564 snap 133 seq_num 564) 2024-03-04T06:13:37.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:37.722+0000 7f7168710640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:37.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:37.729+0000 7f7168710640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:37.742 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:37.745 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:37.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:13:37.813 INFO:tasks.rados.rados.0.smithi064.stdout:1401: finishing write tid 2 to smithi06431743-46 2024-03-04T06:13:37.813 INFO:tasks.rados.rados.0.smithi064.stdout:1402: finishing write tid 1 to smithi06431743-41 2024-03-04T06:13:37.813 INFO:tasks.rados.rados.0.smithi064.stdout:1401: finishing write tid 3 to smithi06431743-46 2024-03-04T06:13:37.813 INFO:tasks.rados.rados.0.smithi064.stdout:1405: finishing write tid 2 to smithi06431743-9 2024-03-04T06:13:37.813 INFO:tasks.rados.rados.0.smithi064.stdout:1407: finishing write tid 1 to smithi06431743-14 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1409: finishing write tid 1 to smithi06431743-44 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1401: finishing write tid 4 to smithi06431743-46 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1402: finishing write tid 2 to smithi06431743-41 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1401: finishing write tid 5 to smithi06431743-46 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1401: oid 46 updating version 0 to 564 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1401: oid 46 version 564 is already newer than 561 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1401: oid 46 version 564 is already newer than 563 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1401: oid 46 version 564 is already newer than 563 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 564 (ObjNum 665 snap 166 seq_num 665) dirty exists 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1401: left oid 46 (ObjNum 665 snap 166 seq_num 665) 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1413: finishing write tid 1 to smithi06431743-13 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1405: finishing write tid 3 to smithi06431743-9 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1409: finishing write tid 2 to smithi06431743-44 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1413: finishing write tid 2 to smithi06431743-13 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1402: finishing write tid 3 to smithi06431743-41 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1402: finishing write tid 4 to smithi06431743-41 2024-03-04T06:13:37.814 INFO:tasks.rados.rados.0.smithi064.stdout:1402: oid 41 updating version 0 to 303 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1402: oid 41 version 303 is already newer than 302 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1402: oid 41 version 303 is already newer than 301 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 303 (ObjNum 666 snap 166 seq_num 666) dirty exists 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1402: left oid 41 (ObjNum 666 snap 166 seq_num 666) 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1412: finishing write tid 1 to smithi06431743-25 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1407: finishing write tid 2 to smithi06431743-14 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1405: finishing write tid 4 to smithi06431743-9 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1405: finishing write tid 5 to smithi06431743-9 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1405: oid 9 updating version 0 to 477 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1405: oid 9 version 477 is already newer than 476 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1405: oid 9 version 477 is already newer than 475 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1405: oid 9 version 477 is already newer than 474 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 477 (ObjNum 667 snap 166 seq_num 667) dirty exists 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1405: left oid 9 (ObjNum 667 snap 166 seq_num 667) 2024-03-04T06:13:37.815 INFO:tasks.rados.rados.0.smithi064.stdout:1410: finishing write tid 1 to smithi06431743-1 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1413: finishing write tid 3 to smithi06431743-13 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1413: finishing write tid 4 to smithi06431743-13 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1409: finishing write tid 3 to smithi06431743-44 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1413: finishing write tid 5 to smithi06431743-13 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1413: oid 13 updating version 0 to 551 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1413: oid 13 version 551 is already newer than 549 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1413: oid 13 updating version 551 to 552 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1413: oid 13 version 552 is already newer than 550 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 552 (ObjNum 673 snap 166 seq_num 673) dirty exists 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1413: left oid 13 (ObjNum 673 snap 166 seq_num 673) 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1407: finishing write tid 3 to smithi06431743-14 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1410: finishing write tid 2 to smithi06431743-1 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1412: finishing write tid 2 to smithi06431743-25 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1409: finishing write tid 4 to smithi06431743-44 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1409: finishing write tid 5 to smithi06431743-44 2024-03-04T06:13:37.816 INFO:tasks.rados.rados.0.smithi064.stdout:1409: oid 44 updating version 0 to 239 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1409: oid 44 updating version 239 to 240 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1409: oid 44 version 240 is already newer than 237 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1409: oid 44 version 240 is already newer than 238 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 240 (ObjNum 670 snap 166 seq_num 670) dirty exists 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1409: left oid 44 (ObjNum 670 snap 166 seq_num 670) 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1407: finishing write tid 4 to smithi06431743-14 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1412: finishing write tid 3 to smithi06431743-25 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1412: finishing write tid 4 to smithi06431743-25 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1412: finishing write tid 5 to smithi06431743-25 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1412: oid 25 updating version 0 to 319 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1412: oid 25 version 319 is already newer than 318 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1412: oid 25 updating version 319 to 321 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1412: oid 25 version 321 is already newer than 321 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 321 (ObjNum 672 snap 166 seq_num 672) dirty exists 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1412: left oid 25 (ObjNum 672 snap 166 seq_num 672) 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1407: finishing write tid 5 to smithi06431743-14 2024-03-04T06:13:37.817 INFO:tasks.rados.rados.0.smithi064.stdout:1407: oid 14 updating version 0 to 352 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1407: oid 14 version 352 is already newer than 351 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1407: oid 14 updating version 352 to 354 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1407: oid 14 version 354 is already newer than 353 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 354 (ObjNum 668 snap 166 seq_num 668) dirty exists 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1407: left oid 14 (ObjNum 668 snap 166 seq_num 668) 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1410: finishing write tid 3 to smithi06431743-1 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1408: finishing write tid 1 to smithi06431743-24 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1408: finishing write tid 2 to smithi06431743-24 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1408: finishing write tid 3 to smithi06431743-24 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1410: finishing write tid 4 to smithi06431743-1 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1408: finishing write tid 4 to smithi06431743-24 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1410: finishing write tid 5 to smithi06431743-1 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1410: oid 1 updating version 0 to 567 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1410: oid 1 version 567 is already newer than 566 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1410: oid 1 version 567 is already newer than 565 2024-03-04T06:13:37.818 INFO:tasks.rados.rados.0.smithi064.stdout:1410: oid 1 updating version 567 to 568 2024-03-04T06:13:37.819 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 568 (ObjNum 671 snap 166 seq_num 671) dirty exists 2024-03-04T06:13:37.819 INFO:tasks.rados.rados.0.smithi064.stdout:1410: left oid 1 (ObjNum 671 snap 166 seq_num 671) 2024-03-04T06:13:37.819 INFO:tasks.rados.rados.0.smithi064.stdout:1408: finishing write tid 5 to smithi06431743-24 2024-03-04T06:13:37.819 INFO:tasks.rados.rados.0.smithi064.stdout:1408: oid 24 updating version 0 to 358 2024-03-04T06:13:37.819 INFO:tasks.rados.rados.0.smithi064.stdout:1408: oid 24 version 358 is already newer than 358 2024-03-04T06:13:37.819 INFO:tasks.rados.rados.0.smithi064.stdout:1408: oid 24 version 358 is already newer than 358 2024-03-04T06:13:37.819 INFO:tasks.rados.rados.0.smithi064.stdout:1408: oid 24 version 358 is already newer than 358 2024-03-04T06:13:37.819 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 358 (ObjNum 669 snap 166 seq_num 669) dirty exists 2024-03-04T06:13:37.819 INFO:tasks.rados.rados.0.smithi064.stdout:1408: left oid 24 (ObjNum 669 snap 166 seq_num 669) 2024-03-04T06:13:37.819 INFO:tasks.rados.rados.0.smithi064.stdout:1411: expect (ObjNum 557 snap 132 seq_num 557) 2024-03-04T06:13:37.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:37.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:37.849+0000 7ff340a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:37.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:37.856+0000 7ff340a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:37.870 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:37.870 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:37.870 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:37.870 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:37.870 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:37.870 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:37.870 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:37.871 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:37.871 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:37.871 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:37.871 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:37.871 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:37.873 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:37.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:13:37.979 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:37.976+0000 7f75e499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:37.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:37.983+0000 7f75e499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:38.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:38.042+0000 7f01b015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:38.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:38.049+0000 7f01b015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:38.068 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:38.071 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:38.071 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:38.228 INFO:tasks.rados.rados.0.smithi064.stdout:1400: done (14 left) 2024-03-04T06:13:38.228 INFO:tasks.rados.rados.0.smithi064.stdout:1401: done (13 left) 2024-03-04T06:13:38.228 INFO:tasks.rados.rados.0.smithi064.stdout:1402: done (12 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1403: done (11 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1404: done (10 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1405: done (9 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1406: done (8 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1407: done (7 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1408: done (6 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1409: done (5 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1410: done (4 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1411: done (3 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1412: done (2 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1413: done (1 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1415: done (0 left) 2024-03-04T06:13:38.229 INFO:tasks.rados.rados.0.smithi064.stdout:1416: snap_create 2024-03-04T06:13:39.229 INFO:tasks.rados.rados.0.smithi064.stdout:1416: done (0 left) 2024-03-04T06:13:39.230 INFO:tasks.rados.rados.0.smithi064.stdout:1417: delete oid 25 current snap is 168 2024-03-04T06:13:39.279 INFO:tasks.rados.rados.0.smithi064.stdout:1417: done (0 left) 2024-03-04T06:13:39.279 INFO:tasks.rados.rados.0.smithi064.stdout:1418: write oid 35 current snap is 168 2024-03-04T06:13:39.279 INFO:tasks.rados.rados.0.smithi064.stdout:1418: seq_num 675 ranges {544725=632139,1716169=522227,2736873=512491} 2024-03-04T06:13:39.285 INFO:tasks.rados.rados.0.smithi064.stdout:1418: writing smithi06431743-35 from 544725 to 1176864 tid 1 2024-03-04T06:13:39.291 INFO:tasks.rados.rados.0.smithi064.stdout:1418: writing smithi06431743-35 from 1716169 to 2238396 tid 2 2024-03-04T06:13:39.296 INFO:tasks.rados.rados.0.smithi064.stdout:1418: writing smithi06431743-35 from 2736873 to 3249364 tid 3 2024-03-04T06:13:39.297 INFO:tasks.rados.rados.0.smithi064.stdout:1419: read oid 12 snap -1 2024-03-04T06:13:39.297 INFO:tasks.rados.rados.0.smithi064.stdout:1419: expect (ObjNum 553 snap 132 seq_num 553) 2024-03-04T06:13:39.297 INFO:tasks.rados.rados.0.smithi064.stdout:1420: write (excl) oid 10 current snap is 168 2024-03-04T06:13:39.297 INFO:tasks.rados.rados.0.smithi064.stdout:1420: seq_num 676 ranges {561532=707388,2058298=782238} 2024-03-04T06:13:39.303 INFO:tasks.rados.rados.0.smithi064.stdout:1420: writing smithi06431743-10 from 561532 to 1268920 tid 1 2024-03-04T06:13:39.312 INFO:tasks.rados.rados.0.smithi064.stdout:1420: writing smithi06431743-10 from 2058298 to 2840536 tid 2 2024-03-04T06:13:39.312 INFO:tasks.rados.rados.0.smithi064.stdout:1421: write oid 2 current snap is 168 2024-03-04T06:13:39.312 INFO:tasks.rados.rados.0.smithi064.stdout:1421: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2024-03-04T06:13:39.318 INFO:tasks.rados.rados.0.smithi064.stdout:1421: writing smithi06431743-2 from 578339 to 1077329 tid 1 2024-03-04T06:13:39.328 INFO:tasks.rados.rados.0.smithi064.stdout:1421: writing smithi06431743-2 from 1833133 to 2359029 tid 2 2024-03-04T06:13:39.335 INFO:tasks.rados.rados.0.smithi064.stdout:1421: writing smithi06431743-2 from 2791719 to 3378339 tid 3 2024-03-04T06:13:39.336 INFO:tasks.rados.rados.0.smithi064.stdout:1422: write oid 40 current snap is 168 2024-03-04T06:13:39.336 INFO:tasks.rados.rados.0.smithi064.stdout:1422: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-03-04T06:13:39.344 INFO:tasks.rados.rados.0.smithi064.stdout:1422: writing smithi06431743-40 from 595146 to 1169385 tid 1 2024-03-04T06:13:39.353 INFO:tasks.rados.rados.0.smithi064.stdout:1422: writing smithi06431743-40 from 1891615 to 2561169 tid 2 2024-03-04T06:13:39.359 INFO:tasks.rados.rados.0.smithi064.stdout:1422: writing smithi06431743-40 from 3219142 to 3395146 tid 3 2024-03-04T06:13:39.359 INFO:tasks.rados.rados.0.smithi064.stdout:1423: read oid 1 snap -1 2024-03-04T06:13:39.359 INFO:tasks.rados.rados.0.smithi064.stdout:1423: expect (ObjNum 671 snap 166 seq_num 671) 2024-03-04T06:13:39.359 INFO:tasks.rados.rados.0.smithi064.stdout:1424: delete oid 26 current snap is 168 2024-03-04T06:13:39.360 INFO:tasks.rados.rados.0.smithi064.stdout:1419: expect (ObjNum 553 snap 132 seq_num 553) 2024-03-04T06:13:39.553 INFO:tasks.rados.rados.0.smithi064.stdout:1420: finishing write tid 1 to smithi06431743-10 2024-03-04T06:13:39.553 INFO:tasks.rados.rados.0.smithi064.stdout:1421: finishing write tid 1 to smithi06431743-2 2024-03-04T06:13:39.553 INFO:tasks.rados.rados.0.smithi064.stdout:1423: expect (ObjNum 671 snap 166 seq_num 671) 2024-03-04T06:13:39.676 INFO:tasks.rados.rados.0.smithi064.stdout:1422: finishing write tid 1 to smithi06431743-40 2024-03-04T06:13:39.676 INFO:tasks.rados.rados.0.smithi064.stdout:1420: finishing write tid 2 to smithi06431743-10 2024-03-04T06:13:39.676 INFO:tasks.rados.rados.0.smithi064.stdout:1418: finishing write tid 1 to smithi06431743-35 2024-03-04T06:13:39.676 INFO:tasks.rados.rados.0.smithi064.stdout:1421: finishing write tid 2 to smithi06431743-2 2024-03-04T06:13:39.676 INFO:tasks.rados.rados.0.smithi064.stdout:1420: finishing write tid 3 to smithi06431743-10 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1420: finishing write tid 4 to smithi06431743-10 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1420: oid 10 updating version 0 to 246 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1420: oid 10 updating version 246 to 247 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1420: oid 10 updating version 247 to 248 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 248 (ObjNum 676 snap 168 seq_num 676) dirty exists 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1420: left oid 10 (ObjNum 676 snap 168 seq_num 676) 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1418: finishing write tid 2 to smithi06431743-35 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1419: done (6 left) 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1420: done (5 left) 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1423: done (4 left) 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1424: done (3 left) 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1425: write oid 38 current snap is 168 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1421: finishing write tid 3 to smithi06431743-2 2024-03-04T06:13:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:1425: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2024-03-04T06:13:39.683 INFO:tasks.rados.rados.0.smithi064.stdout:1425: writing smithi06431743-38 from 628760 to 1353497 tid 1 2024-03-04T06:13:39.691 INFO:tasks.rados.rados.0.smithi064.stdout:1425: writing smithi06431743-38 from 1892226 to 2565449 tid 2 2024-03-04T06:13:39.697 INFO:tasks.rados.rados.0.smithi064.stdout:1425: writing smithi06431743-38 from 3157635 to 3428760 tid 3 2024-03-04T06:13:39.697 INFO:tasks.rados.rados.0.smithi064.stdout:1426: read oid 11 snap -1 2024-03-04T06:13:39.697 INFO:tasks.rados.rados.0.smithi064.stdout:1426: expect (ObjNum 487 snap 114 seq_num 487) 2024-03-04T06:13:39.697 INFO:tasks.rados.rados.0.smithi064.stdout:1427: snap_remove snap 167 2024-03-04T06:13:40.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:40.182 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:40.180+0000 7f4f24d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:40.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:40.184+0000 7f4f24d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:40.457 INFO:tasks.rados.rados.0.smithi064.stdout:1427: done (5 left) 2024-03-04T06:13:40.457 INFO:tasks.rados.rados.0.smithi064.stdout:1428: write (excl) oid 21 current snap is 168 2024-03-04T06:13:40.457 INFO:tasks.rados.rados.0.smithi064.stdout:1421: finishing write tid 4 to smithi06431743-2 2024-03-04T06:13:40.457 INFO:tasks.rados.rados.0.smithi064.stdout:1428: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2024-03-04T06:13:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1428: writing smithi06431743-21 from 645567 to 1445553 tid 1 2024-03-04T06:13:40.471 INFO:tasks.rados.rados.0.smithi064.stdout:1428: writing smithi06431743-21 from 1950708 to 2483942 tid 2 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:1428: writing smithi06431743-21 from 3185058 to 3445567 tid 3 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:1429: delete oid 47 current snap is 168 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:1421: finishing write tid 5 to smithi06431743-2 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:1421: oid 2 updating version 0 to 221 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:1421: oid 2 version 221 is already newer than 218 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:1421: oid 2 version 221 is already newer than 220 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:1421: oid 2 version 221 is already newer than 219 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 221 (ObjNum 677 snap 168 seq_num 677) dirty exists 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:1421: left oid 2 (ObjNum 677 snap 168 seq_num 677) 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:1418: finishing write tid 3 to smithi06431743-35 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:1422: finishing write tid 2 to smithi06431743-40 2024-03-04T06:13:40.478 INFO:tasks.rados.rados.0.smithi064.stdout:1422: finishing write tid 3 to smithi06431743-40 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1418: finishing write tid 4 to smithi06431743-35 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1418: finishing write tid 5 to smithi06431743-35 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1418: oid 35 updating version 0 to 362 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1418: oid 35 updating version 362 to 363 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1418: oid 35 version 363 is already newer than 360 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1418: oid 35 version 363 is already newer than 362 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 363 (ObjNum 675 snap 168 seq_num 675) dirty exists 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1418: left oid 35 (ObjNum 675 snap 168 seq_num 675) 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1422: finishing write tid 4 to smithi06431743-40 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1422: finishing write tid 5 to smithi06431743-40 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1422: oid 40 updating version 0 to 235 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1422: oid 40 updating version 235 to 238 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1422: oid 40 version 238 is already newer than 236 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1422: oid 40 version 238 is already newer than 237 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 238 (ObjNum 678 snap 168 seq_num 678) dirty exists 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1422: left oid 40 (ObjNum 678 snap 168 seq_num 678) 2024-03-04T06:13:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1425: finishing write tid 1 to smithi06431743-38 2024-03-04T06:13:40.480 INFO:tasks.rados.rados.0.smithi064.stdout:1426: expect (ObjNum 487 snap 114 seq_num 487) 2024-03-04T06:13:40.607 INFO:tasks.rados.rados.0.smithi064.stdout:1425: finishing write tid 2 to smithi06431743-38 2024-03-04T06:13:40.607 INFO:tasks.rados.rados.0.smithi064.stdout:1425: finishing write tid 3 to smithi06431743-38 2024-03-04T06:13:40.607 INFO:tasks.rados.rados.0.smithi064.stdout:1425: finishing write tid 4 to smithi06431743-38 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1425: finishing write tid 5 to smithi06431743-38 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1425: oid 38 updating version 0 to 251 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1425: oid 38 version 251 is already newer than 251 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1425: oid 38 updating version 251 to 252 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1425: oid 38 version 252 is already newer than 249 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 252 (ObjNum 680 snap 168 seq_num 680) dirty exists 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1425: left oid 38 (ObjNum 680 snap 168 seq_num 680) 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1418: done (6 left) 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1421: done (5 left) 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1422: done (4 left) 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1425: done (3 left) 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1426: done (2 left) 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1429: done (1 left) 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1430: delete oid 38 current snap is 168 2024-03-04T06:13:40.608 INFO:tasks.rados.rados.0.smithi064.stdout:1428: finishing write tid 1 to smithi06431743-21 2024-03-04T06:13:40.609 INFO:tasks.rados.rados.0.smithi064.stdout:1428: finishing write tid 2 to smithi06431743-21 2024-03-04T06:13:40.609 INFO:tasks.rados.rados.0.smithi064.stdout:1428: finishing write tid 3 to smithi06431743-21 2024-03-04T06:13:40.609 INFO:tasks.rados.rados.0.smithi064.stdout:1428: finishing write tid 4 to smithi06431743-21 2024-03-04T06:13:40.609 INFO:tasks.rados.rados.0.smithi064.stdout:1428: finishing write tid 5 to smithi06431743-21 2024-03-04T06:13:40.609 INFO:tasks.rados.rados.0.smithi064.stdout:1428: oid 21 updating version 0 to 225 2024-03-04T06:13:40.609 INFO:tasks.rados.rados.0.smithi064.stdout:1428: oid 21 version 225 is already newer than 224 2024-03-04T06:13:40.609 INFO:tasks.rados.rados.0.smithi064.stdout:1428: oid 21 updating version 225 to 227 2024-03-04T06:13:40.609 INFO:tasks.rados.rados.0.smithi064.stdout:1428: oid 21 version 227 is already newer than 226 2024-03-04T06:13:40.609 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 227 (ObjNum 681 snap 168 seq_num 681) dirty exists 2024-03-04T06:13:40.609 INFO:tasks.rados.rados.0.smithi064.stdout:1428: left oid 21 (ObjNum 681 snap 168 seq_num 681) 2024-03-04T06:13:40.617 INFO:tasks.rados.rados.0.smithi064.stdout:1428: done (1 left) 2024-03-04T06:13:40.617 INFO:tasks.rados.rados.0.smithi064.stdout:1430: done (0 left) 2024-03-04T06:13:40.617 INFO:tasks.rados.rados.0.smithi064.stdout:1431: write (excl) oid 42 current snap is 168 2024-03-04T06:13:40.617 INFO:tasks.rados.rados.0.smithi064.stdout:1431: seq_num 684 ranges {695988=625733,2009801=680561,3150974=345014} 2024-03-04T06:13:40.623 INFO:tasks.rados.rados.0.smithi064.stdout:1431: writing smithi06431743-42 from 695988 to 1321721 tid 1 2024-03-04T06:13:40.631 INFO:tasks.rados.rados.0.smithi064.stdout:1431: writing smithi06431743-42 from 2009801 to 2690362 tid 2 2024-03-04T06:13:40.636 INFO:tasks.rados.rados.0.smithi064.stdout:1431: writing smithi06431743-42 from 3150974 to 3495988 tid 3 2024-03-04T06:13:40.636 INFO:tasks.rados.rados.0.smithi064.stdout:1432: snap_remove snap 165 2024-03-04T06:13:41.460 INFO:tasks.rados.rados.0.smithi064.stdout:1432: done (1 left) 2024-03-04T06:13:41.460 INFO:tasks.rados.rados.0.smithi064.stdout:1433: write oid 48 current snap is 168 2024-03-04T06:13:41.460 INFO:tasks.rados.rados.0.smithi064.stdout:1431: finishing write tid 1 to smithi06431743-42 2024-03-04T06:13:41.460 INFO:tasks.rados.rados.0.smithi064.stdout:1433: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2024-03-04T06:13:41.466 INFO:tasks.rados.rados.0.smithi064.stdout:1433: writing smithi06431743-48 from 712795 to 1130130 tid 1 2024-03-04T06:13:41.472 INFO:tasks.rados.rados.0.smithi064.stdout:1433: writing smithi06431743-48 from 1784636 to 2325208 tid 2 2024-03-04T06:13:41.479 INFO:tasks.rados.rados.0.smithi064.stdout:1433: writing smithi06431743-48 from 2894750 to 3352667 tid 3 2024-03-04T06:13:41.479 INFO:tasks.rados.rados.0.smithi064.stdout:1434: read oid 20 snap -1 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1434: expect (ObjNum 557 snap 132 seq_num 557) 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1431: finishing write tid 2 to smithi06431743-42 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1431: finishing write tid 3 to smithi06431743-42 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1431: finishing write tid 4 to smithi06431743-42 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1435: read oid 19 snap 163 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1435: expect deleted 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1436: read oid 36 snap -1 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1436: expect (ObjNum 638 snap 154 seq_num 638) 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1437: read oid 21 snap 166 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1437: expect deleted 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1438: read oid 43 snap -1 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1438: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1439: write (excl) oid 31 current snap is 168 2024-03-04T06:13:41.480 INFO:tasks.rados.rados.0.smithi064.stdout:1431: finishing write tid 5 to smithi06431743-42 2024-03-04T06:13:41.481 INFO:tasks.rados.rados.0.smithi064.stdout:1431: oid 42 updating version 0 to 242 2024-03-04T06:13:41.481 INFO:tasks.rados.rados.0.smithi064.stdout:1431: oid 42 updating version 242 to 243 2024-03-04T06:13:41.481 INFO:tasks.rados.rados.0.smithi064.stdout:1431: oid 42 version 243 is already newer than 242 2024-03-04T06:13:41.481 INFO:tasks.rados.rados.0.smithi064.stdout:1431: oid 42 version 243 is already newer than 240 2024-03-04T06:13:41.481 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 243 (ObjNum 684 snap 168 seq_num 684) dirty exists 2024-03-04T06:13:41.481 INFO:tasks.rados.rados.0.smithi064.stdout:1431: left oid 42 (ObjNum 684 snap 168 seq_num 684) 2024-03-04T06:13:41.481 INFO:tasks.rados.rados.0.smithi064.stdout:1439: seq_num 686 ranges {729602=492584,1843118=684230,3322173=207429} 2024-03-04T06:13:41.486 INFO:tasks.rados.rados.0.smithi064.stdout:1439: writing smithi06431743-31 from 729602 to 1222186 tid 1 2024-03-04T06:13:41.493 INFO:tasks.rados.rados.0.smithi064.stdout:1439: writing smithi06431743-31 from 1843118 to 2527348 tid 2 2024-03-04T06:13:41.500 INFO:tasks.rados.rados.0.smithi064.stdout:1439: writing smithi06431743-31 from 3322173 to 3529602 tid 3 2024-03-04T06:13:41.500 INFO:tasks.rados.rados.0.smithi064.stdout:1431: done (7 left) 2024-03-04T06:13:41.500 INFO:tasks.rados.rados.0.smithi064.stdout:1440: write (excl) oid 9 current snap is 168 2024-03-04T06:13:41.500 INFO:tasks.rados.rados.0.smithi064.stdout:1440: seq_num 687 ranges {746409=567833,1901600=544241,3065949=480460} 2024-03-04T06:13:41.509 INFO:tasks.rados.rados.0.smithi064.stdout:1440: writing smithi06431743-9 from 746409 to 1314242 tid 1 2024-03-04T06:13:41.518 INFO:tasks.rados.rados.0.smithi064.stdout:1440: writing smithi06431743-9 from 1901600 to 2445841 tid 2 2024-03-04T06:13:41.525 INFO:tasks.rados.rados.0.smithi064.stdout:1440: writing smithi06431743-9 from 3065949 to 3546409 tid 3 2024-03-04T06:13:41.525 INFO:tasks.rados.rados.0.smithi064.stdout:1441: write (excl) oid 3 current snap is 168 2024-03-04T06:13:41.525 INFO:tasks.rados.rados.0.smithi064.stdout:1441: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2024-03-04T06:13:41.534 INFO:tasks.rados.rados.0.smithi064.stdout:1441: writing smithi06431743-3 from 763216 to 1406298 tid 1 2024-03-04T06:13:41.540 INFO:tasks.rados.rados.0.smithi064.stdout:1441: writing smithi06431743-3 from 1843729 to 2531628 tid 2 2024-03-04T06:13:41.546 INFO:tasks.rados.rados.0.smithi064.stdout:1441: writing smithi06431743-3 from 3260666 to 3563216 tid 3 2024-03-04T06:13:41.546 INFO:tasks.rados.rados.0.smithi064.stdout:1442: read oid 16 snap 166 2024-03-04T06:13:41.546 INFO:tasks.rados.rados.0.smithi064.stdout:1442: expect deleted 2024-03-04T06:13:41.546 INFO:tasks.rados.rados.0.smithi064.stdout:1443: snap_create 2024-03-04T06:13:41.546 INFO:tasks.rados.rados.0.smithi064.stdout:1438: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:13:41.744 INFO:tasks.rados.rados.0.smithi064.stdout:1436: expect (ObjNum 638 snap 154 seq_num 638) 2024-03-04T06:13:41.848 INFO:tasks.rados.rados.0.smithi064.stdout:1440: finishing write tid 1 to smithi06431743-9 2024-03-04T06:13:41.848 INFO:tasks.rados.rados.0.smithi064.stdout:1434: expect (ObjNum 557 snap 132 seq_num 557) 2024-03-04T06:13:42.044 INFO:tasks.rados.rados.0.smithi064.stdout:1439: finishing write tid 1 to smithi06431743-31 2024-03-04T06:13:42.044 INFO:tasks.rados.rados.0.smithi064.stdout:1433: finishing write tid 1 to smithi06431743-48 2024-03-04T06:13:42.044 INFO:tasks.rados.rados.0.smithi064.stdout:1441: finishing write tid 1 to smithi06431743-3 2024-03-04T06:13:42.044 INFO:tasks.rados.rados.0.smithi064.stdout:1439: finishing write tid 2 to smithi06431743-31 2024-03-04T06:13:42.044 INFO:tasks.rados.rados.0.smithi064.stdout:1440: finishing write tid 2 to smithi06431743-9 2024-03-04T06:13:42.044 INFO:tasks.rados.rados.0.smithi064.stdout:1441: finishing write tid 2 to smithi06431743-3 2024-03-04T06:13:42.044 INFO:tasks.rados.rados.0.smithi064.stdout:1440: finishing write tid 3 to smithi06431743-9 2024-03-04T06:13:42.044 INFO:tasks.rados.rados.0.smithi064.stdout:1433: finishing write tid 2 to smithi06431743-48 2024-03-04T06:13:42.044 INFO:tasks.rados.rados.0.smithi064.stdout:1439: finishing write tid 3 to smithi06431743-31 2024-03-04T06:13:42.044 INFO:tasks.rados.rados.0.smithi064.stdout:1440: finishing write tid 4 to smithi06431743-9 2024-03-04T06:13:42.044 INFO:tasks.rados.rados.0.smithi064.stdout:1439: finishing write tid 4 to smithi06431743-31 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1440: finishing write tid 5 to smithi06431743-9 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1440: oid 9 updating version 0 to 488 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1440: oid 9 version 488 is already newer than 487 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1440: oid 9 updating version 488 to 489 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1440: oid 9 updating version 489 to 490 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 490 (ObjNum 687 snap 168 seq_num 687) dirty exists 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1440: left oid 9 (ObjNum 687 snap 168 seq_num 687) 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1439: finishing write tid 5 to smithi06431743-31 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1439: oid 31 updating version 0 to 103 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1439: oid 31 version 103 is already newer than 100 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1439: oid 31 version 103 is already newer than 101 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1439: oid 31 version 103 is already newer than 102 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 103 (ObjNum 686 snap 168 seq_num 686) dirty exists 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1439: left oid 31 (ObjNum 686 snap 168 seq_num 686) 2024-03-04T06:13:42.045 INFO:tasks.rados.rados.0.smithi064.stdout:1441: finishing write tid 3 to smithi06431743-3 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1441: finishing write tid 4 to smithi06431743-3 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1433: finishing write tid 3 to smithi06431743-48 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1441: finishing write tid 5 to smithi06431743-3 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1441: oid 3 updating version 0 to 305 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1441: oid 3 updating version 305 to 308 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1441: oid 3 version 308 is already newer than 308 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1441: oid 3 version 308 is already newer than 306 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 308 (ObjNum 688 snap 168 seq_num 688) dirty exists 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1441: left oid 3 (ObjNum 688 snap 168 seq_num 688) 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1433: finishing write tid 4 to smithi06431743-48 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1433: finishing write tid 5 to smithi06431743-48 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1433: oid 48 updating version 0 to 240 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1433: oid 48 version 240 is already newer than 238 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1433: oid 48 version 240 is already newer than 240 2024-03-04T06:13:42.046 INFO:tasks.rados.rados.0.smithi064.stdout:1433: oid 48 version 240 is already newer than 237 2024-03-04T06:13:42.047 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 240 (ObjNum 685 snap 168 seq_num 685) dirty exists 2024-03-04T06:13:42.047 INFO:tasks.rados.rados.0.smithi064.stdout:1433: left oid 48 (ObjNum 685 snap 168 seq_num 685) 2024-03-04T06:13:42.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:42.359 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:42.357+0000 7f4563b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:42.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:42.361+0000 7f4563b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:42.468 INFO:tasks.rados.rados.0.smithi064.stdout:1433: done (10 left) 2024-03-04T06:13:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:1434: done (9 left) 2024-03-04T06:13:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:1435: done (8 left) 2024-03-04T06:13:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:1436: done (7 left) 2024-03-04T06:13:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:1437: done (6 left) 2024-03-04T06:13:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:1438: done (5 left) 2024-03-04T06:13:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:1439: done (4 left) 2024-03-04T06:13:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:1440: done (3 left) 2024-03-04T06:13:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:1441: done (2 left) 2024-03-04T06:13:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:1442: done (1 left) 2024-03-04T06:13:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:1443: done (0 left) 2024-03-04T06:13:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:1444: delete oid 47 current snap is 169 2024-03-04T06:13:42.499 INFO:tasks.rados.rados.0.smithi064.stdout:1444: done (0 left) 2024-03-04T06:13:42.499 INFO:tasks.rados.rados.0.smithi064.stdout:1445: write oid 27 current snap is 169 2024-03-04T06:13:42.499 INFO:tasks.rados.rados.0.smithi064.stdout:1445: seq_num 690 ranges {796830=793580,2360693=691568} 2024-03-04T06:13:42.507 INFO:tasks.rados.rados.0.smithi064.stdout:1445: writing smithi06431743-27 from 796830 to 1590410 tid 1 2024-03-04T06:13:42.515 INFO:tasks.rados.rados.0.smithi064.stdout:1445: writing smithi06431743-27 from 2360693 to 3052261 tid 2 2024-03-04T06:13:42.515 INFO:tasks.rados.rados.0.smithi064.stdout:1446: delete oid 38 current snap is 169 2024-03-04T06:13:42.518 INFO:tasks.rados.rados.0.smithi064.stdout:1446: done (1 left) 2024-03-04T06:13:42.518 INFO:tasks.rados.rados.0.smithi064.stdout:1447: snap_remove snap 161 2024-03-04T06:13:43.072 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:13:43.073 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:13:43.073 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:13:43.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:13:43.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:43.176+0000 7fcec459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:43.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:43.184+0000 7fcec459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:43.196 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:43.196 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:43.196 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:43.196 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:43.196 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:43.197 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:43.197 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:43.197 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:43.197 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:43.197 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:43.197 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:43.197 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:43.199 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:43.199 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:43.200 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:13:43.229 INFO:tasks.rados.rados.0.smithi064.stdout:1447: done (1 left) 2024-03-04T06:13:43.229 INFO:tasks.rados.rados.0.smithi064.stdout:1448: write (excl) oid 42 current snap is 169 2024-03-04T06:13:43.229 INFO:tasks.rados.rados.0.smithi064.stdout:1445: finishing write tid 1 to smithi06431743-27 2024-03-04T06:13:43.230 INFO:tasks.rados.rados.0.smithi064.stdout:1448: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2024-03-04T06:13:43.235 INFO:tasks.rados.rados.0.smithi064.stdout:1448: writing smithi06431743-42 from 430444 to 1090875 tid 1 2024-03-04T06:13:43.242 INFO:tasks.rados.rados.0.smithi064.stdout:1448: writing smithi06431743-42 from 1677657 to 2372894 tid 2 2024-03-04T06:13:43.249 INFO:tasks.rados.rados.0.smithi064.stdout:1448: writing smithi06431743-42 from 2970358 to 3539885 tid 3 2024-03-04T06:13:43.249 INFO:tasks.rados.rados.0.smithi064.stdout:1449: snap_remove snap 163 2024-03-04T06:13:43.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:43.330+0000 7fbb2439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:43.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:43.338+0000 7fbb2439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:43.355 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:43.356 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:43.356 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:43.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:13:43.505 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:43.500+0000 7f1d7d39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:43.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:43.507+0000 7f1d7d39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:43.521 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:43.522 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:43.522 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:43.524 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:43.524 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:43.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:13:43.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:43.627+0000 7f985975d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:43.635 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:43.634+0000 7f985975d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:43.648 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:43.648 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:43.648 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:43.648 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:43.649 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:43.649 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:43.649 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:43.649 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:43.649 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:43.649 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:43.649 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:43.649 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:43.650 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:43.651 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:44.230 INFO:tasks.rados.rados.0.smithi064.stdout:1449: done (2 left) 2024-03-04T06:13:44.231 INFO:tasks.rados.rados.0.smithi064.stdout:1450: write (excl) oid 44 current snap is 169 2024-03-04T06:13:44.231 INFO:tasks.rados.rados.0.smithi064.stdout:1445: finishing write tid 2 to smithi06431743-27 2024-03-04T06:13:44.231 INFO:tasks.rados.rados.0.smithi064.stdout:1450: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2024-03-04T06:13:44.236 INFO:tasks.rados.rados.0.smithi064.stdout:1450: writing smithi06431743-44 from 447251 to 1182931 tid 1 2024-03-04T06:13:44.242 INFO:tasks.rados.rados.0.smithi064.stdout:1450: writing smithi06431743-44 from 1736139 to 2175034 tid 2 2024-03-04T06:13:44.250 INFO:tasks.rados.rados.0.smithi064.stdout:1450: writing smithi06431743-44 from 2597781 to 3378580 tid 3 2024-03-04T06:13:44.250 INFO:tasks.rados.rados.0.smithi064.stdout:1451: snap_create 2024-03-04T06:13:44.250 INFO:tasks.rados.rados.0.smithi064.stdout:1445: finishing write tid 3 to smithi06431743-27 2024-03-04T06:13:44.250 INFO:tasks.rados.rados.0.smithi064.stdout:1445: finishing write tid 4 to smithi06431743-27 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1445: oid 27 updating version 0 to 202 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1445: oid 27 updating version 202 to 203 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1445: oid 27 updating version 203 to 204 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 204 (ObjNum 690 snap 169 seq_num 690) dirty exists 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1445: left oid 27 (ObjNum 690 snap 169 seq_num 690) 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1448: finishing write tid 1 to smithi06431743-42 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1448: finishing write tid 2 to smithi06431743-42 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1448: finishing write tid 3 to smithi06431743-42 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1448: finishing write tid 4 to smithi06431743-42 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1448: finishing write tid 5 to smithi06431743-42 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1448: oid 42 updating version 0 to 249 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1448: oid 42 updating version 249 to 251 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1448: oid 42 version 251 is already newer than 251 2024-03-04T06:13:44.251 INFO:tasks.rados.rados.0.smithi064.stdout:1448: oid 42 updating version 251 to 252 2024-03-04T06:13:44.252 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 252 (ObjNum 692 snap 169 seq_num 692) dirty exists 2024-03-04T06:13:44.252 INFO:tasks.rados.rados.0.smithi064.stdout:1448: left oid 42 (ObjNum 692 snap 169 seq_num 692) 2024-03-04T06:13:44.311 INFO:tasks.rados.rados.0.smithi064.stdout:1450: finishing write tid 1 to smithi06431743-44 2024-03-04T06:13:44.378 INFO:tasks.rados.rados.0.smithi064.stdout:1450: finishing write tid 2 to smithi06431743-44 2024-03-04T06:13:44.397 INFO:tasks.rados.rados.0.smithi064.stdout:1450: finishing write tid 3 to smithi06431743-44 2024-03-04T06:13:44.405 INFO:tasks.rados.rados.0.smithi064.stdout:1450: finishing write tid 4 to smithi06431743-44 2024-03-04T06:13:44.407 INFO:tasks.rados.rados.0.smithi064.stdout:1450: finishing write tid 5 to smithi06431743-44 2024-03-04T06:13:44.407 INFO:tasks.rados.rados.0.smithi064.stdout:1450: oid 44 updating version 0 to 272 2024-03-04T06:13:44.407 INFO:tasks.rados.rados.0.smithi064.stdout:1450: oid 44 updating version 272 to 274 2024-03-04T06:13:44.407 INFO:tasks.rados.rados.0.smithi064.stdout:1450: oid 44 version 274 is already newer than 273 2024-03-04T06:13:44.407 INFO:tasks.rados.rados.0.smithi064.stdout:1450: oid 44 version 274 is already newer than 271 2024-03-04T06:13:44.407 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 274 (ObjNum 693 snap 169 seq_num 693) dirty exists 2024-03-04T06:13:44.407 INFO:tasks.rados.rados.0.smithi064.stdout:1450: left oid 44 (ObjNum 693 snap 169 seq_num 693) 2024-03-04T06:13:44.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:44.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:44.537+0000 7f6c8735d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:44.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:44.541+0000 7f6c8735d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:45.232 INFO:tasks.rados.rados.0.smithi064.stdout:1445: done (3 left) 2024-03-04T06:13:45.232 INFO:tasks.rados.rados.0.smithi064.stdout:1448: done (2 left) 2024-03-04T06:13:45.232 INFO:tasks.rados.rados.0.smithi064.stdout:1450: done (1 left) 2024-03-04T06:13:45.232 INFO:tasks.rados.rados.0.smithi064.stdout:1451: done (0 left) 2024-03-04T06:13:45.232 INFO:tasks.rados.rados.0.smithi064.stdout:1452: read oid 26 snap 166 2024-03-04T06:13:45.232 INFO:tasks.rados.rados.0.smithi064.stdout:1452: expect (ObjNum 611 snap 146 seq_num 611) 2024-03-04T06:13:45.232 INFO:tasks.rados.rados.0.smithi064.stdout:1453: write (excl) oid 44 current snap is 170 2024-03-04T06:13:45.232 INFO:tasks.rados.rados.0.smithi064.stdout:1453: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2024-03-04T06:13:45.236 INFO:tasks.rados.rados.0.smithi064.stdout:1453: writing smithi06431743-44 from 464058 to 874987 tid 1 2024-03-04T06:13:45.244 INFO:tasks.rados.rados.0.smithi064.stdout:1453: writing smithi06431743-44 from 1394621 to 2093527 tid 2 2024-03-04T06:13:45.250 INFO:tasks.rados.rados.0.smithi064.stdout:1453: writing smithi06431743-44 from 2625204 to 3217275 tid 3 2024-03-04T06:13:45.252 INFO:tasks.rados.rados.0.smithi064.stdout:1453: writing smithi06431743-44 from 3656107 to 3664058 tid 4 2024-03-04T06:13:45.253 INFO:tasks.rados.rados.0.smithi064.stdout:1454: write oid 20 current snap is 170 2024-03-04T06:13:45.253 INFO:tasks.rados.rados.0.smithi064.stdout:1454: seq_num 695 ranges {480865=486178,1453103=442564,2652627=519696} 2024-03-04T06:13:45.259 INFO:tasks.rados.rados.0.smithi064.stdout:1454: writing smithi06431743-20 from 480865 to 967043 tid 1 2024-03-04T06:13:45.265 INFO:tasks.rados.rados.0.smithi064.stdout:1454: writing smithi06431743-20 from 1453103 to 1895667 tid 2 2024-03-04T06:13:45.274 INFO:tasks.rados.rados.0.smithi064.stdout:1454: writing smithi06431743-20 from 2652627 to 3172323 tid 3 2024-03-04T06:13:45.274 INFO:tasks.rados.rados.0.smithi064.stdout:1455: delete oid 16 current snap is 170 2024-03-04T06:13:45.287 INFO:tasks.rados.rados.0.smithi064.stdout:1455: done (3 left) 2024-03-04T06:13:45.288 INFO:tasks.rados.rados.0.smithi064.stdout:1456: snap_create 2024-03-04T06:13:45.300 INFO:tasks.rados.rados.0.smithi064.stdout:1453: finishing write tid 1 to smithi06431743-44 2024-03-04T06:13:45.314 INFO:tasks.rados.rados.0.smithi064.stdout:1452: expect (ObjNum 611 snap 146 seq_num 611) 2024-03-04T06:13:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1454: finishing write tid 1 to smithi06431743-20 2024-03-04T06:13:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1453: finishing write tid 2 to smithi06431743-44 2024-03-04T06:13:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1454: finishing write tid 2 to smithi06431743-20 2024-03-04T06:13:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1453: finishing write tid 3 to smithi06431743-44 2024-03-04T06:13:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1454: finishing write tid 3 to smithi06431743-20 2024-03-04T06:13:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1453: finishing write tid 4 to smithi06431743-44 2024-03-04T06:13:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1453: finishing write tid 5 to smithi06431743-44 2024-03-04T06:13:45.411 INFO:tasks.rados.rados.0.smithi064.stdout:1453: finishing write tid 6 to smithi06431743-44 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1453: oid 44 updating version 0 to 280 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1453: oid 44 version 280 is already newer than 278 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1453: oid 44 updating version 280 to 282 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1453: oid 44 version 282 is already newer than 282 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1453: oid 44 version 282 is already newer than 279 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 282 (ObjNum 694 snap 170 seq_num 694) dirty exists 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1453: left oid 44 (ObjNum 694 snap 170 seq_num 694) 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1454: finishing write tid 4 to smithi06431743-20 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1454: finishing write tid 5 to smithi06431743-20 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1454: oid 20 updating version 0 to 369 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1454: oid 20 updating version 369 to 372 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1454: oid 20 version 372 is already newer than 371 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1454: oid 20 version 372 is already newer than 371 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 372 (ObjNum 695 snap 170 seq_num 695) dirty exists 2024-03-04T06:13:45.412 INFO:tasks.rados.rados.0.smithi064.stdout:1454: left oid 20 (ObjNum 695 snap 170 seq_num 695) 2024-03-04T06:13:46.235 INFO:tasks.rados.rados.0.smithi064.stdout:1452: done (3 left) 2024-03-04T06:13:46.235 INFO:tasks.rados.rados.0.smithi064.stdout:1453: done (2 left) 2024-03-04T06:13:46.235 INFO:tasks.rados.rados.0.smithi064.stdout:1454: done (1 left) 2024-03-04T06:13:46.235 INFO:tasks.rados.rados.0.smithi064.stdout:1456: done (0 left) 2024-03-04T06:13:46.236 INFO:tasks.rados.rados.0.smithi064.stdout:1457: snap_remove snap 169 2024-03-04T06:13:46.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:46.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:46.725+0000 7f23916dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:46.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:46.729+0000 7f23916dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:47.239 INFO:tasks.rados.rados.0.smithi064.stdout:1457: done (0 left) 2024-03-04T06:13:47.239 INFO:tasks.rados.rados.0.smithi064.stdout:1458: read oid 36 snap -1 2024-03-04T06:13:47.239 INFO:tasks.rados.rados.0.smithi064.stdout:1458: expect (ObjNum 638 snap 154 seq_num 638) 2024-03-04T06:13:47.239 INFO:tasks.rados.rados.0.smithi064.stdout:1459: snap_create 2024-03-04T06:13:47.293 INFO:tasks.rados.rados.0.smithi064.stdout:1458: expect (ObjNum 638 snap 154 seq_num 638) 2024-03-04T06:13:48.244 INFO:tasks.rados.rados.0.smithi064.stdout:1458: done (1 left) 2024-03-04T06:13:48.244 INFO:tasks.rados.rados.0.smithi064.stdout:1459: done (0 left) 2024-03-04T06:13:48.244 INFO:tasks.rados.rados.0.smithi064.stdout:1460: write (excl) oid 38 current snap is 172 2024-03-04T06:13:48.244 INFO:tasks.rados.rados.0.smithi064.stdout:1460: seq_num 697 ranges {514479=636676,1853714=446233,2991120=542240} 2024-03-04T06:13:48.250 INFO:tasks.rados.rados.0.smithi064.stdout:1460: writing smithi06431743-38 from 514479 to 1151155 tid 1 2024-03-04T06:13:48.256 INFO:tasks.rados.rados.0.smithi064.stdout:1460: writing smithi06431743-38 from 1853714 to 2299947 tid 2 2024-03-04T06:13:48.263 INFO:tasks.rados.rados.0.smithi064.stdout:1460: writing smithi06431743-38 from 2991120 to 3533360 tid 3 2024-03-04T06:13:48.263 INFO:tasks.rados.rados.0.smithi064.stdout:1461: delete oid 12 current snap is 172 2024-03-04T06:13:48.300 INFO:tasks.rados.rados.0.smithi064.stdout:1460: finishing write tid 1 to smithi06431743-38 2024-03-04T06:13:48.305 INFO:tasks.rados.rados.0.smithi064.stdout:1461: done (1 left) 2024-03-04T06:13:48.305 INFO:tasks.rados.rados.0.smithi064.stdout:1462: write oid 36 current snap is 172 2024-03-04T06:13:48.305 INFO:tasks.rados.rados.0.smithi064.stdout:1462: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2024-03-04T06:13:48.313 INFO:tasks.rados.rados.0.smithi064.stdout:1462: writing smithi06431743-36 from 548093 to 1335267 tid 1 2024-03-04T06:13:48.319 INFO:tasks.rados.rados.0.smithi064.stdout:1462: writing smithi06431743-36 from 1970678 to 2420580 tid 2 2024-03-04T06:13:48.327 INFO:tasks.rados.rados.0.smithi064.stdout:1462: writing smithi06431743-36 from 3045966 to 3727103 tid 3 2024-03-04T06:13:48.328 INFO:tasks.rados.rados.0.smithi064.stdout:1463: read oid 17 snap -1 2024-03-04T06:13:48.328 INFO:tasks.rados.rados.0.smithi064.stdout:1463: expect (ObjNum 655 snap 162 seq_num 655) 2024-03-04T06:13:48.328 INFO:tasks.rados.rados.0.smithi064.stdout:1460: finishing write tid 2 to smithi06431743-38 2024-03-04T06:13:48.328 INFO:tasks.rados.rados.0.smithi064.stdout:1460: finishing write tid 3 to smithi06431743-38 2024-03-04T06:13:48.328 INFO:tasks.rados.rados.0.smithi064.stdout:1464: read oid 40 snap 171 2024-03-04T06:13:48.328 INFO:tasks.rados.rados.0.smithi064.stdout:1464: expect (ObjNum 678 snap 168 seq_num 678) 2024-03-04T06:13:48.328 INFO:tasks.rados.rados.0.smithi064.stdout:1465: read oid 16 snap 166 2024-03-04T06:13:48.328 INFO:tasks.rados.rados.0.smithi064.stdout:1465: expect deleted 2024-03-04T06:13:48.328 INFO:tasks.rados.rados.0.smithi064.stdout:1466: write oid 27 current snap is 172 2024-03-04T06:13:48.328 INFO:tasks.rados.rados.0.smithi064.stdout:1466: seq_num 700 ranges {564900=578776,1629160=709913,3073389=492409} 2024-03-04T06:13:48.334 INFO:tasks.rados.rados.0.smithi064.stdout:1466: writing smithi06431743-27 from 564900 to 1143676 tid 1 2024-03-04T06:13:48.341 INFO:tasks.rados.rados.0.smithi064.stdout:1466: writing smithi06431743-27 from 1629160 to 2339073 tid 2 2024-03-04T06:13:48.348 INFO:tasks.rados.rados.0.smithi064.stdout:1466: writing smithi06431743-27 from 3073389 to 3565798 tid 3 2024-03-04T06:13:48.348 INFO:tasks.rados.rados.0.smithi064.stdout:1467: write oid 22 current snap is 172 2024-03-04T06:13:48.348 INFO:tasks.rados.rados.0.smithi064.stdout:1460: finishing write tid 4 to smithi06431743-38 2024-03-04T06:13:48.348 INFO:tasks.rados.rados.0.smithi064.stdout:1467: seq_num 701 ranges {581707=654025,1687642=453571,2700812=703681} 2024-03-04T06:13:48.356 INFO:tasks.rados.rados.0.smithi064.stdout:1467: writing smithi06431743-22 from 581707 to 1235732 tid 1 2024-03-04T06:13:48.362 INFO:tasks.rados.rados.0.smithi064.stdout:1467: writing smithi06431743-22 from 1687642 to 2141213 tid 2 2024-03-04T06:13:48.372 INFO:tasks.rados.rados.0.smithi064.stdout:1467: writing smithi06431743-22 from 2700812 to 3404493 tid 3 2024-03-04T06:13:48.372 INFO:tasks.rados.rados.0.smithi064.stdout:1468: read oid 9 snap -1 2024-03-04T06:13:48.372 INFO:tasks.rados.rados.0.smithi064.stdout:1468: expect (ObjNum 687 snap 168 seq_num 687) 2024-03-04T06:13:48.372 INFO:tasks.rados.rados.0.smithi064.stdout:1460: finishing write tid 5 to smithi06431743-38 2024-03-04T06:13:48.372 INFO:tasks.rados.rados.0.smithi064.stdout:1460: oid 38 updating version 0 to 285 2024-03-04T06:13:48.372 INFO:tasks.rados.rados.0.smithi064.stdout:1460: oid 38 updating version 285 to 287 2024-03-04T06:13:48.372 INFO:tasks.rados.rados.0.smithi064.stdout:1460: oid 38 version 287 is already newer than 287 2024-03-04T06:13:48.372 INFO:tasks.rados.rados.0.smithi064.stdout:1460: oid 38 updating version 287 to 288 2024-03-04T06:13:48.372 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 288 (ObjNum 697 snap 172 seq_num 697) dirty exists 2024-03-04T06:13:48.373 INFO:tasks.rados.rados.0.smithi064.stdout:1460: left oid 38 (ObjNum 697 snap 172 seq_num 697) 2024-03-04T06:13:48.373 INFO:tasks.rados.rados.0.smithi064.stdout:1464: expect (ObjNum 678 snap 168 seq_num 678) 2024-03-04T06:13:48.509 INFO:tasks.rados.rados.0.smithi064.stdout:1466: finishing write tid 1 to smithi06431743-27 2024-03-04T06:13:48.509 INFO:tasks.rados.rados.0.smithi064.stdout:1467: finishing write tid 1 to smithi06431743-22 2024-03-04T06:13:48.510 INFO:tasks.rados.rados.0.smithi064.stdout:1462: finishing write tid 1 to smithi06431743-36 2024-03-04T06:13:48.510 INFO:tasks.rados.rados.0.smithi064.stdout:1463: expect (ObjNum 655 snap 162 seq_num 655) 2024-03-04T06:13:48.652 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:13:48.652 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:13:48.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:13:48.661 INFO:tasks.rados.rados.0.smithi064.stdout:1466: finishing write tid 2 to smithi06431743-27 2024-03-04T06:13:48.662 INFO:tasks.rados.rados.0.smithi064.stdout:1467: finishing write tid 2 to smithi06431743-22 2024-03-04T06:13:48.662 INFO:tasks.rados.rados.0.smithi064.stdout:1460: done (7 left) 2024-03-04T06:13:48.662 INFO:tasks.rados.rados.0.smithi064.stdout:1463: done (6 left) 2024-03-04T06:13:48.662 INFO:tasks.rados.rados.0.smithi064.stdout:1464: done (5 left) 2024-03-04T06:13:48.662 INFO:tasks.rados.rados.0.smithi064.stdout:1469: write (excl) oid 14 current snap is 172 2024-03-04T06:13:48.662 INFO:tasks.rados.rados.0.smithi064.stdout:1462: finishing write tid 2 to smithi06431743-36 2024-03-04T06:13:48.662 INFO:tasks.rados.rados.0.smithi064.stdout:1469: seq_num 702 ranges {598514=729274,1746124=597229,3128235=514953} 2024-03-04T06:13:48.668 INFO:tasks.rados.rados.0.smithi064.stdout:1469: writing smithi06431743-14 from 598514 to 1327788 tid 1 2024-03-04T06:13:48.674 INFO:tasks.rados.rados.0.smithi064.stdout:1469: writing smithi06431743-14 from 1746124 to 2343353 tid 2 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1469: writing smithi06431743-14 from 3128235 to 3643188 tid 3 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1470: snap_create 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1466: finishing write tid 3 to smithi06431743-27 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1466: finishing write tid 4 to smithi06431743-27 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1466: finishing write tid 5 to smithi06431743-27 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1466: oid 27 updating version 0 to 206 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1466: oid 27 updating version 206 to 209 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1466: oid 27 version 209 is already newer than 207 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1466: oid 27 version 209 is already newer than 209 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 209 (ObjNum 700 snap 172 seq_num 700) dirty exists 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1466: left oid 27 (ObjNum 700 snap 172 seq_num 700) 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1467: finishing write tid 3 to smithi06431743-22 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1462: finishing write tid 3 to smithi06431743-36 2024-03-04T06:13:48.683 INFO:tasks.rados.rados.0.smithi064.stdout:1467: finishing write tid 4 to smithi06431743-22 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1467: finishing write tid 5 to smithi06431743-22 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1467: oid 22 updating version 0 to 498 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1467: oid 22 updating version 498 to 499 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1467: oid 22 updating version 499 to 500 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1467: oid 22 version 500 is already newer than 497 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 500 (ObjNum 701 snap 172 seq_num 701) dirty exists 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1467: left oid 22 (ObjNum 701 snap 172 seq_num 701) 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1462: finishing write tid 4 to smithi06431743-36 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1462: finishing write tid 5 to smithi06431743-36 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1462: oid 36 updating version 0 to 261 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1462: oid 36 updating version 261 to 262 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1462: oid 36 updating version 262 to 263 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1462: oid 36 updating version 263 to 264 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 264 (ObjNum 699 snap 172 seq_num 699) dirty exists 2024-03-04T06:13:48.684 INFO:tasks.rados.rados.0.smithi064.stdout:1462: left oid 36 (ObjNum 699 snap 172 seq_num 699) 2024-03-04T06:13:48.685 INFO:tasks.rados.rados.0.smithi064.stdout:1468: expect (ObjNum 687 snap 168 seq_num 687) 2024-03-04T06:13:48.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:48.783+0000 7f8fd0d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:48.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:48.790+0000 7f8fd0d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:48.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:48.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:48.905+0000 7f2238add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:48.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:48.909+0000 7f2238add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:48.997 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:13:48.998 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:13:49.013 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2024-03-04T06:13:48.781667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108394822,"num_objects":493,"num_object_clones":366,"num_object_copies":1479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":91,"num_read":2742,"num_read_kb":2007306,"num_write":2432,"num_write_kb":627791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4736,"ondisk_log_size":4736,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":384892,"kb_used_data":240156,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377102468,"statfs":{"total":386547056640,"available":386152927232,"internally_reserved":0,"allocated":245919744,"data_stored":262513643,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2134489,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":42,"num_read_kb":26889,"num_write":50,"num_write_kb":15837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000768"},"pg_stats":[{"pgid":"3.f","version":"354'254","reported_seq":1212,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.262850+0000","last_change":"2024-03-04T06:13:44.294883+0000","last_active":"2024-03-04T06:13:45.262850+0000","last_peered":"2024-03-04T06:13:45.262850+0000","last_clean":"2024-03-04T06:13:45.262850+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:13:45.262850+0000","last_undegraded":"2024-03-04T06:13:45.262850+0000","last_fullsized":"2024-03-04T06:13:45.262850+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9361895,"num_objects":35,"num_object_clones":25,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":131,"num_read_kb":88630,"num_write":146,"num_write_kb":39999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"352'204","reported_seq":1176,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.240991+0000","last_change":"2024-03-04T06:13:44.372394+0000","last_active":"2024-03-04T06:13:45.240991+0000","last_peered":"2024-03-04T06:13:45.240991+0000","last_clean":"2024-03-04T06:13:45.240991+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:13:45.240991+0000","last_undegraded":"2024-03-04T06:13:45.240991+0000","last_fullsized":"2024-03-04T06:13:45.240991+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596830,"num_objects":23,"num_object_clones":16,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":86,"num_read_kb":57067,"num_write":90,"num_write_kb":23161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"354'331","reported_seq":1251,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.891150+0000","last_change":"2024-03-04T06:13:45.891150+0000","last_active":"2024-03-04T06:13:45.891150+0000","last_peered":"2024-03-04T06:13:45.891150+0000","last_clean":"2024-03-04T06:13:45.891150+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:13:45.891150+0000","last_undegraded":"2024-03-04T06:13:45.891150+0000","last_fullsized":"2024-03-04T06:13:45.891150+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4840647,"num_objects":38,"num_object_clones":30,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":179,"num_read_kb":125257,"num_write":172,"num_write_kb":47176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"354'237","reported_seq":1210,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.244691+0000","last_change":"2024-03-04T06:13:44.289134+0000","last_active":"2024-03-04T06:13:45.244691+0000","last_peered":"2024-03-04T06:13:45.244691+0000","last_clean":"2024-03-04T06:13:45.244691+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:13:45.244691+0000","last_undegraded":"2024-03-04T06:13:45.244691+0000","last_fullsized":"2024-03-04T06:13:45.244691+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672610,"num_objects":25,"num_object_clones":17,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":148,"num_read_kb":121347,"num_write":130,"num_write_kb":36841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"354'105","reported_seq":1083,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.263161+0000","last_change":"2024-03-04T06:13:45.267528+0000","last_active":"2024-03-04T06:13:45.263161+0000","last_peered":"2024-03-04T06:13:45.263161+0000","last_clean":"2024-03-04T06:13:45.263161+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:13:45.263161+0000","last_undegraded":"2024-03-04T06:13:45.263161+0000","last_fullsized":"2024-03-04T06:13:45.263161+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3529602,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":37,"num_read_kb":17044,"num_write":47,"num_write_kb":11854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"355'280","reported_seq":1224,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.317542+0000","last_change":"2024-03-04T06:13:44.419115+0000","last_active":"2024-03-04T06:13:45.317542+0000","last_peered":"2024-03-04T06:13:45.317542+0000","last_clean":"2024-03-04T06:13:45.317542+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:13:45.317542+0000","last_undegraded":"2024-03-04T06:13:45.317542+0000","last_fullsized":"2024-03-04T06:13:45.317542+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7025590,"num_objects":30,"num_object_clones":23,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":137,"num_read_kb":94393,"num_write":156,"num_write_kb":47268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1025,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.254996+0000","last_change":"2024-03-04T06:13:44.273179+0000","last_active":"2024-03-04T06:13:45.254996+0000","last_peered":"2024-03-04T06:13:45.254996+0000","last_clean":"2024-03-04T06:13:45.254996+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:13:45.254996+0000","last_undegraded":"2024-03-04T06:13:45.254996+0000","last_fullsized":"2024-03-04T06:13:45.254996+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"354'179","reported_seq":1157,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.891621+0000","last_change":"2024-03-04T06:13:45.891621+0000","last_active":"2024-03-04T06:13:45.891621+0000","last_peered":"2024-03-04T06:13:45.891621+0000","last_clean":"2024-03-04T06:13:45.891621+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:13:45.891621+0000","last_undegraded":"2024-03-04T06:13:45.891621+0000","last_fullsized":"2024-03-04T06:13:45.891621+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3008585,"num_objects":20,"num_object_clones":13,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":133,"num_read_kb":125093,"num_write":84,"num_write_kb":22492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1002,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.247032+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:13:45.247032+0000","last_peered":"2024-03-04T06:13:45.247032+0000","last_clean":"2024-03-04T06:13:45.247032+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:13:45.247032+0000","last_undegraded":"2024-03-04T06:13:45.247032+0000","last_fullsized":"2024-03-04T06:13:45.247032+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":863,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.895504+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:13:45.895504+0000","last_peered":"2024-03-04T06:13:45.895504+0000","last_clean":"2024-03-04T06:13:45.895504+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:13:45.895504+0000","last_undegraded":"2024-03-04T06:13:45.895504+0000","last_fullsized":"2024-03-04T06:13:45.895504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"354'310","reported_seq":1226,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.896358+0000","last_change":"2024-03-04T06:13:45.896358+0000","last_active":"2024-03-04T06:13:45.896358+0000","last_peered":"2024-03-04T06:13:45.896358+0000","last_clean":"2024-03-04T06:13:45.896358+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:13:45.896358+0000","last_undegraded":"2024-03-04T06:13:45.896358+0000","last_fullsized":"2024-03-04T06:13:45.896358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6756678,"num_objects":32,"num_object_clones":24,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":167,"num_read_kb":123052,"num_write":147,"num_write_kb":38342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":867,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.256379+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:13:45.256379+0000","last_peered":"2024-03-04T06:13:45.256379+0000","last_clean":"2024-03-04T06:13:45.256379+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:13:45.256379+0000","last_undegraded":"2024-03-04T06:13:45.256379+0000","last_fullsized":"2024-03-04T06:13:45.256379+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"354'242","reported_seq":1174,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.254679+0000","last_change":"2024-03-04T06:13:45.254679+0000","last_active":"2024-03-04T06:13:45.254679+0000","last_peered":"2024-03-04T06:13:45.254679+0000","last_clean":"2024-03-04T06:13:45.254679+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:13:45.254679+0000","last_undegraded":"2024-03-04T06:13:45.254679+0000","last_fullsized":"2024-03-04T06:13:45.254679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7311276,"num_objects":24,"num_object_clones":19,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":129,"num_read_kb":106965,"num_write":101,"num_write_kb":29487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":863,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.886915+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:13:45.886915+0000","last_peered":"2024-03-04T06:13:45.886915+0000","last_clean":"2024-03-04T06:13:45.886915+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:13:45.886915+0000","last_undegraded":"2024-03-04T06:13:45.886915+0000","last_fullsized":"2024-03-04T06:13:45.886915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"355'584","reported_seq":1454,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.309560+0000","last_change":"2024-03-04T06:13:45.311539+0000","last_active":"2024-03-04T06:13:45.309560+0000","last_peered":"2024-03-04T06:13:45.309560+0000","last_clean":"2024-03-04T06:13:45.309560+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:13:45.309560+0000","last_undegraded":"2024-03-04T06:13:45.309560+0000","last_fullsized":"2024-03-04T06:13:45.309560+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454317,"num_objects":59,"num_object_clones":45,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":366,"num_read_kb":296323,"num_write":294,"num_write_kb":78955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":867,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.249913+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:13:45.249913+0000","last_peered":"2024-03-04T06:13:45.249913+0000","last_clean":"2024-03-04T06:13:45.249913+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:13:45.249913+0000","last_undegraded":"2024-03-04T06:13:45.249913+0000","last_fullsized":"2024-03-04T06:13:45.249913+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"354'560","reported_seq":1394,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.247073+0000","last_change":"2024-03-04T06:13:45.253613+0000","last_active":"2024-03-04T06:13:45.247073+0000","last_peered":"2024-03-04T06:13:45.247073+0000","last_clean":"2024-03-04T06:13:45.247073+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:13:45.247073+0000","last_undegraded":"2024-03-04T06:13:45.247073+0000","last_fullsized":"2024-03-04T06:13:45.247073+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11916658,"num_objects":55,"num_object_clones":41,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":236,"num_read_kb":166784,"num_write":263,"num_write_kb":69055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"350'227","reported_seq":1228,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.894537+0000","last_change":"2024-03-04T06:13:44.274867+0000","last_active":"2024-03-04T06:13:45.894537+0000","last_peered":"2024-03-04T06:13:45.894537+0000","last_clean":"2024-03-04T06:13:45.894537+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:13:45.894537+0000","last_undegraded":"2024-03-04T06:13:45.894537+0000","last_fullsized":"2024-03-04T06:13:45.894537+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353310,"num_objects":26,"num_object_clones":18,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":216,"num_read_kb":176020,"num_write":120,"num_write_kb":31838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":901,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.257338+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:13:45.257338+0000","last_peered":"2024-03-04T06:13:45.257338+0000","last_clean":"2024-03-04T06:13:45.257338+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:13:45.257338+0000","last_undegraded":"2024-03-04T06:13:45.257338+0000","last_fullsized":"2024-03-04T06:13:45.257338+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"354'258","reported_seq":1234,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.262929+0000","last_change":"2024-03-04T06:13:45.273249+0000","last_active":"2024-03-04T06:13:45.262929+0000","last_peered":"2024-03-04T06:13:45.262929+0000","last_clean":"2024-03-04T06:13:45.262929+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:13:45.262929+0000","last_undegraded":"2024-03-04T06:13:45.262929+0000","last_fullsized":"2024-03-04T06:13:45.262929+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865907,"num_objects":31,"num_object_clones":20,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":145,"num_read_kb":93706,"num_write":127,"num_write_kb":33476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":869,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.253577+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:13:45.253577+0000","last_peered":"2024-03-04T06:13:45.253577+0000","last_clean":"2024-03-04T06:13:45.253577+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:13:45.253577+0000","last_undegraded":"2024-03-04T06:13:45.253577+0000","last_fullsized":"2024-03-04T06:13:45.253577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"355'372","reported_seq":1299,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.359189+0000","last_change":"2024-03-04T06:13:45.255220+0000","last_active":"2024-03-04T06:13:45.359189+0000","last_peered":"2024-03-04T06:13:45.359189+0000","last_clean":"2024-03-04T06:13:45.359189+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:13:45.359189+0000","last_undegraded":"2024-03-04T06:13:45.359189+0000","last_fullsized":"2024-03-04T06:13:45.359189+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13496549,"num_objects":40,"num_object_clones":29,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":218,"num_read_kb":161055,"num_write":197,"num_write_kb":54817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":901,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.246050+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:13:45.246050+0000","last_peered":"2024-03-04T06:13:45.246050+0000","last_clean":"2024-03-04T06:13:45.246050+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:13:45.246050+0000","last_undegraded":"2024-03-04T06:13:45.246050+0000","last_fullsized":"2024-03-04T06:13:45.246050+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"355'496","reported_seq":1396,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.248083+0000","last_change":"2024-03-04T06:13:45.280959+0000","last_active":"2024-03-04T06:13:45.248083+0000","last_peered":"2024-03-04T06:13:45.248083+0000","last_clean":"2024-03-04T06:13:45.248083+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:13:45.248083+0000","last_undegraded":"2024-03-04T06:13:45.248083+0000","last_fullsized":"2024-03-04T06:13:45.248083+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9614000,"num_objects":44,"num_object_clones":39,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":311,"num_read_kb":254479,"num_write":221,"num_write_kb":61136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":863,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-03-04T06:13:45.891453+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:13:45.891453+0000","last_peered":"2024-03-04T06:13:45.891453+0000","last_clean":"2024-03-04T06:13:45.891453+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:13:45.891453+0000","last_undegraded":"2024-03-04T06:13:45.891453+0000","last_fullsized":"2024-03-04T06:13:45.891453+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":107804454,"num_objects":491,"num_object_clones":366,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":91,"num_read":2639,"num_read_kb":2007215,"num_write":2295,"num_write_kb":625897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4639,"ondisk_log_size":4639,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":111856,"kb_used_data":72552,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94259984,"statfs":{"total":96636764160,"available":96522223616,"internally_reserved":0,"allocated":74293248,"data_stored":78762598,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96040,"kb_used_data":60896,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94275800,"statfs":{"total":96636764160,"available":96538419200,"internally_reserved":0,"allocated":62357504,"data_stored":66804394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83748,"kb_used_data":48604,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94288092,"statfs":{"total":96636764160,"available":96551006208,"internally_reserved":0,"allocated":49770496,"data_stored":53868982,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":93248,"kb_used_data":58104,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94278592,"statfs":{"total":96636764160,"available":96541278208,"internally_reserved":0,"allocated":59498496,"data_stored":63077669,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:13:49.016 INFO:tasks.rados.rados.0.smithi064.stdout:1469: finishing write tid 1 to smithi06431743-14 2024-03-04T06:13:49.016 INFO:tasks.rados.rados.0.smithi064.stdout:1469: finishing write tid 2 to smithi06431743-14 2024-03-04T06:13:49.016 INFO:tasks.rados.rados.0.smithi064.stdout:1469: finishing write tid 3 to smithi06431743-14 2024-03-04T06:13:49.016 INFO:tasks.rados.rados.0.smithi064.stdout:1469: finishing write tid 4 to smithi06431743-14 2024-03-04T06:13:49.016 INFO:tasks.rados.rados.0.smithi064.stdout:1469: finishing write tid 5 to smithi06431743-14 2024-03-04T06:13:49.016 INFO:tasks.rados.rados.0.smithi064.stdout:1469: oid 14 updating version 0 to 379 2024-03-04T06:13:49.016 INFO:tasks.rados.rados.0.smithi064.stdout:1469: oid 14 version 379 is already newer than 376 2024-03-04T06:13:49.016 INFO:tasks.rados.rados.0.smithi064.stdout:1469: oid 14 version 379 is already newer than 379 2024-03-04T06:13:49.016 INFO:tasks.rados.rados.0.smithi064.stdout:1469: oid 14 version 379 is already newer than 377 2024-03-04T06:13:49.016 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 379 (ObjNum 702 snap 172 seq_num 702) dirty exists 2024-03-04T06:13:49.016 INFO:tasks.rados.rados.0.smithi064.stdout:1469: left oid 14 (ObjNum 702 snap 172 seq_num 702) 2024-03-04T06:13:49.258 INFO:tasks.rados.rados.0.smithi064.stdout:1462: done (6 left) 2024-03-04T06:13:49.258 INFO:tasks.rados.rados.0.smithi064.stdout:1465: done (5 left) 2024-03-04T06:13:49.259 INFO:tasks.rados.rados.0.smithi064.stdout:1466: done (4 left) 2024-03-04T06:13:49.259 INFO:tasks.rados.rados.0.smithi064.stdout:1467: done (3 left) 2024-03-04T06:13:49.259 INFO:tasks.rados.rados.0.smithi064.stdout:1468: done (2 left) 2024-03-04T06:13:49.259 INFO:tasks.rados.rados.0.smithi064.stdout:1469: done (1 left) 2024-03-04T06:13:49.259 INFO:tasks.rados.rados.0.smithi064.stdout:1470: done (0 left) 2024-03-04T06:13:49.259 INFO:tasks.rados.rados.0.smithi064.stdout:1471: snap_create 2024-03-04T06:13:50.257 INFO:tasks.rados.rados.0.smithi064.stdout:1471: done (0 left) 2024-03-04T06:13:50.257 INFO:tasks.rados.rados.0.smithi064.stdout:1472: write (excl) oid 47 current snap is 174 2024-03-04T06:13:50.258 INFO:tasks.rados.rados.0.smithi064.stdout:1472: seq_num 703 ranges {615321=404523,1804606=457240,2755658=726225} 2024-03-04T06:13:50.262 INFO:tasks.rados.rados.0.smithi064.stdout:1472: writing smithi06431743-47 from 615321 to 1019844 tid 1 2024-03-04T06:13:50.269 INFO:tasks.rados.rados.0.smithi064.stdout:1472: writing smithi06431743-47 from 1804606 to 2261846 tid 2 2024-03-04T06:13:50.277 INFO:tasks.rados.rados.0.smithi064.stdout:1472: writing smithi06431743-47 from 2755658 to 3481883 tid 3 2024-03-04T06:13:50.277 INFO:tasks.rados.rados.0.smithi064.stdout:1473: read oid 11 snap 171 2024-03-04T06:13:50.277 INFO:tasks.rados.rados.0.smithi064.stdout:1473: expect (ObjNum 487 snap 114 seq_num 487) 2024-03-04T06:13:50.277 INFO:tasks.rados.rados.0.smithi064.stdout:1474: write (excl) oid 42 current snap is 174 2024-03-04T06:13:50.277 INFO:tasks.rados.rados.0.smithi064.stdout:1474: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2024-03-04T06:13:50.283 INFO:tasks.rados.rados.0.smithi064.stdout:1474: writing smithi06431743-42 from 632128 to 1111900 tid 1 2024-03-04T06:13:50.290 INFO:tasks.rados.rados.0.smithi064.stdout:1474: writing smithi06431743-42 from 1746735 to 2347633 tid 2 2024-03-04T06:13:50.299 INFO:tasks.rados.rados.0.smithi064.stdout:1474: writing smithi06431743-42 from 3066728 to 3720578 tid 3 2024-03-04T06:13:50.299 INFO:tasks.rados.rados.0.smithi064.stdout:1475: snap_create 2024-03-04T06:13:50.348 INFO:tasks.rados.rados.0.smithi064.stdout:1473: expect (ObjNum 487 snap 114 seq_num 487) 2024-03-04T06:13:50.481 INFO:tasks.rados.rados.0.smithi064.stdout:1472: finishing write tid 1 to smithi06431743-47 2024-03-04T06:13:50.481 INFO:tasks.rados.rados.0.smithi064.stdout:1474: finishing write tid 1 to smithi06431743-42 2024-03-04T06:13:50.481 INFO:tasks.rados.rados.0.smithi064.stdout:1472: finishing write tid 2 to smithi06431743-47 2024-03-04T06:13:50.481 INFO:tasks.rados.rados.0.smithi064.stdout:1474: finishing write tid 2 to smithi06431743-42 2024-03-04T06:13:50.481 INFO:tasks.rados.rados.0.smithi064.stdout:1474: finishing write tid 3 to smithi06431743-42 2024-03-04T06:13:50.481 INFO:tasks.rados.rados.0.smithi064.stdout:1472: finishing write tid 3 to smithi06431743-47 2024-03-04T06:13:50.481 INFO:tasks.rados.rados.0.smithi064.stdout:1474: finishing write tid 4 to smithi06431743-42 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1474: finishing write tid 5 to smithi06431743-42 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1474: oid 42 updating version 0 to 256 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1474: oid 42 updating version 256 to 259 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1474: oid 42 version 259 is already newer than 258 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1474: oid 42 version 259 is already newer than 258 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 259 (ObjNum 704 snap 174 seq_num 704) dirty exists 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1474: left oid 42 (ObjNum 704 snap 174 seq_num 704) 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1472: finishing write tid 4 to smithi06431743-47 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1472: finishing write tid 5 to smithi06431743-47 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1472: oid 47 updating version 0 to 242 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1472: oid 47 updating version 242 to 243 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1472: oid 47 version 243 is already newer than 240 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1472: oid 47 version 243 is already newer than 241 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 243 (ObjNum 703 snap 174 seq_num 703) dirty exists 2024-03-04T06:13:50.482 INFO:tasks.rados.rados.0.smithi064.stdout:1472: left oid 47 (ObjNum 703 snap 174 seq_num 703) 2024-03-04T06:13:50.986 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:51.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:51.095+0000 7f46a9aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:51.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:51.099+0000 7f46a9aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:51.274 INFO:tasks.rados.rados.0.smithi064.stdout:1472: done (3 left) 2024-03-04T06:13:51.274 INFO:tasks.rados.rados.0.smithi064.stdout:1473: done (2 left) 2024-03-04T06:13:51.274 INFO:tasks.rados.rados.0.smithi064.stdout:1474: done (1 left) 2024-03-04T06:13:51.274 INFO:tasks.rados.rados.0.smithi064.stdout:1475: done (0 left) 2024-03-04T06:13:51.274 INFO:tasks.rados.rados.0.smithi064.stdout:1476: read oid 14 snap -1 2024-03-04T06:13:51.274 INFO:tasks.rados.rados.0.smithi064.stdout:1476: expect (ObjNum 702 snap 172 seq_num 702) 2024-03-04T06:13:51.274 INFO:tasks.rados.rados.0.smithi064.stdout:1477: read oid 23 snap -1 2024-03-04T06:13:51.274 INFO:tasks.rados.rados.0.smithi064.stdout:1477: expect deleted 2024-03-04T06:13:51.274 INFO:tasks.rados.rados.0.smithi064.stdout:1478: snap_create 2024-03-04T06:13:51.323 INFO:tasks.rados.rados.0.smithi064.stdout:1476: expect (ObjNum 702 snap 172 seq_num 702) 2024-03-04T06:13:52.273 INFO:tasks.rados.rados.0.smithi064.stdout:1476: done (2 left) 2024-03-04T06:13:52.273 INFO:tasks.rados.rados.0.smithi064.stdout:1477: done (1 left) 2024-03-04T06:13:52.273 INFO:tasks.rados.rados.0.smithi064.stdout:1478: done (0 left) 2024-03-04T06:13:52.274 INFO:tasks.rados.rados.0.smithi064.stdout:1479: write oid 17 current snap is 176 2024-03-04T06:13:52.274 INFO:tasks.rados.rados.0.smithi064.stdout:1479: seq_num 705 ranges {648935=555021,1805217=460909,2694151=465122,3817442=31493} 2024-03-04T06:13:52.281 INFO:tasks.rados.rados.0.smithi064.stdout:1479: writing smithi06431743-17 from 648935 to 1203956 tid 1 2024-03-04T06:13:52.288 INFO:tasks.rados.rados.0.smithi064.stdout:1479: writing smithi06431743-17 from 1805217 to 2266126 tid 2 2024-03-04T06:13:52.294 INFO:tasks.rados.rados.0.smithi064.stdout:1479: writing smithi06431743-17 from 2694151 to 3159273 tid 3 2024-03-04T06:13:52.298 INFO:tasks.rados.rados.0.smithi064.stdout:1479: writing smithi06431743-17 from 3817442 to 3848935 tid 4 2024-03-04T06:13:52.299 INFO:tasks.rados.rados.0.smithi064.stdout:1480: snap_create 2024-03-04T06:13:52.332 INFO:tasks.rados.rados.0.smithi064.stdout:1479: finishing write tid 1 to smithi06431743-17 2024-03-04T06:13:52.348 INFO:tasks.rados.rados.0.smithi064.stdout:1479: finishing write tid 2 to smithi06431743-17 2024-03-04T06:13:52.354 INFO:tasks.rados.rados.0.smithi064.stdout:1479: finishing write tid 3 to smithi06431743-17 2024-03-04T06:13:52.367 INFO:tasks.rados.rados.0.smithi064.stdout:1479: finishing write tid 4 to smithi06431743-17 2024-03-04T06:13:52.374 INFO:tasks.rados.rados.0.smithi064.stdout:1479: finishing write tid 5 to smithi06431743-17 2024-03-04T06:13:52.375 INFO:tasks.rados.rados.0.smithi064.stdout:1479: finishing write tid 6 to smithi06431743-17 2024-03-04T06:13:52.375 INFO:tasks.rados.rados.0.smithi064.stdout:1479: oid 17 updating version 0 to 184 2024-03-04T06:13:52.375 INFO:tasks.rados.rados.0.smithi064.stdout:1479: oid 17 version 184 is already newer than 181 2024-03-04T06:13:52.375 INFO:tasks.rados.rados.0.smithi064.stdout:1479: oid 17 version 184 is already newer than 183 2024-03-04T06:13:52.375 INFO:tasks.rados.rados.0.smithi064.stdout:1479: oid 17 version 184 is already newer than 183 2024-03-04T06:13:52.375 INFO:tasks.rados.rados.0.smithi064.stdout:1479: oid 17 updating version 184 to 185 2024-03-04T06:13:52.375 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 185 (ObjNum 705 snap 176 seq_num 705) dirty exists 2024-03-04T06:13:52.375 INFO:tasks.rados.rados.0.smithi064.stdout:1479: left oid 17 (ObjNum 705 snap 176 seq_num 705) 2024-03-04T06:13:53.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:53.232 INFO:tasks.rados.rados.0.smithi064.stdout:1479: done (1 left) 2024-03-04T06:13:53.232 INFO:tasks.rados.rados.0.smithi064.stdout:1480: done (0 left) 2024-03-04T06:13:53.232 INFO:tasks.rados.rados.0.smithi064.stdout:1481: write (excl) oid 3 current snap is 177 2024-03-04T06:13:53.232 INFO:tasks.rados.rados.0.smithi064.stdout:1481: seq_num 706 ranges {665742=630270,1863699=604567,3121574=676394} 2024-03-04T06:13:53.239 INFO:tasks.rados.rados.0.smithi064.stdout:1481: writing smithi06431743-3 from 665742 to 1296012 tid 1 2024-03-04T06:13:53.247 INFO:tasks.rados.rados.0.smithi064.stdout:1481: writing smithi06431743-3 from 1863699 to 2468266 tid 2 2024-03-04T06:13:53.256 INFO:tasks.rados.rados.0.smithi064.stdout:1481: writing smithi06431743-3 from 3121574 to 3797968 tid 3 2024-03-04T06:13:53.256 INFO:tasks.rados.rados.0.smithi064.stdout:1482: write (excl) oid 12 current snap is 177 2024-03-04T06:13:53.256 INFO:tasks.rados.rados.0.smithi064.stdout:1482: seq_num 707 ranges {682549=705519,1922181=464578,3148997=487666} 2024-03-04T06:13:53.266 INFO:tasks.rados.rados.0.smithi064.stdout:1482: writing smithi06431743-12 from 682549 to 1388068 tid 1 2024-03-04T06:13:53.271 INFO:tasks.rados.rados.0.smithi064.stdout:1482: writing smithi06431743-12 from 1922181 to 2386759 tid 2 2024-03-04T06:13:53.279 INFO:tasks.rados.rados.0.smithi064.stdout:1482: writing smithi06431743-12 from 3148997 to 3636663 tid 3 2024-03-04T06:13:53.280 INFO:tasks.rados.rados.0.smithi064.stdout:1483: write oid 21 current snap is 177 2024-03-04T06:13:53.280 INFO:tasks.rados.rados.0.smithi064.stdout:1483: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2024-03-04T06:13:53.286 INFO:tasks.rados.rados.0.smithi064.stdout:1483: writing smithi06431743-21 from 699356 to 1196477 tid 1 2024-03-04T06:13:53.294 INFO:tasks.rados.rados.0.smithi064.stdout:1483: writing smithi06431743-21 from 1980663 to 2588899 tid 2 2024-03-04T06:13:53.301 INFO:tasks.rados.rados.0.smithi064.stdout:1483: writing smithi06431743-21 from 3176420 to 3591711 tid 3 2024-03-04T06:13:53.302 INFO:tasks.rados.rados.0.smithi064.stdout:1484: write oid 16 current snap is 177 2024-03-04T06:13:53.302 INFO:tasks.rados.rados.0.smithi064.stdout:1484: seq_num 709 ranges {716163=572370,2039145=468247,3203843=626563} 2024-03-04T06:13:53.303 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:53.300+0000 7fa6ab19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:53.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:53.307+0000 7fa6ab19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:53.310 INFO:tasks.rados.rados.0.smithi064.stdout:1484: writing smithi06431743-16 from 716163 to 1288533 tid 1 2024-03-04T06:13:53.319 INFO:tasks.rados.rados.0.smithi064.stdout:1484: writing smithi06431743-16 from 2039145 to 2507392 tid 2 2024-03-04T06:13:53.328 INFO:tasks.rados.rados.0.smithi064.stdout:1484: writing smithi06431743-16 from 3203843 to 3830406 tid 3 2024-03-04T06:13:53.329 INFO:tasks.rados.rados.0.smithi064.stdout:1485: delete oid 18 current snap is 177 2024-03-04T06:13:53.329 INFO:tasks.rados.rados.0.smithi064.stdout:1483: finishing write tid 1 to smithi06431743-21 2024-03-04T06:13:53.339 INFO:tasks.rados.rados.0.smithi064.stdout:1485: done (4 left) 2024-03-04T06:13:53.339 INFO:tasks.rados.rados.0.smithi064.stdout:1486: snap_create 2024-03-04T06:13:53.340 INFO:tasks.rados.rados.0.smithi064.stdout:1484: finishing write tid 1 to smithi06431743-16 2024-03-04T06:13:53.343 INFO:tasks.rados.rados.0.smithi064.stdout:1483: finishing write tid 2 to smithi06431743-21 2024-03-04T06:13:53.357 INFO:tasks.rados.rados.0.smithi064.stdout:1483: finishing write tid 3 to smithi06431743-21 2024-03-04T06:13:53.361 INFO:tasks.rados.rados.0.smithi064.stdout:1483: finishing write tid 4 to smithi06431743-21 2024-03-04T06:13:53.362 INFO:tasks.rados.rados.0.smithi064.stdout:1483: finishing write tid 5 to smithi06431743-21 2024-03-04T06:13:53.362 INFO:tasks.rados.rados.0.smithi064.stdout:1483: oid 21 updating version 0 to 232 2024-03-04T06:13:53.362 INFO:tasks.rados.rados.0.smithi064.stdout:1483: oid 21 version 232 is already newer than 230 2024-03-04T06:13:53.362 INFO:tasks.rados.rados.0.smithi064.stdout:1483: oid 21 version 232 is already newer than 232 2024-03-04T06:13:53.362 INFO:tasks.rados.rados.0.smithi064.stdout:1483: oid 21 version 232 is already newer than 229 2024-03-04T06:13:53.362 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 232 (ObjNum 708 snap 177 seq_num 708) dirty exists 2024-03-04T06:13:53.362 INFO:tasks.rados.rados.0.smithi064.stdout:1483: left oid 21 (ObjNum 708 snap 177 seq_num 708) 2024-03-04T06:13:53.401 INFO:tasks.rados.rados.0.smithi064.stdout:1481: finishing write tid 1 to smithi06431743-3 2024-03-04T06:13:53.411 INFO:tasks.rados.rados.0.smithi064.stdout:1484: finishing write tid 2 to smithi06431743-16 2024-03-04T06:13:53.415 INFO:tasks.rados.rados.0.smithi064.stdout:1482: finishing write tid 1 to smithi06431743-12 2024-03-04T06:13:53.420 INFO:tasks.rados.rados.0.smithi064.stdout:1481: finishing write tid 2 to smithi06431743-3 2024-03-04T06:13:53.425 INFO:tasks.rados.rados.0.smithi064.stdout:1484: finishing write tid 3 to smithi06431743-16 2024-03-04T06:13:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:1482: finishing write tid 2 to smithi06431743-12 2024-03-04T06:13:53.433 INFO:tasks.rados.rados.0.smithi064.stdout:1484: finishing write tid 4 to smithi06431743-16 2024-03-04T06:13:53.434 INFO:tasks.rados.rados.0.smithi064.stdout:1481: finishing write tid 3 to smithi06431743-3 2024-03-04T06:13:53.436 INFO:tasks.rados.rados.0.smithi064.stdout:1484: finishing write tid 5 to smithi06431743-16 2024-03-04T06:13:53.437 INFO:tasks.rados.rados.0.smithi064.stdout:1484: oid 16 updating version 0 to 266 2024-03-04T06:13:53.437 INFO:tasks.rados.rados.0.smithi064.stdout:1484: oid 16 updating version 266 to 268 2024-03-04T06:13:53.437 INFO:tasks.rados.rados.0.smithi064.stdout:1484: oid 16 version 268 is already newer than 265 2024-03-04T06:13:53.437 INFO:tasks.rados.rados.0.smithi064.stdout:1484: oid 16 version 268 is already newer than 267 2024-03-04T06:13:53.437 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 268 (ObjNum 709 snap 177 seq_num 709) dirty exists 2024-03-04T06:13:53.437 INFO:tasks.rados.rados.0.smithi064.stdout:1484: left oid 16 (ObjNum 709 snap 177 seq_num 709) 2024-03-04T06:13:53.438 INFO:tasks.rados.rados.0.smithi064.stdout:1481: finishing write tid 4 to smithi06431743-3 2024-03-04T06:13:53.439 INFO:tasks.rados.rados.0.smithi064.stdout:1481: finishing write tid 5 to smithi06431743-3 2024-03-04T06:13:53.439 INFO:tasks.rados.rados.0.smithi064.stdout:1481: oid 3 updating version 0 to 315 2024-03-04T06:13:53.439 INFO:tasks.rados.rados.0.smithi064.stdout:1481: oid 3 version 315 is already newer than 315 2024-03-04T06:13:53.439 INFO:tasks.rados.rados.0.smithi064.stdout:1481: oid 3 version 315 is already newer than 313 2024-03-04T06:13:53.439 INFO:tasks.rados.rados.0.smithi064.stdout:1481: oid 3 version 315 is already newer than 312 2024-03-04T06:13:53.439 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 315 (ObjNum 706 snap 177 seq_num 706) dirty exists 2024-03-04T06:13:53.439 INFO:tasks.rados.rados.0.smithi064.stdout:1481: left oid 3 (ObjNum 706 snap 177 seq_num 706) 2024-03-04T06:13:53.443 INFO:tasks.rados.rados.0.smithi064.stdout:1482: finishing write tid 3 to smithi06431743-12 2024-03-04T06:13:53.451 INFO:tasks.rados.rados.0.smithi064.stdout:1482: finishing write tid 4 to smithi06431743-12 2024-03-04T06:13:53.452 INFO:tasks.rados.rados.0.smithi064.stdout:1482: finishing write tid 5 to smithi06431743-12 2024-03-04T06:13:53.452 INFO:tasks.rados.rados.0.smithi064.stdout:1482: oid 12 updating version 0 to 246 2024-03-04T06:13:53.452 INFO:tasks.rados.rados.0.smithi064.stdout:1482: oid 12 updating version 246 to 247 2024-03-04T06:13:53.452 INFO:tasks.rados.rados.0.smithi064.stdout:1482: oid 12 version 247 is already newer than 245 2024-03-04T06:13:53.452 INFO:tasks.rados.rados.0.smithi064.stdout:1482: oid 12 version 247 is already newer than 244 2024-03-04T06:13:53.453 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 247 (ObjNum 707 snap 177 seq_num 707) dirty exists 2024-03-04T06:13:53.453 INFO:tasks.rados.rados.0.smithi064.stdout:1482: left oid 12 (ObjNum 707 snap 177 seq_num 707) 2024-03-04T06:13:54.015 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:13:54.016 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:13:54.016 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:13:54.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:13:54.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:54.120+0000 7fec320ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:54.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:54.127+0000 7fec320ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:54.140 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:54.140 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:54.140 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:54.140 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:54.140 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:54.140 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:54.140 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:54.140 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:54.140 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:54.140 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:54.141 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:54.141 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:54.143 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:54.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:13:54.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:54.245+0000 7f960279d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:54.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:54.253+0000 7f960279d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:54.265 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:54.265 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:54.265 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:54.266 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:54.266 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:54.266 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:54.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:54.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:54.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:54.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:54.266 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:54.266 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:54.269 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:54.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:13:54.285 INFO:tasks.rados.rados.0.smithi064.stdout:1481: done (4 left) 2024-03-04T06:13:54.286 INFO:tasks.rados.rados.0.smithi064.stdout:1482: done (3 left) 2024-03-04T06:13:54.286 INFO:tasks.rados.rados.0.smithi064.stdout:1483: done (2 left) 2024-03-04T06:13:54.286 INFO:tasks.rados.rados.0.smithi064.stdout:1484: done (1 left) 2024-03-04T06:13:54.286 INFO:tasks.rados.rados.0.smithi064.stdout:1486: done (0 left) 2024-03-04T06:13:54.286 INFO:tasks.rados.rados.0.smithi064.stdout:1487: write (excl) oid 45 current snap is 178 2024-03-04T06:13:54.286 INFO:tasks.rados.rados.0.smithi064.stdout:1487: seq_num 711 ranges {749777=722868,2156109=471916,3258689=649107} 2024-03-04T06:13:54.294 INFO:tasks.rados.rados.0.smithi064.stdout:1487: writing smithi06431743-45 from 749777 to 1472645 tid 1 2024-03-04T06:13:54.301 INFO:tasks.rados.rados.0.smithi064.stdout:1487: writing smithi06431743-45 from 2156109 to 2628025 tid 2 2024-03-04T06:13:54.308 INFO:tasks.rados.rados.0.smithi064.stdout:1487: writing smithi06431743-45 from 3258689 to 3907796 tid 3 2024-03-04T06:13:54.308 INFO:tasks.rados.rados.0.smithi064.stdout:1488: snap_remove snap 170 2024-03-04T06:13:54.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:54.389+0000 7f6d1b62e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:54.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:54.397+0000 7f6d1b62e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:54.410 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:54.413 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:54.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:13:54.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:54.556+0000 7fd293514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:54.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:54.564+0000 7fd293514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:13:54.577 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:13:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:13:54.580 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:13:55.290 INFO:tasks.rados.rados.0.smithi064.stdout:1488: done (1 left) 2024-03-04T06:13:55.290 INFO:tasks.rados.rados.0.smithi064.stdout:1489: write oid 31 current snap is 178 2024-03-04T06:13:55.290 INFO:tasks.rados.rados.0.smithi064.stdout:1487: finishing write tid 1 to smithi06431743-45 2024-03-04T06:13:55.291 INFO:tasks.rados.rados.0.smithi064.stdout:1487: finishing write tid 2 to smithi06431743-45 2024-03-04T06:13:55.291 INFO:tasks.rados.rados.0.smithi064.stdout:1487: finishing write tid 3 to smithi06431743-45 2024-03-04T06:13:55.291 INFO:tasks.rados.rados.0.smithi064.stdout:1489: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2024-03-04T06:13:55.298 INFO:tasks.rados.rados.0.smithi064.stdout:1489: writing smithi06431743-31 from 766584 to 1564701 tid 1 2024-03-04T06:13:55.305 INFO:tasks.rados.rados.0.smithi064.stdout:1489: writing smithi06431743-31 from 2098238 to 2713812 tid 2 2024-03-04T06:13:55.311 INFO:tasks.rados.rados.0.smithi064.stdout:1489: writing smithi06431743-31 from 3169759 to 3630138 tid 3 2024-03-04T06:13:55.311 INFO:tasks.rados.rados.0.smithi064.stdout:1490: read oid 40 snap -1 2024-03-04T06:13:55.311 INFO:tasks.rados.rados.0.smithi064.stdout:1490: expect (ObjNum 678 snap 168 seq_num 678) 2024-03-04T06:13:55.312 INFO:tasks.rados.rados.0.smithi064.stdout:1491: read oid 24 snap -1 2024-03-04T06:13:55.312 INFO:tasks.rados.rados.0.smithi064.stdout:1491: expect (ObjNum 669 snap 166 seq_num 669) 2024-03-04T06:13:55.312 INFO:tasks.rados.rados.0.smithi064.stdout:1492: delete oid 9 current snap is 178 2024-03-04T06:13:55.312 INFO:tasks.rados.rados.0.smithi064.stdout:1487: finishing write tid 4 to smithi06431743-45 2024-03-04T06:13:55.312 INFO:tasks.rados.rados.0.smithi064.stdout:1487: finishing write tid 5 to smithi06431743-45 2024-03-04T06:13:55.312 INFO:tasks.rados.rados.0.smithi064.stdout:1487: oid 45 updating version 0 to 335 2024-03-04T06:13:55.312 INFO:tasks.rados.rados.0.smithi064.stdout:1487: oid 45 version 335 is already newer than 333 2024-03-04T06:13:55.312 INFO:tasks.rados.rados.0.smithi064.stdout:1487: oid 45 version 335 is already newer than 332 2024-03-04T06:13:55.312 INFO:tasks.rados.rados.0.smithi064.stdout:1487: oid 45 version 335 is already newer than 334 2024-03-04T06:13:55.312 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 335 (ObjNum 711 snap 178 seq_num 711) dirty exists 2024-03-04T06:13:55.312 INFO:tasks.rados.rados.0.smithi064.stdout:1487: left oid 45 (ObjNum 711 snap 178 seq_num 711) 2024-03-04T06:13:55.350 INFO:tasks.rados.rados.0.smithi064.stdout:1490: expect (ObjNum 678 snap 168 seq_num 678) 2024-03-04T06:13:55.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:55.482 INFO:tasks.rados.rados.0.smithi064.stdout:1491: expect (ObjNum 669 snap 166 seq_num 669) 2024-03-04T06:13:55.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:55.503+0000 7fa21c79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:55.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:55.507+0000 7fa21c79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:55.618 INFO:tasks.rados.rados.0.smithi064.stdout:1489: finishing write tid 1 to smithi06431743-31 2024-03-04T06:13:55.618 INFO:tasks.rados.rados.0.smithi064.stdout:1489: finishing write tid 2 to smithi06431743-31 2024-03-04T06:13:55.618 INFO:tasks.rados.rados.0.smithi064.stdout:1489: finishing write tid 3 to smithi06431743-31 2024-03-04T06:13:55.618 INFO:tasks.rados.rados.0.smithi064.stdout:1489: finishing write tid 4 to smithi06431743-31 2024-03-04T06:13:55.618 INFO:tasks.rados.rados.0.smithi064.stdout:1489: finishing write tid 5 to smithi06431743-31 2024-03-04T06:13:55.618 INFO:tasks.rados.rados.0.smithi064.stdout:1489: oid 31 updating version 0 to 107 2024-03-04T06:13:55.618 INFO:tasks.rados.rados.0.smithi064.stdout:1489: oid 31 updating version 107 to 109 2024-03-04T06:13:55.618 INFO:tasks.rados.rados.0.smithi064.stdout:1489: oid 31 version 109 is already newer than 109 2024-03-04T06:13:55.619 INFO:tasks.rados.rados.0.smithi064.stdout:1489: oid 31 updating version 109 to 110 2024-03-04T06:13:55.619 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 110 (ObjNum 712 snap 178 seq_num 712) dirty exists 2024-03-04T06:13:55.619 INFO:tasks.rados.rados.0.smithi064.stdout:1489: left oid 31 (ObjNum 712 snap 178 seq_num 712) 2024-03-04T06:13:55.619 INFO:tasks.rados.rados.0.smithi064.stdout:1487: done (4 left) 2024-03-04T06:13:55.619 INFO:tasks.rados.rados.0.smithi064.stdout:1489: done (3 left) 2024-03-04T06:13:55.619 INFO:tasks.rados.rados.0.smithi064.stdout:1490: done (2 left) 2024-03-04T06:13:55.619 INFO:tasks.rados.rados.0.smithi064.stdout:1491: done (1 left) 2024-03-04T06:13:55.619 INFO:tasks.rados.rados.0.smithi064.stdout:1492: done (0 left) 2024-03-04T06:13:55.619 INFO:tasks.rados.rados.0.smithi064.stdout:1493: delete oid 36 current snap is 178 2024-03-04T06:13:55.637 INFO:tasks.rados.rados.0.smithi064.stdout:1493: done (0 left) 2024-03-04T06:13:55.638 INFO:tasks.rados.rados.0.smithi064.stdout:1494: snap_remove snap 175 2024-03-04T06:13:56.291 INFO:tasks.rados.rados.0.smithi064.stdout:1494: done (0 left) 2024-03-04T06:13:56.291 INFO:tasks.rados.rados.0.smithi064.stdout:1495: snap_create 2024-03-04T06:13:57.292 INFO:tasks.rados.rados.0.smithi064.stdout:1495: done (0 left) 2024-03-04T06:13:57.292 INFO:tasks.rados.rados.0.smithi064.stdout:1496: write (excl) oid 23 current snap is 179 2024-03-04T06:13:57.293 INFO:tasks.rados.rados.0.smithi064.stdout:1496: seq_num 715 ranges {417005=740217,1590037=426968} 2024-03-04T06:13:57.298 INFO:tasks.rados.rados.0.smithi064.stdout:1496: writing smithi06431743-23 from 417005 to 1157222 tid 1 2024-03-04T06:13:57.303 INFO:tasks.rados.rados.0.smithi064.stdout:1496: writing smithi06431743-23 from 1590037 to 2017005 tid 2 2024-03-04T06:13:57.303 INFO:tasks.rados.rados.0.smithi064.stdout:1497: read oid 33 snap 173 2024-03-04T06:13:57.304 INFO:tasks.rados.rados.0.smithi064.stdout:1497: expect (ObjNum 659 snap 164 seq_num 659) 2024-03-04T06:13:57.304 INFO:tasks.rados.rados.0.smithi064.stdout:1498: write oid 12 current snap is 179 2024-03-04T06:13:57.304 INFO:tasks.rados.rados.0.smithi064.stdout:1498: seq_num 716 ranges {433812=415466,1648519=385293} 2024-03-04T06:13:57.308 INFO:tasks.rados.rados.0.smithi064.stdout:1498: writing smithi06431743-12 from 433812 to 849278 tid 1 2024-03-04T06:13:57.315 INFO:tasks.rados.rados.0.smithi064.stdout:1498: writing smithi06431743-12 from 1648519 to 2033812 tid 2 2024-03-04T06:13:57.315 INFO:tasks.rados.rados.0.smithi064.stdout:1499: read oid 44 snap -1 2024-03-04T06:13:57.315 INFO:tasks.rados.rados.0.smithi064.stdout:1499: expect (ObjNum 694 snap 170 seq_num 694) 2024-03-04T06:13:57.315 INFO:tasks.rados.rados.0.smithi064.stdout:1500: write (excl) oid 5 current snap is 179 2024-03-04T06:13:57.315 INFO:tasks.rados.rados.0.smithi064.stdout:1500: seq_num 717 ranges {450619=490715,1590648=459971} 2024-03-04T06:13:57.320 INFO:tasks.rados.rados.0.smithi064.stdout:1500: writing smithi06431743-5 from 450619 to 941334 tid 1 2024-03-04T06:13:57.326 INFO:tasks.rados.rados.0.smithi064.stdout:1500: writing smithi06431743-5 from 1590648 to 2050619 tid 2 2024-03-04T06:13:57.327 INFO:tasks.rados.rados.0.smithi064.stdout:1501: snap_remove snap 168 2024-03-04T06:13:57.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:13:57.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:57.697+0000 7fb2cff14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:57.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:57.701+0000 7fb2cff14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:58.315 INFO:tasks.rados.rados.0.smithi064.stdout:1501: done (5 left) 2024-03-04T06:13:58.315 INFO:tasks.rados.rados.0.smithi064.stdout:1502: delete oid 1 current snap is 179 2024-03-04T06:13:58.316 INFO:tasks.rados.rados.0.smithi064.stdout:1500: finishing write tid 1 to smithi06431743-5 2024-03-04T06:13:58.316 INFO:tasks.rados.rados.0.smithi064.stdout:1497: expect (ObjNum 659 snap 164 seq_num 659) 2024-03-04T06:13:58.495 INFO:tasks.rados.rados.0.smithi064.stdout:1496: finishing write tid 1 to smithi06431743-23 2024-03-04T06:13:58.495 INFO:tasks.rados.rados.0.smithi064.stdout:1496: finishing write tid 2 to smithi06431743-23 2024-03-04T06:13:58.495 INFO:tasks.rados.rados.0.smithi064.stdout:1498: finishing write tid 1 to smithi06431743-12 2024-03-04T06:13:58.495 INFO:tasks.rados.rados.0.smithi064.stdout:1500: finishing write tid 2 to smithi06431743-5 2024-03-04T06:13:58.495 INFO:tasks.rados.rados.0.smithi064.stdout:1499: expect (ObjNum 694 snap 170 seq_num 694) 2024-03-04T06:13:58.760 INFO:tasks.rados.rados.0.smithi064.stdout:1496: finishing write tid 3 to smithi06431743-23 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1496: finishing write tid 4 to smithi06431743-23 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1496: oid 23 updating version 0 to 320 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1496: oid 23 updating version 320 to 322 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1496: oid 23 version 322 is already newer than 322 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 322 (ObjNum 715 snap 179 seq_num 715) dirty exists 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1496: left oid 23 (ObjNum 715 snap 179 seq_num 715) 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1500: finishing write tid 3 to smithi06431743-5 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1496: done (5 left) 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1497: done (4 left) 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1499: done (3 left) 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1502: done (2 left) 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1503: read oid 45 snap -1 2024-03-04T06:13:58.761 INFO:tasks.rados.rados.0.smithi064.stdout:1503: expect (ObjNum 711 snap 178 seq_num 711) 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1504: write (excl) oid 10 current snap is 179 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1500: finishing write tid 4 to smithi06431743-5 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1500: oid 5 updating version 0 to 564 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1500: oid 5 version 564 is already newer than 562 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1500: oid 5 version 564 is already newer than 563 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 564 (ObjNum 717 snap 179 seq_num 717) dirty exists 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1500: left oid 5 (ObjNum 717 snap 179 seq_num 717) 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1498: finishing write tid 2 to smithi06431743-12 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1498: finishing write tid 3 to smithi06431743-12 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1498: finishing write tid 4 to smithi06431743-12 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1498: oid 12 updating version 0 to 253 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1498: oid 12 version 253 is already newer than 251 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1498: oid 12 version 253 is already newer than 252 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 253 (ObjNum 716 snap 179 seq_num 716) dirty exists 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1498: left oid 12 (ObjNum 716 snap 179 seq_num 716) 2024-03-04T06:13:58.762 INFO:tasks.rados.rados.0.smithi064.stdout:1504: seq_num 719 ranges {484233=641213,1707612=376621} 2024-03-04T06:13:58.766 INFO:tasks.rados.rados.0.smithi064.stdout:1504: writing smithi06431743-10 from 484233 to 1125446 tid 1 2024-03-04T06:13:58.772 INFO:tasks.rados.rados.0.smithi064.stdout:1504: writing smithi06431743-10 from 1707612 to 2084233 tid 2 2024-03-04T06:13:58.772 INFO:tasks.rados.rados.0.smithi064.stdout:1498: done (3 left) 2024-03-04T06:13:58.773 INFO:tasks.rados.rados.0.smithi064.stdout:1500: done (2 left) 2024-03-04T06:13:58.773 INFO:tasks.rados.rados.0.smithi064.stdout:1505: delete oid 4 current snap is 179 2024-03-04T06:13:58.791 INFO:tasks.rados.rados.0.smithi064.stdout:1503: expect (ObjNum 711 snap 178 seq_num 711) 2024-03-04T06:13:58.944 INFO:tasks.rados.rados.0.smithi064.stdout:1504: finishing write tid 1 to smithi06431743-10 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1504: finishing write tid 2 to smithi06431743-10 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1504: finishing write tid 3 to smithi06431743-10 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1504: finishing write tid 4 to smithi06431743-10 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1504: oid 10 updating version 0 to 294 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1504: oid 10 version 294 is already newer than 292 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1504: oid 10 version 294 is already newer than 293 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 294 (ObjNum 719 snap 179 seq_num 719) dirty exists 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1504: left oid 10 (ObjNum 719 snap 179 seq_num 719) 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1503: done (2 left) 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1504: done (1 left) 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1505: done (0 left) 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1506: read oid 44 snap -1 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1506: expect (ObjNum 694 snap 170 seq_num 694) 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1507: read oid 42 snap 176 2024-03-04T06:13:58.945 INFO:tasks.rados.rados.0.smithi064.stdout:1507: expect (ObjNum 704 snap 174 seq_num 704) 2024-03-04T06:13:58.946 INFO:tasks.rados.rados.0.smithi064.stdout:1508: write oid 28 current snap is 179 2024-03-04T06:13:58.946 INFO:tasks.rados.rados.0.smithi064.stdout:1508: seq_num 721 ranges {517847=791711,2108223=9624} 2024-03-04T06:13:58.954 INFO:tasks.rados.rados.0.smithi064.stdout:1508: writing smithi06431743-28 from 517847 to 1309558 tid 1 2024-03-04T06:13:58.959 INFO:tasks.rados.rados.0.smithi064.stdout:1508: writing smithi06431743-28 from 2108223 to 2117847 tid 2 2024-03-04T06:13:58.959 INFO:tasks.rados.rados.0.smithi064.stdout:1509: write oid 7 current snap is 179 2024-03-04T06:13:58.959 INFO:tasks.rados.rados.0.smithi064.stdout:1509: seq_num 722 ranges {534654=466960,1766705=367949} 2024-03-04T06:13:58.965 INFO:tasks.rados.rados.0.smithi064.stdout:1509: writing smithi06431743-7 from 534654 to 1001614 tid 1 2024-03-04T06:13:58.972 INFO:tasks.rados.rados.0.smithi064.stdout:1509: writing smithi06431743-7 from 1766705 to 2134654 tid 2 2024-03-04T06:13:58.972 INFO:tasks.rados.rados.0.smithi064.stdout:1510: read oid 26 snap 171 2024-03-04T06:13:58.972 INFO:tasks.rados.rados.0.smithi064.stdout:1510: expect deleted 2024-03-04T06:13:58.972 INFO:tasks.rados.rados.0.smithi064.stdout:1511: write (excl) oid 34 current snap is 179 2024-03-04T06:13:58.972 INFO:tasks.rados.rados.0.smithi064.stdout:1511: seq_num 723 ranges {551461=658562,1941540=209921} 2024-03-04T06:13:58.980 INFO:tasks.rados.rados.0.smithi064.stdout:1511: writing smithi06431743-34 from 551461 to 1210023 tid 1 2024-03-04T06:13:58.986 INFO:tasks.rados.rados.0.smithi064.stdout:1511: writing smithi06431743-34 from 1941540 to 2151461 tid 2 2024-03-04T06:13:58.986 INFO:tasks.rados.rados.0.smithi064.stdout:1512: snap_remove snap 166 2024-03-04T06:13:59.324 INFO:tasks.rados.rados.0.smithi064.stdout:1512: done (6 left) 2024-03-04T06:13:59.324 INFO:tasks.rados.rados.0.smithi064.stdout:1513: write oid 5 current snap is 179 2024-03-04T06:13:59.324 INFO:tasks.rados.rados.0.smithi064.stdout:1513: seq_num 724 ranges {568268=733811,2000022=168246} 2024-03-04T06:13:59.330 INFO:tasks.rados.rados.0.smithi064.stdout:1513: writing smithi06431743-5 from 568268 to 1302079 tid 1 2024-03-04T06:13:59.336 INFO:tasks.rados.rados.0.smithi064.stdout:1513: writing smithi06431743-5 from 2000022 to 2168268 tid 2 2024-03-04T06:13:59.336 INFO:tasks.rados.rados.0.smithi064.stdout:1514: read oid 2 snap -1 2024-03-04T06:13:59.336 INFO:tasks.rados.rados.0.smithi064.stdout:1514: expect (ObjNum 677 snap 168 seq_num 677) 2024-03-04T06:13:59.336 INFO:tasks.rados.rados.0.smithi064.stdout:1515: write oid 30 current snap is 179 2024-03-04T06:13:59.336 INFO:tasks.rados.rados.0.smithi064.stdout:1515: seq_num 725 ranges {585075=409060,1542151=642924} 2024-03-04T06:13:59.341 INFO:tasks.rados.rados.0.smithi064.stdout:1515: writing smithi06431743-30 from 585075 to 994135 tid 1 2024-03-04T06:13:59.348 INFO:tasks.rados.rados.0.smithi064.stdout:1515: writing smithi06431743-30 from 1542151 to 2185075 tid 2 2024-03-04T06:13:59.348 INFO:tasks.rados.rados.0.smithi064.stdout:1516: delete oid 45 current snap is 179 2024-03-04T06:13:59.348 INFO:tasks.rados.rados.0.smithi064.stdout:1508: finishing write tid 1 to smithi06431743-28 2024-03-04T06:13:59.348 INFO:tasks.rados.rados.0.smithi064.stdout:1506: expect (ObjNum 694 snap 170 seq_num 694) 2024-03-04T06:13:59.580 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:13:59.581 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:13:59.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:13:59.625 INFO:tasks.rados.rados.0.smithi064.stdout:1507: expect (ObjNum 704 snap 174 seq_num 704) 2024-03-04T06:13:59.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:59.684+0000 7faba5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:59.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:59.690+0000 7faba5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:59.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:13:59.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:59.872+0000 7fabed62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:59.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:13:59.876+0000 7fabed62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:13:59.908 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:13:59.909 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:13:59.921 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":547,"stamp":"2024-03-04T06:13:59.320031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":132160613,"num_objects":506,"num_object_clones":375,"num_object_copies":1518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":90,"num_read":2806,"num_read_kb":2052913,"num_write":2506,"num_write_kb":651529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4817,"ondisk_log_size":4817,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":452424,"kb_used_data":307688,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377034936,"statfs":{"total":386547056640,"available":386083774464,"internally_reserved":0,"allocated":315072512,"data_stored":344052365,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16779297,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":18,"num_read_kb":4836,"num_write":44,"num_write_kb":14099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.031937"},"pg_stats":[{"pgid":"3.f","version":"360'259","reported_seq":1245,"reported_epoch":365,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:13:55.325850+0000","last_change":"2024-03-04T06:13:55.325850+0000","last_active":"2024-03-04T06:13:55.325850+0000","last_peered":"2024-03-04T06:13:55.325850+0000","last_clean":"2024-03-04T06:13:55.325850+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:13:55.325850+0000","last_undegraded":"2024-03-04T06:13:55.325850+0000","last_fullsized":"2024-03-04T06:13:55.325850+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9563579,"num_objects":36,"num_object_clones":26,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":139,"num_read_kb":94721,"num_write":151,"num_write_kb":41694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"363'210","reported_seq":1199,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-04T06:13:53.259770+0000","last_change":"2024-03-04T06:13:47.274787+0000","last_active":"2024-03-04T06:13:53.259770+0000","last_peered":"2024-03-04T06:13:53.259770+0000","last_clean":"2024-03-04T06:13:53.259770+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:13:53.259770+0000","last_undegraded":"2024-03-04T06:13:53.259770+0000","last_fullsized":"2024-03-04T06:13:53.259770+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3764900,"num_objects":24,"num_object_clones":17,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":87,"num_read_kb":57068,"num_write":95,"num_write_kb":24902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"364'335","reported_seq":1288,"reported_epoch":366,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:13:56.356159+0000","last_change":"2024-03-04T06:13:56.356159+0000","last_active":"2024-03-04T06:13:56.356159+0000","last_peered":"2024-03-04T06:13:56.356159+0000","last_clean":"2024-03-04T06:13:56.356159+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:13:56.356159+0000","last_undegraded":"2024-03-04T06:13:56.356159+0000","last_fullsized":"2024-03-04T06:13:56.356159+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8790424,"num_objects":39,"num_object_clones":30,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":187,"num_read_kb":129329,"num_write":177,"num_write_kb":48977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"363'247","reported_seq":1246,"reported_epoch":365,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:13:55.315498+0000","last_change":"2024-03-04T06:13:55.315498+0000","last_active":"2024-03-04T06:13:55.315498+0000","last_peered":"2024-03-04T06:13:55.315498+0000","last_clean":"2024-03-04T06:13:55.315498+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:13:55.315498+0000","last_undegraded":"2024-03-04T06:13:55.315498+0000","last_fullsized":"2024-03-04T06:13:55.315498+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11076209,"num_objects":26,"num_object_clones":18,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":152,"num_read_kb":121349,"num_write":141,"num_write_kb":40014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"366'112","reported_seq":1117,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-04T06:13:56.346275+0000","last_change":"2024-03-04T06:13:56.361338+0000","last_active":"2024-03-04T06:13:56.346275+0000","last_peered":"2024-03-04T06:13:56.346275+0000","last_clean":"2024-03-04T06:13:56.346275+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:13:56.346275+0000","last_undegraded":"2024-03-04T06:13:56.346275+0000","last_fullsized":"2024-03-04T06:13:56.346275+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3966584,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":38,"num_read_kb":17045,"num_write":52,"num_write_kb":13686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"358'288","reported_seq":1256,"reported_epoch":365,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:13:55.328208+0000","last_change":"2024-03-04T06:13:55.328208+0000","last_active":"2024-03-04T06:13:55.328208+0000","last_peered":"2024-03-04T06:13:55.328208+0000","last_clean":"2024-03-04T06:13:55.328208+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:13:55.328208+0000","last_undegraded":"2024-03-04T06:13:55.328208+0000","last_fullsized":"2024-03-04T06:13:55.328208+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10740069,"num_objects":31,"num_object_clones":23,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":140,"num_read_kb":94395,"num_write":163,"num_write_kb":48856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1050,"reported_epoch":365,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:13:55.330916+0000","last_change":"2024-03-04T06:13:55.330916+0000","last_active":"2024-03-04T06:13:55.330916+0000","last_peered":"2024-03-04T06:13:55.330916+0000","last_clean":"2024-03-04T06:13:55.330916+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:13:55.330916+0000","last_undegraded":"2024-03-04T06:13:55.330916+0000","last_fullsized":"2024-03-04T06:13:55.330916+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"365'187","reported_seq":1194,"reported_epoch":366,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:13:56.352061+0000","last_change":"2024-03-04T06:13:56.318251+0000","last_active":"2024-03-04T06:13:56.352061+0000","last_peered":"2024-03-04T06:13:56.352061+0000","last_clean":"2024-03-04T06:13:56.352061+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:13:56.352061+0000","last_undegraded":"2024-03-04T06:13:56.352061+0000","last_fullsized":"2024-03-04T06:13:56.352061+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3848935,"num_objects":21,"num_object_clones":14,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":140,"num_read_kb":133003,"num_write":90,"num_write_kb":23972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1027,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-04T06:13:56.340670+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:13:56.340670+0000","last_peered":"2024-03-04T06:13:56.340670+0000","last_clean":"2024-03-04T06:13:56.340670+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:13:56.340670+0000","last_undegraded":"2024-03-04T06:13:56.340670+0000","last_fullsized":"2024-03-04T06:13:56.340670+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":890,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-04T06:13:56.361748+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:13:56.361748+0000","last_peered":"2024-03-04T06:13:56.361748+0000","last_clean":"2024-03-04T06:13:56.361748+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:13:56.361748+0000","last_undegraded":"2024-03-04T06:13:56.361748+0000","last_fullsized":"2024-03-04T06:13:56.361748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"363'315","reported_seq":1256,"reported_epoch":366,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:13:56.327261+0000","last_change":"2024-03-04T06:13:56.327261+0000","last_active":"2024-03-04T06:13:56.327261+0000","last_peered":"2024-03-04T06:13:56.327261+0000","last_clean":"2024-03-04T06:13:56.327261+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:13:56.327261+0000","last_undegraded":"2024-03-04T06:13:56.327261+0000","last_fullsized":"2024-03-04T06:13:56.327261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7059204,"num_objects":33,"num_object_clones":25,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":169,"num_read_kb":123053,"num_write":152,"num_write_kb":40210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":891,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-04T06:13:55.325759+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:13:55.325759+0000","last_peered":"2024-03-04T06:13:55.325759+0000","last_clean":"2024-03-04T06:13:55.325759+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:13:55.325759+0000","last_undegraded":"2024-03-04T06:13:55.325759+0000","last_fullsized":"2024-03-04T06:13:55.325759+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"354'242","reported_seq":1202,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-04T06:13:56.330862+0000","last_change":"2024-03-04T06:13:56.350250+0000","last_active":"2024-03-04T06:13:56.330862+0000","last_peered":"2024-03-04T06:13:56.330862+0000","last_clean":"2024-03-04T06:13:56.330862+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:13:56.330862+0000","last_undegraded":"2024-03-04T06:13:56.330862+0000","last_fullsized":"2024-03-04T06:13:56.330862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7311276,"num_objects":24,"num_object_clones":19,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":129,"num_read_kb":106965,"num_write":101,"num_write_kb":29487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":890,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-04T06:13:56.357746+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:13:56.357746+0000","last_peered":"2024-03-04T06:13:56.357746+0000","last_clean":"2024-03-04T06:13:56.357746+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:13:56.357746+0000","last_undegraded":"2024-03-04T06:13:56.357746+0000","last_fullsized":"2024-03-04T06:13:56.357746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"355'584","reported_seq":1483,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-04T06:13:56.340397+0000","last_change":"2024-03-04T06:13:56.352303+0000","last_active":"2024-03-04T06:13:56.340397+0000","last_peered":"2024-03-04T06:13:56.340397+0000","last_clean":"2024-03-04T06:13:56.340397+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:13:56.340397+0000","last_undegraded":"2024-03-04T06:13:56.340397+0000","last_fullsized":"2024-03-04T06:13:56.340397+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454317,"num_objects":59,"num_object_clones":45,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":366,"num_read_kb":296323,"num_write":294,"num_write_kb":78955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":891,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-04T06:13:55.317943+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:13:55.317943+0000","last_peered":"2024-03-04T06:13:55.317943+0000","last_clean":"2024-03-04T06:13:55.317943+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:13:55.317943+0000","last_undegraded":"2024-03-04T06:13:55.317943+0000","last_fullsized":"2024-03-04T06:13:55.317943+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"354'560","reported_seq":1414,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-04T06:13:55.327450+0000","last_change":"2024-03-04T06:13:47.275262+0000","last_active":"2024-03-04T06:13:55.327450+0000","last_peered":"2024-03-04T06:13:55.327450+0000","last_clean":"2024-03-04T06:13:55.327450+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:13:55.327450+0000","last_undegraded":"2024-03-04T06:13:55.327450+0000","last_fullsized":"2024-03-04T06:13:55.327450+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11916658,"num_objects":55,"num_object_clones":41,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":236,"num_read_kb":166784,"num_write":263,"num_write_kb":69055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"363'232","reported_seq":1258,"reported_epoch":366,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:13:56.335883+0000","last_change":"2024-03-04T06:13:56.335883+0000","last_active":"2024-03-04T06:13:56.335883+0000","last_peered":"2024-03-04T06:13:56.335883+0000","last_clean":"2024-03-04T06:13:56.335883+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:13:56.335883+0000","last_undegraded":"2024-03-04T06:13:56.335883+0000","last_fullsized":"2024-03-04T06:13:56.335883+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6807099,"num_objects":27,"num_object_clones":19,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":217,"num_read_kb":176021,"num_write":125,"num_write_kb":33324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":918,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-04T06:13:53.275971+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:13:53.275971+0000","last_peered":"2024-03-04T06:13:53.275971+0000","last_clean":"2024-03-04T06:13:53.275971+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:13:53.275971+0000","last_undegraded":"2024-03-04T06:13:53.275971+0000","last_fullsized":"2024-03-04T06:13:53.275971+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"363'268","reported_seq":1266,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-04T06:13:53.435427+0000","last_change":"2024-03-04T06:13:47.284629+0000","last_active":"2024-03-04T06:13:53.435427+0000","last_peered":"2024-03-04T06:13:53.435427+0000","last_clean":"2024-03-04T06:13:53.435427+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:13:53.435427+0000","last_undegraded":"2024-03-04T06:13:53.435427+0000","last_fullsized":"2024-03-04T06:13:53.435427+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10807297,"num_objects":33,"num_object_clones":21,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":153,"num_read_kb":98382,"num_write":137,"num_write_kb":36980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":894,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-04T06:13:56.350260+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:13:56.350260+0000","last_peered":"2024-03-04T06:13:56.350260+0000","last_clean":"2024-03-04T06:13:56.350260+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:13:56.350260+0000","last_undegraded":"2024-03-04T06:13:56.350260+0000","last_fullsized":"2024-03-04T06:13:56.350260+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"365'381","reported_seq":1338,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-04T06:13:56.338628+0000","last_change":"2024-03-04T06:13:56.351093+0000","last_active":"2024-03-04T06:13:56.338628+0000","last_peered":"2024-03-04T06:13:56.338628+0000","last_clean":"2024-03-04T06:13:56.338628+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:13:56.338628+0000","last_undegraded":"2024-03-04T06:13:56.338628+0000","last_fullsized":"2024-03-04T06:13:56.338628+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14067987,"num_objects":41,"num_object_clones":30,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":232,"num_read_kb":174217,"num_write":202,"num_write_kb":56617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":919,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-04T06:13:55.323617+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:13:55.323617+0000","last_peered":"2024-03-04T06:13:55.323617+0000","last_clean":"2024-03-04T06:13:55.323617+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:13:55.323617+0000","last_undegraded":"2024-03-04T06:13:55.323617+0000","last_fullsized":"2024-03-04T06:13:55.323617+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"358'500","reported_seq":1423,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-03-04T06:13:53.263717+0000","last_change":"2024-03-04T06:13:47.275727+0000","last_active":"2024-03-04T06:13:53.263717+0000","last_peered":"2024-03-04T06:13:53.263717+0000","last_clean":"2024-03-04T06:13:53.263717+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:13:53.263717+0000","last_undegraded":"2024-03-04T06:13:53.263717+0000","last_fullsized":"2024-03-04T06:13:53.263717+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13395707,"num_objects":45,"num_object_clones":39,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":318,"num_read_kb":264167,"num_write":226,"num_write_kb":62906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":887,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-04T06:13:56.325861+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:13:56.325861+0000","last_peered":"2024-03-04T06:13:56.325861+0000","last_clean":"2024-03-04T06:13:56.325861+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:13:56.325861+0000","last_undegraded":"2024-03-04T06:13:56.325861+0000","last_fullsized":"2024-03-04T06:13:56.325861+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":131570245,"num_objects":504,"num_object_clones":375,"num_object_copies":1512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":90,"num_read":2703,"num_read_kb":2052822,"num_write":2369,"num_write_kb":649635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4720,"ondisk_log_size":4720,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":134232,"kb_used_data":94928,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94237608,"statfs":{"total":96636764160,"available":96499310592,"internally_reserved":0,"allocated":97206272,"data_stored":107062348,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":110132,"kb_used_data":74988,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94261708,"statfs":{"total":96636764160,"available":96523988992,"internally_reserved":0,"allocated":76787712,"data_stored":84373802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":97424,"kb_used_data":62280,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94274416,"statfs":{"total":96636764160,"available":96537001984,"internally_reserved":0,"allocated":63774720,"data_stored":69178851,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706033,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":110636,"kb_used_data":75492,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94261204,"statfs":{"total":96636764160,"available":96523472896,"internally_reserved":0,"allocated":77303808,"data_stored":83437364,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:13:59.926 INFO:tasks.rados.rados.0.smithi064.stdout:1508: finishing write tid 2 to smithi06431743-28 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1509: finishing write tid 1 to smithi06431743-7 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1508: finishing write tid 3 to smithi06431743-28 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1506: done (9 left) 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1507: done (8 left) 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1516: done (7 left) 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1517: write oid 36 current snap is 179 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1508: finishing write tid 4 to smithi06431743-28 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1508: oid 28 updating version 0 to 589 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1508: oid 28 version 589 is already newer than 587 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1508: oid 28 version 589 is already newer than 588 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 589 (ObjNum 721 snap 179 seq_num 721) dirty exists 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1508: left oid 28 (ObjNum 721 snap 179 seq_num 721) 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1509: finishing write tid 2 to smithi06431743-7 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1511: finishing write tid 1 to smithi06431743-34 2024-03-04T06:13:59.927 INFO:tasks.rados.rados.0.smithi064.stdout:1509: finishing write tid 3 to smithi06431743-7 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1509: finishing write tid 4 to smithi06431743-7 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1509: oid 7 updating version 0 to 244 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1509: oid 7 updating version 244 to 245 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1509: oid 7 updating version 245 to 246 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 246 (ObjNum 722 snap 179 seq_num 722) dirty exists 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1509: left oid 7 (ObjNum 722 snap 179 seq_num 722) 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1511: finishing write tid 2 to smithi06431743-34 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1511: finishing write tid 3 to smithi06431743-34 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1511: finishing write tid 4 to smithi06431743-34 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1511: oid 34 updating version 0 to 590 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1511: oid 34 updating version 590 to 592 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1511: oid 34 version 592 is already newer than 591 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 592 (ObjNum 723 snap 179 seq_num 723) dirty exists 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1511: left oid 34 (ObjNum 723 snap 179 seq_num 723) 2024-03-04T06:13:59.928 INFO:tasks.rados.rados.0.smithi064.stdout:1517: seq_num 727 ranges {618689=559558,1659115=559574} 2024-03-04T06:13:59.932 INFO:tasks.rados.rados.0.smithi064.stdout:1517: writing smithi06431743-36 from 618689 to 1178247 tid 1 2024-03-04T06:13:59.938 INFO:tasks.rados.rados.0.smithi064.stdout:1517: writing smithi06431743-36 from 1659115 to 2218689 tid 2 2024-03-04T06:13:59.938 INFO:tasks.rados.rados.0.smithi064.stdout:1508: done (7 left) 2024-03-04T06:13:59.938 INFO:tasks.rados.rados.0.smithi064.stdout:1509: done (6 left) 2024-03-04T06:13:59.938 INFO:tasks.rados.rados.0.smithi064.stdout:1510: done (5 left) 2024-03-04T06:13:59.938 INFO:tasks.rados.rados.0.smithi064.stdout:1511: done (4 left) 2024-03-04T06:13:59.938 INFO:tasks.rados.rados.0.smithi064.stdout:1518: delete oid 19 current snap is 179 2024-03-04T06:13:59.939 INFO:tasks.rados.rados.0.smithi064.stdout:1513: finishing write tid 1 to smithi06431743-5 2024-03-04T06:13:59.939 INFO:tasks.rados.rados.0.smithi064.stdout:1514: expect (ObjNum 677 snap 168 seq_num 677) 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1513: finishing write tid 2 to smithi06431743-5 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1513: finishing write tid 3 to smithi06431743-5 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1513: finishing write tid 4 to smithi06431743-5 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1513: oid 5 updating version 0 to 575 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1513: oid 5 version 575 is already newer than 573 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1513: oid 5 version 575 is already newer than 575 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 575 (ObjNum 724 snap 179 seq_num 724) dirty exists 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1513: left oid 5 (ObjNum 724 snap 179 seq_num 724) 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1515: finishing write tid 1 to smithi06431743-30 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1513: done (4 left) 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1514: done (3 left) 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1518: done (2 left) 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1515: finishing write tid 2 to smithi06431743-30 2024-03-04T06:14:00.134 INFO:tasks.rados.rados.0.smithi064.stdout:1519: read oid 1 snap -1 2024-03-04T06:14:00.135 INFO:tasks.rados.rados.0.smithi064.stdout:1519: expect deleted 2024-03-04T06:14:00.135 INFO:tasks.rados.rados.0.smithi064.stdout:1520: snap_remove snap 172 2024-03-04T06:14:00.326 INFO:tasks.rados.rados.0.smithi064.stdout:1520: done (3 left) 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1521: read oid 9 snap -1 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1521: expect deleted 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1522: read oid 44 snap -1 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1522: expect (ObjNum 694 snap 170 seq_num 694) 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1523: write oid 26 current snap is 179 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1515: finishing write tid 3 to smithi06431743-30 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1515: finishing write tid 4 to smithi06431743-30 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1515: oid 30 updating version 0 to 576 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1515: oid 30 updating version 576 to 578 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1515: oid 30 version 578 is already newer than 578 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 578 (ObjNum 725 snap 179 seq_num 725) dirty exists 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1515: left oid 30 (ObjNum 725 snap 179 seq_num 725) 2024-03-04T06:14:00.327 INFO:tasks.rados.rados.0.smithi064.stdout:1523: seq_num 729 ranges {652303=710056,2059726=192577} 2024-03-04T06:14:00.334 INFO:tasks.rados.rados.0.smithi064.stdout:1523: writing smithi06431743-26 from 652303 to 1362359 tid 1 2024-03-04T06:14:00.339 INFO:tasks.rados.rados.0.smithi064.stdout:1523: writing smithi06431743-26 from 2059726 to 2252303 tid 2 2024-03-04T06:14:00.339 INFO:tasks.rados.rados.0.smithi064.stdout:1515: done (5 left) 2024-03-04T06:14:00.339 INFO:tasks.rados.rados.0.smithi064.stdout:1524: write (excl) oid 15 current snap is 179 2024-03-04T06:14:00.339 INFO:tasks.rados.rados.0.smithi064.stdout:1524: seq_num 730 ranges {669110=501658,1834561=434549} 2024-03-04T06:14:00.346 INFO:tasks.rados.rados.0.smithi064.stdout:1524: writing smithi06431743-15 from 669110 to 1170768 tid 1 2024-03-04T06:14:00.353 INFO:tasks.rados.rados.0.smithi064.stdout:1524: writing smithi06431743-15 from 1834561 to 2269110 tid 2 2024-03-04T06:14:00.353 INFO:tasks.rados.rados.0.smithi064.stdout:1525: write (excl) oid 31 current snap is 179 2024-03-04T06:14:00.353 INFO:tasks.rados.rados.0.smithi064.stdout:1525: seq_num 731 ranges {685917=576907,1893043=392874} 2024-03-04T06:14:00.360 INFO:tasks.rados.rados.0.smithi064.stdout:1525: writing smithi06431743-31 from 685917 to 1262824 tid 1 2024-03-04T06:14:00.367 INFO:tasks.rados.rados.0.smithi064.stdout:1525: writing smithi06431743-31 from 1893043 to 2285917 tid 2 2024-03-04T06:14:00.367 INFO:tasks.rados.rados.0.smithi064.stdout:1526: snap_create 2024-03-04T06:14:00.367 INFO:tasks.rados.rados.0.smithi064.stdout:1517: finishing write tid 1 to smithi06431743-36 2024-03-04T06:14:00.367 INFO:tasks.rados.rados.0.smithi064.stdout:1517: finishing write tid 2 to smithi06431743-36 2024-03-04T06:14:00.367 INFO:tasks.rados.rados.0.smithi064.stdout:1517: finishing write tid 3 to smithi06431743-36 2024-03-04T06:14:00.367 INFO:tasks.rados.rados.0.smithi064.stdout:1517: finishing write tid 4 to smithi06431743-36 2024-03-04T06:14:00.367 INFO:tasks.rados.rados.0.smithi064.stdout:1517: oid 36 updating version 0 to 280 2024-03-04T06:14:00.368 INFO:tasks.rados.rados.0.smithi064.stdout:1517: oid 36 updating version 280 to 281 2024-03-04T06:14:00.368 INFO:tasks.rados.rados.0.smithi064.stdout:1517: oid 36 version 281 is already newer than 279 2024-03-04T06:14:00.368 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 281 (ObjNum 727 snap 179 seq_num 727) dirty exists 2024-03-04T06:14:00.368 INFO:tasks.rados.rados.0.smithi064.stdout:1517: left oid 36 (ObjNum 727 snap 179 seq_num 727) 2024-03-04T06:14:00.400 INFO:tasks.rados.rados.0.smithi064.stdout:1522: expect (ObjNum 694 snap 170 seq_num 694) 2024-03-04T06:14:00.665 INFO:tasks.rados.rados.0.smithi064.stdout:1523: finishing write tid 1 to smithi06431743-26 2024-03-04T06:14:00.665 INFO:tasks.rados.rados.0.smithi064.stdout:1525: finishing write tid 1 to smithi06431743-31 2024-03-04T06:14:00.665 INFO:tasks.rados.rados.0.smithi064.stdout:1524: finishing write tid 1 to smithi06431743-15 2024-03-04T06:14:00.665 INFO:tasks.rados.rados.0.smithi064.stdout:1523: finishing write tid 2 to smithi06431743-26 2024-03-04T06:14:00.665 INFO:tasks.rados.rados.0.smithi064.stdout:1524: finishing write tid 2 to smithi06431743-15 2024-03-04T06:14:00.665 INFO:tasks.rados.rados.0.smithi064.stdout:1523: finishing write tid 3 to smithi06431743-26 2024-03-04T06:14:00.665 INFO:tasks.rados.rados.0.smithi064.stdout:1525: finishing write tid 2 to smithi06431743-31 2024-03-04T06:14:00.665 INFO:tasks.rados.rados.0.smithi064.stdout:1524: finishing write tid 3 to smithi06431743-15 2024-03-04T06:14:00.665 INFO:tasks.rados.rados.0.smithi064.stdout:1524: finishing write tid 4 to smithi06431743-15 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1524: oid 15 updating version 0 to 244 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1524: oid 15 version 244 is already newer than 244 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1524: oid 15 version 244 is already newer than 242 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 244 (ObjNum 730 snap 179 seq_num 730) dirty exists 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1524: left oid 15 (ObjNum 730 snap 179 seq_num 730) 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1523: finishing write tid 4 to smithi06431743-26 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1523: oid 26 updating version 0 to 599 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1523: oid 26 updating version 599 to 601 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1523: oid 26 version 601 is already newer than 600 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 601 (ObjNum 729 snap 179 seq_num 729) dirty exists 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1523: left oid 26 (ObjNum 729 snap 179 seq_num 729) 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1525: finishing write tid 3 to smithi06431743-31 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1525: finishing write tid 4 to smithi06431743-31 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1525: oid 31 updating version 0 to 121 2024-03-04T06:14:00.666 INFO:tasks.rados.rados.0.smithi064.stdout:1525: oid 31 updating version 121 to 122 2024-03-04T06:14:00.667 INFO:tasks.rados.rados.0.smithi064.stdout:1525: oid 31 version 122 is already newer than 120 2024-03-04T06:14:00.667 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 122 (ObjNum 731 snap 179 seq_num 731) dirty exists 2024-03-04T06:14:00.667 INFO:tasks.rados.rados.0.smithi064.stdout:1525: left oid 31 (ObjNum 731 snap 179 seq_num 731) 2024-03-04T06:14:01.329 INFO:tasks.rados.rados.0.smithi064.stdout:1517: done (7 left) 2024-03-04T06:14:01.329 INFO:tasks.rados.rados.0.smithi064.stdout:1519: done (6 left) 2024-03-04T06:14:01.330 INFO:tasks.rados.rados.0.smithi064.stdout:1521: done (5 left) 2024-03-04T06:14:01.330 INFO:tasks.rados.rados.0.smithi064.stdout:1522: done (4 left) 2024-03-04T06:14:01.330 INFO:tasks.rados.rados.0.smithi064.stdout:1523: done (3 left) 2024-03-04T06:14:01.330 INFO:tasks.rados.rados.0.smithi064.stdout:1524: done (2 left) 2024-03-04T06:14:01.330 INFO:tasks.rados.rados.0.smithi064.stdout:1525: done (1 left) 2024-03-04T06:14:01.330 INFO:tasks.rados.rados.0.smithi064.stdout:1526: done (0 left) 2024-03-04T06:14:01.330 INFO:tasks.rados.rados.0.smithi064.stdout:1527: read oid 3 snap -1 2024-03-04T06:14:01.330 INFO:tasks.rados.rados.0.smithi064.stdout:1527: expect (ObjNum 706 snap 177 seq_num 706) 2024-03-04T06:14:01.330 INFO:tasks.rados.rados.0.smithi064.stdout:1528: write oid 37 current snap is 180 2024-03-04T06:14:01.330 INFO:tasks.rados.rados.0.smithi064.stdout:1528: seq_num 732 ranges {702724=652156,1951525=351199} 2024-03-04T06:14:01.337 INFO:tasks.rados.rados.0.smithi064.stdout:1528: writing smithi06431743-37 from 702724 to 1354880 tid 1 2024-03-04T06:14:01.342 INFO:tasks.rados.rados.0.smithi064.stdout:1528: writing smithi06431743-37 from 1951525 to 2302724 tid 2 2024-03-04T06:14:01.342 INFO:tasks.rados.rados.0.smithi064.stdout:1529: snap_remove snap 174 2024-03-04T06:14:01.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:02.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:02.075+0000 7f0178e24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:02.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:02.079+0000 7f0178e24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:02.332 INFO:tasks.rados.rados.0.smithi064.stdout:1529: done (2 left) 2024-03-04T06:14:02.332 INFO:tasks.rados.rados.0.smithi064.stdout:1530: read oid 45 snap -1 2024-03-04T06:14:02.332 INFO:tasks.rados.rados.0.smithi064.stdout:1530: expect deleted 2024-03-04T06:14:02.332 INFO:tasks.rados.rados.0.smithi064.stdout:1531: read oid 6 snap 176 2024-03-04T06:14:02.332 INFO:tasks.rados.rados.0.smithi064.stdout:1531: expect deleted 2024-03-04T06:14:02.332 INFO:tasks.rados.rados.0.smithi064.stdout:1532: snap_remove snap 177 2024-03-04T06:14:03.234 INFO:tasks.rados.rados.0.smithi064.stdout:1532: done (4 left) 2024-03-04T06:14:03.234 INFO:tasks.rados.rados.0.smithi064.stdout:1533: write (excl) oid 12 current snap is 180 2024-03-04T06:14:03.234 INFO:tasks.rados.rados.0.smithi064.stdout:1533: seq_num 733 ranges {719531=727405,1893654=425877} 2024-03-04T06:14:03.241 INFO:tasks.rados.rados.0.smithi064.stdout:1533: writing smithi06431743-12 from 719531 to 1446936 tid 1 2024-03-04T06:14:03.246 INFO:tasks.rados.rados.0.smithi064.stdout:1533: writing smithi06431743-12 from 1893654 to 2319531 tid 2 2024-03-04T06:14:03.246 INFO:tasks.rados.rados.0.smithi064.stdout:1534: write oid 31 current snap is 180 2024-03-04T06:14:03.246 INFO:tasks.rados.rados.0.smithi064.stdout:1534: seq_num 734 ranges {736338=402654,1552136=539580} 2024-03-04T06:14:03.253 INFO:tasks.rados.rados.0.smithi064.stdout:1534: writing smithi06431743-31 from 736338 to 1138992 tid 1 2024-03-04T06:14:03.258 INFO:tasks.rados.rados.0.smithi064.stdout:1534: writing smithi06431743-31 from 1552136 to 2091716 tid 2 2024-03-04T06:14:03.258 INFO:tasks.rados.rados.0.smithi064.stdout:1535: write (excl) oid 19 current snap is 180 2024-03-04T06:14:03.258 INFO:tasks.rados.rados.0.smithi064.stdout:1535: seq_num 735 ranges {753145=477903,2010618=342527} 2024-03-04T06:14:03.265 INFO:tasks.rados.rados.0.smithi064.stdout:1535: writing smithi06431743-19 from 753145 to 1231048 tid 1 2024-03-04T06:14:03.271 INFO:tasks.rados.rados.0.smithi064.stdout:1535: writing smithi06431743-19 from 2010618 to 2353145 tid 2 2024-03-04T06:14:03.271 INFO:tasks.rados.rados.0.smithi064.stdout:1536: snap_create 2024-03-04T06:14:03.272 INFO:tasks.rados.rados.0.smithi064.stdout:1527: expect (ObjNum 706 snap 177 seq_num 706) 2024-03-04T06:14:03.528 INFO:tasks.rados.rados.0.smithi064.stdout:1528: finishing write tid 1 to smithi06431743-37 2024-03-04T06:14:03.528 INFO:tasks.rados.rados.0.smithi064.stdout:1528: finishing write tid 2 to smithi06431743-37 2024-03-04T06:14:03.528 INFO:tasks.rados.rados.0.smithi064.stdout:1528: finishing write tid 3 to smithi06431743-37 2024-03-04T06:14:03.528 INFO:tasks.rados.rados.0.smithi064.stdout:1528: finishing write tid 4 to smithi06431743-37 2024-03-04T06:14:03.528 INFO:tasks.rados.rados.0.smithi064.stdout:1528: oid 37 updating version 0 to 215 2024-03-04T06:14:03.528 INFO:tasks.rados.rados.0.smithi064.stdout:1528: oid 37 version 215 is already newer than 215 2024-03-04T06:14:03.528 INFO:tasks.rados.rados.0.smithi064.stdout:1528: oid 37 version 215 is already newer than 213 2024-03-04T06:14:03.528 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 215 (ObjNum 732 snap 180 seq_num 732) dirty exists 2024-03-04T06:14:03.528 INFO:tasks.rados.rados.0.smithi064.stdout:1528: left oid 37 (ObjNum 732 snap 180 seq_num 732) 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1535: finishing write tid 1 to smithi06431743-19 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1535: finishing write tid 2 to smithi06431743-19 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1535: finishing write tid 3 to smithi06431743-19 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1534: finishing write tid 1 to smithi06431743-31 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1535: finishing write tid 4 to smithi06431743-19 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1535: oid 19 updating version 0 to 594 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1535: oid 19 version 594 is already newer than 594 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1535: oid 19 version 594 is already newer than 592 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 594 (ObjNum 735 snap 180 seq_num 735) dirty exists 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1535: left oid 19 (ObjNum 735 snap 180 seq_num 735) 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1533: finishing write tid 1 to smithi06431743-12 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1534: finishing write tid 2 to smithi06431743-31 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1533: finishing write tid 2 to smithi06431743-12 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1534: finishing write tid 3 to smithi06431743-31 2024-03-04T06:14:03.529 INFO:tasks.rados.rados.0.smithi064.stdout:1533: finishing write tid 3 to smithi06431743-12 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:1534: finishing write tid 4 to smithi06431743-31 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:1534: oid 31 updating version 0 to 129 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:1534: oid 31 version 129 is already newer than 128 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:1534: oid 31 updating version 129 to 130 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 130 (ObjNum 734 snap 180 seq_num 734) dirty exists 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:1534: left oid 31 (ObjNum 734 snap 180 seq_num 734) 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:1533: finishing write tid 4 to smithi06431743-12 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:1533: oid 12 updating version 0 to 267 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:1533: oid 12 version 267 is already newer than 266 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:1533: oid 12 version 267 is already newer than 265 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 267 (ObjNum 733 snap 180 seq_num 733) dirty exists 2024-03-04T06:14:03.530 INFO:tasks.rados.rados.0.smithi064.stdout:1533: left oid 12 (ObjNum 733 snap 180 seq_num 733) 2024-03-04T06:14:04.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:04.233 INFO:tasks.rados.rados.0.smithi064.stdout:1527: done (7 left) 2024-03-04T06:14:04.234 INFO:tasks.rados.rados.0.smithi064.stdout:1528: done (6 left) 2024-03-04T06:14:04.234 INFO:tasks.rados.rados.0.smithi064.stdout:1530: done (5 left) 2024-03-04T06:14:04.234 INFO:tasks.rados.rados.0.smithi064.stdout:1531: done (4 left) 2024-03-04T06:14:04.234 INFO:tasks.rados.rados.0.smithi064.stdout:1533: done (3 left) 2024-03-04T06:14:04.234 INFO:tasks.rados.rados.0.smithi064.stdout:1534: done (2 left) 2024-03-04T06:14:04.234 INFO:tasks.rados.rados.0.smithi064.stdout:1535: done (1 left) 2024-03-04T06:14:04.234 INFO:tasks.rados.rados.0.smithi064.stdout:1536: done (0 left) 2024-03-04T06:14:04.234 INFO:tasks.rados.rados.0.smithi064.stdout:1537: write oid 14 current snap is 181 2024-03-04T06:14:04.234 INFO:tasks.rados.rados.0.smithi064.stdout:1537: seq_num 736 ranges {769952=553152,2069100=300852} 2024-03-04T06:14:04.240 INFO:tasks.rados.rados.0.smithi064.stdout:1537: writing smithi06431743-14 from 769952 to 1323104 tid 1 2024-03-04T06:14:04.246 INFO:tasks.rados.rados.0.smithi064.stdout:1537: writing smithi06431743-14 from 2069100 to 2369952 tid 2 2024-03-04T06:14:04.246 INFO:tasks.rados.rados.0.smithi064.stdout:1538: snap_create 2024-03-04T06:14:04.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:04.257+0000 7fce3fd14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:04.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:04.262+0000 7fce3fd14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:04.309 INFO:tasks.rados.rados.0.smithi064.stdout:1537: finishing write tid 1 to smithi06431743-14 2024-03-04T06:14:04.317 INFO:tasks.rados.rados.0.smithi064.stdout:1537: finishing write tid 2 to smithi06431743-14 2024-03-04T06:14:04.325 INFO:tasks.rados.rados.0.smithi064.stdout:1537: finishing write tid 3 to smithi06431743-14 2024-03-04T06:14:04.326 INFO:tasks.rados.rados.0.smithi064.stdout:1537: finishing write tid 4 to smithi06431743-14 2024-03-04T06:14:04.326 INFO:tasks.rados.rados.0.smithi064.stdout:1537: oid 14 updating version 0 to 395 2024-03-04T06:14:04.327 INFO:tasks.rados.rados.0.smithi064.stdout:1537: oid 14 version 395 is already newer than 394 2024-03-04T06:14:04.327 INFO:tasks.rados.rados.0.smithi064.stdout:1537: oid 14 version 395 is already newer than 394 2024-03-04T06:14:04.327 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 395 (ObjNum 736 snap 181 seq_num 736) dirty exists 2024-03-04T06:14:04.327 INFO:tasks.rados.rados.0.smithi064.stdout:1537: left oid 14 (ObjNum 736 snap 181 seq_num 736) 2024-03-04T06:14:04.923 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:14:04.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:14:05.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:05.030+0000 7f0781a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:05.038 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:05.038+0000 7f0781a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:05.235 INFO:tasks.rados.rados.0.smithi064.stdout:1537: done (1 left) 2024-03-04T06:14:05.236 INFO:tasks.rados.rados.0.smithi064.stdout:1538: done (0 left) 2024-03-04T06:14:05.236 INFO:tasks.rados.rados.0.smithi064.stdout:1539: write oid 42 current snap is 182 2024-03-04T06:14:05.236 INFO:tasks.rados.rados.0.smithi064.stdout:1539: seq_num 737 ranges {786759=628401,2011229=375530} 2024-03-04T06:14:05.243 INFO:tasks.rados.rados.0.smithi064.stdout:1539: writing smithi06431743-42 from 786759 to 1415160 tid 1 2024-03-04T06:14:05.249 INFO:tasks.rados.rados.0.smithi064.stdout:1539: writing smithi06431743-42 from 2011229 to 2386759 tid 2 2024-03-04T06:14:05.249 INFO:tasks.rados.rados.0.smithi064.stdout:1540: snap_remove snap 179 2024-03-04T06:14:06.235 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:14:06.238 INFO:tasks.rados.rados.0.smithi064.stdout:1540: done (1 left) 2024-03-04T06:14:06.238 INFO:tasks.rados.rados.0.smithi064.stdout:1541: read oid 39 snap -1 2024-03-04T06:14:06.238 INFO:tasks.rados.rados.0.smithi064.stdout:1541: expect (ObjNum 634 snap 154 seq_num 634) 2024-03-04T06:14:06.238 INFO:tasks.rados.rados.0.smithi064.stdout:1542: write oid 25 current snap is 182 2024-03-04T06:14:06.238 INFO:tasks.rados.rados.0.smithi064.stdout:1539: finishing write tid 1 to smithi06431743-42 2024-03-04T06:14:06.238 INFO:tasks.rados.rados.0.smithi064.stdout:1542: seq_num 738 ranges {403566=420003,1386064=546918} 2024-03-04T06:14:06.242 INFO:tasks.rados.rados.0.smithi064.stdout:1542: writing smithi06431743-25 from 403566 to 823569 tid 1 2024-03-04T06:14:06.249 INFO:tasks.rados.rados.0.smithi064.stdout:1542: writing smithi06431743-25 from 1386064 to 1932982 tid 2 2024-03-04T06:14:06.249 INFO:tasks.rados.rados.0.smithi064.stdout:1543: read oid 43 snap -1 2024-03-04T06:14:06.249 INFO:tasks.rados.rados.0.smithi064.stdout:1543: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:14:06.249 INFO:tasks.rados.rados.0.smithi064.stdout:1544: snap_create 2024-03-04T06:14:06.249 INFO:tasks.rados.rados.0.smithi064.stdout:1539: finishing write tid 2 to smithi06431743-42 2024-03-04T06:14:06.249 INFO:tasks.rados.rados.0.smithi064.stdout:1539: finishing write tid 3 to smithi06431743-42 2024-03-04T06:14:06.249 INFO:tasks.rados.rados.0.smithi064.stdout:1539: finishing write tid 4 to smithi06431743-42 2024-03-04T06:14:06.249 INFO:tasks.rados.rados.0.smithi064.stdout:1539: oid 42 updating version 0 to 269 2024-03-04T06:14:06.249 INFO:tasks.rados.rados.0.smithi064.stdout:1539: oid 42 updating version 269 to 270 2024-03-04T06:14:06.250 INFO:tasks.rados.rados.0.smithi064.stdout:1539: oid 42 updating version 270 to 271 2024-03-04T06:14:06.250 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 271 (ObjNum 737 snap 182 seq_num 737) dirty exists 2024-03-04T06:14:06.250 INFO:tasks.rados.rados.0.smithi064.stdout:1539: left oid 42 (ObjNum 737 snap 182 seq_num 737) 2024-03-04T06:14:06.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:14:06.302 INFO:tasks.rados.rados.0.smithi064.stdout:1543: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:14:06.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:06.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:06.371+0000 7f6342b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:06.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:06.381+0000 7f6342b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:06.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:06.453+0000 7f01ddcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:06.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:06.458+0000 7f01ddcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:06.501 INFO:tasks.rados.rados.0.smithi064.stdout:1541: expect (ObjNum 634 snap 154 seq_num 634) 2024-03-04T06:14:06.639 INFO:tasks.rados.rados.0.smithi064.stdout:1542: finishing write tid 1 to smithi06431743-25 2024-03-04T06:14:06.639 INFO:tasks.rados.rados.0.smithi064.stdout:1542: finishing write tid 2 to smithi06431743-25 2024-03-04T06:14:06.639 INFO:tasks.rados.rados.0.smithi064.stdout:1542: finishing write tid 3 to smithi06431743-25 2024-03-04T06:14:06.639 INFO:tasks.rados.rados.0.smithi064.stdout:1542: finishing write tid 4 to smithi06431743-25 2024-03-04T06:14:06.639 INFO:tasks.rados.rados.0.smithi064.stdout:1542: oid 25 updating version 0 to 341 2024-03-04T06:14:06.639 INFO:tasks.rados.rados.0.smithi064.stdout:1542: oid 25 version 341 is already newer than 340 2024-03-04T06:14:06.639 INFO:tasks.rados.rados.0.smithi064.stdout:1542: oid 25 updating version 341 to 342 2024-03-04T06:14:06.640 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 342 (ObjNum 738 snap 182 seq_num 738) dirty exists 2024-03-04T06:14:06.640 INFO:tasks.rados.rados.0.smithi064.stdout:1542: left oid 25 (ObjNum 738 snap 182 seq_num 738) 2024-03-04T06:14:07.241 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:14:07.245 INFO:tasks.rados.rados.0.smithi064.stdout:1539: done (4 left) 2024-03-04T06:14:07.245 INFO:tasks.rados.rados.0.smithi064.stdout:1541: done (3 left) 2024-03-04T06:14:07.245 INFO:tasks.rados.rados.0.smithi064.stdout:1542: done (2 left) 2024-03-04T06:14:07.245 INFO:tasks.rados.rados.0.smithi064.stdout:1543: done (1 left) 2024-03-04T06:14:07.245 INFO:tasks.rados.rados.0.smithi064.stdout:1544: done (0 left) 2024-03-04T06:14:07.245 INFO:tasks.rados.rados.0.smithi064.stdout:1545: delete oid 13 current snap is 183 2024-03-04T06:14:07.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:14:07.297 INFO:tasks.rados.rados.0.smithi064.stdout:1545: done (0 left) 2024-03-04T06:14:07.298 INFO:tasks.rados.rados.0.smithi064.stdout:1546: write (excl) oid 10 current snap is 183 2024-03-04T06:14:07.298 INFO:tasks.rados.rados.0.smithi064.stdout:1546: seq_num 740 ranges {437180=570501,1503028=550587} 2024-03-04T06:14:07.303 INFO:tasks.rados.rados.0.smithi064.stdout:1546: writing smithi06431743-10 from 437180 to 1007681 tid 1 2024-03-04T06:14:07.309 INFO:tasks.rados.rados.0.smithi064.stdout:1546: writing smithi06431743-10 from 1503028 to 2053615 tid 2 2024-03-04T06:14:07.309 INFO:tasks.rados.rados.0.smithi064.stdout:1547: write (excl) oid 1 current snap is 183 2024-03-04T06:14:07.309 INFO:tasks.rados.rados.0.smithi064.stdout:1547: seq_num 741 ranges {453987=645750,1845157=608830} 2024-03-04T06:14:07.316 INFO:tasks.rados.rados.0.smithi064.stdout:1547: writing smithi06431743-1 from 453987 to 1099737 tid 1 2024-03-04T06:14:07.324 INFO:tasks.rados.rados.0.smithi064.stdout:1547: writing smithi06431743-1 from 1845157 to 2453987 tid 2 2024-03-04T06:14:07.325 INFO:tasks.rados.rados.0.smithi064.stdout:1548: read oid 29 snap -1 2024-03-04T06:14:07.325 INFO:tasks.rados.rados.0.smithi064.stdout:1548: expect (ObjNum 609 snap 146 seq_num 609) 2024-03-04T06:14:07.325 INFO:tasks.rados.rados.0.smithi064.stdout:1549: snap_create 2024-03-04T06:14:07.328 INFO:tasks.rados.rados.0.smithi064.stdout:1546: finishing write tid 1 to smithi06431743-10 2024-03-04T06:14:07.346 INFO:tasks.rados.rados.0.smithi064.stdout:1548: expect (ObjNum 609 snap 146 seq_num 609) 2024-03-04T06:14:07.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:07.369+0000 7f51e4307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:07.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:07.379+0000 7f51e4307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:07.510 INFO:tasks.rados.rados.0.smithi064.stdout:1546: finishing write tid 2 to smithi06431743-10 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1547: finishing write tid 1 to smithi06431743-1 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1546: finishing write tid 3 to smithi06431743-10 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1546: finishing write tid 4 to smithi06431743-10 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1546: oid 10 updating version 0 to 308 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1546: oid 10 version 308 is already newer than 306 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1546: oid 10 version 308 is already newer than 307 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 308 (ObjNum 740 snap 183 seq_num 740) dirty exists 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1546: left oid 10 (ObjNum 740 snap 183 seq_num 740) 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1547: finishing write tid 2 to smithi06431743-1 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1547: finishing write tid 3 to smithi06431743-1 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1547: finishing write tid 4 to smithi06431743-1 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1547: oid 1 updating version 0 to 608 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1547: oid 1 version 608 is already newer than 607 2024-03-04T06:14:07.511 INFO:tasks.rados.rados.0.smithi064.stdout:1547: oid 1 version 608 is already newer than 606 2024-03-04T06:14:07.512 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 608 (ObjNum 741 snap 183 seq_num 741) dirty exists 2024-03-04T06:14:07.512 INFO:tasks.rados.rados.0.smithi064.stdout:1547: left oid 1 (ObjNum 741 snap 183 seq_num 741) 2024-03-04T06:14:08.243 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:14:08.246 INFO:tasks.rados.rados.0.smithi064.stdout:1546: done (3 left) 2024-03-04T06:14:08.246 INFO:tasks.rados.rados.0.smithi064.stdout:1547: done (2 left) 2024-03-04T06:14:08.246 INFO:tasks.rados.rados.0.smithi064.stdout:1548: done (1 left) 2024-03-04T06:14:08.246 INFO:tasks.rados.rados.0.smithi064.stdout:1549: done (0 left) 2024-03-04T06:14:08.246 INFO:tasks.rados.rados.0.smithi064.stdout:1550: snap_create 2024-03-04T06:14:08.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:14:08.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:08.408+0000 7f06a102d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:08.417 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:08.416+0000 7f06a102d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:08.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:08.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:08.641+0000 7f730a31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:08.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:08.645+0000 7f730a31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:14:09.246 INFO:tasks.rados.rados.0.smithi064.stdout:1550: done (0 left) 2024-03-04T06:14:09.246 INFO:tasks.rados.rados.0.smithi064.stdout:1551: delete oid 29 current snap is 185 2024-03-04T06:14:09.257 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:14:09.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:14:09.294 INFO:tasks.rados.rados.0.smithi064.stdout:1551: done (0 left) 2024-03-04T06:14:09.294 INFO:tasks.rados.rados.0.smithi064.stdout:1552: read oid 43 snap -1 2024-03-04T06:14:09.294 INFO:tasks.rados.rados.0.smithi064.stdout:1552: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:14:09.294 INFO:tasks.rados.rados.0.smithi064.stdout:1553: write oid 3 current snap is 185 2024-03-04T06:14:09.294 INFO:tasks.rados.rados.0.smithi064.stdout:1553: seq_num 743 ranges {487601=796248,1962121=525480} 2024-03-04T06:14:09.300 INFO:tasks.rados.rados.0.smithi064.stdout:1553: writing smithi06431743-3 from 487601 to 1283849 tid 1 2024-03-04T06:14:09.308 INFO:tasks.rados.rados.0.smithi064.stdout:1553: writing smithi06431743-3 from 1962121 to 2487601 tid 2 2024-03-04T06:14:09.308 INFO:tasks.rados.rados.0.smithi064.stdout:1554: read oid 18 snap -1 2024-03-04T06:14:09.308 INFO:tasks.rados.rados.0.smithi064.stdout:1554: expect deleted 2024-03-04T06:14:09.308 INFO:tasks.rados.rados.0.smithi064.stdout:1555: read oid 29 snap -1 2024-03-04T06:14:09.308 INFO:tasks.rados.rados.0.smithi064.stdout:1555: expect deleted 2024-03-04T06:14:09.308 INFO:tasks.rados.rados.0.smithi064.stdout:1556: delete oid 28 current snap is 185 2024-03-04T06:14:09.362 INFO:tasks.rados.rados.0.smithi064.stdout:1554: done (4 left) 2024-03-04T06:14:09.362 INFO:tasks.rados.rados.0.smithi064.stdout:1555: done (3 left) 2024-03-04T06:14:09.362 INFO:tasks.rados.rados.0.smithi064.stdout:1556: done (2 left) 2024-03-04T06:14:09.362 INFO:tasks.rados.rados.0.smithi064.stdout:1557: snap_remove snap 173 2024-03-04T06:14:09.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:09.375+0000 7f67cd59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:09.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:09.383+0000 7f67cd59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:09.578 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:14:09.579 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:14:09.590 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":562,"stamp":"2024-03-04T06:14:09.245090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":123791882,"num_objects":529,"num_object_clones":390,"num_object_copies":1587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":94,"num_read":2896,"num_read_kb":2139862,"num_write":2591,"num_write_kb":670170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5054,"ondisk_log_size":5054,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":453964,"kb_used_data":309228,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377033396,"statfs":{"total":386547056640,"available":386082197504,"internally_reserved":0,"allocated":316649472,"data_stored":344908344,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4521644,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":27930,"num_write":28,"num_write_kb":6592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010416"},"pg_stats":[{"pgid":"3.f","version":"375'271","reported_seq":1292,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.308881+0000","last_change":"2024-03-04T06:14:03.291095+0000","last_active":"2024-03-04T06:14:05.308881+0000","last_peered":"2024-03-04T06:14:05.308881+0000","last_clean":"2024-03-04T06:14:05.308881+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:14:05.308881+0000","last_undegraded":"2024-03-04T06:14:05.308881+0000","last_fullsized":"2024-03-04T06:14:05.308881+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8118210,"num_objects":37,"num_object_clones":27,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":152,"num_read_kb":107848,"num_write":155,"num_write_kb":42675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"371'215","reported_seq":1240,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.383517+0000","last_change":"2024-03-04T06:14:03.300077+0000","last_active":"2024-03-04T06:14:04.383517+0000","last_peered":"2024-03-04T06:14:04.383517+0000","last_clean":"2024-03-04T06:14:04.383517+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:14:04.383517+0000","last_undegraded":"2024-03-04T06:14:04.383517+0000","last_fullsized":"2024-03-04T06:14:04.383517+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067624,"num_objects":25,"num_object_clones":17,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":88,"num_read_kb":57069,"num_write":99,"num_write_kb":25882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"376'342","reported_seq":1334,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.340398+0000","last_change":"2024-03-04T06:14:06.273503+0000","last_active":"2024-03-04T06:14:06.340398+0000","last_peered":"2024-03-04T06:14:06.340398+0000","last_clean":"2024-03-04T06:14:06.340398+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:14:06.340398+0000","last_undegraded":"2024-03-04T06:14:06.340398+0000","last_fullsized":"2024-03-04T06:14:06.340398+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7244213,"num_objects":41,"num_object_clones":31,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":201,"num_read_kb":146762,"num_write":182,"num_write_kb":49923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"373'267","reported_seq":1294,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.265829+0000","last_change":"2024-03-04T06:14:03.396095+0000","last_active":"2024-03-04T06:14:05.265829+0000","last_peered":"2024-03-04T06:14:05.265829+0000","last_clean":"2024-03-04T06:14:05.265829+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:14:05.265829+0000","last_undegraded":"2024-03-04T06:14:05.265829+0000","last_fullsized":"2024-03-04T06:14:05.265829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9513191,"num_objects":28,"num_object_clones":20,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":161,"num_read_kb":130099,"num_write":149,"num_write_kb":41924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"373'130","reported_seq":1155,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.258514+0000","last_change":"2024-03-04T06:14:03.395475+0000","last_active":"2024-03-04T06:14:05.258514+0000","last_peered":"2024-03-04T06:14:05.258514+0000","last_clean":"2024-03-04T06:14:05.258514+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:14:05.258514+0000","last_undegraded":"2024-03-04T06:14:05.258514+0000","last_fullsized":"2024-03-04T06:14:05.258514+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336338,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":17047,"num_write":60,"num_write_kb":15555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"373'304","reported_seq":1306,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.272439+0000","last_change":"2024-03-04T06:14:03.317783+0000","last_active":"2024-03-04T06:14:05.272439+0000","last_peered":"2024-03-04T06:14:05.272439+0000","last_clean":"2024-03-04T06:14:05.272439+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:14:05.272439+0000","last_undegraded":"2024-03-04T06:14:05.272439+0000","last_fullsized":"2024-03-04T06:14:05.272439+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9462770,"num_objects":32,"num_object_clones":24,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":160,"num_read_kb":124271,"num_write":167,"num_write_kb":49851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1087,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.273084+0000","last_change":"2024-03-04T06:14:03.289424+0000","last_active":"2024-03-04T06:14:05.273084+0000","last_peered":"2024-03-04T06:14:05.273084+0000","last_clean":"2024-03-04T06:14:05.273084+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:14:05.273084+0000","last_undegraded":"2024-03-04T06:14:05.273084+0000","last_fullsized":"2024-03-04T06:14:05.273084+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"372'197","reported_seq":1230,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.266653+0000","last_change":"2024-03-04T06:14:06.272942+0000","last_active":"2024-03-04T06:14:06.266653+0000","last_peered":"2024-03-04T06:14:06.266653+0000","last_clean":"2024-03-04T06:14:06.266653+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:14:06.266653+0000","last_undegraded":"2024-03-04T06:14:06.266653+0000","last_fullsized":"2024-03-04T06:14:06.266653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3848935,"num_objects":21,"num_object_clones":14,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":140,"num_read_kb":133003,"num_write":90,"num_write_kb":23972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1052,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.249754+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:14:05.249754+0000","last_peered":"2024-03-04T06:14:05.249754+0000","last_clean":"2024-03-04T06:14:05.249754+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:14:05.249754+0000","last_undegraded":"2024-03-04T06:14:05.249754+0000","last_fullsized":"2024-03-04T06:14:05.249754+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":918,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.282829+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:14:06.282829+0000","last_peered":"2024-03-04T06:14:06.282829+0000","last_clean":"2024-03-04T06:14:06.282829+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:14:06.282829+0000","last_undegraded":"2024-03-04T06:14:06.282829+0000","last_fullsized":"2024-03-04T06:14:06.282829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"372'330","reported_seq":1302,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.278810+0000","last_change":"2024-03-04T06:14:06.283573+0000","last_active":"2024-03-04T06:14:06.278810+0000","last_peered":"2024-03-04T06:14:06.278810+0000","last_clean":"2024-03-04T06:14:06.278810+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:14:06.278810+0000","last_undegraded":"2024-03-04T06:14:06.278810+0000","last_fullsized":"2024-03-04T06:14:06.278810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9076209,"num_objects":34,"num_object_clones":25,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":177,"num_read_kb":134382,"num_write":156,"num_write_kb":41350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":921,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.272930+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:14:05.272930+0000","last_peered":"2024-03-04T06:14:05.272930+0000","last_clean":"2024-03-04T06:14:05.272930+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:14:05.272930+0000","last_undegraded":"2024-03-04T06:14:05.272930+0000","last_fullsized":"2024-03-04T06:14:05.272930+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"373'256","reported_seq":1236,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.250528+0000","last_change":"2024-03-04T06:14:03.308203+0000","last_active":"2024-03-04T06:14:05.250528+0000","last_peered":"2024-03-04T06:14:05.250528+0000","last_clean":"2024-03-04T06:14:05.250528+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:14:05.250528+0000","last_undegraded":"2024-03-04T06:14:05.250528+0000","last_fullsized":"2024-03-04T06:14:05.250528+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647449,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":130,"num_read_kb":106966,"num_write":105,"num_write_kb":30304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":918,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.269009+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:14:06.269009+0000","last_peered":"2024-03-04T06:14:06.269009+0000","last_clean":"2024-03-04T06:14:06.269009+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:14:06.269009+0000","last_undegraded":"2024-03-04T06:14:06.269009+0000","last_fullsized":"2024-03-04T06:14:06.269009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"373'605","reported_seq":1526,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.255489+0000","last_change":"2024-03-04T06:14:03.319004+0000","last_active":"2024-03-04T06:14:05.255489+0000","last_peered":"2024-03-04T06:14:05.255489+0000","last_clean":"2024-03-04T06:14:05.255489+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:14:05.255489+0000","last_undegraded":"2024-03-04T06:14:05.255489+0000","last_fullsized":"2024-03-04T06:14:05.255489+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11698431,"num_objects":63,"num_object_clones":46,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":370,"num_read_kb":296326,"num_write":307,"num_write_kb":81472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":921,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.264987+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:14:05.264987+0000","last_peered":"2024-03-04T06:14:05.264987+0000","last_clean":"2024-03-04T06:14:05.264987+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:14:05.264987+0000","last_undegraded":"2024-03-04T06:14:05.264987+0000","last_fullsized":"2024-03-04T06:14:05.264987+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"373'594","reported_seq":1467,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.387833+0000","last_change":"2024-03-04T06:14:03.375985+0000","last_active":"2024-03-04T06:14:04.387833+0000","last_peered":"2024-03-04T06:14:04.387833+0000","last_clean":"2024-03-04T06:14:04.387833+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:14:04.387833+0000","last_undegraded":"2024-03-04T06:14:04.387833+0000","last_fullsized":"2024-03-04T06:14:04.387833+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12253062,"num_objects":59,"num_object_clones":43,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":242,"num_read_kb":166788,"num_write":280,"num_write_kb":72697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"373'250","reported_seq":1301,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.280163+0000","last_change":"2024-03-04T06:14:06.289537+0000","last_active":"2024-03-04T06:14:06.280163+0000","last_peered":"2024-03-04T06:14:06.280163+0000","last_clean":"2024-03-04T06:14:06.280163+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:14:06.280163+0000","last_undegraded":"2024-03-04T06:14:06.280163+0000","last_fullsized":"2024-03-04T06:14:06.280163+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168466,"num_objects":28,"num_object_clones":20,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":219,"num_read_kb":176022,"num_write":129,"num_write_kb":34239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":948,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.395989+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:14:04.395989+0000","last_peered":"2024-03-04T06:14:04.395989+0000","last_clean":"2024-03-04T06:14:04.395989+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:14:04.395989+0000","last_undegraded":"2024-03-04T06:14:04.395989+0000","last_fullsized":"2024-03-04T06:14:04.395989+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"373'287","reported_seq":1308,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.395270+0000","last_change":"2024-03-04T06:14:03.345324+0000","last_active":"2024-03-04T06:14:04.395270+0000","last_peered":"2024-03-04T06:14:04.395270+0000","last_clean":"2024-03-04T06:14:04.395270+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:14:04.395270+0000","last_undegraded":"2024-03-04T06:14:04.395270+0000","last_fullsized":"2024-03-04T06:14:04.395270+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9277893,"num_objects":34,"num_object_clones":22,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":155,"num_read_kb":98383,"num_write":142,"num_write_kb":38074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":919,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.258124+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:14:05.258124+0000","last_peered":"2024-03-04T06:14:05.258124+0000","last_clean":"2024-03-04T06:14:05.258124+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:14:05.258124+0000","last_undegraded":"2024-03-04T06:14:05.258124+0000","last_fullsized":"2024-03-04T06:14:05.258124+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"374'395","reported_seq":1372,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.252004+0000","last_change":"2024-03-04T06:14:03.281948+0000","last_active":"2024-03-04T06:14:05.252004+0000","last_peered":"2024-03-04T06:14:05.252004+0000","last_clean":"2024-03-04T06:14:05.252004+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:14:05.252004+0000","last_undegraded":"2024-03-04T06:14:05.252004+0000","last_fullsized":"2024-03-04T06:14:05.252004+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12639425,"num_objects":42,"num_object_clones":31,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":233,"num_read_kb":174218,"num_write":206,"num_write_kb":57452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":948,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.386313+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:14:04.386313+0000","last_peered":"2024-03-04T06:14:04.386313+0000","last_clean":"2024-03-04T06:14:04.386313+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:14:04.386313+0000","last_undegraded":"2024-03-04T06:14:04.386313+0000","last_fullsized":"2024-03-04T06:14:04.386313+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"373'514","reported_seq":1464,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.382850+0000","last_change":"2024-03-04T06:14:03.338468+0000","last_active":"2024-03-04T06:14:04.382850+0000","last_peered":"2024-03-04T06:14:04.382850+0000","last_clean":"2024-03-04T06:14:04.382850+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:14:04.382850+0000","last_undegraded":"2024-03-04T06:14:04.382850+0000","last_fullsized":"2024-03-04T06:14:04.382850+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9849298,"num_objects":46,"num_object_clones":40,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":324,"num_read_kb":270587,"num_write":227,"num_write_kb":62906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":918,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.279866+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:14:06.279866+0000","last_peered":"2024-03-04T06:14:06.279866+0000","last_clean":"2024-03-04T06:14:06.279866+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:14:06.279866+0000","last_undegraded":"2024-03-04T06:14:06.279866+0000","last_fullsized":"2024-03-04T06:14:06.279866+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":123201514,"num_objects":527,"num_object_clones":390,"num_object_copies":1581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":94,"num_read":2793,"num_read_kb":2139771,"num_write":2454,"num_write_kb":668276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4957,"ondisk_log_size":4957,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":135544,"kb_used_data":96240,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94236296,"statfs":{"total":96636764160,"available":96497967104,"internally_reserved":0,"allocated":98549760,"data_stored":108139543,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105652,"kb_used_data":70508,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94266188,"statfs":{"total":96636764160,"available":96528576512,"internally_reserved":0,"allocated":72200192,"data_stored":77598442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":103760,"kb_used_data":68616,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94268080,"statfs":{"total":96636764160,"available":96530513920,"internally_reserved":0,"allocated":70262784,"data_stored":76852786,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":109008,"kb_used_data":73864,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94262832,"statfs":{"total":96636764160,"available":96525139968,"internally_reserved":0,"allocated":75636736,"data_stored":82317573,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:14:09.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:14:09.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:09.734+0000 7fe659624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:09.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:09.740+0000 7fe659624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:09.939 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:14:09.939 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:14:09.954 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":562,"stamp":"2024-03-04T06:14:09.245090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":123791882,"num_objects":529,"num_object_clones":390,"num_object_copies":1587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":94,"num_read":2896,"num_read_kb":2139862,"num_write":2591,"num_write_kb":670170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5054,"ondisk_log_size":5054,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":453964,"kb_used_data":309228,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":377033396,"statfs":{"total":386547056640,"available":386082197504,"internally_reserved":0,"allocated":316649472,"data_stored":344908344,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4521644,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":27930,"num_write":28,"num_write_kb":6592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010416"},"pg_stats":[{"pgid":"3.f","version":"375'271","reported_seq":1292,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.308881+0000","last_change":"2024-03-04T06:14:03.291095+0000","last_active":"2024-03-04T06:14:05.308881+0000","last_peered":"2024-03-04T06:14:05.308881+0000","last_clean":"2024-03-04T06:14:05.308881+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:14:05.308881+0000","last_undegraded":"2024-03-04T06:14:05.308881+0000","last_fullsized":"2024-03-04T06:14:05.308881+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8118210,"num_objects":37,"num_object_clones":27,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":152,"num_read_kb":107848,"num_write":155,"num_write_kb":42675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"371'215","reported_seq":1240,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.383517+0000","last_change":"2024-03-04T06:14:03.300077+0000","last_active":"2024-03-04T06:14:04.383517+0000","last_peered":"2024-03-04T06:14:04.383517+0000","last_clean":"2024-03-04T06:14:04.383517+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:14:04.383517+0000","last_undegraded":"2024-03-04T06:14:04.383517+0000","last_fullsized":"2024-03-04T06:14:04.383517+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067624,"num_objects":25,"num_object_clones":17,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":88,"num_read_kb":57069,"num_write":99,"num_write_kb":25882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"376'342","reported_seq":1334,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.340398+0000","last_change":"2024-03-04T06:14:06.273503+0000","last_active":"2024-03-04T06:14:06.340398+0000","last_peered":"2024-03-04T06:14:06.340398+0000","last_clean":"2024-03-04T06:14:06.340398+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:14:06.340398+0000","last_undegraded":"2024-03-04T06:14:06.340398+0000","last_fullsized":"2024-03-04T06:14:06.340398+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7244213,"num_objects":41,"num_object_clones":31,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":201,"num_read_kb":146762,"num_write":182,"num_write_kb":49923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"373'267","reported_seq":1294,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.265829+0000","last_change":"2024-03-04T06:14:03.396095+0000","last_active":"2024-03-04T06:14:05.265829+0000","last_peered":"2024-03-04T06:14:05.265829+0000","last_clean":"2024-03-04T06:14:05.265829+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:14:05.265829+0000","last_undegraded":"2024-03-04T06:14:05.265829+0000","last_fullsized":"2024-03-04T06:14:05.265829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9513191,"num_objects":28,"num_object_clones":20,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":161,"num_read_kb":130099,"num_write":149,"num_write_kb":41924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"373'130","reported_seq":1155,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.258514+0000","last_change":"2024-03-04T06:14:03.395475+0000","last_active":"2024-03-04T06:14:05.258514+0000","last_peered":"2024-03-04T06:14:05.258514+0000","last_clean":"2024-03-04T06:14:05.258514+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:14:05.258514+0000","last_undegraded":"2024-03-04T06:14:05.258514+0000","last_fullsized":"2024-03-04T06:14:05.258514+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336338,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":17047,"num_write":60,"num_write_kb":15555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"373'304","reported_seq":1306,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.272439+0000","last_change":"2024-03-04T06:14:03.317783+0000","last_active":"2024-03-04T06:14:05.272439+0000","last_peered":"2024-03-04T06:14:05.272439+0000","last_clean":"2024-03-04T06:14:05.272439+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:14:05.272439+0000","last_undegraded":"2024-03-04T06:14:05.272439+0000","last_fullsized":"2024-03-04T06:14:05.272439+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9462770,"num_objects":32,"num_object_clones":24,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":160,"num_read_kb":124271,"num_write":167,"num_write_kb":49851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1087,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.273084+0000","last_change":"2024-03-04T06:14:03.289424+0000","last_active":"2024-03-04T06:14:05.273084+0000","last_peered":"2024-03-04T06:14:05.273084+0000","last_clean":"2024-03-04T06:14:05.273084+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:14:05.273084+0000","last_undegraded":"2024-03-04T06:14:05.273084+0000","last_fullsized":"2024-03-04T06:14:05.273084+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"372'197","reported_seq":1230,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.266653+0000","last_change":"2024-03-04T06:14:06.272942+0000","last_active":"2024-03-04T06:14:06.266653+0000","last_peered":"2024-03-04T06:14:06.266653+0000","last_clean":"2024-03-04T06:14:06.266653+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:14:06.266653+0000","last_undegraded":"2024-03-04T06:14:06.266653+0000","last_fullsized":"2024-03-04T06:14:06.266653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3848935,"num_objects":21,"num_object_clones":14,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":140,"num_read_kb":133003,"num_write":90,"num_write_kb":23972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1052,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.249754+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:14:05.249754+0000","last_peered":"2024-03-04T06:14:05.249754+0000","last_clean":"2024-03-04T06:14:05.249754+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:14:05.249754+0000","last_undegraded":"2024-03-04T06:14:05.249754+0000","last_fullsized":"2024-03-04T06:14:05.249754+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":918,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.282829+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:14:06.282829+0000","last_peered":"2024-03-04T06:14:06.282829+0000","last_clean":"2024-03-04T06:14:06.282829+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:14:06.282829+0000","last_undegraded":"2024-03-04T06:14:06.282829+0000","last_fullsized":"2024-03-04T06:14:06.282829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"372'330","reported_seq":1302,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.278810+0000","last_change":"2024-03-04T06:14:06.283573+0000","last_active":"2024-03-04T06:14:06.278810+0000","last_peered":"2024-03-04T06:14:06.278810+0000","last_clean":"2024-03-04T06:14:06.278810+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:14:06.278810+0000","last_undegraded":"2024-03-04T06:14:06.278810+0000","last_fullsized":"2024-03-04T06:14:06.278810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9076209,"num_objects":34,"num_object_clones":25,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":177,"num_read_kb":134382,"num_write":156,"num_write_kb":41350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":921,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.272930+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:14:05.272930+0000","last_peered":"2024-03-04T06:14:05.272930+0000","last_clean":"2024-03-04T06:14:05.272930+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:14:05.272930+0000","last_undegraded":"2024-03-04T06:14:05.272930+0000","last_fullsized":"2024-03-04T06:14:05.272930+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"373'256","reported_seq":1236,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.250528+0000","last_change":"2024-03-04T06:14:03.308203+0000","last_active":"2024-03-04T06:14:05.250528+0000","last_peered":"2024-03-04T06:14:05.250528+0000","last_clean":"2024-03-04T06:14:05.250528+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:14:05.250528+0000","last_undegraded":"2024-03-04T06:14:05.250528+0000","last_fullsized":"2024-03-04T06:14:05.250528+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647449,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":130,"num_read_kb":106966,"num_write":105,"num_write_kb":30304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":918,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.269009+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:14:06.269009+0000","last_peered":"2024-03-04T06:14:06.269009+0000","last_clean":"2024-03-04T06:14:06.269009+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:14:06.269009+0000","last_undegraded":"2024-03-04T06:14:06.269009+0000","last_fullsized":"2024-03-04T06:14:06.269009+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"373'605","reported_seq":1526,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.255489+0000","last_change":"2024-03-04T06:14:03.319004+0000","last_active":"2024-03-04T06:14:05.255489+0000","last_peered":"2024-03-04T06:14:05.255489+0000","last_clean":"2024-03-04T06:14:05.255489+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:14:05.255489+0000","last_undegraded":"2024-03-04T06:14:05.255489+0000","last_fullsized":"2024-03-04T06:14:05.255489+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11698431,"num_objects":63,"num_object_clones":46,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":370,"num_read_kb":296326,"num_write":307,"num_write_kb":81472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":921,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.264987+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:14:05.264987+0000","last_peered":"2024-03-04T06:14:05.264987+0000","last_clean":"2024-03-04T06:14:05.264987+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:14:05.264987+0000","last_undegraded":"2024-03-04T06:14:05.264987+0000","last_fullsized":"2024-03-04T06:14:05.264987+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"373'594","reported_seq":1467,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.387833+0000","last_change":"2024-03-04T06:14:03.375985+0000","last_active":"2024-03-04T06:14:04.387833+0000","last_peered":"2024-03-04T06:14:04.387833+0000","last_clean":"2024-03-04T06:14:04.387833+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:14:04.387833+0000","last_undegraded":"2024-03-04T06:14:04.387833+0000","last_fullsized":"2024-03-04T06:14:04.387833+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12253062,"num_objects":59,"num_object_clones":43,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":242,"num_read_kb":166788,"num_write":280,"num_write_kb":72697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"373'250","reported_seq":1301,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.280163+0000","last_change":"2024-03-04T06:14:06.289537+0000","last_active":"2024-03-04T06:14:06.280163+0000","last_peered":"2024-03-04T06:14:06.280163+0000","last_clean":"2024-03-04T06:14:06.280163+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:14:06.280163+0000","last_undegraded":"2024-03-04T06:14:06.280163+0000","last_fullsized":"2024-03-04T06:14:06.280163+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168466,"num_objects":28,"num_object_clones":20,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":219,"num_read_kb":176022,"num_write":129,"num_write_kb":34239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":948,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.395989+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:14:04.395989+0000","last_peered":"2024-03-04T06:14:04.395989+0000","last_clean":"2024-03-04T06:14:04.395989+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:14:04.395989+0000","last_undegraded":"2024-03-04T06:14:04.395989+0000","last_fullsized":"2024-03-04T06:14:04.395989+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"373'287","reported_seq":1308,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.395270+0000","last_change":"2024-03-04T06:14:03.345324+0000","last_active":"2024-03-04T06:14:04.395270+0000","last_peered":"2024-03-04T06:14:04.395270+0000","last_clean":"2024-03-04T06:14:04.395270+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:14:04.395270+0000","last_undegraded":"2024-03-04T06:14:04.395270+0000","last_fullsized":"2024-03-04T06:14:04.395270+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9277893,"num_objects":34,"num_object_clones":22,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":155,"num_read_kb":98383,"num_write":142,"num_write_kb":38074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":919,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.258124+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:14:05.258124+0000","last_peered":"2024-03-04T06:14:05.258124+0000","last_clean":"2024-03-04T06:14:05.258124+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:14:05.258124+0000","last_undegraded":"2024-03-04T06:14:05.258124+0000","last_fullsized":"2024-03-04T06:14:05.258124+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"374'395","reported_seq":1372,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-03-04T06:14:05.252004+0000","last_change":"2024-03-04T06:14:03.281948+0000","last_active":"2024-03-04T06:14:05.252004+0000","last_peered":"2024-03-04T06:14:05.252004+0000","last_clean":"2024-03-04T06:14:05.252004+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:14:05.252004+0000","last_undegraded":"2024-03-04T06:14:05.252004+0000","last_fullsized":"2024-03-04T06:14:05.252004+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12639425,"num_objects":42,"num_object_clones":31,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":233,"num_read_kb":174218,"num_write":206,"num_write_kb":57452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":948,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.386313+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:14:04.386313+0000","last_peered":"2024-03-04T06:14:04.386313+0000","last_clean":"2024-03-04T06:14:04.386313+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:14:04.386313+0000","last_undegraded":"2024-03-04T06:14:04.386313+0000","last_fullsized":"2024-03-04T06:14:04.386313+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"373'514","reported_seq":1464,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-03-04T06:14:04.382850+0000","last_change":"2024-03-04T06:14:03.338468+0000","last_active":"2024-03-04T06:14:04.382850+0000","last_peered":"2024-03-04T06:14:04.382850+0000","last_clean":"2024-03-04T06:14:04.382850+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:14:04.382850+0000","last_undegraded":"2024-03-04T06:14:04.382850+0000","last_fullsized":"2024-03-04T06:14:04.382850+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9849298,"num_objects":46,"num_object_clones":40,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":324,"num_read_kb":270587,"num_write":227,"num_write_kb":62906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":918,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-03-04T06:14:06.279866+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:14:06.279866+0000","last_peered":"2024-03-04T06:14:06.279866+0000","last_clean":"2024-03-04T06:14:06.279866+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:14:06.279866+0000","last_undegraded":"2024-03-04T06:14:06.279866+0000","last_fullsized":"2024-03-04T06:14:06.279866+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":123201514,"num_objects":527,"num_object_clones":390,"num_object_copies":1581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":94,"num_read":2793,"num_read_kb":2139771,"num_write":2454,"num_write_kb":668276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4957,"ondisk_log_size":4957,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":135544,"kb_used_data":96240,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94236296,"statfs":{"total":96636764160,"available":96497967104,"internally_reserved":0,"allocated":98549760,"data_stored":108139543,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105652,"kb_used_data":70508,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94266188,"statfs":{"total":96636764160,"available":96528576512,"internally_reserved":0,"allocated":72200192,"data_stored":77598442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":103760,"kb_used_data":68616,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94268080,"statfs":{"total":96636764160,"available":96530513920,"internally_reserved":0,"allocated":70262784,"data_stored":76852786,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":109008,"kb_used_data":73864,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94262832,"statfs":{"total":96636764160,"available":96525139968,"internally_reserved":0,"allocated":75636736,"data_stored":82317573,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:14:09.955 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:14:09.955 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:14:09.955 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:14:09.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:14:10.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:10.060+0000 7f53ee26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:10.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:10.067+0000 7f53ee26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:10.079 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:10.080 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:10.080 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:10.080 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:10.080 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:10.080 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:10.080 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:10.080 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:10.080 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:10.080 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:10.080 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:10.080 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:10.082 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:10.083 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:10.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:14:10.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:10.186+0000 7f8be9b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:10.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:10.194+0000 7f8be9b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:10.206 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:10.207 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:10.209 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:10.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:14:10.263 INFO:tasks.rados.rados.0.smithi064.stdout:1557: done (2 left) 2024-03-04T06:14:10.263 INFO:tasks.rados.rados.0.smithi064.stdout:1558: snap_remove snap 181 2024-03-04T06:14:10.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:10.318+0000 7fba5739d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:10.326 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:10.325+0000 7fba5739d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:10.343 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:10.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:14:10.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:10.446+0000 7fb63f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:10.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:10.453+0000 7fb63f66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:10.466 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:10.467 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:10.467 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:10.467 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:10.467 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:10.467 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:10.467 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:10.467 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:10.467 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:10.467 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:10.467 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:10.467 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:10.470 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:10.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:10.820 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:10.818+0000 7f1a7ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:10.823 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:10.823+0000 7f1a7ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:11.269 INFO:tasks.rados.rados.0.smithi064.stdout:1558: done (2 left) 2024-03-04T06:14:11.269 INFO:tasks.rados.rados.0.smithi064.stdout:1559: delete oid 32 current snap is 185 2024-03-04T06:14:11.269 INFO:tasks.rados.rados.0.smithi064.stdout:1552: expect (ObjNum 615 snap 147 seq_num 615) 2024-03-04T06:14:11.451 INFO:tasks.rados.rados.0.smithi064.stdout:1553: finishing write tid 1 to smithi06431743-3 2024-03-04T06:14:11.451 INFO:tasks.rados.rados.0.smithi064.stdout:1553: finishing write tid 2 to smithi06431743-3 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:1553: finishing write tid 3 to smithi06431743-3 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:1553: finishing write tid 4 to smithi06431743-3 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:1553: oid 3 updating version 0 to 334 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:1553: oid 3 version 334 is already newer than 334 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:1553: oid 3 version 334 is already newer than 332 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 334 (ObjNum 743 snap 185 seq_num 743) dirty exists 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:1553: left oid 3 (ObjNum 743 snap 185 seq_num 743) 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:1552: done (2 left) 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:1553: done (1 left) 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:1559: done (0 left) 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:1560: write oid 47 current snap is 185 2024-03-04T06:14:11.452 INFO:tasks.rados.rados.0.smithi064.stdout:1560: seq_num 746 ranges {538022=738348,1737567=561594} 2024-03-04T06:14:11.458 INFO:tasks.rados.rados.0.smithi064.stdout:1560: writing smithi06431743-47 from 538022 to 1276370 tid 1 2024-03-04T06:14:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:1560: writing smithi06431743-47 from 1737567 to 2299161 tid 2 2024-03-04T06:14:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:1561: read oid 20 snap -1 2024-03-04T06:14:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:1561: expect (ObjNum 695 snap 170 seq_num 695) 2024-03-04T06:14:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:1562: snap_create 2024-03-04T06:14:11.491 INFO:tasks.rados.rados.0.smithi064.stdout:1561: expect (ObjNum 695 snap 170 seq_num 695) 2024-03-04T06:14:11.753 INFO:tasks.rados.rados.0.smithi064.stdout:1560: finishing write tid 1 to smithi06431743-47 2024-03-04T06:14:11.753 INFO:tasks.rados.rados.0.smithi064.stdout:1560: finishing write tid 2 to smithi06431743-47 2024-03-04T06:14:11.753 INFO:tasks.rados.rados.0.smithi064.stdout:1560: finishing write tid 3 to smithi06431743-47 2024-03-04T06:14:11.754 INFO:tasks.rados.rados.0.smithi064.stdout:1560: finishing write tid 4 to smithi06431743-47 2024-03-04T06:14:11.754 INFO:tasks.rados.rados.0.smithi064.stdout:1560: oid 47 updating version 0 to 271 2024-03-04T06:14:11.754 INFO:tasks.rados.rados.0.smithi064.stdout:1560: oid 47 updating version 271 to 273 2024-03-04T06:14:11.754 INFO:tasks.rados.rados.0.smithi064.stdout:1560: oid 47 version 273 is already newer than 272 2024-03-04T06:14:11.754 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 273 (ObjNum 746 snap 185 seq_num 746) dirty exists 2024-03-04T06:14:11.754 INFO:tasks.rados.rados.0.smithi064.stdout:1560: left oid 47 (ObjNum 746 snap 185 seq_num 746) 2024-03-04T06:14:12.272 INFO:tasks.rados.rados.0.smithi064.stdout:1560: done (2 left) 2024-03-04T06:14:12.272 INFO:tasks.rados.rados.0.smithi064.stdout:1561: done (1 left) 2024-03-04T06:14:12.272 INFO:tasks.rados.rados.0.smithi064.stdout:1562: done (0 left) 2024-03-04T06:14:12.272 INFO:tasks.rados.rados.0.smithi064.stdout:1563: write oid 6 current snap is 186 2024-03-04T06:14:12.272 INFO:tasks.rados.rados.0.smithi064.stdout:1563: seq_num 747 ranges {554829=413597,1396049=705252} 2024-03-04T06:14:12.279 INFO:tasks.rados.rados.0.smithi064.stdout:1563: writing smithi06431743-6 from 554829 to 968426 tid 1 2024-03-04T06:14:12.287 INFO:tasks.rados.rados.0.smithi064.stdout:1563: writing smithi06431743-6 from 1396049 to 2101301 tid 2 2024-03-04T06:14:12.287 INFO:tasks.rados.rados.0.smithi064.stdout:1564: read oid 20 snap -1 2024-03-04T06:14:12.287 INFO:tasks.rados.rados.0.smithi064.stdout:1564: expect (ObjNum 695 snap 170 seq_num 695) 2024-03-04T06:14:12.287 INFO:tasks.rados.rados.0.smithi064.stdout:1565: delete oid 45 current snap is 186 2024-03-04T06:14:12.322 INFO:tasks.rados.rados.0.smithi064.stdout:1565: done (2 left) 2024-03-04T06:14:12.322 INFO:tasks.rados.rados.0.smithi064.stdout:1566: write (excl) oid 43 current snap is 186 2024-03-04T06:14:12.322 INFO:tasks.rados.rados.0.smithi064.stdout:1566: seq_num 749 ranges {588443=564095,1796660=708921} 2024-03-04T06:14:12.330 INFO:tasks.rados.rados.0.smithi064.stdout:1566: writing smithi06431743-43 from 588443 to 1152538 tid 1 2024-03-04T06:14:12.340 INFO:tasks.rados.rados.0.smithi064.stdout:1566: writing smithi06431743-43 from 1796660 to 2505581 tid 2 2024-03-04T06:14:12.340 INFO:tasks.rados.rados.0.smithi064.stdout:1567: snap_remove snap 182 2024-03-04T06:14:12.894 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:13.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:12.998+0000 7f4431f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:13.003 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:13.003+0000 7f4431f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:13.234 INFO:tasks.rados.rados.0.smithi064.stdout:1567: done (3 left) 2024-03-04T06:14:13.234 INFO:tasks.rados.rados.0.smithi064.stdout:1568: write oid 49 current snap is 186 2024-03-04T06:14:13.234 INFO:tasks.rados.rados.0.smithi064.stdout:1568: seq_num 750 ranges {605250=639344,1855142=452579} 2024-03-04T06:14:13.243 INFO:tasks.rados.rados.0.smithi064.stdout:1568: writing smithi06431743-49 from 605250 to 1244594 tid 1 2024-03-04T06:14:13.250 INFO:tasks.rados.rados.0.smithi064.stdout:1568: writing smithi06431743-49 from 1855142 to 2307721 tid 2 2024-03-04T06:14:13.250 INFO:tasks.rados.rados.0.smithi064.stdout:1569: write (excl) oid 11 current snap is 186 2024-03-04T06:14:13.251 INFO:tasks.rados.rados.0.smithi064.stdout:1563: finishing write tid 1 to smithi06431743-6 2024-03-04T06:14:13.251 INFO:tasks.rados.rados.0.smithi064.stdout:1564: expect (ObjNum 695 snap 170 seq_num 695) 2024-03-04T06:14:13.522 INFO:tasks.rados.rados.0.smithi064.stdout:1563: finishing write tid 2 to smithi06431743-6 2024-03-04T06:14:13.522 INFO:tasks.rados.rados.0.smithi064.stdout:1563: finishing write tid 3 to smithi06431743-6 2024-03-04T06:14:13.522 INFO:tasks.rados.rados.0.smithi064.stdout:1563: finishing write tid 4 to smithi06431743-6 2024-03-04T06:14:13.522 INFO:tasks.rados.rados.0.smithi064.stdout:1563: oid 6 updating version 0 to 253 2024-03-04T06:14:13.523 INFO:tasks.rados.rados.0.smithi064.stdout:1563: oid 6 version 253 is already newer than 252 2024-03-04T06:14:13.523 INFO:tasks.rados.rados.0.smithi064.stdout:1563: oid 6 version 253 is already newer than 251 2024-03-04T06:14:13.523 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 253 (ObjNum 747 snap 186 seq_num 747) dirty exists 2024-03-04T06:14:13.523 INFO:tasks.rados.rados.0.smithi064.stdout:1563: left oid 6 (ObjNum 747 snap 186 seq_num 747) 2024-03-04T06:14:13.523 INFO:tasks.rados.rados.0.smithi064.stdout:1566: finishing write tid 1 to smithi06431743-43 2024-03-04T06:14:13.523 INFO:tasks.rados.rados.0.smithi064.stdout:1569: seq_num 751 ranges {622057=714593,1913624=708433} 2024-03-04T06:14:13.529 INFO:tasks.rados.rados.0.smithi064.stdout:1569: writing smithi06431743-11 from 622057 to 1336650 tid 1 2024-03-04T06:14:13.537 INFO:tasks.rados.rados.0.smithi064.stdout:1569: writing smithi06431743-11 from 1913624 to 2622057 tid 2 2024-03-04T06:14:13.537 INFO:tasks.rados.rados.0.smithi064.stdout:1563: done (4 left) 2024-03-04T06:14:13.537 INFO:tasks.rados.rados.0.smithi064.stdout:1564: done (3 left) 2024-03-04T06:14:13.537 INFO:tasks.rados.rados.0.smithi064.stdout:1570: snap_remove snap 180 2024-03-04T06:14:14.279 INFO:tasks.rados.rados.0.smithi064.stdout:1570: done (3 left) 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:1571: read oid 34 snap -1 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:1571: expect (ObjNum 723 snap 179 seq_num 723) 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:1572: write (excl) oid 37 current snap is 186 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:1566: finishing write tid 2 to smithi06431743-43 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:1566: finishing write tid 3 to smithi06431743-43 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:1566: finishing write tid 4 to smithi06431743-43 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:1566: oid 43 updating version 0 to 279 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:1566: oid 43 updating version 279 to 281 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:1566: oid 43 version 281 is already newer than 280 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 281 (ObjNum 749 snap 186 seq_num 749) dirty exists 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:1566: left oid 43 (ObjNum 749 snap 186 seq_num 749) 2024-03-04T06:14:14.280 INFO:tasks.rados.rados.0.smithi064.stdout:1572: seq_num 752 ranges {638864=789842,1972106=456248} 2024-03-04T06:14:14.289 INFO:tasks.rados.rados.0.smithi064.stdout:1572: writing smithi06431743-37 from 638864 to 1428706 tid 1 2024-03-04T06:14:14.295 INFO:tasks.rados.rados.0.smithi064.stdout:1572: writing smithi06431743-37 from 1972106 to 2428354 tid 2 2024-03-04T06:14:14.295 INFO:tasks.rados.rados.0.smithi064.stdout:1566: done (4 left) 2024-03-04T06:14:14.296 INFO:tasks.rados.rados.0.smithi064.stdout:1573: snap_remove snap 185 2024-03-04T06:14:15.071 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:15.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:15.187+0000 7f9313e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:15.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:15.191+0000 7f9313e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:15.281 INFO:tasks.rados.rados.0.smithi064.stdout:1573: done (4 left) 2024-03-04T06:14:15.281 INFO:tasks.rados.rados.0.smithi064.stdout:1574: snap_create 2024-03-04T06:14:15.281 INFO:tasks.rados.rados.0.smithi064.stdout:1568: finishing write tid 1 to smithi06431743-49 2024-03-04T06:14:15.281 INFO:tasks.rados.rados.0.smithi064.stdout:1568: finishing write tid 2 to smithi06431743-49 2024-03-04T06:14:15.281 INFO:tasks.rados.rados.0.smithi064.stdout:1568: finishing write tid 3 to smithi06431743-49 2024-03-04T06:14:15.281 INFO:tasks.rados.rados.0.smithi064.stdout:1568: finishing write tid 4 to smithi06431743-49 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:1568: oid 49 updating version 0 to 201 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:1568: oid 49 updating version 201 to 202 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:1568: oid 49 version 202 is already newer than 200 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 202 (ObjNum 750 snap 186 seq_num 750) dirty exists 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:1568: left oid 49 (ObjNum 750 snap 186 seq_num 750) 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:1569: finishing write tid 1 to smithi06431743-11 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:1569: finishing write tid 2 to smithi06431743-11 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:1569: finishing write tid 3 to smithi06431743-11 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:1569: finishing write tid 4 to smithi06431743-11 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:1569: oid 11 updating version 0 to 344 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:1569: oid 11 updating version 344 to 346 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:1569: oid 11 version 346 is already newer than 345 2024-03-04T06:14:15.282 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 346 (ObjNum 751 snap 186 seq_num 751) dirty exists 2024-03-04T06:14:15.283 INFO:tasks.rados.rados.0.smithi064.stdout:1569: left oid 11 (ObjNum 751 snap 186 seq_num 751) 2024-03-04T06:14:15.283 INFO:tasks.rados.rados.0.smithi064.stdout:1571: expect (ObjNum 723 snap 179 seq_num 723) 2024-03-04T06:14:15.362 INFO:tasks.rados.rados.0.smithi064.stdout:1572: finishing write tid 1 to smithi06431743-37 2024-03-04T06:14:15.362 INFO:tasks.rados.rados.0.smithi064.stdout:1572: finishing write tid 2 to smithi06431743-37 2024-03-04T06:14:15.362 INFO:tasks.rados.rados.0.smithi064.stdout:1572: finishing write tid 3 to smithi06431743-37 2024-03-04T06:14:15.362 INFO:tasks.rados.rados.0.smithi064.stdout:1572: finishing write tid 4 to smithi06431743-37 2024-03-04T06:14:15.362 INFO:tasks.rados.rados.0.smithi064.stdout:1572: oid 37 updating version 0 to 218 2024-03-04T06:14:15.362 INFO:tasks.rados.rados.0.smithi064.stdout:1572: oid 37 updating version 218 to 219 2024-03-04T06:14:15.362 INFO:tasks.rados.rados.0.smithi064.stdout:1572: oid 37 version 219 is already newer than 217 2024-03-04T06:14:15.362 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 219 (ObjNum 752 snap 186 seq_num 752) dirty exists 2024-03-04T06:14:15.362 INFO:tasks.rados.rados.0.smithi064.stdout:1572: left oid 37 (ObjNum 752 snap 186 seq_num 752) 2024-03-04T06:14:15.470 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:14:15.470 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:14:15.470 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:14:15.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:14:15.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:15.574+0000 7efdcbedd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:15.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:15.581+0000 7efdcbedd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:15.594 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:15.597 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:15.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:14:15.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:15.700+0000 7f39f051d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:15.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:15.708+0000 7f39f051d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:15.720 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:15.720 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:15.720 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:15.720 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:15.720 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:15.720 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:15.721 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:15.721 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:15.721 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:15.721 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:15.721 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:15.721 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:15.724 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:15.724 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:14:15.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:15.827+0000 7f22d235d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:15.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:15.834+0000 7f22d235d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:15.848 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:15.848 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:15.848 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:15.848 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:15.848 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:15.848 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:15.848 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:15.848 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:15.848 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:15.848 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:15.849 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:15.849 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:15.851 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:15.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:14:15.959 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:15.954+0000 7f63d706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:15.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:15.961+0000 7f63d706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:15.975 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:15.977 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:15.978 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:16.284 INFO:tasks.rados.rados.0.smithi064.stdout:1568: done (4 left) 2024-03-04T06:14:16.284 INFO:tasks.rados.rados.0.smithi064.stdout:1569: done (3 left) 2024-03-04T06:14:16.284 INFO:tasks.rados.rados.0.smithi064.stdout:1571: done (2 left) 2024-03-04T06:14:16.285 INFO:tasks.rados.rados.0.smithi064.stdout:1572: done (1 left) 2024-03-04T06:14:16.285 INFO:tasks.rados.rados.0.smithi064.stdout:1574: done (0 left) 2024-03-04T06:14:16.285 INFO:tasks.rados.rados.0.smithi064.stdout:1575: read oid 11 snap 183 2024-03-04T06:14:16.285 INFO:tasks.rados.rados.0.smithi064.stdout:1575: expect (ObjNum 487 snap 114 seq_num 487) 2024-03-04T06:14:16.285 INFO:tasks.rados.rados.0.smithi064.stdout:1576: snap_create 2024-03-04T06:14:16.340 INFO:tasks.rados.rados.0.smithi064.stdout:1575: expect (ObjNum 487 snap 114 seq_num 487) 2024-03-04T06:14:17.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:17.284 INFO:tasks.rados.rados.0.smithi064.stdout:1575: done (1 left) 2024-03-04T06:14:17.284 INFO:tasks.rados.rados.0.smithi064.stdout:1576: done (0 left) 2024-03-04T06:14:17.284 INFO:tasks.rados.rados.0.smithi064.stdout:1577: write oid 50 current snap is 188 2024-03-04T06:14:17.284 INFO:tasks.rados.rados.0.smithi064.stdout:1577: seq_num 753 ranges {655671=581444,2030588=625083} 2024-03-04T06:14:17.290 INFO:tasks.rados.rados.0.smithi064.stdout:1577: writing smithi06431743-50 from 655671 to 1237115 tid 1 2024-03-04T06:14:17.298 INFO:tasks.rados.rados.0.smithi064.stdout:1577: writing smithi06431743-50 from 2030588 to 2655671 tid 2 2024-03-04T06:14:17.298 INFO:tasks.rados.rados.0.smithi064.stdout:1578: delete oid 28 current snap is 188 2024-03-04T06:14:17.331 INFO:tasks.rados.rados.0.smithi064.stdout:1578: done (1 left) 2024-03-04T06:14:17.331 INFO:tasks.rados.rados.0.smithi064.stdout:1579: delete oid 27 current snap is 188 2024-03-04T06:14:17.385 INFO:tasks.rados.rados.0.smithi064.stdout:1577: finishing write tid 1 to smithi06431743-50 2024-03-04T06:14:17.389 INFO:tasks.rados.rados.0.smithi064.stdout:1579: done (1 left) 2024-03-04T06:14:17.389 INFO:tasks.rados.rados.0.smithi064.stdout:1580: delete oid 33 current snap is 188 2024-03-04T06:14:17.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:17.403+0000 7f220cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:17.406 INFO:tasks.rados.rados.0.smithi064.stdout:1577: finishing write tid 2 to smithi06431743-50 2024-03-04T06:14:17.409 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:17.408+0000 7f220cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:17.411 INFO:tasks.rados.rados.0.smithi064.stdout:1577: finishing write tid 3 to smithi06431743-50 2024-03-04T06:14:17.414 INFO:tasks.rados.rados.0.smithi064.stdout:1577: finishing write tid 4 to smithi06431743-50 2024-03-04T06:14:17.414 INFO:tasks.rados.rados.0.smithi064.stdout:1577: oid 50 updating version 0 to 520 2024-03-04T06:14:17.414 INFO:tasks.rados.rados.0.smithi064.stdout:1577: oid 50 version 520 is already newer than 518 2024-03-04T06:14:17.414 INFO:tasks.rados.rados.0.smithi064.stdout:1577: oid 50 version 520 is already newer than 519 2024-03-04T06:14:17.414 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 520 (ObjNum 753 snap 188 seq_num 753) dirty exists 2024-03-04T06:14:17.414 INFO:tasks.rados.rados.0.smithi064.stdout:1577: left oid 50 (ObjNum 753 snap 188 seq_num 753) 2024-03-04T06:14:17.428 INFO:tasks.rados.rados.0.smithi064.stdout:1577: done (1 left) 2024-03-04T06:14:17.428 INFO:tasks.rados.rados.0.smithi064.stdout:1580: done (0 left) 2024-03-04T06:14:17.428 INFO:tasks.rados.rados.0.smithi064.stdout:1581: read oid 36 snap -1 2024-03-04T06:14:17.429 INFO:tasks.rados.rados.0.smithi064.stdout:1581: expect (ObjNum 727 snap 179 seq_num 727) 2024-03-04T06:14:17.429 INFO:tasks.rados.rados.0.smithi064.stdout:1582: read oid 1 snap -1 2024-03-04T06:14:17.429 INFO:tasks.rados.rados.0.smithi064.stdout:1582: expect (ObjNum 741 snap 183 seq_num 741) 2024-03-04T06:14:17.429 INFO:tasks.rados.rados.0.smithi064.stdout:1583: write (excl) oid 46 current snap is 188 2024-03-04T06:14:17.429 INFO:tasks.rados.rados.0.smithi064.stdout:1583: seq_num 757 ranges {722899=482440,1748163=723597} 2024-03-04T06:14:17.435 INFO:tasks.rados.rados.0.smithi064.stdout:1583: writing smithi06431743-46 from 722899 to 1205339 tid 1 2024-03-04T06:14:17.445 INFO:tasks.rados.rados.0.smithi064.stdout:1583: writing smithi06431743-46 from 1748163 to 2471760 tid 2 2024-03-04T06:14:17.445 INFO:tasks.rados.rados.0.smithi064.stdout:1584: snap_create 2024-03-04T06:14:17.449 INFO:tasks.rados.rados.0.smithi064.stdout:1582: expect (ObjNum 741 snap 183 seq_num 741) 2024-03-04T06:14:17.544 INFO:tasks.rados.rados.0.smithi064.stdout:1581: expect (ObjNum 727 snap 179 seq_num 727) 2024-03-04T06:14:17.626 INFO:tasks.rados.rados.0.smithi064.stdout:1583: finishing write tid 1 to smithi06431743-46 2024-03-04T06:14:17.626 INFO:tasks.rados.rados.0.smithi064.stdout:1583: finishing write tid 2 to smithi06431743-46 2024-03-04T06:14:17.626 INFO:tasks.rados.rados.0.smithi064.stdout:1583: finishing write tid 3 to smithi06431743-46 2024-03-04T06:14:17.626 INFO:tasks.rados.rados.0.smithi064.stdout:1583: finishing write tid 4 to smithi06431743-46 2024-03-04T06:14:17.626 INFO:tasks.rados.rados.0.smithi064.stdout:1583: oid 46 updating version 0 to 622 2024-03-04T06:14:17.626 INFO:tasks.rados.rados.0.smithi064.stdout:1583: oid 46 updating version 622 to 624 2024-03-04T06:14:17.626 INFO:tasks.rados.rados.0.smithi064.stdout:1583: oid 46 version 624 is already newer than 623 2024-03-04T06:14:17.627 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 624 (ObjNum 757 snap 188 seq_num 757) dirty exists 2024-03-04T06:14:17.627 INFO:tasks.rados.rados.0.smithi064.stdout:1583: left oid 46 (ObjNum 757 snap 188 seq_num 757) 2024-03-04T06:14:18.303 INFO:tasks.rados.rados.0.smithi064.stdout:1581: done (3 left) 2024-03-04T06:14:18.303 INFO:tasks.rados.rados.0.smithi064.stdout:1582: done (2 left) 2024-03-04T06:14:18.303 INFO:tasks.rados.rados.0.smithi064.stdout:1583: done (1 left) 2024-03-04T06:14:18.303 INFO:tasks.rados.rados.0.smithi064.stdout:1584: done (0 left) 2024-03-04T06:14:18.303 INFO:tasks.rados.rados.0.smithi064.stdout:1585: write (excl) oid 18 current snap is 189 2024-03-04T06:14:18.303 INFO:tasks.rados.rados.0.smithi064.stdout:1585: seq_num 758 ranges {739706=557689,1806645=467255} 2024-03-04T06:14:18.309 INFO:tasks.rados.rados.0.smithi064.stdout:1585: writing smithi06431743-18 from 739706 to 1297395 tid 1 2024-03-04T06:14:18.314 INFO:tasks.rados.rados.0.smithi064.stdout:1585: writing smithi06431743-18 from 1806645 to 2273900 tid 2 2024-03-04T06:14:18.315 INFO:tasks.rados.rados.0.smithi064.stdout:1586: snap_create 2024-03-04T06:14:18.343 INFO:tasks.rados.rados.0.smithi064.stdout:1585: finishing write tid 1 to smithi06431743-18 2024-03-04T06:14:18.355 INFO:tasks.rados.rados.0.smithi064.stdout:1585: finishing write tid 2 to smithi06431743-18 2024-03-04T06:14:18.357 INFO:tasks.rados.rados.0.smithi064.stdout:1585: finishing write tid 3 to smithi06431743-18 2024-03-04T06:14:18.358 INFO:tasks.rados.rados.0.smithi064.stdout:1585: finishing write tid 4 to smithi06431743-18 2024-03-04T06:14:18.358 INFO:tasks.rados.rados.0.smithi064.stdout:1585: oid 18 updating version 0 to 226 2024-03-04T06:14:18.359 INFO:tasks.rados.rados.0.smithi064.stdout:1585: oid 18 version 226 is already newer than 226 2024-03-04T06:14:18.359 INFO:tasks.rados.rados.0.smithi064.stdout:1585: oid 18 version 226 is already newer than 224 2024-03-04T06:14:18.359 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 226 (ObjNum 758 snap 189 seq_num 758) dirty exists 2024-03-04T06:14:18.359 INFO:tasks.rados.rados.0.smithi064.stdout:1585: left oid 18 (ObjNum 758 snap 189 seq_num 758) 2024-03-04T06:14:19.312 INFO:tasks.rados.rados.0.smithi064.stdout:1585: done (1 left) 2024-03-04T06:14:19.312 INFO:tasks.rados.rados.0.smithi064.stdout:1586: done (0 left) 2024-03-04T06:14:19.312 INFO:tasks.rados.rados.0.smithi064.stdout:1587: snap_remove snap 186 2024-03-04T06:14:19.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:19.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:19.616+0000 7f949f46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:19.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:19.621+0000 7f949f46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:20.311 INFO:tasks.rados.rados.0.smithi064.stdout:1587: done (0 left) 2024-03-04T06:14:20.311 INFO:tasks.rados.rados.0.smithi064.stdout:1588: delete oid 28 current snap is 190 2024-03-04T06:14:20.358 INFO:tasks.rados.rados.0.smithi064.stdout:1588: done (0 left) 2024-03-04T06:14:20.358 INFO:tasks.rados.rados.0.smithi064.stdout:1589: write oid 17 current snap is 190 2024-03-04T06:14:20.358 INFO:tasks.rados.rados.0.smithi064.stdout:1589: seq_num 760 ranges {773320=708187,1923609=470924} 2024-03-04T06:14:20.368 INFO:tasks.rados.rados.0.smithi064.stdout:1589: writing smithi06431743-17 from 773320 to 1481507 tid 1 2024-03-04T06:14:20.374 INFO:tasks.rados.rados.0.smithi064.stdout:1589: writing smithi06431743-17 from 1923609 to 2394533 tid 2 2024-03-04T06:14:20.374 INFO:tasks.rados.rados.0.smithi064.stdout:1590: write (excl) oid 9 current snap is 190 2024-03-04T06:14:20.374 INFO:tasks.rados.rados.0.smithi064.stdout:1590: seq_num 761 ranges {790127=499789,1698444=730935} 2024-03-04T06:14:20.383 INFO:tasks.rados.rados.0.smithi064.stdout:1590: writing smithi06431743-9 from 790127 to 1289916 tid 1 2024-03-04T06:14:20.392 INFO:tasks.rados.rados.0.smithi064.stdout:1590: writing smithi06431743-9 from 1698444 to 2429379 tid 2 2024-03-04T06:14:20.392 INFO:tasks.rados.rados.0.smithi064.stdout:1591: read oid 6 snap -1 2024-03-04T06:14:20.392 INFO:tasks.rados.rados.0.smithi064.stdout:1591: expect (ObjNum 747 snap 186 seq_num 747) 2024-03-04T06:14:20.392 INFO:tasks.rados.rados.0.smithi064.stdout:1592: write oid 40 current snap is 190 2024-03-04T06:14:20.392 INFO:tasks.rados.rados.0.smithi064.stdout:1592: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2024-03-04T06:14:20.398 INFO:tasks.rados.rados.0.smithi064.stdout:1592: writing smithi06431743-40 from 406934 to 981972 tid 1 2024-03-04T06:14:20.407 INFO:tasks.rados.rados.0.smithi064.stdout:1592: writing smithi06431743-40 from 1640573 to 2115166 tid 2 2024-03-04T06:14:20.412 INFO:tasks.rados.rados.0.smithi064.stdout:1592: writing smithi06431743-40 from 2642791 to 2806934 tid 3 2024-03-04T06:14:20.412 INFO:tasks.rados.rados.0.smithi064.stdout:1593: write oid 45 current snap is 190 2024-03-04T06:14:20.412 INFO:tasks.rados.rados.0.smithi064.stdout:1593: seq_num 763 ranges {423741=650287,1699055=618251} 2024-03-04T06:14:20.419 INFO:tasks.rados.rados.0.smithi064.stdout:1593: writing smithi06431743-45 from 423741 to 1074028 tid 1 2024-03-04T06:14:20.429 INFO:tasks.rados.rados.0.smithi064.stdout:1593: writing smithi06431743-45 from 1699055 to 2317306 tid 2 2024-03-04T06:14:20.429 INFO:tasks.rados.rados.0.smithi064.stdout:1594: delete oid 18 current snap is 190 2024-03-04T06:14:20.429 INFO:tasks.rados.rados.0.smithi064.stdout:1591: expect (ObjNum 747 snap 186 seq_num 747) 2024-03-04T06:14:20.525 INFO:tasks.rados.rados.0.smithi064.stdout:1589: finishing write tid 1 to smithi06431743-17 2024-03-04T06:14:20.525 INFO:tasks.rados.rados.0.smithi064.stdout:1592: finishing write tid 1 to smithi06431743-40 2024-03-04T06:14:20.525 INFO:tasks.rados.rados.0.smithi064.stdout:1591: done (5 left) 2024-03-04T06:14:20.525 INFO:tasks.rados.rados.0.smithi064.stdout:1594: done (4 left) 2024-03-04T06:14:20.525 INFO:tasks.rados.rados.0.smithi064.stdout:1595: write (excl) oid 42 current snap is 190 2024-03-04T06:14:20.525 INFO:tasks.rados.rados.0.smithi064.stdout:1593: finishing write tid 1 to smithi06431743-45 2024-03-04T06:14:20.525 INFO:tasks.rados.rados.0.smithi064.stdout:1595: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2024-03-04T06:14:20.529 INFO:tasks.rados.rados.0.smithi064.stdout:1595: writing smithi06431743-42 from 457355 to 858140 tid 1 2024-03-04T06:14:20.536 INFO:tasks.rados.rados.0.smithi064.stdout:1595: writing smithi06431743-42 from 1416019 to 2037939 tid 2 2024-03-04T06:14:20.543 INFO:tasks.rados.rados.0.smithi064.stdout:1595: writing smithi06431743-42 from 2725060 to 2857355 tid 3 2024-03-04T06:14:20.543 INFO:tasks.rados.rados.0.smithi064.stdout:1596: read oid 21 snap -1 2024-03-04T06:14:20.543 INFO:tasks.rados.rados.0.smithi064.stdout:1596: expect (ObjNum 708 snap 177 seq_num 708) 2024-03-04T06:14:20.543 INFO:tasks.rados.rados.0.smithi064.stdout:1597: read oid 46 snap -1 2024-03-04T06:14:20.543 INFO:tasks.rados.rados.0.smithi064.stdout:1597: expect (ObjNum 757 snap 188 seq_num 757) 2024-03-04T06:14:20.543 INFO:tasks.rados.rados.0.smithi064.stdout:1598: write oid 44 current snap is 190 2024-03-04T06:14:20.543 INFO:tasks.rados.rados.0.smithi064.stdout:1590: finishing write tid 1 to smithi06431743-9 2024-03-04T06:14:20.543 INFO:tasks.rados.rados.0.smithi064.stdout:1589: finishing write tid 2 to smithi06431743-17 2024-03-04T06:14:20.543 INFO:tasks.rados.rados.0.smithi064.stdout:1589: finishing write tid 3 to smithi06431743-17 2024-03-04T06:14:20.544 INFO:tasks.rados.rados.0.smithi064.stdout:1592: finishing write tid 2 to smithi06431743-40 2024-03-04T06:14:20.544 INFO:tasks.rados.rados.0.smithi064.stdout:1589: finishing write tid 4 to smithi06431743-17 2024-03-04T06:14:20.544 INFO:tasks.rados.rados.0.smithi064.stdout:1589: oid 17 updating version 0 to 206 2024-03-04T06:14:20.544 INFO:tasks.rados.rados.0.smithi064.stdout:1589: oid 17 version 206 is already newer than 204 2024-03-04T06:14:20.544 INFO:tasks.rados.rados.0.smithi064.stdout:1589: oid 17 version 206 is already newer than 206 2024-03-04T06:14:20.544 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 206 (ObjNum 760 snap 190 seq_num 760) dirty exists 2024-03-04T06:14:20.544 INFO:tasks.rados.rados.0.smithi064.stdout:1589: left oid 17 (ObjNum 760 snap 190 seq_num 760) 2024-03-04T06:14:20.544 INFO:tasks.rados.rados.0.smithi064.stdout:1593: finishing write tid 2 to smithi06431743-45 2024-03-04T06:14:20.544 INFO:tasks.rados.rados.0.smithi064.stdout:1598: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2024-03-04T06:14:20.550 INFO:tasks.rados.rados.0.smithi064.stdout:1598: writing smithi06431743-44 from 474162 to 950196 tid 1 2024-03-04T06:14:20.556 INFO:tasks.rados.rados.0.smithi064.stdout:1598: writing smithi06431743-44 from 1358148 to 1840079 tid 2 2024-03-04T06:14:20.564 INFO:tasks.rados.rados.0.smithi064.stdout:1598: writing smithi06431743-44 from 2352483 to 2817786 tid 3 2024-03-04T06:14:20.564 INFO:tasks.rados.rados.0.smithi064.stdout:1589: done (7 left) 2024-03-04T06:14:20.564 INFO:tasks.rados.rados.0.smithi064.stdout:1599: write (excl) oid 36 current snap is 190 2024-03-04T06:14:20.564 INFO:tasks.rados.rados.0.smithi064.stdout:1593: finishing write tid 3 to smithi06431743-45 2024-03-04T06:14:20.564 INFO:tasks.rados.rados.0.smithi064.stdout:1593: finishing write tid 4 to smithi06431743-45 2024-03-04T06:14:20.564 INFO:tasks.rados.rados.0.smithi064.stdout:1593: oid 45 updating version 0 to 352 2024-03-04T06:14:20.564 INFO:tasks.rados.rados.0.smithi064.stdout:1593: oid 45 version 352 is already newer than 351 2024-03-04T06:14:20.564 INFO:tasks.rados.rados.0.smithi064.stdout:1593: oid 45 updating version 352 to 353 2024-03-04T06:14:20.564 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 353 (ObjNum 763 snap 190 seq_num 763) dirty exists 2024-03-04T06:14:20.564 INFO:tasks.rados.rados.0.smithi064.stdout:1593: left oid 45 (ObjNum 763 snap 190 seq_num 763) 2024-03-04T06:14:20.565 INFO:tasks.rados.rados.0.smithi064.stdout:1590: finishing write tid 2 to smithi06431743-9 2024-03-04T06:14:20.565 INFO:tasks.rados.rados.0.smithi064.stdout:1599: seq_num 767 ranges {490969=551283,1816630=625589} 2024-03-04T06:14:20.571 INFO:tasks.rados.rados.0.smithi064.stdout:1599: writing smithi06431743-36 from 490969 to 1042252 tid 1 2024-03-04T06:14:20.579 INFO:tasks.rados.rados.0.smithi064.stdout:1599: writing smithi06431743-36 from 1816630 to 2442219 tid 2 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1593: done (7 left) 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1600: read oid 18 snap -1 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1600: expect deleted 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1601: write (excl) oid 12 current snap is 190 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1592: finishing write tid 3 to smithi06431743-40 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1592: finishing write tid 4 to smithi06431743-40 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1592: finishing write tid 5 to smithi06431743-40 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1592: oid 40 updating version 0 to 289 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1592: oid 40 updating version 289 to 290 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1592: oid 40 updating version 290 to 292 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1592: oid 40 version 292 is already newer than 291 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 292 (ObjNum 762 snap 190 seq_num 762) dirty exists 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1592: left oid 40 (ObjNum 762 snap 190 seq_num 762) 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1590: finishing write tid 3 to smithi06431743-9 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1590: finishing write tid 4 to smithi06431743-9 2024-03-04T06:14:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:1590: oid 9 updating version 0 to 527 2024-03-04T06:14:20.581 INFO:tasks.rados.rados.0.smithi064.stdout:1590: oid 9 updating version 527 to 529 2024-03-04T06:14:20.581 INFO:tasks.rados.rados.0.smithi064.stdout:1590: oid 9 version 529 is already newer than 528 2024-03-04T06:14:20.581 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 529 (ObjNum 761 snap 190 seq_num 761) dirty exists 2024-03-04T06:14:20.581 INFO:tasks.rados.rados.0.smithi064.stdout:1590: left oid 9 (ObjNum 761 snap 190 seq_num 761) 2024-03-04T06:14:20.581 INFO:tasks.rados.rados.0.smithi064.stdout:1601: seq_num 768 ranges {507776=742885,1991465=485600} 2024-03-04T06:14:20.585 INFO:tasks.rados.rados.0.smithi064.stdout:1601: writing smithi06431743-12 from 507776 to 1250661 tid 1 2024-03-04T06:14:20.592 INFO:tasks.rados.rados.0.smithi064.stdout:1601: writing smithi06431743-12 from 1991465 to 2477065 tid 2 2024-03-04T06:14:20.592 INFO:tasks.rados.rados.0.smithi064.stdout:1590: done (8 left) 2024-03-04T06:14:20.593 INFO:tasks.rados.rados.0.smithi064.stdout:1592: done (7 left) 2024-03-04T06:14:20.593 INFO:tasks.rados.rados.0.smithi064.stdout:1602: delete oid 23 current snap is 190 2024-03-04T06:14:20.593 INFO:tasks.rados.rados.0.smithi064.stdout:1597: expect (ObjNum 757 snap 188 seq_num 757) 2024-03-04T06:14:20.783 INFO:tasks.rados.rados.0.smithi064.stdout:1596: expect (ObjNum 708 snap 177 seq_num 708) 2024-03-04T06:14:20.979 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:14:20.979 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:14:20.979 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:14:20.986 INFO:tasks.rados.rados.0.smithi064.stdout:1595: finishing write tid 1 to smithi06431743-42 2024-03-04T06:14:20.986 INFO:tasks.rados.rados.0.smithi064.stdout:1596: done (7 left) 2024-03-04T06:14:20.986 INFO:tasks.rados.rados.0.smithi064.stdout:1597: done (6 left) 2024-03-04T06:14:20.986 INFO:tasks.rados.rados.0.smithi064.stdout:1602: done (5 left) 2024-03-04T06:14:20.986 INFO:tasks.rados.rados.0.smithi064.stdout:1603: write oid 1 current snap is 190 2024-03-04T06:14:20.986 INFO:tasks.rados.rados.0.smithi064.stdout:1599: finishing write tid 1 to smithi06431743-36 2024-03-04T06:14:20.986 INFO:tasks.rados.rados.0.smithi064.stdout:1595: finishing write tid 2 to smithi06431743-42 2024-03-04T06:14:20.986 INFO:tasks.rados.rados.0.smithi064.stdout:1603: seq_num 770 ranges {541390=493383,1592076=489269,2862175=79215} 2024-03-04T06:14:20.991 INFO:tasks.rados.rados.0.smithi064.stdout:1603: writing smithi06431743-1 from 541390 to 1034773 tid 1 2024-03-04T06:14:20.997 INFO:tasks.rados.rados.0.smithi064.stdout:1603: writing smithi06431743-1 from 1592076 to 2081345 tid 2 2024-03-04T06:14:21.002 INFO:tasks.rados.rados.0.smithi064.stdout:1603: writing smithi06431743-1 from 2862175 to 2941390 tid 3 2024-03-04T06:14:21.003 INFO:tasks.rados.rados.0.smithi064.stdout:1600: done (5 left) 2024-03-04T06:14:21.003 INFO:tasks.rados.rados.0.smithi064.stdout:1604: read oid 29 snap -1 2024-03-04T06:14:21.003 INFO:tasks.rados.rados.0.smithi064.stdout:1604: expect deleted 2024-03-04T06:14:21.003 INFO:tasks.rados.rados.0.smithi064.stdout:1605: read oid 28 snap -1 2024-03-04T06:14:21.003 INFO:tasks.rados.rados.0.smithi064.stdout:1605: expect deleted 2024-03-04T06:14:21.003 INFO:tasks.rados.rados.0.smithi064.stdout:1606: write (excl) oid 17 current snap is 190 2024-03-04T06:14:21.003 INFO:tasks.rados.rados.0.smithi064.stdout:1598: finishing write tid 1 to smithi06431743-44 2024-03-04T06:14:21.003 INFO:tasks.rados.rados.0.smithi064.stdout:1599: finishing write tid 2 to smithi06431743-36 2024-03-04T06:14:21.003 INFO:tasks.rados.rados.0.smithi064.stdout:1601: finishing write tid 1 to smithi06431743-12 2024-03-04T06:14:21.003 INFO:tasks.rados.rados.0.smithi064.stdout:1606: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2024-03-04T06:14:21.010 INFO:tasks.rados.rados.0.smithi064.stdout:1606: writing smithi06431743-17 from 558197 to 1126829 tid 1 2024-03-04T06:14:21.018 INFO:tasks.rados.rados.0.smithi064.stdout:1606: writing smithi06431743-17 from 1650558 to 2283485 tid 2 2024-03-04T06:14:21.023 INFO:tasks.rados.rados.0.smithi064.stdout:1606: writing smithi06431743-17 from 2773245 to 2958197 tid 3 2024-03-04T06:14:21.023 INFO:tasks.rados.rados.0.smithi064.stdout:1607: snap_remove snap 178 2024-03-04T06:14:21.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:21.086+0000 7f3682e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:21.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:21.092+0000 7f3682e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:21.283 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:14:21.283 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:14:21.295 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":579,"stamp":"2024-03-04T06:14:20.788691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":116027081,"num_objects":545,"num_object_clones":403,"num_object_copies":1635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":99,"num_read":2965,"num_read_kb":2187822,"num_write":2641,"num_write_kb":683248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5180,"ondisk_log_size":5180,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":489372,"kb_used_data":344636,"kb_used_omap":0,"kb_used_meta":144704,"kb_avail":376997988,"statfs":{"total":386547056640,"available":386045939712,"internally_reserved":0,"allocated":352907264,"data_stored":380842313,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":148176896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4437114,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":9,"num_read_kb":5429,"num_write":10,"num_write_kb":2181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000948"},"pg_stats":[{"pgid":"3.f","version":"387'287","reported_seq":1341,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.326890+0000","last_change":"2024-03-04T06:14:17.371894+0000","last_active":"2024-03-04T06:14:20.326890+0000","last_peered":"2024-03-04T06:14:20.326890+0000","last_clean":"2024-03-04T06:14:20.326890+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:14:20.326890+0000","last_undegraded":"2024-03-04T06:14:20.326890+0000","last_fullsized":"2024-03-04T06:14:20.326890+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8370348,"num_objects":38,"num_object_clones":28,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":166,"num_read_kb":120266,"num_write":159,"num_write_kb":43919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"388'226","reported_seq":1290,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.330221+0000","last_change":"2024-03-04T06:14:15.311038+0000","last_active":"2024-03-04T06:14:20.330221+0000","last_peered":"2024-03-04T06:14:20.330221+0000","last_clean":"2024-03-04T06:14:20.330221+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:14:20.330221+0000","last_undegraded":"2024-03-04T06:14:20.330221+0000","last_fullsized":"2024-03-04T06:14:20.330221+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5378570,"num_objects":28,"num_object_clones":19,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":92,"num_read_kb":57071,"num_write":108,"num_write_kb":28102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"386'350","reported_seq":1371,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.339463+0000","last_change":"2024-03-04T06:14:16.341675+0000","last_active":"2024-03-04T06:14:16.339463+0000","last_peered":"2024-03-04T06:14:16.339463+0000","last_clean":"2024-03-04T06:14:16.339463+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:14:16.339463+0000","last_undegraded":"2024-03-04T06:14:16.339463+0000","last_fullsized":"2024-03-04T06:14:16.339463+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7681261,"num_objects":42,"num_object_clones":32,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":210,"num_read_kb":152388,"num_write":186,"num_write_kb":51313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"384'275","reported_seq":1338,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.320921+0000","last_change":"2024-03-04T06:14:17.318518+0000","last_active":"2024-03-04T06:14:20.320921+0000","last_peered":"2024-03-04T06:14:20.320921+0000","last_clean":"2024-03-04T06:14:20.320921+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:14:20.320921+0000","last_undegraded":"2024-03-04T06:14:20.320921+0000","last_fullsized":"2024-03-04T06:14:20.320921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8235892,"num_objects":29,"num_object_clones":21,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":162,"num_read_kb":130100,"num_write":153,"num_write_kb":43195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"381'134","reported_seq":1188,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.320612+0000","last_change":"2024-03-04T06:14:16.324496+0000","last_active":"2024-03-04T06:14:16.320612+0000","last_peered":"2024-03-04T06:14:16.320612+0000","last_clean":"2024-03-04T06:14:16.320612+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:14:16.320612+0000","last_undegraded":"2024-03-04T06:14:16.320612+0000","last_fullsized":"2024-03-04T06:14:16.320612+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336338,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":41,"num_read_kb":17047,"num_write":60,"num_write_kb":15555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"384'314","reported_seq":1349,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.327699+0000","last_change":"2024-03-04T06:14:17.351013+0000","last_active":"2024-03-04T06:14:20.327699+0000","last_peered":"2024-03-04T06:14:20.327699+0000","last_clean":"2024-03-04T06:14:20.327699+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:14:20.327699+0000","last_undegraded":"2024-03-04T06:14:20.327699+0000","last_fullsized":"2024-03-04T06:14:20.327699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815717,"num_objects":33,"num_object_clones":25,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":162,"num_read_kb":124272,"num_write":171,"num_write_kb":50947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1127,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.324916+0000","last_change":"2024-03-04T06:14:17.351231+0000","last_active":"2024-03-04T06:14:20.324916+0000","last_peered":"2024-03-04T06:14:20.324916+0000","last_clean":"2024-03-04T06:14:20.324916+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:14:20.324916+0000","last_undegraded":"2024-03-04T06:14:20.324916+0000","last_fullsized":"2024-03-04T06:14:20.324916+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"383'202","reported_seq":1263,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.302854+0000","last_change":"2024-03-04T06:14:16.307603+0000","last_active":"2024-03-04T06:14:16.302854+0000","last_peered":"2024-03-04T06:14:16.302854+0000","last_clean":"2024-03-04T06:14:16.302854+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:14:16.302854+0000","last_undegraded":"2024-03-04T06:14:16.302854+0000","last_fullsized":"2024-03-04T06:14:16.302854+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6454185,"num_objects":22,"num_object_clones":14,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":141,"num_read_kb":133004,"num_write":94,"num_write_kb":25039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1079,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.305669+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:14:16.305669+0000","last_peered":"2024-03-04T06:14:16.305669+0000","last_clean":"2024-03-04T06:14:16.305669+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:14:16.305669+0000","last_undegraded":"2024-03-04T06:14:16.305669+0000","last_fullsized":"2024-03-04T06:14:16.305669+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":942,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.310829+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:14:16.310829+0000","last_peered":"2024-03-04T06:14:16.310829+0000","last_clean":"2024-03-04T06:14:16.310829+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:14:16.310829+0000","last_undegraded":"2024-03-04T06:14:16.310829+0000","last_fullsized":"2024-03-04T06:14:16.310829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"384'340","reported_seq":1334,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.312506+0000","last_change":"2024-03-04T06:14:16.314800+0000","last_active":"2024-03-04T06:14:16.312506+0000","last_peered":"2024-03-04T06:14:16.312506+0000","last_clean":"2024-03-04T06:14:16.312506+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:14:16.312506+0000","last_undegraded":"2024-03-04T06:14:16.312506+0000","last_fullsized":"2024-03-04T06:14:16.312506+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7698068,"num_objects":35,"num_object_clones":26,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":178,"num_read_kb":134383,"num_write":160,"num_write_kb":42642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":954,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.324989+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:14:20.324989+0000","last_peered":"2024-03-04T06:14:20.324989+0000","last_clean":"2024-03-04T06:14:20.324989+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:14:20.324989+0000","last_undegraded":"2024-03-04T06:14:20.324989+0000","last_fullsized":"2024-03-04T06:14:20.324989+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"381'258","reported_seq":1269,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.306888+0000","last_change":"2024-03-04T06:14:16.315246+0000","last_active":"2024-03-04T06:14:16.306888+0000","last_peered":"2024-03-04T06:14:16.306888+0000","last_clean":"2024-03-04T06:14:16.306888+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:14:16.306888+0000","last_undegraded":"2024-03-04T06:14:16.306888+0000","last_fullsized":"2024-03-04T06:14:16.306888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647449,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":130,"num_read_kb":106966,"num_write":105,"num_write_kb":30304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":942,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.301158+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:14:16.301158+0000","last_peered":"2024-03-04T06:14:16.301158+0000","last_clean":"2024-03-04T06:14:16.301158+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:14:16.301158+0000","last_undegraded":"2024-03-04T06:14:16.301158+0000","last_fullsized":"2024-03-04T06:14:16.301158+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"384'620","reported_seq":1567,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.325716+0000","last_change":"2024-03-04T06:14:16.328005+0000","last_active":"2024-03-04T06:14:16.325716+0000","last_peered":"2024-03-04T06:14:16.325716+0000","last_clean":"2024-03-04T06:14:16.325716+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:14:16.325716+0000","last_undegraded":"2024-03-04T06:14:16.325716+0000","last_fullsized":"2024-03-04T06:14:16.325716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10034406,"num_objects":65,"num_object_clones":48,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":380,"num_read_kb":298874,"num_write":313,"num_write_kb":82698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":955,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.318958+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:14:20.318958+0000","last_peered":"2024-03-04T06:14:20.318958+0000","last_clean":"2024-03-04T06:14:20.318958+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:14:20.318958+0000","last_undegraded":"2024-03-04T06:14:20.318958+0000","last_fullsized":"2024-03-04T06:14:20.318958+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"384'608","reported_seq":1513,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.333473+0000","last_change":"2024-03-04T06:14:15.308900+0000","last_active":"2024-03-04T06:14:20.333473+0000","last_peered":"2024-03-04T06:14:20.333473+0000","last_clean":"2024-03-04T06:14:20.333473+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:14:20.333473+0000","last_undegraded":"2024-03-04T06:14:20.333473+0000","last_fullsized":"2024-03-04T06:14:20.333473+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6706488,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":249,"num_read_kb":171966,"num_write":282,"num_write_kb":72697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"382'257","reported_seq":1334,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.310218+0000","last_change":"2024-03-04T06:14:16.317203+0000","last_active":"2024-03-04T06:14:16.310218+0000","last_peered":"2024-03-04T06:14:16.310218+0000","last_clean":"2024-03-04T06:14:16.310218+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:14:16.310218+0000","last_undegraded":"2024-03-04T06:14:16.310218+0000","last_fullsized":"2024-03-04T06:14:16.310218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8723295,"num_objects":29,"num_object_clones":20,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":220,"num_read_kb":176023,"num_write":133,"num_write_kb":35332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":983,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.346432+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:14:20.346432+0000","last_peered":"2024-03-04T06:14:20.346432+0000","last_clean":"2024-03-04T06:14:20.346432+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:14:20.346432+0000","last_undegraded":"2024-03-04T06:14:20.346432+0000","last_fullsized":"2024-03-04T06:14:20.346432+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"382'289","reported_seq":1352,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.350766+0000","last_change":"2024-03-04T06:14:15.307778+0000","last_active":"2024-03-04T06:14:20.350766+0000","last_peered":"2024-03-04T06:14:20.350766+0000","last_clean":"2024-03-04T06:14:20.350766+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:14:20.350766+0000","last_undegraded":"2024-03-04T06:14:20.350766+0000","last_fullsized":"2024-03-04T06:14:20.350766+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9277893,"num_objects":34,"num_object_clones":22,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":161,"num_read_kb":103810,"num_write":142,"num_write_kb":38074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":946,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.324964+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:14:16.324964+0000","last_peered":"2024-03-04T06:14:16.324964+0000","last_clean":"2024-03-04T06:14:16.324964+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:14:16.324964+0000","last_undegraded":"2024-03-04T06:14:16.324964+0000","last_fullsized":"2024-03-04T06:14:16.324964+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"384'401","reported_seq":1411,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.316327+0000","last_change":"2024-03-04T06:14:16.316327+0000","last_active":"2024-03-04T06:14:16.316327+0000","last_peered":"2024-03-04T06:14:16.316327+0000","last_clean":"2024-03-04T06:14:16.316327+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:14:16.316327+0000","last_undegraded":"2024-03-04T06:14:16.316327+0000","last_fullsized":"2024-03-04T06:14:16.316327+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":401,"log_dups_size":0,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12639425,"num_objects":42,"num_object_clones":31,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":245,"num_read_kb":190973,"num_write":206,"num_write_kb":57452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":983,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.329391+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:14:20.329391+0000","last_peered":"2024-03-04T06:14:20.329391+0000","last_clean":"2024-03-04T06:14:20.329391+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:14:20.329391+0000","last_undegraded":"2024-03-04T06:14:20.329391+0000","last_fullsized":"2024-03-04T06:14:20.329391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"387'522","reported_seq":1510,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T06:14:20.337619+0000","last_change":"2024-03-04T06:14:15.300178+0000","last_active":"2024-03-04T06:14:20.337619+0000","last_peered":"2024-03-04T06:14:20.337619+0000","last_clean":"2024-03-04T06:14:20.337619+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:14:20.337619+0000","last_undegraded":"2024-03-04T06:14:20.337619+0000","last_fullsized":"2024-03-04T06:14:20.337619+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6437378,"num_objects":48,"num_object_clones":42,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":325,"num_read_kb":270588,"num_write":232,"num_write_kb":64085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":942,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-04T06:14:16.311534+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:14:16.311534+0000","last_peered":"2024-03-04T06:14:16.311534+0000","last_clean":"2024-03-04T06:14:16.311534+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:14:16.311534+0000","last_undegraded":"2024-03-04T06:14:16.311534+0000","last_fullsized":"2024-03-04T06:14:16.311534+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":115436713,"num_objects":543,"num_object_clones":403,"num_object_copies":1629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":99,"num_read":2862,"num_read_kb":2187731,"num_write":2504,"num_write_kb":681354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5083,"ondisk_log_size":5083,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706054,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":145584,"kb_used_data":106280,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94226256,"statfs":{"total":96636764160,"available":96487686144,"internally_reserved":0,"allocated":108830720,"data_stored":117744710,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706054,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":115528,"kb_used_data":80384,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94256312,"statfs":{"total":96636764160,"available":96518463488,"internally_reserved":0,"allocated":82313216,"data_stored":89331007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706058,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112580,"kb_used_data":77436,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94259260,"statfs":{"total":96636764160,"available":96521482240,"internally_reserved":0,"allocated":79294464,"data_stored":84615708,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706058,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":115680,"kb_used_data":80536,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94256160,"statfs":{"total":96636764160,"available":96518307840,"internally_reserved":0,"allocated":82468864,"data_stored":89150888,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1607: done (8 left) 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1608: read oid 38 snap -1 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1608: expect (ObjNum 697 snap 172 seq_num 697) 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1609: snap_create 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1599: finishing write tid 3 to smithi06431743-36 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1599: finishing write tid 4 to smithi06431743-36 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1599: oid 36 updating version 0 to 291 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1599: oid 36 updating version 291 to 293 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1599: oid 36 version 293 is already newer than 292 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 293 (ObjNum 767 snap 190 seq_num 767) dirty exists 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1599: left oid 36 (ObjNum 767 snap 190 seq_num 767) 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1598: finishing write tid 2 to smithi06431743-44 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1595: finishing write tid 3 to smithi06431743-42 2024-03-04T06:14:21.320 INFO:tasks.rados.rados.0.smithi064.stdout:1601: finishing write tid 2 to smithi06431743-12 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1595: finishing write tid 4 to smithi06431743-42 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1595: finishing write tid 5 to smithi06431743-42 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1595: oid 42 updating version 0 to 295 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1595: oid 42 updating version 295 to 297 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1595: oid 42 version 297 is already newer than 296 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1595: oid 42 version 297 is already newer than 294 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 297 (ObjNum 765 snap 190 seq_num 765) dirty exists 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1595: left oid 42 (ObjNum 765 snap 190 seq_num 765) 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1601: finishing write tid 3 to smithi06431743-12 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1601: finishing write tid 4 to smithi06431743-12 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1601: oid 12 updating version 0 to 278 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1601: oid 12 version 278 is already newer than 277 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1601: oid 12 updating version 278 to 279 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 279 (ObjNum 768 snap 190 seq_num 768) dirty exists 2024-03-04T06:14:21.321 INFO:tasks.rados.rados.0.smithi064.stdout:1601: left oid 12 (ObjNum 768 snap 190 seq_num 768) 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1598: finishing write tid 3 to smithi06431743-44 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1598: finishing write tid 4 to smithi06431743-44 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1598: finishing write tid 5 to smithi06431743-44 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1598: oid 44 updating version 0 to 318 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1598: oid 44 version 318 is already newer than 316 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1598: oid 44 updating version 318 to 319 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1598: oid 44 version 319 is already newer than 317 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 319 (ObjNum 766 snap 190 seq_num 766) dirty exists 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1598: left oid 44 (ObjNum 766 snap 190 seq_num 766) 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1603: finishing write tid 1 to smithi06431743-1 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1606: finishing write tid 1 to smithi06431743-17 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1603: finishing write tid 2 to smithi06431743-1 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1603: finishing write tid 3 to smithi06431743-1 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1606: finishing write tid 2 to smithi06431743-17 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1603: finishing write tid 4 to smithi06431743-1 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1603: finishing write tid 5 to smithi06431743-1 2024-03-04T06:14:21.322 INFO:tasks.rados.rados.0.smithi064.stdout:1603: oid 1 updating version 0 to 630 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1603: oid 1 version 630 is already newer than 628 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1603: oid 1 updating version 630 to 631 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1603: oid 1 version 631 is already newer than 629 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 631 (ObjNum 770 snap 190 seq_num 770) dirty exists 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1603: left oid 1 (ObjNum 770 snap 190 seq_num 770) 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1606: finishing write tid 3 to smithi06431743-17 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1606: finishing write tid 4 to smithi06431743-17 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1606: finishing write tid 5 to smithi06431743-17 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1606: oid 17 updating version 0 to 208 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1606: oid 17 version 208 is already newer than 207 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1606: oid 17 updating version 208 to 210 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1606: oid 17 version 210 is already newer than 210 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 210 (ObjNum 771 snap 190 seq_num 771) dirty exists 2024-03-04T06:14:21.323 INFO:tasks.rados.rados.0.smithi064.stdout:1606: left oid 17 (ObjNum 771 snap 190 seq_num 771) 2024-03-04T06:14:21.388 INFO:tasks.rados.rados.0.smithi064.stdout:1608: expect (ObjNum 697 snap 172 seq_num 697) 2024-03-04T06:14:21.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:21.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:21.796+0000 7f6d0879d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:21.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:21.801+0000 7f6d0879d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:22.320 INFO:tasks.rados.rados.0.smithi064.stdout:1595: done (9 left) 2024-03-04T06:14:22.320 INFO:tasks.rados.rados.0.smithi064.stdout:1598: done (8 left) 2024-03-04T06:14:22.320 INFO:tasks.rados.rados.0.smithi064.stdout:1599: done (7 left) 2024-03-04T06:14:22.320 INFO:tasks.rados.rados.0.smithi064.stdout:1601: done (6 left) 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1603: done (5 left) 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1604: done (4 left) 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1605: done (3 left) 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1606: done (2 left) 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1608: done (1 left) 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1609: done (0 left) 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1610: read oid 8 snap -1 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1610: expect (ObjNum 663 snap 165 seq_num 663) 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1611: read oid 38 snap 183 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1611: expect (ObjNum 697 snap 172 seq_num 697) 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1612: write (excl) oid 48 current snap is 191 2024-03-04T06:14:22.321 INFO:tasks.rados.rados.0.smithi064.stdout:1612: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-03-04T06:14:22.328 INFO:tasks.rados.rados.0.smithi064.stdout:1612: writing smithi06431743-48 from 575004 to 1218885 tid 1 2024-03-04T06:14:22.335 INFO:tasks.rados.rados.0.smithi064.stdout:1612: writing smithi06431743-48 from 1709040 to 2201978 tid 2 2024-03-04T06:14:22.340 INFO:tasks.rados.rados.0.smithi064.stdout:1612: writing smithi06431743-48 from 2917021 to 2975004 tid 3 2024-03-04T06:14:22.340 INFO:tasks.rados.rados.0.smithi064.stdout:1613: write oid 19 current snap is 191 2024-03-04T06:14:22.340 INFO:tasks.rados.rados.0.smithi064.stdout:1613: seq_num 773 ranges {591811=719130,1767522=636596,2828091=163720} 2024-03-04T06:14:22.349 INFO:tasks.rados.rados.0.smithi064.stdout:1613: writing smithi06431743-19 from 591811 to 1310941 tid 1 2024-03-04T06:14:22.357 INFO:tasks.rados.rados.0.smithi064.stdout:1613: writing smithi06431743-19 from 1767522 to 2404118 tid 2 2024-03-04T06:14:22.361 INFO:tasks.rados.rados.0.smithi064.stdout:1613: writing smithi06431743-19 from 2828091 to 2991811 tid 3 2024-03-04T06:14:22.362 INFO:tasks.rados.rados.0.smithi064.stdout:1614: snap_create 2024-03-04T06:14:22.375 INFO:tasks.rados.rados.0.smithi064.stdout:1611: expect (ObjNum 697 snap 172 seq_num 697) 2024-03-04T06:14:22.532 INFO:tasks.rados.rados.0.smithi064.stdout:1610: expect (ObjNum 663 snap 165 seq_num 663) 2024-03-04T06:14:22.704 INFO:tasks.rados.rados.0.smithi064.stdout:1612: finishing write tid 1 to smithi06431743-48 2024-03-04T06:14:22.704 INFO:tasks.rados.rados.0.smithi064.stdout:1613: finishing write tid 1 to smithi06431743-19 2024-03-04T06:14:22.704 INFO:tasks.rados.rados.0.smithi064.stdout:1612: finishing write tid 2 to smithi06431743-48 2024-03-04T06:14:22.704 INFO:tasks.rados.rados.0.smithi064.stdout:1613: finishing write tid 2 to smithi06431743-19 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1612: finishing write tid 3 to smithi06431743-48 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1613: finishing write tid 3 to smithi06431743-19 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1613: finishing write tid 4 to smithi06431743-19 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1613: finishing write tid 5 to smithi06431743-19 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1613: oid 19 updating version 0 to 617 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1613: oid 19 version 617 is already newer than 616 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1613: oid 19 version 617 is already newer than 614 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1613: oid 19 version 617 is already newer than 615 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 617 (ObjNum 773 snap 191 seq_num 773) dirty exists 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1613: left oid 19 (ObjNum 773 snap 191 seq_num 773) 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1612: finishing write tid 4 to smithi06431743-48 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1612: finishing write tid 5 to smithi06431743-48 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1612: oid 48 updating version 0 to 263 2024-03-04T06:14:22.705 INFO:tasks.rados.rados.0.smithi064.stdout:1612: oid 48 updating version 263 to 265 2024-03-04T06:14:22.706 INFO:tasks.rados.rados.0.smithi064.stdout:1612: oid 48 version 265 is already newer than 262 2024-03-04T06:14:22.706 INFO:tasks.rados.rados.0.smithi064.stdout:1612: oid 48 version 265 is already newer than 264 2024-03-04T06:14:22.706 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 265 (ObjNum 772 snap 191 seq_num 772) dirty exists 2024-03-04T06:14:22.706 INFO:tasks.rados.rados.0.smithi064.stdout:1612: left oid 48 (ObjNum 772 snap 191 seq_num 772) 2024-03-04T06:14:23.236 INFO:tasks.rados.rados.0.smithi064.stdout:1610: done (4 left) 2024-03-04T06:14:23.236 INFO:tasks.rados.rados.0.smithi064.stdout:1611: done (3 left) 2024-03-04T06:14:23.236 INFO:tasks.rados.rados.0.smithi064.stdout:1612: done (2 left) 2024-03-04T06:14:23.236 INFO:tasks.rados.rados.0.smithi064.stdout:1613: done (1 left) 2024-03-04T06:14:23.237 INFO:tasks.rados.rados.0.smithi064.stdout:1614: done (0 left) 2024-03-04T06:14:23.237 INFO:tasks.rados.rados.0.smithi064.stdout:1615: delete oid 18 current snap is 192 2024-03-04T06:14:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:1615: done (0 left) 2024-03-04T06:14:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:1616: read oid 8 snap -1 2024-03-04T06:14:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:1616: expect (ObjNum 663 snap 165 seq_num 663) 2024-03-04T06:14:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:1617: read oid 24 snap -1 2024-03-04T06:14:23.267 INFO:tasks.rados.rados.0.smithi064.stdout:1617: expect (ObjNum 669 snap 166 seq_num 669) 2024-03-04T06:14:23.267 INFO:tasks.rados.rados.0.smithi064.stdout:1618: delete oid 28 current snap is 192 2024-03-04T06:14:23.285 INFO:tasks.rados.rados.0.smithi064.stdout:1616: expect (ObjNum 663 snap 165 seq_num 663) 2024-03-04T06:14:23.458 INFO:tasks.rados.rados.0.smithi064.stdout:1617: expect (ObjNum 669 snap 166 seq_num 669) 2024-03-04T06:14:23.576 INFO:tasks.rados.rados.0.smithi064.stdout:1616: done (2 left) 2024-03-04T06:14:23.577 INFO:tasks.rados.rados.0.smithi064.stdout:1617: done (1 left) 2024-03-04T06:14:23.577 INFO:tasks.rados.rados.0.smithi064.stdout:1618: done (0 left) 2024-03-04T06:14:23.577 INFO:tasks.rados.rados.0.smithi064.stdout:1619: delete oid 6 current snap is 192 2024-03-04T06:14:23.597 INFO:tasks.rados.rados.0.smithi064.stdout:1619: done (0 left) 2024-03-04T06:14:23.597 INFO:tasks.rados.rados.0.smithi064.stdout:1620: write oid 27 current snap is 192 2024-03-04T06:14:23.597 INFO:tasks.rados.rados.0.smithi064.stdout:1620: seq_num 777 ranges {659039=736479,2001450=643934} 2024-03-04T06:14:23.605 INFO:tasks.rados.rados.0.smithi064.stdout:1620: writing smithi06431743-27 from 659039 to 1395518 tid 1 2024-03-04T06:14:23.612 INFO:tasks.rados.rados.0.smithi064.stdout:1620: writing smithi06431743-27 from 2001450 to 2645384 tid 2 2024-03-04T06:14:23.612 INFO:tasks.rados.rados.0.smithi064.stdout:1621: read oid 40 snap -1 2024-03-04T06:14:23.612 INFO:tasks.rados.rados.0.smithi064.stdout:1621: expect (ObjNum 762 snap 190 seq_num 762) 2024-03-04T06:14:23.612 INFO:tasks.rados.rados.0.smithi064.stdout:1622: write (excl) oid 43 current snap is 192 2024-03-04T06:14:23.612 INFO:tasks.rados.rados.0.smithi064.stdout:1622: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2024-03-04T06:14:23.618 INFO:tasks.rados.rados.0.smithi064.stdout:1622: writing smithi06431743-43 from 675846 to 1087574 tid 1 2024-03-04T06:14:23.625 INFO:tasks.rados.rados.0.smithi064.stdout:1622: writing smithi06431743-43 from 1543579 to 2331171 tid 2 2024-03-04T06:14:23.631 INFO:tasks.rados.rados.0.smithi064.stdout:1622: writing smithi06431743-43 from 2848853 to 3075846 tid 3 2024-03-04T06:14:23.631 INFO:tasks.rados.rados.0.smithi064.stdout:1623: write oid 13 current snap is 192 2024-03-04T06:14:23.631 INFO:tasks.rados.rados.0.smithi064.stdout:1620: finishing write tid 1 to smithi06431743-27 2024-03-04T06:14:23.632 INFO:tasks.rados.rados.0.smithi064.stdout:1623: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2024-03-04T06:14:23.638 INFO:tasks.rados.rados.0.smithi064.stdout:1623: writing smithi06431743-13 from 692653 to 1179630 tid 1 2024-03-04T06:14:23.644 INFO:tasks.rados.rados.0.smithi064.stdout:1623: writing smithi06431743-13 from 1602061 to 2249664 tid 2 2024-03-04T06:14:23.649 INFO:tasks.rados.rados.0.smithi064.stdout:1623: writing smithi06431743-13 from 2992629 to 3092653 tid 3 2024-03-04T06:14:23.649 INFO:tasks.rados.rados.0.smithi064.stdout:1624: delete oid 30 current snap is 192 2024-03-04T06:14:23.650 INFO:tasks.rados.rados.0.smithi064.stdout:1620: finishing write tid 2 to smithi06431743-27 2024-03-04T06:14:23.656 INFO:tasks.rados.rados.0.smithi064.stdout:1623: finishing write tid 1 to smithi06431743-13 2024-03-04T06:14:23.657 INFO:tasks.rados.rados.0.smithi064.stdout:1621: expect (ObjNum 762 snap 190 seq_num 762) 2024-03-04T06:14:23.801 INFO:tasks.rados.rados.0.smithi064.stdout:1620: finishing write tid 3 to smithi06431743-27 2024-03-04T06:14:23.801 INFO:tasks.rados.rados.0.smithi064.stdout:1620: finishing write tid 4 to smithi06431743-27 2024-03-04T06:14:23.801 INFO:tasks.rados.rados.0.smithi064.stdout:1620: oid 27 updating version 0 to 233 2024-03-04T06:14:23.801 INFO:tasks.rados.rados.0.smithi064.stdout:1620: oid 27 updating version 233 to 235 2024-03-04T06:14:23.801 INFO:tasks.rados.rados.0.smithi064.stdout:1620: oid 27 version 235 is already newer than 234 2024-03-04T06:14:23.801 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 235 (ObjNum 777 snap 192 seq_num 777) dirty exists 2024-03-04T06:14:23.801 INFO:tasks.rados.rados.0.smithi064.stdout:1620: left oid 27 (ObjNum 777 snap 192 seq_num 777) 2024-03-04T06:14:23.802 INFO:tasks.rados.rados.0.smithi064.stdout:1622: finishing write tid 1 to smithi06431743-43 2024-03-04T06:14:23.802 INFO:tasks.rados.rados.0.smithi064.stdout:1620: done (4 left) 2024-03-04T06:14:23.802 INFO:tasks.rados.rados.0.smithi064.stdout:1621: done (3 left) 2024-03-04T06:14:23.802 INFO:tasks.rados.rados.0.smithi064.stdout:1624: done (2 left) 2024-03-04T06:14:23.802 INFO:tasks.rados.rados.0.smithi064.stdout:1625: snap_remove snap 187 2024-03-04T06:14:23.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:23.973 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:23.971+0000 7ff9f0b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:23.976 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:23.976+0000 7ff9f0b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:24.326 INFO:tasks.rados.rados.0.smithi064.stdout:1625: done (2 left) 2024-03-04T06:14:24.326 INFO:tasks.rados.rados.0.smithi064.stdout:1626: read oid 38 snap 176 2024-03-04T06:14:24.327 INFO:tasks.rados.rados.0.smithi064.stdout:1626: expect (ObjNum 697 snap 172 seq_num 697) 2024-03-04T06:14:24.327 INFO:tasks.rados.rados.0.smithi064.stdout:1627: snap_remove snap 189 2024-03-04T06:14:25.331 INFO:tasks.rados.rados.0.smithi064.stdout:1627: done (3 left) 2024-03-04T06:14:25.331 INFO:tasks.rados.rados.0.smithi064.stdout:1628: snap_remove snap 188 2024-03-04T06:14:26.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:26.154 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:26.152+0000 7fba7a064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:26.157 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:26.156+0000 7fba7a064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:26.298 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:14:26.298 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:14:26.298 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:14:26.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:14:26.331 INFO:tasks.rados.rados.0.smithi064.stdout:1628: done (3 left) 2024-03-04T06:14:26.331 INFO:tasks.rados.rados.0.smithi064.stdout:1629: write (excl) oid 45 current snap is 192 2024-03-04T06:14:26.331 INFO:tasks.rados.rados.0.smithi064.stdout:1629: seq_num 781 ranges {726267=637475,2119025=651272} 2024-03-04T06:14:26.338 INFO:tasks.rados.rados.0.smithi064.stdout:1629: writing smithi06431743-45 from 726267 to 1363742 tid 1 2024-03-04T06:14:26.346 INFO:tasks.rados.rados.0.smithi064.stdout:1629: writing smithi06431743-45 from 2119025 to 2770297 tid 2 2024-03-04T06:14:26.346 INFO:tasks.rados.rados.0.smithi064.stdout:1630: snap_remove snap 191 2024-03-04T06:14:26.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:26.439+0000 7fbd36806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:26.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:26.450+0000 7fbd36806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:26.469 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:26.471 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:26.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:14:26.620 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:26.615+0000 7faae4d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:26.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:26.622+0000 7faae4d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:26.635 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:26.635 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:26.635 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:26.635 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:26.636 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:26.636 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:26.636 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:26.636 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:26.636 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:26.636 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:26.636 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:26.636 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:26.639 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:26.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:14:26.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:26.742+0000 7fad846d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:26.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:26.749+0000 7fad846d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:26.764 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:26.764 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:26.765 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:26.766 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:26.766 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:14:26.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:26.869+0000 7fbd9379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:26.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:26.877+0000 7fbd9379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:26.891 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:26.891 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:26.891 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:26.891 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:26.891 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:26.891 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:26.891 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:26.891 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:26.891 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:26.891 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:26.891 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:26.892 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:26.894 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1630: done (4 left) 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1631: read oid 20 snap -1 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1631: expect (ObjNum 695 snap 170 seq_num 695) 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1632: snap_create 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1623: finishing write tid 2 to smithi06431743-13 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1622: finishing write tid 2 to smithi06431743-43 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1623: finishing write tid 3 to smithi06431743-13 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1622: finishing write tid 3 to smithi06431743-43 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1623: finishing write tid 4 to smithi06431743-13 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1623: finishing write tid 5 to smithi06431743-13 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1623: oid 13 updating version 0 to 625 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1623: oid 13 version 625 is already newer than 624 2024-03-04T06:14:27.339 INFO:tasks.rados.rados.0.smithi064.stdout:1623: oid 13 version 625 is already newer than 623 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:1623: oid 13 version 625 is already newer than 622 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 625 (ObjNum 779 snap 192 seq_num 779) dirty exists 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:1623: left oid 13 (ObjNum 779 snap 192 seq_num 779) 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:1622: finishing write tid 4 to smithi06431743-43 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:1622: finishing write tid 5 to smithi06431743-43 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:1622: oid 43 updating version 0 to 306 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:1622: oid 43 version 306 is already newer than 305 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:1622: oid 43 updating version 306 to 307 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:1622: oid 43 updating version 307 to 308 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 308 (ObjNum 778 snap 192 seq_num 778) dirty exists 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:1622: left oid 43 (ObjNum 778 snap 192 seq_num 778) 2024-03-04T06:14:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:1626: expect (ObjNum 697 snap 172 seq_num 697) 2024-03-04T06:14:27.478 INFO:tasks.rados.rados.0.smithi064.stdout:1629: finishing write tid 1 to smithi06431743-45 2024-03-04T06:14:27.479 INFO:tasks.rados.rados.0.smithi064.stdout:1629: finishing write tid 2 to smithi06431743-45 2024-03-04T06:14:27.479 INFO:tasks.rados.rados.0.smithi064.stdout:1629: finishing write tid 3 to smithi06431743-45 2024-03-04T06:14:27.479 INFO:tasks.rados.rados.0.smithi064.stdout:1629: finishing write tid 4 to smithi06431743-45 2024-03-04T06:14:27.479 INFO:tasks.rados.rados.0.smithi064.stdout:1629: oid 45 updating version 0 to 360 2024-03-04T06:14:27.479 INFO:tasks.rados.rados.0.smithi064.stdout:1629: oid 45 version 360 is already newer than 359 2024-03-04T06:14:27.479 INFO:tasks.rados.rados.0.smithi064.stdout:1629: oid 45 updating version 360 to 361 2024-03-04T06:14:27.479 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 361 (ObjNum 781 snap 192 seq_num 781) dirty exists 2024-03-04T06:14:27.479 INFO:tasks.rados.rados.0.smithi064.stdout:1629: left oid 45 (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:14:27.479 INFO:tasks.rados.rados.0.smithi064.stdout:1631: expect (ObjNum 695 snap 170 seq_num 695) 2024-03-04T06:14:28.225 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:28.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:28.331+0000 7f7b168dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:28.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:28.335+0000 7f7b168dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:28.338 INFO:tasks.rados.rados.0.smithi064.stdout:1622: done (5 left) 2024-03-04T06:14:28.338 INFO:tasks.rados.rados.0.smithi064.stdout:1623: done (4 left) 2024-03-04T06:14:28.338 INFO:tasks.rados.rados.0.smithi064.stdout:1626: done (3 left) 2024-03-04T06:14:28.338 INFO:tasks.rados.rados.0.smithi064.stdout:1629: done (2 left) 2024-03-04T06:14:28.338 INFO:tasks.rados.rados.0.smithi064.stdout:1631: done (1 left) 2024-03-04T06:14:28.338 INFO:tasks.rados.rados.0.smithi064.stdout:1632: done (0 left) 2024-03-04T06:14:28.338 INFO:tasks.rados.rados.0.smithi064.stdout:1633: snap_remove snap 183 2024-03-04T06:14:29.341 INFO:tasks.rados.rados.0.smithi064.stdout:1633: done (0 left) 2024-03-04T06:14:29.342 INFO:tasks.rados.rados.0.smithi064.stdout:1634: write (excl) oid 35 current snap is 193 2024-03-04T06:14:29.342 INFO:tasks.rados.rados.0.smithi064.stdout:1634: seq_num 782 ranges {743074=712724,2061154=794930} 2024-03-04T06:14:29.349 INFO:tasks.rados.rados.0.smithi064.stdout:1634: writing smithi06431743-35 from 743074 to 1455798 tid 1 2024-03-04T06:14:29.357 INFO:tasks.rados.rados.0.smithi064.stdout:1634: writing smithi06431743-35 from 2061154 to 2856084 tid 2 2024-03-04T06:14:29.357 INFO:tasks.rados.rados.0.smithi064.stdout:1635: write (excl) oid 42 current snap is 193 2024-03-04T06:14:29.357 INFO:tasks.rados.rados.0.smithi064.stdout:1635: seq_num 783 ranges {759881=787973,2119636=654941} 2024-03-04T06:14:29.365 INFO:tasks.rados.rados.0.smithi064.stdout:1635: writing smithi06431743-42 from 759881 to 1547854 tid 1 2024-03-04T06:14:29.374 INFO:tasks.rados.rados.0.smithi064.stdout:1635: writing smithi06431743-42 from 2119636 to 2774577 tid 2 2024-03-04T06:14:29.374 INFO:tasks.rados.rados.0.smithi064.stdout:1636: write (excl) oid 9 current snap is 193 2024-03-04T06:14:29.374 INFO:tasks.rados.rados.0.smithi064.stdout:1636: seq_num 784 ranges {776688=579575,1894471=798599} 2024-03-04T06:14:29.383 INFO:tasks.rados.rados.0.smithi064.stdout:1636: writing smithi06431743-9 from 776688 to 1356263 tid 1 2024-03-04T06:14:29.393 INFO:tasks.rados.rados.0.smithi064.stdout:1636: writing smithi06431743-9 from 1894471 to 2693070 tid 2 2024-03-04T06:14:29.393 INFO:tasks.rados.rados.0.smithi064.stdout:1637: snap_remove snap 192 2024-03-04T06:14:30.350 INFO:tasks.rados.rados.0.smithi064.stdout:1637: done (3 left) 2024-03-04T06:14:30.350 INFO:tasks.rados.rados.0.smithi064.stdout:1638: write oid 49 current snap is 193 2024-03-04T06:14:30.350 INFO:tasks.rados.rados.0.smithi064.stdout:1638: seq_num 785 ranges {793495=654824,1952953=658610,3157167=36328} 2024-03-04T06:14:30.357 INFO:tasks.rados.rados.0.smithi064.stdout:1638: writing smithi06431743-49 from 793495 to 1448319 tid 1 2024-03-04T06:14:30.365 INFO:tasks.rados.rados.0.smithi064.stdout:1638: writing smithi06431743-49 from 1952953 to 2611563 tid 2 2024-03-04T06:14:30.368 INFO:tasks.rados.rados.0.smithi064.stdout:1638: writing smithi06431743-49 from 3157167 to 3193495 tid 3 2024-03-04T06:14:30.368 INFO:tasks.rados.rados.0.smithi064.stdout:1639: snap_remove snap 184 2024-03-04T06:14:30.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:30.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:30.534+0000 7f7ef84dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:30.538 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:30.538+0000 7f7ef84dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:1639: done (4 left) 2024-03-04T06:14:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:1640: read oid 7 snap -1 2024-03-04T06:14:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:1640: expect (ObjNum 722 snap 179 seq_num 722) 2024-03-04T06:14:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:1641: write oid 3 current snap is 193 2024-03-04T06:14:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:1634: finishing write tid 1 to smithi06431743-35 2024-03-04T06:14:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:1641: seq_num 786 ranges {410302=730073,1895082=402268,2951884=258418} 2024-03-04T06:14:31.362 INFO:tasks.rados.rados.0.smithi064.stdout:1641: writing smithi06431743-3 from 410302 to 1140375 tid 1 2024-03-04T06:14:31.368 INFO:tasks.rados.rados.0.smithi064.stdout:1641: writing smithi06431743-3 from 1895082 to 2297350 tid 2 2024-03-04T06:14:31.374 INFO:tasks.rados.rados.0.smithi064.stdout:1641: writing smithi06431743-3 from 2951884 to 3210302 tid 3 2024-03-04T06:14:31.374 INFO:tasks.rados.rados.0.smithi064.stdout:1642: write (excl) oid 30 current snap is 193 2024-03-04T06:14:31.374 INFO:tasks.rados.rados.0.smithi064.stdout:1642: seq_num 787 ranges {427109=405322,1553564=545926,2579307=647802} 2024-03-04T06:14:31.379 INFO:tasks.rados.rados.0.smithi064.stdout:1642: writing smithi06431743-30 from 427109 to 832431 tid 1 2024-03-04T06:14:31.387 INFO:tasks.rados.rados.0.smithi064.stdout:1642: writing smithi06431743-30 from 1553564 to 2099490 tid 2 2024-03-04T06:14:31.400 INFO:tasks.rados.rados.0.smithi064.stdout:1642: writing smithi06431743-30 from 2579307 to 3227109 tid 3 2024-03-04T06:14:31.400 INFO:tasks.rados.rados.0.smithi064.stdout:1643: write (excl) oid 8 current snap is 193 2024-03-04T06:14:31.400 INFO:tasks.rados.rados.0.smithi064.stdout:1643: seq_num 788 ranges {443916=480571,1612046=405937,2606730=495052} 2024-03-04T06:14:31.406 INFO:tasks.rados.rados.0.smithi064.stdout:1643: writing smithi06431743-8 from 443916 to 924487 tid 1 2024-03-04T06:14:31.412 INFO:tasks.rados.rados.0.smithi064.stdout:1643: writing smithi06431743-8 from 1612046 to 2017983 tid 2 2024-03-04T06:14:31.420 INFO:tasks.rados.rados.0.smithi064.stdout:1643: writing smithi06431743-8 from 2606730 to 3101782 tid 3 2024-03-04T06:14:31.420 INFO:tasks.rados.rados.0.smithi064.stdout:1644: read oid 21 snap -1 2024-03-04T06:14:31.420 INFO:tasks.rados.rados.0.smithi064.stdout:1644: expect (ObjNum 708 snap 177 seq_num 708) 2024-03-04T06:14:31.421 INFO:tasks.rados.rados.0.smithi064.stdout:1645: snap_create 2024-03-04T06:14:31.421 INFO:tasks.rados.rados.0.smithi064.stdout:1636: finishing write tid 1 to smithi06431743-9 2024-03-04T06:14:31.421 INFO:tasks.rados.rados.0.smithi064.stdout:1634: finishing write tid 2 to smithi06431743-35 2024-03-04T06:14:31.421 INFO:tasks.rados.rados.0.smithi064.stdout:1636: finishing write tid 2 to smithi06431743-9 2024-03-04T06:14:31.421 INFO:tasks.rados.rados.0.smithi064.stdout:1634: finishing write tid 3 to smithi06431743-35 2024-03-04T06:14:31.421 INFO:tasks.rados.rados.0.smithi064.stdout:1634: finishing write tid 4 to smithi06431743-35 2024-03-04T06:14:31.421 INFO:tasks.rados.rados.0.smithi064.stdout:1634: oid 35 updating version 0 to 407 2024-03-04T06:14:31.421 INFO:tasks.rados.rados.0.smithi064.stdout:1634: oid 35 version 407 is already newer than 407 2024-03-04T06:14:31.421 INFO:tasks.rados.rados.0.smithi064.stdout:1634: oid 35 version 407 is already newer than 405 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 407 (ObjNum 782 snap 193 seq_num 782) dirty exists 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1634: left oid 35 (ObjNum 782 snap 193 seq_num 782) 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1635: finishing write tid 1 to smithi06431743-42 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1636: finishing write tid 3 to smithi06431743-9 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1636: finishing write tid 4 to smithi06431743-9 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1636: oid 9 updating version 0 to 544 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1636: oid 9 version 544 is already newer than 543 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1636: oid 9 updating version 544 to 545 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 545 (ObjNum 784 snap 193 seq_num 784) dirty exists 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1636: left oid 9 (ObjNum 784 snap 193 seq_num 784) 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1635: finishing write tid 2 to smithi06431743-42 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1635: finishing write tid 3 to smithi06431743-42 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1635: finishing write tid 4 to smithi06431743-42 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1635: oid 42 updating version 0 to 335 2024-03-04T06:14:31.422 INFO:tasks.rados.rados.0.smithi064.stdout:1635: oid 42 updating version 335 to 336 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1635: oid 42 version 336 is already newer than 334 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 336 (ObjNum 783 snap 193 seq_num 783) dirty exists 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1635: left oid 42 (ObjNum 783 snap 193 seq_num 783) 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1638: finishing write tid 1 to smithi06431743-49 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1638: finishing write tid 2 to smithi06431743-49 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1638: finishing write tid 3 to smithi06431743-49 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1638: finishing write tid 4 to smithi06431743-49 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1638: finishing write tid 5 to smithi06431743-49 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1638: oid 49 updating version 0 to 222 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1638: oid 49 version 222 is already newer than 220 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1638: oid 49 version 222 is already newer than 222 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1638: oid 49 updating version 222 to 223 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 223 (ObjNum 785 snap 193 seq_num 785) dirty exists 2024-03-04T06:14:31.423 INFO:tasks.rados.rados.0.smithi064.stdout:1638: left oid 49 (ObjNum 785 snap 193 seq_num 785) 2024-03-04T06:14:31.442 INFO:tasks.rados.rados.0.smithi064.stdout:1640: expect (ObjNum 722 snap 179 seq_num 722) 2024-03-04T06:14:31.559 INFO:tasks.rados.rados.0.smithi064.stdout:1644: expect (ObjNum 708 snap 177 seq_num 708) 2024-03-04T06:14:31.764 INFO:tasks.rados.rados.0.smithi064.stdout:1642: finishing write tid 1 to smithi06431743-30 2024-03-04T06:14:31.764 INFO:tasks.rados.rados.0.smithi064.stdout:1643: finishing write tid 1 to smithi06431743-8 2024-03-04T06:14:31.764 INFO:tasks.rados.rados.0.smithi064.stdout:1641: finishing write tid 1 to smithi06431743-3 2024-03-04T06:14:31.764 INFO:tasks.rados.rados.0.smithi064.stdout:1642: finishing write tid 2 to smithi06431743-30 2024-03-04T06:14:31.764 INFO:tasks.rados.rados.0.smithi064.stdout:1641: finishing write tid 2 to smithi06431743-3 2024-03-04T06:14:31.764 INFO:tasks.rados.rados.0.smithi064.stdout:1642: finishing write tid 3 to smithi06431743-30 2024-03-04T06:14:31.764 INFO:tasks.rados.rados.0.smithi064.stdout:1643: finishing write tid 2 to smithi06431743-8 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1642: finishing write tid 4 to smithi06431743-30 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1641: finishing write tid 3 to smithi06431743-3 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1642: finishing write tid 5 to smithi06431743-30 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1642: oid 30 updating version 0 to 651 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1642: oid 30 version 651 is already newer than 648 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1642: oid 30 version 651 is already newer than 649 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1642: oid 30 version 651 is already newer than 650 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 651 (ObjNum 787 snap 193 seq_num 787) dirty exists 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1642: left oid 30 (ObjNum 787 snap 193 seq_num 787) 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1641: finishing write tid 4 to smithi06431743-3 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1641: finishing write tid 5 to smithi06431743-3 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1641: oid 3 updating version 0 to 360 2024-03-04T06:14:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:1641: oid 3 updating version 360 to 361 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:1641: oid 3 version 361 is already newer than 359 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:1641: oid 3 version 361 is already newer than 358 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 361 (ObjNum 786 snap 193 seq_num 786) dirty exists 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:1641: left oid 3 (ObjNum 786 snap 193 seq_num 786) 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:1643: finishing write tid 3 to smithi06431743-8 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:1643: finishing write tid 4 to smithi06431743-8 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:1643: finishing write tid 5 to smithi06431743-8 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:1643: oid 8 updating version 0 to 305 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:1643: oid 8 version 305 is already newer than 303 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:1643: oid 8 version 305 is already newer than 304 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:1643: oid 8 updating version 305 to 306 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 306 (ObjNum 788 snap 193 seq_num 788) dirty exists 2024-03-04T06:14:31.766 INFO:tasks.rados.rados.0.smithi064.stdout:1643: left oid 8 (ObjNum 788 snap 193 seq_num 788) 2024-03-04T06:14:31.894 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:14:31.895 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:14:31.895 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:14:31.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:14:32.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:31.999+0000 7f79d4d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:32.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:32.006+0000 7f79d4d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:32.018 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:32.019 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:32.019 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:32.019 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:32.019 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:32.019 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:32.019 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:32.019 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:32.019 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:32.019 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:32.019 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:32.019 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:32.022 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:32.022 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:14:32.130 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:32.125+0000 7faf85354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:32.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:32.132+0000 7faf85354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:32.145 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:32.145 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:32.145 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:32.145 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:32.145 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:32.145 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:32.145 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:32.146 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:32.146 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:32.146 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:32.146 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:32.146 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:32.148 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:32.148 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:32.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:14:32.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:32.251+0000 7f4ce899d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:32.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:32.259+0000 7f4ce899d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:32.272 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:32.272 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:32.272 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:32.272 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:32.272 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:32.272 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:32.272 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:32.275 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:32.276 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:14:32.355 INFO:tasks.rados.rados.0.smithi064.stdout:1634: done (9 left) 2024-03-04T06:14:32.355 INFO:tasks.rados.rados.0.smithi064.stdout:1635: done (8 left) 2024-03-04T06:14:32.355 INFO:tasks.rados.rados.0.smithi064.stdout:1636: done (7 left) 2024-03-04T06:14:32.356 INFO:tasks.rados.rados.0.smithi064.stdout:1638: done (6 left) 2024-03-04T06:14:32.356 INFO:tasks.rados.rados.0.smithi064.stdout:1640: done (5 left) 2024-03-04T06:14:32.356 INFO:tasks.rados.rados.0.smithi064.stdout:1641: done (4 left) 2024-03-04T06:14:32.356 INFO:tasks.rados.rados.0.smithi064.stdout:1642: done (3 left) 2024-03-04T06:14:32.356 INFO:tasks.rados.rados.0.smithi064.stdout:1643: done (2 left) 2024-03-04T06:14:32.356 INFO:tasks.rados.rados.0.smithi064.stdout:1644: done (1 left) 2024-03-04T06:14:32.356 INFO:tasks.rados.rados.0.smithi064.stdout:1645: done (0 left) 2024-03-04T06:14:32.356 INFO:tasks.rados.rados.0.smithi064.stdout:1646: read oid 26 snap -1 2024-03-04T06:14:32.356 INFO:tasks.rados.rados.0.smithi064.stdout:1646: expect (ObjNum 729 snap 179 seq_num 729) 2024-03-04T06:14:32.356 INFO:tasks.rados.rados.0.smithi064.stdout:1647: snap_create 2024-03-04T06:14:32.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:32.382+0000 7efee6b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:32.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:32.389+0000 7efee6b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:32.404 INFO:tasks.rados.rados.0.smithi064.stdout:1646: expect (ObjNum 729 snap 179 seq_num 729) 2024-03-04T06:14:32.411 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:32.411 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:32.411 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:32.411 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:32.411 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:32.411 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:32.411 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:32.411 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:32.411 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:32.412 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:32.412 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:32.412 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:32.413 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:32.414 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:32.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:32.711 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:32.708+0000 7f5cb186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:32.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:32.712+0000 7f5cb186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:33.238 INFO:tasks.rados.rados.0.smithi064.stdout:1646: done (1 left) 2024-03-04T06:14:33.238 INFO:tasks.rados.rados.0.smithi064.stdout:1647: done (0 left) 2024-03-04T06:14:33.238 INFO:tasks.rados.rados.0.smithi064.stdout:1648: read oid 19 snap -1 2024-03-04T06:14:33.238 INFO:tasks.rados.rados.0.smithi064.stdout:1648: expect (ObjNum 773 snap 191 seq_num 773) 2024-03-04T06:14:33.238 INFO:tasks.rados.rados.0.smithi064.stdout:1649: read oid 36 snap -1 2024-03-04T06:14:33.238 INFO:tasks.rados.rados.0.smithi064.stdout:1649: expect (ObjNum 767 snap 190 seq_num 767) 2024-03-04T06:14:33.238 INFO:tasks.rados.rados.0.smithi064.stdout:1650: write (excl) oid 25 current snap is 195 2024-03-04T06:14:33.238 INFO:tasks.rados.rados.0.smithi064.stdout:1650: seq_num 789 ranges {460723=555820,1670528=549595,2634153=626570} 2024-03-04T06:14:33.243 INFO:tasks.rados.rados.0.smithi064.stdout:1650: writing smithi06431743-25 from 460723 to 1016543 tid 1 2024-03-04T06:14:33.250 INFO:tasks.rados.rados.0.smithi064.stdout:1650: writing smithi06431743-25 from 1670528 to 2220123 tid 2 2024-03-04T06:14:33.256 INFO:tasks.rados.rados.0.smithi064.stdout:1650: writing smithi06431743-25 from 2634153 to 3260723 tid 3 2024-03-04T06:14:33.256 INFO:tasks.rados.rados.0.smithi064.stdout:1651: read oid 45 snap -1 2024-03-04T06:14:33.256 INFO:tasks.rados.rados.0.smithi064.stdout:1651: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:14:33.256 INFO:tasks.rados.rados.0.smithi064.stdout:1652: snap_create 2024-03-04T06:14:33.287 INFO:tasks.rados.rados.0.smithi064.stdout:1649: expect (ObjNum 767 snap 190 seq_num 767) 2024-03-04T06:14:33.433 INFO:tasks.rados.rados.0.smithi064.stdout:1648: expect (ObjNum 773 snap 191 seq_num 773) 2024-03-04T06:14:33.582 INFO:tasks.rados.rados.0.smithi064.stdout:1650: finishing write tid 1 to smithi06431743-25 2024-03-04T06:14:33.583 INFO:tasks.rados.rados.0.smithi064.stdout:1650: finishing write tid 2 to smithi06431743-25 2024-03-04T06:14:33.583 INFO:tasks.rados.rados.0.smithi064.stdout:1650: finishing write tid 3 to smithi06431743-25 2024-03-04T06:14:33.583 INFO:tasks.rados.rados.0.smithi064.stdout:1650: finishing write tid 4 to smithi06431743-25 2024-03-04T06:14:33.583 INFO:tasks.rados.rados.0.smithi064.stdout:1650: finishing write tid 5 to smithi06431743-25 2024-03-04T06:14:33.583 INFO:tasks.rados.rados.0.smithi064.stdout:1650: oid 25 updating version 0 to 371 2024-03-04T06:14:33.583 INFO:tasks.rados.rados.0.smithi064.stdout:1650: oid 25 version 371 is already newer than 370 2024-03-04T06:14:33.583 INFO:tasks.rados.rados.0.smithi064.stdout:1650: oid 25 version 371 is already newer than 369 2024-03-04T06:14:33.583 INFO:tasks.rados.rados.0.smithi064.stdout:1650: oid 25 updating version 371 to 372 2024-03-04T06:14:33.583 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 372 (ObjNum 789 snap 195 seq_num 789) dirty exists 2024-03-04T06:14:33.583 INFO:tasks.rados.rados.0.smithi064.stdout:1650: left oid 25 (ObjNum 789 snap 195 seq_num 789) 2024-03-04T06:14:33.583 INFO:tasks.rados.rados.0.smithi064.stdout:1651: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:14:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:1648: done (4 left) 2024-03-04T06:14:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:1649: done (3 left) 2024-03-04T06:14:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:1650: done (2 left) 2024-03-04T06:14:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:1651: done (1 left) 2024-03-04T06:14:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:1652: done (0 left) 2024-03-04T06:14:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:1653: read oid 33 snap -1 2024-03-04T06:14:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:1653: expect deleted 2024-03-04T06:14:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:1654: read oid 43 snap 193 2024-03-04T06:14:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:1654: expect (ObjNum 778 snap 192 seq_num 778) 2024-03-04T06:14:34.239 INFO:tasks.rados.rados.0.smithi064.stdout:1655: delete oid 38 current snap is 196 2024-03-04T06:14:34.284 INFO:tasks.rados.rados.0.smithi064.stdout:1653: done (2 left) 2024-03-04T06:14:34.284 INFO:tasks.rados.rados.0.smithi064.stdout:1655: done (1 left) 2024-03-04T06:14:34.284 INFO:tasks.rados.rados.0.smithi064.stdout:1656: read oid 50 snap 193 2024-03-04T06:14:34.284 INFO:tasks.rados.rados.0.smithi064.stdout:1656: expect (ObjNum 753 snap 188 seq_num 753) 2024-03-04T06:14:34.284 INFO:tasks.rados.rados.0.smithi064.stdout:1657: write (excl) oid 28 current snap is 196 2024-03-04T06:14:34.284 INFO:tasks.rados.rados.0.smithi064.stdout:1657: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2024-03-04T06:14:34.288 INFO:tasks.rados.rados.0.smithi064.stdout:1657: writing smithi06431743-28 from 494337 to 917008 tid 1 2024-03-04T06:14:34.294 INFO:tasks.rados.rados.0.smithi064.stdout:1657: writing smithi06431743-28 from 1387492 to 1940756 tid 2 2024-03-04T06:14:34.301 INFO:tasks.rados.rados.0.smithi064.stdout:1657: writing smithi06431743-28 from 2688999 to 3134220 tid 3 2024-03-04T06:14:34.301 INFO:tasks.rados.rados.0.smithi064.stdout:1658: snap_remove snap 190 2024-03-04T06:14:34.787 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:34.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:34.891+0000 7f44e766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:34.896 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:34.895+0000 7f44e766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:35.241 INFO:tasks.rados.rados.0.smithi064.stdout:1658: done (3 left) 2024-03-04T06:14:35.241 INFO:tasks.rados.rados.0.smithi064.stdout:1659: delete oid 22 current snap is 196 2024-03-04T06:14:35.241 INFO:tasks.rados.rados.0.smithi064.stdout:1654: expect (ObjNum 778 snap 192 seq_num 778) 2024-03-04T06:14:35.458 INFO:tasks.rados.rados.0.smithi064.stdout:1656: expect (ObjNum 753 snap 188 seq_num 753) 2024-03-04T06:14:35.586 INFO:tasks.rados.rados.0.smithi064.stdout:1657: finishing write tid 1 to smithi06431743-28 2024-03-04T06:14:35.586 INFO:tasks.rados.rados.0.smithi064.stdout:1657: finishing write tid 2 to smithi06431743-28 2024-03-04T06:14:35.586 INFO:tasks.rados.rados.0.smithi064.stdout:1657: finishing write tid 3 to smithi06431743-28 2024-03-04T06:14:35.586 INFO:tasks.rados.rados.0.smithi064.stdout:1654: done (3 left) 2024-03-04T06:14:35.586 INFO:tasks.rados.rados.0.smithi064.stdout:1656: done (2 left) 2024-03-04T06:14:35.586 INFO:tasks.rados.rados.0.smithi064.stdout:1659: done (1 left) 2024-03-04T06:14:35.586 INFO:tasks.rados.rados.0.smithi064.stdout:1660: write oid 41 current snap is 196 2024-03-04T06:14:35.586 INFO:tasks.rados.rados.0.smithi064.stdout:1657: finishing write tid 4 to smithi06431743-28 2024-03-04T06:14:35.586 INFO:tasks.rados.rados.0.smithi064.stdout:1660: seq_num 793 ranges {527951=573169,1504456=556933,2743845=467765} 2024-03-04T06:14:35.591 INFO:tasks.rados.rados.0.smithi064.stdout:1660: writing smithi06431743-41 from 527951 to 1101120 tid 1 2024-03-04T06:14:35.597 INFO:tasks.rados.rados.0.smithi064.stdout:1660: writing smithi06431743-41 from 1504456 to 2061389 tid 2 2024-03-04T06:14:35.604 INFO:tasks.rados.rados.0.smithi064.stdout:1660: writing smithi06431743-41 from 2743845 to 3211610 tid 3 2024-03-04T06:14:35.604 INFO:tasks.rados.rados.0.smithi064.stdout:1661: read oid 23 snap -1 2024-03-04T06:14:35.604 INFO:tasks.rados.rados.0.smithi064.stdout:1661: expect deleted 2024-03-04T06:14:35.604 INFO:tasks.rados.rados.0.smithi064.stdout:1662: snap_create 2024-03-04T06:14:35.604 INFO:tasks.rados.rados.0.smithi064.stdout:1657: finishing write tid 5 to smithi06431743-28 2024-03-04T06:14:35.605 INFO:tasks.rados.rados.0.smithi064.stdout:1657: oid 28 updating version 0 to 663 2024-03-04T06:14:35.605 INFO:tasks.rados.rados.0.smithi064.stdout:1657: oid 28 version 663 is already newer than 662 2024-03-04T06:14:35.605 INFO:tasks.rados.rados.0.smithi064.stdout:1657: oid 28 updating version 663 to 665 2024-03-04T06:14:35.605 INFO:tasks.rados.rados.0.smithi064.stdout:1657: oid 28 version 665 is already newer than 664 2024-03-04T06:14:35.605 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 665 (ObjNum 791 snap 196 seq_num 791) dirty exists 2024-03-04T06:14:35.605 INFO:tasks.rados.rados.0.smithi064.stdout:1657: left oid 28 (ObjNum 791 snap 196 seq_num 791) 2024-03-04T06:14:35.662 INFO:tasks.rados.rados.0.smithi064.stdout:1660: finishing write tid 1 to smithi06431743-41 2024-03-04T06:14:35.676 INFO:tasks.rados.rados.0.smithi064.stdout:1660: finishing write tid 2 to smithi06431743-41 2024-03-04T06:14:35.684 INFO:tasks.rados.rados.0.smithi064.stdout:1660: finishing write tid 3 to smithi06431743-41 2024-03-04T06:14:35.692 INFO:tasks.rados.rados.0.smithi064.stdout:1660: finishing write tid 4 to smithi06431743-41 2024-03-04T06:14:35.693 INFO:tasks.rados.rados.0.smithi064.stdout:1660: finishing write tid 5 to smithi06431743-41 2024-03-04T06:14:35.693 INFO:tasks.rados.rados.0.smithi064.stdout:1660: oid 41 updating version 0 to 367 2024-03-04T06:14:35.693 INFO:tasks.rados.rados.0.smithi064.stdout:1660: oid 41 version 367 is already newer than 365 2024-03-04T06:14:35.693 INFO:tasks.rados.rados.0.smithi064.stdout:1660: oid 41 version 367 is already newer than 366 2024-03-04T06:14:35.693 INFO:tasks.rados.rados.0.smithi064.stdout:1660: oid 41 updating version 367 to 368 2024-03-04T06:14:35.693 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 368 (ObjNum 793 snap 196 seq_num 793) dirty exists 2024-03-04T06:14:35.693 INFO:tasks.rados.rados.0.smithi064.stdout:1660: left oid 41 (ObjNum 793 snap 196 seq_num 793) 2024-03-04T06:14:36.242 INFO:tasks.rados.rados.0.smithi064.stdout:1657: done (3 left) 2024-03-04T06:14:36.242 INFO:tasks.rados.rados.0.smithi064.stdout:1660: done (2 left) 2024-03-04T06:14:36.242 INFO:tasks.rados.rados.0.smithi064.stdout:1661: done (1 left) 2024-03-04T06:14:36.242 INFO:tasks.rados.rados.0.smithi064.stdout:1662: done (0 left) 2024-03-04T06:14:36.242 INFO:tasks.rados.rados.0.smithi064.stdout:1663: read oid 1 snap -1 2024-03-04T06:14:36.243 INFO:tasks.rados.rados.0.smithi064.stdout:1663: expect (ObjNum 770 snap 190 seq_num 770) 2024-03-04T06:14:36.243 INFO:tasks.rados.rados.0.smithi064.stdout:1664: write (excl) oid 31 current snap is 197 2024-03-04T06:14:36.243 INFO:tasks.rados.rados.0.smithi064.stdout:1664: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2024-03-04T06:14:36.248 INFO:tasks.rados.rados.0.smithi064.stdout:1664: writing smithi06431743-31 from 544758 to 1193176 tid 1 2024-03-04T06:14:36.254 INFO:tasks.rados.rados.0.smithi064.stdout:1664: writing smithi06431743-31 from 1846585 to 2263529 tid 2 2024-03-04T06:14:36.260 INFO:tasks.rados.rados.0.smithi064.stdout:1664: writing smithi06431743-31 from 2771268 to 3344758 tid 3 2024-03-04T06:14:36.260 INFO:tasks.rados.rados.0.smithi064.stdout:1665: write oid 10 current snap is 197 2024-03-04T06:14:36.260 INFO:tasks.rados.rados.0.smithi064.stdout:1665: seq_num 795 ranges {561565=723667,1905067=560602,3082338=279227} 2024-03-04T06:14:36.267 INFO:tasks.rados.rados.0.smithi064.stdout:1665: writing smithi06431743-10 from 561565 to 1285232 tid 1 2024-03-04T06:14:36.274 INFO:tasks.rados.rados.0.smithi064.stdout:1665: writing smithi06431743-10 from 1905067 to 2465669 tid 2 2024-03-04T06:14:36.280 INFO:tasks.rados.rados.0.smithi064.stdout:1665: writing smithi06431743-10 from 3082338 to 3361565 tid 3 2024-03-04T06:14:36.280 INFO:tasks.rados.rados.0.smithi064.stdout:1666: write oid 50 current snap is 197 2024-03-04T06:14:36.280 INFO:tasks.rados.rados.0.smithi064.stdout:1666: seq_num 796 ranges {578372=798916,1963549=420613,2826114=417934} 2024-03-04T06:14:36.289 INFO:tasks.rados.rados.0.smithi064.stdout:1666: writing smithi06431743-50 from 578372 to 1377288 tid 1 2024-03-04T06:14:36.296 INFO:tasks.rados.rados.0.smithi064.stdout:1666: writing smithi06431743-50 from 1963549 to 2384162 tid 2 2024-03-04T06:14:36.302 INFO:tasks.rados.rados.0.smithi064.stdout:1666: writing smithi06431743-50 from 2826114 to 3244048 tid 3 2024-03-04T06:14:36.302 INFO:tasks.rados.rados.0.smithi064.stdout:1667: write oid 38 current snap is 197 2024-03-04T06:14:36.302 INFO:tasks.rados.rados.0.smithi064.stdout:1667: seq_num 797 ranges {595179=474165,1622031=564271,2737184=629206} 2024-03-04T06:14:36.309 INFO:tasks.rados.rados.0.smithi064.stdout:1667: writing smithi06431743-38 from 595179 to 1069344 tid 1 2024-03-04T06:14:36.315 INFO:tasks.rados.rados.0.smithi064.stdout:1667: writing smithi06431743-38 from 1622031 to 2186302 tid 2 2024-03-04T06:14:36.322 INFO:tasks.rados.rados.0.smithi064.stdout:1667: writing smithi06431743-38 from 2737184 to 3366390 tid 3 2024-03-04T06:14:36.322 INFO:tasks.rados.rados.0.smithi064.stdout:1668: read oid 37 snap 193 2024-03-04T06:14:36.322 INFO:tasks.rados.rados.0.smithi064.stdout:1668: expect (ObjNum 752 snap 186 seq_num 752) 2024-03-04T06:14:36.323 INFO:tasks.rados.rados.0.smithi064.stdout:1669: write oid 28 current snap is 197 2024-03-04T06:14:36.323 INFO:tasks.rados.rados.0.smithi064.stdout:1669: seq_num 798 ranges {611986=549414,1564160=424282,2764607=440478} 2024-03-04T06:14:36.329 INFO:tasks.rados.rados.0.smithi064.stdout:1669: writing smithi06431743-28 from 611986 to 1161400 tid 1 2024-03-04T06:14:36.334 INFO:tasks.rados.rados.0.smithi064.stdout:1669: writing smithi06431743-28 from 1564160 to 1988442 tid 2 2024-03-04T06:14:36.341 INFO:tasks.rados.rados.0.smithi064.stdout:1669: writing smithi06431743-28 from 2764607 to 3205085 tid 3 2024-03-04T06:14:36.341 INFO:tasks.rados.rados.0.smithi064.stdout:1670: read oid 5 snap -1 2024-03-04T06:14:36.341 INFO:tasks.rados.rados.0.smithi064.stdout:1670: expect (ObjNum 724 snap 179 seq_num 724) 2024-03-04T06:14:36.341 INFO:tasks.rados.rados.0.smithi064.stdout:1671: read oid 13 snap -1 2024-03-04T06:14:36.341 INFO:tasks.rados.rados.0.smithi064.stdout:1671: expect (ObjNum 779 snap 192 seq_num 779) 2024-03-04T06:14:36.341 INFO:tasks.rados.rados.0.smithi064.stdout:1672: snap_create 2024-03-04T06:14:36.341 INFO:tasks.rados.rados.0.smithi064.stdout:1663: expect (ObjNum 770 snap 190 seq_num 770) 2024-03-04T06:14:36.506 INFO:tasks.rados.rados.0.smithi064.stdout:1664: finishing write tid 1 to smithi06431743-31 2024-03-04T06:14:36.506 INFO:tasks.rados.rados.0.smithi064.stdout:1665: finishing write tid 1 to smithi06431743-10 2024-03-04T06:14:36.506 INFO:tasks.rados.rados.0.smithi064.stdout:1669: finishing write tid 1 to smithi06431743-28 2024-03-04T06:14:36.506 INFO:tasks.rados.rados.0.smithi064.stdout:1666: finishing write tid 1 to smithi06431743-50 2024-03-04T06:14:36.506 INFO:tasks.rados.rados.0.smithi064.stdout:1668: expect (ObjNum 752 snap 186 seq_num 752) 2024-03-04T06:14:36.643 INFO:tasks.rados.rados.0.smithi064.stdout:1664: finishing write tid 2 to smithi06431743-31 2024-03-04T06:14:36.643 INFO:tasks.rados.rados.0.smithi064.stdout:1665: finishing write tid 2 to smithi06431743-10 2024-03-04T06:14:36.643 INFO:tasks.rados.rados.0.smithi064.stdout:1670: expect (ObjNum 724 snap 179 seq_num 724) 2024-03-04T06:14:36.752 INFO:tasks.rados.rados.0.smithi064.stdout:1664: finishing write tid 3 to smithi06431743-31 2024-03-04T06:14:36.752 INFO:tasks.rados.rados.0.smithi064.stdout:1666: finishing write tid 2 to smithi06431743-50 2024-03-04T06:14:36.752 INFO:tasks.rados.rados.0.smithi064.stdout:1669: finishing write tid 2 to smithi06431743-28 2024-03-04T06:14:36.752 INFO:tasks.rados.rados.0.smithi064.stdout:1665: finishing write tid 3 to smithi06431743-10 2024-03-04T06:14:36.753 INFO:tasks.rados.rados.0.smithi064.stdout:1664: finishing write tid 4 to smithi06431743-31 2024-03-04T06:14:36.753 INFO:tasks.rados.rados.0.smithi064.stdout:1671: expect (ObjNum 779 snap 192 seq_num 779) 2024-03-04T06:14:36.869 INFO:tasks.rados.rados.0.smithi064.stdout:1664: finishing write tid 5 to smithi06431743-31 2024-03-04T06:14:36.869 INFO:tasks.rados.rados.0.smithi064.stdout:1664: oid 31 updating version 0 to 138 2024-03-04T06:14:36.869 INFO:tasks.rados.rados.0.smithi064.stdout:1664: oid 31 updating version 138 to 139 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1664: oid 31 updating version 139 to 140 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1664: oid 31 updating version 140 to 141 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 141 (ObjNum 794 snap 197 seq_num 794) dirty exists 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1664: left oid 31 (ObjNum 794 snap 197 seq_num 794) 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1666: finishing write tid 3 to smithi06431743-50 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1665: finishing write tid 4 to smithi06431743-10 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1669: finishing write tid 3 to smithi06431743-28 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1665: finishing write tid 5 to smithi06431743-10 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1665: oid 10 updating version 0 to 341 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1665: oid 10 version 341 is already newer than 339 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1665: oid 10 updating version 341 to 342 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1665: oid 10 version 342 is already newer than 340 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 342 (ObjNum 795 snap 197 seq_num 795) dirty exists 2024-03-04T06:14:36.870 INFO:tasks.rados.rados.0.smithi064.stdout:1665: left oid 10 (ObjNum 795 snap 197 seq_num 795) 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1666: finishing write tid 4 to smithi06431743-50 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1666: finishing write tid 5 to smithi06431743-50 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1666: oid 50 updating version 0 to 557 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1666: oid 50 updating version 557 to 560 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1666: oid 50 version 560 is already newer than 559 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1666: oid 50 version 560 is already newer than 559 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 560 (ObjNum 796 snap 197 seq_num 796) dirty exists 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1666: left oid 50 (ObjNum 796 snap 197 seq_num 796) 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1669: finishing write tid 4 to smithi06431743-28 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1669: finishing write tid 5 to smithi06431743-28 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1669: oid 28 updating version 0 to 667 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1669: oid 28 updating version 667 to 670 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1669: oid 28 version 670 is already newer than 668 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1669: oid 28 version 670 is already newer than 669 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 670 (ObjNum 798 snap 197 seq_num 798) dirty exists 2024-03-04T06:14:36.871 INFO:tasks.rados.rados.0.smithi064.stdout:1669: left oid 28 (ObjNum 798 snap 197 seq_num 798) 2024-03-04T06:14:36.872 INFO:tasks.rados.rados.0.smithi064.stdout:1667: finishing write tid 1 to smithi06431743-38 2024-03-04T06:14:36.872 INFO:tasks.rados.rados.0.smithi064.stdout:1667: finishing write tid 2 to smithi06431743-38 2024-03-04T06:14:36.872 INFO:tasks.rados.rados.0.smithi064.stdout:1667: finishing write tid 3 to smithi06431743-38 2024-03-04T06:14:36.872 INFO:tasks.rados.rados.0.smithi064.stdout:1667: finishing write tid 4 to smithi06431743-38 2024-03-04T06:14:36.872 INFO:tasks.rados.rados.0.smithi064.stdout:1667: finishing write tid 5 to smithi06431743-38 2024-03-04T06:14:36.872 INFO:tasks.rados.rados.0.smithi064.stdout:1667: oid 38 updating version 0 to 346 2024-03-04T06:14:36.872 INFO:tasks.rados.rados.0.smithi064.stdout:1667: oid 38 version 346 is already newer than 346 2024-03-04T06:14:36.872 INFO:tasks.rados.rados.0.smithi064.stdout:1667: oid 38 version 346 is already newer than 346 2024-03-04T06:14:36.872 INFO:tasks.rados.rados.0.smithi064.stdout:1667: oid 38 version 346 is already newer than 346 2024-03-04T06:14:36.872 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 346 (ObjNum 797 snap 197 seq_num 797) dirty exists 2024-03-04T06:14:36.872 INFO:tasks.rados.rados.0.smithi064.stdout:1667: left oid 38 (ObjNum 797 snap 197 seq_num 797) 2024-03-04T06:14:36.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:37.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:37.068+0000 7f4115d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:37.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:37.072+0000 7f4115d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:37.246 INFO:tasks.rados.rados.0.smithi064.stdout:1663: done (9 left) 2024-03-04T06:14:37.247 INFO:tasks.rados.rados.0.smithi064.stdout:1664: done (8 left) 2024-03-04T06:14:37.247 INFO:tasks.rados.rados.0.smithi064.stdout:1665: done (7 left) 2024-03-04T06:14:37.247 INFO:tasks.rados.rados.0.smithi064.stdout:1666: done (6 left) 2024-03-04T06:14:37.247 INFO:tasks.rados.rados.0.smithi064.stdout:1667: done (5 left) 2024-03-04T06:14:37.247 INFO:tasks.rados.rados.0.smithi064.stdout:1668: done (4 left) 2024-03-04T06:14:37.247 INFO:tasks.rados.rados.0.smithi064.stdout:1669: done (3 left) 2024-03-04T06:14:37.247 INFO:tasks.rados.rados.0.smithi064.stdout:1670: done (2 left) 2024-03-04T06:14:37.247 INFO:tasks.rados.rados.0.smithi064.stdout:1671: done (1 left) 2024-03-04T06:14:37.247 INFO:tasks.rados.rados.0.smithi064.stdout:1672: done (0 left) 2024-03-04T06:14:37.247 INFO:tasks.rados.rados.0.smithi064.stdout:1673: write (excl) oid 13 current snap is 198 2024-03-04T06:14:37.247 INFO:tasks.rados.rados.0.smithi064.stdout:1673: seq_num 799 ranges {628793=741016,2138995=567940,3192030=236763} 2024-03-04T06:14:37.253 INFO:tasks.rados.rados.0.smithi064.stdout:1673: writing smithi06431743-13 from 628793 to 1369809 tid 1 2024-03-04T06:14:37.261 INFO:tasks.rados.rados.0.smithi064.stdout:1673: writing smithi06431743-13 from 2138995 to 2706935 tid 2 2024-03-04T06:14:37.265 INFO:tasks.rados.rados.0.smithi064.stdout:1673: writing smithi06431743-13 from 3192030 to 3428793 tid 3 2024-03-04T06:14:37.265 INFO:tasks.rados.rados.0.smithi064.stdout:1674: write oid 20 current snap is 198 2024-03-04T06:14:37.265 INFO:tasks.rados.rados.0.smithi064.stdout:1674: seq_num 800 ranges {645600=416265,1797477=711598,3219453=226147} 2024-03-04T06:14:37.270 INFO:tasks.rados.rados.0.smithi064.stdout:1674: writing smithi06431743-20 from 645600 to 1061865 tid 1 2024-03-04T06:14:37.278 INFO:tasks.rados.rados.0.smithi064.stdout:1674: writing smithi06431743-20 from 1797477 to 2509075 tid 2 2024-03-04T06:14:37.283 INFO:tasks.rados.rados.0.smithi064.stdout:1674: writing smithi06431743-20 from 3219453 to 3445600 tid 3 2024-03-04T06:14:37.283 INFO:tasks.rados.rados.0.smithi064.stdout:1675: write (excl) oid 18 current snap is 198 2024-03-04T06:14:37.284 INFO:tasks.rados.rados.0.smithi064.stdout:1675: seq_num 801 ranges {662407=491514,1855959=571609,2846876=615531} 2024-03-04T06:14:37.290 INFO:tasks.rados.rados.0.smithi064.stdout:1675: writing smithi06431743-18 from 662407 to 1153921 tid 1 2024-03-04T06:14:37.296 INFO:tasks.rados.rados.0.smithi064.stdout:1675: writing smithi06431743-18 from 1855959 to 2427568 tid 2 2024-03-04T06:14:37.302 INFO:tasks.rados.rados.0.smithi064.stdout:1675: writing smithi06431743-18 from 2846876 to 3462407 tid 3 2024-03-04T06:14:37.302 INFO:tasks.rados.rados.0.smithi064.stdout:1676: write oid 36 current snap is 198 2024-03-04T06:14:37.303 INFO:tasks.rados.rados.0.smithi064.stdout:1676: seq_num 802 ranges {679214=566763,1798088=715267,3157946=321268} 2024-03-04T06:14:37.309 INFO:tasks.rados.rados.0.smithi064.stdout:1676: writing smithi06431743-36 from 679214 to 1245977 tid 1 2024-03-04T06:14:37.316 INFO:tasks.rados.rados.0.smithi064.stdout:1676: writing smithi06431743-36 from 1798088 to 2513355 tid 2 2024-03-04T06:14:37.321 INFO:tasks.rados.rados.0.smithi064.stdout:1676: writing smithi06431743-36 from 3157946 to 3479214 tid 3 2024-03-04T06:14:37.321 INFO:tasks.rados.rados.0.smithi064.stdout:1677: write oid 7 current snap is 198 2024-03-04T06:14:37.322 INFO:tasks.rados.rados.0.smithi064.stdout:1675: finishing write tid 1 to smithi06431743-18 2024-03-04T06:14:37.322 INFO:tasks.rados.rados.0.smithi064.stdout:1677: seq_num 803 ranges {696021=642012,1856570=575278,3185369=310652} 2024-03-04T06:14:37.329 INFO:tasks.rados.rados.0.smithi064.stdout:1677: writing smithi06431743-7 from 696021 to 1338033 tid 1 2024-03-04T06:14:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:1677: writing smithi06431743-7 from 1856570 to 2431848 tid 2 2024-03-04T06:14:37.342 INFO:tasks.rados.rados.0.smithi064.stdout:1677: writing smithi06431743-7 from 3185369 to 3496021 tid 3 2024-03-04T06:14:37.343 INFO:tasks.rados.rados.0.smithi064.stdout:1678: read oid 4 snap -1 2024-03-04T06:14:37.343 INFO:tasks.rados.rados.0.smithi064.stdout:1678: expect deleted 2024-03-04T06:14:37.343 INFO:tasks.rados.rados.0.smithi064.stdout:1679: read oid 39 snap -1 2024-03-04T06:14:37.343 INFO:tasks.rados.rados.0.smithi064.stdout:1679: expect (ObjNum 634 snap 154 seq_num 634) 2024-03-04T06:14:37.343 INFO:tasks.rados.rados.0.smithi064.stdout:1680: snap_remove snap 194 2024-03-04T06:14:37.415 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:14:37.416 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:14:37.416 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:14:37.525 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:37.521+0000 7fb87346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:37.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:37.528+0000 7fb87346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:37.717 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:14:37.718 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:14:37.729 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":604,"stamp":"2024-03-04T06:14:37.245269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120598915,"num_objects":574,"num_object_clones":431,"num_object_copies":1722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":100,"num_read":3140,"num_read_kb":2324148,"num_write":2775,"num_write_kb":720798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5583,"ondisk_log_size":5583,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":554204,"kb_used_data":401148,"kb_used_omap":0,"kb_used_meta":153024,"kb_avail":376933156,"statfs":{"total":386547056640,"available":385979551744,"internally_reserved":0,"allocated":410775552,"data_stored":453012299,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":156696576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":840482,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":66,"num_read_kb":57014,"num_write":36,"num_write_kb":10909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008980"},"pg_stats":[{"pgid":"3.f","version":"405'350","reported_seq":1423,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.261310+0000","last_change":"2024-03-04T06:14:35.320360+0000","last_active":"2024-03-04T06:14:35.261310+0000","last_peered":"2024-03-04T06:14:35.261310+0000","last_clean":"2024-03-04T06:14:35.261310+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:14:35.261310+0000","last_undegraded":"2024-03-04T06:14:35.261310+0000","last_fullsized":"2024-03-04T06:14:35.261310+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9042661,"num_objects":42,"num_object_clones":32,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":185,"num_read_kb":135824,"num_write":178,"num_write_kb":49041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"401'247","reported_seq":1348,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.250059+0000","last_change":"2024-03-04T06:14:35.267235+0000","last_active":"2024-03-04T06:14:35.250059+0000","last_peered":"2024-03-04T06:14:35.250059+0000","last_clean":"2024-03-04T06:14:35.250059+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:14:35.250059+0000","last_undegraded":"2024-03-04T06:14:35.250059+0000","last_fullsized":"2024-03-04T06:14:35.250059+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":247,"log_dups_size":0,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5697903,"num_objects":29,"num_object_clones":20,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":94,"num_read_kb":57072,"num_write":113,"num_write_kb":29451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"405'376","reported_seq":1445,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.248036+0000","last_change":"2024-03-04T06:14:35.657987+0000","last_active":"2024-03-04T06:14:36.248036+0000","last_peered":"2024-03-04T06:14:36.248036+0000","last_clean":"2024-03-04T06:14:36.248036+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:14:36.248036+0000","last_undegraded":"2024-03-04T06:14:36.248036+0000","last_fullsized":"2024-03-04T06:14:36.248036+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11664685,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":221,"num_read_kb":159374,"num_write":199,"num_write_kb":55505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"401'297","reported_seq":1398,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.258086+0000","last_change":"2024-03-04T06:14:35.263523+0000","last_active":"2024-03-04T06:14:35.258086+0000","last_peered":"2024-03-04T06:14:35.258086+0000","last_clean":"2024-03-04T06:14:35.258086+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:14:35.258086+0000","last_undegraded":"2024-03-04T06:14:35.258086+0000","last_fullsized":"2024-03-04T06:14:35.258086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8824137,"num_objects":30,"num_object_clones":22,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":164,"num_read_kb":130101,"num_write":157,"num_write_kb":44396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"406'138","reported_seq":1254,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.300205+0000","last_change":"2024-03-04T06:14:36.291019+0000","last_active":"2024-03-04T06:14:36.300205+0000","last_peered":"2024-03-04T06:14:36.300205+0000","last_clean":"2024-03-04T06:14:36.300205+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:14:36.300205+0000","last_undegraded":"2024-03-04T06:14:36.300205+0000","last_fullsized":"2024-03-04T06:14:36.300205+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336338,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":42,"num_read_kb":17047,"num_write":62,"num_write_kb":16597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"405'337","reported_seq":1421,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.265101+0000","last_change":"2024-03-04T06:14:35.304602+0000","last_active":"2024-03-04T06:14:35.265101+0000","last_peered":"2024-03-04T06:14:35.265101+0000","last_clean":"2024-03-04T06:14:35.265101+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:14:35.265101+0000","last_undegraded":"2024-03-04T06:14:35.265101+0000","last_fullsized":"2024-03-04T06:14:35.265101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5311342,"num_objects":35,"num_object_clones":27,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":182,"num_read_kb":144685,"num_write":177,"num_write_kb":52338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1183,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.267860+0000","last_change":"2024-03-04T06:14:35.273662+0000","last_active":"2024-03-04T06:14:35.267860+0000","last_peered":"2024-03-04T06:14:35.267860+0000","last_clean":"2024-03-04T06:14:35.267860+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:14:35.267860+0000","last_undegraded":"2024-03-04T06:14:35.267860+0000","last_fullsized":"2024-03-04T06:14:35.267860+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"405'227","reported_seq":1335,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.253413+0000","last_change":"2024-03-04T06:14:35.648990+0000","last_active":"2024-03-04T06:14:36.253413+0000","last_peered":"2024-03-04T06:14:36.253413+0000","last_clean":"2024-03-04T06:14:36.253413+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:14:36.253413+0000","last_undegraded":"2024-03-04T06:14:36.253413+0000","last_fullsized":"2024-03-04T06:14:36.253413+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151692,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":145,"num_read_kb":133007,"num_write":108,"num_write_kb":28867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1133,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.270561+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:14:36.270561+0000","last_peered":"2024-03-04T06:14:36.270561+0000","last_clean":"2024-03-04T06:14:36.270561+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:14:36.270561+0000","last_undegraded":"2024-03-04T06:14:36.270561+0000","last_fullsized":"2024-03-04T06:14:36.270561+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":990,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.260285+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:14:36.260285+0000","last_peered":"2024-03-04T06:14:36.260285+0000","last_clean":"2024-03-04T06:14:36.260285+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:14:36.260285+0000","last_undegraded":"2024-03-04T06:14:36.260285+0000","last_fullsized":"2024-03-04T06:14:36.260285+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"405'368","reported_seq":1404,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.258560+0000","last_change":"2024-03-04T06:14:35.695330+0000","last_active":"2024-03-04T06:14:36.258560+0000","last_peered":"2024-03-04T06:14:36.258560+0000","last_clean":"2024-03-04T06:14:36.258560+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:14:36.258560+0000","last_undegraded":"2024-03-04T06:14:36.258560+0000","last_fullsized":"2024-03-04T06:14:36.258560+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6538253,"num_objects":38,"num_object_clones":29,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":181,"num_read_kb":134385,"num_write":171,"num_write_kb":45562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1001,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.264526+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:14:35.264526+0000","last_peered":"2024-03-04T06:14:35.264526+0000","last_clean":"2024-03-04T06:14:35.264526+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:14:35.264526+0000","last_undegraded":"2024-03-04T06:14:35.264526+0000","last_fullsized":"2024-03-04T06:14:35.264526+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"406'277","reported_seq":1341,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.271811+0000","last_change":"2024-03-04T06:14:36.300031+0000","last_active":"2024-03-04T06:14:36.271811+0000","last_peered":"2024-03-04T06:14:36.271811+0000","last_clean":"2024-03-04T06:14:36.271811+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:14:36.271811+0000","last_undegraded":"2024-03-04T06:14:36.271811+0000","last_fullsized":"2024-03-04T06:14:36.271811+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5109658,"num_objects":26,"num_object_clones":21,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":138,"num_read_kb":111288,"num_write":110,"num_write_kb":31472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":990,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.252135+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:14:36.252135+0000","last_peered":"2024-03-04T06:14:36.252135+0000","last_clean":"2024-03-04T06:14:36.252135+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:14:36.252135+0000","last_undegraded":"2024-03-04T06:14:36.252135+0000","last_fullsized":"2024-03-04T06:14:36.252135+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"406'667","reported_seq":1663,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.334068+0000","last_change":"2024-03-04T06:14:36.289999+0000","last_active":"2024-03-04T06:14:36.334068+0000","last_peered":"2024-03-04T06:14:36.334068+0000","last_clean":"2024-03-04T06:14:36.334068+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:14:36.334068+0000","last_undegraded":"2024-03-04T06:14:36.334068+0000","last_fullsized":"2024-03-04T06:14:36.334068+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13362390,"num_objects":69,"num_object_clones":51,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":410,"num_read_kb":323491,"num_write":329,"num_write_kb":87256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1001,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.251582+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:14:35.251582+0000","last_peered":"2024-03-04T06:14:35.251582+0000","last_clean":"2024-03-04T06:14:35.251582+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:14:35.251582+0000","last_undegraded":"2024-03-04T06:14:35.251582+0000","last_fullsized":"2024-03-04T06:14:35.251582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"405'657","reported_seq":1584,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.253022+0000","last_change":"2024-03-04T06:14:35.310162+0000","last_active":"2024-03-04T06:14:35.253022+0000","last_peered":"2024-03-04T06:14:35.253022+0000","last_clean":"2024-03-04T06:14:35.253022+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:14:35.253022+0000","last_undegraded":"2024-03-04T06:14:35.253022+0000","last_fullsized":"2024-03-04T06:14:35.253022+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11479841,"num_objects":63,"num_object_clones":47,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":259,"num_read_kb":177859,"num_write":298,"num_write_kb":76952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"405'269","reported_seq":1402,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.259596+0000","last_change":"2024-03-04T06:14:35.657332+0000","last_active":"2024-03-04T06:14:36.259596+0000","last_peered":"2024-03-04T06:14:36.259596+0000","last_clean":"2024-03-04T06:14:36.259596+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:14:36.259596+0000","last_undegraded":"2024-03-04T06:14:36.259596+0000","last_fullsized":"2024-03-04T06:14:36.259596+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168466,"num_objects":30,"num_object_clones":21,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":239,"num_read_kb":195564,"num_write":134,"num_write_kb":35332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1025,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.261502+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:14:35.261502+0000","last_peered":"2024-03-04T06:14:35.261502+0000","last_clean":"2024-03-04T06:14:35.261502+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:14:35.261502+0000","last_undegraded":"2024-03-04T06:14:35.261502+0000","last_fullsized":"2024-03-04T06:14:35.261502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"405'308","reported_seq":1422,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.265434+0000","last_change":"2024-03-04T06:14:35.307349+0000","last_active":"2024-03-04T06:14:35.265434+0000","last_peered":"2024-03-04T06:14:35.265434+0000","last_clean":"2024-03-04T06:14:35.265434+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:14:35.265434+0000","last_undegraded":"2024-03-04T06:14:35.265434+0000","last_fullsized":"2024-03-04T06:14:35.265434+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10051048,"num_objects":36,"num_object_clones":24,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":183,"num_read_kb":124550,"num_write":151,"num_write_kb":40575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1000,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.280390+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:14:36.280390+0000","last_peered":"2024-03-04T06:14:36.280390+0000","last_clean":"2024-03-04T06:14:36.280390+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:14:36.280390+0000","last_undegraded":"2024-03-04T06:14:36.280390+0000","last_fullsized":"2024-03-04T06:14:36.280390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"406'413","reported_seq":1485,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.264535+0000","last_change":"2024-03-04T06:14:36.300152+0000","last_active":"2024-03-04T06:14:36.264535+0000","last_peered":"2024-03-04T06:14:36.264535+0000","last_clean":"2024-03-04T06:14:36.264535+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:14:36.264535+0000","last_undegraded":"2024-03-04T06:14:36.264535+0000","last_fullsized":"2024-03-04T06:14:36.264535+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12437774,"num_objects":43,"num_object_clones":32,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":259,"num_read_kb":203376,"num_write":210,"num_write_kb":58926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1025,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.259147+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:14:35.259147+0000","last_peered":"2024-03-04T06:14:35.259147+0000","last_clean":"2024-03-04T06:14:35.259147+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:14:35.259147+0000","last_undegraded":"2024-03-04T06:14:35.259147+0000","last_fullsized":"2024-03-04T06:14:35.259147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"405'555","reported_seq":1574,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-04T06:14:35.288356+0000","last_change":"2024-03-04T06:14:35.305839+0000","last_active":"2024-03-04T06:14:35.288356+0000","last_peered":"2024-03-04T06:14:35.288356+0000","last_clean":"2024-03-04T06:14:35.288356+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:14:35.288356+0000","last_undegraded":"2024-03-04T06:14:35.288356+0000","last_fullsized":"2024-03-04T06:14:35.288356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832359,"num_objects":50,"num_object_clones":44,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":335,"num_read_kb":276434,"num_write":241,"num_write_kb":66634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":990,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-04T06:14:36.259392+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:14:36.259392+0000","last_peered":"2024-03-04T06:14:36.259392+0000","last_clean":"2024-03-04T06:14:36.259392+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:14:36.259392+0000","last_undegraded":"2024-03-04T06:14:36.259392+0000","last_fullsized":"2024-03-04T06:14:36.259392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":120008547,"num_objects":572,"num_object_clones":431,"num_object_copies":1716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":100,"num_read":3037,"num_read_kb":2324057,"num_write":2638,"num_write_kb":718904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5486,"ondisk_log_size":5486,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706074,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":161880,"kb_used_data":122576,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94209960,"statfs":{"total":96636764160,"available":96470999040,"internally_reserved":0,"allocated":125517824,"data_stored":138037090,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706074,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":136252,"kb_used_data":96948,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94235588,"statfs":{"total":96636764160,"available":96497242112,"internally_reserved":0,"allocated":99274752,"data_stored":111063491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706073,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":127296,"kb_used_data":87992,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94244544,"statfs":{"total":96636764160,"available":96506413056,"internally_reserved":0,"allocated":90103808,"data_stored":96751709,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706073,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":128776,"kb_used_data":93632,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94243064,"statfs":{"total":96636764160,"available":96504897536,"internally_reserved":0,"allocated":95879168,"data_stored":107160009,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:14:38.257 INFO:tasks.rados.rados.0.smithi064.stdout:1680: done (7 left) 2024-03-04T06:14:38.257 INFO:tasks.rados.rados.0.smithi064.stdout:1681: read oid 43 snap -1 2024-03-04T06:14:38.258 INFO:tasks.rados.rados.0.smithi064.stdout:1681: expect (ObjNum 778 snap 192 seq_num 778) 2024-03-04T06:14:38.258 INFO:tasks.rados.rados.0.smithi064.stdout:1682: delete oid 47 current snap is 198 2024-03-04T06:14:38.258 INFO:tasks.rados.rados.0.smithi064.stdout:1673: finishing write tid 1 to smithi06431743-13 2024-03-04T06:14:38.258 INFO:tasks.rados.rados.0.smithi064.stdout:1674: finishing write tid 1 to smithi06431743-20 2024-03-04T06:14:38.258 INFO:tasks.rados.rados.0.smithi064.stdout:1675: finishing write tid 2 to smithi06431743-18 2024-03-04T06:14:38.258 INFO:tasks.rados.rados.0.smithi064.stdout:1676: finishing write tid 1 to smithi06431743-36 2024-03-04T06:14:38.258 INFO:tasks.rados.rados.0.smithi064.stdout:1673: finishing write tid 2 to smithi06431743-13 2024-03-04T06:14:38.258 INFO:tasks.rados.rados.0.smithi064.stdout:1675: finishing write tid 3 to smithi06431743-18 2024-03-04T06:14:38.258 INFO:tasks.rados.rados.0.smithi064.stdout:1679: expect (ObjNum 634 snap 154 seq_num 634) 2024-03-04T06:14:38.411 INFO:tasks.rados.rados.0.smithi064.stdout:1676: finishing write tid 2 to smithi06431743-36 2024-03-04T06:14:38.411 INFO:tasks.rados.rados.0.smithi064.stdout:1677: finishing write tid 1 to smithi06431743-7 2024-03-04T06:14:38.411 INFO:tasks.rados.rados.0.smithi064.stdout:1674: finishing write tid 2 to smithi06431743-20 2024-03-04T06:14:38.411 INFO:tasks.rados.rados.0.smithi064.stdout:1673: finishing write tid 3 to smithi06431743-13 2024-03-04T06:14:38.411 INFO:tasks.rados.rados.0.smithi064.stdout:1679: done (8 left) 2024-03-04T06:14:38.411 INFO:tasks.rados.rados.0.smithi064.stdout:1682: done (7 left) 2024-03-04T06:14:38.411 INFO:tasks.rados.rados.0.smithi064.stdout:1683: read oid 6 snap -1 2024-03-04T06:14:38.411 INFO:tasks.rados.rados.0.smithi064.stdout:1683: expect deleted 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1684: delete oid 44 current snap is 198 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1677: finishing write tid 2 to smithi06431743-7 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1675: finishing write tid 4 to smithi06431743-18 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1675: finishing write tid 5 to smithi06431743-18 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1675: oid 18 updating version 0 to 250 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1675: oid 18 version 250 is already newer than 249 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1675: oid 18 version 250 is already newer than 248 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1675: oid 18 updating version 250 to 251 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 251 (ObjNum 801 snap 198 seq_num 801) dirty exists 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1675: left oid 18 (ObjNum 801 snap 198 seq_num 801) 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1674: finishing write tid 3 to smithi06431743-20 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1676: finishing write tid 3 to smithi06431743-36 2024-03-04T06:14:38.412 INFO:tasks.rados.rados.0.smithi064.stdout:1677: finishing write tid 3 to smithi06431743-7 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1677: finishing write tid 4 to smithi06431743-7 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1677: finishing write tid 5 to smithi06431743-7 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1677: oid 7 updating version 0 to 280 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1677: oid 7 version 280 is already newer than 279 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1677: oid 7 updating version 280 to 282 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1677: oid 7 version 282 is already newer than 282 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 282 (ObjNum 803 snap 198 seq_num 803) dirty exists 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1677: left oid 7 (ObjNum 803 snap 198 seq_num 803) 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1676: finishing write tid 4 to smithi06431743-36 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1676: finishing write tid 5 to smithi06431743-36 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1676: oid 36 updating version 0 to 313 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1676: oid 36 version 313 is already newer than 313 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1676: oid 36 version 313 is already newer than 311 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:1676: oid 36 version 313 is already newer than 310 2024-03-04T06:14:38.413 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 313 (ObjNum 802 snap 198 seq_num 802) dirty exists 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1676: left oid 36 (ObjNum 802 snap 198 seq_num 802) 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1674: finishing write tid 4 to smithi06431743-20 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1673: finishing write tid 4 to smithi06431743-13 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1674: finishing write tid 5 to smithi06431743-20 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1674: oid 20 updating version 0 to 415 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1674: oid 20 updating version 415 to 418 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1674: oid 20 version 418 is already newer than 417 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1674: oid 20 version 418 is already newer than 416 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 418 (ObjNum 800 snap 198 seq_num 800) dirty exists 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1674: left oid 20 (ObjNum 800 snap 198 seq_num 800) 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1673: finishing write tid 5 to smithi06431743-13 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1673: oid 13 updating version 0 to 660 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1673: oid 13 version 660 is already newer than 659 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1673: oid 13 updating version 660 to 662 2024-03-04T06:14:38.414 INFO:tasks.rados.rados.0.smithi064.stdout:1673: oid 13 version 662 is already newer than 661 2024-03-04T06:14:38.415 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 662 (ObjNum 799 snap 198 seq_num 799) dirty exists 2024-03-04T06:14:38.415 INFO:tasks.rados.rados.0.smithi064.stdout:1673: left oid 13 (ObjNum 799 snap 198 seq_num 799) 2024-03-04T06:14:38.415 INFO:tasks.rados.rados.0.smithi064.stdout:1681: expect (ObjNum 778 snap 192 seq_num 778) 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1673: done (8 left) 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1674: done (7 left) 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1675: done (6 left) 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1676: done (5 left) 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1677: done (4 left) 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1678: done (3 left) 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1681: done (2 left) 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1683: done (1 left) 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1684: done (0 left) 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1685: read oid 4 snap 195 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1685: expect deleted 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1686: write (excl) oid 28 current snap is 198 2024-03-04T06:14:38.624 INFO:tasks.rados.rados.0.smithi064.stdout:1686: seq_num 806 ranges {746442=584112,1748369=722605,2983991=562451} 2024-03-04T06:14:38.630 INFO:tasks.rados.rados.0.smithi064.stdout:1686: writing smithi06431743-28 from 746442 to 1330554 tid 1 2024-03-04T06:14:38.637 INFO:tasks.rados.rados.0.smithi064.stdout:1686: writing smithi06431743-28 from 1748369 to 2470974 tid 2 2024-03-04T06:14:38.643 INFO:tasks.rados.rados.0.smithi064.stdout:1686: writing smithi06431743-28 from 2983991 to 3546442 tid 3 2024-03-04T06:14:38.643 INFO:tasks.rados.rados.0.smithi064.stdout:1687: read oid 34 snap -1 2024-03-04T06:14:38.644 INFO:tasks.rados.rados.0.smithi064.stdout:1687: expect (ObjNum 723 snap 179 seq_num 723) 2024-03-04T06:14:38.644 INFO:tasks.rados.rados.0.smithi064.stdout:1688: write (excl) oid 24 current snap is 198 2024-03-04T06:14:38.644 INFO:tasks.rados.rados.0.smithi064.stdout:1688: seq_num 807 ranges {763249=659361,2090498=582616,3295061=268188} 2024-03-04T06:14:38.650 INFO:tasks.rados.rados.0.smithi064.stdout:1688: writing smithi06431743-24 from 763249 to 1422610 tid 1 2024-03-04T06:14:38.657 INFO:tasks.rados.rados.0.smithi064.stdout:1688: writing smithi06431743-24 from 2090498 to 2673114 tid 2 2024-03-04T06:14:38.662 INFO:tasks.rados.rados.0.smithi064.stdout:1688: writing smithi06431743-24 from 3295061 to 3563249 tid 3 2024-03-04T06:14:38.662 INFO:tasks.rados.rados.0.smithi064.stdout:1689: write oid 15 current snap is 198 2024-03-04T06:14:38.663 INFO:tasks.rados.rados.0.smithi064.stdout:1689: seq_num 808 ranges {780056=734610,2148980=726274,3322484=257572} 2024-03-04T06:14:38.673 INFO:tasks.rados.rados.0.smithi064.stdout:1689: writing smithi06431743-15 from 780056 to 1514666 tid 1 2024-03-04T06:14:38.682 INFO:tasks.rados.rados.0.smithi064.stdout:1689: writing smithi06431743-15 from 2148980 to 2875254 tid 2 2024-03-04T06:14:38.686 INFO:tasks.rados.rados.0.smithi064.stdout:1689: writing smithi06431743-15 from 3322484 to 3580056 tid 3 2024-03-04T06:14:38.686 INFO:tasks.rados.rados.0.smithi064.stdout:1690: snap_create 2024-03-04T06:14:38.686 INFO:tasks.rados.rados.0.smithi064.stdout:1686: finishing write tid 1 to smithi06431743-28 2024-03-04T06:14:38.686 INFO:tasks.rados.rados.0.smithi064.stdout:1688: finishing write tid 1 to smithi06431743-24 2024-03-04T06:14:38.703 INFO:tasks.rados.rados.0.smithi064.stdout:1686: finishing write tid 2 to smithi06431743-28 2024-03-04T06:14:38.717 INFO:tasks.rados.rados.0.smithi064.stdout:1688: finishing write tid 2 to smithi06431743-24 2024-03-04T06:14:38.724 INFO:tasks.rados.rados.0.smithi064.stdout:1688: finishing write tid 3 to smithi06431743-24 2024-03-04T06:14:38.726 INFO:tasks.rados.rados.0.smithi064.stdout:1686: finishing write tid 3 to smithi06431743-28 2024-03-04T06:14:38.727 INFO:tasks.rados.rados.0.smithi064.stdout:1686: finishing write tid 4 to smithi06431743-28 2024-03-04T06:14:38.727 INFO:tasks.rados.rados.0.smithi064.stdout:1689: finishing write tid 1 to smithi06431743-15 2024-03-04T06:14:38.728 INFO:tasks.rados.rados.0.smithi064.stdout:1686: finishing write tid 5 to smithi06431743-28 2024-03-04T06:14:38.728 INFO:tasks.rados.rados.0.smithi064.stdout:1686: oid 28 updating version 0 to 675 2024-03-04T06:14:38.728 INFO:tasks.rados.rados.0.smithi064.stdout:1686: oid 28 version 675 is already newer than 675 2024-03-04T06:14:38.728 INFO:tasks.rados.rados.0.smithi064.stdout:1686: oid 28 version 675 is already newer than 672 2024-03-04T06:14:38.728 INFO:tasks.rados.rados.0.smithi064.stdout:1686: oid 28 version 675 is already newer than 673 2024-03-04T06:14:38.728 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 675 (ObjNum 806 snap 198 seq_num 806) dirty exists 2024-03-04T06:14:38.728 INFO:tasks.rados.rados.0.smithi064.stdout:1686: left oid 28 (ObjNum 806 snap 198 seq_num 806) 2024-03-04T06:14:38.731 INFO:tasks.rados.rados.0.smithi064.stdout:1688: finishing write tid 4 to smithi06431743-24 2024-03-04T06:14:38.735 INFO:tasks.rados.rados.0.smithi064.stdout:1688: finishing write tid 5 to smithi06431743-24 2024-03-04T06:14:38.735 INFO:tasks.rados.rados.0.smithi064.stdout:1688: oid 24 updating version 0 to 425 2024-03-04T06:14:38.735 INFO:tasks.rados.rados.0.smithi064.stdout:1688: oid 24 version 425 is already newer than 424 2024-03-04T06:14:38.735 INFO:tasks.rados.rados.0.smithi064.stdout:1688: oid 24 version 425 is already newer than 424 2024-03-04T06:14:38.735 INFO:tasks.rados.rados.0.smithi064.stdout:1688: oid 24 version 425 is already newer than 422 2024-03-04T06:14:38.735 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 425 (ObjNum 807 snap 198 seq_num 807) dirty exists 2024-03-04T06:14:38.735 INFO:tasks.rados.rados.0.smithi064.stdout:1688: left oid 24 (ObjNum 807 snap 198 seq_num 807) 2024-03-04T06:14:38.741 INFO:tasks.rados.rados.0.smithi064.stdout:1687: expect (ObjNum 723 snap 179 seq_num 723) 2024-03-04T06:14:38.822 INFO:tasks.rados.rados.0.smithi064.stdout:1689: finishing write tid 2 to smithi06431743-15 2024-03-04T06:14:38.822 INFO:tasks.rados.rados.0.smithi064.stdout:1689: finishing write tid 3 to smithi06431743-15 2024-03-04T06:14:38.822 INFO:tasks.rados.rados.0.smithi064.stdout:1689: finishing write tid 4 to smithi06431743-15 2024-03-04T06:14:38.822 INFO:tasks.rados.rados.0.smithi064.stdout:1689: finishing write tid 5 to smithi06431743-15 2024-03-04T06:14:38.822 INFO:tasks.rados.rados.0.smithi064.stdout:1689: oid 15 updating version 0 to 272 2024-03-04T06:14:38.822 INFO:tasks.rados.rados.0.smithi064.stdout:1689: oid 15 updating version 272 to 274 2024-03-04T06:14:38.822 INFO:tasks.rados.rados.0.smithi064.stdout:1689: oid 15 version 274 is already newer than 273 2024-03-04T06:14:38.823 INFO:tasks.rados.rados.0.smithi064.stdout:1689: oid 15 version 274 is already newer than 271 2024-03-04T06:14:38.823 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 274 (ObjNum 808 snap 198 seq_num 808) dirty exists 2024-03-04T06:14:38.823 INFO:tasks.rados.rados.0.smithi064.stdout:1689: left oid 15 (ObjNum 808 snap 198 seq_num 808) 2024-03-04T06:14:39.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:39.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:39.245+0000 7fb5da514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:39.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:39.249+0000 7fb5da514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:39.266 INFO:tasks.rados.rados.0.smithi064.stdout:1685: done (5 left) 2024-03-04T06:14:39.266 INFO:tasks.rados.rados.0.smithi064.stdout:1686: done (4 left) 2024-03-04T06:14:39.267 INFO:tasks.rados.rados.0.smithi064.stdout:1687: done (3 left) 2024-03-04T06:14:39.267 INFO:tasks.rados.rados.0.smithi064.stdout:1688: done (2 left) 2024-03-04T06:14:39.267 INFO:tasks.rados.rados.0.smithi064.stdout:1689: done (1 left) 2024-03-04T06:14:39.267 INFO:tasks.rados.rados.0.smithi064.stdout:1690: done (0 left) 2024-03-04T06:14:39.267 INFO:tasks.rados.rados.0.smithi064.stdout:1691: read oid 46 snap -1 2024-03-04T06:14:39.267 INFO:tasks.rados.rados.0.smithi064.stdout:1691: expect (ObjNum 757 snap 188 seq_num 757) 2024-03-04T06:14:39.267 INFO:tasks.rados.rados.0.smithi064.stdout:1692: delete oid 41 current snap is 199 2024-03-04T06:14:39.324 INFO:tasks.rados.rados.0.smithi064.stdout:1691: expect (ObjNum 757 snap 188 seq_num 757) 2024-03-04T06:14:39.507 INFO:tasks.rados.rados.0.smithi064.stdout:1691: done (1 left) 2024-03-04T06:14:39.507 INFO:tasks.rados.rados.0.smithi064.stdout:1692: done (0 left) 2024-03-04T06:14:39.507 INFO:tasks.rados.rados.0.smithi064.stdout:1693: write oid 6 current snap is 199 2024-03-04T06:14:39.507 INFO:tasks.rados.rados.0.smithi064.stdout:1693: seq_num 810 ranges {413670=485108,1465944=729943,2977330=524801} 2024-03-04T06:14:39.511 INFO:tasks.rados.rados.0.smithi064.stdout:1693: writing smithi06431743-6 from 413670 to 898778 tid 1 2024-03-04T06:14:39.520 INFO:tasks.rados.rados.0.smithi064.stdout:1693: writing smithi06431743-6 from 1465944 to 2195887 tid 2 2024-03-04T06:14:39.528 INFO:tasks.rados.rados.0.smithi064.stdout:1693: writing smithi06431743-6 from 2977330 to 3502131 tid 3 2024-03-04T06:14:39.528 INFO:tasks.rados.rados.0.smithi064.stdout:1694: write (excl) oid 14 current snap is 199 2024-03-04T06:14:39.528 INFO:tasks.rados.rados.0.smithi064.stdout:1693: finishing write tid 1 to smithi06431743-6 2024-03-04T06:14:39.528 INFO:tasks.rados.rados.0.smithi064.stdout:1694: seq_num 811 ranges {430477=560357,1408073=473601,2488400=736073} 2024-03-04T06:14:39.533 INFO:tasks.rados.rados.0.smithi064.stdout:1694: writing smithi06431743-14 from 430477 to 990834 tid 1 2024-03-04T06:14:39.538 INFO:tasks.rados.rados.0.smithi064.stdout:1694: writing smithi06431743-14 from 1408073 to 1881674 tid 2 2024-03-04T06:14:39.546 INFO:tasks.rados.rados.0.smithi064.stdout:1694: writing smithi06431743-14 from 2488400 to 3224473 tid 3 2024-03-04T06:14:39.546 INFO:tasks.rados.rados.0.smithi064.stdout:1695: write (excl) oid 26 current snap is 199 2024-03-04T06:14:39.546 INFO:tasks.rados.rados.0.smithi064.stdout:1693: finishing write tid 2 to smithi06431743-6 2024-03-04T06:14:39.546 INFO:tasks.rados.rados.0.smithi064.stdout:1695: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2024-03-04T06:14:39.553 INFO:tasks.rados.rados.0.smithi064.stdout:1695: writing smithi06431743-26 from 447284 to 1082890 tid 1 2024-03-04T06:14:39.563 INFO:tasks.rados.rados.0.smithi064.stdout:1695: writing smithi06431743-26 from 1866555 to 2600167 tid 2 2024-03-04T06:14:39.569 INFO:tasks.rados.rados.0.smithi064.stdout:1695: writing smithi06431743-26 from 3315823 to 3647284 tid 3 2024-03-04T06:14:39.570 INFO:tasks.rados.rados.0.smithi064.stdout:1696: write (excl) oid 38 current snap is 199 2024-03-04T06:14:39.570 INFO:tasks.rados.rados.0.smithi064.stdout:1696: seq_num 813 ranges {464091=710855,1925037=477270,2943246=474970} 2024-03-04T06:14:39.578 INFO:tasks.rados.rados.0.smithi064.stdout:1696: writing smithi06431743-38 from 464091 to 1174946 tid 1 2024-03-04T06:14:39.587 INFO:tasks.rados.rados.0.smithi064.stdout:1696: writing smithi06431743-38 from 1925037 to 2402307 tid 2 2024-03-04T06:14:39.594 INFO:tasks.rados.rados.0.smithi064.stdout:1696: writing smithi06431743-38 from 2943246 to 3418216 tid 3 2024-03-04T06:14:39.594 INFO:tasks.rados.rados.0.smithi064.stdout:1697: read oid 35 snap 196 2024-03-04T06:14:39.594 INFO:tasks.rados.rados.0.smithi064.stdout:1697: expect (ObjNum 782 snap 193 seq_num 782) 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1698: delete oid 48 current snap is 199 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1694: finishing write tid 1 to smithi06431743-14 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1693: finishing write tid 3 to smithi06431743-6 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1693: finishing write tid 4 to smithi06431743-6 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1693: finishing write tid 5 to smithi06431743-6 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1693: oid 6 updating version 0 to 275 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1693: oid 6 updating version 275 to 276 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1693: oid 6 updating version 276 to 278 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1693: oid 6 version 278 is already newer than 278 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 278 (ObjNum 810 snap 199 seq_num 810) dirty exists 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1693: left oid 6 (ObjNum 810 snap 199 seq_num 810) 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1694: finishing write tid 2 to smithi06431743-14 2024-03-04T06:14:39.595 INFO:tasks.rados.rados.0.smithi064.stdout:1695: finishing write tid 1 to smithi06431743-26 2024-03-04T06:14:39.621 INFO:tasks.rados.rados.0.smithi064.stdout:1696: finishing write tid 1 to smithi06431743-38 2024-03-04T06:14:39.621 INFO:tasks.rados.rados.0.smithi064.stdout:1694: finishing write tid 3 to smithi06431743-14 2024-03-04T06:14:39.621 INFO:tasks.rados.rados.0.smithi064.stdout:1695: finishing write tid 2 to smithi06431743-26 2024-03-04T06:14:39.631 INFO:tasks.rados.rados.0.smithi064.stdout:1694: finishing write tid 4 to smithi06431743-14 2024-03-04T06:14:39.634 INFO:tasks.rados.rados.0.smithi064.stdout:1694: finishing write tid 5 to smithi06431743-14 2024-03-04T06:14:39.634 INFO:tasks.rados.rados.0.smithi064.stdout:1694: oid 14 updating version 0 to 427 2024-03-04T06:14:39.634 INFO:tasks.rados.rados.0.smithi064.stdout:1694: oid 14 updating version 427 to 430 2024-03-04T06:14:39.634 INFO:tasks.rados.rados.0.smithi064.stdout:1694: oid 14 version 430 is already newer than 430 2024-03-04T06:14:39.634 INFO:tasks.rados.rados.0.smithi064.stdout:1694: oid 14 version 430 is already newer than 428 2024-03-04T06:14:39.634 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 430 (ObjNum 811 snap 199 seq_num 811) dirty exists 2024-03-04T06:14:39.634 INFO:tasks.rados.rados.0.smithi064.stdout:1694: left oid 14 (ObjNum 811 snap 199 seq_num 811) 2024-03-04T06:14:39.640 INFO:tasks.rados.rados.0.smithi064.stdout:1693: done (5 left) 2024-03-04T06:14:39.640 INFO:tasks.rados.rados.0.smithi064.stdout:1694: done (4 left) 2024-03-04T06:14:39.641 INFO:tasks.rados.rados.0.smithi064.stdout:1698: done (3 left) 2024-03-04T06:14:39.641 INFO:tasks.rados.rados.0.smithi064.stdout:1699: snap_remove snap 195 2024-03-04T06:14:40.266 INFO:tasks.rados.rados.0.smithi064.stdout:1699: done (3 left) 2024-03-04T06:14:40.266 INFO:tasks.rados.rados.0.smithi064.stdout:1700: write oid 10 current snap is 199 2024-03-04T06:14:40.266 INFO:tasks.rados.rados.0.smithi064.stdout:1695: finishing write tid 3 to smithi06431743-26 2024-03-04T06:14:40.266 INFO:tasks.rados.rados.0.smithi064.stdout:1700: seq_num 815 ranges {497705=577706,1642001=480939,2598092=497514} 2024-03-04T06:14:40.272 INFO:tasks.rados.rados.0.smithi064.stdout:1700: writing smithi06431743-10 from 497705 to 1075411 tid 1 2024-03-04T06:14:40.279 INFO:tasks.rados.rados.0.smithi064.stdout:1700: writing smithi06431743-10 from 1642001 to 2122940 tid 2 2024-03-04T06:14:40.287 INFO:tasks.rados.rados.0.smithi064.stdout:1700: writing smithi06431743-10 from 2598092 to 3095606 tid 3 2024-03-04T06:14:40.287 INFO:tasks.rados.rados.0.smithi064.stdout:1696: finishing write tid 2 to smithi06431743-38 2024-03-04T06:14:40.287 INFO:tasks.rados.rados.0.smithi064.stdout:1695: finishing write tid 4 to smithi06431743-26 2024-03-04T06:14:40.287 INFO:tasks.rados.rados.0.smithi064.stdout:1697: expect (ObjNum 782 snap 193 seq_num 782) 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1695: finishing write tid 5 to smithi06431743-26 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1695: oid 26 updating version 0 to 679 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1695: oid 26 version 679 is already newer than 678 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1695: oid 26 version 679 is already newer than 677 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1695: oid 26 updating version 679 to 680 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 680 (ObjNum 812 snap 199 seq_num 812) dirty exists 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1695: left oid 26 (ObjNum 812 snap 199 seq_num 812) 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1696: finishing write tid 3 to smithi06431743-38 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1696: finishing write tid 4 to smithi06431743-38 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1701: read oid 23 snap -1 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1701: expect deleted 2024-03-04T06:14:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:1695: done (4 left) 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:1697: done (3 left) 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:1702: write oid 6 current snap is 199 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:1696: finishing write tid 5 to smithi06431743-38 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:1696: oid 38 updating version 0 to 355 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:1696: oid 38 updating version 355 to 356 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:1696: oid 38 version 356 is already newer than 354 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:1696: oid 38 updating version 356 to 357 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 357 (ObjNum 813 snap 199 seq_num 813) dirty exists 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:1696: left oid 38 (ObjNum 813 snap 199 seq_num 813) 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:1700: finishing write tid 1 to smithi06431743-10 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:1700: finishing write tid 2 to smithi06431743-10 2024-03-04T06:14:40.466 INFO:tasks.rados.rados.0.smithi064.stdout:1702: seq_num 816 ranges {514512=652955,1700483=740950,3025515=688997} 2024-03-04T06:14:40.471 INFO:tasks.rados.rados.0.smithi064.stdout:1702: writing smithi06431743-6 from 514512 to 1167467 tid 1 2024-03-04T06:14:40.479 INFO:tasks.rados.rados.0.smithi064.stdout:1702: writing smithi06431743-6 from 1700483 to 2441433 tid 2 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1702: writing smithi06431743-6 from 3025515 to 3714512 tid 3 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1696: done (3 left) 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1703: read oid 11 snap -1 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1703: expect (ObjNum 751 snap 186 seq_num 751) 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1704: read oid 29 snap -1 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1704: expect deleted 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1705: snap_create 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1700: finishing write tid 3 to smithi06431743-10 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1700: finishing write tid 4 to smithi06431743-10 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1700: finishing write tid 5 to smithi06431743-10 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1700: oid 10 updating version 0 to 365 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1700: oid 10 version 365 is already newer than 363 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1700: oid 10 version 365 is already newer than 364 2024-03-04T06:14:40.488 INFO:tasks.rados.rados.0.smithi064.stdout:1700: oid 10 updating version 365 to 366 2024-03-04T06:14:40.489 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 366 (ObjNum 815 snap 199 seq_num 815) dirty exists 2024-03-04T06:14:40.489 INFO:tasks.rados.rados.0.smithi064.stdout:1700: left oid 10 (ObjNum 815 snap 199 seq_num 815) 2024-03-04T06:14:40.495 INFO:tasks.rados.rados.0.smithi064.stdout:1702: finishing write tid 1 to smithi06431743-6 2024-03-04T06:14:40.508 INFO:tasks.rados.rados.0.smithi064.stdout:1703: expect (ObjNum 751 snap 186 seq_num 751) 2024-03-04T06:14:40.664 INFO:tasks.rados.rados.0.smithi064.stdout:1702: finishing write tid 2 to smithi06431743-6 2024-03-04T06:14:40.664 INFO:tasks.rados.rados.0.smithi064.stdout:1702: finishing write tid 3 to smithi06431743-6 2024-03-04T06:14:40.664 INFO:tasks.rados.rados.0.smithi064.stdout:1702: finishing write tid 4 to smithi06431743-6 2024-03-04T06:14:40.665 INFO:tasks.rados.rados.0.smithi064.stdout:1702: finishing write tid 5 to smithi06431743-6 2024-03-04T06:14:40.665 INFO:tasks.rados.rados.0.smithi064.stdout:1702: oid 6 updating version 0 to 283 2024-03-04T06:14:40.665 INFO:tasks.rados.rados.0.smithi064.stdout:1702: oid 6 updating version 283 to 284 2024-03-04T06:14:40.665 INFO:tasks.rados.rados.0.smithi064.stdout:1702: oid 6 version 284 is already newer than 282 2024-03-04T06:14:40.665 INFO:tasks.rados.rados.0.smithi064.stdout:1702: oid 6 version 284 is already newer than 281 2024-03-04T06:14:40.665 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 284 (ObjNum 816 snap 199 seq_num 816) dirty exists 2024-03-04T06:14:40.665 INFO:tasks.rados.rados.0.smithi064.stdout:1702: left oid 6 (ObjNum 816 snap 199 seq_num 816) 2024-03-04T06:14:41.279 INFO:tasks.rados.rados.0.smithi064.stdout:1700: done (5 left) 2024-03-04T06:14:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:1701: done (4 left) 2024-03-04T06:14:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:1702: done (3 left) 2024-03-04T06:14:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:1703: done (2 left) 2024-03-04T06:14:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:1704: done (1 left) 2024-03-04T06:14:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:1705: done (0 left) 2024-03-04T06:14:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:1706: read oid 25 snap -1 2024-03-04T06:14:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:1706: expect (ObjNum 789 snap 195 seq_num 789) 2024-03-04T06:14:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:1707: read oid 32 snap 196 2024-03-04T06:14:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:1707: expect deleted 2024-03-04T06:14:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:1708: snap_remove snap 171 2024-03-04T06:14:41.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:41.453 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:41.451+0000 7f1bc2f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:41.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:41.455+0000 7f1bc2f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:42.280 INFO:tasks.rados.rados.0.smithi064.stdout:1708: done (2 left) 2024-03-04T06:14:42.280 INFO:tasks.rados.rados.0.smithi064.stdout:1709: read oid 47 snap 198 2024-03-04T06:14:42.280 INFO:tasks.rados.rados.0.smithi064.stdout:1709: expect deleted 2024-03-04T06:14:42.280 INFO:tasks.rados.rados.0.smithi064.stdout:1710: delete oid 13 current snap is 200 2024-03-04T06:14:42.280 INFO:tasks.rados.rados.0.smithi064.stdout:1706: expect (ObjNum 789 snap 195 seq_num 789) 2024-03-04T06:14:42.451 INFO:tasks.rados.rados.0.smithi064.stdout:1706: done (3 left) 2024-03-04T06:14:42.451 INFO:tasks.rados.rados.0.smithi064.stdout:1707: done (2 left) 2024-03-04T06:14:42.451 INFO:tasks.rados.rados.0.smithi064.stdout:1709: done (1 left) 2024-03-04T06:14:42.451 INFO:tasks.rados.rados.0.smithi064.stdout:1710: done (0 left) 2024-03-04T06:14:42.451 INFO:tasks.rados.rados.0.smithi064.stdout:1711: write oid 9 current snap is 200 2024-03-04T06:14:42.451 INFO:tasks.rados.rados.0.smithi064.stdout:1711: seq_num 818 ranges {548126=403453,1417447=744619,2680361=447683,3680450=67676} 2024-03-04T06:14:42.456 INFO:tasks.rados.rados.0.smithi064.stdout:1711: writing smithi06431743-9 from 548126 to 951579 tid 1 2024-03-04T06:14:42.463 INFO:tasks.rados.rados.0.smithi064.stdout:1711: writing smithi06431743-9 from 1417447 to 2162066 tid 2 2024-03-04T06:14:42.468 INFO:tasks.rados.rados.0.smithi064.stdout:1711: writing smithi06431743-9 from 2680361 to 3128044 tid 3 2024-03-04T06:14:42.471 INFO:tasks.rados.rados.0.smithi064.stdout:1711: writing smithi06431743-9 from 3680450 to 3748126 tid 4 2024-03-04T06:14:42.472 INFO:tasks.rados.rados.0.smithi064.stdout:1712: read oid 8 snap 197 2024-03-04T06:14:42.472 INFO:tasks.rados.rados.0.smithi064.stdout:1712: expect (ObjNum 788 snap 193 seq_num 788) 2024-03-04T06:14:42.472 INFO:tasks.rados.rados.0.smithi064.stdout:1713: write oid 28 current snap is 200 2024-03-04T06:14:42.472 INFO:tasks.rados.rados.0.smithi064.stdout:1713: seq_num 819 ranges {564933=478702,1759576=488277,2991431=658955} 2024-03-04T06:14:42.477 INFO:tasks.rados.rados.0.smithi064.stdout:1713: writing smithi06431743-28 from 564933 to 1043635 tid 1 2024-03-04T06:14:42.484 INFO:tasks.rados.rados.0.smithi064.stdout:1713: writing smithi06431743-28 from 1759576 to 2247853 tid 2 2024-03-04T06:14:42.492 INFO:tasks.rados.rados.0.smithi064.stdout:1713: writing smithi06431743-28 from 2991431 to 3650386 tid 3 2024-03-04T06:14:42.492 INFO:tasks.rados.rados.0.smithi064.stdout:1714: write oid 18 current snap is 200 2024-03-04T06:14:42.492 INFO:tasks.rados.rados.0.smithi064.stdout:1711: finishing write tid 1 to smithi06431743-9 2024-03-04T06:14:42.492 INFO:tasks.rados.rados.0.smithi064.stdout:1714: seq_num 820 ranges {581740=553951,1818058=748288,3018854=470227} 2024-03-04T06:14:42.498 INFO:tasks.rados.rados.0.smithi064.stdout:1714: writing smithi06431743-18 from 581740 to 1135691 tid 1 2024-03-04T06:14:42.506 INFO:tasks.rados.rados.0.smithi064.stdout:1714: writing smithi06431743-18 from 1818058 to 2566346 tid 2 2024-03-04T06:14:42.511 INFO:tasks.rados.rados.0.smithi064.stdout:1714: writing smithi06431743-18 from 3018854 to 3489081 tid 3 2024-03-04T06:14:42.511 INFO:tasks.rados.rados.0.smithi064.stdout:1715: snap_remove snap 196 2024-03-04T06:14:42.731 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:14:42.731 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:14:42.731 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:14:42.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:14:42.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:42.836+0000 7f9653add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:42.845 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:42.844+0000 7f9653add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:42.856 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:42.856 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:42.856 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:42.856 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:42.857 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:42.857 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:42.857 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:42.857 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:42.857 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:42.857 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:42.857 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:42.857 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:42.859 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:42.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:14:42.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:42.962+0000 7f2f29f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:42.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:42.969+0000 7f2f29f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:42.982 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:42.985 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:42.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:14:43.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:43.087+0000 7fa5e5624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:43.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:43.095+0000 7fa5e5624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:43.108 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:43.108 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:43.108 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:43.109 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:43.109 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:43.109 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:43.109 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:43.109 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:43.109 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:43.109 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:43.109 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:43.109 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:43.111 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:43.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:14:43.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:43.214+0000 7efe300ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:43.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:43.222+0000 7efe300ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:43.235 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:43.235 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:43.235 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:43.235 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:43.235 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:43.235 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:43.236 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:43.236 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:43.236 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:43.236 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:43.236 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:43.236 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:43.237 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:43.238 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:43.239 INFO:tasks.rados.rados.0.smithi064.stdout:1715: done (4 left) 2024-03-04T06:14:43.239 INFO:tasks.rados.rados.0.smithi064.stdout:1716: read oid 42 snap -1 2024-03-04T06:14:43.239 INFO:tasks.rados.rados.0.smithi064.stdout:1716: expect (ObjNum 783 snap 193 seq_num 783) 2024-03-04T06:14:43.239 INFO:tasks.rados.rados.0.smithi064.stdout:1717: snap_create 2024-03-04T06:14:43.239 INFO:tasks.rados.rados.0.smithi064.stdout:1712: expect (ObjNum 788 snap 193 seq_num 788) 2024-03-04T06:14:43.411 INFO:tasks.rados.rados.0.smithi064.stdout:1711: finishing write tid 2 to smithi06431743-9 2024-03-04T06:14:43.411 INFO:tasks.rados.rados.0.smithi064.stdout:1711: finishing write tid 3 to smithi06431743-9 2024-03-04T06:14:43.411 INFO:tasks.rados.rados.0.smithi064.stdout:1714: finishing write tid 1 to smithi06431743-18 2024-03-04T06:14:43.411 INFO:tasks.rados.rados.0.smithi064.stdout:1711: finishing write tid 4 to smithi06431743-9 2024-03-04T06:14:43.411 INFO:tasks.rados.rados.0.smithi064.stdout:1714: finishing write tid 2 to smithi06431743-18 2024-03-04T06:14:43.411 INFO:tasks.rados.rados.0.smithi064.stdout:1713: finishing write tid 1 to smithi06431743-28 2024-03-04T06:14:43.411 INFO:tasks.rados.rados.0.smithi064.stdout:1714: finishing write tid 3 to smithi06431743-18 2024-03-04T06:14:43.411 INFO:tasks.rados.rados.0.smithi064.stdout:1711: finishing write tid 5 to smithi06431743-9 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1711: finishing write tid 6 to smithi06431743-9 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1711: oid 9 updating version 0 to 576 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1711: oid 9 updating version 576 to 578 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1711: oid 9 updating version 578 to 579 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1711: oid 9 version 579 is already newer than 577 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1711: oid 9 updating version 579 to 580 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 580 (ObjNum 818 snap 200 seq_num 818) dirty exists 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1711: left oid 9 (ObjNum 818 snap 200 seq_num 818) 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1713: finishing write tid 2 to smithi06431743-28 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1714: finishing write tid 4 to smithi06431743-18 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1714: finishing write tid 5 to smithi06431743-18 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1714: oid 18 updating version 0 to 258 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1714: oid 18 version 258 is already newer than 257 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1714: oid 18 version 258 is already newer than 255 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:1714: oid 18 version 258 is already newer than 257 2024-03-04T06:14:43.412 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 258 (ObjNum 820 snap 200 seq_num 820) dirty exists 2024-03-04T06:14:43.413 INFO:tasks.rados.rados.0.smithi064.stdout:1714: left oid 18 (ObjNum 820 snap 200 seq_num 820) 2024-03-04T06:14:43.413 INFO:tasks.rados.rados.0.smithi064.stdout:1713: finishing write tid 3 to smithi06431743-28 2024-03-04T06:14:43.413 INFO:tasks.rados.rados.0.smithi064.stdout:1713: finishing write tid 4 to smithi06431743-28 2024-03-04T06:14:43.413 INFO:tasks.rados.rados.0.smithi064.stdout:1713: finishing write tid 5 to smithi06431743-28 2024-03-04T06:14:43.413 INFO:tasks.rados.rados.0.smithi064.stdout:1713: oid 28 updating version 0 to 691 2024-03-04T06:14:43.413 INFO:tasks.rados.rados.0.smithi064.stdout:1713: oid 28 version 691 is already newer than 690 2024-03-04T06:14:43.413 INFO:tasks.rados.rados.0.smithi064.stdout:1713: oid 28 version 691 is already newer than 689 2024-03-04T06:14:43.413 INFO:tasks.rados.rados.0.smithi064.stdout:1713: oid 28 version 691 is already newer than 688 2024-03-04T06:14:43.413 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 691 (ObjNum 819 snap 200 seq_num 819) dirty exists 2024-03-04T06:14:43.413 INFO:tasks.rados.rados.0.smithi064.stdout:1713: left oid 28 (ObjNum 819 snap 200 seq_num 819) 2024-03-04T06:14:43.413 INFO:tasks.rados.rados.0.smithi064.stdout:1716: expect (ObjNum 783 snap 193 seq_num 783) 2024-03-04T06:14:43.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:43.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:43.628+0000 7fc31da6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:43.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:43.632+0000 7fc31da6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:44.240 INFO:tasks.rados.rados.0.smithi064.stdout:1711: done (5 left) 2024-03-04T06:14:44.240 INFO:tasks.rados.rados.0.smithi064.stdout:1712: done (4 left) 2024-03-04T06:14:44.240 INFO:tasks.rados.rados.0.smithi064.stdout:1713: done (3 left) 2024-03-04T06:14:44.240 INFO:tasks.rados.rados.0.smithi064.stdout:1714: done (2 left) 2024-03-04T06:14:44.240 INFO:tasks.rados.rados.0.smithi064.stdout:1716: done (1 left) 2024-03-04T06:14:44.240 INFO:tasks.rados.rados.0.smithi064.stdout:1717: done (0 left) 2024-03-04T06:14:44.240 INFO:tasks.rados.rados.0.smithi064.stdout:1718: write oid 26 current snap is 201 2024-03-04T06:14:44.241 INFO:tasks.rados.rados.0.smithi064.stdout:1718: seq_num 821 ranges {598547=629200,1876540=491946,3046277=681499} 2024-03-04T06:14:44.246 INFO:tasks.rados.rados.0.smithi064.stdout:1718: writing smithi06431743-26 from 598547 to 1227747 tid 1 2024-03-04T06:14:44.252 INFO:tasks.rados.rados.0.smithi064.stdout:1718: writing smithi06431743-26 from 1876540 to 2368486 tid 2 2024-03-04T06:14:44.260 INFO:tasks.rados.rados.0.smithi064.stdout:1718: writing smithi06431743-26 from 3046277 to 3727776 tid 3 2024-03-04T06:14:44.260 INFO:tasks.rados.rados.0.smithi064.stdout:1719: delete oid 47 current snap is 201 2024-03-04T06:14:44.284 INFO:tasks.rados.rados.0.smithi064.stdout:1719: done (1 left) 2024-03-04T06:14:44.284 INFO:tasks.rados.rados.0.smithi064.stdout:1720: snap_create 2024-03-04T06:14:44.360 INFO:tasks.rados.rados.0.smithi064.stdout:1718: finishing write tid 1 to smithi06431743-26 2024-03-04T06:14:44.399 INFO:tasks.rados.rados.0.smithi064.stdout:1718: finishing write tid 2 to smithi06431743-26 2024-03-04T06:14:44.408 INFO:tasks.rados.rados.0.smithi064.stdout:1718: finishing write tid 3 to smithi06431743-26 2024-03-04T06:14:44.413 INFO:tasks.rados.rados.0.smithi064.stdout:1718: finishing write tid 4 to smithi06431743-26 2024-03-04T06:14:44.415 INFO:tasks.rados.rados.0.smithi064.stdout:1718: finishing write tid 5 to smithi06431743-26 2024-03-04T06:14:44.415 INFO:tasks.rados.rados.0.smithi064.stdout:1718: oid 26 updating version 0 to 702 2024-03-04T06:14:44.416 INFO:tasks.rados.rados.0.smithi064.stdout:1718: oid 26 version 702 is already newer than 699 2024-03-04T06:14:44.416 INFO:tasks.rados.rados.0.smithi064.stdout:1718: oid 26 version 702 is already newer than 702 2024-03-04T06:14:44.416 INFO:tasks.rados.rados.0.smithi064.stdout:1718: oid 26 version 702 is already newer than 700 2024-03-04T06:14:44.416 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 702 (ObjNum 821 snap 201 seq_num 821) dirty exists 2024-03-04T06:14:44.416 INFO:tasks.rados.rados.0.smithi064.stdout:1718: left oid 26 (ObjNum 821 snap 201 seq_num 821) 2024-03-04T06:14:45.286 INFO:tasks.rados.rados.0.smithi064.stdout:1718: done (1 left) 2024-03-04T06:14:45.286 INFO:tasks.rados.rados.0.smithi064.stdout:1720: done (0 left) 2024-03-04T06:14:45.286 INFO:tasks.rados.rados.0.smithi064.stdout:1721: write oid 21 current snap is 202 2024-03-04T06:14:45.286 INFO:tasks.rados.rados.0.smithi064.stdout:1721: seq_num 823 ranges {632161=496051,1593504=495615,2701123=420396,3611645=220516} 2024-03-04T06:14:45.292 INFO:tasks.rados.rados.0.smithi064.stdout:1721: writing smithi06431743-21 from 632161 to 1128212 tid 1 2024-03-04T06:14:45.298 INFO:tasks.rados.rados.0.smithi064.stdout:1721: writing smithi06431743-21 from 1593504 to 2089119 tid 2 2024-03-04T06:14:45.305 INFO:tasks.rados.rados.0.smithi064.stdout:1721: writing smithi06431743-21 from 2701123 to 3121519 tid 3 2024-03-04T06:14:45.310 INFO:tasks.rados.rados.0.smithi064.stdout:1721: writing smithi06431743-21 from 3611645 to 3832161 tid 4 2024-03-04T06:14:45.311 INFO:tasks.rados.rados.0.smithi064.stdout:1722: write (excl) oid 3 current snap is 202 2024-03-04T06:14:45.311 INFO:tasks.rados.rados.0.smithi064.stdout:1722: seq_num 824 ranges {648968=571300,1651986=639273,2728546=631668} 2024-03-04T06:14:45.319 INFO:tasks.rados.rados.0.smithi064.stdout:1722: writing smithi06431743-3 from 648968 to 1220268 tid 1 2024-03-04T06:14:45.326 INFO:tasks.rados.rados.0.smithi064.stdout:1722: writing smithi06431743-3 from 1651986 to 2291259 tid 2 2024-03-04T06:14:45.334 INFO:tasks.rados.rados.0.smithi064.stdout:1722: writing smithi06431743-3 from 2728546 to 3360214 tid 3 2024-03-04T06:14:45.334 INFO:tasks.rados.rados.0.smithi064.stdout:1723: write (excl) oid 37 current snap is 202 2024-03-04T06:14:45.335 INFO:tasks.rados.rados.0.smithi064.stdout:1723: seq_num 825 ranges {665775=646549,2110468=499284,3155969=442940} 2024-03-04T06:14:45.343 INFO:tasks.rados.rados.0.smithi064.stdout:1723: writing smithi06431743-37 from 665775 to 1312324 tid 1 2024-03-04T06:14:45.352 INFO:tasks.rados.rados.0.smithi064.stdout:1723: writing smithi06431743-37 from 2110468 to 2609752 tid 2 2024-03-04T06:14:45.359 INFO:tasks.rados.rados.0.smithi064.stdout:1723: writing smithi06431743-37 from 3155969 to 3598909 tid 3 2024-03-04T06:14:45.359 INFO:tasks.rados.rados.0.smithi064.stdout:1724: snap_create 2024-03-04T06:14:45.371 INFO:tasks.rados.rados.0.smithi064.stdout:1721: finishing write tid 1 to smithi06431743-21 2024-03-04T06:14:45.375 INFO:tasks.rados.rados.0.smithi064.stdout:1722: finishing write tid 1 to smithi06431743-3 2024-03-04T06:14:45.394 INFO:tasks.rados.rados.0.smithi064.stdout:1722: finishing write tid 2 to smithi06431743-3 2024-03-04T06:14:45.399 INFO:tasks.rados.rados.0.smithi064.stdout:1721: finishing write tid 2 to smithi06431743-21 2024-03-04T06:14:45.418 INFO:tasks.rados.rados.0.smithi064.stdout:1722: finishing write tid 3 to smithi06431743-3 2024-03-04T06:14:45.420 INFO:tasks.rados.rados.0.smithi064.stdout:1721: finishing write tid 3 to smithi06431743-21 2024-03-04T06:14:45.425 INFO:tasks.rados.rados.0.smithi064.stdout:1723: finishing write tid 1 to smithi06431743-37 2024-03-04T06:14:45.426 INFO:tasks.rados.rados.0.smithi064.stdout:1722: finishing write tid 4 to smithi06431743-3 2024-03-04T06:14:45.426 INFO:tasks.rados.rados.0.smithi064.stdout:1722: finishing write tid 5 to smithi06431743-3 2024-03-04T06:14:45.426 INFO:tasks.rados.rados.0.smithi064.stdout:1722: oid 3 updating version 0 to 382 2024-03-04T06:14:45.426 INFO:tasks.rados.rados.0.smithi064.stdout:1722: oid 3 updating version 382 to 385 2024-03-04T06:14:45.426 INFO:tasks.rados.rados.0.smithi064.stdout:1722: oid 3 version 385 is already newer than 384 2024-03-04T06:14:45.427 INFO:tasks.rados.rados.0.smithi064.stdout:1722: oid 3 version 385 is already newer than 383 2024-03-04T06:14:45.427 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 385 (ObjNum 824 snap 202 seq_num 824) dirty exists 2024-03-04T06:14:45.427 INFO:tasks.rados.rados.0.smithi064.stdout:1722: left oid 3 (ObjNum 824 snap 202 seq_num 824) 2024-03-04T06:14:45.427 INFO:tasks.rados.rados.0.smithi064.stdout:1721: finishing write tid 4 to smithi06431743-21 2024-03-04T06:14:45.434 INFO:tasks.rados.rados.0.smithi064.stdout:1721: finishing write tid 5 to smithi06431743-21 2024-03-04T06:14:45.436 INFO:tasks.rados.rados.0.smithi064.stdout:1721: finishing write tid 6 to smithi06431743-21 2024-03-04T06:14:45.436 INFO:tasks.rados.rados.0.smithi064.stdout:1721: oid 21 updating version 0 to 296 2024-03-04T06:14:45.436 INFO:tasks.rados.rados.0.smithi064.stdout:1721: oid 21 version 296 is already newer than 295 2024-03-04T06:14:45.436 INFO:tasks.rados.rados.0.smithi064.stdout:1721: oid 21 version 296 is already newer than 292 2024-03-04T06:14:45.436 INFO:tasks.rados.rados.0.smithi064.stdout:1721: oid 21 version 296 is already newer than 293 2024-03-04T06:14:45.436 INFO:tasks.rados.rados.0.smithi064.stdout:1721: oid 21 version 296 is already newer than 295 2024-03-04T06:14:45.436 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 296 (ObjNum 823 snap 202 seq_num 823) dirty exists 2024-03-04T06:14:45.436 INFO:tasks.rados.rados.0.smithi064.stdout:1721: left oid 21 (ObjNum 823 snap 202 seq_num 823) 2024-03-04T06:14:45.437 INFO:tasks.rados.rados.0.smithi064.stdout:1723: finishing write tid 2 to smithi06431743-37 2024-03-04T06:14:45.446 INFO:tasks.rados.rados.0.smithi064.stdout:1723: finishing write tid 3 to smithi06431743-37 2024-03-04T06:14:45.452 INFO:tasks.rados.rados.0.smithi064.stdout:1723: finishing write tid 4 to smithi06431743-37 2024-03-04T06:14:45.453 INFO:tasks.rados.rados.0.smithi064.stdout:1723: finishing write tid 5 to smithi06431743-37 2024-03-04T06:14:45.453 INFO:tasks.rados.rados.0.smithi064.stdout:1723: oid 37 updating version 0 to 262 2024-03-04T06:14:45.454 INFO:tasks.rados.rados.0.smithi064.stdout:1723: oid 37 version 262 is already newer than 261 2024-03-04T06:14:45.454 INFO:tasks.rados.rados.0.smithi064.stdout:1723: oid 37 updating version 262 to 263 2024-03-04T06:14:45.454 INFO:tasks.rados.rados.0.smithi064.stdout:1723: oid 37 version 263 is already newer than 260 2024-03-04T06:14:45.454 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 263 (ObjNum 825 snap 202 seq_num 825) dirty exists 2024-03-04T06:14:45.454 INFO:tasks.rados.rados.0.smithi064.stdout:1723: left oid 37 (ObjNum 825 snap 202 seq_num 825) 2024-03-04T06:14:45.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:45.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:45.807+0000 7fd85d8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:45.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:45.811+0000 7fd85d8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:46.289 INFO:tasks.rados.rados.0.smithi064.stdout:1721: done (3 left) 2024-03-04T06:14:46.289 INFO:tasks.rados.rados.0.smithi064.stdout:1722: done (2 left) 2024-03-04T06:14:46.289 INFO:tasks.rados.rados.0.smithi064.stdout:1723: done (1 left) 2024-03-04T06:14:46.289 INFO:tasks.rados.rados.0.smithi064.stdout:1724: done (0 left) 2024-03-04T06:14:46.289 INFO:tasks.rados.rados.0.smithi064.stdout:1725: snap_create 2024-03-04T06:14:47.293 INFO:tasks.rados.rados.0.smithi064.stdout:1725: done (0 left) 2024-03-04T06:14:47.293 INFO:tasks.rados.rados.0.smithi064.stdout:1726: write (excl) oid 19 current snap is 204 2024-03-04T06:14:47.293 INFO:tasks.rados.rados.0.smithi064.stdout:1726: seq_num 826 ranges {682582=721798,2168950=642942,3583392=299190} 2024-03-04T06:14:47.301 INFO:tasks.rados.rados.0.smithi064.stdout:1726: writing smithi06431743-19 from 682582 to 1404380 tid 1 2024-03-04T06:14:47.310 INFO:tasks.rados.rados.0.smithi064.stdout:1726: writing smithi06431743-19 from 2168950 to 2811892 tid 2 2024-03-04T06:14:47.317 INFO:tasks.rados.rados.0.smithi064.stdout:1726: writing smithi06431743-19 from 3583392 to 3882582 tid 3 2024-03-04T06:14:47.317 INFO:tasks.rados.rados.0.smithi064.stdout:1727: delete oid 5 current snap is 204 2024-03-04T06:14:47.364 INFO:tasks.rados.rados.0.smithi064.stdout:1726: finishing write tid 1 to smithi06431743-19 2024-03-04T06:14:47.376 INFO:tasks.rados.rados.0.smithi064.stdout:1726: finishing write tid 2 to smithi06431743-19 2024-03-04T06:14:47.389 INFO:tasks.rados.rados.0.smithi064.stdout:1726: finishing write tid 3 to smithi06431743-19 2024-03-04T06:14:47.392 INFO:tasks.rados.rados.0.smithi064.stdout:1726: finishing write tid 4 to smithi06431743-19 2024-03-04T06:14:47.395 INFO:tasks.rados.rados.0.smithi064.stdout:1726: finishing write tid 5 to smithi06431743-19 2024-03-04T06:14:47.395 INFO:tasks.rados.rados.0.smithi064.stdout:1726: oid 19 updating version 0 to 683 2024-03-04T06:14:47.395 INFO:tasks.rados.rados.0.smithi064.stdout:1726: oid 19 version 683 is already newer than 681 2024-03-04T06:14:47.395 INFO:tasks.rados.rados.0.smithi064.stdout:1726: oid 19 version 683 is already newer than 680 2024-03-04T06:14:47.396 INFO:tasks.rados.rados.0.smithi064.stdout:1726: oid 19 version 683 is already newer than 683 2024-03-04T06:14:47.396 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 683 (ObjNum 826 snap 204 seq_num 826) dirty exists 2024-03-04T06:14:47.396 INFO:tasks.rados.rados.0.smithi064.stdout:1726: left oid 19 (ObjNum 826 snap 204 seq_num 826) 2024-03-04T06:14:47.410 INFO:tasks.rados.rados.0.smithi064.stdout:1726: done (1 left) 2024-03-04T06:14:47.410 INFO:tasks.rados.rados.0.smithi064.stdout:1727: done (0 left) 2024-03-04T06:14:47.410 INFO:tasks.rados.rados.0.smithi064.stdout:1728: snap_remove snap 201 2024-03-04T06:14:47.881 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:48.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:47.998+0000 7fbd99cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:48.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:48.003+0000 7fbd99cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:48.239 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:14:48.239 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:14:48.239 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:14:48.240 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:14:48.294 INFO:tasks.rados.rados.0.smithi064.stdout:1728: done (0 left) 2024-03-04T06:14:48.294 INFO:tasks.rados.rados.0.smithi064.stdout:1729: write (excl) oid 13 current snap is 204 2024-03-04T06:14:48.294 INFO:tasks.rados.rados.0.smithi064.stdout:1729: seq_num 828 ranges {716196=472296,1769561=646611,3121885=793109} 2024-03-04T06:14:48.300 INFO:tasks.rados.rados.0.smithi064.stdout:1729: writing smithi06431743-13 from 716196 to 1188492 tid 1 2024-03-04T06:14:48.308 INFO:tasks.rados.rados.0.smithi064.stdout:1729: writing smithi06431743-13 from 1769561 to 2416172 tid 2 2024-03-04T06:14:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:1729: writing smithi06431743-13 from 3121885 to 3914994 tid 3 2024-03-04T06:14:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:1730: read oid 41 snap -1 2024-03-04T06:14:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:1730: expect deleted 2024-03-04T06:14:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:1731: snap_remove snap 200 2024-03-04T06:14:48.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:48.362+0000 7fceaa507640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:48.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:48.370+0000 7fceaa507640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:48.387 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:48.387 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:48.387 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:48.387 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:48.387 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:48.387 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:48.387 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:48.387 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:48.387 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:48.387 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:48.388 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:48.388 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:48.390 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:48.390 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:48.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:14:48.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:48.493+0000 7f61af42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:48.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:48.500+0000 7f61af42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:48.513 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:48.513 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:48.513 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:48.513 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:48.514 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:48.514 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:48.514 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:48.514 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:48.514 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:48.514 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:48.514 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:48.514 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:48.516 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:48.516 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:48.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:14:48.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:48.619+0000 7faf6695d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:48.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:48.627+0000 7faf6695d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:48.642 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:48.643 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:48.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:14:48.751 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:48.746+0000 7fb6b72ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:48.754 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:48.753+0000 7fb6b72ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:48.766 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:48.766 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:48.766 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:48.766 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:48.767 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:48.767 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:48.767 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:48.767 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:48.767 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:48.767 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:48.767 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:48.767 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:48.769 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:49.301 INFO:tasks.rados.rados.0.smithi064.stdout:1731: done (2 left) 2024-03-04T06:14:49.301 INFO:tasks.rados.rados.0.smithi064.stdout:1732: delete oid 35 current snap is 204 2024-03-04T06:14:49.301 INFO:tasks.rados.rados.0.smithi064.stdout:1729: finishing write tid 1 to smithi06431743-13 2024-03-04T06:14:49.301 INFO:tasks.rados.rados.0.smithi064.stdout:1729: finishing write tid 2 to smithi06431743-13 2024-03-04T06:14:49.301 INFO:tasks.rados.rados.0.smithi064.stdout:1729: finishing write tid 3 to smithi06431743-13 2024-03-04T06:14:49.301 INFO:tasks.rados.rados.0.smithi064.stdout:1729: finishing write tid 4 to smithi06431743-13 2024-03-04T06:14:49.302 INFO:tasks.rados.rados.0.smithi064.stdout:1729: finishing write tid 5 to smithi06431743-13 2024-03-04T06:14:49.302 INFO:tasks.rados.rados.0.smithi064.stdout:1729: oid 13 updating version 0 to 693 2024-03-04T06:14:49.302 INFO:tasks.rados.rados.0.smithi064.stdout:1729: oid 13 version 693 is already newer than 691 2024-03-04T06:14:49.302 INFO:tasks.rados.rados.0.smithi064.stdout:1729: oid 13 version 693 is already newer than 690 2024-03-04T06:14:49.302 INFO:tasks.rados.rados.0.smithi064.stdout:1729: oid 13 version 693 is already newer than 692 2024-03-04T06:14:49.302 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 693 (ObjNum 828 snap 204 seq_num 828) dirty exists 2024-03-04T06:14:49.302 INFO:tasks.rados.rados.0.smithi064.stdout:1729: left oid 13 (ObjNum 828 snap 204 seq_num 828) 2024-03-04T06:14:49.372 INFO:tasks.rados.rados.0.smithi064.stdout:1729: done (2 left) 2024-03-04T06:14:49.372 INFO:tasks.rados.rados.0.smithi064.stdout:1730: done (1 left) 2024-03-04T06:14:49.372 INFO:tasks.rados.rados.0.smithi064.stdout:1732: done (0 left) 2024-03-04T06:14:49.372 INFO:tasks.rados.rados.0.smithi064.stdout:1733: snap_remove snap 198 2024-03-04T06:14:50.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:50.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:50.184+0000 7fdbabd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:50.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:50.188+0000 7fdbabd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:50.302 INFO:tasks.rados.rados.0.smithi064.stdout:1733: done (0 left) 2024-03-04T06:14:50.302 INFO:tasks.rados.rados.0.smithi064.stdout:1734: read oid 48 snap -1 2024-03-04T06:14:50.302 INFO:tasks.rados.rados.0.smithi064.stdout:1734: expect deleted 2024-03-04T06:14:50.302 INFO:tasks.rados.rados.0.smithi064.stdout:1735: read oid 28 snap -1 2024-03-04T06:14:50.302 INFO:tasks.rados.rados.0.smithi064.stdout:1735: expect (ObjNum 819 snap 200 seq_num 819) 2024-03-04T06:14:50.302 INFO:tasks.rados.rados.0.smithi064.stdout:1736: write (excl) oid 6 current snap is 204 2024-03-04T06:14:50.302 INFO:tasks.rados.rados.0.smithi064.stdout:1736: seq_num 830 ranges {749810=739147,2002878=650280,3293084=415653} 2024-03-04T06:14:50.310 INFO:tasks.rados.rados.0.smithi064.stdout:1736: writing smithi06431743-6 from 749810 to 1488957 tid 1 2024-03-04T06:14:50.316 INFO:tasks.rados.rados.0.smithi064.stdout:1736: writing smithi06431743-6 from 2002878 to 2653158 tid 2 2024-03-04T06:14:50.322 INFO:tasks.rados.rados.0.smithi064.stdout:1736: writing smithi06431743-6 from 3293084 to 3708737 tid 3 2024-03-04T06:14:50.322 INFO:tasks.rados.rados.0.smithi064.stdout:1737: snap_remove snap 193 2024-03-04T06:14:51.313 INFO:tasks.rados.rados.0.smithi064.stdout:1737: done (3 left) 2024-03-04T06:14:51.313 INFO:tasks.rados.rados.0.smithi064.stdout:1738: write oid 15 current snap is 204 2024-03-04T06:14:51.313 INFO:tasks.rados.rados.0.smithi064.stdout:1738: seq_num 831 ranges {766617=414396,1945007=510291,3204154=626925} 2024-03-04T06:14:51.319 INFO:tasks.rados.rados.0.smithi064.stdout:1738: writing smithi06431743-15 from 766617 to 1181013 tid 1 2024-03-04T06:14:51.326 INFO:tasks.rados.rados.0.smithi064.stdout:1738: writing smithi06431743-15 from 1945007 to 2455298 tid 2 2024-03-04T06:14:51.334 INFO:tasks.rados.rados.0.smithi064.stdout:1738: writing smithi06431743-15 from 3204154 to 3831079 tid 3 2024-03-04T06:14:51.334 INFO:tasks.rados.rados.0.smithi064.stdout:1739: write oid 9 current snap is 204 2024-03-04T06:14:51.334 INFO:tasks.rados.rados.0.smithi064.stdout:1739: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2024-03-04T06:14:51.341 INFO:tasks.rados.rados.0.smithi064.stdout:1739: writing smithi06431743-9 from 783424 to 1273069 tid 1 2024-03-04T06:14:51.349 INFO:tasks.rados.rados.0.smithi064.stdout:1739: writing smithi06431743-9 from 2003489 to 2657438 tid 2 2024-03-04T06:14:51.356 INFO:tasks.rados.rados.0.smithi064.stdout:1739: writing smithi06431743-9 from 3231577 to 3786127 tid 3 2024-03-04T06:14:51.356 INFO:tasks.rados.rados.0.smithi064.stdout:1740: write (excl) oid 20 current snap is 204 2024-03-04T06:14:51.356 INFO:tasks.rados.rados.0.smithi064.stdout:1735: expect (ObjNum 819 snap 200 seq_num 819) 2024-03-04T06:14:51.644 INFO:tasks.rados.rados.0.smithi064.stdout:1736: finishing write tid 1 to smithi06431743-6 2024-03-04T06:14:51.644 INFO:tasks.rados.rados.0.smithi064.stdout:1736: finishing write tid 2 to smithi06431743-6 2024-03-04T06:14:51.645 INFO:tasks.rados.rados.0.smithi064.stdout:1736: finishing write tid 3 to smithi06431743-6 2024-03-04T06:14:51.645 INFO:tasks.rados.rados.0.smithi064.stdout:1736: finishing write tid 4 to smithi06431743-6 2024-03-04T06:14:51.645 INFO:tasks.rados.rados.0.smithi064.stdout:1736: finishing write tid 5 to smithi06431743-6 2024-03-04T06:14:51.645 INFO:tasks.rados.rados.0.smithi064.stdout:1736: oid 6 updating version 0 to 302 2024-03-04T06:14:51.645 INFO:tasks.rados.rados.0.smithi064.stdout:1736: oid 6 updating version 302 to 303 2024-03-04T06:14:51.645 INFO:tasks.rados.rados.0.smithi064.stdout:1736: oid 6 updating version 303 to 307 2024-03-04T06:14:51.645 INFO:tasks.rados.rados.0.smithi064.stdout:1736: oid 6 version 307 is already newer than 306 2024-03-04T06:14:51.645 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 307 (ObjNum 830 snap 204 seq_num 830) dirty exists 2024-03-04T06:14:51.645 INFO:tasks.rados.rados.0.smithi064.stdout:1736: left oid 6 (ObjNum 830 snap 204 seq_num 830) 2024-03-04T06:14:51.645 INFO:tasks.rados.rados.0.smithi064.stdout:1740: seq_num 833 ranges {400231=564894,1661971=338260} 2024-03-04T06:14:51.649 INFO:tasks.rados.rados.0.smithi064.stdout:1740: writing smithi06431743-20 from 400231 to 965125 tid 1 2024-03-04T06:14:51.655 INFO:tasks.rados.rados.0.smithi064.stdout:1740: writing smithi06431743-20 from 1661971 to 2000231 tid 2 2024-03-04T06:14:51.656 INFO:tasks.rados.rados.0.smithi064.stdout:1734: done (5 left) 2024-03-04T06:14:51.656 INFO:tasks.rados.rados.0.smithi064.stdout:1735: done (4 left) 2024-03-04T06:14:51.656 INFO:tasks.rados.rados.0.smithi064.stdout:1736: done (3 left) 2024-03-04T06:14:51.656 INFO:tasks.rados.rados.0.smithi064.stdout:1741: write (excl) oid 6 current snap is 204 2024-03-04T06:14:51.656 INFO:tasks.rados.rados.0.smithi064.stdout:1738: finishing write tid 1 to smithi06431743-15 2024-03-04T06:14:51.656 INFO:tasks.rados.rados.0.smithi064.stdout:1739: finishing write tid 1 to smithi06431743-9 2024-03-04T06:14:51.656 INFO:tasks.rados.rados.0.smithi064.stdout:1739: finishing write tid 2 to smithi06431743-9 2024-03-04T06:14:51.656 INFO:tasks.rados.rados.0.smithi064.stdout:1738: finishing write tid 2 to smithi06431743-15 2024-03-04T06:14:51.656 INFO:tasks.rados.rados.0.smithi064.stdout:1739: finishing write tid 3 to smithi06431743-9 2024-03-04T06:14:51.656 INFO:tasks.rados.rados.0.smithi064.stdout:1738: finishing write tid 3 to smithi06431743-15 2024-03-04T06:14:51.656 INFO:tasks.rados.rados.0.smithi064.stdout:1741: seq_num 834 ranges {417038=640143,1720453=296585} 2024-03-04T06:14:51.661 INFO:tasks.rados.rados.0.smithi064.stdout:1741: writing smithi06431743-6 from 417038 to 1057181 tid 1 2024-03-04T06:14:51.667 INFO:tasks.rados.rados.0.smithi064.stdout:1741: writing smithi06431743-6 from 1720453 to 2017038 tid 2 2024-03-04T06:14:51.667 INFO:tasks.rados.rados.0.smithi064.stdout:1742: write (excl) oid 38 current snap is 204 2024-03-04T06:14:51.667 INFO:tasks.rados.rados.0.smithi064.stdout:1742: seq_num 835 ranges {433845=715392,1662582=371263} 2024-03-04T06:14:51.673 INFO:tasks.rados.rados.0.smithi064.stdout:1742: writing smithi06431743-38 from 433845 to 1149237 tid 1 2024-03-04T06:14:51.679 INFO:tasks.rados.rados.0.smithi064.stdout:1742: writing smithi06431743-38 from 1662582 to 2033845 tid 2 2024-03-04T06:14:51.679 INFO:tasks.rados.rados.0.smithi064.stdout:1743: snap_create 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1739: finishing write tid 4 to smithi06431743-9 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1739: finishing write tid 5 to smithi06431743-9 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1739: oid 9 updating version 0 to 593 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1739: oid 9 updating version 593 to 595 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1739: oid 9 version 595 is already newer than 595 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1739: oid 9 version 595 is already newer than 593 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 595 (ObjNum 832 snap 204 seq_num 832) dirty exists 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1739: left oid 9 (ObjNum 832 snap 204 seq_num 832) 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1738: finishing write tid 4 to smithi06431743-15 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1738: finishing write tid 5 to smithi06431743-15 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1738: oid 15 updating version 0 to 315 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1738: oid 15 version 315 is already newer than 314 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1738: oid 15 version 315 is already newer than 313 2024-03-04T06:14:51.680 INFO:tasks.rados.rados.0.smithi064.stdout:1738: oid 15 updating version 315 to 316 2024-03-04T06:14:51.681 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 316 (ObjNum 831 snap 204 seq_num 831) dirty exists 2024-03-04T06:14:51.681 INFO:tasks.rados.rados.0.smithi064.stdout:1738: left oid 15 (ObjNum 831 snap 204 seq_num 831) 2024-03-04T06:14:51.710 INFO:tasks.rados.rados.0.smithi064.stdout:1741: finishing write tid 1 to smithi06431743-6 2024-03-04T06:14:51.740 INFO:tasks.rados.rados.0.smithi064.stdout:1741: finishing write tid 2 to smithi06431743-6 2024-03-04T06:14:51.741 INFO:tasks.rados.rados.0.smithi064.stdout:1740: finishing write tid 1 to smithi06431743-20 2024-03-04T06:14:51.757 INFO:tasks.rados.rados.0.smithi064.stdout:1741: finishing write tid 3 to smithi06431743-6 2024-03-04T06:14:51.758 INFO:tasks.rados.rados.0.smithi064.stdout:1741: finishing write tid 4 to smithi06431743-6 2024-03-04T06:14:51.758 INFO:tasks.rados.rados.0.smithi064.stdout:1741: oid 6 updating version 0 to 319 2024-03-04T06:14:51.758 INFO:tasks.rados.rados.0.smithi064.stdout:1741: oid 6 version 319 is already newer than 318 2024-03-04T06:14:51.758 INFO:tasks.rados.rados.0.smithi064.stdout:1741: oid 6 version 319 is already newer than 317 2024-03-04T06:14:51.758 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 319 (ObjNum 834 snap 204 seq_num 834) dirty exists 2024-03-04T06:14:51.758 INFO:tasks.rados.rados.0.smithi064.stdout:1741: left oid 6 (ObjNum 834 snap 204 seq_num 834) 2024-03-04T06:14:51.761 INFO:tasks.rados.rados.0.smithi064.stdout:1740: finishing write tid 2 to smithi06431743-20 2024-03-04T06:14:51.762 INFO:tasks.rados.rados.0.smithi064.stdout:1742: finishing write tid 1 to smithi06431743-38 2024-03-04T06:14:51.774 INFO:tasks.rados.rados.0.smithi064.stdout:1742: finishing write tid 2 to smithi06431743-38 2024-03-04T06:14:51.775 INFO:tasks.rados.rados.0.smithi064.stdout:1740: finishing write tid 3 to smithi06431743-20 2024-03-04T06:14:51.777 INFO:tasks.rados.rados.0.smithi064.stdout:1740: finishing write tid 4 to smithi06431743-20 2024-03-04T06:14:51.777 INFO:tasks.rados.rados.0.smithi064.stdout:1740: oid 20 updating version 0 to 462 2024-03-04T06:14:51.777 INFO:tasks.rados.rados.0.smithi064.stdout:1740: oid 20 updating version 462 to 463 2024-03-04T06:14:51.777 INFO:tasks.rados.rados.0.smithi064.stdout:1740: oid 20 updating version 463 to 464 2024-03-04T06:14:51.777 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 464 (ObjNum 833 snap 204 seq_num 833) dirty exists 2024-03-04T06:14:51.777 INFO:tasks.rados.rados.0.smithi064.stdout:1740: left oid 20 (ObjNum 833 snap 204 seq_num 833) 2024-03-04T06:14:51.782 INFO:tasks.rados.rados.0.smithi064.stdout:1742: finishing write tid 3 to smithi06431743-38 2024-03-04T06:14:51.784 INFO:tasks.rados.rados.0.smithi064.stdout:1742: finishing write tid 4 to smithi06431743-38 2024-03-04T06:14:51.784 INFO:tasks.rados.rados.0.smithi064.stdout:1742: oid 38 updating version 0 to 388 2024-03-04T06:14:51.784 INFO:tasks.rados.rados.0.smithi064.stdout:1742: oid 38 version 388 is already newer than 387 2024-03-04T06:14:51.784 INFO:tasks.rados.rados.0.smithi064.stdout:1742: oid 38 version 388 is already newer than 386 2024-03-04T06:14:51.785 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 388 (ObjNum 835 snap 204 seq_num 835) dirty exists 2024-03-04T06:14:51.785 INFO:tasks.rados.rados.0.smithi064.stdout:1742: left oid 38 (ObjNum 835 snap 204 seq_num 835) 2024-03-04T06:14:52.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:52.312 INFO:tasks.rados.rados.0.smithi064.stdout:1738: done (5 left) 2024-03-04T06:14:52.312 INFO:tasks.rados.rados.0.smithi064.stdout:1739: done (4 left) 2024-03-04T06:14:52.312 INFO:tasks.rados.rados.0.smithi064.stdout:1740: done (3 left) 2024-03-04T06:14:52.312 INFO:tasks.rados.rados.0.smithi064.stdout:1741: done (2 left) 2024-03-04T06:14:52.313 INFO:tasks.rados.rados.0.smithi064.stdout:1742: done (1 left) 2024-03-04T06:14:52.313 INFO:tasks.rados.rados.0.smithi064.stdout:1743: done (0 left) 2024-03-04T06:14:52.313 INFO:tasks.rados.rados.0.smithi064.stdout:1744: write (excl) oid 3 current snap is 205 2024-03-04T06:14:52.313 INFO:tasks.rados.rados.0.smithi064.stdout:1744: seq_num 836 ranges {450652=790641,1721064=329588} 2024-03-04T06:14:52.319 INFO:tasks.rados.rados.0.smithi064.stdout:1744: writing smithi06431743-3 from 450652 to 1241293 tid 1 2024-03-04T06:14:52.324 INFO:tasks.rados.rados.0.smithi064.stdout:1744: writing smithi06431743-3 from 1721064 to 2050652 tid 2 2024-03-04T06:14:52.324 INFO:tasks.rados.rados.0.smithi064.stdout:1745: delete oid 15 current snap is 205 2024-03-04T06:14:52.384 INFO:tasks.rados.rados.0.smithi064.stdout:1744: finishing write tid 1 to smithi06431743-3 2024-03-04T06:14:52.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:52.386+0000 7fcc7ea0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:52.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:52.391+0000 7fcc7ea0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:52.398 INFO:tasks.rados.rados.0.smithi064.stdout:1745: done (1 left) 2024-03-04T06:14:52.398 INFO:tasks.rados.rados.0.smithi064.stdout:1746: write oid 30 current snap is 205 2024-03-04T06:14:52.398 INFO:tasks.rados.rados.0.smithi064.stdout:1746: seq_num 838 ranges {484266=657492,1554381=529885} 2024-03-04T06:14:52.404 INFO:tasks.rados.rados.0.smithi064.stdout:1746: writing smithi06431743-30 from 484266 to 1141758 tid 1 2024-03-04T06:14:52.410 INFO:tasks.rados.rados.0.smithi064.stdout:1746: writing smithi06431743-30 from 1554381 to 2084266 tid 2 2024-03-04T06:14:52.410 INFO:tasks.rados.rados.0.smithi064.stdout:1747: delete oid 10 current snap is 205 2024-03-04T06:14:52.410 INFO:tasks.rados.rados.0.smithi064.stdout:1744: finishing write tid 2 to smithi06431743-3 2024-03-04T06:14:52.434 INFO:tasks.rados.rados.0.smithi064.stdout:1744: finishing write tid 3 to smithi06431743-3 2024-03-04T06:14:52.436 INFO:tasks.rados.rados.0.smithi064.stdout:1744: finishing write tid 4 to smithi06431743-3 2024-03-04T06:14:52.436 INFO:tasks.rados.rados.0.smithi064.stdout:1744: oid 3 updating version 0 to 401 2024-03-04T06:14:52.436 INFO:tasks.rados.rados.0.smithi064.stdout:1744: oid 3 version 401 is already newer than 400 2024-03-04T06:14:52.436 INFO:tasks.rados.rados.0.smithi064.stdout:1744: oid 3 version 401 is already newer than 399 2024-03-04T06:14:52.436 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 401 (ObjNum 836 snap 205 seq_num 836) dirty exists 2024-03-04T06:14:52.436 INFO:tasks.rados.rados.0.smithi064.stdout:1744: left oid 3 (ObjNum 836 snap 205 seq_num 836) 2024-03-04T06:14:52.446 INFO:tasks.rados.rados.0.smithi064.stdout:1744: done (2 left) 2024-03-04T06:14:52.446 INFO:tasks.rados.rados.0.smithi064.stdout:1747: done (1 left) 2024-03-04T06:14:52.446 INFO:tasks.rados.rados.0.smithi064.stdout:1748: snap_remove snap 197 2024-03-04T06:14:53.241 INFO:tasks.rados.rados.0.smithi064.stdout:1748: done (1 left) 2024-03-04T06:14:53.241 INFO:tasks.rados.rados.0.smithi064.stdout:1749: read oid 33 snap -1 2024-03-04T06:14:53.241 INFO:tasks.rados.rados.0.smithi064.stdout:1749: expect deleted 2024-03-04T06:14:53.241 INFO:tasks.rados.rados.0.smithi064.stdout:1750: write (excl) oid 9 current snap is 205 2024-03-04T06:14:53.241 INFO:tasks.rados.rados.0.smithi064.stdout:1746: finishing write tid 1 to smithi06431743-30 2024-03-04T06:14:53.241 INFO:tasks.rados.rados.0.smithi064.stdout:1750: seq_num 840 ranges {517880=407990,1554992=562888} 2024-03-04T06:14:53.245 INFO:tasks.rados.rados.0.smithi064.stdout:1750: writing smithi06431743-9 from 517880 to 925870 tid 1 2024-03-04T06:14:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:1750: writing smithi06431743-9 from 1554992 to 2117880 tid 2 2024-03-04T06:14:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:1751: delete oid 48 current snap is 205 2024-03-04T06:14:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:1746: finishing write tid 2 to smithi06431743-30 2024-03-04T06:14:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:1746: finishing write tid 3 to smithi06431743-30 2024-03-04T06:14:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:1746: finishing write tid 4 to smithi06431743-30 2024-03-04T06:14:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:1746: oid 30 updating version 0 to 712 2024-03-04T06:14:53.252 INFO:tasks.rados.rados.0.smithi064.stdout:1746: oid 30 version 712 is already newer than 711 2024-03-04T06:14:53.253 INFO:tasks.rados.rados.0.smithi064.stdout:1746: oid 30 updating version 712 to 713 2024-03-04T06:14:53.253 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 713 (ObjNum 838 snap 205 seq_num 838) dirty exists 2024-03-04T06:14:53.253 INFO:tasks.rados.rados.0.smithi064.stdout:1746: left oid 30 (ObjNum 838 snap 205 seq_num 838) 2024-03-04T06:14:53.332 INFO:tasks.rados.rados.0.smithi064.stdout:1746: done (3 left) 2024-03-04T06:14:53.332 INFO:tasks.rados.rados.0.smithi064.stdout:1749: done (2 left) 2024-03-04T06:14:53.332 INFO:tasks.rados.rados.0.smithi064.stdout:1751: done (1 left) 2024-03-04T06:14:53.332 INFO:tasks.rados.rados.0.smithi064.stdout:1752: read oid 38 snap 199 2024-03-04T06:14:53.332 INFO:tasks.rados.rados.0.smithi064.stdout:1752: expect (ObjNum 813 snap 199 seq_num 813) 2024-03-04T06:14:53.332 INFO:tasks.rados.rados.0.smithi064.stdout:1753: write (excl) oid 3 current snap is 205 2024-03-04T06:14:53.332 INFO:tasks.rados.rados.0.smithi064.stdout:1753: seq_num 842 ranges {551494=558488,1671956=479538} 2024-03-04T06:14:53.338 INFO:tasks.rados.rados.0.smithi064.stdout:1753: writing smithi06431743-3 from 551494 to 1109982 tid 1 2024-03-04T06:14:53.344 INFO:tasks.rados.rados.0.smithi064.stdout:1753: writing smithi06431743-3 from 1671956 to 2151494 tid 2 2024-03-04T06:14:53.344 INFO:tasks.rados.rados.0.smithi064.stdout:1754: read oid 20 snap -1 2024-03-04T06:14:53.344 INFO:tasks.rados.rados.0.smithi064.stdout:1754: expect (ObjNum 833 snap 204 seq_num 833) 2024-03-04T06:14:53.344 INFO:tasks.rados.rados.0.smithi064.stdout:1755: write oid 21 current snap is 205 2024-03-04T06:14:53.344 INFO:tasks.rados.rados.0.smithi064.stdout:1755: seq_num 843 ranges {568301=633737,1614085=415952} 2024-03-04T06:14:53.350 INFO:tasks.rados.rados.0.smithi064.stdout:1755: writing smithi06431743-21 from 568301 to 1202038 tid 1 2024-03-04T06:14:53.356 INFO:tasks.rados.rados.0.smithi064.stdout:1755: writing smithi06431743-21 from 1614085 to 2030037 tid 2 2024-03-04T06:14:53.356 INFO:tasks.rados.rados.0.smithi064.stdout:1756: write oid 42 current snap is 205 2024-03-04T06:14:53.356 INFO:tasks.rados.rados.0.smithi064.stdout:1756: seq_num 844 ranges {585108=425339,1788920=396188} 2024-03-04T06:14:53.361 INFO:tasks.rados.rados.0.smithi064.stdout:1756: writing smithi06431743-42 from 585108 to 1010447 tid 1 2024-03-04T06:14:53.370 INFO:tasks.rados.rados.0.smithi064.stdout:1756: writing smithi06431743-42 from 1788920 to 2185108 tid 2 2024-03-04T06:14:53.370 INFO:tasks.rados.rados.0.smithi064.stdout:1757: snap_remove snap 204 2024-03-04T06:14:53.770 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:14:53.770 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:14:53.770 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:14:53.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:14:53.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:53.873+0000 7f123775d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:53.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:53.880+0000 7f123775d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:53.894 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:53.894 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:53.894 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:53.894 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:53.894 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:53.894 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:53.894 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:53.894 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:53.895 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:53.897 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:53.897 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:14:54.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:53.999+0000 7fdd8982d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:54.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:54.006+0000 7fdd8982d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:54.019 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:54.020 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:54.020 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:54.020 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:54.020 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:54.020 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:54.020 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:54.020 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:54.020 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:54.020 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:54.020 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:54.020 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:54.023 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:54.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:14:54.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:54.125+0000 7fafe966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:54.134 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:54.132+0000 7fafe966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:54.147 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:54.147 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:54.147 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:54.147 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:54.147 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:54.147 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:54.147 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:54.147 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:54.148 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:54.148 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:54.148 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:54.148 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:54.150 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:54.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:14:54.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:54.252+0000 7f95a14dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:54.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:54.259+0000 7f95a14dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:14:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:14:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:14:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:14:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:14:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:14:54.274 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:14:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:14:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:14:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:14:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:14:54.275 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:14:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:14:54.277 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:14:54.318 INFO:tasks.rados.rados.0.smithi064.stdout:1757: done (6 left) 2024-03-04T06:14:54.318 INFO:tasks.rados.rados.0.smithi064.stdout:1758: read oid 39 snap -1 2024-03-04T06:14:54.318 INFO:tasks.rados.rados.0.smithi064.stdout:1758: expect (ObjNum 634 snap 154 seq_num 634) 2024-03-04T06:14:54.318 INFO:tasks.rados.rados.0.smithi064.stdout:1759: write oid 33 current snap is 205 2024-03-04T06:14:54.318 INFO:tasks.rados.rados.0.smithi064.stdout:1750: finishing write tid 1 to smithi06431743-9 2024-03-04T06:14:54.318 INFO:tasks.rados.rados.0.smithi064.stdout:1759: seq_num 845 ranges {601915=500588,1847402=354513} 2024-03-04T06:14:54.325 INFO:tasks.rados.rados.0.smithi064.stdout:1759: writing smithi06431743-33 from 601915 to 1102503 tid 1 2024-03-04T06:14:54.333 INFO:tasks.rados.rados.0.smithi064.stdout:1759: writing smithi06431743-33 from 1847402 to 2201915 tid 2 2024-03-04T06:14:54.333 INFO:tasks.rados.rados.0.smithi064.stdout:1760: read oid 7 snap 203 2024-03-04T06:14:54.333 INFO:tasks.rados.rados.0.smithi064.stdout:1760: expect (ObjNum 803 snap 198 seq_num 803) 2024-03-04T06:14:54.333 INFO:tasks.rados.rados.0.smithi064.stdout:1761: delete oid 43 current snap is 205 2024-03-04T06:14:54.333 INFO:tasks.rados.rados.0.smithi064.stdout:1754: expect (ObjNum 833 snap 204 seq_num 833) 2024-03-04T06:14:54.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:54.499 INFO:tasks.rados.rados.0.smithi064.stdout:1752: expect (ObjNum 813 snap 199 seq_num 813) 2024-03-04T06:14:54.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:54.594+0000 7fc6d846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:54.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:54.598+0000 7fc6d846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:1750: finishing write tid 2 to smithi06431743-9 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:1756: finishing write tid 1 to smithi06431743-42 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:1750: finishing write tid 3 to smithi06431743-9 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:1753: finishing write tid 1 to smithi06431743-3 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:1750: finishing write tid 4 to smithi06431743-9 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:1750: oid 9 updating version 0 to 602 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:1750: oid 9 updating version 602 to 603 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:1750: oid 9 version 603 is already newer than 601 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 603 (ObjNum 840 snap 205 seq_num 840) dirty exists 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:1750: left oid 9 (ObjNum 840 snap 205 seq_num 840) 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:1756: finishing write tid 2 to smithi06431743-42 2024-03-04T06:14:54.689 INFO:tasks.rados.rados.0.smithi064.stdout:1753: finishing write tid 2 to smithi06431743-3 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1755: finishing write tid 1 to smithi06431743-21 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1756: finishing write tid 3 to smithi06431743-42 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1756: finishing write tid 4 to smithi06431743-42 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1756: oid 42 updating version 0 to 360 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1756: oid 42 version 360 is already newer than 360 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1756: oid 42 version 360 is already newer than 358 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 360 (ObjNum 844 snap 205 seq_num 844) dirty exists 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1756: left oid 42 (ObjNum 844 snap 205 seq_num 844) 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1755: finishing write tid 2 to smithi06431743-21 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1753: finishing write tid 3 to smithi06431743-3 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1753: finishing write tid 4 to smithi06431743-3 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1753: oid 3 updating version 0 to 406 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1753: oid 3 updating version 406 to 407 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1753: oid 3 updating version 407 to 408 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 408 (ObjNum 842 snap 205 seq_num 842) dirty exists 2024-03-04T06:14:54.690 INFO:tasks.rados.rados.0.smithi064.stdout:1753: left oid 3 (ObjNum 842 snap 205 seq_num 842) 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1755: finishing write tid 3 to smithi06431743-21 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1755: finishing write tid 4 to smithi06431743-21 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1755: oid 21 updating version 0 to 329 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1755: oid 21 version 329 is already newer than 327 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1755: oid 21 version 329 is already newer than 329 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 329 (ObjNum 843 snap 205 seq_num 843) dirty exists 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1755: left oid 21 (ObjNum 843 snap 205 seq_num 843) 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1750: done (9 left) 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1752: done (8 left) 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1753: done (7 left) 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1754: done (6 left) 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1755: done (5 left) 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1756: done (4 left) 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1761: done (3 left) 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1762: snap_create 2024-03-04T06:14:54.691 INFO:tasks.rados.rados.0.smithi064.stdout:1758: expect (ObjNum 634 snap 154 seq_num 634) 2024-03-04T06:14:54.826 INFO:tasks.rados.rados.0.smithi064.stdout:1759: finishing write tid 1 to smithi06431743-33 2024-03-04T06:14:54.827 INFO:tasks.rados.rados.0.smithi064.stdout:1759: finishing write tid 2 to smithi06431743-33 2024-03-04T06:14:54.827 INFO:tasks.rados.rados.0.smithi064.stdout:1759: finishing write tid 3 to smithi06431743-33 2024-03-04T06:14:54.827 INFO:tasks.rados.rados.0.smithi064.stdout:1759: finishing write tid 4 to smithi06431743-33 2024-03-04T06:14:54.827 INFO:tasks.rados.rados.0.smithi064.stdout:1759: oid 33 updating version 0 to 608 2024-03-04T06:14:54.827 INFO:tasks.rados.rados.0.smithi064.stdout:1759: oid 33 version 608 is already newer than 608 2024-03-04T06:14:54.827 INFO:tasks.rados.rados.0.smithi064.stdout:1759: oid 33 version 608 is already newer than 606 2024-03-04T06:14:54.827 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 608 (ObjNum 845 snap 205 seq_num 845) dirty exists 2024-03-04T06:14:54.827 INFO:tasks.rados.rados.0.smithi064.stdout:1759: left oid 33 (ObjNum 845 snap 205 seq_num 845) 2024-03-04T06:14:54.827 INFO:tasks.rados.rados.0.smithi064.stdout:1760: expect (ObjNum 803 snap 198 seq_num 803) 2024-03-04T06:14:55.320 INFO:tasks.rados.rados.0.smithi064.stdout:1758: done (3 left) 2024-03-04T06:14:55.320 INFO:tasks.rados.rados.0.smithi064.stdout:1759: done (2 left) 2024-03-04T06:14:55.320 INFO:tasks.rados.rados.0.smithi064.stdout:1760: done (1 left) 2024-03-04T06:14:55.320 INFO:tasks.rados.rados.0.smithi064.stdout:1762: done (0 left) 2024-03-04T06:14:55.320 INFO:tasks.rados.rados.0.smithi064.stdout:1763: snap_remove snap 205 2024-03-04T06:14:56.322 INFO:tasks.rados.rados.0.smithi064.stdout:1763: done (0 left) 2024-03-04T06:14:56.322 INFO:tasks.rados.rados.0.smithi064.stdout:1764: snap_remove snap 203 2024-03-04T06:14:56.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:14:56.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:56.782+0000 7f9ec5ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:56.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:56.787+0000 7f9ec5ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:57.327 INFO:tasks.rados.rados.0.smithi064.stdout:1764: done (0 left) 2024-03-04T06:14:57.327 INFO:tasks.rados.rados.0.smithi064.stdout:1765: read oid 47 snap -1 2024-03-04T06:14:57.327 INFO:tasks.rados.rados.0.smithi064.stdout:1765: expect deleted 2024-03-04T06:14:57.327 INFO:tasks.rados.rados.0.smithi064.stdout:1766: delete oid 25 current snap is 206 2024-03-04T06:14:57.385 INFO:tasks.rados.rados.0.smithi064.stdout:1765: done (1 left) 2024-03-04T06:14:57.385 INFO:tasks.rados.rados.0.smithi064.stdout:1766: done (0 left) 2024-03-04T06:14:57.385 INFO:tasks.rados.rados.0.smithi064.stdout:1767: snap_remove snap 202 2024-03-04T06:14:58.327 INFO:tasks.rados.rados.0.smithi064.stdout:1767: done (0 left) 2024-03-04T06:14:58.327 INFO:tasks.rados.rados.0.smithi064.stdout:1768: delete oid 18 current snap is 206 2024-03-04T06:14:58.397 INFO:tasks.rados.rados.0.smithi064.stdout:1768: done (0 left) 2024-03-04T06:14:58.397 INFO:tasks.rados.rados.0.smithi064.stdout:1769: snap_remove snap 176 2024-03-04T06:14:58.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:14:58.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:58.965+0000 7f206e71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:58.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:58.969+0000 7f206e71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:59.278 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:14:59.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:14:59.333 INFO:tasks.rados.rados.0.smithi064.stdout:1769: done (0 left) 2024-03-04T06:14:59.333 INFO:tasks.rados.rados.0.smithi064.stdout:1770: write (excl) oid 30 current snap is 206 2024-03-04T06:14:59.333 INFO:tasks.rados.rados.0.smithi064.stdout:1770: seq_num 849 ranges {669143=401584,1564977=426959} 2024-03-04T06:14:59.338 INFO:tasks.rados.rados.0.smithi064.stdout:1770: writing smithi06431743-30 from 669143 to 1070727 tid 1 2024-03-04T06:14:59.344 INFO:tasks.rados.rados.0.smithi064.stdout:1770: writing smithi06431743-30 from 1564977 to 1991936 tid 2 2024-03-04T06:14:59.344 INFO:tasks.rados.rados.0.smithi064.stdout:1771: write (excl) oid 29 current snap is 206 2024-03-04T06:14:59.344 INFO:tasks.rados.rados.0.smithi064.stdout:1771: seq_num 850 ranges {685950=476833,1623459=570617} 2024-03-04T06:14:59.350 INFO:tasks.rados.rados.0.smithi064.stdout:1771: writing smithi06431743-29 from 685950 to 1162783 tid 1 2024-03-04T06:14:59.356 INFO:tasks.rados.rados.0.smithi064.stdout:1771: writing smithi06431743-29 from 1623459 to 2194076 tid 2 2024-03-04T06:14:59.356 INFO:tasks.rados.rados.0.smithi064.stdout:1772: read oid 45 snap -1 2024-03-04T06:14:59.356 INFO:tasks.rados.rados.0.smithi064.stdout:1772: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:14:59.356 INFO:tasks.rados.rados.0.smithi064.stdout:1773: delete oid 37 current snap is 206 2024-03-04T06:14:59.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:59.395+0000 7f3d5e79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:59.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:14:59.407+0000 7f3d5e79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:14:59.485 INFO:tasks.rados.rados.0.smithi064.stdout:1770: finishing write tid 1 to smithi06431743-30 2024-03-04T06:14:59.492 INFO:tasks.rados.rados.0.smithi064.stdout:1772: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:14:59.655 INFO:tasks.rados.rados.0.smithi064.stdout:1770: finishing write tid 2 to smithi06431743-30 2024-03-04T06:14:59.655 INFO:tasks.rados.rados.0.smithi064.stdout:1770: finishing write tid 3 to smithi06431743-30 2024-03-04T06:14:59.655 INFO:tasks.rados.rados.0.smithi064.stdout:1770: finishing write tid 4 to smithi06431743-30 2024-03-04T06:14:59.655 INFO:tasks.rados.rados.0.smithi064.stdout:1770: oid 30 updating version 0 to 740 2024-03-04T06:14:59.655 INFO:tasks.rados.rados.0.smithi064.stdout:1770: oid 30 updating version 740 to 742 2024-03-04T06:14:59.655 INFO:tasks.rados.rados.0.smithi064.stdout:1770: oid 30 version 742 is already newer than 741 2024-03-04T06:14:59.655 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 742 (ObjNum 849 snap 206 seq_num 849) dirty exists 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1770: left oid 30 (ObjNum 849 snap 206 seq_num 849) 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1771: finishing write tid 1 to smithi06431743-29 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1771: finishing write tid 2 to smithi06431743-29 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1771: finishing write tid 3 to smithi06431743-29 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1770: done (3 left) 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1772: done (2 left) 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1773: done (1 left) 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1774: write (excl) oid 22 current snap is 206 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1771: finishing write tid 4 to smithi06431743-29 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1771: oid 29 updating version 0 to 745 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1771: oid 29 version 745 is already newer than 743 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1771: oid 29 version 745 is already newer than 745 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 745 (ObjNum 850 snap 206 seq_num 850) dirty exists 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1771: left oid 29 (ObjNum 850 snap 206 seq_num 850) 2024-03-04T06:14:59.656 INFO:tasks.rados.rados.0.smithi064.stdout:1774: seq_num 852 ranges {719564=743684,2140423=179141} 2024-03-04T06:14:59.663 INFO:tasks.rados.rados.0.smithi064.stdout:1774: writing smithi06431743-22 from 719564 to 1463248 tid 1 2024-03-04T06:14:59.668 INFO:tasks.rados.rados.0.smithi064.stdout:1774: writing smithi06431743-22 from 2140423 to 2319564 tid 2 2024-03-04T06:14:59.668 INFO:tasks.rados.rados.0.smithi064.stdout:1771: done (1 left) 2024-03-04T06:14:59.668 INFO:tasks.rados.rados.0.smithi064.stdout:1775: read oid 19 snap 199 2024-03-04T06:14:59.668 INFO:tasks.rados.rados.0.smithi064.stdout:1775: expect (ObjNum 773 snap 191 seq_num 773) 2024-03-04T06:14:59.668 INFO:tasks.rados.rados.0.smithi064.stdout:1776: read oid 44 snap -1 2024-03-04T06:14:59.668 INFO:tasks.rados.rados.0.smithi064.stdout:1776: expect deleted 2024-03-04T06:14:59.668 INFO:tasks.rados.rados.0.smithi064.stdout:1777: snap_create 2024-03-04T06:14:59.685 INFO:tasks.rados.rados.0.smithi064.stdout:1774: finishing write tid 1 to smithi06431743-22 2024-03-04T06:14:59.698 INFO:tasks.rados.rados.0.smithi064.stdout:1775: expect (ObjNum 773 snap 191 seq_num 773) 2024-03-04T06:14:59.848 INFO:tasks.rados.rados.0.smithi064.stdout:1774: finishing write tid 2 to smithi06431743-22 2024-03-04T06:14:59.848 INFO:tasks.rados.rados.0.smithi064.stdout:1774: finishing write tid 3 to smithi06431743-22 2024-03-04T06:14:59.848 INFO:tasks.rados.rados.0.smithi064.stdout:1774: finishing write tid 4 to smithi06431743-22 2024-03-04T06:14:59.848 INFO:tasks.rados.rados.0.smithi064.stdout:1774: oid 22 updating version 0 to 623 2024-03-04T06:14:59.848 INFO:tasks.rados.rados.0.smithi064.stdout:1774: oid 22 version 623 is already newer than 621 2024-03-04T06:14:59.848 INFO:tasks.rados.rados.0.smithi064.stdout:1774: oid 22 version 623 is already newer than 622 2024-03-04T06:14:59.849 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 623 (ObjNum 852 snap 206 seq_num 852) dirty exists 2024-03-04T06:14:59.849 INFO:tasks.rados.rados.0.smithi064.stdout:1774: left oid 22 (ObjNum 852 snap 206 seq_num 852) 2024-03-04T06:15:00.331 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:15:00.334 INFO:tasks.rados.rados.0.smithi064.stdout:1774: done (3 left) 2024-03-04T06:15:00.335 INFO:tasks.rados.rados.0.smithi064.stdout:1775: done (2 left) 2024-03-04T06:15:00.335 INFO:tasks.rados.rados.0.smithi064.stdout:1776: done (1 left) 2024-03-04T06:15:00.335 INFO:tasks.rados.rados.0.smithi064.stdout:1777: done (0 left) 2024-03-04T06:15:00.335 INFO:tasks.rados.rados.0.smithi064.stdout:1778: read oid 35 snap -1 2024-03-04T06:15:00.335 INFO:tasks.rados.rados.0.smithi064.stdout:1778: expect deleted 2024-03-04T06:15:00.335 INFO:tasks.rados.rados.0.smithi064.stdout:1779: read oid 5 snap -1 2024-03-04T06:15:00.335 INFO:tasks.rados.rados.0.smithi064.stdout:1779: expect deleted 2024-03-04T06:15:00.335 INFO:tasks.rados.rados.0.smithi064.stdout:1780: delete oid 11 current snap is 207 2024-03-04T06:15:00.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:15:00.404 INFO:tasks.rados.rados.0.smithi064.stdout:1779: done (2 left) 2024-03-04T06:15:00.404 INFO:tasks.rados.rados.0.smithi064.stdout:1780: done (1 left) 2024-03-04T06:15:00.404 INFO:tasks.rados.rados.0.smithi064.stdout:1781: delete oid 8 current snap is 207 2024-03-04T06:15:00.444 INFO:tasks.rados.rados.0.smithi064.stdout:1778: done (1 left) 2024-03-04T06:15:00.444 INFO:tasks.rados.rados.0.smithi064.stdout:1781: done (0 left) 2024-03-04T06:15:00.444 INFO:tasks.rados.rados.0.smithi064.stdout:1782: write (excl) oid 20 current snap is 207 2024-03-04T06:15:00.444 INFO:tasks.rados.rados.0.smithi064.stdout:1782: seq_num 855 ranges {769985=569431,1915869=437966} 2024-03-04T06:15:00.452 INFO:tasks.rados.rados.0.smithi064.stdout:1782: writing smithi06431743-20 from 769985 to 1339416 tid 1 2024-03-04T06:15:00.459 INFO:tasks.rados.rados.0.smithi064.stdout:1782: writing smithi06431743-20 from 1915869 to 2353835 tid 2 2024-03-04T06:15:00.459 INFO:tasks.rados.rados.0.smithi064.stdout:1783: read oid 26 snap -1 2024-03-04T06:15:00.459 INFO:tasks.rados.rados.0.smithi064.stdout:1783: expect (ObjNum 821 snap 201 seq_num 821) 2024-03-04T06:15:00.459 INFO:tasks.rados.rados.0.smithi064.stdout:1784: read oid 34 snap -1 2024-03-04T06:15:00.459 INFO:tasks.rados.rados.0.smithi064.stdout:1784: expect (ObjNum 723 snap 179 seq_num 723) 2024-03-04T06:15:00.459 INFO:tasks.rados.rados.0.smithi064.stdout:1785: read oid 21 snap -1 2024-03-04T06:15:00.459 INFO:tasks.rados.rados.0.smithi064.stdout:1785: expect (ObjNum 843 snap 205 seq_num 843) 2024-03-04T06:15:00.459 INFO:tasks.rados.rados.0.smithi064.stdout:1786: snap_remove snap 206 2024-03-04T06:15:00.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:00.477+0000 7fe8cee2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:00.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:00.488+0000 7fe8cee2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:01.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:01.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:01.143+0000 7fa38066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:01.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:01.147+0000 7fa38066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:01.338 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:15:01.340 INFO:tasks.rados.rados.0.smithi064.stdout:1786: done (4 left) 2024-03-04T06:15:01.341 INFO:tasks.rados.rados.0.smithi064.stdout:1787: write (excl) oid 37 current snap is 207 2024-03-04T06:15:01.341 INFO:tasks.rados.rados.0.smithi064.stdout:1787: seq_num 856 ranges {786792=644680,1857998=528794} 2024-03-04T06:15:01.348 INFO:tasks.rados.rados.0.smithi064.stdout:1787: writing smithi06431743-37 from 786792 to 1431472 tid 1 2024-03-04T06:15:01.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:15:01.354 INFO:tasks.rados.rados.0.smithi064.stdout:1787: writing smithi06431743-37 from 1857998 to 2386792 tid 2 2024-03-04T06:15:01.354 INFO:tasks.rados.rados.0.smithi064.stdout:1788: write (excl) oid 44 current snap is 207 2024-03-04T06:15:01.354 INFO:tasks.rados.rados.0.smithi064.stdout:1788: seq_num 857 ranges {403599=719929,1916480=441635} 2024-03-04T06:15:01.360 INFO:tasks.rados.rados.0.smithi064.stdout:1788: writing smithi06431743-44 from 403599 to 1123528 tid 1 2024-03-04T06:15:01.368 INFO:tasks.rados.rados.0.smithi064.stdout:1788: writing smithi06431743-44 from 1916480 to 2358115 tid 2 2024-03-04T06:15:01.368 INFO:tasks.rados.rados.0.smithi064.stdout:1789: delete oid 18 current snap is 207 2024-03-04T06:15:01.368 INFO:tasks.rados.rados.0.smithi064.stdout:1785: expect (ObjNum 843 snap 205 seq_num 843) 2024-03-04T06:15:01.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:01.473+0000 7f694d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:01.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:01.482+0000 7f694d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:01.549 INFO:tasks.rados.rados.0.smithi064.stdout:1783: expect (ObjNum 821 snap 201 seq_num 821) 2024-03-04T06:15:01.787 INFO:tasks.rados.rados.0.smithi064.stdout:1782: finishing write tid 1 to smithi06431743-20 2024-03-04T06:15:01.788 INFO:tasks.rados.rados.0.smithi064.stdout:1784: expect (ObjNum 723 snap 179 seq_num 723) 2024-03-04T06:15:01.869 INFO:tasks.rados.rados.0.smithi064.stdout:1782: finishing write tid 2 to smithi06431743-20 2024-03-04T06:15:01.869 INFO:tasks.rados.rados.0.smithi064.stdout:1782: finishing write tid 3 to smithi06431743-20 2024-03-04T06:15:01.869 INFO:tasks.rados.rados.0.smithi064.stdout:1782: finishing write tid 4 to smithi06431743-20 2024-03-04T06:15:01.869 INFO:tasks.rados.rados.0.smithi064.stdout:1782: oid 20 updating version 0 to 488 2024-03-04T06:15:01.869 INFO:tasks.rados.rados.0.smithi064.stdout:1782: oid 20 version 488 is already newer than 486 2024-03-04T06:15:01.869 INFO:tasks.rados.rados.0.smithi064.stdout:1782: oid 20 version 488 is already newer than 488 2024-03-04T06:15:01.869 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 488 (ObjNum 855 snap 207 seq_num 855) dirty exists 2024-03-04T06:15:01.869 INFO:tasks.rados.rados.0.smithi064.stdout:1782: left oid 20 (ObjNum 855 snap 207 seq_num 855) 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1782: done (6 left) 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1783: done (5 left) 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1784: done (4 left) 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1785: done (3 left) 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1789: done (2 left) 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1790: delete oid 11 current snap is 207 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1787: finishing write tid 1 to smithi06431743-37 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1788: finishing write tid 1 to smithi06431743-44 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1787: finishing write tid 2 to smithi06431743-37 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1788: finishing write tid 2 to smithi06431743-44 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1787: finishing write tid 3 to smithi06431743-37 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1788: finishing write tid 3 to smithi06431743-44 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1787: finishing write tid 4 to smithi06431743-37 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1787: oid 37 updating version 0 to 281 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1787: oid 37 version 281 is already newer than 280 2024-03-04T06:15:01.870 INFO:tasks.rados.rados.0.smithi064.stdout:1787: oid 37 updating version 281 to 282 2024-03-04T06:15:01.871 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 282 (ObjNum 856 snap 207 seq_num 856) dirty exists 2024-03-04T06:15:01.871 INFO:tasks.rados.rados.0.smithi064.stdout:1787: left oid 37 (ObjNum 856 snap 207 seq_num 856) 2024-03-04T06:15:01.871 INFO:tasks.rados.rados.0.smithi064.stdout:1788: finishing write tid 4 to smithi06431743-44 2024-03-04T06:15:01.871 INFO:tasks.rados.rados.0.smithi064.stdout:1788: oid 44 updating version 0 to 415 2024-03-04T06:15:01.871 INFO:tasks.rados.rados.0.smithi064.stdout:1788: oid 44 version 415 is already newer than 415 2024-03-04T06:15:01.871 INFO:tasks.rados.rados.0.smithi064.stdout:1788: oid 44 version 415 is already newer than 413 2024-03-04T06:15:01.871 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 415 (ObjNum 857 snap 207 seq_num 857) dirty exists 2024-03-04T06:15:01.871 INFO:tasks.rados.rados.0.smithi064.stdout:1788: left oid 44 (ObjNum 857 snap 207 seq_num 857) 2024-03-04T06:15:01.898 INFO:tasks.rados.rados.0.smithi064.stdout:1787: done (2 left) 2024-03-04T06:15:01.899 INFO:tasks.rados.rados.0.smithi064.stdout:1788: done (1 left) 2024-03-04T06:15:01.899 INFO:tasks.rados.rados.0.smithi064.stdout:1790: done (0 left) 2024-03-04T06:15:01.899 INFO:tasks.rados.rados.0.smithi064.stdout:1791: read oid 39 snap -1 2024-03-04T06:15:01.899 INFO:tasks.rados.rados.0.smithi064.stdout:1791: expect (ObjNum 634 snap 154 seq_num 634) 2024-03-04T06:15:01.899 INFO:tasks.rados.rados.0.smithi064.stdout:1792: read oid 13 snap -1 2024-03-04T06:15:01.899 INFO:tasks.rados.rados.0.smithi064.stdout:1792: expect (ObjNum 828 snap 204 seq_num 828) 2024-03-04T06:15:01.899 INFO:tasks.rados.rados.0.smithi064.stdout:1793: write oid 36 current snap is 207 2024-03-04T06:15:01.899 INFO:tasks.rados.rados.0.smithi064.stdout:1793: seq_num 860 ranges {454020=662029,1691926=588962} 2024-03-04T06:15:01.904 INFO:tasks.rados.rados.0.smithi064.stdout:1793: writing smithi06431743-36 from 454020 to 1116049 tid 1 2024-03-04T06:15:01.911 INFO:tasks.rados.rados.0.smithi064.stdout:1793: writing smithi06431743-36 from 1691926 to 2280888 tid 2 2024-03-04T06:15:01.911 INFO:tasks.rados.rados.0.smithi064.stdout:1794: write (excl) oid 43 current snap is 207 2024-03-04T06:15:01.911 INFO:tasks.rados.rados.0.smithi064.stdout:1794: seq_num 861 ranges {470827=737278,1750408=720419} 2024-03-04T06:15:01.918 INFO:tasks.rados.rados.0.smithi064.stdout:1794: writing smithi06431743-43 from 470827 to 1208105 tid 1 2024-03-04T06:15:01.928 INFO:tasks.rados.rados.0.smithi064.stdout:1794: writing smithi06431743-43 from 1750408 to 2470827 tid 2 2024-03-04T06:15:01.928 INFO:tasks.rados.rados.0.smithi064.stdout:1795: write oid 47 current snap is 207 2024-03-04T06:15:01.928 INFO:tasks.rados.rados.0.smithi064.stdout:1795: seq_num 862 ranges {487634=412527,1408890=592631} 2024-03-04T06:15:01.934 INFO:tasks.rados.rados.0.smithi064.stdout:1795: writing smithi06431743-47 from 487634 to 900161 tid 1 2024-03-04T06:15:01.942 INFO:tasks.rados.rados.0.smithi064.stdout:1795: writing smithi06431743-47 from 1408890 to 2001521 tid 2 2024-03-04T06:15:01.942 INFO:tasks.rados.rados.0.smithi064.stdout:1796: write oid 23 current snap is 207 2024-03-04T06:15:01.942 INFO:tasks.rados.rados.0.smithi064.stdout:1791: expect (ObjNum 634 snap 154 seq_num 634) 2024-03-04T06:15:02.087 INFO:tasks.rados.rados.0.smithi064.stdout:1792: expect (ObjNum 828 snap 204 seq_num 828) 2024-03-04T06:15:02.231 INFO:tasks.rados.rados.0.smithi064.stdout:1793: finishing write tid 1 to smithi06431743-36 2024-03-04T06:15:02.231 INFO:tasks.rados.rados.0.smithi064.stdout:1794: finishing write tid 1 to smithi06431743-43 2024-03-04T06:15:02.231 INFO:tasks.rados.rados.0.smithi064.stdout:1795: finishing write tid 1 to smithi06431743-47 2024-03-04T06:15:02.231 INFO:tasks.rados.rados.0.smithi064.stdout:1793: finishing write tid 2 to smithi06431743-36 2024-03-04T06:15:02.231 INFO:tasks.rados.rados.0.smithi064.stdout:1794: finishing write tid 2 to smithi06431743-43 2024-03-04T06:15:02.231 INFO:tasks.rados.rados.0.smithi064.stdout:1795: finishing write tid 2 to smithi06431743-47 2024-03-04T06:15:02.231 INFO:tasks.rados.rados.0.smithi064.stdout:1793: finishing write tid 3 to smithi06431743-36 2024-03-04T06:15:02.232 INFO:tasks.rados.rados.0.smithi064.stdout:1793: finishing write tid 4 to smithi06431743-36 2024-03-04T06:15:02.232 INFO:tasks.rados.rados.0.smithi064.stdout:1793: oid 36 updating version 0 to 339 2024-03-04T06:15:02.232 INFO:tasks.rados.rados.0.smithi064.stdout:1793: oid 36 version 339 is already newer than 337 2024-03-04T06:15:02.232 INFO:tasks.rados.rados.0.smithi064.stdout:1793: oid 36 version 339 is already newer than 338 2024-03-04T06:15:02.232 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 339 (ObjNum 860 snap 207 seq_num 860) dirty exists 2024-03-04T06:15:02.232 INFO:tasks.rados.rados.0.smithi064.stdout:1793: left oid 36 (ObjNum 860 snap 207 seq_num 860) 2024-03-04T06:15:02.232 INFO:tasks.rados.rados.0.smithi064.stdout:1794: finishing write tid 3 to smithi06431743-43 2024-03-04T06:15:02.232 INFO:tasks.rados.rados.0.smithi064.stdout:1796: seq_num 863 ranges {504441=487776,1467372=736289} 2024-03-04T06:15:02.236 INFO:tasks.rados.rados.0.smithi064.stdout:1796: writing smithi06431743-23 from 504441 to 992217 tid 1 2024-03-04T06:15:02.245 INFO:tasks.rados.rados.0.smithi064.stdout:1796: writing smithi06431743-23 from 1467372 to 2203661 tid 2 2024-03-04T06:15:02.245 INFO:tasks.rados.rados.0.smithi064.stdout:1791: done (5 left) 2024-03-04T06:15:02.245 INFO:tasks.rados.rados.0.smithi064.stdout:1792: done (4 left) 2024-03-04T06:15:02.245 INFO:tasks.rados.rados.0.smithi064.stdout:1793: done (3 left) 2024-03-04T06:15:02.245 INFO:tasks.rados.rados.0.smithi064.stdout:1797: read oid 30 snap 199 2024-03-04T06:15:02.245 INFO:tasks.rados.rados.0.smithi064.stdout:1797: expect (ObjNum 787 snap 193 seq_num 787) 2024-03-04T06:15:02.245 INFO:tasks.rados.rados.0.smithi064.stdout:1798: read oid 18 snap 199 2024-03-04T06:15:02.245 INFO:tasks.rados.rados.0.smithi064.stdout:1798: expect (ObjNum 801 snap 198 seq_num 801) 2024-03-04T06:15:02.245 INFO:tasks.rados.rados.0.smithi064.stdout:1799: snap_create 2024-03-04T06:15:02.245 INFO:tasks.rados.rados.0.smithi064.stdout:1795: finishing write tid 3 to smithi06431743-47 2024-03-04T06:15:02.245 INFO:tasks.rados.rados.0.smithi064.stdout:1794: finishing write tid 4 to smithi06431743-43 2024-03-04T06:15:02.246 INFO:tasks.rados.rados.0.smithi064.stdout:1794: oid 43 updating version 0 to 380 2024-03-04T06:15:02.246 INFO:tasks.rados.rados.0.smithi064.stdout:1794: oid 43 version 380 is already newer than 379 2024-03-04T06:15:02.246 INFO:tasks.rados.rados.0.smithi064.stdout:1794: oid 43 updating version 380 to 381 2024-03-04T06:15:02.246 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 381 (ObjNum 861 snap 207 seq_num 861) dirty exists 2024-03-04T06:15:02.246 INFO:tasks.rados.rados.0.smithi064.stdout:1794: left oid 43 (ObjNum 861 snap 207 seq_num 861) 2024-03-04T06:15:02.246 INFO:tasks.rados.rados.0.smithi064.stdout:1795: finishing write tid 4 to smithi06431743-47 2024-03-04T06:15:02.246 INFO:tasks.rados.rados.0.smithi064.stdout:1795: oid 47 updating version 0 to 312 2024-03-04T06:15:02.246 INFO:tasks.rados.rados.0.smithi064.stdout:1795: oid 47 updating version 312 to 313 2024-03-04T06:15:02.246 INFO:tasks.rados.rados.0.smithi064.stdout:1795: oid 47 updating version 313 to 314 2024-03-04T06:15:02.246 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 314 (ObjNum 862 snap 207 seq_num 862) dirty exists 2024-03-04T06:15:02.246 INFO:tasks.rados.rados.0.smithi064.stdout:1795: left oid 47 (ObjNum 862 snap 207 seq_num 862) 2024-03-04T06:15:02.249 INFO:tasks.rados.rados.0.smithi064.stdout:1796: finishing write tid 1 to smithi06431743-23 2024-03-04T06:15:02.261 INFO:tasks.rados.rados.0.smithi064.stdout:1796: finishing write tid 2 to smithi06431743-23 2024-03-04T06:15:02.270 INFO:tasks.rados.rados.0.smithi064.stdout:1796: finishing write tid 3 to smithi06431743-23 2024-03-04T06:15:02.270 INFO:tasks.rados.rados.0.smithi064.stdout:1797: expect (ObjNum 787 snap 193 seq_num 787) 2024-03-04T06:15:02.339 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:15:02.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:15:02.400 INFO:tasks.rados.rados.0.smithi064.stdout:1796: finishing write tid 4 to smithi06431743-23 2024-03-04T06:15:02.400 INFO:tasks.rados.rados.0.smithi064.stdout:1796: oid 23 updating version 0 to 419 2024-03-04T06:15:02.400 INFO:tasks.rados.rados.0.smithi064.stdout:1796: oid 23 updating version 419 to 421 2024-03-04T06:15:02.400 INFO:tasks.rados.rados.0.smithi064.stdout:1796: oid 23 version 421 is already newer than 420 2024-03-04T06:15:02.400 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 421 (ObjNum 863 snap 207 seq_num 863) dirty exists 2024-03-04T06:15:02.400 INFO:tasks.rados.rados.0.smithi064.stdout:1796: left oid 23 (ObjNum 863 snap 207 seq_num 863) 2024-03-04T06:15:02.400 INFO:tasks.rados.rados.0.smithi064.stdout:1794: done (5 left) 2024-03-04T06:15:02.400 INFO:tasks.rados.rados.0.smithi064.stdout:1795: done (4 left) 2024-03-04T06:15:02.400 INFO:tasks.rados.rados.0.smithi064.stdout:1796: done (3 left) 2024-03-04T06:15:02.401 INFO:tasks.rados.rados.0.smithi064.stdout:1797: done (2 left) 2024-03-04T06:15:02.401 INFO:tasks.rados.rados.0.smithi064.stdout:1799: done (1 left) 2024-03-04T06:15:02.401 INFO:tasks.rados.rados.0.smithi064.stdout:1800: snap_remove snap 207 2024-03-04T06:15:02.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:02.514+0000 7f2391d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:02.522 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:02.522+0000 7f2391d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:03.215 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:03.239 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:15:03.243 INFO:tasks.rados.rados.0.smithi064.stdout:1800: done (1 left) 2024-03-04T06:15:03.243 INFO:tasks.rados.rados.0.smithi064.stdout:1801: delete oid 34 current snap is 208 2024-03-04T06:15:03.243 INFO:tasks.rados.rados.0.smithi064.stdout:1798: expect (ObjNum 801 snap 198 seq_num 801) 2024-03-04T06:15:03.258 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:15:03.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:15:03.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:03.331+0000 7f2ddb4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:03.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:03.335+0000 7f2ddb4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:03.386 INFO:tasks.rados.rados.0.smithi064.stdout:1798: done (1 left) 2024-03-04T06:15:03.386 INFO:tasks.rados.rados.0.smithi064.stdout:1801: done (0 left) 2024-03-04T06:15:03.386 INFO:tasks.rados.rados.0.smithi064.stdout:1802: write (excl) oid 31 current snap is 208 2024-03-04T06:15:03.386 INFO:tasks.rados.rados.0.smithi064.stdout:1802: seq_num 865 ranges {538055=638274,1867983=670072} 2024-03-04T06:15:03.392 INFO:tasks.rados.rados.0.smithi064.stdout:1802: writing smithi06431743-31 from 538055 to 1176329 tid 1 2024-03-04T06:15:03.402 INFO:tasks.rados.rados.0.smithi064.stdout:1802: writing smithi06431743-31 from 1867983 to 2538055 tid 2 2024-03-04T06:15:03.402 INFO:tasks.rados.rados.0.smithi064.stdout:1803: write (excl) oid 44 current snap is 208 2024-03-04T06:15:03.402 INFO:tasks.rados.rados.0.smithi064.stdout:1803: seq_num 866 ranges {554862=713523,1926465=599969} 2024-03-04T06:15:03.409 INFO:tasks.rados.rados.0.smithi064.stdout:1803: writing smithi06431743-44 from 554862 to 1268385 tid 1 2024-03-04T06:15:03.416 INFO:tasks.rados.rados.0.smithi064.stdout:1803: writing smithi06431743-44 from 1926465 to 2526434 tid 2 2024-03-04T06:15:03.416 INFO:tasks.rados.rados.0.smithi064.stdout:1804: delete oid 7 current snap is 208 2024-03-04T06:15:03.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:03.417+0000 7f3eecb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:03.425 INFO:tasks.rados.rados.0.smithi064.stdout:1802: finishing write tid 1 to smithi06431743-31 2024-03-04T06:15:03.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:03.425+0000 7f3eecb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:03.461 INFO:tasks.rados.rados.0.smithi064.stdout:1802: finishing write tid 2 to smithi06431743-31 2024-03-04T06:15:03.471 INFO:tasks.rados.rados.0.smithi064.stdout:1804: done (2 left) 2024-03-04T06:15:03.471 INFO:tasks.rados.rados.0.smithi064.stdout:1805: snap_create 2024-03-04T06:15:03.479 INFO:tasks.rados.rados.0.smithi064.stdout:1802: finishing write tid 3 to smithi06431743-31 2024-03-04T06:15:03.479 INFO:tasks.rados.rados.0.smithi064.stdout:1802: finishing write tid 4 to smithi06431743-31 2024-03-04T06:15:03.479 INFO:tasks.rados.rados.0.smithi064.stdout:1802: oid 31 updating version 0 to 157 2024-03-04T06:15:03.479 INFO:tasks.rados.rados.0.smithi064.stdout:1802: oid 31 version 157 is already newer than 156 2024-03-04T06:15:03.479 INFO:tasks.rados.rados.0.smithi064.stdout:1802: oid 31 version 157 is already newer than 155 2024-03-04T06:15:03.480 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 157 (ObjNum 865 snap 208 seq_num 865) dirty exists 2024-03-04T06:15:03.480 INFO:tasks.rados.rados.0.smithi064.stdout:1802: left oid 31 (ObjNum 865 snap 208 seq_num 865) 2024-03-04T06:15:03.481 INFO:tasks.rados.rados.0.smithi064.stdout:1803: finishing write tid 1 to smithi06431743-44 2024-03-04T06:15:03.493 INFO:tasks.rados.rados.0.smithi064.stdout:1803: finishing write tid 2 to smithi06431743-44 2024-03-04T06:15:03.501 INFO:tasks.rados.rados.0.smithi064.stdout:1803: finishing write tid 3 to smithi06431743-44 2024-03-04T06:15:03.501 INFO:tasks.rados.rados.0.smithi064.stdout:1803: finishing write tid 4 to smithi06431743-44 2024-03-04T06:15:03.501 INFO:tasks.rados.rados.0.smithi064.stdout:1803: oid 44 updating version 0 to 418 2024-03-04T06:15:03.502 INFO:tasks.rados.rados.0.smithi064.stdout:1803: oid 44 updating version 418 to 419 2024-03-04T06:15:03.502 INFO:tasks.rados.rados.0.smithi064.stdout:1803: oid 44 updating version 419 to 420 2024-03-04T06:15:03.502 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 420 (ObjNum 866 snap 208 seq_num 866) dirty exists 2024-03-04T06:15:03.502 INFO:tasks.rados.rados.0.smithi064.stdout:1803: left oid 44 (ObjNum 866 snap 208 seq_num 866) 2024-03-04T06:15:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:15:03.649 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:15:03.661 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":643,"stamp":"2024-03-04T06:15:03.240651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":100026454,"num_objects":621,"num_object_clones":475,"num_object_copies":1863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":110,"num_read":3347,"num_read_kb":2475014,"num_write":2998,"num_write_kb":782903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6211,"ondisk_log_size":6211,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":507008,"kb_used_data":342624,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":376980352,"statfs":{"total":386547056640,"available":386027880448,"internally_reserved":0,"allocated":350846976,"data_stored":373125019,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8370150,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":40,"num_read_kb":32774,"num_write":20,"num_write_kb":3721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.909533"},"pg_stats":[{"pgid":"3.f","version":"427'368","reported_seq":1512,"reported_epoch":429,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:14:59.374948+0000","last_change":"2024-03-04T06:14:59.374948+0000","last_active":"2024-03-04T06:14:59.374948+0000","last_peered":"2024-03-04T06:14:59.374948+0000","last_clean":"2024-03-04T06:14:59.374948+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:14:59.374948+0000","last_undegraded":"2024-03-04T06:14:59.374948+0000","last_fullsized":"2024-03-04T06:14:59.374948+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4992042,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":198,"num_read_kb":151919,"num_write":183,"num_write_kb":49844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"429'279","reported_seq":1451,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.470586+0000","last_change":"2024-03-04T06:14:59.470586+0000","last_active":"2024-03-04T06:14:59.470586+0000","last_peered":"2024-03-04T06:14:59.470586+0000","last_clean":"2024-03-04T06:14:59.470586+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:14:59.470586+0000","last_undegraded":"2024-03-04T06:14:59.470586+0000","last_fullsized":"2024-03-04T06:14:59.470586+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3059039,"num_objects":32,"num_object_clones":22,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":106,"num_read_kb":64809,"num_write":130,"num_write_kb":34377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"429'388","reported_seq":1544,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.345828+0000","last_change":"2024-03-04T06:14:59.495627+0000","last_active":"2024-03-04T06:15:00.345828+0000","last_peered":"2024-03-04T06:15:00.345828+0000","last_clean":"2024-03-04T06:15:00.345828+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:15:00.345828+0000","last_undegraded":"2024-03-04T06:15:00.345828+0000","last_fullsized":"2024-03-04T06:15:00.345828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5781905,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":252,"num_read_kb":192838,"num_write":201,"num_write_kb":55505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"423'309","reported_seq":1478,"reported_epoch":429,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:14:59.356415+0000","last_change":"2024-03-04T06:14:59.356415+0000","last_active":"2024-03-04T06:14:59.356415+0000","last_peered":"2024-03-04T06:14:59.356415+0000","last_clean":"2024-03-04T06:14:59.356415+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:14:59.356415+0000","last_undegraded":"2024-03-04T06:14:59.356415+0000","last_fullsized":"2024-03-04T06:14:59.356415+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6286115,"num_objects":31,"num_object_clones":23,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":165,"num_read_kb":130101,"num_write":158,"num_write_kb":44396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"421'151","reported_seq":1327,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.373927+0000","last_change":"2024-03-04T06:15:00.373927+0000","last_active":"2024-03-04T06:15:00.373927+0000","last_peered":"2024-03-04T06:15:00.373927+0000","last_clean":"2024-03-04T06:15:00.373927+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:15:00.373927+0000","last_undegraded":"2024-03-04T06:15:00.373927+0000","last_fullsized":"2024-03-04T06:15:00.373927+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3344758,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":43,"num_read_kb":17048,"num_write":65,"num_write_kb":17158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"427'404","reported_seq":1528,"reported_epoch":429,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:14:59.373494+0000","last_change":"2024-03-04T06:14:59.373494+0000","last_active":"2024-03-04T06:14:59.373494+0000","last_peered":"2024-03-04T06:14:59.373494+0000","last_clean":"2024-03-04T06:14:59.373494+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:14:59.373494+0000","last_undegraded":"2024-03-04T06:14:59.373494+0000","last_fullsized":"2024-03-04T06:14:59.373494+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033845,"num_objects":41,"num_object_clones":33,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":195,"num_read_kb":153970,"num_write":203,"num_write_kb":59706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1263,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.380907+0000","last_change":"2024-03-04T06:14:59.384006+0000","last_active":"2024-03-04T06:14:59.380907+0000","last_peered":"2024-03-04T06:14:59.380907+0000","last_clean":"2024-03-04T06:14:59.380907+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:14:59.380907+0000","last_undegraded":"2024-03-04T06:14:59.380907+0000","last_fullsized":"2024-03-04T06:14:59.380907+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"429'231","reported_seq":1417,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.349831+0000","last_change":"2024-03-04T06:14:59.496627+0000","last_active":"2024-03-04T06:15:00.349831+0000","last_peered":"2024-03-04T06:15:00.349831+0000","last_clean":"2024-03-04T06:15:00.349831+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:15:00.349831+0000","last_undegraded":"2024-03-04T06:15:00.349831+0000","last_fullsized":"2024-03-04T06:15:00.349831+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151692,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":145,"num_read_kb":133007,"num_write":108,"num_write_kb":28867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1191,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.362385+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:15:00.362385+0000","last_peered":"2024-03-04T06:15:00.362385+0000","last_clean":"2024-03-04T06:15:00.362385+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:15:00.362385+0000","last_undegraded":"2024-03-04T06:15:00.362385+0000","last_fullsized":"2024-03-04T06:15:00.362385+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.354390+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:15:00.354390+0000","last_peered":"2024-03-04T06:15:00.354390+0000","last_clean":"2024-03-04T06:15:00.354390+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:15:00.354390+0000","last_undegraded":"2024-03-04T06:15:00.354390+0000","last_fullsized":"2024-03-04T06:15:00.354390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"429'418","reported_seq":1500,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.353891+0000","last_change":"2024-03-04T06:14:59.496088+0000","last_active":"2024-03-04T06:15:00.353891+0000","last_peered":"2024-03-04T06:15:00.353891+0000","last_clean":"2024-03-04T06:15:00.353891+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:15:00.353891+0000","last_undegraded":"2024-03-04T06:15:00.353891+0000","last_fullsized":"2024-03-04T06:15:00.353891+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151494,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":188,"num_read_kb":134388,"num_write":185,"num_write_kb":49472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1067,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.367866+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:14:59.367866+0000","last_peered":"2024-03-04T06:14:59.367866+0000","last_clean":"2024-03-04T06:14:59.367866+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:14:59.367866+0000","last_undegraded":"2024-03-04T06:14:59.367866+0000","last_fullsized":"2024-03-04T06:14:59.367866+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"423'308","reported_seq":1421,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.359417+0000","last_change":"2024-03-04T06:15:00.359417+0000","last_active":"2024-03-04T06:15:00.359417+0000","last_peered":"2024-03-04T06:15:00.359417+0000","last_clean":"2024-03-04T06:15:00.359417+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:15:00.359417+0000","last_undegraded":"2024-03-04T06:15:00.359417+0000","last_fullsized":"2024-03-04T06:15:00.359417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3496021,"num_objects":28,"num_object_clones":23,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":147,"num_read_kb":118180,"num_write":116,"num_write_kb":32965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.347540+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:15:00.347540+0000","last_peered":"2024-03-04T06:15:00.347540+0000","last_clean":"2024-03-04T06:15:00.347540+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:15:00.347540+0000","last_undegraded":"2024-03-04T06:15:00.347540+0000","last_fullsized":"2024-03-04T06:15:00.347540+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"430'718","reported_seq":1771,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.487585+0000","last_change":"2024-03-04T06:15:00.369904+0000","last_active":"2024-03-04T06:15:00.487585+0000","last_peered":"2024-03-04T06:15:00.487585+0000","last_clean":"2024-03-04T06:15:00.487585+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:15:00.487585+0000","last_undegraded":"2024-03-04T06:15:00.487585+0000","last_fullsized":"2024-03-04T06:15:00.487585+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15379230,"num_objects":73,"num_object_clones":55,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":447,"num_read_kb":356462,"num_write":352,"num_write_kb":94527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1067,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.359983+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:14:59.359983+0000","last_peered":"2024-03-04T06:14:59.359983+0000","last_clean":"2024-03-04T06:14:59.359983+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:14:59.359983+0000","last_undegraded":"2024-03-04T06:14:59.359983+0000","last_fullsized":"2024-03-04T06:14:59.359983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"429'749","reported_seq":1707,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.342096+0000","last_change":"2024-03-04T06:14:59.576525+0000","last_active":"2024-03-04T06:15:00.342096+0000","last_peered":"2024-03-04T06:15:00.342096+0000","last_clean":"2024-03-04T06:15:00.342096+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:15:00.342096+0000","last_undegraded":"2024-03-04T06:15:00.342096+0000","last_fullsized":"2024-03-04T06:15:00.342096+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":749,"log_dups_size":0,"ondisk_log_size":749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12353871,"num_objects":68,"num_object_clones":52,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":288,"num_read_kb":198073,"num_write":327,"num_write_kb":84953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"429'349","reported_seq":1527,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.479189+0000","last_change":"2024-03-04T06:14:59.487685+0000","last_active":"2024-03-04T06:15:00.479189+0000","last_peered":"2024-03-04T06:15:00.479189+0000","last_clean":"2024-03-04T06:15:00.479189+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:15:00.479189+0000","last_undegraded":"2024-03-04T06:15:00.479189+0000","last_fullsized":"2024-03-04T06:15:00.479189+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4185339,"num_objects":37,"num_object_clones":27,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":201227,"num_write":174,"num_write_kb":47566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1093,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.364621+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:14:59.364621+0000","last_peered":"2024-03-04T06:14:59.364621+0000","last_clean":"2024-03-04T06:14:59.364621+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:14:59.364621+0000","last_undegraded":"2024-03-04T06:14:59.364621+0000","last_fullsized":"2024-03-04T06:14:59.364621+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"429'331","reported_seq":1512,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.374918+0000","last_change":"2024-03-04T06:14:59.469912+0000","last_active":"2024-03-04T06:14:59.374918+0000","last_peered":"2024-03-04T06:14:59.374918+0000","last_clean":"2024-03-04T06:14:59.374918+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:14:59.374918+0000","last_undegraded":"2024-03-04T06:14:59.374918+0000","last_fullsized":"2024-03-04T06:14:59.374918+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639293,"num_objects":37,"num_object_clones":25,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":190,"num_read_kb":130950,"num_write":156,"num_write_kb":42142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.376220+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:15:00.376220+0000","last_peered":"2024-03-04T06:15:00.376220+0000","last_clean":"2024-03-04T06:15:00.376220+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:15:00.376220+0000","last_undegraded":"2024-03-04T06:15:00.376220+0000","last_fullsized":"2024-03-04T06:15:00.376220+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"430'488","reported_seq":1585,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.492997+0000","last_change":"2024-03-04T06:15:00.357595+0000","last_active":"2024-03-04T06:15:00.492997+0000","last_peered":"2024-03-04T06:15:00.492997+0000","last_clean":"2024-03-04T06:15:00.492997+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:15:00.492997+0000","last_undegraded":"2024-03-04T06:15:00.492997+0000","last_fullsized":"2024-03-04T06:15:00.492997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9563711,"num_objects":49,"num_object_clones":38,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":281,"num_read_kb":215511,"num_write":234,"num_write_kb":65322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1094,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.359072+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:14:59.359072+0000","last_peered":"2024-03-04T06:14:59.359072+0000","last_clean":"2024-03-04T06:14:59.359072+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:14:59.359072+0000","last_undegraded":"2024-03-04T06:14:59.359072+0000","last_fullsized":"2024-03-04T06:14:59.359072+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"429'623","reported_seq":1687,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.345488+0000","last_change":"2024-03-04T06:14:59.530975+0000","last_active":"2024-03-04T06:15:00.345488+0000","last_peered":"2024-03-04T06:15:00.345488+0000","last_clean":"2024-03-04T06:15:00.345488+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:15:00.345488+0000","last_undegraded":"2024-03-04T06:15:00.345488+0000","last_fullsized":"2024-03-04T06:15:00.345488+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10017731,"num_objects":55,"num_object_clones":48,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":343,"num_read_kb":276440,"num_write":269,"num_write_kb":74209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.354684+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:15:00.354684+0000","last_peered":"2024-03-04T06:15:00.354684+0000","last_clean":"2024-03-04T06:15:00.354684+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:15:00.354684+0000","last_undegraded":"2024-03-04T06:15:00.354684+0000","last_fullsized":"2024-03-04T06:15:00.354684+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":99436086,"num_objects":619,"num_object_clones":475,"num_object_copies":1857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":110,"num_read":3244,"num_read_kb":2474923,"num_write":2861,"num_write_kb":781009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6114,"ondisk_log_size":6114,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706099,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":145136,"kb_used_data":100264,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94226704,"statfs":{"total":96636764160,"available":96488144896,"internally_reserved":0,"allocated":102670336,"data_stored":109897618,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706099,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123388,"kb_used_data":84084,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94248452,"statfs":{"total":96636764160,"available":96510414848,"internally_reserved":0,"allocated":86102016,"data_stored":92279557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706098,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":118860,"kb_used_data":78756,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94252980,"statfs":{"total":96636764160,"available":96515051520,"internally_reserved":0,"allocated":80646144,"data_stored":83690593,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706098,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":119624,"kb_used_data":79520,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94252216,"statfs":{"total":96636764160,"available":96514269184,"internally_reserved":0,"allocated":81428480,"data_stored":87257251,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:15:03.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:15:03.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:03.804+0000 7fea53e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:03.811 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:03.810+0000 7fea53e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:04.000 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:15:04.000 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:15:04.012 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":643,"stamp":"2024-03-04T06:15:03.240651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":100026454,"num_objects":621,"num_object_clones":475,"num_object_copies":1863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":110,"num_read":3347,"num_read_kb":2475014,"num_write":2998,"num_write_kb":782903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6211,"ondisk_log_size":6211,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":507008,"kb_used_data":342624,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":376980352,"statfs":{"total":386547056640,"available":386027880448,"internally_reserved":0,"allocated":350846976,"data_stored":373125019,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8370150,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":40,"num_read_kb":32774,"num_write":20,"num_write_kb":3721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.909533"},"pg_stats":[{"pgid":"3.f","version":"427'368","reported_seq":1512,"reported_epoch":429,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:14:59.374948+0000","last_change":"2024-03-04T06:14:59.374948+0000","last_active":"2024-03-04T06:14:59.374948+0000","last_peered":"2024-03-04T06:14:59.374948+0000","last_clean":"2024-03-04T06:14:59.374948+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:14:59.374948+0000","last_undegraded":"2024-03-04T06:14:59.374948+0000","last_fullsized":"2024-03-04T06:14:59.374948+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4992042,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":198,"num_read_kb":151919,"num_write":183,"num_write_kb":49844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"429'279","reported_seq":1451,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.470586+0000","last_change":"2024-03-04T06:14:59.470586+0000","last_active":"2024-03-04T06:14:59.470586+0000","last_peered":"2024-03-04T06:14:59.470586+0000","last_clean":"2024-03-04T06:14:59.470586+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:14:59.470586+0000","last_undegraded":"2024-03-04T06:14:59.470586+0000","last_fullsized":"2024-03-04T06:14:59.470586+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3059039,"num_objects":32,"num_object_clones":22,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":106,"num_read_kb":64809,"num_write":130,"num_write_kb":34377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"429'388","reported_seq":1544,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.345828+0000","last_change":"2024-03-04T06:14:59.495627+0000","last_active":"2024-03-04T06:15:00.345828+0000","last_peered":"2024-03-04T06:15:00.345828+0000","last_clean":"2024-03-04T06:15:00.345828+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:15:00.345828+0000","last_undegraded":"2024-03-04T06:15:00.345828+0000","last_fullsized":"2024-03-04T06:15:00.345828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5781905,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":252,"num_read_kb":192838,"num_write":201,"num_write_kb":55505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"423'309","reported_seq":1478,"reported_epoch":429,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:14:59.356415+0000","last_change":"2024-03-04T06:14:59.356415+0000","last_active":"2024-03-04T06:14:59.356415+0000","last_peered":"2024-03-04T06:14:59.356415+0000","last_clean":"2024-03-04T06:14:59.356415+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:14:59.356415+0000","last_undegraded":"2024-03-04T06:14:59.356415+0000","last_fullsized":"2024-03-04T06:14:59.356415+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6286115,"num_objects":31,"num_object_clones":23,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":165,"num_read_kb":130101,"num_write":158,"num_write_kb":44396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"421'151","reported_seq":1327,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.373927+0000","last_change":"2024-03-04T06:15:00.373927+0000","last_active":"2024-03-04T06:15:00.373927+0000","last_peered":"2024-03-04T06:15:00.373927+0000","last_clean":"2024-03-04T06:15:00.373927+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:15:00.373927+0000","last_undegraded":"2024-03-04T06:15:00.373927+0000","last_fullsized":"2024-03-04T06:15:00.373927+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3344758,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":43,"num_read_kb":17048,"num_write":65,"num_write_kb":17158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"427'404","reported_seq":1528,"reported_epoch":429,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:14:59.373494+0000","last_change":"2024-03-04T06:14:59.373494+0000","last_active":"2024-03-04T06:14:59.373494+0000","last_peered":"2024-03-04T06:14:59.373494+0000","last_clean":"2024-03-04T06:14:59.373494+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:14:59.373494+0000","last_undegraded":"2024-03-04T06:14:59.373494+0000","last_fullsized":"2024-03-04T06:14:59.373494+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033845,"num_objects":41,"num_object_clones":33,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":195,"num_read_kb":153970,"num_write":203,"num_write_kb":59706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1263,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.380907+0000","last_change":"2024-03-04T06:14:59.384006+0000","last_active":"2024-03-04T06:14:59.380907+0000","last_peered":"2024-03-04T06:14:59.380907+0000","last_clean":"2024-03-04T06:14:59.380907+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:14:59.380907+0000","last_undegraded":"2024-03-04T06:14:59.380907+0000","last_fullsized":"2024-03-04T06:14:59.380907+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"429'231","reported_seq":1417,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.349831+0000","last_change":"2024-03-04T06:14:59.496627+0000","last_active":"2024-03-04T06:15:00.349831+0000","last_peered":"2024-03-04T06:15:00.349831+0000","last_clean":"2024-03-04T06:15:00.349831+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:15:00.349831+0000","last_undegraded":"2024-03-04T06:15:00.349831+0000","last_fullsized":"2024-03-04T06:15:00.349831+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151692,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":145,"num_read_kb":133007,"num_write":108,"num_write_kb":28867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1191,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.362385+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:15:00.362385+0000","last_peered":"2024-03-04T06:15:00.362385+0000","last_clean":"2024-03-04T06:15:00.362385+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:15:00.362385+0000","last_undegraded":"2024-03-04T06:15:00.362385+0000","last_fullsized":"2024-03-04T06:15:00.362385+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.354390+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:15:00.354390+0000","last_peered":"2024-03-04T06:15:00.354390+0000","last_clean":"2024-03-04T06:15:00.354390+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:15:00.354390+0000","last_undegraded":"2024-03-04T06:15:00.354390+0000","last_fullsized":"2024-03-04T06:15:00.354390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"429'418","reported_seq":1500,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.353891+0000","last_change":"2024-03-04T06:14:59.496088+0000","last_active":"2024-03-04T06:15:00.353891+0000","last_peered":"2024-03-04T06:15:00.353891+0000","last_clean":"2024-03-04T06:15:00.353891+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:15:00.353891+0000","last_undegraded":"2024-03-04T06:15:00.353891+0000","last_fullsized":"2024-03-04T06:15:00.353891+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151494,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":188,"num_read_kb":134388,"num_write":185,"num_write_kb":49472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1067,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.367866+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:14:59.367866+0000","last_peered":"2024-03-04T06:14:59.367866+0000","last_clean":"2024-03-04T06:14:59.367866+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:14:59.367866+0000","last_undegraded":"2024-03-04T06:14:59.367866+0000","last_fullsized":"2024-03-04T06:14:59.367866+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"423'308","reported_seq":1421,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.359417+0000","last_change":"2024-03-04T06:15:00.359417+0000","last_active":"2024-03-04T06:15:00.359417+0000","last_peered":"2024-03-04T06:15:00.359417+0000","last_clean":"2024-03-04T06:15:00.359417+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:15:00.359417+0000","last_undegraded":"2024-03-04T06:15:00.359417+0000","last_fullsized":"2024-03-04T06:15:00.359417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3496021,"num_objects":28,"num_object_clones":23,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":147,"num_read_kb":118180,"num_write":116,"num_write_kb":32965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.347540+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:15:00.347540+0000","last_peered":"2024-03-04T06:15:00.347540+0000","last_clean":"2024-03-04T06:15:00.347540+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:15:00.347540+0000","last_undegraded":"2024-03-04T06:15:00.347540+0000","last_fullsized":"2024-03-04T06:15:00.347540+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"430'718","reported_seq":1771,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.487585+0000","last_change":"2024-03-04T06:15:00.369904+0000","last_active":"2024-03-04T06:15:00.487585+0000","last_peered":"2024-03-04T06:15:00.487585+0000","last_clean":"2024-03-04T06:15:00.487585+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:15:00.487585+0000","last_undegraded":"2024-03-04T06:15:00.487585+0000","last_fullsized":"2024-03-04T06:15:00.487585+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15379230,"num_objects":73,"num_object_clones":55,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":447,"num_read_kb":356462,"num_write":352,"num_write_kb":94527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1067,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.359983+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:14:59.359983+0000","last_peered":"2024-03-04T06:14:59.359983+0000","last_clean":"2024-03-04T06:14:59.359983+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:14:59.359983+0000","last_undegraded":"2024-03-04T06:14:59.359983+0000","last_fullsized":"2024-03-04T06:14:59.359983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"429'749","reported_seq":1707,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.342096+0000","last_change":"2024-03-04T06:14:59.576525+0000","last_active":"2024-03-04T06:15:00.342096+0000","last_peered":"2024-03-04T06:15:00.342096+0000","last_clean":"2024-03-04T06:15:00.342096+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:15:00.342096+0000","last_undegraded":"2024-03-04T06:15:00.342096+0000","last_fullsized":"2024-03-04T06:15:00.342096+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":749,"log_dups_size":0,"ondisk_log_size":749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12353871,"num_objects":68,"num_object_clones":52,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":288,"num_read_kb":198073,"num_write":327,"num_write_kb":84953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"429'349","reported_seq":1527,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.479189+0000","last_change":"2024-03-04T06:14:59.487685+0000","last_active":"2024-03-04T06:15:00.479189+0000","last_peered":"2024-03-04T06:15:00.479189+0000","last_clean":"2024-03-04T06:15:00.479189+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:15:00.479189+0000","last_undegraded":"2024-03-04T06:15:00.479189+0000","last_fullsized":"2024-03-04T06:15:00.479189+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4185339,"num_objects":37,"num_object_clones":27,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":201227,"num_write":174,"num_write_kb":47566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1093,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.364621+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:14:59.364621+0000","last_peered":"2024-03-04T06:14:59.364621+0000","last_clean":"2024-03-04T06:14:59.364621+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:14:59.364621+0000","last_undegraded":"2024-03-04T06:14:59.364621+0000","last_fullsized":"2024-03-04T06:14:59.364621+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"429'331","reported_seq":1512,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.374918+0000","last_change":"2024-03-04T06:14:59.469912+0000","last_active":"2024-03-04T06:14:59.374918+0000","last_peered":"2024-03-04T06:14:59.374918+0000","last_clean":"2024-03-04T06:14:59.374918+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:14:59.374918+0000","last_undegraded":"2024-03-04T06:14:59.374918+0000","last_fullsized":"2024-03-04T06:14:59.374918+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639293,"num_objects":37,"num_object_clones":25,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":190,"num_read_kb":130950,"num_write":156,"num_write_kb":42142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.376220+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:15:00.376220+0000","last_peered":"2024-03-04T06:15:00.376220+0000","last_clean":"2024-03-04T06:15:00.376220+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:15:00.376220+0000","last_undegraded":"2024-03-04T06:15:00.376220+0000","last_fullsized":"2024-03-04T06:15:00.376220+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"430'488","reported_seq":1585,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.492997+0000","last_change":"2024-03-04T06:15:00.357595+0000","last_active":"2024-03-04T06:15:00.492997+0000","last_peered":"2024-03-04T06:15:00.492997+0000","last_clean":"2024-03-04T06:15:00.492997+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:15:00.492997+0000","last_undegraded":"2024-03-04T06:15:00.492997+0000","last_fullsized":"2024-03-04T06:15:00.492997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9563711,"num_objects":49,"num_object_clones":38,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":281,"num_read_kb":215511,"num_write":234,"num_write_kb":65322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1094,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.359072+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:14:59.359072+0000","last_peered":"2024-03-04T06:14:59.359072+0000","last_clean":"2024-03-04T06:14:59.359072+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:14:59.359072+0000","last_undegraded":"2024-03-04T06:14:59.359072+0000","last_fullsized":"2024-03-04T06:14:59.359072+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"429'623","reported_seq":1687,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.345488+0000","last_change":"2024-03-04T06:14:59.530975+0000","last_active":"2024-03-04T06:15:00.345488+0000","last_peered":"2024-03-04T06:15:00.345488+0000","last_clean":"2024-03-04T06:15:00.345488+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:15:00.345488+0000","last_undegraded":"2024-03-04T06:15:00.345488+0000","last_fullsized":"2024-03-04T06:15:00.345488+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10017731,"num_objects":55,"num_object_clones":48,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":343,"num_read_kb":276440,"num_write":269,"num_write_kb":74209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.354684+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:15:00.354684+0000","last_peered":"2024-03-04T06:15:00.354684+0000","last_clean":"2024-03-04T06:15:00.354684+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:15:00.354684+0000","last_undegraded":"2024-03-04T06:15:00.354684+0000","last_fullsized":"2024-03-04T06:15:00.354684+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":99436086,"num_objects":619,"num_object_clones":475,"num_object_copies":1857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":110,"num_read":3244,"num_read_kb":2474923,"num_write":2861,"num_write_kb":781009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6114,"ondisk_log_size":6114,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706099,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":145136,"kb_used_data":100264,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94226704,"statfs":{"total":96636764160,"available":96488144896,"internally_reserved":0,"allocated":102670336,"data_stored":109897618,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706099,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123388,"kb_used_data":84084,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94248452,"statfs":{"total":96636764160,"available":96510414848,"internally_reserved":0,"allocated":86102016,"data_stored":92279557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706098,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":118860,"kb_used_data":78756,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94252980,"statfs":{"total":96636764160,"available":96515051520,"internally_reserved":0,"allocated":80646144,"data_stored":83690593,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706098,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":119624,"kb_used_data":79520,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94252216,"statfs":{"total":96636764160,"available":96514269184,"internally_reserved":0,"allocated":81428480,"data_stored":87257251,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:15:04.013 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:15:04.013 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:15:04.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:15:04.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:04.116+0000 7f3212a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:04.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:04.123+0000 7f3212a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:04.315 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:15:04.315 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:15:04.328 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":643,"stamp":"2024-03-04T06:15:03.240651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":100026454,"num_objects":621,"num_object_clones":475,"num_object_copies":1863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":110,"num_read":3347,"num_read_kb":2475014,"num_write":2998,"num_write_kb":782903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6211,"ondisk_log_size":6211,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":507008,"kb_used_data":342624,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":376980352,"statfs":{"total":386547056640,"available":386027880448,"internally_reserved":0,"allocated":350846976,"data_stored":373125019,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8370150,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":40,"num_read_kb":32774,"num_write":20,"num_write_kb":3721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.909533"},"pg_stats":[{"pgid":"3.f","version":"427'368","reported_seq":1512,"reported_epoch":429,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:14:59.374948+0000","last_change":"2024-03-04T06:14:59.374948+0000","last_active":"2024-03-04T06:14:59.374948+0000","last_peered":"2024-03-04T06:14:59.374948+0000","last_clean":"2024-03-04T06:14:59.374948+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:14:59.374948+0000","last_undegraded":"2024-03-04T06:14:59.374948+0000","last_fullsized":"2024-03-04T06:14:59.374948+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4992042,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":198,"num_read_kb":151919,"num_write":183,"num_write_kb":49844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"429'279","reported_seq":1451,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.470586+0000","last_change":"2024-03-04T06:14:59.470586+0000","last_active":"2024-03-04T06:14:59.470586+0000","last_peered":"2024-03-04T06:14:59.470586+0000","last_clean":"2024-03-04T06:14:59.470586+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:14:59.470586+0000","last_undegraded":"2024-03-04T06:14:59.470586+0000","last_fullsized":"2024-03-04T06:14:59.470586+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3059039,"num_objects":32,"num_object_clones":22,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":106,"num_read_kb":64809,"num_write":130,"num_write_kb":34377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"429'388","reported_seq":1544,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.345828+0000","last_change":"2024-03-04T06:14:59.495627+0000","last_active":"2024-03-04T06:15:00.345828+0000","last_peered":"2024-03-04T06:15:00.345828+0000","last_clean":"2024-03-04T06:15:00.345828+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:15:00.345828+0000","last_undegraded":"2024-03-04T06:15:00.345828+0000","last_fullsized":"2024-03-04T06:15:00.345828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5781905,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":252,"num_read_kb":192838,"num_write":201,"num_write_kb":55505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"423'309","reported_seq":1478,"reported_epoch":429,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:14:59.356415+0000","last_change":"2024-03-04T06:14:59.356415+0000","last_active":"2024-03-04T06:14:59.356415+0000","last_peered":"2024-03-04T06:14:59.356415+0000","last_clean":"2024-03-04T06:14:59.356415+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:14:59.356415+0000","last_undegraded":"2024-03-04T06:14:59.356415+0000","last_fullsized":"2024-03-04T06:14:59.356415+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6286115,"num_objects":31,"num_object_clones":23,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":165,"num_read_kb":130101,"num_write":158,"num_write_kb":44396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"421'151","reported_seq":1327,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.373927+0000","last_change":"2024-03-04T06:15:00.373927+0000","last_active":"2024-03-04T06:15:00.373927+0000","last_peered":"2024-03-04T06:15:00.373927+0000","last_clean":"2024-03-04T06:15:00.373927+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:15:00.373927+0000","last_undegraded":"2024-03-04T06:15:00.373927+0000","last_fullsized":"2024-03-04T06:15:00.373927+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3344758,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":43,"num_read_kb":17048,"num_write":65,"num_write_kb":17158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"427'404","reported_seq":1528,"reported_epoch":429,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:14:59.373494+0000","last_change":"2024-03-04T06:14:59.373494+0000","last_active":"2024-03-04T06:14:59.373494+0000","last_peered":"2024-03-04T06:14:59.373494+0000","last_clean":"2024-03-04T06:14:59.373494+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:14:59.373494+0000","last_undegraded":"2024-03-04T06:14:59.373494+0000","last_fullsized":"2024-03-04T06:14:59.373494+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033845,"num_objects":41,"num_object_clones":33,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":195,"num_read_kb":153970,"num_write":203,"num_write_kb":59706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1263,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.380907+0000","last_change":"2024-03-04T06:14:59.384006+0000","last_active":"2024-03-04T06:14:59.380907+0000","last_peered":"2024-03-04T06:14:59.380907+0000","last_clean":"2024-03-04T06:14:59.380907+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:14:59.380907+0000","last_undegraded":"2024-03-04T06:14:59.380907+0000","last_fullsized":"2024-03-04T06:14:59.380907+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"429'231","reported_seq":1417,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.349831+0000","last_change":"2024-03-04T06:14:59.496627+0000","last_active":"2024-03-04T06:15:00.349831+0000","last_peered":"2024-03-04T06:15:00.349831+0000","last_clean":"2024-03-04T06:15:00.349831+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:15:00.349831+0000","last_undegraded":"2024-03-04T06:15:00.349831+0000","last_fullsized":"2024-03-04T06:15:00.349831+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151692,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":145,"num_read_kb":133007,"num_write":108,"num_write_kb":28867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1191,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.362385+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:15:00.362385+0000","last_peered":"2024-03-04T06:15:00.362385+0000","last_clean":"2024-03-04T06:15:00.362385+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:15:00.362385+0000","last_undegraded":"2024-03-04T06:15:00.362385+0000","last_fullsized":"2024-03-04T06:15:00.362385+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.354390+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:15:00.354390+0000","last_peered":"2024-03-04T06:15:00.354390+0000","last_clean":"2024-03-04T06:15:00.354390+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:15:00.354390+0000","last_undegraded":"2024-03-04T06:15:00.354390+0000","last_fullsized":"2024-03-04T06:15:00.354390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"429'418","reported_seq":1500,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.353891+0000","last_change":"2024-03-04T06:14:59.496088+0000","last_active":"2024-03-04T06:15:00.353891+0000","last_peered":"2024-03-04T06:15:00.353891+0000","last_clean":"2024-03-04T06:15:00.353891+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:15:00.353891+0000","last_undegraded":"2024-03-04T06:15:00.353891+0000","last_fullsized":"2024-03-04T06:15:00.353891+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151494,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":188,"num_read_kb":134388,"num_write":185,"num_write_kb":49472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1067,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.367866+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:14:59.367866+0000","last_peered":"2024-03-04T06:14:59.367866+0000","last_clean":"2024-03-04T06:14:59.367866+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:14:59.367866+0000","last_undegraded":"2024-03-04T06:14:59.367866+0000","last_fullsized":"2024-03-04T06:14:59.367866+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"423'308","reported_seq":1421,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.359417+0000","last_change":"2024-03-04T06:15:00.359417+0000","last_active":"2024-03-04T06:15:00.359417+0000","last_peered":"2024-03-04T06:15:00.359417+0000","last_clean":"2024-03-04T06:15:00.359417+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:15:00.359417+0000","last_undegraded":"2024-03-04T06:15:00.359417+0000","last_fullsized":"2024-03-04T06:15:00.359417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3496021,"num_objects":28,"num_object_clones":23,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":147,"num_read_kb":118180,"num_write":116,"num_write_kb":32965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.347540+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:15:00.347540+0000","last_peered":"2024-03-04T06:15:00.347540+0000","last_clean":"2024-03-04T06:15:00.347540+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:15:00.347540+0000","last_undegraded":"2024-03-04T06:15:00.347540+0000","last_fullsized":"2024-03-04T06:15:00.347540+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"430'718","reported_seq":1771,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.487585+0000","last_change":"2024-03-04T06:15:00.369904+0000","last_active":"2024-03-04T06:15:00.487585+0000","last_peered":"2024-03-04T06:15:00.487585+0000","last_clean":"2024-03-04T06:15:00.487585+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:15:00.487585+0000","last_undegraded":"2024-03-04T06:15:00.487585+0000","last_fullsized":"2024-03-04T06:15:00.487585+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15379230,"num_objects":73,"num_object_clones":55,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":447,"num_read_kb":356462,"num_write":352,"num_write_kb":94527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1067,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.359983+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:14:59.359983+0000","last_peered":"2024-03-04T06:14:59.359983+0000","last_clean":"2024-03-04T06:14:59.359983+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:14:59.359983+0000","last_undegraded":"2024-03-04T06:14:59.359983+0000","last_fullsized":"2024-03-04T06:14:59.359983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"429'749","reported_seq":1707,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.342096+0000","last_change":"2024-03-04T06:14:59.576525+0000","last_active":"2024-03-04T06:15:00.342096+0000","last_peered":"2024-03-04T06:15:00.342096+0000","last_clean":"2024-03-04T06:15:00.342096+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:15:00.342096+0000","last_undegraded":"2024-03-04T06:15:00.342096+0000","last_fullsized":"2024-03-04T06:15:00.342096+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":749,"log_dups_size":0,"ondisk_log_size":749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12353871,"num_objects":68,"num_object_clones":52,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":288,"num_read_kb":198073,"num_write":327,"num_write_kb":84953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"429'349","reported_seq":1527,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.479189+0000","last_change":"2024-03-04T06:14:59.487685+0000","last_active":"2024-03-04T06:15:00.479189+0000","last_peered":"2024-03-04T06:15:00.479189+0000","last_clean":"2024-03-04T06:15:00.479189+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:15:00.479189+0000","last_undegraded":"2024-03-04T06:15:00.479189+0000","last_fullsized":"2024-03-04T06:15:00.479189+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4185339,"num_objects":37,"num_object_clones":27,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":201227,"num_write":174,"num_write_kb":47566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1093,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.364621+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:14:59.364621+0000","last_peered":"2024-03-04T06:14:59.364621+0000","last_clean":"2024-03-04T06:14:59.364621+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:14:59.364621+0000","last_undegraded":"2024-03-04T06:14:59.364621+0000","last_fullsized":"2024-03-04T06:14:59.364621+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"429'331","reported_seq":1512,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.374918+0000","last_change":"2024-03-04T06:14:59.469912+0000","last_active":"2024-03-04T06:14:59.374918+0000","last_peered":"2024-03-04T06:14:59.374918+0000","last_clean":"2024-03-04T06:14:59.374918+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:14:59.374918+0000","last_undegraded":"2024-03-04T06:14:59.374918+0000","last_fullsized":"2024-03-04T06:14:59.374918+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639293,"num_objects":37,"num_object_clones":25,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":190,"num_read_kb":130950,"num_write":156,"num_write_kb":42142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.376220+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:15:00.376220+0000","last_peered":"2024-03-04T06:15:00.376220+0000","last_clean":"2024-03-04T06:15:00.376220+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:15:00.376220+0000","last_undegraded":"2024-03-04T06:15:00.376220+0000","last_fullsized":"2024-03-04T06:15:00.376220+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"430'488","reported_seq":1585,"reported_epoch":430,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:00.492997+0000","last_change":"2024-03-04T06:15:00.357595+0000","last_active":"2024-03-04T06:15:00.492997+0000","last_peered":"2024-03-04T06:15:00.492997+0000","last_clean":"2024-03-04T06:15:00.492997+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:15:00.492997+0000","last_undegraded":"2024-03-04T06:15:00.492997+0000","last_fullsized":"2024-03-04T06:15:00.492997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9563711,"num_objects":49,"num_object_clones":38,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":281,"num_read_kb":215511,"num_write":234,"num_write_kb":65322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1094,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:14:59.359072+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:14:59.359072+0000","last_peered":"2024-03-04T06:14:59.359072+0000","last_clean":"2024-03-04T06:14:59.359072+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:14:59.359072+0000","last_undegraded":"2024-03-04T06:14:59.359072+0000","last_fullsized":"2024-03-04T06:14:59.359072+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"429'623","reported_seq":1687,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.345488+0000","last_change":"2024-03-04T06:14:59.530975+0000","last_active":"2024-03-04T06:15:00.345488+0000","last_peered":"2024-03-04T06:15:00.345488+0000","last_clean":"2024-03-04T06:15:00.345488+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:15:00.345488+0000","last_undegraded":"2024-03-04T06:15:00.345488+0000","last_fullsized":"2024-03-04T06:15:00.345488+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10017731,"num_objects":55,"num_object_clones":48,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":343,"num_read_kb":276440,"num_write":269,"num_write_kb":74209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1058,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-04T06:15:00.354684+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:15:00.354684+0000","last_peered":"2024-03-04T06:15:00.354684+0000","last_clean":"2024-03-04T06:15:00.354684+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:15:00.354684+0000","last_undegraded":"2024-03-04T06:15:00.354684+0000","last_fullsized":"2024-03-04T06:15:00.354684+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":99436086,"num_objects":619,"num_object_clones":475,"num_object_copies":1857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":110,"num_read":3244,"num_read_kb":2474923,"num_write":2861,"num_write_kb":781009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6114,"ondisk_log_size":6114,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706099,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":145136,"kb_used_data":100264,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94226704,"statfs":{"total":96636764160,"available":96488144896,"internally_reserved":0,"allocated":102670336,"data_stored":109897618,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706099,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123388,"kb_used_data":84084,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94248452,"statfs":{"total":96636764160,"available":96510414848,"internally_reserved":0,"allocated":86102016,"data_stored":92279557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706098,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":118860,"kb_used_data":78756,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94252980,"statfs":{"total":96636764160,"available":96515051520,"internally_reserved":0,"allocated":80646144,"data_stored":83690593,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706098,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":119624,"kb_used_data":79520,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94252216,"statfs":{"total":96636764160,"available":96514269184,"internally_reserved":0,"allocated":81428480,"data_stored":87257251,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:15:04.348 INFO:tasks.rados.rados.0.smithi064.stdout:1802: done (2 left) 2024-03-04T06:15:04.348 INFO:tasks.rados.rados.0.smithi064.stdout:1803: done (1 left) 2024-03-04T06:15:04.348 INFO:tasks.rados.rados.0.smithi064.stdout:1805: done (0 left) 2024-03-04T06:15:04.348 INFO:tasks.rados.rados.0.smithi064.stdout:1806: write (excl) oid 32 current snap is 209 2024-03-04T06:15:04.348 INFO:tasks.rados.rados.0.smithi064.stdout:1806: seq_num 868 ranges {588476=580374,1643429=603638} 2024-03-04T06:15:04.354 INFO:tasks.rados.rados.0.smithi064.stdout:1806: writing smithi06431743-32 from 588476 to 1168850 tid 1 2024-03-04T06:15:04.360 INFO:tasks.rados.rados.0.smithi064.stdout:1806: writing smithi06431743-32 from 1643429 to 2247067 tid 2 2024-03-04T06:15:04.360 INFO:tasks.rados.rados.0.smithi064.stdout:1807: write oid 7 current snap is 209 2024-03-04T06:15:04.360 INFO:tasks.rados.rados.0.smithi064.stdout:1807: seq_num 869 ranges {605283=655623,1701911=747296} 2024-03-04T06:15:04.366 INFO:tasks.rados.rados.0.smithi064.stdout:1807: writing smithi06431743-7 from 605283 to 1260906 tid 1 2024-03-04T06:15:04.373 INFO:tasks.rados.rados.0.smithi064.stdout:1807: writing smithi06431743-7 from 1701911 to 2449207 tid 2 2024-03-04T06:15:04.373 INFO:tasks.rados.rados.0.smithi064.stdout:1808: delete oid 27 current snap is 209 2024-03-04T06:15:04.417 INFO:tasks.rados.rados.0.smithi064.stdout:1808: done (2 left) 2024-03-04T06:15:04.417 INFO:tasks.rados.rados.0.smithi064.stdout:1809: read oid 19 snap -1 2024-03-04T06:15:04.417 INFO:tasks.rados.rados.0.smithi064.stdout:1809: expect (ObjNum 826 snap 204 seq_num 826) 2024-03-04T06:15:04.417 INFO:tasks.rados.rados.0.smithi064.stdout:1810: read oid 47 snap -1 2024-03-04T06:15:04.417 INFO:tasks.rados.rados.0.smithi064.stdout:1810: expect (ObjNum 862 snap 207 seq_num 862) 2024-03-04T06:15:04.418 INFO:tasks.rados.rados.0.smithi064.stdout:1811: write (excl) oid 13 current snap is 209 2024-03-04T06:15:04.418 INFO:tasks.rados.rados.0.smithi064.stdout:1811: seq_num 871 ranges {638897=406121,1818875=750965} 2024-03-04T06:15:04.424 INFO:tasks.rados.rados.0.smithi064.stdout:1811: writing smithi06431743-13 from 638897 to 1045018 tid 1 2024-03-04T06:15:04.434 INFO:tasks.rados.rados.0.smithi064.stdout:1811: writing smithi06431743-13 from 1818875 to 2569840 tid 2 2024-03-04T06:15:04.434 INFO:tasks.rados.rados.0.smithi064.stdout:1812: write oid 20 current snap is 209 2024-03-04T06:15:04.434 INFO:tasks.rados.rados.0.smithi064.stdout:1807: finishing write tid 1 to smithi06431743-7 2024-03-04T06:15:04.434 INFO:tasks.rados.rados.0.smithi064.stdout:1812: seq_num 872 ranges {655704=481370,1761004=494623} 2024-03-04T06:15:04.441 INFO:tasks.rados.rados.0.smithi064.stdout:1812: writing smithi06431743-20 from 655704 to 1137074 tid 1 2024-03-04T06:15:04.449 INFO:tasks.rados.rados.0.smithi064.stdout:1812: writing smithi06431743-20 from 1761004 to 2255627 tid 2 2024-03-04T06:15:04.449 INFO:tasks.rados.rados.0.smithi064.stdout:1813: write oid 26 current snap is 209 2024-03-04T06:15:04.449 INFO:tasks.rados.rados.0.smithi064.stdout:1806: finishing write tid 1 to smithi06431743-32 2024-03-04T06:15:04.449 INFO:tasks.rados.rados.0.smithi064.stdout:1813: seq_num 873 ranges {672511=556619,1819486=754634} 2024-03-04T06:15:04.458 INFO:tasks.rados.rados.0.smithi064.stdout:1813: writing smithi06431743-26 from 672511 to 1229130 tid 1 2024-03-04T06:15:04.468 INFO:tasks.rados.rados.0.smithi064.stdout:1813: writing smithi06431743-26 from 1819486 to 2574120 tid 2 2024-03-04T06:15:04.468 INFO:tasks.rados.rados.0.smithi064.stdout:1814: write (excl) oid 4 current snap is 209 2024-03-04T06:15:04.469 INFO:tasks.rados.rados.0.smithi064.stdout:1806: finishing write tid 2 to smithi06431743-32 2024-03-04T06:15:04.469 INFO:tasks.rados.rados.0.smithi064.stdout:1807: finishing write tid 2 to smithi06431743-7 2024-03-04T06:15:04.469 INFO:tasks.rados.rados.0.smithi064.stdout:1809: expect (ObjNum 826 snap 204 seq_num 826) 2024-03-04T06:15:04.718 INFO:tasks.rados.rados.0.smithi064.stdout:1806: finishing write tid 3 to smithi06431743-32 2024-03-04T06:15:04.718 INFO:tasks.rados.rados.0.smithi064.stdout:1806: finishing write tid 4 to smithi06431743-32 2024-03-04T06:15:04.718 INFO:tasks.rados.rados.0.smithi064.stdout:1806: oid 32 updating version 0 to 725 2024-03-04T06:15:04.718 INFO:tasks.rados.rados.0.smithi064.stdout:1806: oid 32 version 725 is already newer than 725 2024-03-04T06:15:04.718 INFO:tasks.rados.rados.0.smithi064.stdout:1806: oid 32 version 725 is already newer than 723 2024-03-04T06:15:04.718 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 725 (ObjNum 868 snap 209 seq_num 868) dirty exists 2024-03-04T06:15:04.719 INFO:tasks.rados.rados.0.smithi064.stdout:1806: left oid 32 (ObjNum 868 snap 209 seq_num 868) 2024-03-04T06:15:04.719 INFO:tasks.rados.rados.0.smithi064.stdout:1807: finishing write tid 3 to smithi06431743-7 2024-03-04T06:15:04.719 INFO:tasks.rados.rados.0.smithi064.stdout:1814: seq_num 874 ranges {689318=631868,1877968=498292} 2024-03-04T06:15:04.725 INFO:tasks.rados.rados.0.smithi064.stdout:1814: writing smithi06431743-4 from 689318 to 1321186 tid 1 2024-03-04T06:15:04.732 INFO:tasks.rados.rados.0.smithi064.stdout:1814: writing smithi06431743-4 from 1877968 to 2376260 tid 2 2024-03-04T06:15:04.732 INFO:tasks.rados.rados.0.smithi064.stdout:1806: done (7 left) 2024-03-04T06:15:04.732 INFO:tasks.rados.rados.0.smithi064.stdout:1809: done (6 left) 2024-03-04T06:15:04.732 INFO:tasks.rados.rados.0.smithi064.stdout:1815: delete oid 30 current snap is 209 2024-03-04T06:15:04.732 INFO:tasks.rados.rados.0.smithi064.stdout:1807: finishing write tid 4 to smithi06431743-7 2024-03-04T06:15:04.732 INFO:tasks.rados.rados.0.smithi064.stdout:1807: oid 7 updating version 0 to 316 2024-03-04T06:15:04.732 INFO:tasks.rados.rados.0.smithi064.stdout:1807: oid 7 version 316 is already newer than 315 2024-03-04T06:15:04.732 INFO:tasks.rados.rados.0.smithi064.stdout:1807: oid 7 updating version 316 to 317 2024-03-04T06:15:04.732 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 317 (ObjNum 869 snap 209 seq_num 869) dirty exists 2024-03-04T06:15:04.732 INFO:tasks.rados.rados.0.smithi064.stdout:1807: left oid 7 (ObjNum 869 snap 209 seq_num 869) 2024-03-04T06:15:04.732 INFO:tasks.rados.rados.0.smithi064.stdout:1810: expect (ObjNum 862 snap 207 seq_num 862) 2024-03-04T06:15:04.825 INFO:tasks.rados.rados.0.smithi064.stdout:1812: finishing write tid 1 to smithi06431743-20 2024-03-04T06:15:04.825 INFO:tasks.rados.rados.0.smithi064.stdout:1811: finishing write tid 1 to smithi06431743-13 2024-03-04T06:15:04.825 INFO:tasks.rados.rados.0.smithi064.stdout:1813: finishing write tid 1 to smithi06431743-26 2024-03-04T06:15:04.825 INFO:tasks.rados.rados.0.smithi064.stdout:1812: finishing write tid 2 to smithi06431743-20 2024-03-04T06:15:04.825 INFO:tasks.rados.rados.0.smithi064.stdout:1812: finishing write tid 3 to smithi06431743-20 2024-03-04T06:15:04.825 INFO:tasks.rados.rados.0.smithi064.stdout:1811: finishing write tid 2 to smithi06431743-13 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:1812: finishing write tid 4 to smithi06431743-20 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:1812: oid 20 updating version 0 to 502 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:1812: oid 20 version 502 is already newer than 500 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:1812: oid 20 version 502 is already newer than 501 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 502 (ObjNum 872 snap 209 seq_num 872) dirty exists 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:1812: left oid 20 (ObjNum 872 snap 209 seq_num 872) 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:1811: finishing write tid 3 to smithi06431743-13 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:1807: done (6 left) 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:1810: done (5 left) 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:1812: done (4 left) 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:1815: done (3 left) 2024-03-04T06:15:04.826 INFO:tasks.rados.rados.0.smithi064.stdout:1816: snap_remove snap 199 2024-03-04T06:15:05.353 INFO:tasks.rados.rados.0.smithi064.stdout:1816: done (3 left) 2024-03-04T06:15:05.353 INFO:tasks.rados.rados.0.smithi064.stdout:1817: snap_create 2024-03-04T06:15:05.353 INFO:tasks.rados.rados.0.smithi064.stdout:1811: finishing write tid 4 to smithi06431743-13 2024-03-04T06:15:05.353 INFO:tasks.rados.rados.0.smithi064.stdout:1811: oid 13 updating version 0 to 751 2024-03-04T06:15:05.353 INFO:tasks.rados.rados.0.smithi064.stdout:1811: oid 13 updating version 751 to 753 2024-03-04T06:15:05.353 INFO:tasks.rados.rados.0.smithi064.stdout:1811: oid 13 version 753 is already newer than 752 2024-03-04T06:15:05.353 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 753 (ObjNum 871 snap 209 seq_num 871) dirty exists 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1811: left oid 13 (ObjNum 871 snap 209 seq_num 871) 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1813: finishing write tid 2 to smithi06431743-26 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1813: finishing write tid 3 to smithi06431743-26 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1813: finishing write tid 4 to smithi06431743-26 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1813: oid 26 updating version 0 to 728 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1813: oid 26 version 728 is already newer than 727 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1813: oid 26 updating version 728 to 729 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 729 (ObjNum 873 snap 209 seq_num 873) dirty exists 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1813: left oid 26 (ObjNum 873 snap 209 seq_num 873) 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1814: finishing write tid 1 to smithi06431743-4 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1814: finishing write tid 2 to smithi06431743-4 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1814: finishing write tid 3 to smithi06431743-4 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1814: finishing write tid 4 to smithi06431743-4 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1814: oid 4 updating version 0 to 755 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1814: oid 4 updating version 755 to 756 2024-03-04T06:15:05.354 INFO:tasks.rados.rados.0.smithi064.stdout:1814: oid 4 version 756 is already newer than 754 2024-03-04T06:15:05.355 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 756 (ObjNum 874 snap 209 seq_num 874) dirty exists 2024-03-04T06:15:05.355 INFO:tasks.rados.rados.0.smithi064.stdout:1814: left oid 4 (ObjNum 874 snap 209 seq_num 874) 2024-03-04T06:15:05.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:05.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:05.541+0000 7fc67a79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:05.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:05.545+0000 7fc67a79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:06.354 INFO:tasks.rados.rados.0.smithi064.stdout:1811: done (3 left) 2024-03-04T06:15:06.354 INFO:tasks.rados.rados.0.smithi064.stdout:1813: done (2 left) 2024-03-04T06:15:06.354 INFO:tasks.rados.rados.0.smithi064.stdout:1814: done (1 left) 2024-03-04T06:15:06.355 INFO:tasks.rados.rados.0.smithi064.stdout:1817: done (0 left) 2024-03-04T06:15:06.355 INFO:tasks.rados.rados.0.smithi064.stdout:1818: read oid 41 snap -1 2024-03-04T06:15:06.355 INFO:tasks.rados.rados.0.smithi064.stdout:1818: expect deleted 2024-03-04T06:15:06.355 INFO:tasks.rados.rados.0.smithi064.stdout:1819: snap_remove snap 208 2024-03-04T06:15:07.360 INFO:tasks.rados.rados.0.smithi064.stdout:1819: done (1 left) 2024-03-04T06:15:07.360 INFO:tasks.rados.rados.0.smithi064.stdout:1820: delete oid 31 current snap is 210 2024-03-04T06:15:07.421 INFO:tasks.rados.rados.0.smithi064.stdout:1818: done (1 left) 2024-03-04T06:15:07.421 INFO:tasks.rados.rados.0.smithi064.stdout:1820: done (0 left) 2024-03-04T06:15:07.421 INFO:tasks.rados.rados.0.smithi064.stdout:1821: snap_create 2024-03-04T06:15:07.613 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:07.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:07.730+0000 7f6bb0edd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:07.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:07.735+0000 7f6bb0edd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:08.362 INFO:tasks.rados.rados.0.smithi064.stdout:1821: done (0 left) 2024-03-04T06:15:08.362 INFO:tasks.rados.rados.0.smithi064.stdout:1822: delete oid 2 current snap is 211 2024-03-04T06:15:08.443 INFO:tasks.rados.rados.0.smithi064.stdout:1822: done (0 left) 2024-03-04T06:15:08.443 INFO:tasks.rados.rados.0.smithi064.stdout:1823: snap_remove snap 209 2024-03-04T06:15:09.330 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:15:09.330 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:15:09.331 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:15:09.368 INFO:tasks.rados.rados.0.smithi064.stdout:1823: done (0 left) 2024-03-04T06:15:09.368 INFO:tasks.rados.rados.0.smithi064.stdout:1824: read oid 49 snap -1 2024-03-04T06:15:09.368 INFO:tasks.rados.rados.0.smithi064.stdout:1824: expect (ObjNum 785 snap 193 seq_num 785) 2024-03-04T06:15:09.369 INFO:tasks.rados.rados.0.smithi064.stdout:1825: read oid 46 snap -1 2024-03-04T06:15:09.369 INFO:tasks.rados.rados.0.smithi064.stdout:1825: expect (ObjNum 757 snap 188 seq_num 757) 2024-03-04T06:15:09.369 INFO:tasks.rados.rados.0.smithi064.stdout:1826: write oid 35 current snap is 211 2024-03-04T06:15:09.369 INFO:tasks.rados.rados.0.smithi064.stdout:1826: seq_num 878 ranges {756546=649217,2111896=505630} 2024-03-04T06:15:09.376 INFO:tasks.rados.rados.0.smithi064.stdout:1826: writing smithi06431743-35 from 756546 to 1405763 tid 1 2024-03-04T06:15:09.383 INFO:tasks.rados.rados.0.smithi064.stdout:1826: writing smithi06431743-35 from 2111896 to 2617526 tid 2 2024-03-04T06:15:09.383 INFO:tasks.rados.rados.0.smithi064.stdout:1827: write oid 23 current snap is 211 2024-03-04T06:15:09.383 INFO:tasks.rados.rados.0.smithi064.stdout:1827: seq_num 879 ranges {773353=724466,2170378=602975} 2024-03-04T06:15:09.391 INFO:tasks.rados.rados.0.smithi064.stdout:1827: writing smithi06431743-23 from 773353 to 1497819 tid 1 2024-03-04T06:15:09.398 INFO:tasks.rados.rados.0.smithi064.stdout:1827: writing smithi06431743-23 from 2170378 to 2773353 tid 2 2024-03-04T06:15:09.398 INFO:tasks.rados.rados.0.smithi064.stdout:1828: delete oid 3 current snap is 211 2024-03-04T06:15:09.432 INFO:tasks.rados.rados.0.smithi064.stdout:1825: expect (ObjNum 757 snap 188 seq_num 757) 2024-03-04T06:15:09.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:09.476+0000 7f2cc5d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:09.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:09.484+0000 7f2cc5d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:09.629 INFO:tasks.rados.rados.0.smithi064.stdout:1826: finishing write tid 1 to smithi06431743-35 2024-03-04T06:15:09.629 INFO:tasks.rados.rados.0.smithi064.stdout:1827: finishing write tid 1 to smithi06431743-23 2024-03-04T06:15:09.629 INFO:tasks.rados.rados.0.smithi064.stdout:1826: finishing write tid 2 to smithi06431743-35 2024-03-04T06:15:09.629 INFO:tasks.rados.rados.0.smithi064.stdout:1824: expect (ObjNum 785 snap 193 seq_num 785) 2024-03-04T06:15:09.680 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:15:09.681 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:15:09.692 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":652,"stamp":"2024-03-04T06:15:09.365849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":102144994,"num_objects":635,"num_object_clones":485,"num_object_copies":1905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":111,"num_read":3409,"num_read_kb":2520364,"num_write":3059,"num_write_kb":799574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6336,"ondisk_log_size":6336,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":488836,"kb_used_data":324336,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":376998524,"statfs":{"total":386547056640,"available":386046488576,"internally_reserved":0,"allocated":332120064,"data_stored":345835954,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":403698,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":15,"num_read_kb":6826,"num_write":26,"num_write_kb":7240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014114"},"pg_stats":[{"pgid":"3.f","version":"431'381","reported_seq":1535,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.451697+0000","last_change":"2024-03-04T06:15:03.450897+0000","last_active":"2024-03-04T06:15:04.451697+0000","last_peered":"2024-03-04T06:15:04.451697+0000","last_clean":"2024-03-04T06:15:04.451697+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:15:04.451697+0000","last_undegraded":"2024-03-04T06:15:04.451697+0000","last_fullsized":"2024-03-04T06:15:04.451697+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462869,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":200,"num_read_kb":151920,"num_write":187,"num_write_kb":51268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"431'282","reported_seq":1474,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.378039+0000","last_change":"2024-03-04T06:15:04.378039+0000","last_active":"2024-03-04T06:15:04.378039+0000","last_peered":"2024-03-04T06:15:04.378039+0000","last_clean":"2024-03-04T06:15:04.378039+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:15:04.378039+0000","last_undegraded":"2024-03-04T06:15:04.378039+0000","last_fullsized":"2024-03-04T06:15:04.378039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2386792,"num_objects":33,"num_object_clones":23,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":116,"num_read_kb":74956,"num_write":135,"num_write_kb":35524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"435'396","reported_seq":1565,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.993789+0000","last_change":"2024-03-04T06:15:05.993789+0000","last_active":"2024-03-04T06:15:05.993789+0000","last_peered":"2024-03-04T06:15:05.993789+0000","last_clean":"2024-03-04T06:15:05.993789+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:15:05.993789+0000","last_undegraded":"2024-03-04T06:15:05.993789+0000","last_fullsized":"2024-03-04T06:15:05.993789+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":396,"log_dups_size":0,"ondisk_log_size":396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5781905,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":259,"num_read_kb":199658,"num_write":201,"num_write_kb":55505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"431'314","reported_seq":1504,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.477406+0000","last_change":"2024-03-04T06:15:03.440427+0000","last_active":"2024-03-04T06:15:04.477406+0000","last_peered":"2024-03-04T06:15:04.477406+0000","last_clean":"2024-03-04T06:15:04.477406+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:15:04.477406+0000","last_undegraded":"2024-03-04T06:15:04.477406+0000","last_fullsized":"2024-03-04T06:15:04.477406+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8773749,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":172,"num_read_kb":135944,"num_write":162,"num_write_kb":45378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"435'159","reported_seq":1351,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.373461+0000","last_change":"2024-03-04T06:15:05.403404+0000","last_active":"2024-03-04T06:15:05.373461+0000","last_peered":"2024-03-04T06:15:05.373461+0000","last_clean":"2024-03-04T06:15:05.373461+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:15:05.373461+0000","last_undegraded":"2024-03-04T06:15:05.373461+0000","last_fullsized":"2024-03-04T06:15:05.373461+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2538055,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":45,"num_read_kb":17049,"num_write":69,"num_write_kb":18437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"433'420","reported_seq":1555,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.452984+0000","last_change":"2024-03-04T06:15:03.446193+0000","last_active":"2024-03-04T06:15:04.452984+0000","last_peered":"2024-03-04T06:15:04.452984+0000","last_clean":"2024-03-04T06:15:04.452984+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:15:04.452984+0000","last_undegraded":"2024-03-04T06:15:04.452984+0000","last_fullsized":"2024-03-04T06:15:04.452984+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588707,"num_objects":41,"num_object_clones":33,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":199,"num_read_kb":153972,"num_write":211,"num_write_kb":62125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1280,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.449256+0000","last_change":"2024-03-04T06:15:03.447665+0000","last_active":"2024-03-04T06:15:04.449256+0000","last_peered":"2024-03-04T06:15:04.449256+0000","last_clean":"2024-03-04T06:15:04.449256+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:15:04.449256+0000","last_undegraded":"2024-03-04T06:15:04.449256+0000","last_fullsized":"2024-03-04T06:15:04.449256+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"429'231","reported_seq":1436,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-04T06:15:06.359578+0000","last_change":"2024-03-04T06:15:05.901681+0000","last_active":"2024-03-04T06:15:06.359578+0000","last_peered":"2024-03-04T06:15:06.359578+0000","last_clean":"2024-03-04T06:15:06.359578+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:15:06.359578+0000","last_undegraded":"2024-03-04T06:15:06.359578+0000","last_fullsized":"2024-03-04T06:15:06.359578+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151692,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":145,"num_read_kb":133007,"num_write":108,"num_write_kb":28867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1206,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.360037+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:15:05.360037+0000","last_peered":"2024-03-04T06:15:05.360037+0000","last_clean":"2024-03-04T06:15:05.360037+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:15:05.360037+0000","last_undegraded":"2024-03-04T06:15:05.360037+0000","last_fullsized":"2024-03-04T06:15:05.360037+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1071,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.908069+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:15:05.908069+0000","last_peered":"2024-03-04T06:15:05.908069+0000","last_clean":"2024-03-04T06:15:05.908069+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:15:05.908069+0000","last_undegraded":"2024-03-04T06:15:05.908069+0000","last_fullsized":"2024-03-04T06:15:05.908069+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"435'423","reported_seq":1520,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.900598+0000","last_change":"2024-03-04T06:15:05.979191+0000","last_active":"2024-03-04T06:15:05.900598+0000","last_peered":"2024-03-04T06:15:05.900598+0000","last_clean":"2024-03-04T06:15:05.900598+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:15:05.900598+0000","last_undegraded":"2024-03-04T06:15:05.900598+0000","last_fullsized":"2024-03-04T06:15:05.900598+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4655935,"num_objects":42,"num_object_clones":32,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":189,"num_read_kb":134389,"num_write":189,"num_write_kb":50669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1082,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.439906+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:15:04.439906+0000","last_peered":"2024-03-04T06:15:04.439906+0000","last_clean":"2024-03-04T06:15:04.439906+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:15:04.439906+0000","last_undegraded":"2024-03-04T06:15:04.439906+0000","last_fullsized":"2024-03-04T06:15:04.439906+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"435'319","reported_seq":1446,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.362783+0000","last_change":"2024-03-04T06:15:05.417660+0000","last_active":"2024-03-04T06:15:05.362783+0000","last_peered":"2024-03-04T06:15:05.362783+0000","last_clean":"2024-03-04T06:15:05.362783+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:15:05.362783+0000","last_undegraded":"2024-03-04T06:15:05.362783+0000","last_fullsized":"2024-03-04T06:15:05.362783+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605283,"num_objects":29,"num_object_clones":24,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":148,"num_read_kb":118181,"num_write":121,"num_write_kb":34336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1072,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.894295+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:15:05.894295+0000","last_peered":"2024-03-04T06:15:05.894295+0000","last_clean":"2024-03-04T06:15:05.894295+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:15:05.894295+0000","last_undegraded":"2024-03-04T06:15:05.894295+0000","last_fullsized":"2024-03-04T06:15:05.894295+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"435'735","reported_seq":1800,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.374704+0000","last_change":"2024-03-04T06:15:05.437984+0000","last_active":"2024-03-04T06:15:05.374704+0000","last_peered":"2024-03-04T06:15:05.374704+0000","last_clean":"2024-03-04T06:15:05.374704+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:15:05.374704+0000","last_undegraded":"2024-03-04T06:15:05.374704+0000","last_fullsized":"2024-03-04T06:15:05.374704+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":735,"log_dups_size":0,"ondisk_log_size":735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14690209,"num_objects":76,"num_object_clones":57,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":450,"num_read_kb":356464,"num_write":361,"num_write_kb":96965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1082,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.439936+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:15:04.439936+0000","last_peered":"2024-03-04T06:15:04.439936+0000","last_clean":"2024-03-04T06:15:04.439936+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:15:04.439936+0000","last_undegraded":"2024-03-04T06:15:04.439936+0000","last_fullsized":"2024-03-04T06:15:04.439936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"434'756","reported_seq":1737,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.762385+0000","last_change":"2024-03-04T06:15:04.376537+0000","last_active":"2024-03-04T06:15:04.762385+0000","last_peered":"2024-03-04T06:15:04.762385+0000","last_clean":"2024-03-04T06:15:04.762385+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:15:04.762385+0000","last_undegraded":"2024-03-04T06:15:04.762385+0000","last_fullsized":"2024-03-04T06:15:04.762385+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11496747,"num_objects":71,"num_object_clones":54,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":310,"num_read_kb":220603,"num_write":336,"num_write_kb":87189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"435'355","reported_seq":1543,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.910430+0000","last_change":"2024-03-04T06:15:06.026792+0000","last_active":"2024-03-04T06:15:05.910430+0000","last_peered":"2024-03-04T06:15:05.910430+0000","last_clean":"2024-03-04T06:15:05.910430+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:15:05.910430+0000","last_undegraded":"2024-03-04T06:15:05.910430+0000","last_fullsized":"2024-03-04T06:15:05.910430+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4185339,"num_objects":37,"num_object_clones":27,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":201227,"num_write":174,"num_write_kb":47566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1106,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.378297+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:15:04.378297+0000","last_peered":"2024-03-04T06:15:04.378297+0000","last_clean":"2024-03-04T06:15:04.378297+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:15:04.378297+0000","last_undegraded":"2024-03-04T06:15:04.378297+0000","last_fullsized":"2024-03-04T06:15:04.378297+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"431'339","reported_seq":1532,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.381066+0000","last_change":"2024-03-04T06:15:04.396456+0000","last_active":"2024-03-04T06:15:04.381066+0000","last_peered":"2024-03-04T06:15:04.381066+0000","last_clean":"2024-03-04T06:15:04.381066+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:15:04.381066+0000","last_undegraded":"2024-03-04T06:15:04.381066+0000","last_fullsized":"2024-03-04T06:15:04.381066+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6370183,"num_objects":39,"num_object_clones":27,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":192,"num_read_kb":130951,"num_write":161,"num_write_kb":43365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1073,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.377631+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:15:05.377631+0000","last_peered":"2024-03-04T06:15:05.377631+0000","last_clean":"2024-03-04T06:15:05.377631+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:15:05.377631+0000","last_undegraded":"2024-03-04T06:15:05.377631+0000","last_fullsized":"2024-03-04T06:15:05.377631+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"435'506","reported_seq":1609,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.364754+0000","last_change":"2024-03-04T06:15:05.425176+0000","last_active":"2024-03-04T06:15:05.364754+0000","last_peered":"2024-03-04T06:15:05.364754+0000","last_clean":"2024-03-04T06:15:05.364754+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:15:05.364754+0000","last_undegraded":"2024-03-04T06:15:05.364754+0000","last_fullsized":"2024-03-04T06:15:05.364754+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9849430,"num_objects":50,"num_object_clones":39,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":282,"num_read_kb":215512,"num_write":238,"num_write_kb":66277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1106,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.369358+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:15:04.369358+0000","last_peered":"2024-03-04T06:15:04.369358+0000","last_clean":"2024-03-04T06:15:04.369358+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:15:04.369358+0000","last_undegraded":"2024-03-04T06:15:04.369358+0000","last_fullsized":"2024-03-04T06:15:04.369358+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"429'623","reported_seq":1699,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-04T06:15:04.363066+0000","last_change":"2024-03-04T06:15:04.377739+0000","last_active":"2024-03-04T06:15:04.363066+0000","last_peered":"2024-03-04T06:15:04.363066+0000","last_clean":"2024-03-04T06:15:04.363066+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:15:04.363066+0000","last_undegraded":"2024-03-04T06:15:04.363066+0000","last_fullsized":"2024-03-04T06:15:04.363066+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10017731,"num_objects":55,"num_object_clones":48,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":343,"num_read_kb":276440,"num_write":269,"num_write_kb":74209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1071,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-04T06:15:05.903934+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:15:05.903934+0000","last_peered":"2024-03-04T06:15:05.903934+0000","last_clean":"2024-03-04T06:15:05.903934+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:15:05.903934+0000","last_undegraded":"2024-03-04T06:15:05.903934+0000","last_fullsized":"2024-03-04T06:15:05.903934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":101554626,"num_objects":633,"num_object_clones":485,"num_object_copies":1899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":111,"num_read":3306,"num_read_kb":2520273,"num_write":2922,"num_write_kb":797680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6239,"ondisk_log_size":6239,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706104,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":141764,"kb_used_data":96892,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94230076,"statfs":{"total":96636764160,"available":96491597824,"internally_reserved":0,"allocated":99217408,"data_stored":102379321,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706104,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114844,"kb_used_data":75540,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94256996,"statfs":{"total":96636764160,"available":96519163904,"internally_reserved":0,"allocated":77352960,"data_stored":79009237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706103,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":117320,"kb_used_data":77276,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94254520,"statfs":{"total":96636764160,"available":96516628480,"internally_reserved":0,"allocated":79130624,"data_stored":84528394,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706103,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114908,"kb_used_data":74628,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94256932,"statfs":{"total":96636764160,"available":96519098368,"internally_reserved":0,"allocated":76419072,"data_stored":79919002,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:15:09.790 INFO:tasks.rados.rados.0.smithi064.stdout:1826: finishing write tid 3 to smithi06431743-35 2024-03-04T06:15:09.790 INFO:tasks.rados.rados.0.smithi064.stdout:1826: finishing write tid 4 to smithi06431743-35 2024-03-04T06:15:09.790 INFO:tasks.rados.rados.0.smithi064.stdout:1826: oid 35 updating version 0 to 511 2024-03-04T06:15:09.790 INFO:tasks.rados.rados.0.smithi064.stdout:1826: oid 35 version 511 is already newer than 509 2024-03-04T06:15:09.790 INFO:tasks.rados.rados.0.smithi064.stdout:1826: oid 35 version 511 is already newer than 510 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 511 (ObjNum 878 snap 211 seq_num 878) dirty exists 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:1826: left oid 35 (ObjNum 878 snap 211 seq_num 878) 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:1827: finishing write tid 2 to smithi06431743-23 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:1824: done (4 left) 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:1825: done (3 left) 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:1826: done (2 left) 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:1828: done (1 left) 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:1829: snap_create 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:1827: finishing write tid 3 to smithi06431743-23 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:1827: finishing write tid 4 to smithi06431743-23 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:1827: oid 23 updating version 0 to 426 2024-03-04T06:15:09.791 INFO:tasks.rados.rados.0.smithi064.stdout:1827: oid 23 updating version 426 to 427 2024-03-04T06:15:09.792 INFO:tasks.rados.rados.0.smithi064.stdout:1827: oid 23 version 427 is already newer than 426 2024-03-04T06:15:09.792 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 427 (ObjNum 879 snap 211 seq_num 879) dirty exists 2024-03-04T06:15:09.792 INFO:tasks.rados.rados.0.smithi064.stdout:1827: left oid 23 (ObjNum 879 snap 211 seq_num 879) 2024-03-04T06:15:09.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:09.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:09.908+0000 7fa2a6b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:09.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:09.912+0000 7fa2a6b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:10.369 INFO:tasks.rados.rados.0.smithi064.stdout:1827: done (1 left) 2024-03-04T06:15:10.369 INFO:tasks.rados.rados.0.smithi064.stdout:1829: done (0 left) 2024-03-04T06:15:10.369 INFO:tasks.rados.rados.0.smithi064.stdout:1830: read oid 33 snap -1 2024-03-04T06:15:10.369 INFO:tasks.rados.rados.0.smithi064.stdout:1830: expect (ObjNum 845 snap 205 seq_num 845) 2024-03-04T06:15:10.369 INFO:tasks.rados.rados.0.smithi064.stdout:1831: read oid 13 snap -1 2024-03-04T06:15:10.369 INFO:tasks.rados.rados.0.smithi064.stdout:1831: expect (ObjNum 871 snap 209 seq_num 871) 2024-03-04T06:15:10.369 INFO:tasks.rados.rados.0.smithi064.stdout:1832: delete oid 10 current snap is 212 2024-03-04T06:15:10.403 INFO:tasks.rados.rados.0.smithi064.stdout:1832: done (2 left) 2024-03-04T06:15:10.403 INFO:tasks.rados.rados.0.smithi064.stdout:1833: write oid 1 current snap is 212 2024-03-04T06:15:10.403 INFO:tasks.rados.rados.0.smithi064.stdout:1833: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2024-03-04T06:15:10.411 INFO:tasks.rados.rados.0.smithi064.stdout:1833: writing smithi06431743-1 from 423774 to 1090340 tid 1 2024-03-04T06:15:10.418 INFO:tasks.rados.rados.0.smithi064.stdout:1833: writing smithi06431743-1 from 1545824 to 2058792 tid 2 2024-03-04T06:15:10.422 INFO:tasks.rados.rados.0.smithi064.stdout:1833: writing smithi06431743-1 from 2704609 to 2823774 tid 3 2024-03-04T06:15:10.423 INFO:tasks.rados.rados.0.smithi064.stdout:1834: snap_create 2024-03-04T06:15:10.461 INFO:tasks.rados.rados.0.smithi064.stdout:1830: expect (ObjNum 845 snap 205 seq_num 845) 2024-03-04T06:15:10.547 INFO:tasks.rados.rados.0.smithi064.stdout:1831: expect (ObjNum 871 snap 209 seq_num 871) 2024-03-04T06:15:10.673 INFO:tasks.rados.rados.0.smithi064.stdout:1833: finishing write tid 1 to smithi06431743-1 2024-03-04T06:15:10.673 INFO:tasks.rados.rados.0.smithi064.stdout:1833: finishing write tid 2 to smithi06431743-1 2024-03-04T06:15:10.673 INFO:tasks.rados.rados.0.smithi064.stdout:1833: finishing write tid 3 to smithi06431743-1 2024-03-04T06:15:10.673 INFO:tasks.rados.rados.0.smithi064.stdout:1833: finishing write tid 4 to smithi06431743-1 2024-03-04T06:15:10.673 INFO:tasks.rados.rados.0.smithi064.stdout:1833: finishing write tid 5 to smithi06431743-1 2024-03-04T06:15:10.673 INFO:tasks.rados.rados.0.smithi064.stdout:1833: oid 1 updating version 0 to 742 2024-03-04T06:15:10.673 INFO:tasks.rados.rados.0.smithi064.stdout:1833: oid 1 version 742 is already newer than 740 2024-03-04T06:15:10.674 INFO:tasks.rados.rados.0.smithi064.stdout:1833: oid 1 version 742 is already newer than 739 2024-03-04T06:15:10.674 INFO:tasks.rados.rados.0.smithi064.stdout:1833: oid 1 version 742 is already newer than 742 2024-03-04T06:15:10.674 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 742 (ObjNum 882 snap 212 seq_num 882) dirty exists 2024-03-04T06:15:10.674 INFO:tasks.rados.rados.0.smithi064.stdout:1833: left oid 1 (ObjNum 882 snap 212 seq_num 882) 2024-03-04T06:15:11.375 INFO:tasks.rados.rados.0.smithi064.stdout:1830: done (3 left) 2024-03-04T06:15:11.375 INFO:tasks.rados.rados.0.smithi064.stdout:1831: done (2 left) 2024-03-04T06:15:11.375 INFO:tasks.rados.rados.0.smithi064.stdout:1833: done (1 left) 2024-03-04T06:15:11.376 INFO:tasks.rados.rados.0.smithi064.stdout:1834: done (0 left) 2024-03-04T06:15:11.376 INFO:tasks.rados.rados.0.smithi064.stdout:1835: write (excl) oid 26 current snap is 213 2024-03-04T06:15:11.376 INFO:tasks.rados.rados.0.smithi064.stdout:1835: seq_num 883 ranges {440581=741815,1604306=656626,2732032=108549} 2024-03-04T06:15:11.381 INFO:tasks.rados.rados.0.smithi064.stdout:1835: writing smithi06431743-26 from 440581 to 1182396 tid 1 2024-03-04T06:15:11.387 INFO:tasks.rados.rados.0.smithi064.stdout:1835: writing smithi06431743-26 from 1604306 to 2260932 tid 2 2024-03-04T06:15:11.390 INFO:tasks.rados.rados.0.smithi064.stdout:1835: writing smithi06431743-26 from 2732032 to 2840581 tid 3 2024-03-04T06:15:11.390 INFO:tasks.rados.rados.0.smithi064.stdout:1836: snap_remove snap 211 2024-03-04T06:15:11.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:12.101 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:12.099+0000 7f8aa2d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:12.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:12.103+0000 7f8aa2d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:12.379 INFO:tasks.rados.rados.0.smithi064.stdout:1836: done (1 left) 2024-03-04T06:15:12.379 INFO:tasks.rados.rados.0.smithi064.stdout:1837: snap_remove snap 212 2024-03-04T06:15:13.245 INFO:tasks.rados.rados.0.smithi064.stdout:1837: done (1 left) 2024-03-04T06:15:13.245 INFO:tasks.rados.rados.0.smithi064.stdout:1838: snap_create 2024-03-04T06:15:13.245 INFO:tasks.rados.rados.0.smithi064.stdout:1835: finishing write tid 1 to smithi06431743-26 2024-03-04T06:15:13.245 INFO:tasks.rados.rados.0.smithi064.stdout:1835: finishing write tid 2 to smithi06431743-26 2024-03-04T06:15:13.245 INFO:tasks.rados.rados.0.smithi064.stdout:1835: finishing write tid 3 to smithi06431743-26 2024-03-04T06:15:13.245 INFO:tasks.rados.rados.0.smithi064.stdout:1835: finishing write tid 4 to smithi06431743-26 2024-03-04T06:15:13.245 INFO:tasks.rados.rados.0.smithi064.stdout:1835: finishing write tid 5 to smithi06431743-26 2024-03-04T06:15:13.245 INFO:tasks.rados.rados.0.smithi064.stdout:1835: oid 26 updating version 0 to 746 2024-03-04T06:15:13.246 INFO:tasks.rados.rados.0.smithi064.stdout:1835: oid 26 version 746 is already newer than 746 2024-03-04T06:15:13.246 INFO:tasks.rados.rados.0.smithi064.stdout:1835: oid 26 updating version 746 to 747 2024-03-04T06:15:13.246 INFO:tasks.rados.rados.0.smithi064.stdout:1835: oid 26 version 747 is already newer than 744 2024-03-04T06:15:13.246 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 747 (ObjNum 883 snap 213 seq_num 883) dirty exists 2024-03-04T06:15:13.246 INFO:tasks.rados.rados.0.smithi064.stdout:1835: left oid 26 (ObjNum 883 snap 213 seq_num 883) 2024-03-04T06:15:14.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:14.245 INFO:tasks.rados.rados.0.smithi064.stdout:1835: done (1 left) 2024-03-04T06:15:14.245 INFO:tasks.rados.rados.0.smithi064.stdout:1838: done (0 left) 2024-03-04T06:15:14.245 INFO:tasks.rados.rados.0.smithi064.stdout:1839: write oid 8 current snap is 214 2024-03-04T06:15:14.245 INFO:tasks.rados.rados.0.smithi064.stdout:1839: seq_num 884 ranges {457388=417064,1546435=516637,2643102=214286} 2024-03-04T06:15:14.249 INFO:tasks.rados.rados.0.smithi064.stdout:1839: writing smithi06431743-8 from 457388 to 874452 tid 1 2024-03-04T06:15:14.256 INFO:tasks.rados.rados.0.smithi064.stdout:1839: writing smithi06431743-8 from 1546435 to 2063072 tid 2 2024-03-04T06:15:14.261 INFO:tasks.rados.rados.0.smithi064.stdout:1839: writing smithi06431743-8 from 2643102 to 2857388 tid 3 2024-03-04T06:15:14.261 INFO:tasks.rados.rados.0.smithi064.stdout:1840: snap_create 2024-03-04T06:15:14.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:14.292+0000 7f732375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:14.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:14.296+0000 7f732375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:14.298 INFO:tasks.rados.rados.0.smithi064.stdout:1839: finishing write tid 1 to smithi06431743-8 2024-03-04T06:15:14.306 INFO:tasks.rados.rados.0.smithi064.stdout:1839: finishing write tid 2 to smithi06431743-8 2024-03-04T06:15:14.323 INFO:tasks.rados.rados.0.smithi064.stdout:1839: finishing write tid 3 to smithi06431743-8 2024-03-04T06:15:14.341 INFO:tasks.rados.rados.0.smithi064.stdout:1839: finishing write tid 4 to smithi06431743-8 2024-03-04T06:15:14.341 INFO:tasks.rados.rados.0.smithi064.stdout:1839: finishing write tid 5 to smithi06431743-8 2024-03-04T06:15:14.341 INFO:tasks.rados.rados.0.smithi064.stdout:1839: oid 8 updating version 0 to 345 2024-03-04T06:15:14.341 INFO:tasks.rados.rados.0.smithi064.stdout:1839: oid 8 version 345 is already newer than 344 2024-03-04T06:15:14.341 INFO:tasks.rados.rados.0.smithi064.stdout:1839: oid 8 updating version 345 to 347 2024-03-04T06:15:14.341 INFO:tasks.rados.rados.0.smithi064.stdout:1839: oid 8 version 347 is already newer than 346 2024-03-04T06:15:14.341 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 347 (ObjNum 884 snap 214 seq_num 884) dirty exists 2024-03-04T06:15:14.341 INFO:tasks.rados.rados.0.smithi064.stdout:1839: left oid 8 (ObjNum 884 snap 214 seq_num 884) 2024-03-04T06:15:14.694 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:15:14.695 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:15:14.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:15:14.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:14.801+0000 7fbb02f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:14.808 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:14.807+0000 7fbb02f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:15:14.997 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:15:15.008 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":660,"stamp":"2024-03-04T06:15:14.800306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":96984948,"num_objects":641,"num_object_clones":490,"num_object_copies":1923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":114,"num_read":3425,"num_read_kb":2535390,"num_write":3075,"num_write_kb":803269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6396,"ondisk_log_size":6396,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":356180,"kb_used_data":193396,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":377131180,"statfs":{"total":386547056640,"available":386182328320,"internally_reserved":0,"allocated":198037504,"data_stored":198137881,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1781707,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":15,"num_read_kb":15026,"num_write":15,"num_write_kb":3695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000938"},"pg_stats":[{"pgid":"3.f","version":"435'385","reported_seq":1549,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.394924+0000","last_change":"2024-03-04T06:15:08.357904+0000","last_active":"2024-03-04T06:15:08.394924+0000","last_peered":"2024-03-04T06:15:08.394924+0000","last_clean":"2024-03-04T06:15:08.394924+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:15:08.394924+0000","last_undegraded":"2024-03-04T06:15:08.394924+0000","last_fullsized":"2024-03-04T06:15:08.394924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462869,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":200,"num_read_kb":151920,"num_write":187,"num_write_kb":51268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"437'292","reported_seq":1483,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.341358+0000","last_change":"2024-03-04T06:15:08.380269+0000","last_active":"2024-03-04T06:15:08.341358+0000","last_peered":"2024-03-04T06:15:08.341358+0000","last_clean":"2024-03-04T06:15:08.341358+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:15:08.341358+0000","last_undegraded":"2024-03-04T06:15:08.341358+0000","last_fullsized":"2024-03-04T06:15:08.341358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2386792,"num_objects":33,"num_object_clones":23,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":116,"num_read_kb":74956,"num_write":135,"num_write_kb":35524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"435'396","reported_seq":1581,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.409789+0000","last_change":"2024-03-04T06:15:09.388678+0000","last_active":"2024-03-04T06:15:10.409789+0000","last_peered":"2024-03-04T06:15:10.409789+0000","last_clean":"2024-03-04T06:15:10.409789+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:15:10.409789+0000","last_undegraded":"2024-03-04T06:15:10.409789+0000","last_fullsized":"2024-03-04T06:15:10.409789+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":396,"log_dups_size":0,"ondisk_log_size":396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5781905,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":259,"num_read_kb":199658,"num_write":201,"num_write_kb":55505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"431'314","reported_seq":1519,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.393556+0000","last_change":"2024-03-04T06:15:08.348586+0000","last_active":"2024-03-04T06:15:08.393556+0000","last_peered":"2024-03-04T06:15:08.393556+0000","last_clean":"2024-03-04T06:15:08.393556+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:15:08.393556+0000","last_undegraded":"2024-03-04T06:15:08.393556+0000","last_fullsized":"2024-03-04T06:15:08.393556+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395410,"num_objects":33,"num_object_clones":24,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":173,"num_read_kb":135944,"num_write":163,"num_write_kb":45378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"439'163","reported_seq":1367,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.439592+0000","last_change":"2024-03-04T06:15:09.443630+0000","last_active":"2024-03-04T06:15:10.439592+0000","last_peered":"2024-03-04T06:15:10.439592+0000","last_clean":"2024-03-04T06:15:10.439592+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:15:10.439592+0000","last_undegraded":"2024-03-04T06:15:10.439592+0000","last_fullsized":"2024-03-04T06:15:10.439592+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":45,"num_read_kb":17049,"num_write":70,"num_write_kb":18437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"435'424","reported_seq":1569,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.397253+0000","last_change":"2024-03-04T06:15:08.357525+0000","last_active":"2024-03-04T06:15:08.397253+0000","last_peered":"2024-03-04T06:15:08.397253+0000","last_clean":"2024-03-04T06:15:08.397253+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:15:08.397253+0000","last_undegraded":"2024-03-04T06:15:08.397253+0000","last_fullsized":"2024-03-04T06:15:08.397253+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588707,"num_objects":41,"num_object_clones":33,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":199,"num_read_kb":153972,"num_write":211,"num_write_kb":62125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1294,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.393607+0000","last_change":"2024-03-04T06:15:08.358190+0000","last_active":"2024-03-04T06:15:08.393607+0000","last_peered":"2024-03-04T06:15:08.393607+0000","last_clean":"2024-03-04T06:15:08.393607+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:15:08.393607+0000","last_undegraded":"2024-03-04T06:15:08.393607+0000","last_fullsized":"2024-03-04T06:15:08.393607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"429'231","reported_seq":1453,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.404456+0000","last_change":"2024-03-04T06:15:09.388175+0000","last_active":"2024-03-04T06:15:10.404456+0000","last_peered":"2024-03-04T06:15:10.404456+0000","last_clean":"2024-03-04T06:15:10.404456+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:15:10.404456+0000","last_undegraded":"2024-03-04T06:15:10.404456+0000","last_fullsized":"2024-03-04T06:15:10.404456+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151692,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":151,"num_read_kb":140843,"num_write":108,"num_write_kb":28867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1219,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.429838+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:15:10.429838+0000","last_peered":"2024-03-04T06:15:10.429838+0000","last_clean":"2024-03-04T06:15:10.429838+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:15:10.429838+0000","last_undegraded":"2024-03-04T06:15:10.429838+0000","last_fullsized":"2024-03-04T06:15:10.429838+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1086,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.423915+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:15:10.423915+0000","last_peered":"2024-03-04T06:15:10.423915+0000","last_clean":"2024-03-04T06:15:10.423915+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:15:10.423915+0000","last_undegraded":"2024-03-04T06:15:10.423915+0000","last_fullsized":"2024-03-04T06:15:10.423915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"439'429","reported_seq":1542,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.424063+0000","last_change":"2024-03-04T06:15:09.395950+0000","last_active":"2024-03-04T06:15:10.424063+0000","last_peered":"2024-03-04T06:15:10.424063+0000","last_clean":"2024-03-04T06:15:10.424063+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:15:10.424063+0000","last_undegraded":"2024-03-04T06:15:10.424063+0000","last_fullsized":"2024-03-04T06:15:10.424063+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2773353,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":190,"num_read_kb":134390,"num_write":194,"num_write_kb":51966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1094,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.399590+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:15:08.399590+0000","last_peered":"2024-03-04T06:15:08.399590+0000","last_clean":"2024-03-04T06:15:08.399590+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:15:08.399590+0000","last_undegraded":"2024-03-04T06:15:08.399590+0000","last_fullsized":"2024-03-04T06:15:08.399590+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"435'319","reported_seq":1461,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.430642+0000","last_change":"2024-03-04T06:15:09.402613+0000","last_active":"2024-03-04T06:15:10.430642+0000","last_peered":"2024-03-04T06:15:10.430642+0000","last_clean":"2024-03-04T06:15:10.430642+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:15:10.430642+0000","last_undegraded":"2024-03-04T06:15:10.430642+0000","last_fullsized":"2024-03-04T06:15:10.430642+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605283,"num_objects":29,"num_object_clones":24,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":148,"num_read_kb":118181,"num_write":121,"num_write_kb":34336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1086,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.403253+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:15:10.403253+0000","last_peered":"2024-03-04T06:15:10.403253+0000","last_clean":"2024-03-04T06:15:10.403253+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:15:10.403253+0000","last_undegraded":"2024-03-04T06:15:10.403253+0000","last_fullsized":"2024-03-04T06:15:10.403253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"440'742","reported_seq":1823,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.494082+0000","last_change":"2024-03-04T06:15:09.405062+0000","last_active":"2024-03-04T06:15:10.494082+0000","last_peered":"2024-03-04T06:15:10.494082+0000","last_clean":"2024-03-04T06:15:10.494082+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:15:10.494082+0000","last_undegraded":"2024-03-04T06:15:10.494082+0000","last_fullsized":"2024-03-04T06:15:10.494082+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14572593,"num_objects":77,"num_object_clones":58,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":457,"num_read_kb":363652,"num_write":366,"num_write_kb":98234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1094,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.395486+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:15:08.395486+0000","last_peered":"2024-03-04T06:15:08.395486+0000","last_clean":"2024-03-04T06:15:08.395486+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:15:08.395486+0000","last_undegraded":"2024-03-04T06:15:08.395486+0000","last_fullsized":"2024-03-04T06:15:08.395486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"437'770","reported_seq":1746,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.344724+0000","last_change":"2024-03-04T06:15:08.404919+0000","last_active":"2024-03-04T06:15:08.344724+0000","last_peered":"2024-03-04T06:15:08.344724+0000","last_clean":"2024-03-04T06:15:08.344724+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:15:08.344724+0000","last_undegraded":"2024-03-04T06:15:08.344724+0000","last_fullsized":"2024-03-04T06:15:08.344724+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":770,"log_dups_size":0,"ondisk_log_size":770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11496747,"num_objects":71,"num_object_clones":54,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":310,"num_read_kb":220603,"num_write":336,"num_write_kb":87189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"435'355","reported_seq":1560,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.425572+0000","last_change":"2024-03-04T06:15:09.409883+0000","last_active":"2024-03-04T06:15:10.425572+0000","last_peered":"2024-03-04T06:15:10.425572+0000","last_clean":"2024-03-04T06:15:10.425572+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:15:10.425572+0000","last_undegraded":"2024-03-04T06:15:10.425572+0000","last_fullsized":"2024-03-04T06:15:10.425572+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4185339,"num_objects":37,"num_object_clones":27,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":201227,"num_write":174,"num_write_kb":47566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1113,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.359059+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:15:08.359059+0000","last_peered":"2024-03-04T06:15:08.359059+0000","last_clean":"2024-03-04T06:15:08.359059+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:15:08.359059+0000","last_undegraded":"2024-03-04T06:15:08.359059+0000","last_fullsized":"2024-03-04T06:15:08.359059+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"435'343","reported_seq":1541,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.357640+0000","last_change":"2024-03-04T06:15:08.363540+0000","last_active":"2024-03-04T06:15:08.357640+0000","last_peered":"2024-03-04T06:15:08.357640+0000","last_clean":"2024-03-04T06:15:08.357640+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:15:08.357640+0000","last_undegraded":"2024-03-04T06:15:08.357640+0000","last_fullsized":"2024-03-04T06:15:08.357640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6370183,"num_objects":39,"num_object_clones":27,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":192,"num_read_kb":130951,"num_write":161,"num_write_kb":43365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1086,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.442451+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:15:10.442451+0000","last_peered":"2024-03-04T06:15:10.442451+0000","last_clean":"2024-03-04T06:15:10.442451+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:15:10.442451+0000","last_undegraded":"2024-03-04T06:15:10.442451+0000","last_fullsized":"2024-03-04T06:15:10.442451+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"439'511","reported_seq":1628,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.434729+0000","last_change":"2024-03-04T06:15:09.402416+0000","last_active":"2024-03-04T06:15:10.434729+0000","last_peered":"2024-03-04T06:15:10.434729+0000","last_clean":"2024-03-04T06:15:10.434729+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:15:10.434729+0000","last_undegraded":"2024-03-04T06:15:10.434729+0000","last_fullsized":"2024-03-04T06:15:10.434729+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12605976,"num_objects":51,"num_object_clones":39,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":215513,"num_write":242,"num_write_kb":67406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1113,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.340462+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:15:08.340462+0000","last_peered":"2024-03-04T06:15:08.340462+0000","last_clean":"2024-03-04T06:15:08.340462+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:15:08.340462+0000","last_undegraded":"2024-03-04T06:15:08.340462+0000","last_fullsized":"2024-03-04T06:15:08.340462+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"435'625","reported_seq":1708,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-03-04T06:15:08.336894+0000","last_change":"2024-03-04T06:15:08.350978+0000","last_active":"2024-03-04T06:15:08.336894+0000","last_peered":"2024-03-04T06:15:08.336894+0000","last_clean":"2024-03-04T06:15:08.336894+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:15:08.336894+0000","last_undegraded":"2024-03-04T06:15:08.336894+0000","last_fullsized":"2024-03-04T06:15:08.336894+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10017731,"num_objects":55,"num_object_clones":48,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":343,"num_read_kb":276440,"num_write":269,"num_write_kb":74209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1086,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-04T06:15:10.423409+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:15:10.423409+0000","last_peered":"2024-03-04T06:15:10.423409+0000","last_clean":"2024-03-04T06:15:10.423409+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:15:10.423409+0000","last_undegraded":"2024-03-04T06:15:10.423409+0000","last_fullsized":"2024-03-04T06:15:10.423409+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":96394580,"num_objects":639,"num_object_clones":490,"num_object_copies":1917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":114,"num_read":3322,"num_read_kb":2535299,"num_write":2938,"num_write_kb":801375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6299,"ondisk_log_size":6299,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":106764,"kb_used_data":61892,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94265076,"statfs":{"total":96636764160,"available":96527437824,"internally_reserved":0,"allocated":63377408,"data_stored":63797630,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706109,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89760,"kb_used_data":50456,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94282080,"statfs":{"total":96636764160,"available":96544849920,"internally_reserved":0,"allocated":51666944,"data_stored":51688159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706108,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":80036,"kb_used_data":40732,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94291804,"statfs":{"total":96636764160,"available":96554807296,"internally_reserved":0,"allocated":41709568,"data_stored":41039742,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706108,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":79620,"kb_used_data":40316,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94292220,"statfs":{"total":96636764160,"available":96555233280,"internally_reserved":0,"allocated":41283584,"data_stored":41612350,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:15:15.247 INFO:tasks.rados.rados.0.smithi064.stdout:1839: done (1 left) 2024-03-04T06:15:15.247 INFO:tasks.rados.rados.0.smithi064.stdout:1840: done (0 left) 2024-03-04T06:15:15.247 INFO:tasks.rados.rados.0.smithi064.stdout:1841: delete oid 8 current snap is 215 2024-03-04T06:15:15.298 INFO:tasks.rados.rados.0.smithi064.stdout:1841: done (0 left) 2024-03-04T06:15:15.298 INFO:tasks.rados.rados.0.smithi064.stdout:1842: read oid 34 snap -1 2024-03-04T06:15:15.299 INFO:tasks.rados.rados.0.smithi064.stdout:1842: expect deleted 2024-03-04T06:15:15.299 INFO:tasks.rados.rados.0.smithi064.stdout:1843: read oid 11 snap 210 2024-03-04T06:15:15.299 INFO:tasks.rados.rados.0.smithi064.stdout:1843: expect deleted 2024-03-04T06:15:15.299 INFO:tasks.rados.rados.0.smithi064.stdout:1844: snap_create 2024-03-04T06:15:16.249 INFO:tasks.rados.rados.0.smithi064.stdout:1842: done (2 left) 2024-03-04T06:15:16.249 INFO:tasks.rados.rados.0.smithi064.stdout:1843: done (1 left) 2024-03-04T06:15:16.249 INFO:tasks.rados.rados.0.smithi064.stdout:1844: done (0 left) 2024-03-04T06:15:16.250 INFO:tasks.rados.rados.0.smithi064.stdout:1845: read oid 42 snap -1 2024-03-04T06:15:16.250 INFO:tasks.rados.rados.0.smithi064.stdout:1845: expect (ObjNum 844 snap 205 seq_num 844) 2024-03-04T06:15:16.250 INFO:tasks.rados.rados.0.smithi064.stdout:1846: read oid 37 snap -1 2024-03-04T06:15:16.250 INFO:tasks.rados.rados.0.smithi064.stdout:1846: expect (ObjNum 856 snap 207 seq_num 856) 2024-03-04T06:15:16.250 INFO:tasks.rados.rados.0.smithi064.stdout:1847: delete oid 18 current snap is 216 2024-03-04T06:15:16.296 INFO:tasks.rados.rados.0.smithi064.stdout:1846: expect (ObjNum 856 snap 207 seq_num 856) 2024-03-04T06:15:16.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:16.385 INFO:tasks.rados.rados.0.smithi064.stdout:1845: expect (ObjNum 844 snap 205 seq_num 844) 2024-03-04T06:15:16.483 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:16.481+0000 7f3c9639d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:16.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:16.485+0000 7f3c9639d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:16.523 INFO:tasks.rados.rados.0.smithi064.stdout:1845: done (2 left) 2024-03-04T06:15:16.523 INFO:tasks.rados.rados.0.smithi064.stdout:1846: done (1 left) 2024-03-04T06:15:16.523 INFO:tasks.rados.rados.0.smithi064.stdout:1847: done (0 left) 2024-03-04T06:15:16.523 INFO:tasks.rados.rados.0.smithi064.stdout:1848: read oid 16 snap -1 2024-03-04T06:15:16.523 INFO:tasks.rados.rados.0.smithi064.stdout:1848: expect (ObjNum 709 snap 177 seq_num 709) 2024-03-04T06:15:16.523 INFO:tasks.rados.rados.0.smithi064.stdout:1849: write (excl) oid 46 current snap is 216 2024-03-04T06:15:16.523 INFO:tasks.rados.rados.0.smithi064.stdout:1849: seq_num 887 ranges {507809=642811,1721881=663964} 2024-03-04T06:15:16.528 INFO:tasks.rados.rados.0.smithi064.stdout:1849: writing smithi06431743-46 from 507809 to 1150620 tid 1 2024-03-04T06:15:16.535 INFO:tasks.rados.rados.0.smithi064.stdout:1849: writing smithi06431743-46 from 1721881 to 2385845 tid 2 2024-03-04T06:15:16.536 INFO:tasks.rados.rados.0.smithi064.stdout:1850: write oid 44 current snap is 216 2024-03-04T06:15:16.536 INFO:tasks.rados.rados.0.smithi064.stdout:1850: seq_num 888 ranges {524616=718060,1664010=523975,2752794=171822} 2024-03-04T06:15:16.542 INFO:tasks.rados.rados.0.smithi064.stdout:1850: writing smithi06431743-44 from 524616 to 1242676 tid 1 2024-03-04T06:15:16.551 INFO:tasks.rados.rados.0.smithi064.stdout:1850: writing smithi06431743-44 from 1664010 to 2187985 tid 2 2024-03-04T06:15:16.556 INFO:tasks.rados.rados.0.smithi064.stdout:1850: writing smithi06431743-44 from 2752794 to 2924616 tid 3 2024-03-04T06:15:16.556 INFO:tasks.rados.rados.0.smithi064.stdout:1851: write (excl) oid 48 current snap is 216 2024-03-04T06:15:16.556 INFO:tasks.rados.rados.0.smithi064.stdout:1851: seq_num 889 ranges {541423=793309,2122492=667633} 2024-03-04T06:15:16.564 INFO:tasks.rados.rados.0.smithi064.stdout:1851: writing smithi06431743-48 from 541423 to 1334732 tid 1 2024-03-04T06:15:16.572 INFO:tasks.rados.rados.0.smithi064.stdout:1851: writing smithi06431743-48 from 2122492 to 2790125 tid 2 2024-03-04T06:15:16.572 INFO:tasks.rados.rados.0.smithi064.stdout:1852: read oid 6 snap -1 2024-03-04T06:15:16.572 INFO:tasks.rados.rados.0.smithi064.stdout:1852: expect (ObjNum 834 snap 204 seq_num 834) 2024-03-04T06:15:16.573 INFO:tasks.rados.rados.0.smithi064.stdout:1853: snap_remove snap 210 2024-03-04T06:15:17.254 INFO:tasks.rados.rados.0.smithi064.stdout:1853: done (5 left) 2024-03-04T06:15:17.255 INFO:tasks.rados.rados.0.smithi064.stdout:1854: delete oid 2 current snap is 216 2024-03-04T06:15:17.255 INFO:tasks.rados.rados.0.smithi064.stdout:1850: finishing write tid 1 to smithi06431743-44 2024-03-04T06:15:17.255 INFO:tasks.rados.rados.0.smithi064.stdout:1848: expect (ObjNum 709 snap 177 seq_num 709) 2024-03-04T06:15:17.417 INFO:tasks.rados.rados.0.smithi064.stdout:1849: finishing write tid 1 to smithi06431743-46 2024-03-04T06:15:17.417 INFO:tasks.rados.rados.0.smithi064.stdout:1851: finishing write tid 1 to smithi06431743-48 2024-03-04T06:15:17.417 INFO:tasks.rados.rados.0.smithi064.stdout:1850: finishing write tid 2 to smithi06431743-44 2024-03-04T06:15:17.417 INFO:tasks.rados.rados.0.smithi064.stdout:1849: finishing write tid 2 to smithi06431743-46 2024-03-04T06:15:17.417 INFO:tasks.rados.rados.0.smithi064.stdout:1849: finishing write tid 3 to smithi06431743-46 2024-03-04T06:15:17.418 INFO:tasks.rados.rados.0.smithi064.stdout:1849: finishing write tid 4 to smithi06431743-46 2024-03-04T06:15:17.418 INFO:tasks.rados.rados.0.smithi064.stdout:1849: oid 46 updating version 0 to 755 2024-03-04T06:15:17.418 INFO:tasks.rados.rados.0.smithi064.stdout:1849: oid 46 version 755 is already newer than 755 2024-03-04T06:15:17.418 INFO:tasks.rados.rados.0.smithi064.stdout:1849: oid 46 version 755 is already newer than 753 2024-03-04T06:15:17.418 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 755 (ObjNum 887 snap 216 seq_num 887) dirty exists 2024-03-04T06:15:17.418 INFO:tasks.rados.rados.0.smithi064.stdout:1849: left oid 46 (ObjNum 887 snap 216 seq_num 887) 2024-03-04T06:15:17.418 INFO:tasks.rados.rados.0.smithi064.stdout:1850: finishing write tid 3 to smithi06431743-44 2024-03-04T06:15:17.418 INFO:tasks.rados.rados.0.smithi064.stdout:1848: done (5 left) 2024-03-04T06:15:17.418 INFO:tasks.rados.rados.0.smithi064.stdout:1849: done (4 left) 2024-03-04T06:15:17.418 INFO:tasks.rados.rados.0.smithi064.stdout:1854: done (3 left) 2024-03-04T06:15:17.418 INFO:tasks.rados.rados.0.smithi064.stdout:1855: snap_remove snap 214 2024-03-04T06:15:18.257 INFO:tasks.rados.rados.0.smithi064.stdout:1855: done (3 left) 2024-03-04T06:15:18.258 INFO:tasks.rados.rados.0.smithi064.stdout:1856: read oid 24 snap -1 2024-03-04T06:15:18.258 INFO:tasks.rados.rados.0.smithi064.stdout:1856: expect (ObjNum 807 snap 198 seq_num 807) 2024-03-04T06:15:18.258 INFO:tasks.rados.rados.0.smithi064.stdout:1857: delete oid 9 current snap is 216 2024-03-04T06:15:18.258 INFO:tasks.rados.rados.0.smithi064.stdout:1851: finishing write tid 2 to smithi06431743-48 2024-03-04T06:15:18.258 INFO:tasks.rados.rados.0.smithi064.stdout:1850: finishing write tid 4 to smithi06431743-44 2024-03-04T06:15:18.258 INFO:tasks.rados.rados.0.smithi064.stdout:1850: finishing write tid 5 to smithi06431743-44 2024-03-04T06:15:18.258 INFO:tasks.rados.rados.0.smithi064.stdout:1850: oid 44 updating version 0 to 429 2024-03-04T06:15:18.258 INFO:tasks.rados.rados.0.smithi064.stdout:1850: oid 44 version 429 is already newer than 427 2024-03-04T06:15:18.258 INFO:tasks.rados.rados.0.smithi064.stdout:1850: oid 44 version 429 is already newer than 428 2024-03-04T06:15:18.258 INFO:tasks.rados.rados.0.smithi064.stdout:1850: oid 44 updating version 429 to 430 2024-03-04T06:15:18.258 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 430 (ObjNum 888 snap 216 seq_num 888) dirty exists 2024-03-04T06:15:18.259 INFO:tasks.rados.rados.0.smithi064.stdout:1850: left oid 44 (ObjNum 888 snap 216 seq_num 888) 2024-03-04T06:15:18.259 INFO:tasks.rados.rados.0.smithi064.stdout:1851: finishing write tid 3 to smithi06431743-48 2024-03-04T06:15:18.259 INFO:tasks.rados.rados.0.smithi064.stdout:1852: expect (ObjNum 834 snap 204 seq_num 834) 2024-03-04T06:15:18.415 INFO:tasks.rados.rados.0.smithi064.stdout:1851: finishing write tid 4 to smithi06431743-48 2024-03-04T06:15:18.415 INFO:tasks.rados.rados.0.smithi064.stdout:1851: oid 48 updating version 0 to 321 2024-03-04T06:15:18.415 INFO:tasks.rados.rados.0.smithi064.stdout:1851: oid 48 version 321 is already newer than 320 2024-03-04T06:15:18.415 INFO:tasks.rados.rados.0.smithi064.stdout:1851: oid 48 updating version 321 to 322 2024-03-04T06:15:18.415 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 322 (ObjNum 889 snap 216 seq_num 889) dirty exists 2024-03-04T06:15:18.415 INFO:tasks.rados.rados.0.smithi064.stdout:1851: left oid 48 (ObjNum 889 snap 216 seq_num 889) 2024-03-04T06:15:18.415 INFO:tasks.rados.rados.0.smithi064.stdout:1850: done (4 left) 2024-03-04T06:15:18.416 INFO:tasks.rados.rados.0.smithi064.stdout:1851: done (3 left) 2024-03-04T06:15:18.416 INFO:tasks.rados.rados.0.smithi064.stdout:1852: done (2 left) 2024-03-04T06:15:18.416 INFO:tasks.rados.rados.0.smithi064.stdout:1857: done (1 left) 2024-03-04T06:15:18.416 INFO:tasks.rados.rados.0.smithi064.stdout:1858: delete oid 39 current snap is 216 2024-03-04T06:15:18.416 INFO:tasks.rados.rados.0.smithi064.stdout:1856: expect (ObjNum 807 snap 198 seq_num 807) 2024-03-04T06:15:18.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:18.605 INFO:tasks.rados.rados.0.smithi064.stdout:1856: done (1 left) 2024-03-04T06:15:18.618 INFO:tasks.rados.rados.0.smithi064.stdout:1858: done (0 left) 2024-03-04T06:15:18.618 INFO:tasks.rados.rados.0.smithi064.stdout:1859: read oid 48 snap -1 2024-03-04T06:15:18.619 INFO:tasks.rados.rados.0.smithi064.stdout:1859: expect (ObjNum 889 snap 216 seq_num 889) 2024-03-04T06:15:18.619 INFO:tasks.rados.rados.0.smithi064.stdout:1860: snap_create 2024-03-04T06:15:18.619 INFO:tasks.rados.rados.0.smithi064.stdout:1859: expect (ObjNum 889 snap 216 seq_num 889) 2024-03-04T06:15:18.676 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:18.674+0000 7fc3de51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:18.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:18.679+0000 7fc3de51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:19.260 INFO:tasks.rados.rados.0.smithi064.stdout:1859: done (1 left) 2024-03-04T06:15:19.260 INFO:tasks.rados.rados.0.smithi064.stdout:1860: done (0 left) 2024-03-04T06:15:19.260 INFO:tasks.rados.rados.0.smithi064.stdout:1861: write (excl) oid 41 current snap is 217 2024-03-04T06:15:19.260 INFO:tasks.rados.rados.0.smithi064.stdout:1861: seq_num 893 ranges {608651=410658,1556420=674971,2773556=235095} 2024-03-04T06:15:19.265 INFO:tasks.rados.rados.0.smithi064.stdout:1861: writing smithi06431743-41 from 608651 to 1019309 tid 1 2024-03-04T06:15:19.272 INFO:tasks.rados.rados.0.smithi064.stdout:1861: writing smithi06431743-41 from 1556420 to 2231391 tid 2 2024-03-04T06:15:19.278 INFO:tasks.rados.rados.0.smithi064.stdout:1861: writing smithi06431743-41 from 2773556 to 3008651 tid 3 2024-03-04T06:15:19.278 INFO:tasks.rados.rados.0.smithi064.stdout:1862: read oid 39 snap -1 2024-03-04T06:15:19.278 INFO:tasks.rados.rados.0.smithi064.stdout:1862: expect deleted 2024-03-04T06:15:19.278 INFO:tasks.rados.rados.0.smithi064.stdout:1863: read oid 34 snap -1 2024-03-04T06:15:19.279 INFO:tasks.rados.rados.0.smithi064.stdout:1863: expect deleted 2024-03-04T06:15:19.279 INFO:tasks.rados.rados.0.smithi064.stdout:1864: snap_remove snap 215 2024-03-04T06:15:20.010 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:15:20.010 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:15:20.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:15:20.132 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:20.128+0000 7f8911a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:20.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:20.135+0000 7f8911a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:20.262 INFO:tasks.rados.rados.0.smithi064.stdout:1864: done (3 left) 2024-03-04T06:15:20.262 INFO:tasks.rados.rados.0.smithi064.stdout:1865: snap_create 2024-03-04T06:15:20.263 INFO:tasks.rados.rados.0.smithi064.stdout:1861: finishing write tid 1 to smithi06431743-41 2024-03-04T06:15:20.263 INFO:tasks.rados.rados.0.smithi064.stdout:1861: finishing write tid 2 to smithi06431743-41 2024-03-04T06:15:20.263 INFO:tasks.rados.rados.0.smithi064.stdout:1861: finishing write tid 3 to smithi06431743-41 2024-03-04T06:15:20.263 INFO:tasks.rados.rados.0.smithi064.stdout:1861: finishing write tid 4 to smithi06431743-41 2024-03-04T06:15:20.263 INFO:tasks.rados.rados.0.smithi064.stdout:1861: finishing write tid 5 to smithi06431743-41 2024-03-04T06:15:20.263 INFO:tasks.rados.rados.0.smithi064.stdout:1861: oid 41 updating version 0 to 434 2024-03-04T06:15:20.263 INFO:tasks.rados.rados.0.smithi064.stdout:1861: oid 41 updating version 434 to 436 2024-03-04T06:15:20.263 INFO:tasks.rados.rados.0.smithi064.stdout:1861: oid 41 updating version 436 to 437 2024-03-04T06:15:20.263 INFO:tasks.rados.rados.0.smithi064.stdout:1861: oid 41 version 437 is already newer than 436 2024-03-04T06:15:20.263 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 437 (ObjNum 893 snap 217 seq_num 893) dirty exists 2024-03-04T06:15:20.263 INFO:tasks.rados.rados.0.smithi064.stdout:1861: left oid 41 (ObjNum 893 snap 217 seq_num 893) 2024-03-04T06:15:20.340 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:15:20.341 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:15:20.353 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2024-03-04T06:15:20.261463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":97153018,"num_objects":644,"num_object_clones":492,"num_object_copies":1932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":115,"num_read":3441,"num_read_kb":2545690,"num_write":3086,"num_write_kb":805866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6414,"ondisk_log_size":6414,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":365480,"kb_used_data":202696,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":377121880,"statfs":{"total":386547056640,"available":386172805120,"internally_reserved":0,"allocated":207560704,"data_stored":209447396,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":168070,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014491"},"pg_stats":[{"pgid":"3.f","version":"435'385","reported_seq":1564,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-04T06:15:14.275331+0000","last_change":"2024-03-04T06:15:13.295415+0000","last_active":"2024-03-04T06:15:14.275331+0000","last_peered":"2024-03-04T06:15:14.275331+0000","last_clean":"2024-03-04T06:15:14.275331+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:15:14.275331+0000","last_undegraded":"2024-03-04T06:15:14.275331+0000","last_fullsized":"2024-03-04T06:15:14.275331+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462869,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":200,"num_read_kb":151920,"num_write":187,"num_write_kb":51268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"437'292","reported_seq":1504,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.266566+0000","last_change":"2024-03-04T06:15:13.926534+0000","last_active":"2024-03-04T06:15:15.266566+0000","last_peered":"2024-03-04T06:15:15.266566+0000","last_clean":"2024-03-04T06:15:15.266566+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:15:15.266566+0000","last_undegraded":"2024-03-04T06:15:15.266566+0000","last_fullsized":"2024-03-04T06:15:15.266566+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2386792,"num_objects":33,"num_object_clones":23,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":116,"num_read_kb":74956,"num_write":135,"num_write_kb":35524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"435'396","reported_seq":1596,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.311625+0000","last_change":"2024-03-04T06:15:13.951059+0000","last_active":"2024-03-04T06:15:15.311625+0000","last_peered":"2024-03-04T06:15:15.311625+0000","last_clean":"2024-03-04T06:15:15.311625+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:15:15.311625+0000","last_undegraded":"2024-03-04T06:15:15.311625+0000","last_fullsized":"2024-03-04T06:15:15.311625+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":396,"log_dups_size":0,"ondisk_log_size":396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5781905,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":259,"num_read_kb":199658,"num_write":201,"num_write_kb":55505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"440'318","reported_seq":1534,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-04T06:15:14.269849+0000","last_change":"2024-03-04T06:15:13.278229+0000","last_active":"2024-03-04T06:15:14.269849+0000","last_peered":"2024-03-04T06:15:14.269849+0000","last_clean":"2024-03-04T06:15:14.269849+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:15:14.269849+0000","last_undegraded":"2024-03-04T06:15:14.269849+0000","last_fullsized":"2024-03-04T06:15:14.269849+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395410,"num_objects":33,"num_object_clones":24,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":173,"num_read_kb":135944,"num_write":163,"num_write_kb":45378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"439'163","reported_seq":1382,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.276657+0000","last_change":"2024-03-04T06:15:13.935296+0000","last_active":"2024-03-04T06:15:15.276657+0000","last_peered":"2024-03-04T06:15:15.276657+0000","last_clean":"2024-03-04T06:15:15.276657+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:15:15.276657+0000","last_undegraded":"2024-03-04T06:15:15.276657+0000","last_fullsized":"2024-03-04T06:15:15.276657+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":45,"num_read_kb":17049,"num_write":70,"num_write_kb":18437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"440'425","reported_seq":1584,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-04T06:15:14.280053+0000","last_change":"2024-03-04T06:15:13.295936+0000","last_active":"2024-03-04T06:15:14.280053+0000","last_peered":"2024-03-04T06:15:14.280053+0000","last_clean":"2024-03-04T06:15:14.280053+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:15:14.280053+0000","last_undegraded":"2024-03-04T06:15:14.280053+0000","last_fullsized":"2024-03-04T06:15:14.280053+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588707,"num_objects":41,"num_object_clones":33,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":199,"num_read_kb":153972,"num_write":211,"num_write_kb":62125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1309,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-04T06:15:14.279314+0000","last_change":"2024-03-04T06:15:13.296052+0000","last_active":"2024-03-04T06:15:14.279314+0000","last_peered":"2024-03-04T06:15:14.279314+0000","last_clean":"2024-03-04T06:15:14.279314+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:15:14.279314+0000","last_undegraded":"2024-03-04T06:15:14.279314+0000","last_fullsized":"2024-03-04T06:15:14.279314+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"429'231","reported_seq":1468,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.307569+0000","last_change":"2024-03-04T06:15:13.951194+0000","last_active":"2024-03-04T06:15:15.307569+0000","last_peered":"2024-03-04T06:15:15.307569+0000","last_clean":"2024-03-04T06:15:15.307569+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:15:15.307569+0000","last_undegraded":"2024-03-04T06:15:15.307569+0000","last_fullsized":"2024-03-04T06:15:15.307569+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151692,"num_objects":24,"num_object_clones":16,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":151,"num_read_kb":140843,"num_write":108,"num_write_kb":28867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1232,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.261005+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:15:15.261005+0000","last_peered":"2024-03-04T06:15:15.261005+0000","last_clean":"2024-03-04T06:15:15.261005+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:15:15.261005+0000","last_undegraded":"2024-03-04T06:15:15.261005+0000","last_fullsized":"2024-03-04T06:15:15.261005+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1099,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.318340+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:15:15.318340+0000","last_peered":"2024-03-04T06:15:15.318340+0000","last_clean":"2024-03-04T06:15:15.318340+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:15:15.318340+0000","last_undegraded":"2024-03-04T06:15:15.318340+0000","last_fullsized":"2024-03-04T06:15:15.318340+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"439'429","reported_seq":1557,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.317548+0000","last_change":"2024-03-04T06:15:13.949215+0000","last_active":"2024-03-04T06:15:15.317548+0000","last_peered":"2024-03-04T06:15:15.317548+0000","last_clean":"2024-03-04T06:15:15.317548+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:15:15.317548+0000","last_undegraded":"2024-03-04T06:15:15.317548+0000","last_fullsized":"2024-03-04T06:15:15.317548+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2773353,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":190,"num_read_kb":134390,"num_write":194,"num_write_kb":51966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1106,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-04T06:15:14.279180+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:15:14.279180+0000","last_peered":"2024-03-04T06:15:14.279180+0000","last_clean":"2024-03-04T06:15:14.279180+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:15:14.279180+0000","last_undegraded":"2024-03-04T06:15:14.279180+0000","last_fullsized":"2024-03-04T06:15:14.279180+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"435'319","reported_seq":1476,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.264134+0000","last_change":"2024-03-04T06:15:13.913228+0000","last_active":"2024-03-04T06:15:15.264134+0000","last_peered":"2024-03-04T06:15:15.264134+0000","last_clean":"2024-03-04T06:15:15.264134+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:15:15.264134+0000","last_undegraded":"2024-03-04T06:15:15.264134+0000","last_fullsized":"2024-03-04T06:15:15.264134+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605283,"num_objects":29,"num_object_clones":24,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":148,"num_read_kb":118181,"num_write":121,"num_write_kb":34336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1099,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.310476+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:15:15.310476+0000","last_peered":"2024-03-04T06:15:15.310476+0000","last_clean":"2024-03-04T06:15:15.310476+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:15:15.310476+0000","last_undegraded":"2024-03-04T06:15:15.310476+0000","last_fullsized":"2024-03-04T06:15:15.310476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"443'751","reported_seq":1843,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.269481+0000","last_change":"2024-03-04T06:15:14.338599+0000","last_active":"2024-03-04T06:15:15.269481+0000","last_peered":"2024-03-04T06:15:15.269481+0000","last_clean":"2024-03-04T06:15:15.269481+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:15:15.269481+0000","last_undegraded":"2024-03-04T06:15:15.269481+0000","last_fullsized":"2024-03-04T06:15:15.269481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14740663,"num_objects":78,"num_object_clones":59,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":459,"num_read_kb":363653,"num_write":371,"num_write_kb":99708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1106,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-03-04T06:15:14.266804+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:15:14.266804+0000","last_peered":"2024-03-04T06:15:14.266804+0000","last_clean":"2024-03-04T06:15:14.266804+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:15:14.266804+0000","last_undegraded":"2024-03-04T06:15:14.266804+0000","last_fullsized":"2024-03-04T06:15:14.266804+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"437'770","reported_seq":1770,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.257300+0000","last_change":"2024-03-04T06:15:13.925512+0000","last_active":"2024-03-04T06:15:15.257300+0000","last_peered":"2024-03-04T06:15:15.257300+0000","last_clean":"2024-03-04T06:15:15.257300+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:15:15.257300+0000","last_undegraded":"2024-03-04T06:15:15.257300+0000","last_fullsized":"2024-03-04T06:15:15.257300+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":770,"log_dups_size":0,"ondisk_log_size":770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11496747,"num_objects":71,"num_object_clones":54,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":316,"num_read_kb":227078,"num_write":336,"num_write_kb":87189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"435'355","reported_seq":1575,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.317156+0000","last_change":"2024-03-04T06:15:13.953985+0000","last_active":"2024-03-04T06:15:15.317156+0000","last_peered":"2024-03-04T06:15:15.317156+0000","last_clean":"2024-03-04T06:15:15.317156+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:15:15.317156+0000","last_undegraded":"2024-03-04T06:15:15.317156+0000","last_fullsized":"2024-03-04T06:15:15.317156+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4185339,"num_objects":37,"num_object_clones":27,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":201227,"num_write":174,"num_write_kb":47566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1131,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.274804+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:15:15.274804+0000","last_peered":"2024-03-04T06:15:15.274804+0000","last_clean":"2024-03-04T06:15:15.274804+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:15:15.274804+0000","last_undegraded":"2024-03-04T06:15:15.274804+0000","last_fullsized":"2024-03-04T06:15:15.274804+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"444'347","reported_seq":1568,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.272802+0000","last_change":"2024-03-04T06:15:13.938521+0000","last_active":"2024-03-04T06:15:15.272802+0000","last_peered":"2024-03-04T06:15:15.272802+0000","last_clean":"2024-03-04T06:15:15.272802+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:15:15.272802+0000","last_undegraded":"2024-03-04T06:15:15.272802+0000","last_fullsized":"2024-03-04T06:15:15.272802+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6370183,"num_objects":41,"num_object_clones":28,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":194,"num_read_kb":130952,"num_write":167,"num_write_kb":44488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1099,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.268931+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:15:15.268931+0000","last_peered":"2024-03-04T06:15:15.268931+0000","last_clean":"2024-03-04T06:15:15.268931+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:15:15.268931+0000","last_undegraded":"2024-03-04T06:15:15.268931+0000","last_fullsized":"2024-03-04T06:15:15.268931+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"439'511","reported_seq":1643,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.263150+0000","last_change":"2024-03-04T06:15:13.912815+0000","last_active":"2024-03-04T06:15:15.263150+0000","last_peered":"2024-03-04T06:15:15.263150+0000","last_clean":"2024-03-04T06:15:15.263150+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:15:15.263150+0000","last_undegraded":"2024-03-04T06:15:15.263150+0000","last_fullsized":"2024-03-04T06:15:15.263150+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12605976,"num_objects":51,"num_object_clones":39,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":215513,"num_write":242,"num_write_kb":67406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1131,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.264303+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:15:15.264303+0000","last_peered":"2024-03-04T06:15:15.264303+0000","last_clean":"2024-03-04T06:15:15.264303+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:15:15.264303+0000","last_undegraded":"2024-03-04T06:15:15.264303+0000","last_fullsized":"2024-03-04T06:15:15.264303+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"435'625","reported_seq":1732,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.261980+0000","last_change":"2024-03-04T06:15:13.926300+0000","last_active":"2024-03-04T06:15:15.261980+0000","last_peered":"2024-03-04T06:15:15.261980+0000","last_clean":"2024-03-04T06:15:15.261980+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:15:15.261980+0000","last_undegraded":"2024-03-04T06:15:15.261980+0000","last_fullsized":"2024-03-04T06:15:15.261980+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10017731,"num_objects":55,"num_object_clones":48,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":349,"num_read_kb":280263,"num_write":269,"num_write_kb":74209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1099,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-03-04T06:15:15.318179+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:15:15.318179+0000","last_peered":"2024-03-04T06:15:15.318179+0000","last_clean":"2024-03-04T06:15:15.318179+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:15:15.318179+0000","last_undegraded":"2024-03-04T06:15:15.318179+0000","last_fullsized":"2024-03-04T06:15:15.318179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":96562650,"num_objects":642,"num_object_clones":492,"num_object_copies":1926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":115,"num_read":3338,"num_read_kb":2545599,"num_write":2949,"num_write_kb":803972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6317,"ondisk_log_size":6317,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706114,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":109416,"kb_used_data":64544,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94262424,"statfs":{"total":96636764160,"available":96524722176,"internally_reserved":0,"allocated":66093056,"data_stored":66990006,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706114,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91280,"kb_used_data":51976,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94280560,"statfs":{"total":96636764160,"available":96543293440,"internally_reserved":0,"allocated":53223424,"data_stored":53732548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706113,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83984,"kb_used_data":44680,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94287856,"statfs":{"total":96636764160,"available":96550764544,"internally_reserved":0,"allocated":45752320,"data_stored":45942659,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706113,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":80800,"kb_used_data":41496,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94291040,"statfs":{"total":96636764160,"available":96554024960,"internally_reserved":0,"allocated":42491904,"data_stored":42782183,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:15:20.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:20.854 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:20.852+0000 7ff1c1b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:20.857 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:20.856+0000 7ff1c1b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:21.267 INFO:tasks.rados.rados.0.smithi064.stdout:1861: done (3 left) 2024-03-04T06:15:21.267 INFO:tasks.rados.rados.0.smithi064.stdout:1862: done (2 left) 2024-03-04T06:15:21.267 INFO:tasks.rados.rados.0.smithi064.stdout:1863: done (1 left) 2024-03-04T06:15:21.267 INFO:tasks.rados.rados.0.smithi064.stdout:1865: done (0 left) 2024-03-04T06:15:21.267 INFO:tasks.rados.rados.0.smithi064.stdout:1866: write oid 33 current snap is 218 2024-03-04T06:15:21.267 INFO:tasks.rados.rados.0.smithi064.stdout:1866: seq_num 894 ranges {625458=485907,1614902=534982,2917332=108126} 2024-03-04T06:15:21.273 INFO:tasks.rados.rados.0.smithi064.stdout:1866: writing smithi06431743-33 from 625458 to 1111365 tid 1 2024-03-04T06:15:21.278 INFO:tasks.rados.rados.0.smithi064.stdout:1866: writing smithi06431743-33 from 1614902 to 2149884 tid 2 2024-03-04T06:15:21.283 INFO:tasks.rados.rados.0.smithi064.stdout:1866: writing smithi06431743-33 from 2917332 to 3025458 tid 3 2024-03-04T06:15:21.283 INFO:tasks.rados.rados.0.smithi064.stdout:1867: read oid 29 snap 217 2024-03-04T06:15:21.283 INFO:tasks.rados.rados.0.smithi064.stdout:1867: expect (ObjNum 850 snap 206 seq_num 850) 2024-03-04T06:15:21.283 INFO:tasks.rados.rados.0.smithi064.stdout:1868: snap_create 2024-03-04T06:15:21.328 INFO:tasks.rados.rados.0.smithi064.stdout:1867: expect (ObjNum 850 snap 206 seq_num 850) 2024-03-04T06:15:21.417 INFO:tasks.rados.rados.0.smithi064.stdout:1866: finishing write tid 1 to smithi06431743-33 2024-03-04T06:15:21.417 INFO:tasks.rados.rados.0.smithi064.stdout:1866: finishing write tid 2 to smithi06431743-33 2024-03-04T06:15:21.418 INFO:tasks.rados.rados.0.smithi064.stdout:1866: finishing write tid 3 to smithi06431743-33 2024-03-04T06:15:21.418 INFO:tasks.rados.rados.0.smithi064.stdout:1866: finishing write tid 4 to smithi06431743-33 2024-03-04T06:15:21.418 INFO:tasks.rados.rados.0.smithi064.stdout:1866: finishing write tid 5 to smithi06431743-33 2024-03-04T06:15:21.418 INFO:tasks.rados.rados.0.smithi064.stdout:1866: oid 33 updating version 0 to 631 2024-03-04T06:15:21.418 INFO:tasks.rados.rados.0.smithi064.stdout:1866: oid 33 updating version 631 to 633 2024-03-04T06:15:21.418 INFO:tasks.rados.rados.0.smithi064.stdout:1866: oid 33 version 633 is already newer than 632 2024-03-04T06:15:21.418 INFO:tasks.rados.rados.0.smithi064.stdout:1866: oid 33 updating version 633 to 634 2024-03-04T06:15:21.418 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 634 (ObjNum 894 snap 218 seq_num 894) dirty exists 2024-03-04T06:15:21.418 INFO:tasks.rados.rados.0.smithi064.stdout:1866: left oid 33 (ObjNum 894 snap 218 seq_num 894) 2024-03-04T06:15:22.267 INFO:tasks.rados.rados.0.smithi064.stdout:1866: done (2 left) 2024-03-04T06:15:22.268 INFO:tasks.rados.rados.0.smithi064.stdout:1867: done (1 left) 2024-03-04T06:15:22.268 INFO:tasks.rados.rados.0.smithi064.stdout:1868: done (0 left) 2024-03-04T06:15:22.268 INFO:tasks.rados.rados.0.smithi064.stdout:1869: snap_remove snap 213 2024-03-04T06:15:22.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:23.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:23.045+0000 7f672015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:23.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:23.049+0000 7f672015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:23.247 INFO:tasks.rados.rados.0.smithi064.stdout:1869: done (0 left) 2024-03-04T06:15:23.247 INFO:tasks.rados.rados.0.smithi064.stdout:1870: read oid 7 snap -1 2024-03-04T06:15:23.247 INFO:tasks.rados.rados.0.smithi064.stdout:1870: expect (ObjNum 869 snap 209 seq_num 869) 2024-03-04T06:15:23.247 INFO:tasks.rados.rados.0.smithi064.stdout:1871: write (excl) oid 13 current snap is 219 2024-03-04T06:15:23.247 INFO:tasks.rados.rados.0.smithi064.stdout:1871: seq_num 895 ranges {642265=561156,1673384=678640,2828402=213863} 2024-03-04T06:15:23.253 INFO:tasks.rados.rados.0.smithi064.stdout:1871: writing smithi06431743-13 from 642265 to 1203421 tid 1 2024-03-04T06:15:23.261 INFO:tasks.rados.rados.0.smithi064.stdout:1871: writing smithi06431743-13 from 1673384 to 2352024 tid 2 2024-03-04T06:15:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:1871: writing smithi06431743-13 from 2828402 to 3042265 tid 3 2024-03-04T06:15:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:1872: write (excl) oid 25 current snap is 219 2024-03-04T06:15:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:1872: seq_num 896 ranges {659072=636405,1731866=422298,2855825=203247} 2024-03-04T06:15:23.275 INFO:tasks.rados.rados.0.smithi064.stdout:1872: writing smithi06431743-25 from 659072 to 1295477 tid 1 2024-03-04T06:15:23.280 INFO:tasks.rados.rados.0.smithi064.stdout:1872: writing smithi06431743-25 from 1731866 to 2154164 tid 2 2024-03-04T06:15:23.286 INFO:tasks.rados.rados.0.smithi064.stdout:1872: writing smithi06431743-25 from 2855825 to 3059072 tid 3 2024-03-04T06:15:23.286 INFO:tasks.rados.rados.0.smithi064.stdout:1873: read oid 22 snap -1 2024-03-04T06:15:23.286 INFO:tasks.rados.rados.0.smithi064.stdout:1873: expect (ObjNum 852 snap 206 seq_num 852) 2024-03-04T06:15:23.286 INFO:tasks.rados.rados.0.smithi064.stdout:1874: read oid 45 snap 217 2024-03-04T06:15:23.286 INFO:tasks.rados.rados.0.smithi064.stdout:1874: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:15:23.286 INFO:tasks.rados.rados.0.smithi064.stdout:1875: delete oid 37 current snap is 219 2024-03-04T06:15:23.332 INFO:tasks.rados.rados.0.smithi064.stdout:1870: expect (ObjNum 869 snap 209 seq_num 869) 2024-03-04T06:15:23.431 INFO:tasks.rados.rados.0.smithi064.stdout:1872: finishing write tid 1 to smithi06431743-25 2024-03-04T06:15:23.431 INFO:tasks.rados.rados.0.smithi064.stdout:1871: finishing write tid 1 to smithi06431743-13 2024-03-04T06:15:23.431 INFO:tasks.rados.rados.0.smithi064.stdout:1872: finishing write tid 2 to smithi06431743-25 2024-03-04T06:15:23.431 INFO:tasks.rados.rados.0.smithi064.stdout:1873: expect (ObjNum 852 snap 206 seq_num 852) 2024-03-04T06:15:23.518 INFO:tasks.rados.rados.0.smithi064.stdout:1872: finishing write tid 3 to smithi06431743-25 2024-03-04T06:15:23.518 INFO:tasks.rados.rados.0.smithi064.stdout:1871: finishing write tid 2 to smithi06431743-13 2024-03-04T06:15:23.518 INFO:tasks.rados.rados.0.smithi064.stdout:1870: done (5 left) 2024-03-04T06:15:23.518 INFO:tasks.rados.rados.0.smithi064.stdout:1873: done (4 left) 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:1875: done (3 left) 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:1876: snap_create 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:1872: finishing write tid 4 to smithi06431743-25 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:1872: finishing write tid 5 to smithi06431743-25 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:1872: oid 25 updating version 0 to 405 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:1872: oid 25 updating version 405 to 406 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:1872: oid 25 version 406 is already newer than 405 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:1872: oid 25 version 406 is already newer than 403 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 406 (ObjNum 896 snap 219 seq_num 896) dirty exists 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:1872: left oid 25 (ObjNum 896 snap 219 seq_num 896) 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:1871: finishing write tid 3 to smithi06431743-13 2024-03-04T06:15:23.519 INFO:tasks.rados.rados.0.smithi064.stdout:1874: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:15:23.675 INFO:tasks.rados.rados.0.smithi064.stdout:1871: finishing write tid 4 to smithi06431743-13 2024-03-04T06:15:23.675 INFO:tasks.rados.rados.0.smithi064.stdout:1871: finishing write tid 5 to smithi06431743-13 2024-03-04T06:15:23.675 INFO:tasks.rados.rados.0.smithi064.stdout:1871: oid 13 updating version 0 to 772 2024-03-04T06:15:23.676 INFO:tasks.rados.rados.0.smithi064.stdout:1871: oid 13 updating version 772 to 775 2024-03-04T06:15:23.676 INFO:tasks.rados.rados.0.smithi064.stdout:1871: oid 13 version 775 is already newer than 773 2024-03-04T06:15:23.676 INFO:tasks.rados.rados.0.smithi064.stdout:1871: oid 13 version 775 is already newer than 774 2024-03-04T06:15:23.676 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 775 (ObjNum 895 snap 219 seq_num 895) dirty exists 2024-03-04T06:15:23.676 INFO:tasks.rados.rados.0.smithi064.stdout:1871: left oid 13 (ObjNum 895 snap 219 seq_num 895) 2024-03-04T06:15:24.276 INFO:tasks.rados.rados.0.smithi064.stdout:1871: done (3 left) 2024-03-04T06:15:24.276 INFO:tasks.rados.rados.0.smithi064.stdout:1872: done (2 left) 2024-03-04T06:15:24.276 INFO:tasks.rados.rados.0.smithi064.stdout:1874: done (1 left) 2024-03-04T06:15:24.276 INFO:tasks.rados.rados.0.smithi064.stdout:1876: done (0 left) 2024-03-04T06:15:24.276 INFO:tasks.rados.rados.0.smithi064.stdout:1877: read oid 2 snap -1 2024-03-04T06:15:24.276 INFO:tasks.rados.rados.0.smithi064.stdout:1877: expect deleted 2024-03-04T06:15:24.276 INFO:tasks.rados.rados.0.smithi064.stdout:1878: write (excl) oid 40 current snap is 220 2024-03-04T06:15:24.276 INFO:tasks.rados.rados.0.smithi064.stdout:1878: seq_num 898 ranges {692686=503256,1848830=425967,2910671=182015} 2024-03-04T06:15:24.283 INFO:tasks.rados.rados.0.smithi064.stdout:1878: writing smithi06431743-40 from 692686 to 1195942 tid 1 2024-03-04T06:15:24.289 INFO:tasks.rados.rados.0.smithi064.stdout:1878: writing smithi06431743-40 from 1848830 to 2274797 tid 2 2024-03-04T06:15:24.293 INFO:tasks.rados.rados.0.smithi064.stdout:1878: writing smithi06431743-40 from 2910671 to 3092686 tid 3 2024-03-04T06:15:24.294 INFO:tasks.rados.rados.0.smithi064.stdout:1879: read oid 29 snap -1 2024-03-04T06:15:24.294 INFO:tasks.rados.rados.0.smithi064.stdout:1879: expect (ObjNum 850 snap 206 seq_num 850) 2024-03-04T06:15:24.294 INFO:tasks.rados.rados.0.smithi064.stdout:1880: write oid 5 current snap is 220 2024-03-04T06:15:24.294 INFO:tasks.rados.rados.0.smithi064.stdout:1880: seq_num 899 ranges {709493=578505,1907312=685978} 2024-03-04T06:15:24.301 INFO:tasks.rados.rados.0.smithi064.stdout:1880: writing smithi06431743-5 from 709493 to 1287998 tid 1 2024-03-04T06:15:24.308 INFO:tasks.rados.rados.0.smithi064.stdout:1880: writing smithi06431743-5 from 1907312 to 2593290 tid 2 2024-03-04T06:15:24.309 INFO:tasks.rados.rados.0.smithi064.stdout:1881: write (excl) oid 49 current snap is 220 2024-03-04T06:15:24.309 INFO:tasks.rados.rados.0.smithi064.stdout:1881: seq_num 900 ranges {726300=653754,1965794=429636,2965517=160783} 2024-03-04T06:15:24.318 INFO:tasks.rados.rados.0.smithi064.stdout:1881: writing smithi06431743-49 from 726300 to 1380054 tid 1 2024-03-04T06:15:24.324 INFO:tasks.rados.rados.0.smithi064.stdout:1881: writing smithi06431743-49 from 1965794 to 2395430 tid 2 2024-03-04T06:15:24.329 INFO:tasks.rados.rados.0.smithi064.stdout:1881: writing smithi06431743-49 from 2965517 to 3126300 tid 3 2024-03-04T06:15:24.329 INFO:tasks.rados.rados.0.smithi064.stdout:1882: read oid 45 snap 219 2024-03-04T06:15:24.329 INFO:tasks.rados.rados.0.smithi064.stdout:1882: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:15:24.329 INFO:tasks.rados.rados.0.smithi064.stdout:1883: snap_create 2024-03-04T06:15:24.332 INFO:tasks.rados.rados.0.smithi064.stdout:1878: finishing write tid 1 to smithi06431743-40 2024-03-04T06:15:24.347 INFO:tasks.rados.rados.0.smithi064.stdout:1879: expect (ObjNum 850 snap 206 seq_num 850) 2024-03-04T06:15:24.433 INFO:tasks.rados.rados.0.smithi064.stdout:1878: finishing write tid 2 to smithi06431743-40 2024-03-04T06:15:24.433 INFO:tasks.rados.rados.0.smithi064.stdout:1880: finishing write tid 1 to smithi06431743-5 2024-03-04T06:15:24.433 INFO:tasks.rados.rados.0.smithi064.stdout:1878: finishing write tid 3 to smithi06431743-40 2024-03-04T06:15:24.433 INFO:tasks.rados.rados.0.smithi064.stdout:1878: finishing write tid 4 to smithi06431743-40 2024-03-04T06:15:24.433 INFO:tasks.rados.rados.0.smithi064.stdout:1880: finishing write tid 2 to smithi06431743-5 2024-03-04T06:15:24.433 INFO:tasks.rados.rados.0.smithi064.stdout:1878: finishing write tid 5 to smithi06431743-40 2024-03-04T06:15:24.433 INFO:tasks.rados.rados.0.smithi064.stdout:1878: oid 40 updating version 0 to 387 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:1878: oid 40 updating version 387 to 390 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:1878: oid 40 version 390 is already newer than 390 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:1878: oid 40 version 390 is already newer than 388 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 390 (ObjNum 898 snap 220 seq_num 898) dirty exists 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:1878: left oid 40 (ObjNum 898 snap 220 seq_num 898) 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:1880: finishing write tid 3 to smithi06431743-5 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:1880: finishing write tid 4 to smithi06431743-5 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:1880: oid 5 updating version 0 to 778 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:1880: oid 5 version 778 is already newer than 776 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:1880: oid 5 version 778 is already newer than 778 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 778 (ObjNum 899 snap 220 seq_num 899) dirty exists 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:1880: left oid 5 (ObjNum 899 snap 220 seq_num 899) 2024-03-04T06:15:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:1882: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:15:24.590 INFO:tasks.rados.rados.0.smithi064.stdout:1881: finishing write tid 1 to smithi06431743-49 2024-03-04T06:15:24.590 INFO:tasks.rados.rados.0.smithi064.stdout:1881: finishing write tid 2 to smithi06431743-49 2024-03-04T06:15:24.590 INFO:tasks.rados.rados.0.smithi064.stdout:1881: finishing write tid 3 to smithi06431743-49 2024-03-04T06:15:24.590 INFO:tasks.rados.rados.0.smithi064.stdout:1881: finishing write tid 4 to smithi06431743-49 2024-03-04T06:15:24.590 INFO:tasks.rados.rados.0.smithi064.stdout:1881: finishing write tid 5 to smithi06431743-49 2024-03-04T06:15:24.590 INFO:tasks.rados.rados.0.smithi064.stdout:1881: oid 49 updating version 0 to 236 2024-03-04T06:15:24.590 INFO:tasks.rados.rados.0.smithi064.stdout:1881: oid 49 version 236 is already newer than 233 2024-03-04T06:15:24.590 INFO:tasks.rados.rados.0.smithi064.stdout:1881: oid 49 version 236 is already newer than 236 2024-03-04T06:15:24.590 INFO:tasks.rados.rados.0.smithi064.stdout:1881: oid 49 version 236 is already newer than 234 2024-03-04T06:15:24.590 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 236 (ObjNum 900 snap 220 seq_num 900) dirty exists 2024-03-04T06:15:24.590 INFO:tasks.rados.rados.0.smithi064.stdout:1881: left oid 49 (ObjNum 900 snap 220 seq_num 900) 2024-03-04T06:15:25.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:25.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:25.233+0000 7fa85f75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:25.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:25.237+0000 7fa85f75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:25.278 INFO:tasks.rados.rados.0.smithi064.stdout:1877: done (6 left) 2024-03-04T06:15:25.278 INFO:tasks.rados.rados.0.smithi064.stdout:1878: done (5 left) 2024-03-04T06:15:25.279 INFO:tasks.rados.rados.0.smithi064.stdout:1879: done (4 left) 2024-03-04T06:15:25.279 INFO:tasks.rados.rados.0.smithi064.stdout:1880: done (3 left) 2024-03-04T06:15:25.279 INFO:tasks.rados.rados.0.smithi064.stdout:1881: done (2 left) 2024-03-04T06:15:25.279 INFO:tasks.rados.rados.0.smithi064.stdout:1882: done (1 left) 2024-03-04T06:15:25.279 INFO:tasks.rados.rados.0.smithi064.stdout:1883: done (0 left) 2024-03-04T06:15:25.279 INFO:tasks.rados.rados.0.smithi064.stdout:1884: write (excl) oid 16 current snap is 221 2024-03-04T06:15:25.279 INFO:tasks.rados.rados.0.smithi064.stdout:1884: seq_num 901 ranges {743107=729003,1907923=689647} 2024-03-04T06:15:25.286 INFO:tasks.rados.rados.0.smithi064.stdout:1884: writing smithi06431743-16 from 743107 to 1472110 tid 1 2024-03-04T06:15:25.293 INFO:tasks.rados.rados.0.smithi064.stdout:1884: writing smithi06431743-16 from 1907923 to 2597570 tid 2 2024-03-04T06:15:25.293 INFO:tasks.rados.rados.0.smithi064.stdout:1885: read oid 44 snap -1 2024-03-04T06:15:25.293 INFO:tasks.rados.rados.0.smithi064.stdout:1885: expect (ObjNum 888 snap 216 seq_num 888) 2024-03-04T06:15:25.293 INFO:tasks.rados.rados.0.smithi064.stdout:1886: read oid 29 snap -1 2024-03-04T06:15:25.293 INFO:tasks.rados.rados.0.smithi064.stdout:1886: expect (ObjNum 850 snap 206 seq_num 850) 2024-03-04T06:15:25.293 INFO:tasks.rados.rados.0.smithi064.stdout:1887: snap_create 2024-03-04T06:15:25.321 INFO:tasks.rados.rados.0.smithi064.stdout:1885: expect (ObjNum 888 snap 216 seq_num 888) 2024-03-04T06:15:25.355 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:15:25.355 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:15:25.355 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:15:25.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:15:25.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:25.458+0000 7fb3d4f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:25.466 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:25.465+0000 7fb3d4f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:25.479 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:25.479 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:25.479 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:25.479 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:25.479 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:25.480 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:25.480 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:25.480 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:25.480 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:25.480 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:25.480 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:25.480 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:25.482 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:25.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:15:25.520 INFO:tasks.rados.rados.0.smithi064.stdout:1886: expect (ObjNum 850 snap 206 seq_num 850) 2024-03-04T06:15:25.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:25.586+0000 7f990f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:25.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:25.593+0000 7f990f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:25.606 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:25.606 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:25.606 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:25.606 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:25.607 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:25.607 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:25.607 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:25.607 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:25.607 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:25.607 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:25.608 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:25.608 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:25.609 INFO:tasks.rados.rados.0.smithi064.stdout:1884: finishing write tid 1 to smithi06431743-16 2024-03-04T06:15:25.609 INFO:tasks.rados.rados.0.smithi064.stdout:1884: finishing write tid 2 to smithi06431743-16 2024-03-04T06:15:25.609 INFO:tasks.rados.rados.0.smithi064.stdout:1884: finishing write tid 3 to smithi06431743-16 2024-03-04T06:15:25.609 INFO:tasks.rados.rados.0.smithi064.stdout:1884: finishing write tid 4 to smithi06431743-16 2024-03-04T06:15:25.610 INFO:tasks.rados.rados.0.smithi064.stdout:1884: oid 16 updating version 0 to 354 2024-03-04T06:15:25.610 INFO:tasks.rados.rados.0.smithi064.stdout:1884: oid 16 updating version 354 to 355 2024-03-04T06:15:25.610 INFO:tasks.rados.rados.0.smithi064.stdout:1884: oid 16 version 355 is already newer than 353 2024-03-04T06:15:25.610 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 355 (ObjNum 901 snap 221 seq_num 901) dirty exists 2024-03-04T06:15:25.610 INFO:tasks.rados.rados.0.smithi064.stdout:1884: left oid 16 (ObjNum 901 snap 221 seq_num 901) 2024-03-04T06:15:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:25.611 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:25.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:15:25.759 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:25.754+0000 7fe76c554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:25.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:25.761+0000 7fe76c554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:25.775 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:25.775 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:25.775 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:25.775 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:25.775 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:25.775 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:25.776 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:25.776 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:25.776 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:25.776 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:25.776 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:25.776 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:25.778 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:25.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:15:25.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:25.882+0000 7fa5e5b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:25.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:25.889+0000 7fa5e5b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:25.903 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:25.903 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:25.903 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:25.903 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:25.903 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:25.904 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:25.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:25.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:25.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:25.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:25.904 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:25.904 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:25.906 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:26.279 INFO:tasks.rados.rados.0.smithi064.stdout:1884: done (3 left) 2024-03-04T06:15:26.280 INFO:tasks.rados.rados.0.smithi064.stdout:1885: done (2 left) 2024-03-04T06:15:26.280 INFO:tasks.rados.rados.0.smithi064.stdout:1886: done (1 left) 2024-03-04T06:15:26.280 INFO:tasks.rados.rados.0.smithi064.stdout:1887: done (0 left) 2024-03-04T06:15:26.280 INFO:tasks.rados.rados.0.smithi064.stdout:1888: write oid 5 current snap is 222 2024-03-04T06:15:26.280 INFO:tasks.rados.rados.0.smithi064.stdout:1888: seq_num 902 ranges {759914=404252,1566405=433305,2504010=655904} 2024-03-04T06:15:26.285 INFO:tasks.rados.rados.0.smithi064.stdout:1888: writing smithi06431743-5 from 759914 to 1164166 tid 1 2024-03-04T06:15:26.290 INFO:tasks.rados.rados.0.smithi064.stdout:1888: writing smithi06431743-5 from 1566405 to 1999710 tid 2 2024-03-04T06:15:26.297 INFO:tasks.rados.rados.0.smithi064.stdout:1888: writing smithi06431743-5 from 2504010 to 3159914 tid 3 2024-03-04T06:15:26.297 INFO:tasks.rados.rados.0.smithi064.stdout:1889: read oid 14 snap -1 2024-03-04T06:15:26.297 INFO:tasks.rados.rados.0.smithi064.stdout:1889: expect (ObjNum 811 snap 199 seq_num 811) 2024-03-04T06:15:26.297 INFO:tasks.rados.rados.0.smithi064.stdout:1890: write oid 42 current snap is 222 2024-03-04T06:15:26.297 INFO:tasks.rados.rados.0.smithi064.stdout:1890: seq_num 903 ranges {776721=479501,2024887=693316} 2024-03-04T06:15:26.303 INFO:tasks.rados.rados.0.smithi064.stdout:1890: writing smithi06431743-42 from 776721 to 1256222 tid 1 2024-03-04T06:15:26.312 INFO:tasks.rados.rados.0.smithi064.stdout:1890: writing smithi06431743-42 from 2024887 to 2718203 tid 2 2024-03-04T06:15:26.312 INFO:tasks.rados.rados.0.smithi064.stdout:1891: read oid 7 snap -1 2024-03-04T06:15:26.312 INFO:tasks.rados.rados.0.smithi064.stdout:1891: expect (ObjNum 869 snap 209 seq_num 869) 2024-03-04T06:15:26.313 INFO:tasks.rados.rados.0.smithi064.stdout:1892: read oid 17 snap -1 2024-03-04T06:15:26.313 INFO:tasks.rados.rados.0.smithi064.stdout:1892: expect (ObjNum 771 snap 190 seq_num 771) 2024-03-04T06:15:26.313 INFO:tasks.rados.rados.0.smithi064.stdout:1893: read oid 45 snap -1 2024-03-04T06:15:26.313 INFO:tasks.rados.rados.0.smithi064.stdout:1893: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:15:26.313 INFO:tasks.rados.rados.0.smithi064.stdout:1894: snap_create 2024-03-04T06:15:26.368 INFO:tasks.rados.rados.0.smithi064.stdout:1889: expect (ObjNum 811 snap 199 seq_num 811) 2024-03-04T06:15:26.594 INFO:tasks.rados.rados.0.smithi064.stdout:1891: expect (ObjNum 869 snap 209 seq_num 869) 2024-03-04T06:15:26.688 INFO:tasks.rados.rados.0.smithi064.stdout:1890: finishing write tid 1 to smithi06431743-42 2024-03-04T06:15:26.688 INFO:tasks.rados.rados.0.smithi064.stdout:1888: finishing write tid 1 to smithi06431743-5 2024-03-04T06:15:26.688 INFO:tasks.rados.rados.0.smithi064.stdout:1893: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:15:26.845 INFO:tasks.rados.rados.0.smithi064.stdout:1892: expect (ObjNum 771 snap 190 seq_num 771) 2024-03-04T06:15:27.098 INFO:tasks.rados.rados.0.smithi064.stdout:1888: finishing write tid 2 to smithi06431743-5 2024-03-04T06:15:27.099 INFO:tasks.rados.rados.0.smithi064.stdout:1890: finishing write tid 2 to smithi06431743-42 2024-03-04T06:15:27.099 INFO:tasks.rados.rados.0.smithi064.stdout:1890: finishing write tid 3 to smithi06431743-42 2024-03-04T06:15:27.099 INFO:tasks.rados.rados.0.smithi064.stdout:1890: finishing write tid 4 to smithi06431743-42 2024-03-04T06:15:27.099 INFO:tasks.rados.rados.0.smithi064.stdout:1890: oid 42 updating version 0 to 394 2024-03-04T06:15:27.099 INFO:tasks.rados.rados.0.smithi064.stdout:1890: oid 42 version 394 is already newer than 392 2024-03-04T06:15:27.099 INFO:tasks.rados.rados.0.smithi064.stdout:1890: oid 42 version 394 is already newer than 393 2024-03-04T06:15:27.099 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 394 (ObjNum 903 snap 222 seq_num 903) dirty exists 2024-03-04T06:15:27.099 INFO:tasks.rados.rados.0.smithi064.stdout:1890: left oid 42 (ObjNum 903 snap 222 seq_num 903) 2024-03-04T06:15:27.099 INFO:tasks.rados.rados.0.smithi064.stdout:1888: finishing write tid 3 to smithi06431743-5 2024-03-04T06:15:27.100 INFO:tasks.rados.rados.0.smithi064.stdout:1888: finishing write tid 4 to smithi06431743-5 2024-03-04T06:15:27.100 INFO:tasks.rados.rados.0.smithi064.stdout:1888: finishing write tid 5 to smithi06431743-5 2024-03-04T06:15:27.100 INFO:tasks.rados.rados.0.smithi064.stdout:1888: oid 5 updating version 0 to 783 2024-03-04T06:15:27.100 INFO:tasks.rados.rados.0.smithi064.stdout:1888: oid 5 version 783 is already newer than 783 2024-03-04T06:15:27.100 INFO:tasks.rados.rados.0.smithi064.stdout:1888: oid 5 version 783 is already newer than 781 2024-03-04T06:15:27.100 INFO:tasks.rados.rados.0.smithi064.stdout:1888: oid 5 version 783 is already newer than 780 2024-03-04T06:15:27.100 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 783 (ObjNum 902 snap 222 seq_num 902) dirty exists 2024-03-04T06:15:27.100 INFO:tasks.rados.rados.0.smithi064.stdout:1888: left oid 5 (ObjNum 902 snap 222 seq_num 902) 2024-03-04T06:15:27.284 INFO:tasks.rados.rados.0.smithi064.stdout:1888: done (6 left) 2024-03-04T06:15:27.284 INFO:tasks.rados.rados.0.smithi064.stdout:1889: done (5 left) 2024-03-04T06:15:27.284 INFO:tasks.rados.rados.0.smithi064.stdout:1890: done (4 left) 2024-03-04T06:15:27.284 INFO:tasks.rados.rados.0.smithi064.stdout:1891: done (3 left) 2024-03-04T06:15:27.284 INFO:tasks.rados.rados.0.smithi064.stdout:1892: done (2 left) 2024-03-04T06:15:27.284 INFO:tasks.rados.rados.0.smithi064.stdout:1893: done (1 left) 2024-03-04T06:15:27.284 INFO:tasks.rados.rados.0.smithi064.stdout:1894: done (0 left) 2024-03-04T06:15:27.284 INFO:tasks.rados.rados.0.smithi064.stdout:1895: snap_remove snap 220 2024-03-04T06:15:27.309 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:27.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:27.413+0000 7fe2e1e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:27.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:27.417+0000 7fe2e1e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:28.286 INFO:tasks.rados.rados.0.smithi064.stdout:1895: done (0 left) 2024-03-04T06:15:28.286 INFO:tasks.rados.rados.0.smithi064.stdout:1896: write (excl) oid 19 current snap is 223 2024-03-04T06:15:28.287 INFO:tasks.rados.rados.0.smithi064.stdout:1896: seq_num 904 ranges {793528=554750,2083369=436974,2958856=234672} 2024-03-04T06:15:28.294 INFO:tasks.rados.rados.0.smithi064.stdout:1896: writing smithi06431743-19 from 793528 to 1348278 tid 1 2024-03-04T06:15:28.300 INFO:tasks.rados.rados.0.smithi064.stdout:1896: writing smithi06431743-19 from 2083369 to 2520343 tid 2 2024-03-04T06:15:28.304 INFO:tasks.rados.rados.0.smithi064.stdout:1896: writing smithi06431743-19 from 2958856 to 3193528 tid 3 2024-03-04T06:15:28.304 INFO:tasks.rados.rados.0.smithi064.stdout:1897: read oid 50 snap 219 2024-03-04T06:15:28.304 INFO:tasks.rados.rados.0.smithi064.stdout:1897: expect (ObjNum 796 snap 197 seq_num 796) 2024-03-04T06:15:28.304 INFO:tasks.rados.rados.0.smithi064.stdout:1898: delete oid 6 current snap is 223 2024-03-04T06:15:28.377 INFO:tasks.rados.rados.0.smithi064.stdout:1898: done (2 left) 2024-03-04T06:15:28.377 INFO:tasks.rados.rados.0.smithi064.stdout:1899: read oid 6 snap -1 2024-03-04T06:15:28.377 INFO:tasks.rados.rados.0.smithi064.stdout:1899: expect deleted 2024-03-04T06:15:28.377 INFO:tasks.rados.rados.0.smithi064.stdout:1900: read oid 14 snap 217 2024-03-04T06:15:28.377 INFO:tasks.rados.rados.0.smithi064.stdout:1900: expect (ObjNum 811 snap 199 seq_num 811) 2024-03-04T06:15:28.377 INFO:tasks.rados.rados.0.smithi064.stdout:1901: read oid 8 snap -1 2024-03-04T06:15:28.377 INFO:tasks.rados.rados.0.smithi064.stdout:1901: expect deleted 2024-03-04T06:15:28.377 INFO:tasks.rados.rados.0.smithi064.stdout:1902: snap_remove snap 222 2024-03-04T06:15:29.293 INFO:tasks.rados.rados.0.smithi064.stdout:1902: done (5 left) 2024-03-04T06:15:29.293 INFO:tasks.rados.rados.0.smithi064.stdout:1903: snap_remove snap 216 2024-03-04T06:15:29.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:29.607 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:29.605+0000 7f26a7b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:29.610 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:29.609+0000 7f26a7b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:30.293 INFO:tasks.rados.rados.0.smithi064.stdout:1903: done (5 left) 2024-03-04T06:15:30.293 INFO:tasks.rados.rados.0.smithi064.stdout:1904: write oid 39 current snap is 223 2024-03-04T06:15:30.293 INFO:tasks.rados.rados.0.smithi064.stdout:1904: seq_num 906 ranges {427142=421601,1400333=440643,2613702=450326} 2024-03-04T06:15:30.297 INFO:tasks.rados.rados.0.smithi064.stdout:1904: writing smithi06431743-39 from 427142 to 848743 tid 1 2024-03-04T06:15:30.302 INFO:tasks.rados.rados.0.smithi064.stdout:1904: writing smithi06431743-39 from 1400333 to 1840976 tid 2 2024-03-04T06:15:30.311 INFO:tasks.rados.rados.0.smithi064.stdout:1904: writing smithi06431743-39 from 2613702 to 3064028 tid 3 2024-03-04T06:15:30.311 INFO:tasks.rados.rados.0.smithi064.stdout:1905: read oid 48 snap -1 2024-03-04T06:15:30.311 INFO:tasks.rados.rados.0.smithi064.stdout:1905: expect (ObjNum 889 snap 216 seq_num 889) 2024-03-04T06:15:30.311 INFO:tasks.rados.rados.0.smithi064.stdout:1906: write oid 33 current snap is 223 2024-03-04T06:15:30.311 INFO:tasks.rados.rados.0.smithi064.stdout:1906: seq_num 907 ranges {443949=496850,1458815=584301,2641125=602824} 2024-03-04T06:15:30.316 INFO:tasks.rados.rados.0.smithi064.stdout:1906: writing smithi06431743-33 from 443949 to 940799 tid 1 2024-03-04T06:15:30.323 INFO:tasks.rados.rados.0.smithi064.stdout:1906: writing smithi06431743-33 from 1458815 to 2043116 tid 2 2024-03-04T06:15:30.332 INFO:tasks.rados.rados.0.smithi064.stdout:1906: writing smithi06431743-33 from 2641125 to 3243949 tid 3 2024-03-04T06:15:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:1907: read oid 47 snap -1 2024-03-04T06:15:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:1907: expect (ObjNum 862 snap 207 seq_num 862) 2024-03-04T06:15:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:1908: write (excl) oid 3 current snap is 223 2024-03-04T06:15:30.334 INFO:tasks.rados.rados.0.smithi064.stdout:1897: expect (ObjNum 796 snap 197 seq_num 796) 2024-03-04T06:15:30.541 INFO:tasks.rados.rados.0.smithi064.stdout:1896: finishing write tid 1 to smithi06431743-19 2024-03-04T06:15:30.542 INFO:tasks.rados.rados.0.smithi064.stdout:1896: finishing write tid 2 to smithi06431743-19 2024-03-04T06:15:30.542 INFO:tasks.rados.rados.0.smithi064.stdout:1908: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2024-03-04T06:15:30.547 INFO:tasks.rados.rados.0.smithi064.stdout:1908: writing smithi06431743-3 from 460756 to 1032855 tid 1 2024-03-04T06:15:30.552 INFO:tasks.rados.rados.0.smithi064.stdout:1908: writing smithi06431743-3 from 1517297 to 1961609 tid 2 2024-03-04T06:15:30.561 INFO:tasks.rados.rados.0.smithi064.stdout:1908: writing smithi06431743-3 from 2668548 to 3141418 tid 3 2024-03-04T06:15:30.561 INFO:tasks.rados.rados.0.smithi064.stdout:1897: done (9 left) 2024-03-04T06:15:30.561 INFO:tasks.rados.rados.0.smithi064.stdout:1899: done (8 left) 2024-03-04T06:15:30.561 INFO:tasks.rados.rados.0.smithi064.stdout:1901: done (7 left) 2024-03-04T06:15:30.561 INFO:tasks.rados.rados.0.smithi064.stdout:1909: write (excl) oid 20 current snap is 223 2024-03-04T06:15:30.561 INFO:tasks.rados.rados.0.smithi064.stdout:1909: seq_num 909 ranges {477563=647348,1859426=587970,2979618=297945} 2024-03-04T06:15:30.568 INFO:tasks.rados.rados.0.smithi064.stdout:1909: writing smithi06431743-20 from 477563 to 1124911 tid 1 2024-03-04T06:15:30.577 INFO:tasks.rados.rados.0.smithi064.stdout:1909: writing smithi06431743-20 from 1859426 to 2447396 tid 2 2024-03-04T06:15:30.582 INFO:tasks.rados.rados.0.smithi064.stdout:1909: writing smithi06431743-20 from 2979618 to 3277563 tid 3 2024-03-04T06:15:30.582 INFO:tasks.rados.rados.0.smithi064.stdout:1910: snap_create 2024-03-04T06:15:30.583 INFO:tasks.rados.rados.0.smithi064.stdout:1896: finishing write tid 3 to smithi06431743-19 2024-03-04T06:15:30.583 INFO:tasks.rados.rados.0.smithi064.stdout:1900: expect (ObjNum 811 snap 199 seq_num 811) 2024-03-04T06:15:30.808 INFO:tasks.rados.rados.0.smithi064.stdout:1896: finishing write tid 4 to smithi06431743-19 2024-03-04T06:15:30.809 INFO:tasks.rados.rados.0.smithi064.stdout:1896: finishing write tid 5 to smithi06431743-19 2024-03-04T06:15:30.809 INFO:tasks.rados.rados.0.smithi064.stdout:1896: oid 19 updating version 0 to 790 2024-03-04T06:15:30.809 INFO:tasks.rados.rados.0.smithi064.stdout:1896: oid 19 version 790 is already newer than 788 2024-03-04T06:15:30.809 INFO:tasks.rados.rados.0.smithi064.stdout:1896: oid 19 version 790 is already newer than 790 2024-03-04T06:15:30.809 INFO:tasks.rados.rados.0.smithi064.stdout:1896: oid 19 version 790 is already newer than 787 2024-03-04T06:15:30.809 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 790 (ObjNum 904 snap 223 seq_num 904) dirty exists 2024-03-04T06:15:30.809 INFO:tasks.rados.rados.0.smithi064.stdout:1896: left oid 19 (ObjNum 904 snap 223 seq_num 904) 2024-03-04T06:15:30.809 INFO:tasks.rados.rados.0.smithi064.stdout:1904: finishing write tid 1 to smithi06431743-39 2024-03-04T06:15:30.809 INFO:tasks.rados.rados.0.smithi064.stdout:1905: expect (ObjNum 889 snap 216 seq_num 889) 2024-03-04T06:15:30.907 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:15:30.907 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:15:30.907 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:15:30.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:15:30.917 INFO:tasks.rados.rados.0.smithi064.stdout:1907: expect (ObjNum 862 snap 207 seq_num 862) 2024-03-04T06:15:31.011 INFO:tasks.rados.rados.0.smithi064.stdout:1906: finishing write tid 1 to smithi06431743-33 2024-03-04T06:15:31.011 INFO:tasks.rados.rados.0.smithi064.stdout:1904: finishing write tid 2 to smithi06431743-39 2024-03-04T06:15:31.011 INFO:tasks.rados.rados.0.smithi064.stdout:1904: finishing write tid 3 to smithi06431743-39 2024-03-04T06:15:31.011 INFO:tasks.rados.rados.0.smithi064.stdout:1906: finishing write tid 2 to smithi06431743-33 2024-03-04T06:15:31.012 INFO:tasks.rados.rados.0.smithi064.stdout:1904: finishing write tid 4 to smithi06431743-39 2024-03-04T06:15:31.012 INFO:tasks.rados.rados.0.smithi064.stdout:1904: finishing write tid 5 to smithi06431743-39 2024-03-04T06:15:31.012 INFO:tasks.rados.rados.0.smithi064.stdout:1904: oid 39 updating version 0 to 410 2024-03-04T06:15:31.012 INFO:tasks.rados.rados.0.smithi064.stdout:1904: oid 39 version 410 is already newer than 408 2024-03-04T06:15:31.012 INFO:tasks.rados.rados.0.smithi064.stdout:1904: oid 39 version 410 is already newer than 409 2024-03-04T06:15:31.012 INFO:tasks.rados.rados.0.smithi064.stdout:1904: oid 39 version 410 is already newer than 408 2024-03-04T06:15:31.012 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 410 (ObjNum 906 snap 223 seq_num 906) dirty exists 2024-03-04T06:15:31.012 INFO:tasks.rados.rados.0.smithi064.stdout:1904: left oid 39 (ObjNum 906 snap 223 seq_num 906) 2024-03-04T06:15:31.012 INFO:tasks.rados.rados.0.smithi064.stdout:1906: finishing write tid 3 to smithi06431743-33 2024-03-04T06:15:31.013 INFO:tasks.rados.rados.0.smithi064.stdout:1906: finishing write tid 4 to smithi06431743-33 2024-03-04T06:15:31.013 INFO:tasks.rados.rados.0.smithi064.stdout:1906: finishing write tid 5 to smithi06431743-33 2024-03-04T06:15:31.013 INFO:tasks.rados.rados.0.smithi064.stdout:1906: oid 33 updating version 0 to 644 2024-03-04T06:15:31.013 INFO:tasks.rados.rados.0.smithi064.stdout:1906: oid 33 version 644 is already newer than 643 2024-03-04T06:15:31.013 INFO:tasks.rados.rados.0.smithi064.stdout:1906: oid 33 version 644 is already newer than 642 2024-03-04T06:15:31.013 INFO:tasks.rados.rados.0.smithi064.stdout:1906: oid 33 updating version 644 to 645 2024-03-04T06:15:31.013 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 645 (ObjNum 907 snap 223 seq_num 907) dirty exists 2024-03-04T06:15:31.013 INFO:tasks.rados.rados.0.smithi064.stdout:1906: left oid 33 (ObjNum 907 snap 223 seq_num 907) 2024-03-04T06:15:31.013 INFO:tasks.rados.rados.0.smithi064.stdout:1908: finishing write tid 1 to smithi06431743-3 2024-03-04T06:15:31.013 INFO:tasks.rados.rados.0.smithi064.stdout:1908: finishing write tid 2 to smithi06431743-3 2024-03-04T06:15:31.013 INFO:tasks.rados.rados.0.smithi064.stdout:1908: finishing write tid 3 to smithi06431743-3 2024-03-04T06:15:31.014 INFO:tasks.rados.rados.0.smithi064.stdout:1909: finishing write tid 1 to smithi06431743-20 2024-03-04T06:15:31.014 INFO:tasks.rados.rados.0.smithi064.stdout:1908: finishing write tid 4 to smithi06431743-3 2024-03-04T06:15:31.014 INFO:tasks.rados.rados.0.smithi064.stdout:1908: finishing write tid 5 to smithi06431743-3 2024-03-04T06:15:31.014 INFO:tasks.rados.rados.0.smithi064.stdout:1908: oid 3 updating version 0 to 441 2024-03-04T06:15:31.014 INFO:tasks.rados.rados.0.smithi064.stdout:1908: oid 3 version 441 is already newer than 440 2024-03-04T06:15:31.014 INFO:tasks.rados.rados.0.smithi064.stdout:1908: oid 3 version 441 is already newer than 438 2024-03-04T06:15:31.014 INFO:tasks.rados.rados.0.smithi064.stdout:1908: oid 3 version 441 is already newer than 439 2024-03-04T06:15:31.014 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 441 (ObjNum 908 snap 223 seq_num 908) dirty exists 2024-03-04T06:15:31.014 INFO:tasks.rados.rados.0.smithi064.stdout:1908: left oid 3 (ObjNum 908 snap 223 seq_num 908) 2024-03-04T06:15:31.014 INFO:tasks.rados.rados.0.smithi064.stdout:1909: finishing write tid 2 to smithi06431743-20 2024-03-04T06:15:31.015 INFO:tasks.rados.rados.0.smithi064.stdout:1909: finishing write tid 3 to smithi06431743-20 2024-03-04T06:15:31.015 INFO:tasks.rados.rados.0.smithi064.stdout:1909: finishing write tid 4 to smithi06431743-20 2024-03-04T06:15:31.015 INFO:tasks.rados.rados.0.smithi064.stdout:1909: finishing write tid 5 to smithi06431743-20 2024-03-04T06:15:31.015 INFO:tasks.rados.rados.0.smithi064.stdout:1909: oid 20 updating version 0 to 514 2024-03-04T06:15:31.015 INFO:tasks.rados.rados.0.smithi064.stdout:1909: oid 20 updating version 514 to 515 2024-03-04T06:15:31.015 INFO:tasks.rados.rados.0.smithi064.stdout:1909: oid 20 version 515 is already newer than 513 2024-03-04T06:15:31.015 INFO:tasks.rados.rados.0.smithi064.stdout:1909: oid 20 updating version 515 to 516 2024-03-04T06:15:31.015 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 516 (ObjNum 909 snap 223 seq_num 909) dirty exists 2024-03-04T06:15:31.015 INFO:tasks.rados.rados.0.smithi064.stdout:1909: left oid 20 (ObjNum 909 snap 223 seq_num 909) 2024-03-04T06:15:31.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:31.011+0000 7fb46e79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:31.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:31.018+0000 7fb46e79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:31.031 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:31.031 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:31.031 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:31.031 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:31.031 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:31.032 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:31.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:31.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:31.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:31.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:31.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:31.032 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:31.034 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:31.035 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:31.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:15:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:31.178+0000 7fd44295d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:31.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:31.185+0000 7fd44295d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:31.198 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:31.198 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:31.198 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:31.199 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:31.199 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:31.199 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:31.199 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:31.199 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:31.199 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:31.199 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:31.199 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:31.199 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:31.202 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:31.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:15:31.299 INFO:tasks.rados.rados.0.smithi064.stdout:1896: done (8 left) 2024-03-04T06:15:31.299 INFO:tasks.rados.rados.0.smithi064.stdout:1900: done (7 left) 2024-03-04T06:15:31.299 INFO:tasks.rados.rados.0.smithi064.stdout:1904: done (6 left) 2024-03-04T06:15:31.299 INFO:tasks.rados.rados.0.smithi064.stdout:1905: done (5 left) 2024-03-04T06:15:31.299 INFO:tasks.rados.rados.0.smithi064.stdout:1906: done (4 left) 2024-03-04T06:15:31.299 INFO:tasks.rados.rados.0.smithi064.stdout:1907: done (3 left) 2024-03-04T06:15:31.299 INFO:tasks.rados.rados.0.smithi064.stdout:1908: done (2 left) 2024-03-04T06:15:31.299 INFO:tasks.rados.rados.0.smithi064.stdout:1909: done (1 left) 2024-03-04T06:15:31.300 INFO:tasks.rados.rados.0.smithi064.stdout:1910: done (0 left) 2024-03-04T06:15:31.300 INFO:tasks.rados.rados.0.smithi064.stdout:1911: delete oid 8 current snap is 224 2024-03-04T06:15:31.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:31.308+0000 7f875b02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:31.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:31.315+0000 7f875b02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:31.329 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:31.329 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:31.329 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:31.329 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:31.329 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:31.329 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:31.330 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:31.330 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:31.330 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:31.330 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:31.330 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:31.330 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:31.332 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:31.333 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:31.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:15:31.336 INFO:tasks.rados.rados.0.smithi064.stdout:1911: done (0 left) 2024-03-04T06:15:31.337 INFO:tasks.rados.rados.0.smithi064.stdout:1912: read oid 7 snap -1 2024-03-04T06:15:31.337 INFO:tasks.rados.rados.0.smithi064.stdout:1912: expect (ObjNum 869 snap 209 seq_num 869) 2024-03-04T06:15:31.337 INFO:tasks.rados.rados.0.smithi064.stdout:1913: read oid 50 snap 218 2024-03-04T06:15:31.337 INFO:tasks.rados.rados.0.smithi064.stdout:1913: expect (ObjNum 796 snap 197 seq_num 796) 2024-03-04T06:15:31.337 INFO:tasks.rados.rados.0.smithi064.stdout:1914: snap_remove snap 217 2024-03-04T06:15:31.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:31.439+0000 7f81e286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:31.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:31.446+0000 7f81e286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:31.459 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:31.459 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:31.459 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:31.460 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:31.460 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:31.460 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:31.460 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:31.460 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:31.460 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:31.460 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:31.460 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:31.460 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:31.462 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:31.462 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:31.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:31.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:31.785+0000 7f4e3895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:31.790 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:31.790+0000 7f4e3895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:32.301 INFO:tasks.rados.rados.0.smithi064.stdout:1914: done (2 left) 2024-03-04T06:15:32.302 INFO:tasks.rados.rados.0.smithi064.stdout:1915: delete oid 14 current snap is 224 2024-03-04T06:15:32.302 INFO:tasks.rados.rados.0.smithi064.stdout:1913: expect (ObjNum 796 snap 197 seq_num 796) 2024-03-04T06:15:32.517 INFO:tasks.rados.rados.0.smithi064.stdout:1912: expect (ObjNum 869 snap 209 seq_num 869) 2024-03-04T06:15:32.614 INFO:tasks.rados.rados.0.smithi064.stdout:1912: done (2 left) 2024-03-04T06:15:32.614 INFO:tasks.rados.rados.0.smithi064.stdout:1913: done (1 left) 2024-03-04T06:15:32.614 INFO:tasks.rados.rados.0.smithi064.stdout:1915: done (0 left) 2024-03-04T06:15:32.614 INFO:tasks.rados.rados.0.smithi064.stdout:1916: write oid 31 current snap is 224 2024-03-04T06:15:32.614 INFO:tasks.rados.rados.0.smithi064.stdout:1916: seq_num 912 ranges {527984=473095,1634872=451650,2661887=634311} 2024-03-04T06:15:32.619 INFO:tasks.rados.rados.0.smithi064.stdout:1916: writing smithi06431743-31 from 527984 to 1001079 tid 1 2024-03-04T06:15:32.625 INFO:tasks.rados.rados.0.smithi064.stdout:1916: writing smithi06431743-31 from 1634872 to 2086522 tid 2 2024-03-04T06:15:32.632 INFO:tasks.rados.rados.0.smithi064.stdout:1916: writing smithi06431743-31 from 2661887 to 3296198 tid 3 2024-03-04T06:15:32.632 INFO:tasks.rados.rados.0.smithi064.stdout:1917: snap_remove snap 221 2024-03-04T06:15:33.249 INFO:tasks.rados.rados.0.smithi064.stdout:1917: done (1 left) 2024-03-04T06:15:33.250 INFO:tasks.rados.rados.0.smithi064.stdout:1918: write (excl) oid 48 current snap is 224 2024-03-04T06:15:33.250 INFO:tasks.rados.rados.0.smithi064.stdout:1916: finishing write tid 1 to smithi06431743-31 2024-03-04T06:15:33.250 INFO:tasks.rados.rados.0.smithi064.stdout:1918: seq_num 913 ranges {544791=664697,1693354=595308,2689310=445583} 2024-03-04T06:15:33.256 INFO:tasks.rados.rados.0.smithi064.stdout:1918: writing smithi06431743-48 from 544791 to 1209488 tid 1 2024-03-04T06:15:33.262 INFO:tasks.rados.rados.0.smithi064.stdout:1918: writing smithi06431743-48 from 1693354 to 2288662 tid 2 2024-03-04T06:15:33.266 INFO:tasks.rados.rados.0.smithi064.stdout:1918: writing smithi06431743-48 from 2689310 to 3134893 tid 3 2024-03-04T06:15:33.267 INFO:tasks.rados.rados.0.smithi064.stdout:1919: write (excl) oid 5 current snap is 224 2024-03-04T06:15:33.267 INFO:tasks.rados.rados.0.smithi064.stdout:1916: finishing write tid 2 to smithi06431743-31 2024-03-04T06:15:33.267 INFO:tasks.rados.rados.0.smithi064.stdout:1916: finishing write tid 3 to smithi06431743-31 2024-03-04T06:15:33.267 INFO:tasks.rados.rados.0.smithi064.stdout:1919: seq_num 914 ranges {561598=739946,1751836=455319,2716733=644865} 2024-03-04T06:15:33.273 INFO:tasks.rados.rados.0.smithi064.stdout:1919: writing smithi06431743-5 from 561598 to 1301544 tid 1 2024-03-04T06:15:33.279 INFO:tasks.rados.rados.0.smithi064.stdout:1919: writing smithi06431743-5 from 1751836 to 2207155 tid 2 2024-03-04T06:15:33.285 INFO:tasks.rados.rados.0.smithi064.stdout:1919: writing smithi06431743-5 from 2716733 to 3361598 tid 3 2024-03-04T06:15:33.285 INFO:tasks.rados.rados.0.smithi064.stdout:1920: read oid 42 snap -1 2024-03-04T06:15:33.285 INFO:tasks.rados.rados.0.smithi064.stdout:1920: expect (ObjNum 903 snap 222 seq_num 903) 2024-03-04T06:15:33.285 INFO:tasks.rados.rados.0.smithi064.stdout:1921: read oid 11 snap -1 2024-03-04T06:15:33.285 INFO:tasks.rados.rados.0.smithi064.stdout:1921: expect deleted 2024-03-04T06:15:33.285 INFO:tasks.rados.rados.0.smithi064.stdout:1922: snap_remove snap 223 2024-03-04T06:15:33.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:33.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:33.967+0000 7f747dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:33.995 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:33.971+0000 7f747dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:34.250 INFO:tasks.rados.rados.0.smithi064.stdout:1922: done (5 left) 2024-03-04T06:15:34.250 INFO:tasks.rados.rados.0.smithi064.stdout:1923: delete oid 49 current snap is 224 2024-03-04T06:15:34.250 INFO:tasks.rados.rados.0.smithi064.stdout:1916: finishing write tid 4 to smithi06431743-31 2024-03-04T06:15:34.250 INFO:tasks.rados.rados.0.smithi064.stdout:1916: finishing write tid 5 to smithi06431743-31 2024-03-04T06:15:34.250 INFO:tasks.rados.rados.0.smithi064.stdout:1916: oid 31 updating version 0 to 165 2024-03-04T06:15:34.250 INFO:tasks.rados.rados.0.smithi064.stdout:1916: oid 31 version 165 is already newer than 164 2024-03-04T06:15:34.250 INFO:tasks.rados.rados.0.smithi064.stdout:1916: oid 31 updating version 165 to 166 2024-03-04T06:15:34.251 INFO:tasks.rados.rados.0.smithi064.stdout:1916: oid 31 updating version 166 to 167 2024-03-04T06:15:34.251 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 167 (ObjNum 912 snap 224 seq_num 912) dirty exists 2024-03-04T06:15:34.251 INFO:tasks.rados.rados.0.smithi064.stdout:1916: left oid 31 (ObjNum 912 snap 224 seq_num 912) 2024-03-04T06:15:34.251 INFO:tasks.rados.rados.0.smithi064.stdout:1918: finishing write tid 1 to smithi06431743-48 2024-03-04T06:15:34.251 INFO:tasks.rados.rados.0.smithi064.stdout:1920: expect (ObjNum 903 snap 222 seq_num 903) 2024-03-04T06:15:34.481 INFO:tasks.rados.rados.0.smithi064.stdout:1919: finishing write tid 1 to smithi06431743-5 2024-03-04T06:15:34.482 INFO:tasks.rados.rados.0.smithi064.stdout:1918: finishing write tid 2 to smithi06431743-48 2024-03-04T06:15:34.482 INFO:tasks.rados.rados.0.smithi064.stdout:1919: finishing write tid 2 to smithi06431743-5 2024-03-04T06:15:34.482 INFO:tasks.rados.rados.0.smithi064.stdout:1918: finishing write tid 3 to smithi06431743-48 2024-03-04T06:15:34.482 INFO:tasks.rados.rados.0.smithi064.stdout:1918: finishing write tid 4 to smithi06431743-48 2024-03-04T06:15:34.482 INFO:tasks.rados.rados.0.smithi064.stdout:1918: finishing write tid 5 to smithi06431743-48 2024-03-04T06:15:34.482 INFO:tasks.rados.rados.0.smithi064.stdout:1918: oid 48 updating version 0 to 327 2024-03-04T06:15:34.482 INFO:tasks.rados.rados.0.smithi064.stdout:1918: oid 48 version 327 is already newer than 324 2024-03-04T06:15:34.482 INFO:tasks.rados.rados.0.smithi064.stdout:1918: oid 48 version 327 is already newer than 325 2024-03-04T06:15:34.482 INFO:tasks.rados.rados.0.smithi064.stdout:1918: oid 48 version 327 is already newer than 326 2024-03-04T06:15:34.483 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 327 (ObjNum 913 snap 224 seq_num 913) dirty exists 2024-03-04T06:15:34.483 INFO:tasks.rados.rados.0.smithi064.stdout:1918: left oid 48 (ObjNum 913 snap 224 seq_num 913) 2024-03-04T06:15:34.483 INFO:tasks.rados.rados.0.smithi064.stdout:1919: finishing write tid 3 to smithi06431743-5 2024-03-04T06:15:34.483 INFO:tasks.rados.rados.0.smithi064.stdout:1919: finishing write tid 4 to smithi06431743-5 2024-03-04T06:15:34.483 INFO:tasks.rados.rados.0.smithi064.stdout:1919: finishing write tid 5 to smithi06431743-5 2024-03-04T06:15:34.483 INFO:tasks.rados.rados.0.smithi064.stdout:1919: oid 5 updating version 0 to 807 2024-03-04T06:15:34.483 INFO:tasks.rados.rados.0.smithi064.stdout:1919: oid 5 updating version 807 to 809 2024-03-04T06:15:34.483 INFO:tasks.rados.rados.0.smithi064.stdout:1919: oid 5 version 809 is already newer than 806 2024-03-04T06:15:34.484 INFO:tasks.rados.rados.0.smithi064.stdout:1919: oid 5 version 809 is already newer than 809 2024-03-04T06:15:34.484 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 809 (ObjNum 914 snap 224 seq_num 914) dirty exists 2024-03-04T06:15:34.484 INFO:tasks.rados.rados.0.smithi064.stdout:1919: left oid 5 (ObjNum 914 snap 224 seq_num 914) 2024-03-04T06:15:34.484 INFO:tasks.rados.rados.0.smithi064.stdout:1916: done (5 left) 2024-03-04T06:15:34.484 INFO:tasks.rados.rados.0.smithi064.stdout:1918: done (4 left) 2024-03-04T06:15:34.484 INFO:tasks.rados.rados.0.smithi064.stdout:1919: done (3 left) 2024-03-04T06:15:34.484 INFO:tasks.rados.rados.0.smithi064.stdout:1920: done (2 left) 2024-03-04T06:15:34.484 INFO:tasks.rados.rados.0.smithi064.stdout:1921: done (1 left) 2024-03-04T06:15:34.484 INFO:tasks.rados.rados.0.smithi064.stdout:1923: done (0 left) 2024-03-04T06:15:34.485 INFO:tasks.rados.rados.0.smithi064.stdout:1924: snap_remove snap 218 2024-03-04T06:15:35.319 INFO:tasks.rados.rados.0.smithi064.stdout:1924: done (0 left) 2024-03-04T06:15:35.320 INFO:tasks.rados.rados.0.smithi064.stdout:1925: write oid 36 current snap is 224 2024-03-04T06:15:35.320 INFO:tasks.rados.rados.0.smithi064.stdout:1925: seq_num 916 ranges {595212=490444,1868800=458988,2887932=507280} 2024-03-04T06:15:35.325 INFO:tasks.rados.rados.0.smithi064.stdout:1925: writing smithi06431743-36 from 595212 to 1085656 tid 1 2024-03-04T06:15:35.332 INFO:tasks.rados.rados.0.smithi064.stdout:1925: writing smithi06431743-36 from 1868800 to 2327788 tid 2 2024-03-04T06:15:35.339 INFO:tasks.rados.rados.0.smithi064.stdout:1925: writing smithi06431743-36 from 2887932 to 3395212 tid 3 2024-03-04T06:15:35.339 INFO:tasks.rados.rados.0.smithi064.stdout:1926: read oid 21 snap -1 2024-03-04T06:15:35.339 INFO:tasks.rados.rados.0.smithi064.stdout:1926: expect (ObjNum 843 snap 205 seq_num 843) 2024-03-04T06:15:35.339 INFO:tasks.rados.rados.0.smithi064.stdout:1927: read oid 47 snap -1 2024-03-04T06:15:35.339 INFO:tasks.rados.rados.0.smithi064.stdout:1927: expect (ObjNum 862 snap 207 seq_num 862) 2024-03-04T06:15:35.339 INFO:tasks.rados.rados.0.smithi064.stdout:1928: read oid 28 snap -1 2024-03-04T06:15:35.339 INFO:tasks.rados.rados.0.smithi064.stdout:1928: expect (ObjNum 819 snap 200 seq_num 819) 2024-03-04T06:15:35.339 INFO:tasks.rados.rados.0.smithi064.stdout:1929: write oid 11 current snap is 224 2024-03-04T06:15:35.339 INFO:tasks.rados.rados.0.smithi064.stdout:1929: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2024-03-04T06:15:35.345 INFO:tasks.rados.rados.0.smithi064.stdout:1929: writing smithi06431743-11 from 612019 to 1177712 tid 1 2024-03-04T06:15:35.352 INFO:tasks.rados.rados.0.smithi064.stdout:1929: writing smithi06431743-11 from 1810929 to 2413575 tid 2 2024-03-04T06:15:35.358 INFO:tasks.rados.rados.0.smithi064.stdout:1929: writing smithi06431743-11 from 3082649 to 3412019 tid 3 2024-03-04T06:15:35.358 INFO:tasks.rados.rados.0.smithi064.stdout:1930: write oid 14 current snap is 224 2024-03-04T06:15:35.358 INFO:tasks.rados.rados.0.smithi064.stdout:1930: seq_num 918 ranges {628826=640942,1869411=462657,2826425=534649} 2024-03-04T06:15:35.368 INFO:tasks.rados.rados.0.smithi064.stdout:1930: writing smithi06431743-14 from 628826 to 1269768 tid 1 2024-03-04T06:15:35.377 INFO:tasks.rados.rados.0.smithi064.stdout:1930: writing smithi06431743-14 from 1869411 to 2332068 tid 2 2024-03-04T06:15:35.385 INFO:tasks.rados.rados.0.smithi064.stdout:1930: writing smithi06431743-14 from 2826425 to 3361074 tid 3 2024-03-04T06:15:35.385 INFO:tasks.rados.rados.0.smithi064.stdout:1931: snap_remove snap 219 2024-03-04T06:15:36.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:36.155 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:36.153+0000 7f728795d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:36.158 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:36.158+0000 7f728795d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:36.325 INFO:tasks.rados.rados.0.smithi064.stdout:1931: done (6 left) 2024-03-04T06:15:36.325 INFO:tasks.rados.rados.0.smithi064.stdout:1932: read oid 32 snap -1 2024-03-04T06:15:36.325 INFO:tasks.rados.rados.0.smithi064.stdout:1932: expect (ObjNum 868 snap 209 seq_num 868) 2024-03-04T06:15:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:1933: read oid 3 snap -1 2024-03-04T06:15:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:1933: expect (ObjNum 908 snap 223 seq_num 908) 2024-03-04T06:15:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:1934: read oid 42 snap -1 2024-03-04T06:15:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:1934: expect (ObjNum 903 snap 222 seq_num 903) 2024-03-04T06:15:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:1935: delete oid 46 current snap is 224 2024-03-04T06:15:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:1928: expect (ObjNum 819 snap 200 seq_num 819) 2024-03-04T06:15:36.463 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:15:36.463 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:15:36.463 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:15:36.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:15:36.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:36.567+0000 7fd8d395d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:36.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:36.574+0000 7fd8d395d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:36.588 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:36.588 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:36.592 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:36.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:15:36.630 INFO:tasks.rados.rados.0.smithi064.stdout:1926: expect (ObjNum 843 snap 205 seq_num 843) 2024-03-04T06:15:36.700 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:36.695+0000 7f4ae706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:36.703 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:36.703+0000 7f4ae706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:36.715 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:36.716 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:36.716 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:36.716 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:36.716 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:36.716 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:36.716 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:36.716 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:36.716 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:36.716 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:36.716 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:36.717 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:36.719 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:36.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:15:36.800 INFO:tasks.rados.rados.0.smithi064.stdout:1929: finishing write tid 1 to smithi06431743-11 2024-03-04T06:15:36.800 INFO:tasks.rados.rados.0.smithi064.stdout:1927: expect (ObjNum 862 snap 207 seq_num 862) 2024-03-04T06:15:36.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:36.862+0000 7f632559d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:36.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:36.869+0000 7f632559d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:36.883 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:36.883 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:36.883 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:36.883 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:36.883 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:36.884 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:36.884 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:36.884 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:36.884 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:36.884 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:36.884 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:36.884 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:36.886 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:36.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:15:36.897 INFO:tasks.rados.rados.0.smithi064.stdout:1929: finishing write tid 2 to smithi06431743-11 2024-03-04T06:15:36.897 INFO:tasks.rados.rados.0.smithi064.stdout:1930: finishing write tid 1 to smithi06431743-14 2024-03-04T06:15:36.897 INFO:tasks.rados.rados.0.smithi064.stdout:1925: finishing write tid 1 to smithi06431743-36 2024-03-04T06:15:36.897 INFO:tasks.rados.rados.0.smithi064.stdout:1930: finishing write tid 2 to smithi06431743-14 2024-03-04T06:15:36.897 INFO:tasks.rados.rados.0.smithi064.stdout:1925: finishing write tid 2 to smithi06431743-36 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:1929: finishing write tid 3 to smithi06431743-11 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:1929: finishing write tid 4 to smithi06431743-11 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:1930: finishing write tid 3 to smithi06431743-14 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:1929: finishing write tid 5 to smithi06431743-11 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:1929: oid 11 updating version 0 to 412 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:1929: oid 11 updating version 412 to 413 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:1929: oid 11 updating version 413 to 414 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:1929: oid 11 version 414 is already newer than 412 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 414 (ObjNum 917 snap 224 seq_num 917) dirty exists 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:1929: left oid 11 (ObjNum 917 snap 224 seq_num 917) 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:1925: finishing write tid 3 to smithi06431743-36 2024-03-04T06:15:36.898 INFO:tasks.rados.rados.0.smithi064.stdout:1926: done (9 left) 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1927: done (8 left) 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1928: done (7 left) 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1929: done (6 left) 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1935: done (5 left) 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1936: snap_create 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1925: finishing write tid 4 to smithi06431743-36 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1930: finishing write tid 4 to smithi06431743-14 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1925: finishing write tid 5 to smithi06431743-36 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1925: oid 36 updating version 0 to 369 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1925: oid 36 version 369 is already newer than 367 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1925: oid 36 version 369 is already newer than 367 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:1925: oid 36 version 369 is already newer than 368 2024-03-04T06:15:36.899 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 369 (ObjNum 916 snap 224 seq_num 916) dirty exists 2024-03-04T06:15:36.900 INFO:tasks.rados.rados.0.smithi064.stdout:1925: left oid 36 (ObjNum 916 snap 224 seq_num 916) 2024-03-04T06:15:36.900 INFO:tasks.rados.rados.0.smithi064.stdout:1930: finishing write tid 5 to smithi06431743-14 2024-03-04T06:15:36.900 INFO:tasks.rados.rados.0.smithi064.stdout:1930: oid 14 updating version 0 to 532 2024-03-04T06:15:36.900 INFO:tasks.rados.rados.0.smithi064.stdout:1930: oid 14 updating version 532 to 534 2024-03-04T06:15:36.900 INFO:tasks.rados.rados.0.smithi064.stdout:1930: oid 14 version 534 is already newer than 533 2024-03-04T06:15:36.900 INFO:tasks.rados.rados.0.smithi064.stdout:1930: oid 14 version 534 is already newer than 531 2024-03-04T06:15:36.900 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 534 (ObjNum 918 snap 224 seq_num 918) dirty exists 2024-03-04T06:15:36.900 INFO:tasks.rados.rados.0.smithi064.stdout:1930: left oid 14 (ObjNum 918 snap 224 seq_num 918) 2024-03-04T06:15:36.900 INFO:tasks.rados.rados.0.smithi064.stdout:1933: expect (ObjNum 908 snap 223 seq_num 908) 2024-03-04T06:15:37.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:37.002+0000 7f332c613640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:37.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:37.010+0000 7f332c613640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:37.019 INFO:tasks.rados.rados.0.smithi064.stdout:1934: expect (ObjNum 903 snap 222 seq_num 903) 2024-03-04T06:15:37.023 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:37.023 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:37.023 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:37.023 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:37.024 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:37.024 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:37.024 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:37.024 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:37.024 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:37.024 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:37.024 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:37.024 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:37.026 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:37.026 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:37.235 INFO:tasks.rados.rados.0.smithi064.stdout:1932: expect (ObjNum 868 snap 209 seq_num 868) 2024-03-04T06:15:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:1925: done (5 left) 2024-03-04T06:15:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:1930: done (4 left) 2024-03-04T06:15:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:1932: done (3 left) 2024-03-04T06:15:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:1933: done (2 left) 2024-03-04T06:15:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:1934: done (1 left) 2024-03-04T06:15:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:1936: done (0 left) 2024-03-04T06:15:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:1937: write oid 10 current snap is 225 2024-03-04T06:15:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:1937: seq_num 920 ranges {662440=507793,1702728=749973,2881271=557193} 2024-03-04T06:15:37.340 INFO:tasks.rados.rados.0.smithi064.stdout:1937: writing smithi06431743-10 from 662440 to 1170233 tid 1 2024-03-04T06:15:37.349 INFO:tasks.rados.rados.0.smithi064.stdout:1937: writing smithi06431743-10 from 1702728 to 2452701 tid 2 2024-03-04T06:15:37.356 INFO:tasks.rados.rados.0.smithi064.stdout:1937: writing smithi06431743-10 from 2881271 to 3438464 tid 3 2024-03-04T06:15:37.356 INFO:tasks.rados.rados.0.smithi064.stdout:1938: write (excl) oid 37 current snap is 225 2024-03-04T06:15:37.356 INFO:tasks.rados.rados.0.smithi064.stdout:1938: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2024-03-04T06:15:37.363 INFO:tasks.rados.rados.0.smithi064.stdout:1938: writing smithi06431743-37 from 679247 to 1262289 tid 1 2024-03-04T06:15:37.371 INFO:tasks.rados.rados.0.smithi064.stdout:1938: writing smithi06431743-37 from 2044857 to 2654841 tid 2 2024-03-04T06:15:37.376 INFO:tasks.rados.rados.0.smithi064.stdout:1938: writing smithi06431743-37 from 3192341 to 3479247 tid 3 2024-03-04T06:15:37.377 INFO:tasks.rados.rados.0.smithi064.stdout:1939: snap_create 2024-03-04T06:15:37.377 INFO:tasks.rados.rados.0.smithi064.stdout:1937: finishing write tid 1 to smithi06431743-10 2024-03-04T06:15:37.410 INFO:tasks.rados.rados.0.smithi064.stdout:1937: finishing write tid 2 to smithi06431743-10 2024-03-04T06:15:37.418 INFO:tasks.rados.rados.0.smithi064.stdout:1938: finishing write tid 1 to smithi06431743-37 2024-03-04T06:15:37.420 INFO:tasks.rados.rados.0.smithi064.stdout:1937: finishing write tid 3 to smithi06431743-10 2024-03-04T06:15:37.433 INFO:tasks.rados.rados.0.smithi064.stdout:1937: finishing write tid 4 to smithi06431743-10 2024-03-04T06:15:37.433 INFO:tasks.rados.rados.0.smithi064.stdout:1938: finishing write tid 2 to smithi06431743-37 2024-03-04T06:15:37.434 INFO:tasks.rados.rados.0.smithi064.stdout:1937: finishing write tid 5 to smithi06431743-10 2024-03-04T06:15:37.434 INFO:tasks.rados.rados.0.smithi064.stdout:1937: oid 10 updating version 0 to 442 2024-03-04T06:15:37.434 INFO:tasks.rados.rados.0.smithi064.stdout:1937: oid 10 version 442 is already newer than 440 2024-03-04T06:15:37.434 INFO:tasks.rados.rados.0.smithi064.stdout:1937: oid 10 version 442 is already newer than 441 2024-03-04T06:15:37.434 INFO:tasks.rados.rados.0.smithi064.stdout:1937: oid 10 version 442 is already newer than 439 2024-03-04T06:15:37.434 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 442 (ObjNum 920 snap 225 seq_num 920) dirty exists 2024-03-04T06:15:37.434 INFO:tasks.rados.rados.0.smithi064.stdout:1937: left oid 10 (ObjNum 920 snap 225 seq_num 920) 2024-03-04T06:15:37.441 INFO:tasks.rados.rados.0.smithi064.stdout:1938: finishing write tid 3 to smithi06431743-37 2024-03-04T06:15:37.448 INFO:tasks.rados.rados.0.smithi064.stdout:1938: finishing write tid 4 to smithi06431743-37 2024-03-04T06:15:37.449 INFO:tasks.rados.rados.0.smithi064.stdout:1938: finishing write tid 5 to smithi06431743-37 2024-03-04T06:15:37.449 INFO:tasks.rados.rados.0.smithi064.stdout:1938: oid 37 updating version 0 to 302 2024-03-04T06:15:37.449 INFO:tasks.rados.rados.0.smithi064.stdout:1938: oid 37 updating version 302 to 304 2024-03-04T06:15:37.450 INFO:tasks.rados.rados.0.smithi064.stdout:1938: oid 37 updating version 304 to 305 2024-03-04T06:15:37.450 INFO:tasks.rados.rados.0.smithi064.stdout:1938: oid 37 version 305 is already newer than 304 2024-03-04T06:15:37.450 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 305 (ObjNum 921 snap 225 seq_num 921) dirty exists 2024-03-04T06:15:37.450 INFO:tasks.rados.rados.0.smithi064.stdout:1938: left oid 37 (ObjNum 921 snap 225 seq_num 921) 2024-03-04T06:15:38.226 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:38.339 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:38.337+0000 7f712f26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:38.343 INFO:tasks.rados.rados.0.smithi064.stdout:1937: done (2 left) 2024-03-04T06:15:38.343 INFO:tasks.rados.rados.0.smithi064.stdout:1938: done (1 left) 2024-03-04T06:15:38.343 INFO:tasks.rados.rados.0.smithi064.stdout:1939: done (0 left) 2024-03-04T06:15:38.343 INFO:tasks.rados.rados.0.smithi064.stdout:1940: read oid 21 snap -1 2024-03-04T06:15:38.343 INFO:tasks.rados.rados.0.smithi064.stdout:1940: expect (ObjNum 843 snap 205 seq_num 843) 2024-03-04T06:15:38.343 INFO:tasks.rados.rados.0.smithi064.stdout:1941: read oid 5 snap 225 2024-03-04T06:15:38.343 INFO:tasks.rados.rados.0.smithi064.stdout:1941: expect (ObjNum 914 snap 224 seq_num 914) 2024-03-04T06:15:38.343 INFO:tasks.rados.rados.0.smithi064.stdout:1942: write oid 30 current snap is 226 2024-03-04T06:15:38.343 INFO:tasks.rados.rados.0.smithi064.stdout:1942: seq_num 922 ranges {696054=658291,2103339=753642} 2024-03-04T06:15:38.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:38.343+0000 7f712f26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:38.351 INFO:tasks.rados.rados.0.smithi064.stdout:1942: writing smithi06431743-30 from 696054 to 1354345 tid 1 2024-03-04T06:15:38.359 INFO:tasks.rados.rados.0.smithi064.stdout:1942: writing smithi06431743-30 from 2103339 to 2856981 tid 2 2024-03-04T06:15:38.359 INFO:tasks.rados.rados.0.smithi064.stdout:1943: write oid 23 current snap is 226 2024-03-04T06:15:38.359 INFO:tasks.rados.rados.0.smithi064.stdout:1943: seq_num 923 ranges {712861=733540,2161821=613653,3247187=265674} 2024-03-04T06:15:38.368 INFO:tasks.rados.rados.0.smithi064.stdout:1943: writing smithi06431743-23 from 712861 to 1446401 tid 1 2024-03-04T06:15:38.377 INFO:tasks.rados.rados.0.smithi064.stdout:1943: writing smithi06431743-23 from 2161821 to 2775474 tid 2 2024-03-04T06:15:38.382 INFO:tasks.rados.rados.0.smithi064.stdout:1943: writing smithi06431743-23 from 3247187 to 3512861 tid 3 2024-03-04T06:15:38.382 INFO:tasks.rados.rados.0.smithi064.stdout:1944: delete oid 36 current snap is 226 2024-03-04T06:15:38.403 INFO:tasks.rados.rados.0.smithi064.stdout:1940: expect (ObjNum 843 snap 205 seq_num 843) 2024-03-04T06:15:38.578 INFO:tasks.rados.rados.0.smithi064.stdout:1941: expect (ObjNum 914 snap 224 seq_num 914) 2024-03-04T06:15:38.779 INFO:tasks.rados.rados.0.smithi064.stdout:1943: finishing write tid 1 to smithi06431743-23 2024-03-04T06:15:38.779 INFO:tasks.rados.rados.0.smithi064.stdout:1942: finishing write tid 1 to smithi06431743-30 2024-03-04T06:15:38.779 INFO:tasks.rados.rados.0.smithi064.stdout:1943: finishing write tid 2 to smithi06431743-23 2024-03-04T06:15:38.779 INFO:tasks.rados.rados.0.smithi064.stdout:1942: finishing write tid 2 to smithi06431743-30 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1943: finishing write tid 3 to smithi06431743-23 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1942: finishing write tid 3 to smithi06431743-30 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1942: finishing write tid 4 to smithi06431743-30 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1942: oid 30 updating version 0 to 819 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1942: oid 30 updating version 819 to 820 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1942: oid 30 version 820 is already newer than 819 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 820 (ObjNum 922 snap 226 seq_num 922) dirty exists 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1942: left oid 30 (ObjNum 922 snap 226 seq_num 922) 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1943: finishing write tid 4 to smithi06431743-23 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1940: done (4 left) 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1941: done (3 left) 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1942: done (2 left) 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1944: done (1 left) 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1945: delete oid 2 current snap is 226 2024-03-04T06:15:38.780 INFO:tasks.rados.rados.0.smithi064.stdout:1943: finishing write tid 5 to smithi06431743-23 2024-03-04T06:15:38.781 INFO:tasks.rados.rados.0.smithi064.stdout:1943: oid 23 updating version 0 to 444 2024-03-04T06:15:38.781 INFO:tasks.rados.rados.0.smithi064.stdout:1943: oid 23 updating version 444 to 446 2024-03-04T06:15:38.781 INFO:tasks.rados.rados.0.smithi064.stdout:1943: oid 23 version 446 is already newer than 443 2024-03-04T06:15:38.781 INFO:tasks.rados.rados.0.smithi064.stdout:1943: oid 23 version 446 is already newer than 445 2024-03-04T06:15:38.781 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 446 (ObjNum 923 snap 226 seq_num 923) dirty exists 2024-03-04T06:15:38.781 INFO:tasks.rados.rados.0.smithi064.stdout:1943: left oid 23 (ObjNum 923 snap 226 seq_num 923) 2024-03-04T06:15:38.783 INFO:tasks.rados.rados.0.smithi064.stdout:1943: done (1 left) 2024-03-04T06:15:38.783 INFO:tasks.rados.rados.0.smithi064.stdout:1945: done (0 left) 2024-03-04T06:15:38.783 INFO:tasks.rados.rados.0.smithi064.stdout:1946: write (excl) oid 41 current snap is 226 2024-03-04T06:15:38.783 INFO:tasks.rados.rados.0.smithi064.stdout:1946: seq_num 926 ranges {763282=559287,1820914=760980,3213103=350179} 2024-03-04T06:15:38.790 INFO:tasks.rados.rados.0.smithi064.stdout:1946: writing smithi06431743-41 from 763282 to 1322569 tid 1 2024-03-04T06:15:38.798 INFO:tasks.rados.rados.0.smithi064.stdout:1946: writing smithi06431743-41 from 1820914 to 2581894 tid 2 2024-03-04T06:15:38.804 INFO:tasks.rados.rados.0.smithi064.stdout:1946: writing smithi06431743-41 from 3213103 to 3563282 tid 3 2024-03-04T06:15:38.805 INFO:tasks.rados.rados.0.smithi064.stdout:1947: snap_create 2024-03-04T06:15:38.808 INFO:tasks.rados.rados.0.smithi064.stdout:1946: finishing write tid 1 to smithi06431743-41 2024-03-04T06:15:38.820 INFO:tasks.rados.rados.0.smithi064.stdout:1946: finishing write tid 2 to smithi06431743-41 2024-03-04T06:15:38.831 INFO:tasks.rados.rados.0.smithi064.stdout:1946: finishing write tid 3 to smithi06431743-41 2024-03-04T06:15:38.838 INFO:tasks.rados.rados.0.smithi064.stdout:1946: finishing write tid 4 to smithi06431743-41 2024-03-04T06:15:38.838 INFO:tasks.rados.rados.0.smithi064.stdout:1946: finishing write tid 5 to smithi06431743-41 2024-03-04T06:15:38.839 INFO:tasks.rados.rados.0.smithi064.stdout:1946: oid 41 updating version 0 to 448 2024-03-04T06:15:38.839 INFO:tasks.rados.rados.0.smithi064.stdout:1946: oid 41 updating version 448 to 451 2024-03-04T06:15:38.839 INFO:tasks.rados.rados.0.smithi064.stdout:1946: oid 41 version 451 is already newer than 450 2024-03-04T06:15:38.839 INFO:tasks.rados.rados.0.smithi064.stdout:1946: oid 41 version 451 is already newer than 449 2024-03-04T06:15:38.839 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 451 (ObjNum 926 snap 226 seq_num 926) dirty exists 2024-03-04T06:15:38.839 INFO:tasks.rados.rados.0.smithi064.stdout:1946: left oid 41 (ObjNum 926 snap 226 seq_num 926) 2024-03-04T06:15:39.347 INFO:tasks.rados.rados.0.smithi064.stdout:1946: done (1 left) 2024-03-04T06:15:39.347 INFO:tasks.rados.rados.0.smithi064.stdout:1947: done (0 left) 2024-03-04T06:15:39.347 INFO:tasks.rados.rados.0.smithi064.stdout:1948: read oid 45 snap -1 2024-03-04T06:15:39.347 INFO:tasks.rados.rados.0.smithi064.stdout:1948: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:15:39.347 INFO:tasks.rados.rados.0.smithi064.stdout:1949: read oid 9 snap -1 2024-03-04T06:15:39.347 INFO:tasks.rados.rados.0.smithi064.stdout:1949: expect deleted 2024-03-04T06:15:39.347 INFO:tasks.rados.rados.0.smithi064.stdout:1950: write oid 5 current snap is 227 2024-03-04T06:15:39.347 INFO:tasks.rados.rados.0.smithi064.stdout:1950: seq_num 927 ranges {780089=634536,1879396=620991,3240526=339563} 2024-03-04T06:15:39.354 INFO:tasks.rados.rados.0.smithi064.stdout:1950: writing smithi06431743-5 from 780089 to 1414625 tid 1 2024-03-04T06:15:39.361 INFO:tasks.rados.rados.0.smithi064.stdout:1950: writing smithi06431743-5 from 1879396 to 2500387 tid 2 2024-03-04T06:15:39.368 INFO:tasks.rados.rados.0.smithi064.stdout:1950: writing smithi06431743-5 from 3240526 to 3580089 tid 3 2024-03-04T06:15:39.368 INFO:tasks.rados.rados.0.smithi064.stdout:1951: write (excl) oid 7 current snap is 227 2024-03-04T06:15:39.369 INFO:tasks.rados.rados.0.smithi064.stdout:1951: seq_num 928 ranges {796896=426138,1654231=764649,2984302=480075} 2024-03-04T06:15:39.376 INFO:tasks.rados.rados.0.smithi064.stdout:1951: writing smithi06431743-7 from 796896 to 1223034 tid 1 2024-03-04T06:15:39.385 INFO:tasks.rados.rados.0.smithi064.stdout:1951: writing smithi06431743-7 from 1654231 to 2418880 tid 2 2024-03-04T06:15:39.392 INFO:tasks.rados.rados.0.smithi064.stdout:1951: writing smithi06431743-7 from 2984302 to 3464377 tid 3 2024-03-04T06:15:39.393 INFO:tasks.rados.rados.0.smithi064.stdout:1952: snap_create 2024-03-04T06:15:39.393 INFO:tasks.rados.rados.0.smithi064.stdout:1948: expect (ObjNum 781 snap 192 seq_num 781) 2024-03-04T06:15:39.560 INFO:tasks.rados.rados.0.smithi064.stdout:1950: finishing write tid 1 to smithi06431743-5 2024-03-04T06:15:39.560 INFO:tasks.rados.rados.0.smithi064.stdout:1951: finishing write tid 1 to smithi06431743-7 2024-03-04T06:15:39.560 INFO:tasks.rados.rados.0.smithi064.stdout:1950: finishing write tid 2 to smithi06431743-5 2024-03-04T06:15:39.560 INFO:tasks.rados.rados.0.smithi064.stdout:1951: finishing write tid 2 to smithi06431743-7 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1950: finishing write tid 3 to smithi06431743-5 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1950: finishing write tid 4 to smithi06431743-5 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1950: finishing write tid 5 to smithi06431743-5 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1950: oid 5 updating version 0 to 825 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1950: oid 5 version 825 is already newer than 825 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1950: oid 5 version 825 is already newer than 822 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1950: oid 5 version 825 is already newer than 823 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 825 (ObjNum 927 snap 227 seq_num 927) dirty exists 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1950: left oid 5 (ObjNum 927 snap 227 seq_num 927) 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1951: finishing write tid 3 to smithi06431743-7 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1951: finishing write tid 4 to smithi06431743-7 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1951: finishing write tid 5 to smithi06431743-7 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1951: oid 7 updating version 0 to 337 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1951: oid 7 updating version 337 to 338 2024-03-04T06:15:39.561 INFO:tasks.rados.rados.0.smithi064.stdout:1951: oid 7 version 338 is already newer than 335 2024-03-04T06:15:39.562 INFO:tasks.rados.rados.0.smithi064.stdout:1951: oid 7 version 338 is already newer than 336 2024-03-04T06:15:39.562 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 338 (ObjNum 928 snap 227 seq_num 928) dirty exists 2024-03-04T06:15:39.562 INFO:tasks.rados.rados.0.smithi064.stdout:1951: left oid 7 (ObjNum 928 snap 227 seq_num 928) 2024-03-04T06:15:40.348 INFO:tasks.rados.rados.0.smithi064.stdout:1948: done (4 left) 2024-03-04T06:15:40.348 INFO:tasks.rados.rados.0.smithi064.stdout:1949: done (3 left) 2024-03-04T06:15:40.348 INFO:tasks.rados.rados.0.smithi064.stdout:1950: done (2 left) 2024-03-04T06:15:40.348 INFO:tasks.rados.rados.0.smithi064.stdout:1951: done (1 left) 2024-03-04T06:15:40.348 INFO:tasks.rados.rados.0.smithi064.stdout:1952: done (0 left) 2024-03-04T06:15:40.348 INFO:tasks.rados.rados.0.smithi064.stdout:1953: snap_remove snap 226 2024-03-04T06:15:40.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:40.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:40.572+0000 7feacd6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:40.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:40.577+0000 7feacd6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:41.351 INFO:tasks.rados.rados.0.smithi064.stdout:1953: done (0 left) 2024-03-04T06:15:41.352 INFO:tasks.rados.rados.0.smithi064.stdout:1954: write (excl) oid 13 current snap is 228 2024-03-04T06:15:41.352 INFO:tasks.rados.rados.0.smithi064.stdout:1954: seq_num 929 ranges {413703=501387,1596360=624660,2895372=691347} 2024-03-04T06:15:41.356 INFO:tasks.rados.rados.0.smithi064.stdout:1954: writing smithi06431743-13 from 413703 to 915090 tid 1 2024-03-04T06:15:41.363 INFO:tasks.rados.rados.0.smithi064.stdout:1954: writing smithi06431743-13 from 1596360 to 2221020 tid 2 2024-03-04T06:15:41.372 INFO:tasks.rados.rados.0.smithi064.stdout:1954: writing smithi06431743-13 from 2895372 to 3586719 tid 3 2024-03-04T06:15:41.372 INFO:tasks.rados.rados.0.smithi064.stdout:1955: read oid 44 snap 225 2024-03-04T06:15:41.372 INFO:tasks.rados.rados.0.smithi064.stdout:1955: expect (ObjNum 888 snap 216 seq_num 888) 2024-03-04T06:15:41.372 INFO:tasks.rados.rados.0.smithi064.stdout:1956: read oid 6 snap -1 2024-03-04T06:15:41.372 INFO:tasks.rados.rados.0.smithi064.stdout:1956: expect deleted 2024-03-04T06:15:41.372 INFO:tasks.rados.rados.0.smithi064.stdout:1957: read oid 43 snap -1 2024-03-04T06:15:41.372 INFO:tasks.rados.rados.0.smithi064.stdout:1957: expect (ObjNum 861 snap 207 seq_num 861) 2024-03-04T06:15:41.372 INFO:tasks.rados.rados.0.smithi064.stdout:1958: snap_create 2024-03-04T06:15:41.434 INFO:tasks.rados.rados.0.smithi064.stdout:1955: expect (ObjNum 888 snap 216 seq_num 888) 2024-03-04T06:15:41.622 INFO:tasks.rados.rados.0.smithi064.stdout:1954: finishing write tid 1 to smithi06431743-13 2024-03-04T06:15:41.622 INFO:tasks.rados.rados.0.smithi064.stdout:1957: expect (ObjNum 861 snap 207 seq_num 861) 2024-03-04T06:15:41.711 INFO:tasks.rados.rados.0.smithi064.stdout:1954: finishing write tid 2 to smithi06431743-13 2024-03-04T06:15:41.712 INFO:tasks.rados.rados.0.smithi064.stdout:1954: finishing write tid 3 to smithi06431743-13 2024-03-04T06:15:41.712 INFO:tasks.rados.rados.0.smithi064.stdout:1954: finishing write tid 4 to smithi06431743-13 2024-03-04T06:15:41.712 INFO:tasks.rados.rados.0.smithi064.stdout:1954: finishing write tid 5 to smithi06431743-13 2024-03-04T06:15:41.712 INFO:tasks.rados.rados.0.smithi064.stdout:1954: oid 13 updating version 0 to 830 2024-03-04T06:15:41.712 INFO:tasks.rados.rados.0.smithi064.stdout:1954: oid 13 version 830 is already newer than 829 2024-03-04T06:15:41.712 INFO:tasks.rados.rados.0.smithi064.stdout:1954: oid 13 updating version 830 to 832 2024-03-04T06:15:41.712 INFO:tasks.rados.rados.0.smithi064.stdout:1954: oid 13 version 832 is already newer than 831 2024-03-04T06:15:41.712 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 832 (ObjNum 929 snap 228 seq_num 929) dirty exists 2024-03-04T06:15:41.712 INFO:tasks.rados.rados.0.smithi064.stdout:1954: left oid 13 (ObjNum 929 snap 228 seq_num 929) 2024-03-04T06:15:42.027 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:15:42.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:15:42.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:15:42.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:42.132+0000 7f3b22f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:42.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:42.138+0000 7f3b22f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:42.330 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:15:42.330 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:15:42.342 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":700,"stamp":"2024-03-04T06:15:41.349785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":121221929,"num_objects":677,"num_object_clones":514,"num_object_copies":2031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":121,"num_read":3669,"num_read_kb":2746569,"num_write":3227,"num_write_kb":845255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6695,"ondisk_log_size":6695,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":405100,"kb_used_data":242316,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":377082260,"statfs":{"total":386547056640,"available":386132234240,"internally_reserved":0,"allocated":248131584,"data_stored":257042040,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8202212,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":25,"num_read_kb":21655,"num_write":25,"num_write_kb":7583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020248"},"pg_stats":[{"pgid":"3.f","version":"466'414","reported_seq":1663,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.383164+0000","last_change":"2024-03-04T06:15:36.443650+0000","last_active":"2024-03-04T06:15:39.383164+0000","last_peered":"2024-03-04T06:15:39.383164+0000","last_clean":"2024-03-04T06:15:39.383164+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:15:39.383164+0000","last_undegraded":"2024-03-04T06:15:39.383164+0000","last_fullsized":"2024-03-04T06:15:39.383164+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8740234,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":221,"num_read_kb":172630,"num_write":196,"num_write_kb":53501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"467'305","reported_seq":1591,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.361675+0000","last_change":"2024-03-04T06:15:36.345485+0000","last_active":"2024-03-04T06:15:39.361675+0000","last_peered":"2024-03-04T06:15:39.361675+0000","last_clean":"2024-03-04T06:15:39.361675+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:15:39.361675+0000","last_undegraded":"2024-03-04T06:15:39.361675+0000","last_fullsized":"2024-03-04T06:15:39.361675+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3479247,"num_objects":35,"num_object_clones":24,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":125,"num_read_kb":80765,"num_write":141,"num_write_kb":36971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"465'414","reported_seq":1690,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-04T06:15:36.343964+0000","last_change":"2024-03-04T06:15:36.351860+0000","last_active":"2024-03-04T06:15:36.343964+0000","last_peered":"2024-03-04T06:15:36.343964+0000","last_clean":"2024-03-04T06:15:36.343964+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:15:36.343964+0000","last_undegraded":"2024-03-04T06:15:36.343964+0000","last_fullsized":"2024-03-04T06:15:36.343964+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12824500,"num_objects":50,"num_object_clones":37,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":282,"num_read_kb":221698,"num_write":217,"num_write_kb":59486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"468'322","reported_seq":1621,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.364187+0000","last_change":"2024-03-04T06:15:36.342281+0000","last_active":"2024-03-04T06:15:39.364187+0000","last_peered":"2024-03-04T06:15:39.364187+0000","last_clean":"2024-03-04T06:15:39.364187+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:15:39.364187+0000","last_undegraded":"2024-03-04T06:15:39.364187+0000","last_fullsized":"2024-03-04T06:15:39.364187+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395410,"num_objects":33,"num_object_clones":24,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":185,"num_read_kb":144732,"num_write":163,"num_write_kb":45378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1454,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-04T06:15:36.354459+0000","last_change":"2024-03-04T06:15:36.364229+0000","last_active":"2024-03-04T06:15:36.354459+0000","last_peered":"2024-03-04T06:15:36.354459+0000","last_clean":"2024-03-04T06:15:36.354459+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:15:36.354459+0000","last_undegraded":"2024-03-04T06:15:36.354459+0000","last_fullsized":"2024-03-04T06:15:36.354459+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":46,"num_read_kb":17050,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"467'442","reported_seq":1678,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.378061+0000","last_change":"2024-03-04T06:15:36.352269+0000","last_active":"2024-03-04T06:15:39.378061+0000","last_peered":"2024-03-04T06:15:39.378061+0000","last_clean":"2024-03-04T06:15:39.378061+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:15:39.378061+0000","last_undegraded":"2024-03-04T06:15:39.378061+0000","last_fullsized":"2024-03-04T06:15:39.378061+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8420901,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":207,"num_read_kb":160541,"num_write":221,"num_write_kb":65281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1390,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.379052+0000","last_change":"2024-03-04T06:15:36.355585+0000","last_active":"2024-03-04T06:15:39.379052+0000","last_peered":"2024-03-04T06:15:39.379052+0000","last_clean":"2024-03-04T06:15:39.379052+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:15:39.379052+0000","last_undegraded":"2024-03-04T06:15:39.379052+0000","last_fullsized":"2024-03-04T06:15:39.379052+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"465'243","reported_seq":1545,"reported_epoch":466,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:36.342273+0000","last_change":"2024-03-04T06:15:36.342273+0000","last_active":"2024-03-04T06:15:36.342273+0000","last_peered":"2024-03-04T06:15:36.342273+0000","last_clean":"2024-03-04T06:15:36.342273+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:15:36.342273+0000","last_undegraded":"2024-03-04T06:15:36.342273+0000","last_fullsized":"2024-03-04T06:15:36.342273+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2958197,"num_objects":25,"num_object_clones":17,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":160,"num_read_kb":148475,"num_write":114,"num_write_kb":30084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1287,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-04T06:15:36.343236+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:15:36.343236+0000","last_peered":"2024-03-04T06:15:36.343236+0000","last_clean":"2024-03-04T06:15:36.343236+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:15:36.343236+0000","last_undegraded":"2024-03-04T06:15:36.343236+0000","last_fullsized":"2024-03-04T06:15:36.343236+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1156,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-04T06:15:36.351224+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:15:36.351224+0000","last_peered":"2024-03-04T06:15:36.351224+0000","last_clean":"2024-03-04T06:15:36.351224+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:15:36.351224+0000","last_undegraded":"2024-03-04T06:15:36.351224+0000","last_fullsized":"2024-03-04T06:15:36.351224+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"460'441","reported_seq":1636,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-04T06:15:36.348944+0000","last_change":"2024-03-04T06:15:36.354823+0000","last_active":"2024-03-04T06:15:36.348944+0000","last_peered":"2024-03-04T06:15:36.348944+0000","last_clean":"2024-03-04T06:15:36.348944+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:15:36.348944+0000","last_undegraded":"2024-03-04T06:15:36.348944+0000","last_fullsized":"2024-03-04T06:15:36.348944+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9042760,"num_objects":46,"num_object_clones":34,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":194,"num_read_kb":134392,"num_write":204,"num_write_kb":54713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1175,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.383540+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:15:39.383540+0000","last_peered":"2024-03-04T06:15:39.383540+0000","last_clean":"2024-03-04T06:15:39.383540+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:15:39.383540+0000","last_undegraded":"2024-03-04T06:15:39.383540+0000","last_fullsized":"2024-03-04T06:15:39.383540+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"465'331","reported_seq":1566,"reported_epoch":466,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:36.335732+0000","last_change":"2024-03-04T06:15:36.335732+0000","last_active":"2024-03-04T06:15:36.335732+0000","last_peered":"2024-03-04T06:15:36.335732+0000","last_clean":"2024-03-04T06:15:36.335732+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:15:36.335732+0000","last_undegraded":"2024-03-04T06:15:36.335732+0000","last_fullsized":"2024-03-04T06:15:36.335732+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5950074,"num_objects":31,"num_object_clones":25,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":182,"num_read_kb":141704,"num_write":130,"num_write_kb":37431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1156,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-04T06:15:36.338265+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:15:36.338265+0000","last_peered":"2024-03-04T06:15:36.338265+0000","last_clean":"2024-03-04T06:15:36.338265+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:15:36.338265+0000","last_undegraded":"2024-03-04T06:15:36.338265+0000","last_fullsized":"2024-03-04T06:15:36.338265+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"453'765","reported_seq":1917,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-04T06:15:36.353182+0000","last_change":"2024-03-04T06:15:36.363429+0000","last_active":"2024-03-04T06:15:36.353182+0000","last_peered":"2024-03-04T06:15:36.353182+0000","last_clean":"2024-03-04T06:15:36.353182+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:15:36.353182+0000","last_undegraded":"2024-03-04T06:15:36.353182+0000","last_fullsized":"2024-03-04T06:15:36.353182+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":765,"log_dups_size":0,"ondisk_log_size":765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14925573,"num_objects":79,"num_object_clones":60,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":467,"num_read_kb":372959,"num_write":375,"num_write_kb":100985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1175,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.373500+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:15:39.373500+0000","last_peered":"2024-03-04T06:15:39.373500+0000","last_clean":"2024-03-04T06:15:39.373500+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:15:39.373500+0000","last_undegraded":"2024-03-04T06:15:39.373500+0000","last_fullsized":"2024-03-04T06:15:39.373500+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"469'825","reported_seq":1893,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.462303+0000","last_change":"2024-03-04T06:15:36.413663+0000","last_active":"2024-03-04T06:15:39.462303+0000","last_peered":"2024-03-04T06:15:39.462303+0000","last_clean":"2024-03-04T06:15:39.462303+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:15:39.462303+0000","last_undegraded":"2024-03-04T06:15:39.462303+0000","last_fullsized":"2024-03-04T06:15:39.462303+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18287204,"num_objects":78,"num_object_clones":59,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":350,"num_read_kb":252997,"num_write":369,"num_write_kb":97240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"465'367","reported_seq":1650,"reported_epoch":466,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:36.353980+0000","last_change":"2024-03-04T06:15:36.353980+0000","last_active":"2024-03-04T06:15:36.353980+0000","last_peered":"2024-03-04T06:15:36.353980+0000","last_clean":"2024-03-04T06:15:36.353980+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:15:36.353980+0000","last_undegraded":"2024-03-04T06:15:36.353980+0000","last_fullsized":"2024-03-04T06:15:36.353980+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2168301,"num_objects":38,"num_object_clones":28,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":269,"num_read_kb":212388,"num_write":175,"num_write_kb":47566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1197,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.372098+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:15:39.372098+0000","last_peered":"2024-03-04T06:15:39.372098+0000","last_clean":"2024-03-04T06:15:39.372098+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:15:39.372098+0000","last_undegraded":"2024-03-04T06:15:39.372098+0000","last_fullsized":"2024-03-04T06:15:39.372098+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"468'375","reported_seq":1659,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.367908+0000","last_change":"2024-03-04T06:15:36.429359+0000","last_active":"2024-03-04T06:15:39.367908+0000","last_peered":"2024-03-04T06:15:39.367908+0000","last_clean":"2024-03-04T06:15:39.367908+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:15:39.367908+0000","last_undegraded":"2024-03-04T06:15:39.367908+0000","last_fullsized":"2024-03-04T06:15:39.367908+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3143107,"num_objects":44,"num_object_clones":31,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":204,"num_read_kb":142429,"num_write":177,"num_write_kb":47298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1154,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-04T06:15:36.352105+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:15:36.352105+0000","last_peered":"2024-03-04T06:15:36.352105+0000","last_clean":"2024-03-04T06:15:36.352105+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:15:36.352105+0000","last_undegraded":"2024-03-04T06:15:36.352105+0000","last_fullsized":"2024-03-04T06:15:36.352105+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"465'534","reported_seq":1729,"reported_epoch":466,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:36.345615+0000","last_change":"2024-03-04T06:15:36.345615+0000","last_active":"2024-03-04T06:15:36.345615+0000","last_peered":"2024-03-04T06:15:36.345615+0000","last_clean":"2024-03-04T06:15:36.345615+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:15:36.345615+0000","last_undegraded":"2024-03-04T06:15:36.345615+0000","last_fullsized":"2024-03-04T06:15:36.345615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":534,"log_dups_size":0,"ondisk_log_size":534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13026184,"num_objects":53,"num_object_clones":41,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":305,"num_read_kb":241815,"num_write":253,"num_write_kb":70506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1197,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.363437+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:15:39.363437+0000","last_peered":"2024-03-04T06:15:39.363437+0000","last_clean":"2024-03-04T06:15:39.363437+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:15:39.363437+0000","last_undegraded":"2024-03-04T06:15:39.363437+0000","last_fullsized":"2024-03-04T06:15:39.363437+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"466'653","reported_seq":1830,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-03-04T06:15:39.360742+0000","last_change":"2024-03-04T06:15:36.424686+0000","last_active":"2024-03-04T06:15:39.360742+0000","last_peered":"2024-03-04T06:15:39.360742+0000","last_clean":"2024-03-04T06:15:39.360742+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:15:39.360742+0000","last_undegraded":"2024-03-04T06:15:39.360742+0000","last_fullsized":"2024-03-04T06:15:39.360742+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":653,"log_dups_size":0,"ondisk_log_size":653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8941885,"num_objects":58,"num_object_clones":51,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":369,"num_read_kb":301903,"num_write":280,"num_write_kb":76959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1156,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-04T06:15:36.353140+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:15:36.353140+0000","last_peered":"2024-03-04T06:15:36.353140+0000","last_clean":"2024-03-04T06:15:36.353140+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:15:36.353140+0000","last_undegraded":"2024-03-04T06:15:36.353140+0000","last_fullsized":"2024-03-04T06:15:36.353140+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":120631561,"num_objects":675,"num_object_clones":514,"num_object_copies":2025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":121,"num_read":3566,"num_read_kb":2746478,"num_write":3090,"num_write_kb":843361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6598,"ondisk_log_size":6598,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123420,"kb_used_data":78548,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94248420,"statfs":{"total":96636764160,"available":96510382080,"internally_reserved":0,"allocated":80433152,"data_stored":84575725,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706134,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100624,"kb_used_data":61320,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94271216,"statfs":{"total":96636764160,"available":96533725184,"internally_reserved":0,"allocated":62791680,"data_stored":64907825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706138,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83116,"kb_used_data":43812,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94288724,"statfs":{"total":96636764160,"available":96551653376,"internally_reserved":0,"allocated":44863488,"data_stored":45305723,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706138,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":97940,"kb_used_data":58636,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94273900,"statfs":{"total":96636764160,"available":96536473600,"internally_reserved":0,"allocated":60043264,"data_stored":62252767,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:15:42.356 INFO:tasks.rados.rados.0.smithi064.stdout:1954: done (4 left) 2024-03-04T06:15:42.356 INFO:tasks.rados.rados.0.smithi064.stdout:1955: done (3 left) 2024-03-04T06:15:42.356 INFO:tasks.rados.rados.0.smithi064.stdout:1956: done (2 left) 2024-03-04T06:15:42.356 INFO:tasks.rados.rados.0.smithi064.stdout:1957: done (1 left) 2024-03-04T06:15:42.356 INFO:tasks.rados.rados.0.smithi064.stdout:1958: done (0 left) 2024-03-04T06:15:42.356 INFO:tasks.rados.rados.0.smithi064.stdout:1959: write (excl) oid 44 current snap is 229 2024-03-04T06:15:42.356 INFO:tasks.rados.rados.0.smithi064.stdout:1959: seq_num 930 ranges {430510=576636,1654842=768318,2922795=502619} 2024-03-04T06:15:42.361 INFO:tasks.rados.rados.0.smithi064.stdout:1959: writing smithi06431743-44 from 430510 to 1007146 tid 1 2024-03-04T06:15:42.371 INFO:tasks.rados.rados.0.smithi064.stdout:1959: writing smithi06431743-44 from 1654842 to 2423160 tid 2 2024-03-04T06:15:42.379 INFO:tasks.rados.rados.0.smithi064.stdout:1959: writing smithi06431743-44 from 2922795 to 3425414 tid 3 2024-03-04T06:15:42.379 INFO:tasks.rados.rados.0.smithi064.stdout:1960: delete oid 45 current snap is 229 2024-03-04T06:15:42.427 INFO:tasks.rados.rados.0.smithi064.stdout:1959: finishing write tid 1 to smithi06431743-44 2024-03-04T06:15:42.443 INFO:tasks.rados.rados.0.smithi064.stdout:1959: finishing write tid 2 to smithi06431743-44 2024-03-04T06:15:42.454 INFO:tasks.rados.rados.0.smithi064.stdout:1959: finishing write tid 3 to smithi06431743-44 2024-03-04T06:15:42.457 INFO:tasks.rados.rados.0.smithi064.stdout:1960: done (1 left) 2024-03-04T06:15:42.457 INFO:tasks.rados.rados.0.smithi064.stdout:1961: read oid 11 snap -1 2024-03-04T06:15:42.457 INFO:tasks.rados.rados.0.smithi064.stdout:1961: expect (ObjNum 917 snap 224 seq_num 917) 2024-03-04T06:15:42.458 INFO:tasks.rados.rados.0.smithi064.stdout:1962: delete oid 5 current snap is 229 2024-03-04T06:15:42.464 INFO:tasks.rados.rados.0.smithi064.stdout:1959: finishing write tid 4 to smithi06431743-44 2024-03-04T06:15:42.465 INFO:tasks.rados.rados.0.smithi064.stdout:1959: finishing write tid 5 to smithi06431743-44 2024-03-04T06:15:42.465 INFO:tasks.rados.rados.0.smithi064.stdout:1959: oid 44 updating version 0 to 445 2024-03-04T06:15:42.465 INFO:tasks.rados.rados.0.smithi064.stdout:1959: oid 44 updating version 445 to 446 2024-03-04T06:15:42.466 INFO:tasks.rados.rados.0.smithi064.stdout:1959: oid 44 updating version 446 to 447 2024-03-04T06:15:42.466 INFO:tasks.rados.rados.0.smithi064.stdout:1959: oid 44 version 447 is already newer than 444 2024-03-04T06:15:42.466 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 447 (ObjNum 930 snap 229 seq_num 930) dirty exists 2024-03-04T06:15:42.466 INFO:tasks.rados.rados.0.smithi064.stdout:1959: left oid 44 (ObjNum 930 snap 229 seq_num 930) 2024-03-04T06:15:42.473 INFO:tasks.rados.rados.0.smithi064.stdout:1961: expect (ObjNum 917 snap 224 seq_num 917) 2024-03-04T06:15:42.601 INFO:tasks.rados.rados.0.smithi064.stdout:1959: done (2 left) 2024-03-04T06:15:42.601 INFO:tasks.rados.rados.0.smithi064.stdout:1961: done (1 left) 2024-03-04T06:15:42.601 INFO:tasks.rados.rados.0.smithi064.stdout:1962: done (0 left) 2024-03-04T06:15:42.601 INFO:tasks.rados.rados.0.smithi064.stdout:1963: read oid 49 snap -1 2024-03-04T06:15:42.601 INFO:tasks.rados.rados.0.smithi064.stdout:1963: expect deleted 2024-03-04T06:15:42.601 INFO:tasks.rados.rados.0.smithi064.stdout:1964: snap_create 2024-03-04T06:15:42.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:42.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:42.749+0000 7f86c026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:42.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:42.753+0000 7f86c026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:43.249 INFO:tasks.rados.rados.0.smithi064.stdout:1963: done (1 left) 2024-03-04T06:15:43.249 INFO:tasks.rados.rados.0.smithi064.stdout:1964: done (0 left) 2024-03-04T06:15:43.250 INFO:tasks.rados.rados.0.smithi064.stdout:1965: read oid 19 snap -1 2024-03-04T06:15:43.250 INFO:tasks.rados.rados.0.smithi064.stdout:1965: expect (ObjNum 904 snap 223 seq_num 904) 2024-03-04T06:15:43.250 INFO:tasks.rados.rados.0.smithi064.stdout:1966: write (excl) oid 32 current snap is 230 2024-03-04T06:15:43.250 INFO:tasks.rados.rados.0.smithi064.stdout:1966: seq_num 933 ranges {480931=402383,1313935=515645,2488711=452788,3443230=237701} 2024-03-04T06:15:43.254 INFO:tasks.rados.rados.0.smithi064.stdout:1966: writing smithi06431743-32 from 480931 to 883314 tid 1 2024-03-04T06:15:43.259 INFO:tasks.rados.rados.0.smithi064.stdout:1966: writing smithi06431743-32 from 1313935 to 1829580 tid 2 2024-03-04T06:15:43.265 INFO:tasks.rados.rados.0.smithi064.stdout:1966: writing smithi06431743-32 from 2488711 to 2941499 tid 3 2024-03-04T06:15:43.269 INFO:tasks.rados.rados.0.smithi064.stdout:1966: writing smithi06431743-32 from 3443230 to 3680931 tid 4 2024-03-04T06:15:43.269 INFO:tasks.rados.rados.0.smithi064.stdout:1967: snap_remove snap 224 2024-03-04T06:15:44.253 INFO:tasks.rados.rados.0.smithi064.stdout:1967: done (2 left) 2024-03-04T06:15:44.253 INFO:tasks.rados.rados.0.smithi064.stdout:1968: read oid 46 snap 227 2024-03-04T06:15:44.253 INFO:tasks.rados.rados.0.smithi064.stdout:1968: expect deleted 2024-03-04T06:15:44.253 INFO:tasks.rados.rados.0.smithi064.stdout:1969: read oid 48 snap -1 2024-03-04T06:15:44.253 INFO:tasks.rados.rados.0.smithi064.stdout:1969: expect (ObjNum 913 snap 224 seq_num 913) 2024-03-04T06:15:44.253 INFO:tasks.rados.rados.0.smithi064.stdout:1970: delete oid 13 current snap is 230 2024-03-04T06:15:44.253 INFO:tasks.rados.rados.0.smithi064.stdout:1966: finishing write tid 1 to smithi06431743-32 2024-03-04T06:15:44.254 INFO:tasks.rados.rados.0.smithi064.stdout:1965: expect (ObjNum 904 snap 223 seq_num 904) 2024-03-04T06:15:44.519 INFO:tasks.rados.rados.0.smithi064.stdout:1966: finishing write tid 2 to smithi06431743-32 2024-03-04T06:15:44.519 INFO:tasks.rados.rados.0.smithi064.stdout:1966: finishing write tid 3 to smithi06431743-32 2024-03-04T06:15:44.519 INFO:tasks.rados.rados.0.smithi064.stdout:1966: finishing write tid 4 to smithi06431743-32 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1966: finishing write tid 5 to smithi06431743-32 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1966: finishing write tid 6 to smithi06431743-32 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1966: oid 32 updating version 0 to 771 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1966: oid 32 updating version 771 to 772 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1966: oid 32 version 772 is already newer than 769 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1966: oid 32 version 772 is already newer than 768 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1966: oid 32 version 772 is already newer than 770 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 772 (ObjNum 933 snap 230 seq_num 933) dirty exists 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1966: left oid 32 (ObjNum 933 snap 230 seq_num 933) 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1965: done (4 left) 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1966: done (3 left) 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1970: done (2 left) 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1971: snap_create 2024-03-04T06:15:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:1969: expect (ObjNum 913 snap 224 seq_num 913) 2024-03-04T06:15:44.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:44.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:44.927+0000 7f6f4699d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:44.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:44.931+0000 7f6f4699d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:45.254 INFO:tasks.rados.rados.0.smithi064.stdout:1968: done (2 left) 2024-03-04T06:15:45.254 INFO:tasks.rados.rados.0.smithi064.stdout:1969: done (1 left) 2024-03-04T06:15:45.254 INFO:tasks.rados.rados.0.smithi064.stdout:1971: done (0 left) 2024-03-04T06:15:45.254 INFO:tasks.rados.rados.0.smithi064.stdout:1972: write (excl) oid 1 current snap is 231 2024-03-04T06:15:45.254 INFO:tasks.rados.rados.0.smithi064.stdout:1972: seq_num 935 ranges {514545=552881,1830899=519314,2943557=475332} 2024-03-04T06:15:45.260 INFO:tasks.rados.rados.0.smithi064.stdout:1972: writing smithi06431743-1 from 514545 to 1067426 tid 1 2024-03-04T06:15:45.267 INFO:tasks.rados.rados.0.smithi064.stdout:1972: writing smithi06431743-1 from 1830899 to 2350213 tid 2 2024-03-04T06:15:45.272 INFO:tasks.rados.rados.0.smithi064.stdout:1972: writing smithi06431743-1 from 2943557 to 3418889 tid 3 2024-03-04T06:15:45.272 INFO:tasks.rados.rados.0.smithi064.stdout:1973: delete oid 33 current snap is 231 2024-03-04T06:15:45.333 INFO:tasks.rados.rados.0.smithi064.stdout:1972: finishing write tid 1 to smithi06431743-1 2024-03-04T06:15:45.342 INFO:tasks.rados.rados.0.smithi064.stdout:1973: done (1 left) 2024-03-04T06:15:45.342 INFO:tasks.rados.rados.0.smithi064.stdout:1974: write oid 11 current snap is 231 2024-03-04T06:15:45.342 INFO:tasks.rados.rados.0.smithi064.stdout:1974: seq_num 937 ranges {548159=419732,1664216=522983,2714756=614229} 2024-03-04T06:15:45.347 INFO:tasks.rados.rados.0.smithi064.stdout:1974: writing smithi06431743-11 from 548159 to 967891 tid 1 2024-03-04T06:15:45.356 INFO:tasks.rados.rados.0.smithi064.stdout:1974: writing smithi06431743-11 from 1664216 to 2187199 tid 2 2024-03-04T06:15:45.364 INFO:tasks.rados.rados.0.smithi064.stdout:1974: writing smithi06431743-11 from 2714756 to 3328985 tid 3 2024-03-04T06:15:45.364 INFO:tasks.rados.rados.0.smithi064.stdout:1975: write (excl) oid 50 current snap is 231 2024-03-04T06:15:45.364 INFO:tasks.rados.rados.0.smithi064.stdout:1975: seq_num 938 ranges {564966=494981,1606345=782994,3025826=425501} 2024-03-04T06:15:45.371 INFO:tasks.rados.rados.0.smithi064.stdout:1975: writing smithi06431743-50 from 564966 to 1059947 tid 1 2024-03-04T06:15:45.380 INFO:tasks.rados.rados.0.smithi064.stdout:1975: writing smithi06431743-50 from 1606345 to 2389339 tid 2 2024-03-04T06:15:45.388 INFO:tasks.rados.rados.0.smithi064.stdout:1975: writing smithi06431743-50 from 3025826 to 3451327 tid 3 2024-03-04T06:15:45.388 INFO:tasks.rados.rados.0.smithi064.stdout:1976: write (excl) oid 20 current snap is 231 2024-03-04T06:15:45.388 INFO:tasks.rados.rados.0.smithi064.stdout:1972: finishing write tid 2 to smithi06431743-1 2024-03-04T06:15:45.388 INFO:tasks.rados.rados.0.smithi064.stdout:1976: seq_num 939 ranges {581773=570230,1664827=526652,2653249=636773} 2024-03-04T06:15:45.394 INFO:tasks.rados.rados.0.smithi064.stdout:1976: writing smithi06431743-20 from 581773 to 1152003 tid 1 2024-03-04T06:15:45.402 INFO:tasks.rados.rados.0.smithi064.stdout:1976: writing smithi06431743-20 from 1664827 to 2191479 tid 2 2024-03-04T06:15:45.410 INFO:tasks.rados.rados.0.smithi064.stdout:1976: writing smithi06431743-20 from 2653249 to 3290022 tid 3 2024-03-04T06:15:45.410 INFO:tasks.rados.rados.0.smithi064.stdout:1977: snap_remove snap 229 2024-03-04T06:15:46.258 INFO:tasks.rados.rados.0.smithi064.stdout:1977: done (4 left) 2024-03-04T06:15:46.258 INFO:tasks.rados.rados.0.smithi064.stdout:1978: read oid 9 snap -1 2024-03-04T06:15:46.258 INFO:tasks.rados.rados.0.smithi064.stdout:1978: expect deleted 2024-03-04T06:15:46.258 INFO:tasks.rados.rados.0.smithi064.stdout:1979: snap_remove snap 228 2024-03-04T06:15:47.001 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:47.119 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:47.117+0000 7fe09131d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:47.123 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:47.122+0000 7fe09131d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:47.262 INFO:tasks.rados.rados.0.smithi064.stdout:1979: done (5 left) 2024-03-04T06:15:47.262 INFO:tasks.rados.rados.0.smithi064.stdout:1980: delete oid 40 current snap is 231 2024-03-04T06:15:47.262 INFO:tasks.rados.rados.0.smithi064.stdout:1972: finishing write tid 3 to smithi06431743-1 2024-03-04T06:15:47.262 INFO:tasks.rados.rados.0.smithi064.stdout:1972: finishing write tid 4 to smithi06431743-1 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1972: finishing write tid 5 to smithi06431743-1 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1972: oid 1 updating version 0 to 779 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1972: oid 1 version 779 is already newer than 777 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1972: oid 1 version 779 is already newer than 779 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1972: oid 1 version 779 is already newer than 776 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 779 (ObjNum 935 snap 231 seq_num 935) dirty exists 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1972: left oid 1 (ObjNum 935 snap 231 seq_num 935) 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1975: finishing write tid 1 to smithi06431743-50 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1976: finishing write tid 1 to smithi06431743-20 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1974: finishing write tid 1 to smithi06431743-11 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1974: finishing write tid 2 to smithi06431743-11 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1975: finishing write tid 2 to smithi06431743-50 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1974: finishing write tid 3 to smithi06431743-11 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1974: finishing write tid 4 to smithi06431743-11 2024-03-04T06:15:47.263 INFO:tasks.rados.rados.0.smithi064.stdout:1974: finishing write tid 5 to smithi06431743-11 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1974: oid 11 updating version 0 to 423 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1974: oid 11 version 423 is already newer than 421 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1974: oid 11 version 423 is already newer than 421 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1974: oid 11 version 423 is already newer than 423 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 423 (ObjNum 937 snap 231 seq_num 937) dirty exists 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1974: left oid 11 (ObjNum 937 snap 231 seq_num 937) 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1976: finishing write tid 2 to smithi06431743-20 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1975: finishing write tid 3 to smithi06431743-50 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1975: finishing write tid 4 to smithi06431743-50 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1976: finishing write tid 3 to smithi06431743-20 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1975: finishing write tid 5 to smithi06431743-50 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1975: oid 50 updating version 0 to 657 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1975: oid 50 updating version 657 to 659 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1975: oid 50 version 659 is already newer than 658 2024-03-04T06:15:47.264 INFO:tasks.rados.rados.0.smithi064.stdout:1975: oid 50 updating version 659 to 660 2024-03-04T06:15:47.265 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 660 (ObjNum 938 snap 231 seq_num 938) dirty exists 2024-03-04T06:15:47.265 INFO:tasks.rados.rados.0.smithi064.stdout:1975: left oid 50 (ObjNum 938 snap 231 seq_num 938) 2024-03-04T06:15:47.265 INFO:tasks.rados.rados.0.smithi064.stdout:1976: finishing write tid 4 to smithi06431743-20 2024-03-04T06:15:47.265 INFO:tasks.rados.rados.0.smithi064.stdout:1976: finishing write tid 5 to smithi06431743-20 2024-03-04T06:15:47.265 INFO:tasks.rados.rados.0.smithi064.stdout:1976: oid 20 updating version 0 to 540 2024-03-04T06:15:47.265 INFO:tasks.rados.rados.0.smithi064.stdout:1976: oid 20 updating version 540 to 542 2024-03-04T06:15:47.265 INFO:tasks.rados.rados.0.smithi064.stdout:1976: oid 20 updating version 542 to 543 2024-03-04T06:15:47.265 INFO:tasks.rados.rados.0.smithi064.stdout:1976: oid 20 version 543 is already newer than 541 2024-03-04T06:15:47.265 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 543 (ObjNum 939 snap 231 seq_num 939) dirty exists 2024-03-04T06:15:47.265 INFO:tasks.rados.rados.0.smithi064.stdout:1976: left oid 20 (ObjNum 939 snap 231 seq_num 939) 2024-03-04T06:15:47.323 INFO:tasks.rados.rados.0.smithi064.stdout:1972: done (5 left) 2024-03-04T06:15:47.323 INFO:tasks.rados.rados.0.smithi064.stdout:1974: done (4 left) 2024-03-04T06:15:47.323 INFO:tasks.rados.rados.0.smithi064.stdout:1975: done (3 left) 2024-03-04T06:15:47.323 INFO:tasks.rados.rados.0.smithi064.stdout:1976: done (2 left) 2024-03-04T06:15:47.323 INFO:tasks.rados.rados.0.smithi064.stdout:1978: done (1 left) 2024-03-04T06:15:47.323 INFO:tasks.rados.rados.0.smithi064.stdout:1980: done (0 left) 2024-03-04T06:15:47.323 INFO:tasks.rados.rados.0.smithi064.stdout:1981: delete oid 35 current snap is 231 2024-03-04T06:15:47.339 INFO:tasks.rados.rados.0.smithi064.stdout:1981: done (0 left) 2024-03-04T06:15:47.339 INFO:tasks.rados.rados.0.smithi064.stdout:1982: read oid 32 snap -1 2024-03-04T06:15:47.339 INFO:tasks.rados.rados.0.smithi064.stdout:1982: expect (ObjNum 933 snap 230 seq_num 933) 2024-03-04T06:15:47.339 INFO:tasks.rados.rados.0.smithi064.stdout:1983: read oid 7 snap -1 2024-03-04T06:15:47.339 INFO:tasks.rados.rados.0.smithi064.stdout:1983: expect (ObjNum 928 snap 227 seq_num 928) 2024-03-04T06:15:47.339 INFO:tasks.rados.rados.0.smithi064.stdout:1984: delete oid 48 current snap is 231 2024-03-04T06:15:47.344 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:15:47.344 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:15:47.344 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:15:47.372 INFO:tasks.rados.rados.0.smithi064.stdout:1982: expect (ObjNum 933 snap 230 seq_num 933) 2024-03-04T06:15:47.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:47.455+0000 7fbc0446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:47.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:47.461+0000 7fbc0446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:47.556 INFO:tasks.rados.rados.0.smithi064.stdout:1983: expect (ObjNum 928 snap 227 seq_num 928) 2024-03-04T06:15:47.655 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:15:47.656 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:15:47.668 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2024-03-04T06:15:47.260348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":111136772,"num_objects":689,"num_object_clones":527,"num_object_copies":2067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":125,"num_read":3743,"num_read_kb":2811516,"num_write":3278,"num_write_kb":859982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6780,"ondisk_log_size":6780,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":423216,"kb_used_data":260432,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":377064144,"statfs":{"total":386547056640,"available":386113683456,"internally_reserved":0,"allocated":266682368,"data_stored":282852707,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9463100,"num_objects":10,"num_object_clones":10,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":44,"num_read_kb":37210,"num_write":35,"num_write_kb":9884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012402"},"pg_stats":[{"pgid":"3.f","version":"466'414","reported_seq":1679,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-04T06:15:44.390377+0000","last_change":"2024-03-04T06:15:44.402849+0000","last_active":"2024-03-04T06:15:44.390377+0000","last_peered":"2024-03-04T06:15:44.390377+0000","last_clean":"2024-03-04T06:15:44.390377+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:15:44.390377+0000","last_undegraded":"2024-03-04T06:15:44.390377+0000","last_fullsized":"2024-03-04T06:15:44.390377+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8740234,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":227,"num_read_kb":178880,"num_write":196,"num_write_kb":53501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"467'305","reported_seq":1609,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.293443+0000","last_change":"2024-03-04T06:15:44.282081+0000","last_active":"2024-03-04T06:15:45.293443+0000","last_peered":"2024-03-04T06:15:45.293443+0000","last_clean":"2024-03-04T06:15:45.293443+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:15:45.293443+0000","last_undegraded":"2024-03-04T06:15:45.293443+0000","last_fullsized":"2024-03-04T06:15:45.293443+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3479247,"num_objects":35,"num_object_clones":24,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":125,"num_read_kb":80765,"num_write":141,"num_write_kb":36971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"475'423","reported_seq":1729,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.456391+0000","last_change":"2024-03-04T06:15:44.399984+0000","last_active":"2024-03-04T06:15:45.456391+0000","last_peered":"2024-03-04T06:15:45.456391+0000","last_clean":"2024-03-04T06:15:45.456391+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:15:45.456391+0000","last_undegraded":"2024-03-04T06:15:45.456391+0000","last_fullsized":"2024-03-04T06:15:45.456391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10034373,"num_objects":52,"num_object_clones":39,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":295,"num_read_kb":234941,"num_write":223,"num_write_kb":61007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"468'322","reported_seq":1634,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-04T06:15:44.343598+0000","last_change":"2024-03-04T06:15:44.363255+0000","last_active":"2024-03-04T06:15:44.343598+0000","last_peered":"2024-03-04T06:15:44.343598+0000","last_clean":"2024-03-04T06:15:44.343598+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:15:44.343598+0000","last_undegraded":"2024-03-04T06:15:44.343598+0000","last_fullsized":"2024-03-04T06:15:44.343598+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395410,"num_objects":33,"num_object_clones":24,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":185,"num_read_kb":144732,"num_write":163,"num_write_kb":45378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1481,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.281294+0000","last_change":"2024-03-04T06:15:44.286391+0000","last_active":"2024-03-04T06:15:45.281294+0000","last_peered":"2024-03-04T06:15:45.281294+0000","last_clean":"2024-03-04T06:15:45.281294+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:15:45.281294+0000","last_undegraded":"2024-03-04T06:15:45.281294+0000","last_fullsized":"2024-03-04T06:15:45.281294+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":46,"num_read_kb":17050,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"474'449","reported_seq":1699,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-04T06:15:44.386604+0000","last_change":"2024-03-04T06:15:44.409452+0000","last_active":"2024-03-04T06:15:44.386604+0000","last_peered":"2024-03-04T06:15:44.386604+0000","last_clean":"2024-03-04T06:15:44.386604+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:15:44.386604+0000","last_undegraded":"2024-03-04T06:15:44.386604+0000","last_fullsized":"2024-03-04T06:15:44.386604+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9126795,"num_objects":44,"num_object_clones":35,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":215,"num_read_kb":168111,"num_write":226,"num_write_kb":67087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1403,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-04T06:15:44.384883+0000","last_change":"2024-03-04T06:15:44.399825+0000","last_active":"2024-03-04T06:15:44.384883+0000","last_peered":"2024-03-04T06:15:44.384883+0000","last_clean":"2024-03-04T06:15:44.384883+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:15:44.384883+0000","last_undegraded":"2024-03-04T06:15:44.384883+0000","last_fullsized":"2024-03-04T06:15:44.384883+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"466'245","reported_seq":1573,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.383404+0000","last_change":"2024-03-04T06:15:44.365277+0000","last_active":"2024-03-04T06:15:45.383404+0000","last_peered":"2024-03-04T06:15:45.383404+0000","last_clean":"2024-03-04T06:15:45.383404+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:15:45.383404+0000","last_undegraded":"2024-03-04T06:15:45.383404+0000","last_fullsized":"2024-03-04T06:15:45.383404+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2958197,"num_objects":25,"num_object_clones":17,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":160,"num_read_kb":148475,"num_write":114,"num_write_kb":30084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1312,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.266091+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:15:45.266091+0000","last_peered":"2024-03-04T06:15:45.266091+0000","last_clean":"2024-03-04T06:15:45.266091+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:15:45.266091+0000","last_undegraded":"2024-03-04T06:15:45.266091+0000","last_fullsized":"2024-03-04T06:15:45.266091+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1181,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.393321+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:15:45.393321+0000","last_peered":"2024-03-04T06:15:45.393321+0000","last_clean":"2024-03-04T06:15:45.393321+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:15:45.393321+0000","last_undegraded":"2024-03-04T06:15:45.393321+0000","last_fullsized":"2024-03-04T06:15:45.393321+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"474'455","reported_seq":1676,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.392052+0000","last_change":"2024-03-04T06:15:44.401713+0000","last_active":"2024-03-04T06:15:45.392052+0000","last_peered":"2024-03-04T06:15:45.392052+0000","last_clean":"2024-03-04T06:15:45.392052+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:15:45.392052+0000","last_undegraded":"2024-03-04T06:15:45.392052+0000","last_fullsized":"2024-03-04T06:15:45.392052+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10336899,"num_objects":48,"num_object_clones":36,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":203,"num_read_kb":143949,"num_write":214,"num_write_kb":57923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1186,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-04T06:15:44.391367+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:15:44.391367+0000","last_peered":"2024-03-04T06:15:44.391367+0000","last_clean":"2024-03-04T06:15:44.391367+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:15:44.391367+0000","last_undegraded":"2024-03-04T06:15:44.391367+0000","last_fullsized":"2024-03-04T06:15:44.391367+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"474'342","reported_seq":1602,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.268636+0000","last_change":"2024-03-04T06:15:44.292835+0000","last_active":"2024-03-04T06:15:45.268636+0000","last_peered":"2024-03-04T06:15:45.268636+0000","last_clean":"2024-03-04T06:15:45.268636+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:15:45.268636+0000","last_undegraded":"2024-03-04T06:15:45.268636+0000","last_fullsized":"2024-03-04T06:15:45.268636+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6941687,"num_objects":32,"num_object_clones":26,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":190,"num_read_kb":150422,"num_write":135,"num_write_kb":39064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1181,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.385712+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:15:45.385712+0000","last_peered":"2024-03-04T06:15:45.385712+0000","last_clean":"2024-03-04T06:15:45.385712+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:15:45.385712+0000","last_undegraded":"2024-03-04T06:15:45.385712+0000","last_fullsized":"2024-03-04T06:15:45.385712+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"475'779","reported_seq":1959,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.346799+0000","last_change":"2024-03-04T06:15:44.320709+0000","last_active":"2024-03-04T06:15:45.346799+0000","last_peered":"2024-03-04T06:15:45.346799+0000","last_clean":"2024-03-04T06:15:45.346799+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:15:45.346799+0000","last_undegraded":"2024-03-04T06:15:45.346799+0000","last_fullsized":"2024-03-04T06:15:45.346799+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14000990,"num_objects":80,"num_object_clones":62,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":478,"num_read_kb":377803,"num_write":387,"num_write_kb":104071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1186,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-04T06:15:44.341806+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:15:44.341806+0000","last_peered":"2024-03-04T06:15:44.341806+0000","last_clean":"2024-03-04T06:15:44.341806+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:15:44.341806+0000","last_undegraded":"2024-03-04T06:15:44.341806+0000","last_fullsized":"2024-03-04T06:15:44.341806+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"474'840","reported_seq":1921,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.290315+0000","last_change":"2024-03-04T06:15:44.325597+0000","last_active":"2024-03-04T06:15:45.290315+0000","last_peered":"2024-03-04T06:15:45.290315+0000","last_clean":"2024-03-04T06:15:45.290315+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:15:45.290315+0000","last_undegraded":"2024-03-04T06:15:45.290315+0000","last_fullsized":"2024-03-04T06:15:45.290315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":840,"log_dups_size":0,"ondisk_log_size":840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11664850,"num_objects":81,"num_object_clones":62,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":360,"num_read_kb":261407,"num_write":376,"num_write_kb":99017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"466'369","reported_seq":1681,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.392576+0000","last_change":"2024-03-04T06:15:44.377815+0000","last_active":"2024-03-04T06:15:45.392576+0000","last_peered":"2024-03-04T06:15:45.392576+0000","last_clean":"2024-03-04T06:15:45.392576+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:15:45.392576+0000","last_undegraded":"2024-03-04T06:15:45.392576+0000","last_fullsized":"2024-03-04T06:15:45.392576+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2168301,"num_objects":38,"num_object_clones":28,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":275,"num_read_kb":218742,"num_write":175,"num_write_kb":47566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1213,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.304754+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:15:45.304754+0000","last_peered":"2024-03-04T06:15:45.304754+0000","last_clean":"2024-03-04T06:15:45.304754+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:15:45.304754+0000","last_undegraded":"2024-03-04T06:15:45.304754+0000","last_fullsized":"2024-03-04T06:15:45.304754+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"474'377","reported_seq":1677,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.300888+0000","last_change":"2024-03-04T06:15:44.322189+0000","last_active":"2024-03-04T06:15:45.300888+0000","last_peered":"2024-03-04T06:15:45.300888+0000","last_clean":"2024-03-04T06:15:45.300888+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:15:45.300888+0000","last_undegraded":"2024-03-04T06:15:45.300888+0000","last_fullsized":"2024-03-04T06:15:45.300888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3143107,"num_objects":44,"num_object_clones":31,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":204,"num_read_kb":142429,"num_write":177,"num_write_kb":47298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1179,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.282337+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:15:45.282337+0000","last_peered":"2024-03-04T06:15:45.282337+0000","last_clean":"2024-03-04T06:15:45.282337+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:15:45.282337+0000","last_undegraded":"2024-03-04T06:15:45.282337+0000","last_fullsized":"2024-03-04T06:15:45.282337+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"475'543","reported_seq":1762,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.496423+0000","last_change":"2024-03-04T06:15:44.274558+0000","last_active":"2024-03-04T06:15:45.496423+0000","last_peered":"2024-03-04T06:15:45.496423+0000","last_clean":"2024-03-04T06:15:45.496423+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:15:45.496423+0000","last_undegraded":"2024-03-04T06:15:45.496423+0000","last_fullsized":"2024-03-04T06:15:45.496423+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13530394,"num_objects":54,"num_object_clones":42,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":307,"num_read_kb":241816,"num_write":258,"num_write_kb":72200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1213,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.286831+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:15:45.286831+0000","last_peered":"2024-03-04T06:15:45.286831+0000","last_clean":"2024-03-04T06:15:45.286831+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:15:45.286831+0000","last_undegraded":"2024-03-04T06:15:45.286831+0000","last_fullsized":"2024-03-04T06:15:45.286831+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"466'653","reported_seq":1849,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.292502+0000","last_change":"2024-03-04T06:15:44.281704+0000","last_active":"2024-03-04T06:15:45.292502+0000","last_peered":"2024-03-04T06:15:45.292502+0000","last_clean":"2024-03-04T06:15:45.292502+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:15:45.292502+0000","last_undegraded":"2024-03-04T06:15:45.292502+0000","last_fullsized":"2024-03-04T06:15:45.292502+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":653,"log_dups_size":0,"ondisk_log_size":653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5697936,"num_objects":59,"num_object_clones":52,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":370,"num_read_kb":301903,"num_write":281,"num_write_kb":76959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1181,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-04T06:15:45.393832+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:15:45.393832+0000","last_peered":"2024-03-04T06:15:45.393832+0000","last_clean":"2024-03-04T06:15:45.393832+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:15:45.393832+0000","last_undegraded":"2024-03-04T06:15:45.393832+0000","last_fullsized":"2024-03-04T06:15:45.393832+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":110546404,"num_objects":687,"num_object_clones":527,"num_object_copies":2061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":125,"num_read":3640,"num_read_kb":2811425,"num_write":3141,"num_write_kb":858088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6683,"ondisk_log_size":6683,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706144,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126704,"kb_used_data":81832,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94245136,"statfs":{"total":96636764160,"available":96507019264,"internally_reserved":0,"allocated":83795968,"data_stored":89877730,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706144,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":106588,"kb_used_data":67284,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94265252,"statfs":{"total":96636764160,"available":96527618048,"internally_reserved":0,"allocated":68898816,"data_stored":73207519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706143,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88344,"kb_used_data":49040,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94283496,"statfs":{"total":96636764160,"available":96546299904,"internally_reserved":0,"allocated":50216960,"data_stored":52565977,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706143,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":101580,"kb_used_data":62276,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94270260,"statfs":{"total":96636764160,"available":96532746240,"internally_reserved":0,"allocated":63770624,"data_stored":67201481,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:15:47.743 INFO:tasks.rados.rados.0.smithi064.stdout:1982: done (2 left) 2024-03-04T06:15:47.743 INFO:tasks.rados.rados.0.smithi064.stdout:1983: done (1 left) 2024-03-04T06:15:47.743 INFO:tasks.rados.rados.0.smithi064.stdout:1984: done (0 left) 2024-03-04T06:15:47.743 INFO:tasks.rados.rados.0.smithi064.stdout:1985: snap_remove snap 225 2024-03-04T06:15:48.266 INFO:tasks.rados.rados.0.smithi064.stdout:1985: done (0 left) 2024-03-04T06:15:48.266 INFO:tasks.rados.rados.0.smithi064.stdout:1986: snap_create 2024-03-04T06:15:49.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:49.269 INFO:tasks.rados.rados.0.smithi064.stdout:1986: done (0 left) 2024-03-04T06:15:49.270 INFO:tasks.rados.rados.0.smithi064.stdout:1987: snap_remove snap 231 2024-03-04T06:15:49.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:49.311+0000 7fa84b413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:49.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:49.315+0000 7fa84b413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:50.271 INFO:tasks.rados.rados.0.smithi064.stdout:1987: done (0 left) 2024-03-04T06:15:50.272 INFO:tasks.rados.rados.0.smithi064.stdout:1988: read oid 9 snap -1 2024-03-04T06:15:50.272 INFO:tasks.rados.rados.0.smithi064.stdout:1988: expect deleted 2024-03-04T06:15:50.272 INFO:tasks.rados.rados.0.smithi064.stdout:1989: read oid 26 snap -1 2024-03-04T06:15:50.272 INFO:tasks.rados.rados.0.smithi064.stdout:1989: expect (ObjNum 883 snap 213 seq_num 883) 2024-03-04T06:15:50.272 INFO:tasks.rados.rados.0.smithi064.stdout:1990: write (excl) oid 25 current snap is 232 2024-03-04T06:15:50.272 INFO:tasks.rados.rados.0.smithi064.stdout:1990: seq_num 943 ranges {649001=587579,1898755=533990,3162941=686060} 2024-03-04T06:15:50.278 INFO:tasks.rados.rados.0.smithi064.stdout:1990: writing smithi06431743-25 from 649001 to 1236580 tid 1 2024-03-04T06:15:50.284 INFO:tasks.rados.rados.0.smithi064.stdout:1990: writing smithi06431743-25 from 1898755 to 2432745 tid 2 2024-03-04T06:15:50.293 INFO:tasks.rados.rados.0.smithi064.stdout:1990: writing smithi06431743-25 from 3162941 to 3849001 tid 3 2024-03-04T06:15:50.293 INFO:tasks.rados.rados.0.smithi064.stdout:1991: snap_remove snap 227 2024-03-04T06:15:51.276 INFO:tasks.rados.rados.0.smithi064.stdout:1991: done (3 left) 2024-03-04T06:15:51.276 INFO:tasks.rados.rados.0.smithi064.stdout:1992: snap_remove snap 230 2024-03-04T06:15:51.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:51.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:51.490+0000 7f1621664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:51.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:51.494+0000 7f1621664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:52.277 INFO:tasks.rados.rados.0.smithi064.stdout:1992: done (3 left) 2024-03-04T06:15:52.277 INFO:tasks.rados.rados.0.smithi064.stdout:1993: snap_create 2024-03-04T06:15:52.277 INFO:tasks.rados.rados.0.smithi064.stdout:1989: expect (ObjNum 883 snap 213 seq_num 883) 2024-03-04T06:15:52.507 INFO:tasks.rados.rados.0.smithi064.stdout:1990: finishing write tid 1 to smithi06431743-25 2024-03-04T06:15:52.507 INFO:tasks.rados.rados.0.smithi064.stdout:1990: finishing write tid 2 to smithi06431743-25 2024-03-04T06:15:52.507 INFO:tasks.rados.rados.0.smithi064.stdout:1990: finishing write tid 3 to smithi06431743-25 2024-03-04T06:15:52.507 INFO:tasks.rados.rados.0.smithi064.stdout:1990: finishing write tid 4 to smithi06431743-25 2024-03-04T06:15:52.507 INFO:tasks.rados.rados.0.smithi064.stdout:1990: finishing write tid 5 to smithi06431743-25 2024-03-04T06:15:52.508 INFO:tasks.rados.rados.0.smithi064.stdout:1990: oid 25 updating version 0 to 431 2024-03-04T06:15:52.508 INFO:tasks.rados.rados.0.smithi064.stdout:1990: oid 25 version 431 is already newer than 429 2024-03-04T06:15:52.508 INFO:tasks.rados.rados.0.smithi064.stdout:1990: oid 25 version 431 is already newer than 430 2024-03-04T06:15:52.508 INFO:tasks.rados.rados.0.smithi064.stdout:1990: oid 25 updating version 431 to 432 2024-03-04T06:15:52.508 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 432 (ObjNum 943 snap 232 seq_num 943) dirty exists 2024-03-04T06:15:52.508 INFO:tasks.rados.rados.0.smithi064.stdout:1990: left oid 25 (ObjNum 943 snap 232 seq_num 943) 2024-03-04T06:15:52.670 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:15:52.670 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:15:52.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:15:52.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:52.774+0000 7f637055d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:52.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:52.780+0000 7f637055d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:52.969 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:15:52.970 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:15:52.981 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2024-03-04T06:15:52.808301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103119272,"num_objects":694,"num_object_clones":532,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":128,"num_read":3767,"num_read_kb":2836611,"num_write":3291,"num_write_kb":863413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6842,"ondisk_log_size":6842,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":414348,"kb_used_data":251564,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":377073012,"statfs":{"total":386547056640,"available":386122764288,"internally_reserved":0,"allocated":257601536,"data_stored":268906676,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8017500,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":24,"num_read_kb":25095,"num_write":13,"num_write_kb":3431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000762"},"pg_stats":[{"pgid":"3.f","version":"478'418","reported_seq":1697,"reported_epoch":480,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:50.354072+0000","last_change":"2024-03-04T06:15:50.354072+0000","last_active":"2024-03-04T06:15:50.354072+0000","last_peered":"2024-03-04T06:15:50.354072+0000","last_clean":"2024-03-04T06:15:50.354072+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:15:50.354072+0000","last_undegraded":"2024-03-04T06:15:50.354072+0000","last_fullsized":"2024-03-04T06:15:50.354072+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647548,"num_objects":47,"num_object_clones":37,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":228,"num_read_kb":178880,"num_write":197,"num_write_kb":53501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"467'305","reported_seq":1624,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.301617+0000","last_change":"2024-03-04T06:15:50.308348+0000","last_active":"2024-03-04T06:15:50.301617+0000","last_peered":"2024-03-04T06:15:50.301617+0000","last_clean":"2024-03-04T06:15:50.301617+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:15:50.301617+0000","last_undegraded":"2024-03-04T06:15:50.301617+0000","last_fullsized":"2024-03-04T06:15:50.301617+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3479247,"num_objects":35,"num_object_clones":24,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":125,"num_read_kb":80765,"num_write":141,"num_write_kb":36971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"480'432","reported_seq":1741,"reported_epoch":480,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:50.418803+0000","last_change":"2024-03-04T06:15:50.315436+0000","last_active":"2024-03-04T06:15:50.418803+0000","last_peered":"2024-03-04T06:15:50.418803+0000","last_clean":"2024-03-04T06:15:50.418803+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:15:50.418803+0000","last_undegraded":"2024-03-04T06:15:50.418803+0000","last_fullsized":"2024-03-04T06:15:50.418803+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10824302,"num_objects":53,"num_object_clones":40,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":297,"num_read_kb":234942,"num_write":228,"num_write_kb":62773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"468'322","reported_seq":1651,"reported_epoch":480,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:50.348621+0000","last_change":"2024-03-04T06:15:50.348621+0000","last_active":"2024-03-04T06:15:50.348621+0000","last_peered":"2024-03-04T06:15:50.348621+0000","last_clean":"2024-03-04T06:15:50.348621+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:15:50.348621+0000","last_undegraded":"2024-03-04T06:15:50.348621+0000","last_fullsized":"2024-03-04T06:15:50.348621+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395410,"num_objects":33,"num_object_clones":24,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":185,"num_read_kb":144732,"num_write":163,"num_write_kb":45378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1498,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.290951+0000","last_change":"2024-03-04T06:15:50.301589+0000","last_active":"2024-03-04T06:15:50.290951+0000","last_peered":"2024-03-04T06:15:50.290951+0000","last_clean":"2024-03-04T06:15:50.290951+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:15:50.290951+0000","last_undegraded":"2024-03-04T06:15:50.290951+0000","last_fullsized":"2024-03-04T06:15:50.290951+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":46,"num_read_kb":17050,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"478'453","reported_seq":1716,"reported_epoch":480,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:50.359827+0000","last_change":"2024-03-04T06:15:50.359827+0000","last_active":"2024-03-04T06:15:50.359827+0000","last_peered":"2024-03-04T06:15:50.359827+0000","last_clean":"2024-03-04T06:15:50.359827+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:15:50.359827+0000","last_undegraded":"2024-03-04T06:15:50.359827+0000","last_fullsized":"2024-03-04T06:15:50.359827+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9126795,"num_objects":44,"num_object_clones":35,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":215,"num_read_kb":168111,"num_write":226,"num_write_kb":67087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1420,"reported_epoch":480,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:50.353219+0000","last_change":"2024-03-04T06:15:50.353219+0000","last_active":"2024-03-04T06:15:50.353219+0000","last_peered":"2024-03-04T06:15:50.353219+0000","last_clean":"2024-03-04T06:15:50.353219+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:15:50.353219+0000","last_undegraded":"2024-03-04T06:15:50.353219+0000","last_fullsized":"2024-03-04T06:15:50.353219+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"466'245","reported_seq":1584,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.357906+0000","last_change":"2024-03-04T06:15:50.371773+0000","last_active":"2024-03-04T06:15:50.357906+0000","last_peered":"2024-03-04T06:15:50.357906+0000","last_clean":"2024-03-04T06:15:50.357906+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:15:50.357906+0000","last_undegraded":"2024-03-04T06:15:50.357906+0000","last_fullsized":"2024-03-04T06:15:50.357906+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2958197,"num_objects":25,"num_object_clones":17,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":160,"num_read_kb":148475,"num_write":114,"num_write_kb":30084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1325,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.288410+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:15:50.288410+0000","last_peered":"2024-03-04T06:15:50.288410+0000","last_clean":"2024-03-04T06:15:50.288410+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:15:50.288410+0000","last_undegraded":"2024-03-04T06:15:50.288410+0000","last_fullsized":"2024-03-04T06:15:50.288410+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1188,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.358156+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:15:50.358156+0000","last_peered":"2024-03-04T06:15:50.358156+0000","last_clean":"2024-03-04T06:15:50.358156+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:15:50.358156+0000","last_undegraded":"2024-03-04T06:15:50.358156+0000","last_fullsized":"2024-03-04T06:15:50.358156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"474'455","reported_seq":1683,"reported_epoch":480,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:50.337287+0000","last_change":"2024-03-04T06:15:50.337287+0000","last_active":"2024-03-04T06:15:50.337287+0000","last_peered":"2024-03-04T06:15:50.337287+0000","last_clean":"2024-03-04T06:15:50.337287+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:15:50.337287+0000","last_undegraded":"2024-03-04T06:15:50.337287+0000","last_fullsized":"2024-03-04T06:15:50.337287+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10336899,"num_objects":48,"num_object_clones":36,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":203,"num_read_kb":143949,"num_write":214,"num_write_kb":57923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1200,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.355479+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:15:50.355479+0000","last_peered":"2024-03-04T06:15:50.355479+0000","last_clean":"2024-03-04T06:15:50.355479+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:15:50.355479+0000","last_undegraded":"2024-03-04T06:15:50.355479+0000","last_fullsized":"2024-03-04T06:15:50.355479+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"478'348","reported_seq":1623,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.285486+0000","last_change":"2024-03-04T06:15:50.293168+0000","last_active":"2024-03-04T06:15:50.285486+0000","last_peered":"2024-03-04T06:15:50.285486+0000","last_clean":"2024-03-04T06:15:50.285486+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:15:50.285486+0000","last_undegraded":"2024-03-04T06:15:50.285486+0000","last_fullsized":"2024-03-04T06:15:50.285486+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596896,"num_objects":33,"num_object_clones":27,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":196,"num_read_kb":159334,"num_write":136,"num_write_kb":39064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1188,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.325938+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:15:50.325938+0000","last_peered":"2024-03-04T06:15:50.325938+0000","last_clean":"2024-03-04T06:15:50.325938+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:15:50.325938+0000","last_undegraded":"2024-03-04T06:15:50.325938+0000","last_fullsized":"2024-03-04T06:15:50.325938+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"478'787","reported_seq":1982,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.322272+0000","last_change":"2024-03-04T06:15:50.301020+0000","last_active":"2024-03-04T06:15:50.322272+0000","last_peered":"2024-03-04T06:15:50.322272+0000","last_clean":"2024-03-04T06:15:50.322272+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:15:50.322272+0000","last_undegraded":"2024-03-04T06:15:50.322272+0000","last_fullsized":"2024-03-04T06:15:50.322272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14000990,"num_objects":80,"num_object_clones":62,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":490,"num_read_kb":393984,"num_write":387,"num_write_kb":104071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1200,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.350300+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:15:50.350300+0000","last_peered":"2024-03-04T06:15:50.350300+0000","last_clean":"2024-03-04T06:15:50.350300+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:15:50.350300+0000","last_undegraded":"2024-03-04T06:15:50.350300+0000","last_fullsized":"2024-03-04T06:15:50.350300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"478'846","reported_seq":1933,"reported_epoch":480,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:15:50.293729+0000","last_change":"2024-03-04T06:15:48.420210+0000","last_active":"2024-03-04T06:15:50.293729+0000","last_peered":"2024-03-04T06:15:50.293729+0000","last_clean":"2024-03-04T06:15:50.293729+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:15:50.293729+0000","last_undegraded":"2024-03-04T06:15:50.293729+0000","last_fullsized":"2024-03-04T06:15:50.293729+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":846,"log_dups_size":0,"ondisk_log_size":846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11664850,"num_objects":81,"num_object_clones":62,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":360,"num_read_kb":261407,"num_write":376,"num_write_kb":99017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"466'369","reported_seq":1692,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.365942+0000","last_change":"2024-03-04T06:15:50.369758+0000","last_active":"2024-03-04T06:15:50.365942+0000","last_peered":"2024-03-04T06:15:50.365942+0000","last_clean":"2024-03-04T06:15:50.365942+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:15:50.365942+0000","last_undegraded":"2024-03-04T06:15:50.365942+0000","last_fullsized":"2024-03-04T06:15:50.365942+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2168301,"num_objects":38,"num_object_clones":28,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":275,"num_read_kb":218742,"num_write":175,"num_write_kb":47566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1224,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.311167+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:15:50.311167+0000","last_peered":"2024-03-04T06:15:50.311167+0000","last_clean":"2024-03-04T06:15:50.311167+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:15:50.311167+0000","last_undegraded":"2024-03-04T06:15:50.311167+0000","last_fullsized":"2024-03-04T06:15:50.311167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"478'379","reported_seq":1691,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.320748+0000","last_change":"2024-03-04T06:15:50.323689+0000","last_active":"2024-03-04T06:15:50.320748+0000","last_peered":"2024-03-04T06:15:50.320748+0000","last_clean":"2024-03-04T06:15:50.320748+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:15:50.320748+0000","last_undegraded":"2024-03-04T06:15:50.320748+0000","last_fullsized":"2024-03-04T06:15:50.320748+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3143107,"num_objects":44,"num_object_clones":31,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":204,"num_read_kb":142429,"num_write":177,"num_write_kb":47298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1192,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.296947+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:15:50.296947+0000","last_peered":"2024-03-04T06:15:50.296947+0000","last_clean":"2024-03-04T06:15:50.296947+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:15:50.296947+0000","last_undegraded":"2024-03-04T06:15:50.296947+0000","last_fullsized":"2024-03-04T06:15:50.296947+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"478'551","reported_seq":1780,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.293503+0000","last_change":"2024-03-04T06:15:50.293503+0000","last_active":"2024-03-04T06:15:50.293503+0000","last_peered":"2024-03-04T06:15:50.293503+0000","last_clean":"2024-03-04T06:15:50.293503+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:15:50.293503+0000","last_undegraded":"2024-03-04T06:15:50.293503+0000","last_fullsized":"2024-03-04T06:15:50.293503+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10773848,"num_objects":55,"num_object_clones":43,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":308,"num_read_kb":241816,"num_write":259,"num_write_kb":72200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1224,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.295310+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:15:50.295310+0000","last_peered":"2024-03-04T06:15:50.295310+0000","last_clean":"2024-03-04T06:15:50.295310+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:15:50.295310+0000","last_undegraded":"2024-03-04T06:15:50.295310+0000","last_fullsized":"2024-03-04T06:15:50.295310+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"478'668","reported_seq":1868,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.310141+0000","last_change":"2024-03-04T06:15:50.319996+0000","last_active":"2024-03-04T06:15:50.310141+0000","last_peered":"2024-03-04T06:15:50.310141+0000","last_clean":"2024-03-04T06:15:50.310141+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:15:50.310141+0000","last_undegraded":"2024-03-04T06:15:50.310141+0000","last_fullsized":"2024-03-04T06:15:50.310141+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084530,"num_objects":60,"num_object_clones":53,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":372,"num_read_kb":301904,"num_write":286,"num_write_kb":78624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1188,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-03-04T06:15:50.335588+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:15:50.335588+0000","last_peered":"2024-03-04T06:15:50.335588+0000","last_clean":"2024-03-04T06:15:50.335588+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:15:50.335588+0000","last_undegraded":"2024-03-04T06:15:50.335588+0000","last_fullsized":"2024-03-04T06:15:50.335588+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102528904,"num_objects":692,"num_object_clones":532,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":128,"num_read":3664,"num_read_kb":2836520,"num_write":3154,"num_write_kb":861519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6745,"ondisk_log_size":6745,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":120336,"kb_used_data":75464,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94251504,"statfs":{"total":96636764160,"available":96513540096,"internally_reserved":0,"allocated":77275136,"data_stored":79989418,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105320,"kb_used_data":66016,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94266520,"statfs":{"total":96636764160,"available":96528916480,"internally_reserved":0,"allocated":67600384,"data_stored":70470003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88264,"kb_used_data":48960,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94283576,"statfs":{"total":96636764160,"available":96546381824,"internally_reserved":0,"allocated":50135040,"data_stored":52616864,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100428,"kb_used_data":61124,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94271412,"statfs":{"total":96636764160,"available":96533925888,"internally_reserved":0,"allocated":62590976,"data_stored":65830391,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:15:53.254 INFO:tasks.rados.rados.0.smithi064.stdout:1988: done (3 left) 2024-03-04T06:15:53.254 INFO:tasks.rados.rados.0.smithi064.stdout:1989: done (2 left) 2024-03-04T06:15:53.254 INFO:tasks.rados.rados.0.smithi064.stdout:1990: done (1 left) 2024-03-04T06:15:53.254 INFO:tasks.rados.rados.0.smithi064.stdout:1993: done (0 left) 2024-03-04T06:15:53.254 INFO:tasks.rados.rados.0.smithi064.stdout:1994: delete oid 37 current snap is 233 2024-03-04T06:15:53.339 INFO:tasks.rados.rados.0.smithi064.stdout:1994: done (0 left) 2024-03-04T06:15:53.339 INFO:tasks.rados.rados.0.smithi064.stdout:1995: read oid 43 snap -1 2024-03-04T06:15:53.339 INFO:tasks.rados.rados.0.smithi064.stdout:1995: expect (ObjNum 861 snap 207 seq_num 861) 2024-03-04T06:15:53.339 INFO:tasks.rados.rados.0.smithi064.stdout:1996: snap_remove snap 232 2024-03-04T06:15:53.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:53.671 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:53.669+0000 7f1fc202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:53.674 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:53.673+0000 7f1fc202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:54.290 INFO:tasks.rados.rados.0.smithi064.stdout:1996: done (1 left) 2024-03-04T06:15:54.290 INFO:tasks.rados.rados.0.smithi064.stdout:1997: snap_create 2024-03-04T06:15:54.290 INFO:tasks.rados.rados.0.smithi064.stdout:1995: expect (ObjNum 861 snap 207 seq_num 861) 2024-03-04T06:15:55.294 INFO:tasks.rados.rados.0.smithi064.stdout:1995: done (1 left) 2024-03-04T06:15:55.294 INFO:tasks.rados.rados.0.smithi064.stdout:1997: done (0 left) 2024-03-04T06:15:55.294 INFO:tasks.rados.rados.0.smithi064.stdout:1998: delete oid 30 current snap is 234 2024-03-04T06:15:55.361 INFO:tasks.rados.rados.0.smithi064.stdout:1998: done (0 left) 2024-03-04T06:15:55.361 INFO:tasks.rados.rados.0.smithi064.stdout:1999: read oid 3 snap 233 2024-03-04T06:15:55.361 INFO:tasks.rados.rados.0.smithi064.stdout:1999: expect (ObjNum 908 snap 223 seq_num 908) 2024-03-04T06:15:55.361 INFO:tasks.rados.rados.0.smithi064.stdout:2000: read oid 38 snap -1 2024-03-04T06:15:55.361 INFO:tasks.rados.rados.0.smithi064.stdout:2000: expect (ObjNum 835 snap 204 seq_num 835) 2024-03-04T06:15:55.361 INFO:tasks.rados.rados.0.smithi064.stdout:2001: read oid 42 snap -1 2024-03-04T06:15:55.362 INFO:tasks.rados.rados.0.smithi064.stdout:2001: expect (ObjNum 903 snap 222 seq_num 903) 2024-03-04T06:15:55.362 INFO:tasks.rados.rados.0.smithi064.stdout:2002: read oid 7 snap 233 2024-03-04T06:15:55.362 INFO:tasks.rados.rados.0.smithi064.stdout:2002: expect (ObjNum 928 snap 227 seq_num 928) 2024-03-04T06:15:55.362 INFO:tasks.rados.rados.0.smithi064.stdout:2003: read oid 45 snap 233 2024-03-04T06:15:55.362 INFO:tasks.rados.rados.0.smithi064.stdout:2003: expect deleted 2024-03-04T06:15:55.362 INFO:tasks.rados.rados.0.smithi064.stdout:2004: snap_create 2024-03-04T06:15:55.374 INFO:tasks.rados.rados.0.smithi064.stdout:1999: expect (ObjNum 908 snap 223 seq_num 908) 2024-03-04T06:15:55.501 INFO:tasks.rados.rados.0.smithi064.stdout:2002: expect (ObjNum 928 snap 227 seq_num 928) 2024-03-04T06:15:55.689 INFO:tasks.rados.rados.0.smithi064.stdout:2000: expect (ObjNum 835 snap 204 seq_num 835) 2024-03-04T06:15:55.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:15:55.808 INFO:tasks.rados.rados.0.smithi064.stdout:2001: expect (ObjNum 903 snap 222 seq_num 903) 2024-03-04T06:15:55.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:55.846+0000 7fea3f91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:55.851 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:55.851+0000 7fea3f91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:56.295 INFO:tasks.rados.rados.0.smithi064.stdout:1999: done (5 left) 2024-03-04T06:15:56.295 INFO:tasks.rados.rados.0.smithi064.stdout:2000: done (4 left) 2024-03-04T06:15:56.295 INFO:tasks.rados.rados.0.smithi064.stdout:2001: done (3 left) 2024-03-04T06:15:56.295 INFO:tasks.rados.rados.0.smithi064.stdout:2002: done (2 left) 2024-03-04T06:15:56.296 INFO:tasks.rados.rados.0.smithi064.stdout:2003: done (1 left) 2024-03-04T06:15:56.296 INFO:tasks.rados.rados.0.smithi064.stdout:2004: done (0 left) 2024-03-04T06:15:56.296 INFO:tasks.rados.rados.0.smithi064.stdout:2005: read oid 27 snap -1 2024-03-04T06:15:56.296 INFO:tasks.rados.rados.0.smithi064.stdout:2005: expect deleted 2024-03-04T06:15:56.296 INFO:tasks.rados.rados.0.smithi064.stdout:2006: read oid 17 snap -1 2024-03-04T06:15:56.296 INFO:tasks.rados.rados.0.smithi064.stdout:2006: expect (ObjNum 771 snap 190 seq_num 771) 2024-03-04T06:15:56.296 INFO:tasks.rados.rados.0.smithi064.stdout:2007: snap_remove snap 234 2024-03-04T06:15:57.300 INFO:tasks.rados.rados.0.smithi064.stdout:2007: done (2 left) 2024-03-04T06:15:57.300 INFO:tasks.rados.rados.0.smithi064.stdout:2008: write oid 11 current snap is 235 2024-03-04T06:15:57.301 INFO:tasks.rados.rados.0.smithi064.stdout:2008: seq_num 946 ranges {699422=413326,1557848=681317,3012504=748383} 2024-03-04T06:15:57.306 INFO:tasks.rados.rados.0.smithi064.stdout:2008: writing smithi06431743-11 from 699422 to 1112748 tid 1 2024-03-04T06:15:57.312 INFO:tasks.rados.rados.0.smithi064.stdout:2008: writing smithi06431743-11 from 1557848 to 2239165 tid 2 2024-03-04T06:15:57.321 INFO:tasks.rados.rados.0.smithi064.stdout:2008: writing smithi06431743-11 from 3012504 to 3760887 tid 3 2024-03-04T06:15:57.321 INFO:tasks.rados.rados.0.smithi064.stdout:2009: snap_remove snap 233 2024-03-04T06:15:57.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:15:57.983 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:15:57.984 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:15:57.984 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:15:57.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:15:58.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:58.045+0000 7fb29399d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:58.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:58.050+0000 7fb29399d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:58.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:58.102+0000 7f4ca2224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:58.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:58.110+0000 7f4ca2224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:58.127 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:58.128 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:58.130 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:58.130 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:58.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:15:58.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:58.275+0000 7f4fb3425640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:58.282+0000 7f4fb3425640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:58.295 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:58.296 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:58.297 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:58.298 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:58.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:15:58.302 INFO:tasks.rados.rados.0.smithi064.stdout:2009: done (3 left) 2024-03-04T06:15:58.302 INFO:tasks.rados.rados.0.smithi064.stdout:2010: write oid 37 current snap is 235 2024-03-04T06:15:58.302 INFO:tasks.rados.rados.0.smithi064.stdout:2010: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2024-03-04T06:15:58.308 INFO:tasks.rados.rados.0.smithi064.stdout:2010: writing smithi06431743-37 from 716229 to 1204804 tid 1 2024-03-04T06:15:58.313 INFO:tasks.rados.rados.0.smithi064.stdout:2010: writing smithi06431743-37 from 1616330 to 2157658 tid 2 2024-03-04T06:15:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:2010: writing smithi06431743-37 from 2756280 to 3315935 tid 3 2024-03-04T06:15:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:2011: read oid 29 snap -1 2024-03-04T06:15:58.321 INFO:tasks.rados.rados.0.smithi064.stdout:2011: expect (ObjNum 850 snap 206 seq_num 850) 2024-03-04T06:15:58.321 INFO:tasks.rados.rados.0.smithi064.stdout:2012: write (excl) oid 3 current snap is 235 2024-03-04T06:15:58.321 INFO:tasks.rados.rados.0.smithi064.stdout:2006: expect (ObjNum 771 snap 190 seq_num 771) 2024-03-04T06:15:58.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:58.411+0000 7f82046dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:58.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:58.418+0000 7f82046dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:58.438 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:58.438 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:58.438 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:58.438 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:58.438 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:58.438 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:58.438 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:58.438 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:58.438 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:58.438 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:58.439 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:58.439 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:58.442 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:58.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:15:58.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:58.586+0000 7fefe126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:58.593 INFO:tasks.rados.rados.0.smithi064.stdout:2012: seq_num 948 ranges {733036=563824,2074812=684986,3183703=749333} 2024-03-04T06:15:58.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:15:58.593+0000 7fefe126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:15:58.601 INFO:tasks.rados.rados.0.smithi064.stdout:2012: writing smithi06431743-3 from 733036 to 1296860 tid 1 2024-03-04T06:15:58.609 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:15:58.609 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:15:58.610 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:15:58.610 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:15:58.610 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:15:58.610 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:15:58.610 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:15:58.610 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:15:58.610 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:15:58.610 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:15:58.610 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:15:58.610 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:15:58.611 INFO:tasks.rados.rados.0.smithi064.stdout:2012: writing smithi06431743-3 from 2074812 to 2759798 tid 2 2024-03-04T06:15:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:15:58.615 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:15:58.619 INFO:tasks.rados.rados.0.smithi064.stdout:2012: writing smithi06431743-3 from 3183703 to 3933036 tid 3 2024-03-04T06:15:58.619 INFO:tasks.rados.rados.0.smithi064.stdout:2005: done (5 left) 2024-03-04T06:15:58.619 INFO:tasks.rados.rados.0.smithi064.stdout:2006: done (4 left) 2024-03-04T06:15:58.620 INFO:tasks.rados.rados.0.smithi064.stdout:2013: read oid 47 snap -1 2024-03-04T06:15:58.620 INFO:tasks.rados.rados.0.smithi064.stdout:2013: expect (ObjNum 862 snap 207 seq_num 862) 2024-03-04T06:15:58.620 INFO:tasks.rados.rados.0.smithi064.stdout:2014: read oid 17 snap -1 2024-03-04T06:15:58.620 INFO:tasks.rados.rados.0.smithi064.stdout:2014: expect (ObjNum 771 snap 190 seq_num 771) 2024-03-04T06:15:58.620 INFO:tasks.rados.rados.0.smithi064.stdout:2015: delete oid 49 current snap is 235 2024-03-04T06:15:58.620 INFO:tasks.rados.rados.0.smithi064.stdout:2008: finishing write tid 1 to smithi06431743-11 2024-03-04T06:15:58.620 INFO:tasks.rados.rados.0.smithi064.stdout:2008: finishing write tid 2 to smithi06431743-11 2024-03-04T06:15:58.620 INFO:tasks.rados.rados.0.smithi064.stdout:2008: finishing write tid 3 to smithi06431743-11 2024-03-04T06:15:58.620 INFO:tasks.rados.rados.0.smithi064.stdout:2008: finishing write tid 4 to smithi06431743-11 2024-03-04T06:15:58.620 INFO:tasks.rados.rados.0.smithi064.stdout:2008: finishing write tid 5 to smithi06431743-11 2024-03-04T06:15:58.621 INFO:tasks.rados.rados.0.smithi064.stdout:2008: oid 11 updating version 0 to 440 2024-03-04T06:15:58.621 INFO:tasks.rados.rados.0.smithi064.stdout:2008: oid 11 updating version 440 to 442 2024-03-04T06:15:58.621 INFO:tasks.rados.rados.0.smithi064.stdout:2008: oid 11 updating version 442 to 443 2024-03-04T06:15:58.621 INFO:tasks.rados.rados.0.smithi064.stdout:2008: oid 11 version 443 is already newer than 441 2024-03-04T06:15:58.621 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 443 (ObjNum 946 snap 235 seq_num 946) dirty exists 2024-03-04T06:15:58.621 INFO:tasks.rados.rados.0.smithi064.stdout:2008: left oid 11 (ObjNum 946 snap 235 seq_num 946) 2024-03-04T06:15:58.621 INFO:tasks.rados.rados.0.smithi064.stdout:2010: finishing write tid 1 to smithi06431743-37 2024-03-04T06:15:58.621 INFO:tasks.rados.rados.0.smithi064.stdout:2011: expect (ObjNum 850 snap 206 seq_num 850) 2024-03-04T06:15:58.703 INFO:tasks.rados.rados.0.smithi064.stdout:2010: finishing write tid 2 to smithi06431743-37 2024-03-04T06:15:58.703 INFO:tasks.rados.rados.0.smithi064.stdout:2010: finishing write tid 3 to smithi06431743-37 2024-03-04T06:15:58.703 INFO:tasks.rados.rados.0.smithi064.stdout:2010: finishing write tid 4 to smithi06431743-37 2024-03-04T06:15:58.704 INFO:tasks.rados.rados.0.smithi064.stdout:2010: finishing write tid 5 to smithi06431743-37 2024-03-04T06:15:58.704 INFO:tasks.rados.rados.0.smithi064.stdout:2010: oid 37 updating version 0 to 310 2024-03-04T06:15:58.704 INFO:tasks.rados.rados.0.smithi064.stdout:2010: oid 37 updating version 310 to 313 2024-03-04T06:15:58.704 INFO:tasks.rados.rados.0.smithi064.stdout:2010: oid 37 version 313 is already newer than 313 2024-03-04T06:15:58.704 INFO:tasks.rados.rados.0.smithi064.stdout:2010: oid 37 version 313 is already newer than 311 2024-03-04T06:15:58.704 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 313 (ObjNum 947 snap 235 seq_num 947) dirty exists 2024-03-04T06:15:58.704 INFO:tasks.rados.rados.0.smithi064.stdout:2010: left oid 37 (ObjNum 947 snap 235 seq_num 947) 2024-03-04T06:15:58.704 INFO:tasks.rados.rados.0.smithi064.stdout:2008: done (6 left) 2024-03-04T06:15:58.704 INFO:tasks.rados.rados.0.smithi064.stdout:2010: done (5 left) 2024-03-04T06:15:58.705 INFO:tasks.rados.rados.0.smithi064.stdout:2011: done (4 left) 2024-03-04T06:15:58.705 INFO:tasks.rados.rados.0.smithi064.stdout:2015: done (3 left) 2024-03-04T06:15:58.705 INFO:tasks.rados.rados.0.smithi064.stdout:2016: snap_create 2024-03-04T06:15:58.705 INFO:tasks.rados.rados.0.smithi064.stdout:2012: finishing write tid 1 to smithi06431743-3 2024-03-04T06:15:58.705 INFO:tasks.rados.rados.0.smithi064.stdout:2013: expect (ObjNum 862 snap 207 seq_num 862) 2024-03-04T06:15:58.796 INFO:tasks.rados.rados.0.smithi064.stdout:2012: finishing write tid 2 to smithi06431743-3 2024-03-04T06:15:58.796 INFO:tasks.rados.rados.0.smithi064.stdout:2014: expect (ObjNum 771 snap 190 seq_num 771) 2024-03-04T06:15:59.050 INFO:tasks.rados.rados.0.smithi064.stdout:2012: finishing write tid 3 to smithi06431743-3 2024-03-04T06:15:59.050 INFO:tasks.rados.rados.0.smithi064.stdout:2012: finishing write tid 4 to smithi06431743-3 2024-03-04T06:15:59.050 INFO:tasks.rados.rados.0.smithi064.stdout:2012: finishing write tid 5 to smithi06431743-3 2024-03-04T06:15:59.050 INFO:tasks.rados.rados.0.smithi064.stdout:2012: oid 3 updating version 0 to 463 2024-03-04T06:15:59.050 INFO:tasks.rados.rados.0.smithi064.stdout:2012: oid 3 version 463 is already newer than 460 2024-03-04T06:15:59.050 INFO:tasks.rados.rados.0.smithi064.stdout:2012: oid 3 version 463 is already newer than 462 2024-03-04T06:15:59.050 INFO:tasks.rados.rados.0.smithi064.stdout:2012: oid 3 version 463 is already newer than 461 2024-03-04T06:15:59.050 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 463 (ObjNum 948 snap 235 seq_num 948) dirty exists 2024-03-04T06:15:59.051 INFO:tasks.rados.rados.0.smithi064.stdout:2012: left oid 3 (ObjNum 948 snap 235 seq_num 948) 2024-03-04T06:15:59.305 INFO:tasks.rados.rados.0.smithi064.stdout:2012: done (3 left) 2024-03-04T06:15:59.305 INFO:tasks.rados.rados.0.smithi064.stdout:2013: done (2 left) 2024-03-04T06:15:59.305 INFO:tasks.rados.rados.0.smithi064.stdout:2014: done (1 left) 2024-03-04T06:15:59.306 INFO:tasks.rados.rados.0.smithi064.stdout:2016: done (0 left) 2024-03-04T06:15:59.306 INFO:tasks.rados.rados.0.smithi064.stdout:2017: write (excl) oid 46 current snap is 236 2024-03-04T06:15:59.306 INFO:tasks.rados.rados.0.smithi064.stdout:2017: seq_num 950 ranges {766650=714322,2075423=688655,3522196=444454} 2024-03-04T06:15:59.313 INFO:tasks.rados.rados.0.smithi064.stdout:2017: writing smithi06431743-46 from 766650 to 1480972 tid 1 2024-03-04T06:15:59.322 INFO:tasks.rados.rados.0.smithi064.stdout:2017: writing smithi06431743-46 from 2075423 to 2764078 tid 2 2024-03-04T06:15:59.330 INFO:tasks.rados.rados.0.smithi064.stdout:2017: writing smithi06431743-46 from 3522196 to 3966650 tid 3 2024-03-04T06:15:59.330 INFO:tasks.rados.rados.0.smithi064.stdout:2018: snap_create 2024-03-04T06:15:59.368 INFO:tasks.rados.rados.0.smithi064.stdout:2017: finishing write tid 1 to smithi06431743-46 2024-03-04T06:15:59.379 INFO:tasks.rados.rados.0.smithi064.stdout:2017: finishing write tid 2 to smithi06431743-46 2024-03-04T06:15:59.392 INFO:tasks.rados.rados.0.smithi064.stdout:2017: finishing write tid 3 to smithi06431743-46 2024-03-04T06:15:59.394 INFO:tasks.rados.rados.0.smithi064.stdout:2017: finishing write tid 4 to smithi06431743-46 2024-03-04T06:15:59.394 INFO:tasks.rados.rados.0.smithi064.stdout:2017: finishing write tid 5 to smithi06431743-46 2024-03-04T06:15:59.395 INFO:tasks.rados.rados.0.smithi064.stdout:2017: oid 46 updating version 0 to 795 2024-03-04T06:15:59.395 INFO:tasks.rados.rados.0.smithi064.stdout:2017: oid 46 version 795 is already newer than 795 2024-03-04T06:15:59.395 INFO:tasks.rados.rados.0.smithi064.stdout:2017: oid 46 version 795 is already newer than 792 2024-03-04T06:15:59.395 INFO:tasks.rados.rados.0.smithi064.stdout:2017: oid 46 version 795 is already newer than 793 2024-03-04T06:15:59.395 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 795 (ObjNum 950 snap 236 seq_num 950) dirty exists 2024-03-04T06:15:59.395 INFO:tasks.rados.rados.0.smithi064.stdout:2017: left oid 46 (ObjNum 950 snap 236 seq_num 950) 2024-03-04T06:16:00.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:00.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:00.239+0000 7fe7cda25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:00.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:00.243+0000 7fe7cda25640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:00.308 INFO:tasks.rados.rados.0.smithi064.stdout:2017: done (1 left) 2024-03-04T06:16:00.308 INFO:tasks.rados.rados.0.smithi064.stdout:2018: done (0 left) 2024-03-04T06:16:00.308 INFO:tasks.rados.rados.0.smithi064.stdout:2019: write (excl) oid 5 current snap is 237 2024-03-04T06:16:00.308 INFO:tasks.rados.rados.0.smithi064.stdout:2019: seq_num 951 ranges {783457=505924,1850258=548666,2865972=721096} 2024-03-04T06:16:00.315 INFO:tasks.rados.rados.0.smithi064.stdout:2019: writing smithi06431743-5 from 783457 to 1289381 tid 1 2024-03-04T06:16:00.321 INFO:tasks.rados.rados.0.smithi064.stdout:2019: writing smithi06431743-5 from 1850258 to 2398924 tid 2 2024-03-04T06:16:00.329 INFO:tasks.rados.rados.0.smithi064.stdout:2019: writing smithi06431743-5 from 2865972 to 3587068 tid 3 2024-03-04T06:16:00.329 INFO:tasks.rados.rados.0.smithi064.stdout:2020: write (excl) oid 18 current snap is 237 2024-03-04T06:16:00.329 INFO:tasks.rados.rados.0.smithi064.stdout:2020: seq_num 952 ranges {400264=581173,1508740=491524} 2024-03-04T06:16:00.334 INFO:tasks.rados.rados.0.smithi064.stdout:2020: writing smithi06431743-18 from 400264 to 981437 tid 1 2024-03-04T06:16:00.340 INFO:tasks.rados.rados.0.smithi064.stdout:2020: writing smithi06431743-18 from 1508740 to 2000264 tid 2 2024-03-04T06:16:00.340 INFO:tasks.rados.rados.0.smithi064.stdout:2021: write oid 33 current snap is 237 2024-03-04T06:16:00.340 INFO:tasks.rados.rados.0.smithi064.stdout:2021: seq_num 953 ranges {417071=656422,1567222=449849} 2024-03-04T06:16:00.345 INFO:tasks.rados.rados.0.smithi064.stdout:2021: writing smithi06431743-33 from 417071 to 1073493 tid 1 2024-03-04T06:16:00.351 INFO:tasks.rados.rados.0.smithi064.stdout:2021: writing smithi06431743-33 from 1567222 to 2017071 tid 2 2024-03-04T06:16:00.351 INFO:tasks.rados.rados.0.smithi064.stdout:2022: read oid 3 snap 235 2024-03-04T06:16:00.352 INFO:tasks.rados.rados.0.smithi064.stdout:2022: expect (ObjNum 948 snap 235 seq_num 948) 2024-03-04T06:16:00.352 INFO:tasks.rados.rados.0.smithi064.stdout:2023: read oid 42 snap -1 2024-03-04T06:16:00.352 INFO:tasks.rados.rados.0.smithi064.stdout:2023: expect (ObjNum 903 snap 222 seq_num 903) 2024-03-04T06:16:00.352 INFO:tasks.rados.rados.0.smithi064.stdout:2024: snap_remove snap 236 2024-03-04T06:16:01.313 INFO:tasks.rados.rados.0.smithi064.stdout:2024: done (5 left) 2024-03-04T06:16:01.314 INFO:tasks.rados.rados.0.smithi064.stdout:2025: read oid 14 snap -1 2024-03-04T06:16:01.314 INFO:tasks.rados.rados.0.smithi064.stdout:2025: expect (ObjNum 918 snap 224 seq_num 918) 2024-03-04T06:16:01.314 INFO:tasks.rados.rados.0.smithi064.stdout:2026: read oid 8 snap -1 2024-03-04T06:16:01.314 INFO:tasks.rados.rados.0.smithi064.stdout:2026: expect deleted 2024-03-04T06:16:01.314 INFO:tasks.rados.rados.0.smithi064.stdout:2027: snap_create 2024-03-04T06:16:01.314 INFO:tasks.rados.rados.0.smithi064.stdout:2023: expect (ObjNum 903 snap 222 seq_num 903) 2024-03-04T06:16:01.532 INFO:tasks.rados.rados.0.smithi064.stdout:2019: finishing write tid 1 to smithi06431743-5 2024-03-04T06:16:01.532 INFO:tasks.rados.rados.0.smithi064.stdout:2022: expect (ObjNum 948 snap 235 seq_num 948) 2024-03-04T06:16:01.718 INFO:tasks.rados.rados.0.smithi064.stdout:2020: finishing write tid 1 to smithi06431743-18 2024-03-04T06:16:01.718 INFO:tasks.rados.rados.0.smithi064.stdout:2021: finishing write tid 1 to smithi06431743-33 2024-03-04T06:16:01.719 INFO:tasks.rados.rados.0.smithi064.stdout:2019: finishing write tid 2 to smithi06431743-5 2024-03-04T06:16:01.719 INFO:tasks.rados.rados.0.smithi064.stdout:2020: finishing write tid 2 to smithi06431743-18 2024-03-04T06:16:01.719 INFO:tasks.rados.rados.0.smithi064.stdout:2019: finishing write tid 3 to smithi06431743-5 2024-03-04T06:16:01.719 INFO:tasks.rados.rados.0.smithi064.stdout:2021: finishing write tid 2 to smithi06431743-33 2024-03-04T06:16:01.719 INFO:tasks.rados.rados.0.smithi064.stdout:2020: finishing write tid 3 to smithi06431743-18 2024-03-04T06:16:01.719 INFO:tasks.rados.rados.0.smithi064.stdout:2020: finishing write tid 4 to smithi06431743-18 2024-03-04T06:16:01.719 INFO:tasks.rados.rados.0.smithi064.stdout:2020: oid 18 updating version 0 to 316 2024-03-04T06:16:01.719 INFO:tasks.rados.rados.0.smithi064.stdout:2020: oid 18 version 316 is already newer than 314 2024-03-04T06:16:01.719 INFO:tasks.rados.rados.0.smithi064.stdout:2020: oid 18 version 316 is already newer than 315 2024-03-04T06:16:01.720 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 316 (ObjNum 952 snap 237 seq_num 952) dirty exists 2024-03-04T06:16:01.720 INFO:tasks.rados.rados.0.smithi064.stdout:2020: left oid 18 (ObjNum 952 snap 237 seq_num 952) 2024-03-04T06:16:01.720 INFO:tasks.rados.rados.0.smithi064.stdout:2019: finishing write tid 4 to smithi06431743-5 2024-03-04T06:16:01.720 INFO:tasks.rados.rados.0.smithi064.stdout:2019: finishing write tid 5 to smithi06431743-5 2024-03-04T06:16:01.720 INFO:tasks.rados.rados.0.smithi064.stdout:2019: oid 5 updating version 0 to 862 2024-03-04T06:16:01.720 INFO:tasks.rados.rados.0.smithi064.stdout:2019: oid 5 version 862 is already newer than 859 2024-03-04T06:16:01.720 INFO:tasks.rados.rados.0.smithi064.stdout:2019: oid 5 version 862 is already newer than 861 2024-03-04T06:16:01.720 INFO:tasks.rados.rados.0.smithi064.stdout:2019: oid 5 version 862 is already newer than 861 2024-03-04T06:16:01.720 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 862 (ObjNum 951 snap 237 seq_num 951) dirty exists 2024-03-04T06:16:01.720 INFO:tasks.rados.rados.0.smithi064.stdout:2019: left oid 5 (ObjNum 951 snap 237 seq_num 951) 2024-03-04T06:16:01.720 INFO:tasks.rados.rados.0.smithi064.stdout:2021: finishing write tid 3 to smithi06431743-33 2024-03-04T06:16:01.721 INFO:tasks.rados.rados.0.smithi064.stdout:2021: finishing write tid 4 to smithi06431743-33 2024-03-04T06:16:01.721 INFO:tasks.rados.rados.0.smithi064.stdout:2021: oid 33 updating version 0 to 674 2024-03-04T06:16:01.721 INFO:tasks.rados.rados.0.smithi064.stdout:2021: oid 33 updating version 674 to 675 2024-03-04T06:16:01.721 INFO:tasks.rados.rados.0.smithi064.stdout:2021: oid 33 version 675 is already newer than 673 2024-03-04T06:16:01.721 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 675 (ObjNum 953 snap 237 seq_num 953) dirty exists 2024-03-04T06:16:01.721 INFO:tasks.rados.rados.0.smithi064.stdout:2021: left oid 33 (ObjNum 953 snap 237 seq_num 953) 2024-03-04T06:16:01.721 INFO:tasks.rados.rados.0.smithi064.stdout:2025: expect (ObjNum 918 snap 224 seq_num 918) 2024-03-04T06:16:02.314 INFO:tasks.rados.rados.0.smithi064.stdout:2019: done (7 left) 2024-03-04T06:16:02.314 INFO:tasks.rados.rados.0.smithi064.stdout:2020: done (6 left) 2024-03-04T06:16:02.314 INFO:tasks.rados.rados.0.smithi064.stdout:2021: done (5 left) 2024-03-04T06:16:02.314 INFO:tasks.rados.rados.0.smithi064.stdout:2022: done (4 left) 2024-03-04T06:16:02.314 INFO:tasks.rados.rados.0.smithi064.stdout:2023: done (3 left) 2024-03-04T06:16:02.314 INFO:tasks.rados.rados.0.smithi064.stdout:2025: done (2 left) 2024-03-04T06:16:02.314 INFO:tasks.rados.rados.0.smithi064.stdout:2026: done (1 left) 2024-03-04T06:16:02.315 INFO:tasks.rados.rados.0.smithi064.stdout:2027: done (0 left) 2024-03-04T06:16:02.315 INFO:tasks.rados.rados.0.smithi064.stdout:2028: write (excl) oid 5 current snap is 238 2024-03-04T06:16:02.315 INFO:tasks.rados.rados.0.smithi064.stdout:2028: seq_num 954 ranges {433878=731671,1909351=124527} 2024-03-04T06:16:02.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:02.320 INFO:tasks.rados.rados.0.smithi064.stdout:2028: writing smithi06431743-5 from 433878 to 1165549 tid 1 2024-03-04T06:16:02.324 INFO:tasks.rados.rados.0.smithi064.stdout:2028: writing smithi06431743-5 from 1909351 to 2033878 tid 2 2024-03-04T06:16:02.325 INFO:tasks.rados.rados.0.smithi064.stdout:2029: snap_create 2024-03-04T06:16:02.384 INFO:tasks.rados.rados.0.smithi064.stdout:2028: finishing write tid 1 to smithi06431743-5 2024-03-04T06:16:02.399 INFO:tasks.rados.rados.0.smithi064.stdout:2028: finishing write tid 2 to smithi06431743-5 2024-03-04T06:16:02.409 INFO:tasks.rados.rados.0.smithi064.stdout:2028: finishing write tid 3 to smithi06431743-5 2024-03-04T06:16:02.410 INFO:tasks.rados.rados.0.smithi064.stdout:2028: finishing write tid 4 to smithi06431743-5 2024-03-04T06:16:02.410 INFO:tasks.rados.rados.0.smithi064.stdout:2028: oid 5 updating version 0 to 865 2024-03-04T06:16:02.410 INFO:tasks.rados.rados.0.smithi064.stdout:2028: oid 5 updating version 865 to 866 2024-03-04T06:16:02.410 INFO:tasks.rados.rados.0.smithi064.stdout:2028: oid 5 version 866 is already newer than 864 2024-03-04T06:16:02.410 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 866 (ObjNum 954 snap 238 seq_num 954) dirty exists 2024-03-04T06:16:02.410 INFO:tasks.rados.rados.0.smithi064.stdout:2028: left oid 5 (ObjNum 954 snap 238 seq_num 954) 2024-03-04T06:16:02.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:02.435+0000 7f3dfa307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:02.440 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:02.439+0000 7f3dfa307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:03.256 INFO:tasks.rados.rados.0.smithi064.stdout:2028: done (1 left) 2024-03-04T06:16:03.256 INFO:tasks.rados.rados.0.smithi064.stdout:2029: done (0 left) 2024-03-04T06:16:03.256 INFO:tasks.rados.rados.0.smithi064.stdout:2030: snap_create 2024-03-04T06:16:03.616 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:16:03.617 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:16:03.617 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:16:03.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:16:03.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:03.735+0000 7f8f3aa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:03.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:03.742+0000 7f8f3aa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:03.756 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:03.756 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:03.756 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:03.756 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:03.757 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:03.757 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:03.757 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:03.757 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:03.757 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:03.757 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:03.757 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:03.757 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:03.758 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:03.759 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:03.759 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:16:03.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:03.864+0000 7fbca9217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:03.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:03.871+0000 7fbca9217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:03.884 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:03.888 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:03.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:16:03.997 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:03.992+0000 7f35cd00f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:04.000 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:03.999+0000 7f35cd00f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:04.013 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:04.013 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:04.013 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:04.013 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:04.014 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:04.014 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:04.014 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:04.014 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:04.014 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:04.014 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:04.014 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:04.014 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:04.016 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:04.017 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:16:04.125 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:04.120+0000 7f84e955d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:04.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:04.127+0000 7f84e955d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:04.141 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:04.141 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:04.141 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:04.141 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:04.142 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:04.142 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:04.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:04.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:04.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:04.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:04.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:04.142 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:04.144 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:04.256 INFO:tasks.rados.rados.0.smithi064.stdout:2030: done (0 left) 2024-03-04T06:16:04.256 INFO:tasks.rados.rados.0.smithi064.stdout:2031: write oid 14 current snap is 240 2024-03-04T06:16:04.256 INFO:tasks.rados.rados.0.smithi064.stdout:2031: seq_num 955 ranges {450685=406920,1567833=482852} 2024-03-04T06:16:04.262 INFO:tasks.rados.rados.0.smithi064.stdout:2031: writing smithi06431743-14 from 450685 to 857605 tid 1 2024-03-04T06:16:04.270 INFO:tasks.rados.rados.0.smithi064.stdout:2031: writing smithi06431743-14 from 1567833 to 2050685 tid 2 2024-03-04T06:16:04.270 INFO:tasks.rados.rados.0.smithi064.stdout:2032: snap_remove snap 237 2024-03-04T06:16:04.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:04.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:04.610+0000 7f8d07864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:04.615 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:04.615+0000 7f8d07864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:05.257 INFO:tasks.rados.rados.0.smithi064.stdout:2032: done (1 left) 2024-03-04T06:16:05.258 INFO:tasks.rados.rados.0.smithi064.stdout:2033: write (excl) oid 11 current snap is 240 2024-03-04T06:16:05.258 INFO:tasks.rados.rados.0.smithi064.stdout:2033: seq_num 956 ranges {467492=482169,1626315=441177} 2024-03-04T06:16:05.262 INFO:tasks.rados.rados.0.smithi064.stdout:2033: writing smithi06431743-11 from 467492 to 949661 tid 1 2024-03-04T06:16:05.270 INFO:tasks.rados.rados.0.smithi064.stdout:2033: writing smithi06431743-11 from 1626315 to 2067492 tid 2 2024-03-04T06:16:05.270 INFO:tasks.rados.rados.0.smithi064.stdout:2034: write oid 44 current snap is 240 2024-03-04T06:16:05.270 INFO:tasks.rados.rados.0.smithi064.stdout:2034: seq_num 957 ranges {484299=557418,1684797=399502} 2024-03-04T06:16:05.277 INFO:tasks.rados.rados.0.smithi064.stdout:2034: writing smithi06431743-44 from 484299 to 1041717 tid 1 2024-03-04T06:16:05.284 INFO:tasks.rados.rados.0.smithi064.stdout:2034: writing smithi06431743-44 from 1684797 to 2084299 tid 2 2024-03-04T06:16:05.284 INFO:tasks.rados.rados.0.smithi064.stdout:2035: write (excl) oid 16 current snap is 240 2024-03-04T06:16:05.285 INFO:tasks.rados.rados.0.smithi064.stdout:2031: finishing write tid 1 to smithi06431743-14 2024-03-04T06:16:05.285 INFO:tasks.rados.rados.0.smithi064.stdout:2035: seq_num 958 ranges {501106=749020,1743279=357827} 2024-03-04T06:16:05.293 INFO:tasks.rados.rados.0.smithi064.stdout:2035: writing smithi06431743-16 from 501106 to 1250126 tid 1 2024-03-04T06:16:05.299 INFO:tasks.rados.rados.0.smithi064.stdout:2035: writing smithi06431743-16 from 1743279 to 2101106 tid 2 2024-03-04T06:16:05.299 INFO:tasks.rados.rados.0.smithi064.stdout:2036: write oid 5 current snap is 240 2024-03-04T06:16:05.299 INFO:tasks.rados.rados.0.smithi064.stdout:2031: finishing write tid 2 to smithi06431743-14 2024-03-04T06:16:05.299 INFO:tasks.rados.rados.0.smithi064.stdout:2031: finishing write tid 3 to smithi06431743-14 2024-03-04T06:16:05.299 INFO:tasks.rados.rados.0.smithi064.stdout:2031: finishing write tid 4 to smithi06431743-14 2024-03-04T06:16:05.299 INFO:tasks.rados.rados.0.smithi064.stdout:2031: oid 14 updating version 0 to 557 2024-03-04T06:16:05.299 INFO:tasks.rados.rados.0.smithi064.stdout:2031: oid 14 updating version 557 to 558 2024-03-04T06:16:05.299 INFO:tasks.rados.rados.0.smithi064.stdout:2031: oid 14 updating version 558 to 559 2024-03-04T06:16:05.299 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 559 (ObjNum 955 snap 240 seq_num 955) dirty exists 2024-03-04T06:16:05.300 INFO:tasks.rados.rados.0.smithi064.stdout:2031: left oid 14 (ObjNum 955 snap 240 seq_num 955) 2024-03-04T06:16:05.300 INFO:tasks.rados.rados.0.smithi064.stdout:2036: seq_num 959 ranges {517913=424269,1401761=446989} 2024-03-04T06:16:05.305 INFO:tasks.rados.rados.0.smithi064.stdout:2036: writing smithi06431743-5 from 517913 to 942182 tid 1 2024-03-04T06:16:05.312 INFO:tasks.rados.rados.0.smithi064.stdout:2036: writing smithi06431743-5 from 1401761 to 1848750 tid 2 2024-03-04T06:16:05.312 INFO:tasks.rados.rados.0.smithi064.stdout:2031: done (4 left) 2024-03-04T06:16:05.312 INFO:tasks.rados.rados.0.smithi064.stdout:2037: snap_create 2024-03-04T06:16:05.346 INFO:tasks.rados.rados.0.smithi064.stdout:2033: finishing write tid 1 to smithi06431743-11 2024-03-04T06:16:05.359 INFO:tasks.rados.rados.0.smithi064.stdout:2034: finishing write tid 1 to smithi06431743-44 2024-03-04T06:16:05.379 INFO:tasks.rados.rados.0.smithi064.stdout:2035: finishing write tid 1 to smithi06431743-16 2024-03-04T06:16:05.382 INFO:tasks.rados.rados.0.smithi064.stdout:2034: finishing write tid 2 to smithi06431743-44 2024-03-04T06:16:05.386 INFO:tasks.rados.rados.0.smithi064.stdout:2033: finishing write tid 2 to smithi06431743-11 2024-03-04T06:16:05.388 INFO:tasks.rados.rados.0.smithi064.stdout:2036: finishing write tid 1 to smithi06431743-5 2024-03-04T06:16:05.395 INFO:tasks.rados.rados.0.smithi064.stdout:2035: finishing write tid 2 to smithi06431743-16 2024-03-04T06:16:05.403 INFO:tasks.rados.rados.0.smithi064.stdout:2034: finishing write tid 3 to smithi06431743-44 2024-03-04T06:16:05.404 INFO:tasks.rados.rados.0.smithi064.stdout:2034: finishing write tid 4 to smithi06431743-44 2024-03-04T06:16:05.404 INFO:tasks.rados.rados.0.smithi064.stdout:2034: oid 44 updating version 0 to 457 2024-03-04T06:16:05.404 INFO:tasks.rados.rados.0.smithi064.stdout:2034: oid 44 updating version 457 to 459 2024-03-04T06:16:05.404 INFO:tasks.rados.rados.0.smithi064.stdout:2034: oid 44 version 459 is already newer than 458 2024-03-04T06:16:05.404 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 459 (ObjNum 957 snap 240 seq_num 957) dirty exists 2024-03-04T06:16:05.404 INFO:tasks.rados.rados.0.smithi064.stdout:2034: left oid 44 (ObjNum 957 snap 240 seq_num 957) 2024-03-04T06:16:05.405 INFO:tasks.rados.rados.0.smithi064.stdout:2036: finishing write tid 2 to smithi06431743-5 2024-03-04T06:16:05.405 INFO:tasks.rados.rados.0.smithi064.stdout:2033: finishing write tid 3 to smithi06431743-11 2024-03-04T06:16:05.406 INFO:tasks.rados.rados.0.smithi064.stdout:2033: finishing write tid 4 to smithi06431743-11 2024-03-04T06:16:05.406 INFO:tasks.rados.rados.0.smithi064.stdout:2033: oid 11 updating version 0 to 449 2024-03-04T06:16:05.406 INFO:tasks.rados.rados.0.smithi064.stdout:2033: oid 11 version 449 is already newer than 448 2024-03-04T06:16:05.406 INFO:tasks.rados.rados.0.smithi064.stdout:2033: oid 11 version 449 is already newer than 447 2024-03-04T06:16:05.406 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 449 (ObjNum 956 snap 240 seq_num 956) dirty exists 2024-03-04T06:16:05.406 INFO:tasks.rados.rados.0.smithi064.stdout:2033: left oid 11 (ObjNum 956 snap 240 seq_num 956) 2024-03-04T06:16:05.406 INFO:tasks.rados.rados.0.smithi064.stdout:2035: finishing write tid 3 to smithi06431743-16 2024-03-04T06:16:05.408 INFO:tasks.rados.rados.0.smithi064.stdout:2035: finishing write tid 4 to smithi06431743-16 2024-03-04T06:16:05.408 INFO:tasks.rados.rados.0.smithi064.stdout:2035: oid 16 updating version 0 to 383 2024-03-04T06:16:05.408 INFO:tasks.rados.rados.0.smithi064.stdout:2035: oid 16 version 383 is already newer than 382 2024-03-04T06:16:05.408 INFO:tasks.rados.rados.0.smithi064.stdout:2035: oid 16 version 383 is already newer than 381 2024-03-04T06:16:05.408 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 383 (ObjNum 958 snap 240 seq_num 958) dirty exists 2024-03-04T06:16:05.408 INFO:tasks.rados.rados.0.smithi064.stdout:2035: left oid 16 (ObjNum 958 snap 240 seq_num 958) 2024-03-04T06:16:05.411 INFO:tasks.rados.rados.0.smithi064.stdout:2036: finishing write tid 3 to smithi06431743-5 2024-03-04T06:16:05.412 INFO:tasks.rados.rados.0.smithi064.stdout:2036: finishing write tid 4 to smithi06431743-5 2024-03-04T06:16:05.412 INFO:tasks.rados.rados.0.smithi064.stdout:2036: oid 5 updating version 0 to 871 2024-03-04T06:16:05.412 INFO:tasks.rados.rados.0.smithi064.stdout:2036: oid 5 updating version 871 to 872 2024-03-04T06:16:05.412 INFO:tasks.rados.rados.0.smithi064.stdout:2036: oid 5 version 872 is already newer than 870 2024-03-04T06:16:05.413 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 872 (ObjNum 959 snap 240 seq_num 959) dirty exists 2024-03-04T06:16:05.413 INFO:tasks.rados.rados.0.smithi064.stdout:2036: left oid 5 (ObjNum 959 snap 240 seq_num 959) 2024-03-04T06:16:06.261 INFO:tasks.rados.rados.0.smithi064.stdout:2033: done (4 left) 2024-03-04T06:16:06.261 INFO:tasks.rados.rados.0.smithi064.stdout:2034: done (3 left) 2024-03-04T06:16:06.261 INFO:tasks.rados.rados.0.smithi064.stdout:2035: done (2 left) 2024-03-04T06:16:06.261 INFO:tasks.rados.rados.0.smithi064.stdout:2036: done (1 left) 2024-03-04T06:16:06.261 INFO:tasks.rados.rados.0.smithi064.stdout:2037: done (0 left) 2024-03-04T06:16:06.261 INFO:tasks.rados.rados.0.smithi064.stdout:2038: write (excl) oid 5 current snap is 241 2024-03-04T06:16:06.261 INFO:tasks.rados.rados.0.smithi064.stdout:2038: seq_num 960 ranges {534720=499518,1460243=674477} 2024-03-04T06:16:06.266 INFO:tasks.rados.rados.0.smithi064.stdout:2038: writing smithi06431743-5 from 534720 to 1034238 tid 1 2024-03-04T06:16:06.272 INFO:tasks.rados.rados.0.smithi064.stdout:2038: writing smithi06431743-5 from 1460243 to 2134720 tid 2 2024-03-04T06:16:06.272 INFO:tasks.rados.rados.0.smithi064.stdout:2039: delete oid 48 current snap is 241 2024-03-04T06:16:06.308 INFO:tasks.rados.rados.0.smithi064.stdout:2039: done (1 left) 2024-03-04T06:16:06.308 INFO:tasks.rados.rados.0.smithi064.stdout:2040: read oid 15 snap -1 2024-03-04T06:16:06.309 INFO:tasks.rados.rados.0.smithi064.stdout:2040: expect deleted 2024-03-04T06:16:06.309 INFO:tasks.rados.rados.0.smithi064.stdout:2041: delete oid 7 current snap is 241 2024-03-04T06:16:06.336 INFO:tasks.rados.rados.0.smithi064.stdout:2038: finishing write tid 1 to smithi06431743-5 2024-03-04T06:16:06.353 INFO:tasks.rados.rados.0.smithi064.stdout:2040: done (2 left) 2024-03-04T06:16:06.353 INFO:tasks.rados.rados.0.smithi064.stdout:2041: done (1 left) 2024-03-04T06:16:06.353 INFO:tasks.rados.rados.0.smithi064.stdout:2042: write (excl) oid 9 current snap is 241 2024-03-04T06:16:06.353 INFO:tasks.rados.rados.0.smithi064.stdout:2042: seq_num 963 ranges {585141=725265,1919336=265805} 2024-03-04T06:16:06.360 INFO:tasks.rados.rados.0.smithi064.stdout:2042: writing smithi06431743-9 from 585141 to 1310406 tid 1 2024-03-04T06:16:06.366 INFO:tasks.rados.rados.0.smithi064.stdout:2042: writing smithi06431743-9 from 1919336 to 2185141 tid 2 2024-03-04T06:16:06.366 INFO:tasks.rados.rados.0.smithi064.stdout:2043: read oid 22 snap -1 2024-03-04T06:16:06.366 INFO:tasks.rados.rados.0.smithi064.stdout:2043: expect (ObjNum 852 snap 206 seq_num 852) 2024-03-04T06:16:06.366 INFO:tasks.rados.rados.0.smithi064.stdout:2044: write oid 3 current snap is 241 2024-03-04T06:16:06.366 INFO:tasks.rados.rados.0.smithi064.stdout:2038: finishing write tid 2 to smithi06431743-5 2024-03-04T06:16:06.366 INFO:tasks.rados.rados.0.smithi064.stdout:2044: seq_num 964 ranges {601948=400514,1577818=624130} 2024-03-04T06:16:06.371 INFO:tasks.rados.rados.0.smithi064.stdout:2044: writing smithi06431743-3 from 601948 to 1002462 tid 1 2024-03-04T06:16:06.380 INFO:tasks.rados.rados.0.smithi064.stdout:2044: writing smithi06431743-3 from 1577818 to 2201948 tid 2 2024-03-04T06:16:06.380 INFO:tasks.rados.rados.0.smithi064.stdout:2045: read oid 11 snap -1 2024-03-04T06:16:06.380 INFO:tasks.rados.rados.0.smithi064.stdout:2045: expect (ObjNum 956 snap 240 seq_num 956) 2024-03-04T06:16:06.380 INFO:tasks.rados.rados.0.smithi064.stdout:2046: snap_remove snap 235 2024-03-04T06:16:06.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:06.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:06.796+0000 7fe66139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:06.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:06.801+0000 7fe66139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:07.264 INFO:tasks.rados.rados.0.smithi064.stdout:2046: done (5 left) 2024-03-04T06:16:07.264 INFO:tasks.rados.rados.0.smithi064.stdout:2047: write oid 33 current snap is 241 2024-03-04T06:16:07.264 INFO:tasks.rados.rados.0.smithi064.stdout:2047: seq_num 965 ranges {618755=475763,1636300=457996} 2024-03-04T06:16:07.269 INFO:tasks.rados.rados.0.smithi064.stdout:2047: writing smithi06431743-33 from 618755 to 1094518 tid 1 2024-03-04T06:16:07.276 INFO:tasks.rados.rados.0.smithi064.stdout:2047: writing smithi06431743-33 from 1636300 to 2094296 tid 2 2024-03-04T06:16:07.276 INFO:tasks.rados.rados.0.smithi064.stdout:2048: read oid 49 snap 239 2024-03-04T06:16:07.276 INFO:tasks.rados.rados.0.smithi064.stdout:2048: expect deleted 2024-03-04T06:16:07.276 INFO:tasks.rados.rados.0.smithi064.stdout:2049: snap_create 2024-03-04T06:16:07.276 INFO:tasks.rados.rados.0.smithi064.stdout:2038: finishing write tid 3 to smithi06431743-5 2024-03-04T06:16:07.276 INFO:tasks.rados.rados.0.smithi064.stdout:2038: finishing write tid 4 to smithi06431743-5 2024-03-04T06:16:07.276 INFO:tasks.rados.rados.0.smithi064.stdout:2038: oid 5 updating version 0 to 876 2024-03-04T06:16:07.277 INFO:tasks.rados.rados.0.smithi064.stdout:2038: oid 5 version 876 is already newer than 874 2024-03-04T06:16:07.277 INFO:tasks.rados.rados.0.smithi064.stdout:2038: oid 5 version 876 is already newer than 876 2024-03-04T06:16:07.277 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 876 (ObjNum 960 snap 241 seq_num 960) dirty exists 2024-03-04T06:16:07.277 INFO:tasks.rados.rados.0.smithi064.stdout:2038: left oid 5 (ObjNum 960 snap 241 seq_num 960) 2024-03-04T06:16:07.277 INFO:tasks.rados.rados.0.smithi064.stdout:2042: finishing write tid 1 to smithi06431743-9 2024-03-04T06:16:07.277 INFO:tasks.rados.rados.0.smithi064.stdout:2044: finishing write tid 1 to smithi06431743-3 2024-03-04T06:16:07.277 INFO:tasks.rados.rados.0.smithi064.stdout:2044: finishing write tid 2 to smithi06431743-3 2024-03-04T06:16:07.277 INFO:tasks.rados.rados.0.smithi064.stdout:2042: finishing write tid 2 to smithi06431743-9 2024-03-04T06:16:07.277 INFO:tasks.rados.rados.0.smithi064.stdout:2042: finishing write tid 3 to smithi06431743-9 2024-03-04T06:16:07.277 INFO:tasks.rados.rados.0.smithi064.stdout:2045: expect (ObjNum 956 snap 240 seq_num 956) 2024-03-04T06:16:07.413 INFO:tasks.rados.rados.0.smithi064.stdout:2042: finishing write tid 4 to smithi06431743-9 2024-03-04T06:16:07.413 INFO:tasks.rados.rados.0.smithi064.stdout:2042: oid 9 updating version 0 to 678 2024-03-04T06:16:07.414 INFO:tasks.rados.rados.0.smithi064.stdout:2042: oid 9 version 678 is already newer than 676 2024-03-04T06:16:07.414 INFO:tasks.rados.rados.0.smithi064.stdout:2042: oid 9 version 678 is already newer than 678 2024-03-04T06:16:07.414 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 678 (ObjNum 963 snap 241 seq_num 963) dirty exists 2024-03-04T06:16:07.414 INFO:tasks.rados.rados.0.smithi064.stdout:2042: left oid 9 (ObjNum 963 snap 241 seq_num 963) 2024-03-04T06:16:07.414 INFO:tasks.rados.rados.0.smithi064.stdout:2044: finishing write tid 3 to smithi06431743-3 2024-03-04T06:16:07.414 INFO:tasks.rados.rados.0.smithi064.stdout:2044: finishing write tid 4 to smithi06431743-3 2024-03-04T06:16:07.414 INFO:tasks.rados.rados.0.smithi064.stdout:2044: oid 3 updating version 0 to 467 2024-03-04T06:16:07.414 INFO:tasks.rados.rados.0.smithi064.stdout:2044: oid 3 version 467 is already newer than 465 2024-03-04T06:16:07.414 INFO:tasks.rados.rados.0.smithi064.stdout:2044: oid 3 version 467 is already newer than 467 2024-03-04T06:16:07.414 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 467 (ObjNum 964 snap 241 seq_num 964) dirty exists 2024-03-04T06:16:07.414 INFO:tasks.rados.rados.0.smithi064.stdout:2044: left oid 3 (ObjNum 964 snap 241 seq_num 964) 2024-03-04T06:16:07.415 INFO:tasks.rados.rados.0.smithi064.stdout:2043: expect (ObjNum 852 snap 206 seq_num 852) 2024-03-04T06:16:07.500 INFO:tasks.rados.rados.0.smithi064.stdout:2047: finishing write tid 1 to smithi06431743-33 2024-03-04T06:16:07.500 INFO:tasks.rados.rados.0.smithi064.stdout:2047: finishing write tid 2 to smithi06431743-33 2024-03-04T06:16:07.500 INFO:tasks.rados.rados.0.smithi064.stdout:2047: finishing write tid 3 to smithi06431743-33 2024-03-04T06:16:07.501 INFO:tasks.rados.rados.0.smithi064.stdout:2047: finishing write tid 4 to smithi06431743-33 2024-03-04T06:16:07.501 INFO:tasks.rados.rados.0.smithi064.stdout:2047: oid 33 updating version 0 to 682 2024-03-04T06:16:07.501 INFO:tasks.rados.rados.0.smithi064.stdout:2047: oid 33 version 682 is already newer than 680 2024-03-04T06:16:07.501 INFO:tasks.rados.rados.0.smithi064.stdout:2047: oid 33 version 682 is already newer than 682 2024-03-04T06:16:07.501 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 682 (ObjNum 965 snap 241 seq_num 965) dirty exists 2024-03-04T06:16:07.501 INFO:tasks.rados.rados.0.smithi064.stdout:2047: left oid 33 (ObjNum 965 snap 241 seq_num 965) 2024-03-04T06:16:08.270 INFO:tasks.rados.rados.0.smithi064.stdout:2038: done (7 left) 2024-03-04T06:16:08.270 INFO:tasks.rados.rados.0.smithi064.stdout:2042: done (6 left) 2024-03-04T06:16:08.270 INFO:tasks.rados.rados.0.smithi064.stdout:2043: done (5 left) 2024-03-04T06:16:08.271 INFO:tasks.rados.rados.0.smithi064.stdout:2044: done (4 left) 2024-03-04T06:16:08.271 INFO:tasks.rados.rados.0.smithi064.stdout:2045: done (3 left) 2024-03-04T06:16:08.271 INFO:tasks.rados.rados.0.smithi064.stdout:2047: done (2 left) 2024-03-04T06:16:08.271 INFO:tasks.rados.rados.0.smithi064.stdout:2048: done (1 left) 2024-03-04T06:16:08.271 INFO:tasks.rados.rados.0.smithi064.stdout:2049: done (0 left) 2024-03-04T06:16:08.271 INFO:tasks.rados.rados.0.smithi064.stdout:2050: write oid 16 current snap is 242 2024-03-04T06:16:08.271 INFO:tasks.rados.rados.0.smithi064.stdout:2050: seq_num 966 ranges {635562=667365,2094782=140780} 2024-03-04T06:16:08.277 INFO:tasks.rados.rados.0.smithi064.stdout:2050: writing smithi06431743-16 from 635562 to 1302927 tid 1 2024-03-04T06:16:08.282 INFO:tasks.rados.rados.0.smithi064.stdout:2050: writing smithi06431743-16 from 2094782 to 2235562 tid 2 2024-03-04T06:16:08.282 INFO:tasks.rados.rados.0.smithi064.stdout:2051: delete oid 32 current snap is 242 2024-03-04T06:16:08.353 INFO:tasks.rados.rados.0.smithi064.stdout:2051: done (1 left) 2024-03-04T06:16:08.353 INFO:tasks.rados.rados.0.smithi064.stdout:2052: write oid 29 current snap is 242 2024-03-04T06:16:08.353 INFO:tasks.rados.rados.0.smithi064.stdout:2052: seq_num 968 ranges {669176=417863,1811746=457430} 2024-03-04T06:16:08.359 INFO:tasks.rados.rados.0.smithi064.stdout:2052: writing smithi06431743-29 from 669176 to 1087039 tid 1 2024-03-04T06:16:08.366 INFO:tasks.rados.rados.0.smithi064.stdout:2052: writing smithi06431743-29 from 1811746 to 2269176 tid 2 2024-03-04T06:16:08.367 INFO:tasks.rados.rados.0.smithi064.stdout:2053: write oid 43 current snap is 242 2024-03-04T06:16:08.367 INFO:tasks.rados.rados.0.smithi064.stdout:2053: seq_num 969 ranges {685983=493112,1870228=415755} 2024-03-04T06:16:08.373 INFO:tasks.rados.rados.0.smithi064.stdout:2053: writing smithi06431743-43 from 685983 to 1179095 tid 1 2024-03-04T06:16:08.380 INFO:tasks.rados.rados.0.smithi064.stdout:2053: writing smithi06431743-43 from 1870228 to 2285983 tid 2 2024-03-04T06:16:08.380 INFO:tasks.rados.rados.0.smithi064.stdout:2054: read oid 35 snap 238 2024-03-04T06:16:08.380 INFO:tasks.rados.rados.0.smithi064.stdout:2054: expect deleted 2024-03-04T06:16:08.380 INFO:tasks.rados.rados.0.smithi064.stdout:2055: delete oid 21 current snap is 242 2024-03-04T06:16:08.381 INFO:tasks.rados.rados.0.smithi064.stdout:2050: finishing write tid 1 to smithi06431743-16 2024-03-04T06:16:08.381 INFO:tasks.rados.rados.0.smithi064.stdout:2050: finishing write tid 2 to smithi06431743-16 2024-03-04T06:16:08.381 INFO:tasks.rados.rados.0.smithi064.stdout:2050: finishing write tid 3 to smithi06431743-16 2024-03-04T06:16:08.381 INFO:tasks.rados.rados.0.smithi064.stdout:2050: finishing write tid 4 to smithi06431743-16 2024-03-04T06:16:08.381 INFO:tasks.rados.rados.0.smithi064.stdout:2050: oid 16 updating version 0 to 389 2024-03-04T06:16:08.381 INFO:tasks.rados.rados.0.smithi064.stdout:2050: oid 16 version 389 is already newer than 389 2024-03-04T06:16:08.381 INFO:tasks.rados.rados.0.smithi064.stdout:2050: oid 16 version 389 is already newer than 387 2024-03-04T06:16:08.381 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 389 (ObjNum 966 snap 242 seq_num 966) dirty exists 2024-03-04T06:16:08.382 INFO:tasks.rados.rados.0.smithi064.stdout:2050: left oid 16 (ObjNum 966 snap 242 seq_num 966) 2024-03-04T06:16:08.384 INFO:tasks.rados.rados.0.smithi064.stdout:2052: finishing write tid 1 to smithi06431743-29 2024-03-04T06:16:08.404 INFO:tasks.rados.rados.0.smithi064.stdout:2053: finishing write tid 1 to smithi06431743-43 2024-03-04T06:16:08.411 INFO:tasks.rados.rados.0.smithi064.stdout:2052: finishing write tid 2 to smithi06431743-29 2024-03-04T06:16:08.417 INFO:tasks.rados.rados.0.smithi064.stdout:2050: done (4 left) 2024-03-04T06:16:08.417 INFO:tasks.rados.rados.0.smithi064.stdout:2054: done (3 left) 2024-03-04T06:16:08.417 INFO:tasks.rados.rados.0.smithi064.stdout:2055: done (2 left) 2024-03-04T06:16:08.417 INFO:tasks.rados.rados.0.smithi064.stdout:2056: read oid 30 snap -1 2024-03-04T06:16:08.417 INFO:tasks.rados.rados.0.smithi064.stdout:2056: expect deleted 2024-03-04T06:16:08.417 INFO:tasks.rados.rados.0.smithi064.stdout:2057: snap_create 2024-03-04T06:16:08.418 INFO:tasks.rados.rados.0.smithi064.stdout:2052: finishing write tid 3 to smithi06431743-29 2024-03-04T06:16:08.419 INFO:tasks.rados.rados.0.smithi064.stdout:2052: finishing write tid 4 to smithi06431743-29 2024-03-04T06:16:08.419 INFO:tasks.rados.rados.0.smithi064.stdout:2052: oid 29 updating version 0 to 878 2024-03-04T06:16:08.419 INFO:tasks.rados.rados.0.smithi064.stdout:2052: oid 29 updating version 878 to 880 2024-03-04T06:16:08.419 INFO:tasks.rados.rados.0.smithi064.stdout:2052: oid 29 version 880 is already newer than 880 2024-03-04T06:16:08.419 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 880 (ObjNum 968 snap 242 seq_num 968) dirty exists 2024-03-04T06:16:08.419 INFO:tasks.rados.rados.0.smithi064.stdout:2052: left oid 29 (ObjNum 968 snap 242 seq_num 968) 2024-03-04T06:16:08.421 INFO:tasks.rados.rados.0.smithi064.stdout:2053: finishing write tid 2 to smithi06431743-43 2024-03-04T06:16:08.427 INFO:tasks.rados.rados.0.smithi064.stdout:2053: finishing write tid 3 to smithi06431743-43 2024-03-04T06:16:08.427 INFO:tasks.rados.rados.0.smithi064.stdout:2053: finishing write tid 4 to smithi06431743-43 2024-03-04T06:16:08.427 INFO:tasks.rados.rados.0.smithi064.stdout:2053: oid 43 updating version 0 to 425 2024-03-04T06:16:08.427 INFO:tasks.rados.rados.0.smithi064.stdout:2053: oid 43 updating version 425 to 426 2024-03-04T06:16:08.427 INFO:tasks.rados.rados.0.smithi064.stdout:2053: oid 43 version 426 is already newer than 424 2024-03-04T06:16:08.427 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 426 (ObjNum 969 snap 242 seq_num 969) dirty exists 2024-03-04T06:16:08.427 INFO:tasks.rados.rados.0.smithi064.stdout:2053: left oid 43 (ObjNum 969 snap 242 seq_num 969) 2024-03-04T06:16:08.869 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:08.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:08.979+0000 7efc8175d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:08.984 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:08.983+0000 7efc8175d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:09.145 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:16:09.145 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:16:09.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:16:09.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:09.248+0000 7f10e2c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:09.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:09.254+0000 7f10e2c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:09.272 INFO:tasks.rados.rados.0.smithi064.stdout:2052: done (3 left) 2024-03-04T06:16:09.272 INFO:tasks.rados.rados.0.smithi064.stdout:2053: done (2 left) 2024-03-04T06:16:09.272 INFO:tasks.rados.rados.0.smithi064.stdout:2056: done (1 left) 2024-03-04T06:16:09.272 INFO:tasks.rados.rados.0.smithi064.stdout:2057: done (0 left) 2024-03-04T06:16:09.272 INFO:tasks.rados.rados.0.smithi064.stdout:2058: snap_remove snap 240 2024-03-04T06:16:09.444 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:16:09.445 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:16:09.456 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":742,"stamp":"2024-03-04T06:16:09.271080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":104094639,"num_objects":709,"num_object_clones":542,"num_object_copies":2127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":131,"num_read":3857,"num_read_kb":2917055,"num_write":3343,"num_write_kb":879085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6937,"ondisk_log_size":6937,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":375020,"kb_used_data":212236,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":377112340,"statfs":{"total":386547056640,"available":386163036160,"internally_reserved":0,"allocated":217329664,"data_stored":216145650,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-755754,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":8447,"num_write":30,"num_write_kb":8559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014842"},"pg_stats":[{"pgid":"3.f","version":"483'422","reported_seq":1751,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-04T06:16:05.305394+0000","last_change":"2024-03-04T06:16:05.314516+0000","last_active":"2024-03-04T06:16:05.305394+0000","last_peered":"2024-03-04T06:16:05.305394+0000","last_clean":"2024-03-04T06:16:05.305394+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:16:05.305394+0000","last_undegraded":"2024-03-04T06:16:05.305394+0000","last_fullsized":"2024-03-04T06:16:05.305394+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647548,"num_objects":47,"num_object_clones":37,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":246,"num_read_kb":201023,"num_write":197,"num_write_kb":53501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"490'316","reported_seq":1674,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-04T06:16:05.307793+0000","last_change":"2024-03-04T06:16:05.324319+0000","last_active":"2024-03-04T06:16:05.307793+0000","last_peered":"2024-03-04T06:16:05.307793+0000","last_clean":"2024-03-04T06:16:05.307793+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:16:05.307793+0000","last_undegraded":"2024-03-04T06:16:05.307793+0000","last_fullsized":"2024-03-04T06:16:05.307793+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916493,"num_objects":38,"num_object_clones":25,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":129,"num_read_kb":80767,"num_write":151,"num_write_kb":39574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"495'449","reported_seq":1805,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.325864+0000","last_change":"2024-03-04T06:16:05.297627+0000","last_active":"2024-03-04T06:16:06.325864+0000","last_peered":"2024-03-04T06:16:06.325864+0000","last_clean":"2024-03-04T06:16:06.325864+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:16:06.325864+0000","last_undegraded":"2024-03-04T06:16:06.325864+0000","last_fullsized":"2024-03-04T06:16:06.325864+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9143635,"num_objects":55,"num_object_clones":42,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":300,"num_read_kb":234944,"num_write":237,"num_write_kb":65476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"468'322","reported_seq":1699,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-04T06:16:05.286534+0000","last_change":"2024-03-04T06:16:05.303352+0000","last_active":"2024-03-04T06:16:05.286534+0000","last_peered":"2024-03-04T06:16:05.286534+0000","last_clean":"2024-03-04T06:16:05.286534+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:16:05.286534+0000","last_undegraded":"2024-03-04T06:16:05.286534+0000","last_fullsized":"2024-03-04T06:16:05.286534+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395410,"num_objects":33,"num_object_clones":24,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":191,"num_read_kb":149126,"num_write":163,"num_write_kb":45378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1543,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.299000+0000","last_change":"2024-03-04T06:16:05.284835+0000","last_active":"2024-03-04T06:16:06.299000+0000","last_peered":"2024-03-04T06:16:06.299000+0000","last_clean":"2024-03-04T06:16:06.299000+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:16:06.299000+0000","last_undegraded":"2024-03-04T06:16:06.299000+0000","last_fullsized":"2024-03-04T06:16:06.299000+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":46,"num_read_kb":17050,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"495'457","reported_seq":1766,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-04T06:16:05.315290+0000","last_change":"2024-03-04T06:16:05.313614+0000","last_active":"2024-03-04T06:16:05.315290+0000","last_peered":"2024-03-04T06:16:05.315290+0000","last_clean":"2024-03-04T06:16:05.315290+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:16:05.315290+0000","last_undegraded":"2024-03-04T06:16:05.315290+0000","last_fullsized":"2024-03-04T06:16:05.315290+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9126795,"num_objects":45,"num_object_clones":36,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":221,"num_read_kb":172350,"num_write":228,"num_write_kb":68023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1465,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-04T06:16:05.303462+0000","last_change":"2024-03-04T06:16:05.315430+0000","last_active":"2024-03-04T06:16:05.303462+0000","last_peered":"2024-03-04T06:16:05.303462+0000","last_clean":"2024-03-04T06:16:05.303462+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:16:05.303462+0000","last_undegraded":"2024-03-04T06:16:05.303462+0000","last_fullsized":"2024-03-04T06:16:05.303462+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"488'246","reported_seq":1641,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.319874+0000","last_change":"2024-03-04T06:16:05.298043+0000","last_active":"2024-03-04T06:16:06.319874+0000","last_peered":"2024-03-04T06:16:06.319874+0000","last_clean":"2024-03-04T06:16:06.319874+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:16:06.319874+0000","last_undegraded":"2024-03-04T06:16:06.319874+0000","last_fullsized":"2024-03-04T06:16:06.319874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":246,"log_dups_size":0,"ondisk_log_size":246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2958197,"num_objects":25,"num_object_clones":17,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":172,"num_read_kb":163219,"num_write":114,"num_write_kb":30084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1363,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.285247+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:16:06.285247+0000","last_peered":"2024-03-04T06:16:06.285247+0000","last_clean":"2024-03-04T06:16:06.285247+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:16:06.285247+0000","last_undegraded":"2024-03-04T06:16:06.285247+0000","last_fullsized":"2024-03-04T06:16:06.285247+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1232,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.336866+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:16:06.336866+0000","last_peered":"2024-03-04T06:16:06.336866+0000","last_clean":"2024-03-04T06:16:06.336866+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:16:06.336866+0000","last_undegraded":"2024-03-04T06:16:06.336866+0000","last_fullsized":"2024-03-04T06:16:06.336866+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"488'463","reported_seq":1749,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.333562+0000","last_change":"2024-03-04T06:16:05.307137+0000","last_active":"2024-03-04T06:16:06.333562+0000","last_peered":"2024-03-04T06:16:06.333562+0000","last_clean":"2024-03-04T06:16:06.333562+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:16:06.333562+0000","last_undegraded":"2024-03-04T06:16:06.333562+0000","last_fullsized":"2024-03-04T06:16:06.333562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11009179,"num_objects":48,"num_object_clones":36,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":217,"num_read_kb":160751,"num_write":219,"num_write_kb":59875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1237,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-04T06:16:05.302322+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:16:05.302322+0000","last_peered":"2024-03-04T06:16:05.302322+0000","last_clean":"2024-03-04T06:16:05.302322+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:16:05.302322+0000","last_undegraded":"2024-03-04T06:16:05.302322+0000","last_fullsized":"2024-03-04T06:16:05.302322+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"496'351","reported_seq":1672,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.288898+0000","last_change":"2024-03-04T06:16:05.276266+0000","last_active":"2024-03-04T06:16:06.288898+0000","last_peered":"2024-03-04T06:16:06.288898+0000","last_clean":"2024-03-04T06:16:06.288898+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:16:06.288898+0000","last_undegraded":"2024-03-04T06:16:06.288898+0000","last_fullsized":"2024-03-04T06:16:06.288898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":28,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":202,"num_read_kb":166619,"num_write":137,"num_write_kb":39064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1232,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.327784+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:16:06.327784+0000","last_peered":"2024-03-04T06:16:06.327784+0000","last_clean":"2024-03-04T06:16:06.327784+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:16:06.327784+0000","last_undegraded":"2024-03-04T06:16:06.327784+0000","last_fullsized":"2024-03-04T06:16:06.327784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"489'795","reported_seq":2032,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.300819+0000","last_change":"2024-03-04T06:16:05.285130+0000","last_active":"2024-03-04T06:16:06.300819+0000","last_peered":"2024-03-04T06:16:06.300819+0000","last_clean":"2024-03-04T06:16:06.300819+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:16:06.300819+0000","last_undegraded":"2024-03-04T06:16:06.300819+0000","last_fullsized":"2024-03-04T06:16:06.300819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17967640,"num_objects":81,"num_object_clones":62,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":492,"num_read_kb":393985,"num_write":392,"num_write_kb":105877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1237,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-04T06:16:05.293538+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:16:05.293538+0000","last_peered":"2024-03-04T06:16:05.293538+0000","last_clean":"2024-03-04T06:16:05.293538+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:16:05.293538+0000","last_undegraded":"2024-03-04T06:16:05.293538+0000","last_fullsized":"2024-03-04T06:16:05.293538+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"495'868","reported_seq":1989,"reported_epoch":495,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:05.310173+0000","last_change":"2024-03-04T06:16:05.309995+0000","last_active":"2024-03-04T06:16:05.310173+0000","last_peered":"2024-03-04T06:16:05.310173+0000","last_clean":"2024-03-04T06:16:05.310173+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:16:05.310173+0000","last_undegraded":"2024-03-04T06:16:05.310173+0000","last_fullsized":"2024-03-04T06:16:05.310173+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":868,"log_dups_size":0,"ondisk_log_size":868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10202674,"num_objects":85,"num_object_clones":65,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":370,"num_read_kb":265688,"num_write":387,"num_write_kb":102005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"466'369","reported_seq":1743,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.333645+0000","last_change":"2024-03-04T06:16:05.306475+0000","last_active":"2024-03-04T06:16:06.333645+0000","last_peered":"2024-03-04T06:16:06.333645+0000","last_clean":"2024-03-04T06:16:06.333645+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:16:06.333645+0000","last_undegraded":"2024-03-04T06:16:06.333645+0000","last_fullsized":"2024-03-04T06:16:06.333645+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2168301,"num_objects":38,"num_object_clones":28,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":275,"num_read_kb":218742,"num_write":175,"num_write_kb":47566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1257,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-04T06:16:05.319497+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:16:05.319497+0000","last_peered":"2024-03-04T06:16:05.319497+0000","last_clean":"2024-03-04T06:16:05.319497+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:16:05.319497+0000","last_undegraded":"2024-03-04T06:16:05.319497+0000","last_fullsized":"2024-03-04T06:16:05.319497+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"478'379","reported_seq":1733,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-04T06:16:05.327160+0000","last_change":"2024-03-04T06:16:05.327160+0000","last_active":"2024-03-04T06:16:05.327160+0000","last_peered":"2024-03-04T06:16:05.327160+0000","last_clean":"2024-03-04T06:16:05.327160+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:16:05.327160+0000","last_undegraded":"2024-03-04T06:16:05.327160+0000","last_fullsized":"2024-03-04T06:16:05.327160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3143107,"num_objects":45,"num_object_clones":32,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":205,"num_read_kb":142429,"num_write":178,"num_write_kb":48030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1230,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.294247+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:16:06.294247+0000","last_peered":"2024-03-04T06:16:06.294247+0000","last_clean":"2024-03-04T06:16:06.294247+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:16:06.294247+0000","last_undegraded":"2024-03-04T06:16:06.294247+0000","last_fullsized":"2024-03-04T06:16:06.294247+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"495'561","reported_seq":1832,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.289634+0000","last_change":"2024-03-04T06:16:05.313162+0000","last_active":"2024-03-04T06:16:06.289634+0000","last_peered":"2024-03-04T06:16:06.289634+0000","last_clean":"2024-03-04T06:16:06.289634+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:16:06.289634+0000","last_undegraded":"2024-03-04T06:16:06.289634+0000","last_fullsized":"2024-03-04T06:16:06.289634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9395707,"num_objects":56,"num_object_clones":44,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":315,"num_read_kb":248366,"num_write":263,"num_write_kb":73070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1257,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-04T06:16:05.314058+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:16:05.314058+0000","last_peered":"2024-03-04T06:16:05.314058+0000","last_clean":"2024-03-04T06:16:05.314058+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:16:05.314058+0000","last_undegraded":"2024-03-04T06:16:05.314058+0000","last_fullsized":"2024-03-04T06:16:05.314058+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"490'675","reported_seq":1913,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-04T06:16:05.309332+0000","last_change":"2024-03-04T06:16:05.324635+0000","last_active":"2024-03-04T06:16:05.309332+0000","last_peered":"2024-03-04T06:16:05.309332+0000","last_clean":"2024-03-04T06:16:05.309332+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:16:05.309332+0000","last_undegraded":"2024-03-04T06:16:05.309332+0000","last_fullsized":"2024-03-04T06:16:05.309332+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101601,"num_objects":61,"num_object_clones":53,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":373,"num_read_kb":301905,"num_write":290,"num_write_kb":79706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1232,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-04T06:16:06.332756+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:16:06.332756+0000","last_peered":"2024-03-04T06:16:06.332756+0000","last_clean":"2024-03-04T06:16:06.332756+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:16:06.332756+0000","last_undegraded":"2024-03-04T06:16:06.332756+0000","last_fullsized":"2024-03-04T06:16:06.332756+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":103504271,"num_objects":707,"num_object_clones":542,"num_object_copies":2121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":131,"num_read":3754,"num_read_kb":2916964,"num_write":3206,"num_write_kb":877191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6840,"ondisk_log_size":6840,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":109996,"kb_used_data":65124,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94261844,"statfs":{"total":96636764160,"available":96524128256,"internally_reserved":0,"allocated":66686976,"data_stored":67080870,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":97512,"kb_used_data":58208,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94274328,"statfs":{"total":96636764160,"available":96536911872,"internally_reserved":0,"allocated":59604992,"data_stored":59056087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81144,"kb_used_data":41840,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94290696,"statfs":{"total":96636764160,"available":96553672704,"internally_reserved":0,"allocated":42844160,"data_stored":42115822,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86368,"kb_used_data":47064,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94285472,"statfs":{"total":96636764160,"available":96548323328,"internally_reserved":0,"allocated":48193536,"data_stored":47892871,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:16:10.273 INFO:tasks.rados.rados.0.smithi064.stdout:2058: done (0 left) 2024-03-04T06:16:10.273 INFO:tasks.rados.rados.0.smithi064.stdout:2059: snap_remove snap 242 2024-03-04T06:16:11.054 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:11.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:11.164+0000 7f9fb5cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:11.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:11.168+0000 7f9fb5cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:11.278 INFO:tasks.rados.rados.0.smithi064.stdout:2059: done (0 left) 2024-03-04T06:16:11.279 INFO:tasks.rados.rados.0.smithi064.stdout:2060: read oid 34 snap 238 2024-03-04T06:16:11.279 INFO:tasks.rados.rados.0.smithi064.stdout:2060: expect deleted 2024-03-04T06:16:11.279 INFO:tasks.rados.rados.0.smithi064.stdout:2061: delete oid 26 current snap is 243 2024-03-04T06:16:11.351 INFO:tasks.rados.rados.0.smithi064.stdout:2060: done (1 left) 2024-03-04T06:16:11.351 INFO:tasks.rados.rados.0.smithi064.stdout:2061: done (0 left) 2024-03-04T06:16:11.351 INFO:tasks.rados.rados.0.smithi064.stdout:2062: read oid 38 snap 238 2024-03-04T06:16:11.351 INFO:tasks.rados.rados.0.smithi064.stdout:2062: expect (ObjNum 835 snap 204 seq_num 835) 2024-03-04T06:16:11.351 INFO:tasks.rados.rados.0.smithi064.stdout:2063: snap_create 2024-03-04T06:16:11.384 INFO:tasks.rados.rados.0.smithi064.stdout:2062: expect (ObjNum 835 snap 204 seq_num 835) 2024-03-04T06:16:12.282 INFO:tasks.rados.rados.0.smithi064.stdout:2062: done (1 left) 2024-03-04T06:16:12.282 INFO:tasks.rados.rados.0.smithi064.stdout:2063: done (0 left) 2024-03-04T06:16:12.282 INFO:tasks.rados.rados.0.smithi064.stdout:2064: read oid 3 snap 238 2024-03-04T06:16:12.282 INFO:tasks.rados.rados.0.smithi064.stdout:2064: expect (ObjNum 948 snap 235 seq_num 948) 2024-03-04T06:16:12.282 INFO:tasks.rados.rados.0.smithi064.stdout:2065: write oid 21 current snap is 244 2024-03-04T06:16:12.282 INFO:tasks.rados.rados.0.smithi064.stdout:2065: seq_num 972 ranges {736404=718859,1929321=407083} 2024-03-04T06:16:12.291 INFO:tasks.rados.rados.0.smithi064.stdout:2065: writing smithi06431743-21 from 736404 to 1455263 tid 1 2024-03-04T06:16:12.298 INFO:tasks.rados.rados.0.smithi064.stdout:2065: writing smithi06431743-21 from 1929321 to 2336404 tid 2 2024-03-04T06:16:12.298 INFO:tasks.rados.rados.0.smithi064.stdout:2066: snap_remove snap 241 2024-03-04T06:16:13.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:13.255 INFO:tasks.rados.rados.0.smithi064.stdout:2066: done (2 left) 2024-03-04T06:16:13.255 INFO:tasks.rados.rados.0.smithi064.stdout:2067: write (excl) oid 39 current snap is 244 2024-03-04T06:16:13.256 INFO:tasks.rados.rados.0.smithi064.stdout:2067: seq_num 973 ranges {753211=794108,1987803=365408} 2024-03-04T06:16:13.263 INFO:tasks.rados.rados.0.smithi064.stdout:2067: writing smithi06431743-39 from 753211 to 1547319 tid 1 2024-03-04T06:16:13.269 INFO:tasks.rados.rados.0.smithi064.stdout:2067: writing smithi06431743-39 from 1987803 to 2353211 tid 2 2024-03-04T06:16:13.270 INFO:tasks.rados.rados.0.smithi064.stdout:2068: read oid 44 snap -1 2024-03-04T06:16:13.270 INFO:tasks.rados.rados.0.smithi064.stdout:2068: expect (ObjNum 957 snap 240 seq_num 957) 2024-03-04T06:16:13.270 INFO:tasks.rados.rados.0.smithi064.stdout:2069: write oid 34 current snap is 244 2024-03-04T06:16:13.270 INFO:tasks.rados.rados.0.smithi064.stdout:2069: seq_num 974 ranges {770018=585710,2046285=323733} 2024-03-04T06:16:13.279 INFO:tasks.rados.rados.0.smithi064.stdout:2069: writing smithi06431743-34 from 770018 to 1355728 tid 1 2024-03-04T06:16:13.286 INFO:tasks.rados.rados.0.smithi064.stdout:2069: writing smithi06431743-34 from 2046285 to 2370018 tid 2 2024-03-04T06:16:13.286 INFO:tasks.rados.rados.0.smithi064.stdout:2070: write (excl) oid 50 current snap is 244 2024-03-04T06:16:13.286 INFO:tasks.rados.rados.0.smithi064.stdout:2064: expect (ObjNum 948 snap 235 seq_num 948) 2024-03-04T06:16:13.397 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:13.395+0000 7f57a6c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:13.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:13.400+0000 7f57a6c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:13.505 INFO:tasks.rados.rados.0.smithi064.stdout:2065: finishing write tid 1 to smithi06431743-21 2024-03-04T06:16:13.505 INFO:tasks.rados.rados.0.smithi064.stdout:2065: finishing write tid 2 to smithi06431743-21 2024-03-04T06:16:13.506 INFO:tasks.rados.rados.0.smithi064.stdout:2065: finishing write tid 3 to smithi06431743-21 2024-03-04T06:16:13.506 INFO:tasks.rados.rados.0.smithi064.stdout:2065: finishing write tid 4 to smithi06431743-21 2024-03-04T06:16:13.506 INFO:tasks.rados.rados.0.smithi064.stdout:2065: oid 21 updating version 0 to 375 2024-03-04T06:16:13.506 INFO:tasks.rados.rados.0.smithi064.stdout:2065: oid 21 version 375 is already newer than 375 2024-03-04T06:16:13.506 INFO:tasks.rados.rados.0.smithi064.stdout:2065: oid 21 updating version 375 to 376 2024-03-04T06:16:13.506 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 376 (ObjNum 972 snap 244 seq_num 972) dirty exists 2024-03-04T06:16:13.506 INFO:tasks.rados.rados.0.smithi064.stdout:2065: left oid 21 (ObjNum 972 snap 244 seq_num 972) 2024-03-04T06:16:13.506 INFO:tasks.rados.rados.0.smithi064.stdout:2070: seq_num 975 ranges {786825=660959,2104767=282058} 2024-03-04T06:16:13.513 INFO:tasks.rados.rados.0.smithi064.stdout:2070: writing smithi06431743-50 from 786825 to 1447784 tid 1 2024-03-04T06:16:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:2070: writing smithi06431743-50 from 2104767 to 2386825 tid 2 2024-03-04T06:16:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:2064: done (5 left) 2024-03-04T06:16:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:2065: done (4 left) 2024-03-04T06:16:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:2071: snap_remove snap 239 2024-03-04T06:16:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:2069: finishing write tid 1 to smithi06431743-34 2024-03-04T06:16:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:2067: finishing write tid 1 to smithi06431743-39 2024-03-04T06:16:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:2068: expect (ObjNum 957 snap 240 seq_num 957) 2024-03-04T06:16:13.675 INFO:tasks.rados.rados.0.smithi064.stdout:2069: finishing write tid 2 to smithi06431743-34 2024-03-04T06:16:13.675 INFO:tasks.rados.rados.0.smithi064.stdout:2067: finishing write tid 2 to smithi06431743-39 2024-03-04T06:16:13.675 INFO:tasks.rados.rados.0.smithi064.stdout:2069: finishing write tid 3 to smithi06431743-34 2024-03-04T06:16:13.675 INFO:tasks.rados.rados.0.smithi064.stdout:2069: finishing write tid 4 to smithi06431743-34 2024-03-04T06:16:13.675 INFO:tasks.rados.rados.0.smithi064.stdout:2069: oid 34 updating version 0 to 808 2024-03-04T06:16:13.675 INFO:tasks.rados.rados.0.smithi064.stdout:2069: oid 34 version 808 is already newer than 807 2024-03-04T06:16:13.676 INFO:tasks.rados.rados.0.smithi064.stdout:2069: oid 34 version 808 is already newer than 806 2024-03-04T06:16:13.676 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 808 (ObjNum 974 snap 244 seq_num 974) dirty exists 2024-03-04T06:16:13.676 INFO:tasks.rados.rados.0.smithi064.stdout:2069: left oid 34 (ObjNum 974 snap 244 seq_num 974) 2024-03-04T06:16:13.676 INFO:tasks.rados.rados.0.smithi064.stdout:2067: finishing write tid 3 to smithi06431743-39 2024-03-04T06:16:14.289 INFO:tasks.rados.rados.0.smithi064.stdout:2068: done (4 left) 2024-03-04T06:16:14.289 INFO:tasks.rados.rados.0.smithi064.stdout:2069: done (3 left) 2024-03-04T06:16:14.289 INFO:tasks.rados.rados.0.smithi064.stdout:2071: done (2 left) 2024-03-04T06:16:14.289 INFO:tasks.rados.rados.0.smithi064.stdout:2072: read oid 27 snap -1 2024-03-04T06:16:14.289 INFO:tasks.rados.rados.0.smithi064.stdout:2072: expect deleted 2024-03-04T06:16:14.290 INFO:tasks.rados.rados.0.smithi064.stdout:2073: snap_remove snap 238 2024-03-04T06:16:14.459 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:16:14.459 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:16:14.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:16:14.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:14.567+0000 7f6ede26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:14.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:14.573+0000 7f6ede26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:14.764 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:16:14.764 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:16:14.776 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2024-03-04T06:16:14.287100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93506031,"num_objects":719,"num_object_clones":551,"num_object_copies":2157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":134,"num_read":3882,"num_read_kb":2927731,"num_write":3382,"num_write_kb":886438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6996,"ondisk_log_size":6996,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":398552,"kb_used_data":235768,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":377088808,"statfs":{"total":386547056640,"available":386138939392,"internally_reserved":0,"allocated":241426432,"data_stored":249631442,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10588608,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":25,"num_read_kb":10676,"num_write":39,"num_write_kb":7353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015052"},"pg_stats":[{"pgid":"3.f","version":"500'428","reported_seq":1772,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-04T06:16:10.293417+0000","last_change":"2024-03-04T06:16:10.313170+0000","last_active":"2024-03-04T06:16:10.293417+0000","last_peered":"2024-03-04T06:16:10.293417+0000","last_clean":"2024-03-04T06:16:10.293417+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:16:10.293417+0000","last_undegraded":"2024-03-04T06:16:10.293417+0000","last_fullsized":"2024-03-04T06:16:10.293417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5462704,"num_objects":48,"num_object_clones":38,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":247,"num_read_kb":201024,"num_write":201,"num_write_kb":54390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"490'316","reported_seq":1687,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-04T06:16:09.898903+0000","last_change":"2024-03-04T06:16:07.297250+0000","last_active":"2024-03-04T06:16:09.898903+0000","last_peered":"2024-03-04T06:16:09.898903+0000","last_clean":"2024-03-04T06:16:09.898903+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:16:09.898903+0000","last_undegraded":"2024-03-04T06:16:09.898903+0000","last_fullsized":"2024-03-04T06:16:09.898903+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916493,"num_objects":38,"num_object_clones":25,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":129,"num_read_kb":80767,"num_write":151,"num_write_kb":39574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"497'451","reported_seq":1824,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-04T06:16:11.297646+0000","last_change":"2024-03-04T06:16:11.300685+0000","last_active":"2024-03-04T06:16:11.297646+0000","last_peered":"2024-03-04T06:16:11.297646+0000","last_clean":"2024-03-04T06:16:11.297646+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:16:11.297646+0000","last_undegraded":"2024-03-04T06:16:11.297646+0000","last_fullsized":"2024-03-04T06:16:11.297646+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9143635,"num_objects":55,"num_object_clones":42,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":306,"num_read_kb":240176,"num_write":237,"num_write_kb":65476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"468'322","reported_seq":1716,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-04T06:16:10.280617+0000","last_change":"2024-03-04T06:16:10.289493+0000","last_active":"2024-03-04T06:16:10.280617+0000","last_peered":"2024-03-04T06:16:10.280617+0000","last_clean":"2024-03-04T06:16:10.280617+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:16:10.280617+0000","last_undegraded":"2024-03-04T06:16:10.280617+0000","last_fullsized":"2024-03-04T06:16:10.280617+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395410,"num_objects":33,"num_object_clones":24,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":191,"num_read_kb":149126,"num_write":163,"num_write_kb":45378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1557,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-04T06:16:11.322434+0000","last_change":"2024-03-04T06:16:11.331072+0000","last_active":"2024-03-04T06:16:11.322434+0000","last_peered":"2024-03-04T06:16:11.322434+0000","last_clean":"2024-03-04T06:16:11.322434+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:16:11.322434+0000","last_undegraded":"2024-03-04T06:16:11.322434+0000","last_fullsized":"2024-03-04T06:16:11.322434+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":46,"num_read_kb":17050,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"497'461","reported_seq":1785,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-04T06:16:10.295364+0000","last_change":"2024-03-04T06:16:10.300431+0000","last_active":"2024-03-04T06:16:10.295364+0000","last_peered":"2024-03-04T06:16:10.295364+0000","last_clean":"2024-03-04T06:16:10.295364+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:16:10.295364+0000","last_undegraded":"2024-03-04T06:16:10.295364+0000","last_fullsized":"2024-03-04T06:16:10.295364+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7580584,"num_objects":45,"num_object_clones":36,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":222,"num_read_kb":172351,"num_write":230,"num_write_kb":68023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1482,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-04T06:16:10.290800+0000","last_change":"2024-03-04T06:16:10.300554+0000","last_active":"2024-03-04T06:16:10.290800+0000","last_peered":"2024-03-04T06:16:10.290800+0000","last_clean":"2024-03-04T06:16:10.290800+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:16:10.290800+0000","last_undegraded":"2024-03-04T06:16:10.290800+0000","last_fullsized":"2024-03-04T06:16:10.290800+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"488'246","reported_seq":1657,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-04T06:16:11.296536+0000","last_change":"2024-03-04T06:16:11.300204+0000","last_active":"2024-03-04T06:16:11.296536+0000","last_peered":"2024-03-04T06:16:11.296536+0000","last_clean":"2024-03-04T06:16:11.296536+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:16:11.296536+0000","last_undegraded":"2024-03-04T06:16:11.296536+0000","last_fullsized":"2024-03-04T06:16:11.296536+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":246,"log_dups_size":0,"ondisk_log_size":246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2958197,"num_objects":25,"num_object_clones":17,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":172,"num_read_kb":163219,"num_write":114,"num_write_kb":30084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1374,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-04T06:16:11.302995+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:16:11.302995+0000","last_peered":"2024-03-04T06:16:11.302995+0000","last_clean":"2024-03-04T06:16:11.302995+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:16:11.302995+0000","last_undegraded":"2024-03-04T06:16:11.302995+0000","last_fullsized":"2024-03-04T06:16:11.302995+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1245,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-04T06:16:11.305686+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:16:11.305686+0000","last_peered":"2024-03-04T06:16:11.305686+0000","last_clean":"2024-03-04T06:16:11.305686+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:16:11.305686+0000","last_undegraded":"2024-03-04T06:16:11.305686+0000","last_fullsized":"2024-03-04T06:16:11.305686+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"497'469","reported_seq":1767,"reported_epoch":501,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:11.300755+0000","last_change":"2024-03-04T06:16:11.300755+0000","last_active":"2024-03-04T06:16:11.300755+0000","last_peered":"2024-03-04T06:16:11.300755+0000","last_clean":"2024-03-04T06:16:11.300755+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:16:11.300755+0000","last_undegraded":"2024-03-04T06:16:11.300755+0000","last_fullsized":"2024-03-04T06:16:11.300755+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9278091,"num_objects":49,"num_object_clones":37,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":218,"num_read_kb":160752,"num_write":223,"num_write_kb":60877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1252,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-04T06:16:10.291153+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:16:10.291153+0000","last_peered":"2024-03-04T06:16:10.291153+0000","last_clean":"2024-03-04T06:16:10.291153+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:16:10.291153+0000","last_undegraded":"2024-03-04T06:16:10.291153+0000","last_fullsized":"2024-03-04T06:16:10.291153+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"498'355","reported_seq":1684,"reported_epoch":501,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:11.302000+0000","last_change":"2024-03-04T06:16:11.302000+0000","last_active":"2024-03-04T06:16:11.302000+0000","last_peered":"2024-03-04T06:16:11.302000+0000","last_clean":"2024-03-04T06:16:11.302000+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:16:11.302000+0000","last_undegraded":"2024-03-04T06:16:11.302000+0000","last_fullsized":"2024-03-04T06:16:11.302000+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":28,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":202,"num_read_kb":166619,"num_write":137,"num_write_kb":39064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1245,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-04T06:16:11.287226+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:16:11.287226+0000","last_peered":"2024-03-04T06:16:11.287226+0000","last_clean":"2024-03-04T06:16:11.287226+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:16:11.287226+0000","last_undegraded":"2024-03-04T06:16:11.287226+0000","last_fullsized":"2024-03-04T06:16:11.287226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"501'799","reported_seq":2046,"reported_epoch":501,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:11.314190+0000","last_change":"2024-03-04T06:16:11.314018+0000","last_active":"2024-03-04T06:16:11.314190+0000","last_peered":"2024-03-04T06:16:11.314190+0000","last_clean":"2024-03-04T06:16:11.314190+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:16:11.314190+0000","last_undegraded":"2024-03-04T06:16:11.314190+0000","last_fullsized":"2024-03-04T06:16:11.314190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11446128,"num_objects":83,"num_object_clones":64,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":493,"num_read_kb":393985,"num_write":394,"num_write_kb":105877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1252,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-03-04T06:16:10.285344+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:16:10.285344+0000","last_peered":"2024-03-04T06:16:10.285344+0000","last_clean":"2024-03-04T06:16:10.285344+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:16:10.285344+0000","last_undegraded":"2024-03-04T06:16:10.285344+0000","last_fullsized":"2024-03-04T06:16:10.285344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"498'880","reported_seq":2014,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-04T06:16:09.895982+0000","last_change":"2024-03-04T06:16:07.297462+0000","last_active":"2024-03-04T06:16:09.895982+0000","last_peered":"2024-03-04T06:16:09.895982+0000","last_clean":"2024-03-04T06:16:09.895982+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:16:09.895982+0000","last_undegraded":"2024-03-04T06:16:09.895982+0000","last_fullsized":"2024-03-04T06:16:09.895982+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10286742,"num_objects":87,"num_object_clones":67,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":374,"num_read_kb":265691,"num_write":398,"num_write_kb":104445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"498'371","reported_seq":1758,"reported_epoch":501,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:11.304435+0000","last_change":"2024-03-04T06:16:11.304435+0000","last_active":"2024-03-04T06:16:11.304435+0000","last_peered":"2024-03-04T06:16:11.304435+0000","last_clean":"2024-03-04T06:16:11.304435+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:16:11.304435+0000","last_undegraded":"2024-03-04T06:16:11.304435+0000","last_fullsized":"2024-03-04T06:16:11.304435+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":29,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":275,"num_read_kb":218742,"num_write":176,"num_write_kb":47566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1269,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-04T06:16:09.910225+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:16:09.910225+0000","last_peered":"2024-03-04T06:16:09.910225+0000","last_clean":"2024-03-04T06:16:09.910225+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:16:09.910225+0000","last_undegraded":"2024-03-04T06:16:09.910225+0000","last_fullsized":"2024-03-04T06:16:09.910225+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"498'389","reported_seq":1753,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-04T06:16:09.904217+0000","last_change":"2024-03-04T06:16:07.320105+0000","last_active":"2024-03-04T06:16:09.904217+0000","last_peered":"2024-03-04T06:16:09.904217+0000","last_clean":"2024-03-04T06:16:09.904217+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:16:09.904217+0000","last_undegraded":"2024-03-04T06:16:09.904217+0000","last_fullsized":"2024-03-04T06:16:09.904217+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235562,"num_objects":46,"num_object_clones":33,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":207,"num_read_kb":142431,"num_write":185,"num_write_kb":49170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1241,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-04T06:16:11.314555+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:16:11.314555+0000","last_peered":"2024-03-04T06:16:11.314555+0000","last_clean":"2024-03-04T06:16:11.314555+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:16:11.314555+0000","last_undegraded":"2024-03-04T06:16:11.314555+0000","last_fullsized":"2024-03-04T06:16:11.314555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"498'563","reported_seq":1846,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-04T06:16:11.308344+0000","last_change":"2024-03-04T06:16:11.316997+0000","last_active":"2024-03-04T06:16:11.308344+0000","last_peered":"2024-03-04T06:16:11.308344+0000","last_clean":"2024-03-04T06:16:11.308344+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:16:11.308344+0000","last_undegraded":"2024-03-04T06:16:11.308344+0000","last_fullsized":"2024-03-04T06:16:11.308344+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9395707,"num_objects":56,"num_object_clones":44,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":315,"num_read_kb":248366,"num_write":263,"num_write_kb":73070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1269,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-04T06:16:09.898162+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:16:09.898162+0000","last_peered":"2024-03-04T06:16:09.898162+0000","last_clean":"2024-03-04T06:16:09.898162+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:16:09.898162+0000","last_undegraded":"2024-03-04T06:16:09.898162+0000","last_fullsized":"2024-03-04T06:16:09.898162+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"497'682","reported_seq":1937,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-03-04T06:16:09.894239+0000","last_change":"2024-03-04T06:16:07.297624+0000","last_active":"2024-03-04T06:16:09.894239+0000","last_peered":"2024-03-04T06:16:09.894239+0000","last_clean":"2024-03-04T06:16:09.894239+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:16:09.894239+0000","last_undegraded":"2024-03-04T06:16:09.894239+0000","last_fullsized":"2024-03-04T06:16:09.894239+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10488426,"num_objects":63,"num_object_clones":54,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":5,"num_read":382,"num_read_kb":307341,"num_write":298,"num_write_kb":81588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1245,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-04T06:16:11.301611+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:16:11.301611+0000","last_peered":"2024-03-04T06:16:11.301611+0000","last_clean":"2024-03-04T06:16:11.301611+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:16:11.301611+0000","last_undegraded":"2024-03-04T06:16:11.301611+0000","last_fullsized":"2024-03-04T06:16:11.301611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":92915663,"num_objects":717,"num_object_clones":551,"num_object_copies":2151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":134,"num_read":3779,"num_read_kb":2927640,"num_write":3245,"num_write_kb":884544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6899,"ondisk_log_size":6899,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706169,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":116640,"kb_used_data":71768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94255200,"statfs":{"total":96636764160,"available":96517324800,"internally_reserved":0,"allocated":73490432,"data_stored":76275634,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706169,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":101344,"kb_used_data":62040,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94270496,"statfs":{"total":96636764160,"available":96532987904,"internally_reserved":0,"allocated":63528960,"data_stored":64059778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87216,"kb_used_data":47912,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94284624,"statfs":{"total":96636764160,"available":96547454976,"internally_reserved":0,"allocated":49061888,"data_stored":50850550,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706168,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":93352,"kb_used_data":54048,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94278488,"statfs":{"total":96636764160,"available":96541171712,"internally_reserved":0,"allocated":55345152,"data_stored":58445480,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:16:15.291 INFO:tasks.rados.rados.0.smithi064.stdout:2073: done (3 left) 2024-03-04T06:16:15.291 INFO:tasks.rados.rados.0.smithi064.stdout:2074: snap_remove snap 243 2024-03-04T06:16:15.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:15.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:15.581+0000 7f15a106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:15.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:15.585+0000 7f15a106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:16.295 INFO:tasks.rados.rados.0.smithi064.stdout:2074: done (3 left) 2024-03-04T06:16:16.297 INFO:tasks.rados.rados.0.smithi064.stdout:2075: write oid 6 current snap is 244 2024-03-04T06:16:16.298 INFO:tasks.rados.rados.0.smithi064.stdout:2067: finishing write tid 4 to smithi06431743-39 2024-03-04T06:16:16.298 INFO:tasks.rados.rados.0.smithi064.stdout:2067: oid 39 updating version 0 to 455 2024-03-04T06:16:16.298 INFO:tasks.rados.rados.0.smithi064.stdout:2067: oid 39 version 455 is already newer than 453 2024-03-04T06:16:16.298 INFO:tasks.rados.rados.0.smithi064.stdout:2067: oid 39 version 455 is already newer than 454 2024-03-04T06:16:16.298 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 455 (ObjNum 973 snap 244 seq_num 973) dirty exists 2024-03-04T06:16:16.298 INFO:tasks.rados.rados.0.smithi064.stdout:2067: left oid 39 (ObjNum 973 snap 244 seq_num 973) 2024-03-04T06:16:16.298 INFO:tasks.rados.rados.0.smithi064.stdout:2075: seq_num 976 ranges {403632=736208,1763249=619999} 2024-03-04T06:16:16.300 INFO:tasks.rados.rados.0.smithi064.stdout:2075: writing smithi06431743-6 from 403632 to 1139840 tid 1 2024-03-04T06:16:16.307 INFO:tasks.rados.rados.0.smithi064.stdout:2075: writing smithi06431743-6 from 1763249 to 2383248 tid 2 2024-03-04T06:16:16.307 INFO:tasks.rados.rados.0.smithi064.stdout:2067: done (3 left) 2024-03-04T06:16:16.307 INFO:tasks.rados.rados.0.smithi064.stdout:2076: snap_create 2024-03-04T06:16:16.307 INFO:tasks.rados.rados.0.smithi064.stdout:2070: finishing write tid 1 to smithi06431743-50 2024-03-04T06:16:16.308 INFO:tasks.rados.rados.0.smithi064.stdout:2070: finishing write tid 2 to smithi06431743-50 2024-03-04T06:16:16.308 INFO:tasks.rados.rados.0.smithi064.stdout:2070: finishing write tid 3 to smithi06431743-50 2024-03-04T06:16:16.308 INFO:tasks.rados.rados.0.smithi064.stdout:2070: finishing write tid 4 to smithi06431743-50 2024-03-04T06:16:16.308 INFO:tasks.rados.rados.0.smithi064.stdout:2070: oid 50 updating version 0 to 687 2024-03-04T06:16:16.308 INFO:tasks.rados.rados.0.smithi064.stdout:2070: oid 50 updating version 687 to 688 2024-03-04T06:16:16.308 INFO:tasks.rados.rados.0.smithi064.stdout:2070: oid 50 version 688 is already newer than 686 2024-03-04T06:16:16.308 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 688 (ObjNum 975 snap 244 seq_num 975) dirty exists 2024-03-04T06:16:16.308 INFO:tasks.rados.rados.0.smithi064.stdout:2070: left oid 50 (ObjNum 975 snap 244 seq_num 975) 2024-03-04T06:16:16.342 INFO:tasks.rados.rados.0.smithi064.stdout:2075: finishing write tid 1 to smithi06431743-6 2024-03-04T06:16:16.364 INFO:tasks.rados.rados.0.smithi064.stdout:2075: finishing write tid 2 to smithi06431743-6 2024-03-04T06:16:16.370 INFO:tasks.rados.rados.0.smithi064.stdout:2075: finishing write tid 3 to smithi06431743-6 2024-03-04T06:16:16.371 INFO:tasks.rados.rados.0.smithi064.stdout:2075: finishing write tid 4 to smithi06431743-6 2024-03-04T06:16:16.371 INFO:tasks.rados.rados.0.smithi064.stdout:2075: oid 6 updating version 0 to 385 2024-03-04T06:16:16.371 INFO:tasks.rados.rados.0.smithi064.stdout:2075: oid 6 version 385 is already newer than 383 2024-03-04T06:16:16.371 INFO:tasks.rados.rados.0.smithi064.stdout:2075: oid 6 version 385 is already newer than 384 2024-03-04T06:16:16.371 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 385 (ObjNum 976 snap 244 seq_num 976) dirty exists 2024-03-04T06:16:16.371 INFO:tasks.rados.rados.0.smithi064.stdout:2075: left oid 6 (ObjNum 976 snap 244 seq_num 976) 2024-03-04T06:16:17.300 INFO:tasks.rados.rados.0.smithi064.stdout:2070: done (3 left) 2024-03-04T06:16:17.300 INFO:tasks.rados.rados.0.smithi064.stdout:2072: done (2 left) 2024-03-04T06:16:17.300 INFO:tasks.rados.rados.0.smithi064.stdout:2075: done (1 left) 2024-03-04T06:16:17.300 INFO:tasks.rados.rados.0.smithi064.stdout:2076: done (0 left) 2024-03-04T06:16:17.300 INFO:tasks.rados.rados.0.smithi064.stdout:2077: read oid 45 snap -1 2024-03-04T06:16:17.300 INFO:tasks.rados.rados.0.smithi064.stdout:2077: expect deleted 2024-03-04T06:16:17.300 INFO:tasks.rados.rados.0.smithi064.stdout:2078: write (excl) oid 49 current snap is 245 2024-03-04T06:16:17.300 INFO:tasks.rados.rados.0.smithi064.stdout:2078: seq_num 977 ranges {420439=411457,1421731=480010} 2024-03-04T06:16:17.304 INFO:tasks.rados.rados.0.smithi064.stdout:2078: writing smithi06431743-49 from 420439 to 831896 tid 1 2024-03-04T06:16:17.311 INFO:tasks.rados.rados.0.smithi064.stdout:2078: writing smithi06431743-49 from 1421731 to 1901741 tid 2 2024-03-04T06:16:17.311 INFO:tasks.rados.rados.0.smithi064.stdout:2079: read oid 11 snap -1 2024-03-04T06:16:17.311 INFO:tasks.rados.rados.0.smithi064.stdout:2079: expect (ObjNum 956 snap 240 seq_num 956) 2024-03-04T06:16:17.311 INFO:tasks.rados.rados.0.smithi064.stdout:2080: read oid 3 snap -1 2024-03-04T06:16:17.311 INFO:tasks.rados.rados.0.smithi064.stdout:2080: expect (ObjNum 964 snap 241 seq_num 964) 2024-03-04T06:16:17.312 INFO:tasks.rados.rados.0.smithi064.stdout:2081: read oid 10 snap -1 2024-03-04T06:16:17.312 INFO:tasks.rados.rados.0.smithi064.stdout:2081: expect (ObjNum 920 snap 225 seq_num 920) 2024-03-04T06:16:17.312 INFO:tasks.rados.rados.0.smithi064.stdout:2082: delete oid 16 current snap is 245 2024-03-04T06:16:17.374 INFO:tasks.rados.rados.0.smithi064.stdout:2079: expect (ObjNum 956 snap 240 seq_num 956) 2024-03-04T06:16:17.496 INFO:tasks.rados.rados.0.smithi064.stdout:2080: expect (ObjNum 964 snap 241 seq_num 964) 2024-03-04T06:16:17.620 INFO:tasks.rados.rados.0.smithi064.stdout:2078: finishing write tid 1 to smithi06431743-49 2024-03-04T06:16:17.620 INFO:tasks.rados.rados.0.smithi064.stdout:2081: expect (ObjNum 920 snap 225 seq_num 920) 2024-03-04T06:16:17.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:17.749 INFO:tasks.rados.rados.0.smithi064.stdout:2078: finishing write tid 2 to smithi06431743-49 2024-03-04T06:16:17.749 INFO:tasks.rados.rados.0.smithi064.stdout:2078: finishing write tid 3 to smithi06431743-49 2024-03-04T06:16:17.749 INFO:tasks.rados.rados.0.smithi064.stdout:2078: finishing write tid 4 to smithi06431743-49 2024-03-04T06:16:17.750 INFO:tasks.rados.rados.0.smithi064.stdout:2078: oid 49 updating version 0 to 247 2024-03-04T06:16:17.750 INFO:tasks.rados.rados.0.smithi064.stdout:2078: oid 49 updating version 247 to 248 2024-03-04T06:16:17.750 INFO:tasks.rados.rados.0.smithi064.stdout:2078: oid 49 updating version 248 to 249 2024-03-04T06:16:17.750 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 249 (ObjNum 977 snap 245 seq_num 977) dirty exists 2024-03-04T06:16:17.750 INFO:tasks.rados.rados.0.smithi064.stdout:2078: left oid 49 (ObjNum 977 snap 245 seq_num 977) 2024-03-04T06:16:17.750 INFO:tasks.rados.rados.0.smithi064.stdout:2077: done (5 left) 2024-03-04T06:16:17.750 INFO:tasks.rados.rados.0.smithi064.stdout:2078: done (4 left) 2024-03-04T06:16:17.750 INFO:tasks.rados.rados.0.smithi064.stdout:2079: done (3 left) 2024-03-04T06:16:17.750 INFO:tasks.rados.rados.0.smithi064.stdout:2080: done (2 left) 2024-03-04T06:16:17.750 INFO:tasks.rados.rados.0.smithi064.stdout:2081: done (1 left) 2024-03-04T06:16:17.750 INFO:tasks.rados.rados.0.smithi064.stdout:2082: done (0 left) 2024-03-04T06:16:17.751 INFO:tasks.rados.rados.0.smithi064.stdout:2083: snap_remove snap 244 2024-03-04T06:16:17.763 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:17.761+0000 7f333cd07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:17.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:17.765+0000 7f333cd07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:18.300 INFO:tasks.rados.rados.0.smithi064.stdout:2083: done (0 left) 2024-03-04T06:16:18.300 INFO:tasks.rados.rados.0.smithi064.stdout:2084: write (excl) oid 8 current snap is 245 2024-03-04T06:16:18.300 INFO:tasks.rados.rados.0.smithi064.stdout:2084: seq_num 979 ranges {454053=561955,1422342=483679,2368404=85649} 2024-03-04T06:16:18.308 INFO:tasks.rados.rados.0.smithi064.stdout:2084: writing smithi06431743-8 from 454053 to 1016008 tid 1 2024-03-04T06:16:18.311 INFO:tasks.rados.rados.0.smithi064.stdout:2084: writing smithi06431743-8 from 1422342 to 1906021 tid 2 2024-03-04T06:16:18.314 INFO:tasks.rados.rados.0.smithi064.stdout:2084: writing smithi06431743-8 from 2368404 to 2454053 tid 3 2024-03-04T06:16:18.314 INFO:tasks.rados.rados.0.smithi064.stdout:2085: snap_create 2024-03-04T06:16:18.354 INFO:tasks.rados.rados.0.smithi064.stdout:2084: finishing write tid 1 to smithi06431743-8 2024-03-04T06:16:18.368 INFO:tasks.rados.rados.0.smithi064.stdout:2084: finishing write tid 2 to smithi06431743-8 2024-03-04T06:16:18.375 INFO:tasks.rados.rados.0.smithi064.stdout:2084: finishing write tid 3 to smithi06431743-8 2024-03-04T06:16:18.378 INFO:tasks.rados.rados.0.smithi064.stdout:2084: finishing write tid 4 to smithi06431743-8 2024-03-04T06:16:18.380 INFO:tasks.rados.rados.0.smithi064.stdout:2084: finishing write tid 5 to smithi06431743-8 2024-03-04T06:16:18.381 INFO:tasks.rados.rados.0.smithi064.stdout:2084: oid 8 updating version 0 to 401 2024-03-04T06:16:18.381 INFO:tasks.rados.rados.0.smithi064.stdout:2084: oid 8 updating version 401 to 403 2024-03-04T06:16:18.381 INFO:tasks.rados.rados.0.smithi064.stdout:2084: oid 8 version 403 is already newer than 403 2024-03-04T06:16:18.381 INFO:tasks.rados.rados.0.smithi064.stdout:2084: oid 8 version 403 is already newer than 400 2024-03-04T06:16:18.381 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 403 (ObjNum 979 snap 245 seq_num 979) dirty exists 2024-03-04T06:16:18.381 INFO:tasks.rados.rados.0.smithi064.stdout:2084: left oid 8 (ObjNum 979 snap 245 seq_num 979) 2024-03-04T06:16:19.306 INFO:tasks.rados.rados.0.smithi064.stdout:2084: done (1 left) 2024-03-04T06:16:19.307 INFO:tasks.rados.rados.0.smithi064.stdout:2085: done (0 left) 2024-03-04T06:16:19.307 INFO:tasks.rados.rados.0.smithi064.stdout:2086: snap_remove snap 245 2024-03-04T06:16:19.779 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:16:19.779 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:16:19.780 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:16:19.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:16:19.836 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:19.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:19.899+0000 7f1b95d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:19.906+0000 7f1b95d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:19.926 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:19.928 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:19.931 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:19.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:16:19.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:19.951+0000 7f0098e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:19.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:19.956+0000 7f0098e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:20.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:20.038+0000 7fc79bf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:20.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:20.045+0000 7fc79bf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:20.061 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:20.062 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:20.063 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:20.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:16:20.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:20.206+0000 7fc4a6f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:20.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:20.213+0000 7fc4a6f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:20.227 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:20.229 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:20.230 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:20.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:16:20.308 INFO:tasks.rados.rados.0.smithi064.stdout:2086: done (0 left) 2024-03-04T06:16:20.308 INFO:tasks.rados.rados.0.smithi064.stdout:2087: read oid 5 snap -1 2024-03-04T06:16:20.308 INFO:tasks.rados.rados.0.smithi064.stdout:2087: expect (ObjNum 960 snap 241 seq_num 960) 2024-03-04T06:16:20.308 INFO:tasks.rados.rados.0.smithi064.stdout:2088: snap_create 2024-03-04T06:16:20.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:20.342+0000 7f762086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:20.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:20.352+0000 7f762086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:20.382 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:20.382 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:20.382 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:20.382 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:20.382 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:20.382 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:20.382 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:20.382 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:20.383 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:20.383 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:20.383 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:20.383 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:20.385 INFO:tasks.rados.rados.0.smithi064.stdout:2087: expect (ObjNum 960 snap 241 seq_num 960) 2024-03-04T06:16:20.385 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:20.386 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:21.313 INFO:tasks.rados.rados.0.smithi064.stdout:2087: done (1 left) 2024-03-04T06:16:21.313 INFO:tasks.rados.rados.0.smithi064.stdout:2088: done (0 left) 2024-03-04T06:16:21.313 INFO:tasks.rados.rados.0.smithi064.stdout:2089: read oid 31 snap 246 2024-03-04T06:16:21.313 INFO:tasks.rados.rados.0.smithi064.stdout:2089: expect (ObjNum 912 snap 224 seq_num 912) 2024-03-04T06:16:21.313 INFO:tasks.rados.rados.0.smithi064.stdout:2090: snap_create 2024-03-04T06:16:21.365 INFO:tasks.rados.rados.0.smithi064.stdout:2089: expect (ObjNum 912 snap 224 seq_num 912) 2024-03-04T06:16:22.029 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:22.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:22.146+0000 7fd40506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:22.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:22.151+0000 7fd40506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:22.313 INFO:tasks.rados.rados.0.smithi064.stdout:2089: done (1 left) 2024-03-04T06:16:22.313 INFO:tasks.rados.rados.0.smithi064.stdout:2090: done (0 left) 2024-03-04T06:16:22.313 INFO:tasks.rados.rados.0.smithi064.stdout:2091: snap_create 2024-03-04T06:16:23.256 INFO:tasks.rados.rados.0.smithi064.stdout:2091: done (0 left) 2024-03-04T06:16:23.257 INFO:tasks.rados.rados.0.smithi064.stdout:2092: delete oid 40 current snap is 249 2024-03-04T06:16:23.288 INFO:tasks.rados.rados.0.smithi064.stdout:2092: done (0 left) 2024-03-04T06:16:23.288 INFO:tasks.rados.rados.0.smithi064.stdout:2093: write oid 22 current snap is 249 2024-03-04T06:16:23.288 INFO:tasks.rados.rados.0.smithi064.stdout:2093: seq_num 981 ranges {487667=428806,1655659=770995} 2024-03-04T06:16:23.292 INFO:tasks.rados.rados.0.smithi064.stdout:2093: writing smithi06431743-22 from 487667 to 916473 tid 1 2024-03-04T06:16:23.301 INFO:tasks.rados.rados.0.smithi064.stdout:2093: writing smithi06431743-22 from 1655659 to 2426654 tid 2 2024-03-04T06:16:23.301 INFO:tasks.rados.rados.0.smithi064.stdout:2094: write (excl) oid 28 current snap is 249 2024-03-04T06:16:23.301 INFO:tasks.rados.rados.0.smithi064.stdout:2094: seq_num 982 ranges {504474=504055,1714141=631006} 2024-03-04T06:16:23.306 INFO:tasks.rados.rados.0.smithi064.stdout:2094: writing smithi06431743-28 from 504474 to 1008529 tid 1 2024-03-04T06:16:23.313 INFO:tasks.rados.rados.0.smithi064.stdout:2094: writing smithi06431743-28 from 1714141 to 2345147 tid 2 2024-03-04T06:16:23.314 INFO:tasks.rados.rados.0.smithi064.stdout:2095: write (excl) oid 16 current snap is 249 2024-03-04T06:16:23.314 INFO:tasks.rados.rados.0.smithi064.stdout:2095: seq_num 983 ranges {521281=579304,1656270=774664} 2024-03-04T06:16:23.319 INFO:tasks.rados.rados.0.smithi064.stdout:2095: writing smithi06431743-16 from 521281 to 1100585 tid 1 2024-03-04T06:16:23.327 INFO:tasks.rados.rados.0.smithi064.stdout:2095: writing smithi06431743-16 from 1656270 to 2430934 tid 2 2024-03-04T06:16:23.327 INFO:tasks.rados.rados.0.smithi064.stdout:2096: snap_create 2024-03-04T06:16:23.363 INFO:tasks.rados.rados.0.smithi064.stdout:2094: finishing write tid 1 to smithi06431743-28 2024-03-04T06:16:23.364 INFO:tasks.rados.rados.0.smithi064.stdout:2093: finishing write tid 1 to smithi06431743-22 2024-03-04T06:16:23.370 INFO:tasks.rados.rados.0.smithi064.stdout:2095: finishing write tid 1 to smithi06431743-16 2024-03-04T06:16:23.385 INFO:tasks.rados.rados.0.smithi064.stdout:2093: finishing write tid 2 to smithi06431743-22 2024-03-04T06:16:23.391 INFO:tasks.rados.rados.0.smithi064.stdout:2094: finishing write tid 2 to smithi06431743-28 2024-03-04T06:16:23.393 INFO:tasks.rados.rados.0.smithi064.stdout:2093: finishing write tid 3 to smithi06431743-22 2024-03-04T06:16:23.393 INFO:tasks.rados.rados.0.smithi064.stdout:2095: finishing write tid 2 to smithi06431743-16 2024-03-04T06:16:23.395 INFO:tasks.rados.rados.0.smithi064.stdout:2093: finishing write tid 4 to smithi06431743-22 2024-03-04T06:16:23.395 INFO:tasks.rados.rados.0.smithi064.stdout:2093: oid 22 updating version 0 to 699 2024-03-04T06:16:23.395 INFO:tasks.rados.rados.0.smithi064.stdout:2093: oid 22 updating version 699 to 700 2024-03-04T06:16:23.395 INFO:tasks.rados.rados.0.smithi064.stdout:2093: oid 22 version 700 is already newer than 698 2024-03-04T06:16:23.395 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 700 (ObjNum 981 snap 249 seq_num 981) dirty exists 2024-03-04T06:16:23.395 INFO:tasks.rados.rados.0.smithi064.stdout:2093: left oid 22 (ObjNum 981 snap 249 seq_num 981) 2024-03-04T06:16:23.401 INFO:tasks.rados.rados.0.smithi064.stdout:2095: finishing write tid 3 to smithi06431743-16 2024-03-04T06:16:23.402 INFO:tasks.rados.rados.0.smithi064.stdout:2095: finishing write tid 4 to smithi06431743-16 2024-03-04T06:16:23.402 INFO:tasks.rados.rados.0.smithi064.stdout:2095: oid 16 updating version 0 to 408 2024-03-04T06:16:23.402 INFO:tasks.rados.rados.0.smithi064.stdout:2095: oid 16 version 408 is already newer than 407 2024-03-04T06:16:23.403 INFO:tasks.rados.rados.0.smithi064.stdout:2095: oid 16 version 408 is already newer than 406 2024-03-04T06:16:23.403 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 408 (ObjNum 983 snap 249 seq_num 983) dirty exists 2024-03-04T06:16:23.403 INFO:tasks.rados.rados.0.smithi064.stdout:2095: left oid 16 (ObjNum 983 snap 249 seq_num 983) 2024-03-04T06:16:23.403 INFO:tasks.rados.rados.0.smithi064.stdout:2094: finishing write tid 3 to smithi06431743-28 2024-03-04T06:16:23.408 INFO:tasks.rados.rados.0.smithi064.stdout:2094: finishing write tid 4 to smithi06431743-28 2024-03-04T06:16:23.408 INFO:tasks.rados.rados.0.smithi064.stdout:2094: oid 28 updating version 0 to 820 2024-03-04T06:16:23.408 INFO:tasks.rados.rados.0.smithi064.stdout:2094: oid 28 version 820 is already newer than 818 2024-03-04T06:16:23.408 INFO:tasks.rados.rados.0.smithi064.stdout:2094: oid 28 version 820 is already newer than 819 2024-03-04T06:16:23.408 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 820 (ObjNum 982 snap 249 seq_num 982) dirty exists 2024-03-04T06:16:23.408 INFO:tasks.rados.rados.0.smithi064.stdout:2094: left oid 28 (ObjNum 982 snap 249 seq_num 982) 2024-03-04T06:16:24.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:24.321 INFO:tasks.rados.rados.0.smithi064.stdout:2093: done (3 left) 2024-03-04T06:16:24.321 INFO:tasks.rados.rados.0.smithi064.stdout:2094: done (2 left) 2024-03-04T06:16:24.321 INFO:tasks.rados.rados.0.smithi064.stdout:2095: done (1 left) 2024-03-04T06:16:24.321 INFO:tasks.rados.rados.0.smithi064.stdout:2096: done (0 left) 2024-03-04T06:16:24.321 INFO:tasks.rados.rados.0.smithi064.stdout:2097: read oid 13 snap -1 2024-03-04T06:16:24.321 INFO:tasks.rados.rados.0.smithi064.stdout:2097: expect deleted 2024-03-04T06:16:24.321 INFO:tasks.rados.rados.0.smithi064.stdout:2098: delete oid 8 current snap is 250 2024-03-04T06:16:24.343 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:24.341+0000 7f90e6d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:24.346+0000 7f90e6d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:24.371 INFO:tasks.rados.rados.0.smithi064.stdout:2097: done (1 left) 2024-03-04T06:16:24.371 INFO:tasks.rados.rados.0.smithi064.stdout:2098: done (0 left) 2024-03-04T06:16:24.371 INFO:tasks.rados.rados.0.smithi064.stdout:2099: read oid 33 snap -1 2024-03-04T06:16:24.371 INFO:tasks.rados.rados.0.smithi064.stdout:2099: expect (ObjNum 965 snap 241 seq_num 965) 2024-03-04T06:16:24.371 INFO:tasks.rados.rados.0.smithi064.stdout:2100: snap_create 2024-03-04T06:16:24.386 INFO:tasks.rados.rados.0.smithi064.stdout:2099: expect (ObjNum 965 snap 241 seq_num 965) 2024-03-04T06:16:25.326 INFO:tasks.rados.rados.0.smithi064.stdout:2099: done (1 left) 2024-03-04T06:16:25.326 INFO:tasks.rados.rados.0.smithi064.stdout:2100: done (0 left) 2024-03-04T06:16:25.326 INFO:tasks.rados.rados.0.smithi064.stdout:2101: write (excl) oid 41 current snap is 251 2024-03-04T06:16:25.326 INFO:tasks.rados.rados.0.smithi064.stdout:2101: seq_num 985 ranges {554895=729802,1773234=778333} 2024-03-04T06:16:25.333 INFO:tasks.rados.rados.0.smithi064.stdout:2101: writing smithi06431743-41 from 554895 to 1284697 tid 1 2024-03-04T06:16:25.342 INFO:tasks.rados.rados.0.smithi064.stdout:2101: writing smithi06431743-41 from 1773234 to 2551567 tid 2 2024-03-04T06:16:25.342 INFO:tasks.rados.rados.0.smithi064.stdout:2102: write (excl) oid 6 current snap is 251 2024-03-04T06:16:25.342 INFO:tasks.rados.rados.0.smithi064.stdout:2102: seq_num 986 ranges {571702=405051,1431716=638344,2560365=11337} 2024-03-04T06:16:25.348 INFO:tasks.rados.rados.0.smithi064.stdout:2102: writing smithi06431743-6 from 571702 to 976753 tid 1 2024-03-04T06:16:25.355 INFO:tasks.rados.rados.0.smithi064.stdout:2102: writing smithi06431743-6 from 1431716 to 2070060 tid 2 2024-03-04T06:16:25.359 INFO:tasks.rados.rados.0.smithi064.stdout:2102: writing smithi06431743-6 from 2560365 to 2571702 tid 3 2024-03-04T06:16:25.359 INFO:tasks.rados.rados.0.smithi064.stdout:2103: read oid 38 snap -1 2024-03-04T06:16:25.360 INFO:tasks.rados.rados.0.smithi064.stdout:2103: expect (ObjNum 835 snap 204 seq_num 835) 2024-03-04T06:16:25.360 INFO:tasks.rados.rados.0.smithi064.stdout:2104: write oid 5 current snap is 251 2024-03-04T06:16:25.360 INFO:tasks.rados.rados.0.smithi064.stdout:2104: seq_num 987 ranges {588509=480300,1773845=782002} 2024-03-04T06:16:25.366 INFO:tasks.rados.rados.0.smithi064.stdout:2104: writing smithi06431743-5 from 588509 to 1068809 tid 1 2024-03-04T06:16:25.374 INFO:tasks.rados.rados.0.smithi064.stdout:2104: writing smithi06431743-5 from 1773845 to 2555847 tid 2 2024-03-04T06:16:25.375 INFO:tasks.rados.rados.0.smithi064.stdout:2105: snap_create 2024-03-04T06:16:25.386 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:16:25.386 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:16:25.386 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:16:25.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:16:25.399 INFO:tasks.rados.rados.0.smithi064.stdout:2101: finishing write tid 1 to smithi06431743-41 2024-03-04T06:16:25.405 INFO:tasks.rados.rados.0.smithi064.stdout:2102: finishing write tid 1 to smithi06431743-6 2024-03-04T06:16:25.423 INFO:tasks.rados.rados.0.smithi064.stdout:2101: finishing write tid 2 to smithi06431743-41 2024-03-04T06:16:25.424 INFO:tasks.rados.rados.0.smithi064.stdout:2102: finishing write tid 2 to smithi06431743-6 2024-03-04T06:16:25.428 INFO:tasks.rados.rados.0.smithi064.stdout:2103: expect (ObjNum 835 snap 204 seq_num 835) 2024-03-04T06:16:25.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:25.511+0000 7f610be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:25.520 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:25.519+0000 7f610be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:25.532 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:25.532 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:25.532 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:25.532 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:25.532 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:25.532 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:25.533 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:25.533 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:25.533 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:25.533 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:25.533 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:25.533 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:25.535 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:25.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:16:25.548 INFO:tasks.rados.rados.0.smithi064.stdout:2102: finishing write tid 3 to smithi06431743-6 2024-03-04T06:16:25.548 INFO:tasks.rados.rados.0.smithi064.stdout:2102: finishing write tid 4 to smithi06431743-6 2024-03-04T06:16:25.549 INFO:tasks.rados.rados.0.smithi064.stdout:2102: finishing write tid 5 to smithi06431743-6 2024-03-04T06:16:25.549 INFO:tasks.rados.rados.0.smithi064.stdout:2102: oid 6 updating version 0 to 390 2024-03-04T06:16:25.549 INFO:tasks.rados.rados.0.smithi064.stdout:2102: oid 6 version 390 is already newer than 390 2024-03-04T06:16:25.549 INFO:tasks.rados.rados.0.smithi064.stdout:2102: oid 6 version 390 is already newer than 388 2024-03-04T06:16:25.549 INFO:tasks.rados.rados.0.smithi064.stdout:2102: oid 6 version 390 is already newer than 387 2024-03-04T06:16:25.549 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 390 (ObjNum 986 snap 251 seq_num 986) dirty exists 2024-03-04T06:16:25.549 INFO:tasks.rados.rados.0.smithi064.stdout:2102: left oid 6 (ObjNum 986 snap 251 seq_num 986) 2024-03-04T06:16:25.549 INFO:tasks.rados.rados.0.smithi064.stdout:2104: finishing write tid 1 to smithi06431743-5 2024-03-04T06:16:25.549 INFO:tasks.rados.rados.0.smithi064.stdout:2104: finishing write tid 2 to smithi06431743-5 2024-03-04T06:16:25.549 INFO:tasks.rados.rados.0.smithi064.stdout:2101: finishing write tid 3 to smithi06431743-41 2024-03-04T06:16:25.549 INFO:tasks.rados.rados.0.smithi064.stdout:2101: finishing write tid 4 to smithi06431743-41 2024-03-04T06:16:25.550 INFO:tasks.rados.rados.0.smithi064.stdout:2101: oid 41 updating version 0 to 479 2024-03-04T06:16:25.550 INFO:tasks.rados.rados.0.smithi064.stdout:2101: oid 41 version 479 is already newer than 479 2024-03-04T06:16:25.550 INFO:tasks.rados.rados.0.smithi064.stdout:2101: oid 41 version 479 is already newer than 477 2024-03-04T06:16:25.550 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 479 (ObjNum 985 snap 251 seq_num 985) dirty exists 2024-03-04T06:16:25.550 INFO:tasks.rados.rados.0.smithi064.stdout:2101: left oid 41 (ObjNum 985 snap 251 seq_num 985) 2024-03-04T06:16:25.550 INFO:tasks.rados.rados.0.smithi064.stdout:2104: finishing write tid 3 to smithi06431743-5 2024-03-04T06:16:25.550 INFO:tasks.rados.rados.0.smithi064.stdout:2104: finishing write tid 4 to smithi06431743-5 2024-03-04T06:16:25.550 INFO:tasks.rados.rados.0.smithi064.stdout:2104: oid 5 updating version 0 to 898 2024-03-04T06:16:25.550 INFO:tasks.rados.rados.0.smithi064.stdout:2104: oid 5 version 898 is already newer than 897 2024-03-04T06:16:25.550 INFO:tasks.rados.rados.0.smithi064.stdout:2104: oid 5 version 898 is already newer than 897 2024-03-04T06:16:25.550 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 898 (ObjNum 987 snap 251 seq_num 987) dirty exists 2024-03-04T06:16:25.551 INFO:tasks.rados.rados.0.smithi064.stdout:2104: left oid 5 (ObjNum 987 snap 251 seq_num 987) 2024-03-04T06:16:25.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:25.653+0000 7fb47166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:25.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:25.661+0000 7fb47166d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:25.673 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:25.673 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:25.674 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:25.674 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:25.674 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:25.674 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:25.674 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:25.674 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:25.674 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:25.674 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:25.674 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:25.674 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:25.677 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:25.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:16:25.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:25.779+0000 7fe4c731d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:25.787 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:25.787+0000 7fe4c731d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:25.799 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:25.800 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:25.803 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:25.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:16:25.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:25.906+0000 7f90eb224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:25.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:25.913+0000 7f90eb224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:25.927 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:25.930 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:26.326 INFO:tasks.rados.rados.0.smithi064.stdout:2101: done (4 left) 2024-03-04T06:16:26.326 INFO:tasks.rados.rados.0.smithi064.stdout:2102: done (3 left) 2024-03-04T06:16:26.326 INFO:tasks.rados.rados.0.smithi064.stdout:2103: done (2 left) 2024-03-04T06:16:26.326 INFO:tasks.rados.rados.0.smithi064.stdout:2104: done (1 left) 2024-03-04T06:16:26.326 INFO:tasks.rados.rados.0.smithi064.stdout:2105: done (0 left) 2024-03-04T06:16:26.327 INFO:tasks.rados.rados.0.smithi064.stdout:2106: snap_create 2024-03-04T06:16:26.433 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:26.539 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:26.537+0000 7fce728ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:26.542 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:26.542+0000 7fce728ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:27.327 INFO:tasks.rados.rados.0.smithi064.stdout:2106: done (0 left) 2024-03-04T06:16:27.327 INFO:tasks.rados.rados.0.smithi064.stdout:2107: write (excl) oid 37 current snap is 253 2024-03-04T06:16:27.327 INFO:tasks.rados.rados.0.smithi064.stdout:2107: seq_num 988 ranges {605316=555549,1832327=642013} 2024-03-04T06:16:27.334 INFO:tasks.rados.rados.0.smithi064.stdout:2107: writing smithi06431743-37 from 605316 to 1160865 tid 1 2024-03-04T06:16:27.341 INFO:tasks.rados.rados.0.smithi064.stdout:2107: writing smithi06431743-37 from 1832327 to 2474340 tid 2 2024-03-04T06:16:27.341 INFO:tasks.rados.rados.0.smithi064.stdout:2108: write (excl) oid 33 current snap is 253 2024-03-04T06:16:27.341 INFO:tasks.rados.rados.0.smithi064.stdout:2108: seq_num 989 ranges {622123=747151,2007162=614961} 2024-03-04T06:16:27.348 INFO:tasks.rados.rados.0.smithi064.stdout:2108: writing smithi06431743-33 from 622123 to 1369274 tid 1 2024-03-04T06:16:27.355 INFO:tasks.rados.rados.0.smithi064.stdout:2108: writing smithi06431743-33 from 2007162 to 2622123 tid 2 2024-03-04T06:16:27.356 INFO:tasks.rados.rados.0.smithi064.stdout:2109: delete oid 22 current snap is 253 2024-03-04T06:16:27.397 INFO:tasks.rados.rados.0.smithi064.stdout:2108: finishing write tid 1 to smithi06431743-33 2024-03-04T06:16:27.402 INFO:tasks.rados.rados.0.smithi064.stdout:2107: finishing write tid 1 to smithi06431743-37 2024-03-04T06:16:27.414 INFO:tasks.rados.rados.0.smithi064.stdout:2108: finishing write tid 2 to smithi06431743-33 2024-03-04T06:16:27.422 INFO:tasks.rados.rados.0.smithi064.stdout:2107: finishing write tid 2 to smithi06431743-37 2024-03-04T06:16:27.428 INFO:tasks.rados.rados.0.smithi064.stdout:2108: finishing write tid 3 to smithi06431743-33 2024-03-04T06:16:27.430 INFO:tasks.rados.rados.0.smithi064.stdout:2108: finishing write tid 4 to smithi06431743-33 2024-03-04T06:16:27.430 INFO:tasks.rados.rados.0.smithi064.stdout:2108: oid 33 updating version 0 to 704 2024-03-04T06:16:27.431 INFO:tasks.rados.rados.0.smithi064.stdout:2108: oid 33 version 704 is already newer than 703 2024-03-04T06:16:27.431 INFO:tasks.rados.rados.0.smithi064.stdout:2108: oid 33 version 704 is already newer than 702 2024-03-04T06:16:27.431 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 704 (ObjNum 989 snap 253 seq_num 989) dirty exists 2024-03-04T06:16:27.431 INFO:tasks.rados.rados.0.smithi064.stdout:2108: left oid 33 (ObjNum 989 snap 253 seq_num 989) 2024-03-04T06:16:27.434 INFO:tasks.rados.rados.0.smithi064.stdout:2107: finishing write tid 3 to smithi06431743-37 2024-03-04T06:16:27.438 INFO:tasks.rados.rados.0.smithi064.stdout:2107: finishing write tid 4 to smithi06431743-37 2024-03-04T06:16:27.438 INFO:tasks.rados.rados.0.smithi064.stdout:2107: oid 37 updating version 0 to 319 2024-03-04T06:16:27.438 INFO:tasks.rados.rados.0.smithi064.stdout:2107: oid 37 updating version 319 to 320 2024-03-04T06:16:27.438 INFO:tasks.rados.rados.0.smithi064.stdout:2107: oid 37 version 320 is already newer than 318 2024-03-04T06:16:27.438 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 320 (ObjNum 988 snap 253 seq_num 988) dirty exists 2024-03-04T06:16:27.438 INFO:tasks.rados.rados.0.smithi064.stdout:2107: left oid 37 (ObjNum 988 snap 253 seq_num 988) 2024-03-04T06:16:27.445 INFO:tasks.rados.rados.0.smithi064.stdout:2107: done (2 left) 2024-03-04T06:16:27.446 INFO:tasks.rados.rados.0.smithi064.stdout:2108: done (1 left) 2024-03-04T06:16:27.446 INFO:tasks.rados.rados.0.smithi064.stdout:2109: done (0 left) 2024-03-04T06:16:27.446 INFO:tasks.rados.rados.0.smithi064.stdout:2110: delete oid 7 current snap is 253 2024-03-04T06:16:27.473 INFO:tasks.rados.rados.0.smithi064.stdout:2110: done (0 left) 2024-03-04T06:16:27.473 INFO:tasks.rados.rados.0.smithi064.stdout:2111: delete oid 32 current snap is 253 2024-03-04T06:16:27.478 INFO:tasks.rados.rados.0.smithi064.stdout:2111: done (0 left) 2024-03-04T06:16:27.478 INFO:tasks.rados.rados.0.smithi064.stdout:2112: snap_create 2024-03-04T06:16:28.334 INFO:tasks.rados.rados.0.smithi064.stdout:2112: done (0 left) 2024-03-04T06:16:28.334 INFO:tasks.rados.rados.0.smithi064.stdout:2113: write (excl) oid 44 current snap is 254 2024-03-04T06:16:28.335 INFO:tasks.rados.rados.0.smithi064.stdout:2113: seq_num 993 ranges {689351=648147,2124737=564614} 2024-03-04T06:16:28.341 INFO:tasks.rados.rados.0.smithi064.stdout:2113: writing smithi06431743-44 from 689351 to 1337498 tid 1 2024-03-04T06:16:28.350 INFO:tasks.rados.rados.0.smithi064.stdout:2113: writing smithi06431743-44 from 2124737 to 2689351 tid 2 2024-03-04T06:16:28.350 INFO:tasks.rados.rados.0.smithi064.stdout:2114: snap_remove snap 248 2024-03-04T06:16:28.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:28.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:28.719+0000 7f7c89a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:28.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:28.724+0000 7f7c89a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:29.344 INFO:tasks.rados.rados.0.smithi064.stdout:2114: done (1 left) 2024-03-04T06:16:29.344 INFO:tasks.rados.rados.0.smithi064.stdout:2115: write oid 42 current snap is 254 2024-03-04T06:16:29.345 INFO:tasks.rados.rados.0.smithi064.stdout:2115: seq_num 994 ranges {706158=723396,2183219=522939} 2024-03-04T06:16:29.352 INFO:tasks.rados.rados.0.smithi064.stdout:2115: writing smithi06431743-42 from 706158 to 1429554 tid 1 2024-03-04T06:16:29.360 INFO:tasks.rados.rados.0.smithi064.stdout:2115: writing smithi06431743-42 from 2183219 to 2706158 tid 2 2024-03-04T06:16:29.360 INFO:tasks.rados.rados.0.smithi064.stdout:2116: read oid 21 snap 249 2024-03-04T06:16:29.360 INFO:tasks.rados.rados.0.smithi064.stdout:2116: expect (ObjNum 972 snap 244 seq_num 972) 2024-03-04T06:16:29.361 INFO:tasks.rados.rados.0.smithi064.stdout:2117: delete oid 7 current snap is 254 2024-03-04T06:16:29.361 INFO:tasks.rados.rados.0.smithi064.stdout:2113: finishing write tid 1 to smithi06431743-44 2024-03-04T06:16:29.361 INFO:tasks.rados.rados.0.smithi064.stdout:2113: finishing write tid 2 to smithi06431743-44 2024-03-04T06:16:29.361 INFO:tasks.rados.rados.0.smithi064.stdout:2113: finishing write tid 3 to smithi06431743-44 2024-03-04T06:16:29.361 INFO:tasks.rados.rados.0.smithi064.stdout:2113: finishing write tid 4 to smithi06431743-44 2024-03-04T06:16:29.361 INFO:tasks.rados.rados.0.smithi064.stdout:2113: oid 44 updating version 0 to 467 2024-03-04T06:16:29.361 INFO:tasks.rados.rados.0.smithi064.stdout:2113: oid 44 updating version 467 to 469 2024-03-04T06:16:29.361 INFO:tasks.rados.rados.0.smithi064.stdout:2113: oid 44 version 469 is already newer than 468 2024-03-04T06:16:29.361 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 469 (ObjNum 993 snap 254 seq_num 993) dirty exists 2024-03-04T06:16:29.361 INFO:tasks.rados.rados.0.smithi064.stdout:2113: left oid 44 (ObjNum 993 snap 254 seq_num 993) 2024-03-04T06:16:29.416 INFO:tasks.rados.rados.0.smithi064.stdout:2113: done (3 left) 2024-03-04T06:16:29.416 INFO:tasks.rados.rados.0.smithi064.stdout:2117: done (2 left) 2024-03-04T06:16:29.416 INFO:tasks.rados.rados.0.smithi064.stdout:2118: snap_remove snap 252 2024-03-04T06:16:30.344 INFO:tasks.rados.rados.0.smithi064.stdout:2118: done (2 left) 2024-03-04T06:16:30.344 INFO:tasks.rados.rados.0.smithi064.stdout:2119: read oid 35 snap -1 2024-03-04T06:16:30.344 INFO:tasks.rados.rados.0.smithi064.stdout:2119: expect deleted 2024-03-04T06:16:30.344 INFO:tasks.rados.rados.0.smithi064.stdout:2120: write (excl) oid 16 current snap is 254 2024-03-04T06:16:30.344 INFO:tasks.rados.rados.0.smithi064.stdout:2120: seq_num 996 ranges {739772=590247,1900183=540336} 2024-03-04T06:16:30.351 INFO:tasks.rados.rados.0.smithi064.stdout:2120: writing smithi06431743-16 from 739772 to 1330019 tid 1 2024-03-04T06:16:30.357 INFO:tasks.rados.rados.0.smithi064.stdout:2120: writing smithi06431743-16 from 1900183 to 2440519 tid 2 2024-03-04T06:16:30.357 INFO:tasks.rados.rados.0.smithi064.stdout:2121: snap_create 2024-03-04T06:16:30.357 INFO:tasks.rados.rados.0.smithi064.stdout:2116: expect (ObjNum 972 snap 244 seq_num 972) 2024-03-04T06:16:30.451 INFO:tasks.rados.rados.0.smithi064.stdout:2115: finishing write tid 1 to smithi06431743-42 2024-03-04T06:16:30.451 INFO:tasks.rados.rados.0.smithi064.stdout:2115: finishing write tid 2 to smithi06431743-42 2024-03-04T06:16:30.451 INFO:tasks.rados.rados.0.smithi064.stdout:2115: finishing write tid 3 to smithi06431743-42 2024-03-04T06:16:30.451 INFO:tasks.rados.rados.0.smithi064.stdout:2115: finishing write tid 4 to smithi06431743-42 2024-03-04T06:16:30.452 INFO:tasks.rados.rados.0.smithi064.stdout:2115: oid 42 updating version 0 to 438 2024-03-04T06:16:30.452 INFO:tasks.rados.rados.0.smithi064.stdout:2115: oid 42 updating version 438 to 439 2024-03-04T06:16:30.452 INFO:tasks.rados.rados.0.smithi064.stdout:2115: oid 42 version 439 is already newer than 438 2024-03-04T06:16:30.452 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 439 (ObjNum 994 snap 254 seq_num 994) dirty exists 2024-03-04T06:16:30.452 INFO:tasks.rados.rados.0.smithi064.stdout:2115: left oid 42 (ObjNum 994 snap 254 seq_num 994) 2024-03-04T06:16:30.475 INFO:tasks.rados.rados.0.smithi064.stdout:2120: finishing write tid 1 to smithi06431743-16 2024-03-04T06:16:30.487 INFO:tasks.rados.rados.0.smithi064.stdout:2120: finishing write tid 2 to smithi06431743-16 2024-03-04T06:16:30.493 INFO:tasks.rados.rados.0.smithi064.stdout:2120: finishing write tid 3 to smithi06431743-16 2024-03-04T06:16:30.494 INFO:tasks.rados.rados.0.smithi064.stdout:2120: finishing write tid 4 to smithi06431743-16 2024-03-04T06:16:30.494 INFO:tasks.rados.rados.0.smithi064.stdout:2120: oid 16 updating version 0 to 416 2024-03-04T06:16:30.494 INFO:tasks.rados.rados.0.smithi064.stdout:2120: oid 16 version 416 is already newer than 414 2024-03-04T06:16:30.495 INFO:tasks.rados.rados.0.smithi064.stdout:2120: oid 16 version 416 is already newer than 415 2024-03-04T06:16:30.495 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 416 (ObjNum 996 snap 254 seq_num 996) dirty exists 2024-03-04T06:16:30.495 INFO:tasks.rados.rados.0.smithi064.stdout:2120: left oid 16 (ObjNum 996 snap 254 seq_num 996) 2024-03-04T06:16:30.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:30.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:30.898+0000 7f103f02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:30.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:30.902+0000 7f103f02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:30.931 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:16:30.931 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:16:30.931 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:16:30.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:16:31.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:31.034+0000 7f245c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:31.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:31.041+0000 7f245c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:31.055 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:31.058 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:31.058 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:31.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:16:31.166 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:31.160+0000 7fb85642d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:31.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:31.167+0000 7fb85642d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:31.184 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:31.184 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:31.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:16:31.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:31.286+0000 7f34fa22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:31.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:31.293+0000 7f34fa22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:31.308 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:31.308 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:31.308 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:31.308 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:31.308 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:31.308 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:31.308 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:31.308 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:31.308 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:31.308 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:31.309 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:31.309 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:31.311 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:31.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:16:31.348 INFO:tasks.rados.rados.0.smithi064.stdout:2115: done (4 left) 2024-03-04T06:16:31.349 INFO:tasks.rados.rados.0.smithi064.stdout:2116: done (3 left) 2024-03-04T06:16:31.349 INFO:tasks.rados.rados.0.smithi064.stdout:2119: done (2 left) 2024-03-04T06:16:31.349 INFO:tasks.rados.rados.0.smithi064.stdout:2120: done (1 left) 2024-03-04T06:16:31.349 INFO:tasks.rados.rados.0.smithi064.stdout:2121: done (0 left) 2024-03-04T06:16:31.349 INFO:tasks.rados.rados.0.smithi064.stdout:2122: delete oid 3 current snap is 255 2024-03-04T06:16:31.397 INFO:tasks.rados.rados.0.smithi064.stdout:2122: done (0 left) 2024-03-04T06:16:31.397 INFO:tasks.rados.rados.0.smithi064.stdout:2123: snap_create 2024-03-04T06:16:31.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:31.428+0000 7f12add14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:31.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:31.436+0000 7f12add14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:31.451 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:31.453 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:31.454 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:32.352 INFO:tasks.rados.rados.0.smithi064.stdout:2123: done (0 left) 2024-03-04T06:16:32.352 INFO:tasks.rados.rados.0.smithi064.stdout:2124: read oid 1 snap -1 2024-03-04T06:16:32.352 INFO:tasks.rados.rados.0.smithi064.stdout:2124: expect (ObjNum 935 snap 231 seq_num 935) 2024-03-04T06:16:32.352 INFO:tasks.rados.rados.0.smithi064.stdout:2125: snap_create 2024-03-04T06:16:32.434 INFO:tasks.rados.rados.0.smithi064.stdout:2124: expect (ObjNum 935 snap 231 seq_num 935) 2024-03-04T06:16:32.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:33.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:33.076+0000 7fdc2a0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:33.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:33.081+0000 7fdc2a0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:33.260 INFO:tasks.rados.rados.0.smithi064.stdout:2124: done (1 left) 2024-03-04T06:16:33.260 INFO:tasks.rados.rados.0.smithi064.stdout:2125: done (0 left) 2024-03-04T06:16:33.260 INFO:tasks.rados.rados.0.smithi064.stdout:2126: snap_create 2024-03-04T06:16:34.259 INFO:tasks.rados.rados.0.smithi064.stdout:2126: done (0 left) 2024-03-04T06:16:34.259 INFO:tasks.rados.rados.0.smithi064.stdout:2127: read oid 34 snap -1 2024-03-04T06:16:34.259 INFO:tasks.rados.rados.0.smithi064.stdout:2127: expect (ObjNum 974 snap 244 seq_num 974) 2024-03-04T06:16:34.259 INFO:tasks.rados.rados.0.smithi064.stdout:2128: read oid 33 snap -1 2024-03-04T06:16:34.259 INFO:tasks.rados.rados.0.smithi064.stdout:2128: expect (ObjNum 989 snap 253 seq_num 989) 2024-03-04T06:16:34.259 INFO:tasks.rados.rados.0.smithi064.stdout:2129: write (excl) oid 20 current snap is 258 2024-03-04T06:16:34.259 INFO:tasks.rados.rados.0.smithi064.stdout:2129: seq_num 998 ranges {773386=740745,2017147=544005} 2024-03-04T06:16:34.267 INFO:tasks.rados.rados.0.smithi064.stdout:2129: writing smithi06431743-20 from 773386 to 1514131 tid 1 2024-03-04T06:16:34.273 INFO:tasks.rados.rados.0.smithi064.stdout:2129: writing smithi06431743-20 from 2017147 to 2561152 tid 2 2024-03-04T06:16:34.273 INFO:tasks.rados.rados.0.smithi064.stdout:2130: read oid 25 snap -1 2024-03-04T06:16:34.273 INFO:tasks.rados.rados.0.smithi064.stdout:2130: expect (ObjNum 943 snap 232 seq_num 943) 2024-03-04T06:16:34.273 INFO:tasks.rados.rados.0.smithi064.stdout:2131: snap_remove snap 255 2024-03-04T06:16:35.151 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:35.263 INFO:tasks.rados.rados.0.smithi064.stdout:2131: done (4 left) 2024-03-04T06:16:35.263 INFO:tasks.rados.rados.0.smithi064.stdout:2132: delete oid 19 current snap is 258 2024-03-04T06:16:35.263 INFO:tasks.rados.rados.0.smithi064.stdout:2128: expect (ObjNum 989 snap 253 seq_num 989) 2024-03-04T06:16:35.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:35.269+0000 7f678fc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:35.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:35.274+0000 7f678fc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:35.420 INFO:tasks.rados.rados.0.smithi064.stdout:2127: expect (ObjNum 974 snap 244 seq_num 974) 2024-03-04T06:16:35.509 INFO:tasks.rados.rados.0.smithi064.stdout:2130: expect (ObjNum 943 snap 232 seq_num 943) 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2129: finishing write tid 1 to smithi06431743-20 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2129: finishing write tid 2 to smithi06431743-20 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2129: finishing write tid 3 to smithi06431743-20 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2129: finishing write tid 4 to smithi06431743-20 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2129: oid 20 updating version 0 to 571 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2129: oid 20 version 571 is already newer than 571 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2129: oid 20 version 571 is already newer than 569 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 571 (ObjNum 998 snap 258 seq_num 998) dirty exists 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2129: left oid 20 (ObjNum 998 snap 258 seq_num 998) 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2127: done (4 left) 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2128: done (3 left) 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2129: done (2 left) 2024-03-04T06:16:35.698 INFO:tasks.rados.rados.0.smithi064.stdout:2130: done (1 left) 2024-03-04T06:16:35.699 INFO:tasks.rados.rados.0.smithi064.stdout:2132: done (0 left) 2024-03-04T06:16:35.699 INFO:tasks.rados.rados.0.smithi064.stdout:2133: snap_create 2024-03-04T06:16:36.264 INFO:tasks.rados.rados.0.smithi064.stdout:2133: done (0 left) 2024-03-04T06:16:36.264 INFO:tasks.rados.rados.0.smithi064.stdout:2134: read oid 6 snap -1 2024-03-04T06:16:36.264 INFO:tasks.rados.rados.0.smithi064.stdout:2134: expect (ObjNum 986 snap 251 seq_num 986) 2024-03-04T06:16:36.264 INFO:tasks.rados.rados.0.smithi064.stdout:2135: delete oid 24 current snap is 259 2024-03-04T06:16:36.311 INFO:tasks.rados.rados.0.smithi064.stdout:2134: expect (ObjNum 986 snap 251 seq_num 986) 2024-03-04T06:16:36.449 INFO:tasks.rados.rados.0.smithi064.stdout:2134: done (1 left) 2024-03-04T06:16:36.449 INFO:tasks.rados.rados.0.smithi064.stdout:2135: done (0 left) 2024-03-04T06:16:36.450 INFO:tasks.rados.rados.0.smithi064.stdout:2136: read oid 9 snap 251 2024-03-04T06:16:36.450 INFO:tasks.rados.rados.0.smithi064.stdout:2136: expect (ObjNum 963 snap 241 seq_num 963) 2024-03-04T06:16:36.450 INFO:tasks.rados.rados.0.smithi064.stdout:2137: write (excl) oid 33 current snap is 259 2024-03-04T06:16:36.450 INFO:tasks.rados.rados.0.smithi064.stdout:2137: seq_num 1001 ranges {423807=566492,1676240=407685,2739004=84803} 2024-03-04T06:16:36.454 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:16:36.454 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:16:36.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:16:36.455 INFO:tasks.rados.rados.0.smithi064.stdout:2137: writing smithi06431743-33 from 423807 to 990299 tid 1 2024-03-04T06:16:36.461 INFO:tasks.rados.rados.0.smithi064.stdout:2137: writing smithi06431743-33 from 1676240 to 2083925 tid 2 2024-03-04T06:16:36.465 INFO:tasks.rados.rados.0.smithi064.stdout:2137: writing smithi06431743-33 from 2739004 to 2823807 tid 3 2024-03-04T06:16:36.465 INFO:tasks.rados.rados.0.smithi064.stdout:2138: delete oid 46 current snap is 259 2024-03-04T06:16:36.472 INFO:tasks.rados.rados.0.smithi064.stdout:2136: expect (ObjNum 963 snap 241 seq_num 963) 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2137: finishing write tid 1 to smithi06431743-33 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2137: finishing write tid 2 to smithi06431743-33 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2137: finishing write tid 3 to smithi06431743-33 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2137: finishing write tid 4 to smithi06431743-33 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2137: finishing write tid 5 to smithi06431743-33 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2137: oid 33 updating version 0 to 714 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2137: oid 33 updating version 714 to 716 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2137: oid 33 updating version 716 to 717 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2137: oid 33 version 717 is already newer than 716 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 717 (ObjNum 1001 snap 259 seq_num 1001) dirty exists 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2137: left oid 33 (ObjNum 1001 snap 259 seq_num 1001) 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2136: done (2 left) 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2137: done (1 left) 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2138: done (0 left) 2024-03-04T06:16:36.555 INFO:tasks.rados.rados.0.smithi064.stdout:2139: read oid 7 snap 256 2024-03-04T06:16:36.556 INFO:tasks.rados.rados.0.smithi064.stdout:2139: expect deleted 2024-03-04T06:16:36.556 INFO:tasks.rados.rados.0.smithi064.stdout:2140: write oid 14 current snap is 259 2024-03-04T06:16:36.556 INFO:tasks.rados.rados.0.smithi064.stdout:2140: seq_num 1003 ranges {457421=716990,1676851=411354,2677497=179924} 2024-03-04T06:16:36.561 INFO:tasks.rados.rados.0.smithi064.stdout:2140: writing smithi06431743-14 from 457421 to 1174411 tid 1 2024-03-04T06:16:36.566 INFO:tasks.rados.rados.0.smithi064.stdout:2140: writing smithi06431743-14 from 1676851 to 2088205 tid 2 2024-03-04T06:16:36.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:36.565+0000 7f4be0a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:36.571 INFO:tasks.rados.rados.0.smithi064.stdout:2140: writing smithi06431743-14 from 2677497 to 2857421 tid 3 2024-03-04T06:16:36.571 INFO:tasks.rados.rados.0.smithi064.stdout:2141: write oid 40 current snap is 259 2024-03-04T06:16:36.571 INFO:tasks.rados.rados.0.smithi064.stdout:2141: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2024-03-04T06:16:36.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:36.572+0000 7f4be0a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:36.577 INFO:tasks.rados.rados.0.smithi064.stdout:2141: writing smithi06431743-40 from 474228 to 982820 tid 1 2024-03-04T06:16:36.584 INFO:tasks.rados.rados.0.smithi064.stdout:2141: writing smithi06431743-40 from 1451686 to 2006698 tid 2 2024-03-04T06:16:36.589 INFO:tasks.rados.rados.0.smithi064.stdout:2141: writing smithi06431743-40 from 2704920 to 2874228 tid 3 2024-03-04T06:16:36.589 INFO:tasks.rados.rados.0.smithi064.stdout:2139: done (2 left) 2024-03-04T06:16:36.589 INFO:tasks.rados.rados.0.smithi064.stdout:2142: write oid 10 current snap is 259 2024-03-04T06:16:36.589 INFO:tasks.rados.rados.0.smithi064.stdout:2142: seq_num 1005 ranges {491035=583841,1510168=698670,2732343=158692} 2024-03-04T06:16:36.595 INFO:tasks.rados.rados.0.smithi064.stdout:2142: writing smithi06431743-10 from 491035 to 1074876 tid 1 2024-03-04T06:16:36.603 INFO:tasks.rados.rados.0.smithi064.stdout:2142: writing smithi06431743-10 from 1510168 to 2208838 tid 2 2024-03-04T06:16:36.608 INFO:tasks.rados.rados.0.smithi064.stdout:2142: writing smithi06431743-10 from 2732343 to 2891035 tid 3 2024-03-04T06:16:36.608 INFO:tasks.rados.rados.0.smithi064.stdout:2143: read oid 8 snap -1 2024-03-04T06:16:36.608 INFO:tasks.rados.rados.0.smithi064.stdout:2143: expect deleted 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2144: read oid 7 snap -1 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2144: expect deleted 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2145: read oid 46 snap 257 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2145: expect (ObjNum 950 snap 236 seq_num 950) 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2146: read oid 24 snap -1 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2146: expect deleted 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2147: read oid 28 snap -1 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2147: expect (ObjNum 982 snap 249 seq_num 982) 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2140: finishing write tid 1 to smithi06431743-14 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2148: read oid 19 snap -1 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2148: expect deleted 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2149: write (excl) oid 1 current snap is 259 2024-03-04T06:16:36.609 INFO:tasks.rados.rados.0.smithi064.stdout:2149: seq_num 1006 ranges {507842=659090,1568650=558681,2759766=148076} 2024-03-04T06:16:36.617 INFO:tasks.rados.rados.0.smithi064.stdout:2149: writing smithi06431743-1 from 507842 to 1166932 tid 1 2024-03-04T06:16:36.624 INFO:tasks.rados.rados.0.smithi064.stdout:2149: writing smithi06431743-1 from 1568650 to 2127331 tid 2 2024-03-04T06:16:36.629 INFO:tasks.rados.rados.0.smithi064.stdout:2149: writing smithi06431743-1 from 2759766 to 2907842 tid 3 2024-03-04T06:16:36.630 INFO:tasks.rados.rados.0.smithi064.stdout:2150: write (excl) oid 20 current snap is 259 2024-03-04T06:16:36.630 INFO:tasks.rados.rados.0.smithi064.stdout:2150: seq_num 1007 ranges {524649=734339,1910779=702339} 2024-03-04T06:16:36.638 INFO:tasks.rados.rados.0.smithi064.stdout:2150: writing smithi06431743-20 from 524649 to 1258988 tid 1 2024-03-04T06:16:36.649 INFO:tasks.rados.rados.0.smithi064.stdout:2150: writing smithi06431743-20 from 1910779 to 2613118 tid 2 2024-03-04T06:16:36.649 INFO:tasks.rados.rados.0.smithi064.stdout:2151: write oid 18 current snap is 259 2024-03-04T06:16:36.649 INFO:tasks.rados.rados.0.smithi064.stdout:2141: finishing write tid 1 to smithi06431743-40 2024-03-04T06:16:36.649 INFO:tasks.rados.rados.0.smithi064.stdout:2151: seq_num 1008 ranges {541456=409588,1569261=562350,2814612=126844} 2024-03-04T06:16:36.654 INFO:tasks.rados.rados.0.smithi064.stdout:2151: writing smithi06431743-18 from 541456 to 951044 tid 1 2024-03-04T06:16:36.661 INFO:tasks.rados.rados.0.smithi064.stdout:2151: writing smithi06431743-18 from 1569261 to 2131611 tid 2 2024-03-04T06:16:36.668 INFO:tasks.rados.rados.0.smithi064.stdout:2151: writing smithi06431743-18 from 2814612 to 2941456 tid 3 2024-03-04T06:16:36.668 INFO:tasks.rados.rados.0.smithi064.stdout:2152: delete oid 38 current snap is 259 2024-03-04T06:16:36.668 INFO:tasks.rados.rados.0.smithi064.stdout:2140: finishing write tid 2 to smithi06431743-14 2024-03-04T06:16:36.668 INFO:tasks.rados.rados.0.smithi064.stdout:2141: finishing write tid 2 to smithi06431743-40 2024-03-04T06:16:36.668 INFO:tasks.rados.rados.0.smithi064.stdout:2145: expect (ObjNum 950 snap 236 seq_num 950) 2024-03-04T06:16:36.819 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:16:36.819 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:16:36.831 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":782,"stamp":"2024-03-04T06:16:36.262214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93355527,"num_objects":740,"num_object_clones":567,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":137,"num_read":3986,"num_read_kb":2998511,"num_write":3460,"num_write_kb":907139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7190,"ondisk_log_size":7190,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":386652,"kb_used_data":223868,"kb_used_omap":0,"kb_used_meta":162752,"kb_avail":377100708,"statfs":{"total":386547056640,"available":386151124992,"internally_reserved":0,"allocated":229240832,"data_stored":243382731,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":166658048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2975037,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":9,"num_read_kb":7084,"num_write":5,"num_write_kb":1105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.913082"},"pg_stats":[{"pgid":"3.f","version":"520'441","reported_seq":1847,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.300191+0000","last_change":"2024-03-04T06:16:35.314102+0000","last_active":"2024-03-04T06:16:35.300191+0000","last_peered":"2024-03-04T06:16:35.300191+0000","last_clean":"2024-03-04T06:16:35.300191+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:16:35.300191+0000","last_undegraded":"2024-03-04T06:16:35.300191+0000","last_fullsized":"2024-03-04T06:16:35.300191+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4992141,"num_objects":49,"num_object_clones":39,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":248,"num_read_kb":201025,"num_write":205,"num_write_kb":55608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"520'322","reported_seq":1764,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.278928+0000","last_change":"2024-03-04T06:16:35.287066+0000","last_active":"2024-03-04T06:16:35.278928+0000","last_peered":"2024-03-04T06:16:35.278928+0000","last_clean":"2024-03-04T06:16:35.278928+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:16:35.278928+0000","last_undegraded":"2024-03-04T06:16:35.278928+0000","last_fullsized":"2024-03-04T06:16:35.278928+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4605580,"num_objects":39,"num_object_clones":26,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":131,"num_read_kb":80768,"num_write":155,"num_write_kb":40744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"506'465","reported_seq":1891,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T06:16:31.362101+0000","last_change":"2024-03-04T06:16:30.447957+0000","last_active":"2024-03-04T06:16:31.362101+0000","last_peered":"2024-03-04T06:16:31.362101+0000","last_clean":"2024-03-04T06:16:31.362101+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:16:31.362101+0000","last_undegraded":"2024-03-04T06:16:31.362101+0000","last_fullsized":"2024-03-04T06:16:31.362101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269704,"num_objects":56,"num_object_clones":43,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":314,"num_read_kb":244581,"num_write":241,"num_write_kb":66609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"468'322","reported_seq":1787,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.288825+0000","last_change":"2024-03-04T06:16:35.295529+0000","last_active":"2024-03-04T06:16:35.288825+0000","last_peered":"2024-03-04T06:16:35.288825+0000","last_clean":"2024-03-04T06:16:35.288825+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:16:35.288825+0000","last_undegraded":"2024-03-04T06:16:35.288825+0000","last_fullsized":"2024-03-04T06:16:35.288825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5395410,"num_objects":33,"num_object_clones":24,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":191,"num_read_kb":149126,"num_write":163,"num_write_kb":45378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1619,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T06:16:30.378202+0000","last_change":"2024-03-04T06:16:30.381240+0000","last_active":"2024-03-04T06:16:30.378202+0000","last_peered":"2024-03-04T06:16:30.378202+0000","last_clean":"2024-03-04T06:16:30.378202+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:16:30.378202+0000","last_undegraded":"2024-03-04T06:16:30.378202+0000","last_fullsized":"2024-03-04T06:16:30.378202+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":52,"num_read_kb":25073,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"520'473","reported_seq":1872,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.298856+0000","last_change":"2024-03-04T06:16:35.311119+0000","last_active":"2024-03-04T06:16:35.298856+0000","last_peered":"2024-03-04T06:16:35.298856+0000","last_clean":"2024-03-04T06:16:35.298856+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:16:35.298856+0000","last_undegraded":"2024-03-04T06:16:35.298856+0000","last_fullsized":"2024-03-04T06:16:35.298856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8185636,"num_objects":46,"num_object_clones":37,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":248,"num_read_kb":192233,"num_write":234,"num_write_kb":69208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1553,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.301651+0000","last_change":"2024-03-04T06:16:35.314448+0000","last_active":"2024-03-04T06:16:35.301651+0000","last_peered":"2024-03-04T06:16:35.301651+0000","last_clean":"2024-03-04T06:16:35.301651+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:16:35.301651+0000","last_undegraded":"2024-03-04T06:16:35.301651+0000","last_fullsized":"2024-03-04T06:16:35.301651+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"507'249","reported_seq":1718,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T06:16:30.440871+0000","last_change":"2024-03-04T06:16:30.448470+0000","last_active":"2024-03-04T06:16:30.440871+0000","last_peered":"2024-03-04T06:16:30.440871+0000","last_clean":"2024-03-04T06:16:30.440871+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:16:30.440871+0000","last_undegraded":"2024-03-04T06:16:30.440871+0000","last_fullsized":"2024-03-04T06:16:30.440871+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5378636,"num_objects":26,"num_object_clones":17,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":174,"num_read_kb":163220,"num_write":118,"num_write_kb":30955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1425,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T06:16:30.360969+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:16:30.360969+0000","last_peered":"2024-03-04T06:16:30.360969+0000","last_clean":"2024-03-04T06:16:30.360969+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:16:30.360969+0000","last_undegraded":"2024-03-04T06:16:30.360969+0000","last_fullsized":"2024-03-04T06:16:30.360969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1294,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T06:16:30.448456+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:16:30.448456+0000","last_peered":"2024-03-04T06:16:30.448456+0000","last_clean":"2024-03-04T06:16:30.448456+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:16:30.448456+0000","last_undegraded":"2024-03-04T06:16:30.448456+0000","last_fullsized":"2024-03-04T06:16:30.448456+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"519'481","reported_seq":1836,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T06:16:30.450116+0000","last_change":"2024-03-04T06:16:30.454146+0000","last_active":"2024-03-04T06:16:30.450116+0000","last_peered":"2024-03-04T06:16:30.450116+0000","last_clean":"2024-03-04T06:16:30.450116+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:16:30.450116+0000","last_undegraded":"2024-03-04T06:16:30.450116+0000","last_fullsized":"2024-03-04T06:16:30.450116+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269704,"num_objects":50,"num_object_clones":38,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":232,"num_read_kb":177399,"num_write":227,"num_write_kb":62351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1313,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.302557+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:16:35.302557+0000","last_peered":"2024-03-04T06:16:35.302557+0000","last_clean":"2024-03-04T06:16:35.302557+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:16:35.302557+0000","last_undegraded":"2024-03-04T06:16:35.302557+0000","last_fullsized":"2024-03-04T06:16:35.302557+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"519'363","reported_seq":1745,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T06:16:30.359591+0000","last_change":"2024-03-04T06:16:30.365817+0000","last_active":"2024-03-04T06:16:30.359591+0000","last_peered":"2024-03-04T06:16:30.359591+0000","last_clean":"2024-03-04T06:16:30.359591+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:16:30.359591+0000","last_undegraded":"2024-03-04T06:16:30.359591+0000","last_fullsized":"2024-03-04T06:16:30.359591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":28,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":202,"num_read_kb":166619,"num_write":137,"num_write_kb":39064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1296,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T06:16:31.360987+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:16:31.360987+0000","last_peered":"2024-03-04T06:16:31.360987+0000","last_clean":"2024-03-04T06:16:31.360987+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:16:31.360987+0000","last_undegraded":"2024-03-04T06:16:31.360987+0000","last_fullsized":"2024-03-04T06:16:31.360987+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"519'823","reported_seq":2115,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T06:16:30.367703+0000","last_change":"2024-03-04T06:16:30.380809+0000","last_active":"2024-03-04T06:16:30.367703+0000","last_peered":"2024-03-04T06:16:30.367703+0000","last_clean":"2024-03-04T06:16:30.367703+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:16:30.367703+0000","last_undegraded":"2024-03-04T06:16:30.367703+0000","last_fullsized":"2024-03-04T06:16:30.367703+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12555687,"num_objects":85,"num_object_clones":65,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":496,"num_read_kb":393987,"num_write":402,"num_write_kb":107876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1313,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.291934+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:16:35.291934+0000","last_peered":"2024-03-04T06:16:35.291934+0000","last_clean":"2024-03-04T06:16:35.291934+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:16:35.291934+0000","last_undegraded":"2024-03-04T06:16:35.291934+0000","last_fullsized":"2024-03-04T06:16:35.291934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"520'900","reported_seq":2095,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.288057+0000","last_change":"2024-03-04T06:16:35.288057+0000","last_active":"2024-03-04T06:16:35.288057+0000","last_peered":"2024-03-04T06:16:35.288057+0000","last_clean":"2024-03-04T06:16:35.288057+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:16:35.288057+0000","last_undegraded":"2024-03-04T06:16:35.288057+0000","last_fullsized":"2024-03-04T06:16:35.288057+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7547003,"num_objects":89,"num_object_clones":69,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":382,"num_read_kb":271414,"num_write":403,"num_write_kb":105679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"519'392","reported_seq":1833,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T06:16:30.452520+0000","last_change":"2024-03-04T06:16:30.454565+0000","last_active":"2024-03-04T06:16:30.452520+0000","last_peered":"2024-03-04T06:16:30.452520+0000","last_clean":"2024-03-04T06:16:30.452520+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:16:30.452520+0000","last_undegraded":"2024-03-04T06:16:30.452520+0000","last_fullsized":"2024-03-04T06:16:30.452520+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4908106,"num_objects":41,"num_object_clones":30,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":285,"num_read_kb":223227,"num_write":189,"num_write_kb":51024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1331,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.283343+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:16:35.283343+0000","last_peered":"2024-03-04T06:16:35.283343+0000","last_clean":"2024-03-04T06:16:35.283343+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:16:35.283343+0000","last_undegraded":"2024-03-04T06:16:35.283343+0000","last_fullsized":"2024-03-04T06:16:35.283343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"520'416","reported_seq":1841,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.286176+0000","last_change":"2024-03-04T06:16:35.290847+0000","last_active":"2024-03-04T06:16:35.286176+0000","last_peered":"2024-03-04T06:16:35.286176+0000","last_clean":"2024-03-04T06:16:35.286176+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:16:35.286176+0000","last_undegraded":"2024-03-04T06:16:35.286176+0000","last_fullsized":"2024-03-04T06:16:35.286176+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2739772,"num_objects":51,"num_object_clones":36,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":214,"num_read_kb":142434,"num_write":200,"num_write_kb":52704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1292,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T06:16:30.374971+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:16:30.374971+0000","last_peered":"2024-03-04T06:16:30.374971+0000","last_clean":"2024-03-04T06:16:30.374971+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:16:30.374971+0000","last_undegraded":"2024-03-04T06:16:30.374971+0000","last_fullsized":"2024-03-04T06:16:30.374971+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"507'567","reported_seq":1905,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T06:16:30.356866+0000","last_change":"2024-03-04T06:16:30.366463+0000","last_active":"2024-03-04T06:16:30.356866+0000","last_peered":"2024-03-04T06:16:30.356866+0000","last_clean":"2024-03-04T06:16:30.356866+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:16:30.356866+0000","last_undegraded":"2024-03-04T06:16:30.356866+0000","last_fullsized":"2024-03-04T06:16:30.356866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":567,"log_dups_size":0,"ondisk_log_size":567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9395707,"num_objects":56,"num_object_clones":44,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":315,"num_read_kb":248366,"num_write":263,"num_write_kb":73070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1331,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.279803+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:16:35.279803+0000","last_peered":"2024-03-04T06:16:35.279803+0000","last_clean":"2024-03-04T06:16:35.279803+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:16:35.279803+0000","last_undegraded":"2024-03-04T06:16:35.279803+0000","last_fullsized":"2024-03-04T06:16:35.279803+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"521'712","reported_seq":2029,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-04T06:16:35.277150+0000","last_change":"2024-03-04T06:16:35.287896+0000","last_active":"2024-03-04T06:16:35.277150+0000","last_peered":"2024-03-04T06:16:35.277150+0000","last_clean":"2024-03-04T06:16:35.277150+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:16:35.277150+0000","last_undegraded":"2024-03-04T06:16:35.277150+0000","last_fullsized":"2024-03-04T06:16:35.277150+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7194089,"num_objects":67,"num_object_clones":58,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":399,"num_read_kb":318948,"num_write":311,"num_write_kb":85013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1294,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T06:16:30.450782+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:16:30.450782+0000","last_peered":"2024-03-04T06:16:30.450782+0000","last_clean":"2024-03-04T06:16:30.450782+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:16:30.450782+0000","last_undegraded":"2024-03-04T06:16:30.450782+0000","last_fullsized":"2024-03-04T06:16:30.450782+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":92765159,"num_objects":738,"num_object_clones":567,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":137,"num_read":3883,"num_read_kb":2998420,"num_write":3323,"num_write_kb":905245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7093,"ondisk_log_size":7093,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":113012,"kb_used_data":68140,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94258828,"statfs":{"total":96636764160,"available":96521039872,"internally_reserved":0,"allocated":69775360,"data_stored":73877041,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":97624,"kb_used_data":58320,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94274216,"statfs":{"total":96636764160,"available":96536797184,"internally_reserved":0,"allocated":59719680,"data_stored":62902727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88544,"kb_used_data":49240,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94283296,"statfs":{"total":96636764160,"available":96546095104,"internally_reserved":0,"allocated":50421760,"data_stored":54134932,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":87472,"kb_used_data":48168,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94284368,"statfs":{"total":96636764160,"available":96547192832,"internally_reserved":0,"allocated":49324032,"data_stored":52468031,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:16:36.836 INFO:tasks.rados.rados.0.smithi064.stdout:2142: finishing write tid 1 to smithi06431743-10 2024-03-04T06:16:36.836 INFO:tasks.rados.rados.0.smithi064.stdout:2147: expect (ObjNum 982 snap 249 seq_num 982) 2024-03-04T06:16:37.034 INFO:tasks.rados.rados.0.smithi064.stdout:2141: finishing write tid 3 to smithi06431743-40 2024-03-04T06:16:37.034 INFO:tasks.rados.rados.0.smithi064.stdout:2140: finishing write tid 3 to smithi06431743-14 2024-03-04T06:16:37.034 INFO:tasks.rados.rados.0.smithi064.stdout:2142: finishing write tid 2 to smithi06431743-10 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2141: finishing write tid 4 to smithi06431743-40 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2141: finishing write tid 5 to smithi06431743-40 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2141: oid 40 updating version 0 to 445 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2141: oid 40 version 445 is already newer than 445 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2141: oid 40 version 445 is already newer than 443 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2141: oid 40 version 445 is already newer than 442 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 445 (ObjNum 1004 snap 259 seq_num 1004) dirty exists 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2141: left oid 40 (ObjNum 1004 snap 259 seq_num 1004) 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2142: finishing write tid 3 to smithi06431743-10 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2149: finishing write tid 1 to smithi06431743-1 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2140: finishing write tid 4 to smithi06431743-14 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2141: done (12 left) 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2143: done (11 left) 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2144: done (10 left) 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2145: done (9 left) 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2147: done (8 left) 2024-03-04T06:16:37.035 INFO:tasks.rados.rados.0.smithi064.stdout:2148: done (7 left) 2024-03-04T06:16:37.036 INFO:tasks.rados.rados.0.smithi064.stdout:2152: done (6 left) 2024-03-04T06:16:37.036 INFO:tasks.rados.rados.0.smithi064.stdout:2153: snap_remove snap 250 2024-03-04T06:16:37.269 INFO:tasks.rados.rados.0.smithi064.stdout:2153: done (6 left) 2024-03-04T06:16:37.269 INFO:tasks.rados.rados.0.smithi064.stdout:2154: read oid 31 snap 257 2024-03-04T06:16:37.269 INFO:tasks.rados.rados.0.smithi064.stdout:2154: expect (ObjNum 912 snap 224 seq_num 912) 2024-03-04T06:16:37.269 INFO:tasks.rados.rados.0.smithi064.stdout:2155: write oid 21 current snap is 259 2024-03-04T06:16:37.270 INFO:tasks.rados.rados.0.smithi064.stdout:2151: finishing write tid 1 to smithi06431743-18 2024-03-04T06:16:37.270 INFO:tasks.rados.rados.0.smithi064.stdout:2155: seq_num 1010 ranges {575070=560086,1686225=566019,2869458=105612} 2024-03-04T06:16:37.275 INFO:tasks.rados.rados.0.smithi064.stdout:2155: writing smithi06431743-21 from 575070 to 1135156 tid 1 2024-03-04T06:16:37.283 INFO:tasks.rados.rados.0.smithi064.stdout:2155: writing smithi06431743-21 from 1686225 to 2252244 tid 2 2024-03-04T06:16:37.288 INFO:tasks.rados.rados.0.smithi064.stdout:2155: writing smithi06431743-21 from 2869458 to 2975070 tid 3 2024-03-04T06:16:37.288 INFO:tasks.rados.rados.0.smithi064.stdout:2156: write (excl) oid 26 current snap is 259 2024-03-04T06:16:37.288 INFO:tasks.rados.rados.0.smithi064.stdout:2140: finishing write tid 5 to smithi06431743-14 2024-03-04T06:16:37.288 INFO:tasks.rados.rados.0.smithi064.stdout:2140: oid 14 updating version 0 to 577 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2140: oid 14 updating version 577 to 580 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2140: oid 14 version 580 is already newer than 578 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2140: oid 14 version 580 is already newer than 579 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 580 (ObjNum 1003 snap 259 seq_num 1003) dirty exists 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2140: left oid 14 (ObjNum 1003 snap 259 seq_num 1003) 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2142: finishing write tid 4 to smithi06431743-10 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2142: finishing write tid 5 to smithi06431743-10 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2142: oid 10 updating version 0 to 478 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2142: oid 10 version 478 is already newer than 477 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2142: oid 10 version 478 is already newer than 475 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2142: oid 10 version 478 is already newer than 476 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 478 (ObjNum 1005 snap 259 seq_num 1005) dirty exists 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2142: left oid 10 (ObjNum 1005 snap 259 seq_num 1005) 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2149: finishing write tid 2 to smithi06431743-1 2024-03-04T06:16:37.289 INFO:tasks.rados.rados.0.smithi064.stdout:2156: seq_num 1011 ranges {591877=635335,1628354=709677} 2024-03-04T06:16:37.296 INFO:tasks.rados.rados.0.smithi064.stdout:2156: writing smithi06431743-26 from 591877 to 1227212 tid 1 2024-03-04T06:16:37.305 INFO:tasks.rados.rados.0.smithi064.stdout:2156: writing smithi06431743-26 from 1628354 to 2338031 tid 2 2024-03-04T06:16:37.305 INFO:tasks.rados.rados.0.smithi064.stdout:2140: done (8 left) 2024-03-04T06:16:37.305 INFO:tasks.rados.rados.0.smithi064.stdout:2142: done (7 left) 2024-03-04T06:16:37.305 INFO:tasks.rados.rados.0.smithi064.stdout:2146: done (6 left) 2024-03-04T06:16:37.305 INFO:tasks.rados.rados.0.smithi064.stdout:2157: delete oid 6 current snap is 259 2024-03-04T06:16:37.305 INFO:tasks.rados.rados.0.smithi064.stdout:2151: finishing write tid 2 to smithi06431743-18 2024-03-04T06:16:37.305 INFO:tasks.rados.rados.0.smithi064.stdout:2149: finishing write tid 3 to smithi06431743-1 2024-03-04T06:16:37.305 INFO:tasks.rados.rados.0.smithi064.stdout:2151: finishing write tid 3 to smithi06431743-18 2024-03-04T06:16:37.305 INFO:tasks.rados.rados.0.smithi064.stdout:2150: finishing write tid 1 to smithi06431743-20 2024-03-04T06:16:37.305 INFO:tasks.rados.rados.0.smithi064.stdout:2151: finishing write tid 4 to smithi06431743-18 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2151: finishing write tid 5 to smithi06431743-18 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2151: oid 18 updating version 0 to 327 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2151: oid 18 version 327 is already newer than 324 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2151: oid 18 version 327 is already newer than 327 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2151: oid 18 version 327 is already newer than 325 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 327 (ObjNum 1008 snap 259 seq_num 1008) dirty exists 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2151: left oid 18 (ObjNum 1008 snap 259 seq_num 1008) 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2149: finishing write tid 4 to smithi06431743-1 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2149: finishing write tid 5 to smithi06431743-1 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2149: oid 1 updating version 0 to 827 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2149: oid 1 updating version 827 to 828 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2149: oid 1 updating version 828 to 829 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2149: oid 1 updating version 829 to 830 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 830 (ObjNum 1006 snap 259 seq_num 1006) dirty exists 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2149: left oid 1 (ObjNum 1006 snap 259 seq_num 1006) 2024-03-04T06:16:37.306 INFO:tasks.rados.rados.0.smithi064.stdout:2150: finishing write tid 2 to smithi06431743-20 2024-03-04T06:16:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2150: finishing write tid 3 to smithi06431743-20 2024-03-04T06:16:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2150: finishing write tid 4 to smithi06431743-20 2024-03-04T06:16:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2150: oid 20 updating version 0 to 584 2024-03-04T06:16:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2150: oid 20 version 584 is already newer than 584 2024-03-04T06:16:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2150: oid 20 version 584 is already newer than 582 2024-03-04T06:16:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 584 (ObjNum 1007 snap 259 seq_num 1007) dirty exists 2024-03-04T06:16:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2150: left oid 20 (ObjNum 1007 snap 259 seq_num 1007) 2024-03-04T06:16:37.339 INFO:tasks.rados.rados.0.smithi064.stdout:2154: expect (ObjNum 912 snap 224 seq_num 912) 2024-03-04T06:16:37.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:2156: finishing write tid 1 to smithi06431743-26 2024-03-04T06:16:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:2156: finishing write tid 2 to smithi06431743-26 2024-03-04T06:16:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:2149: done (6 left) 2024-03-04T06:16:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:2150: done (5 left) 2024-03-04T06:16:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:2151: done (4 left) 2024-03-04T06:16:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:2154: done (3 left) 2024-03-04T06:16:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:2157: done (2 left) 2024-03-04T06:16:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:2158: snap_create 2024-03-04T06:16:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:2155: finishing write tid 1 to smithi06431743-21 2024-03-04T06:16:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:2156: finishing write tid 3 to smithi06431743-26 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2156: finishing write tid 4 to smithi06431743-26 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2156: oid 26 updating version 0 to 831 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2156: oid 26 updating version 831 to 837 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2156: oid 26 version 837 is already newer than 836 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 837 (ObjNum 1011 snap 259 seq_num 1011) dirty exists 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2156: left oid 26 (ObjNum 1011 snap 259 seq_num 1011) 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2155: finishing write tid 2 to smithi06431743-21 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2155: finishing write tid 3 to smithi06431743-21 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2155: finishing write tid 4 to smithi06431743-21 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2155: finishing write tid 5 to smithi06431743-21 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2155: oid 21 updating version 0 to 398 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2155: oid 21 updating version 398 to 399 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2155: oid 21 version 399 is already newer than 397 2024-03-04T06:16:37.474 INFO:tasks.rados.rados.0.smithi064.stdout:2155: oid 21 version 399 is already newer than 396 2024-03-04T06:16:37.475 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 399 (ObjNum 1010 snap 259 seq_num 1010) dirty exists 2024-03-04T06:16:37.475 INFO:tasks.rados.rados.0.smithi064.stdout:2155: left oid 21 (ObjNum 1010 snap 259 seq_num 1010) 2024-03-04T06:16:37.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:37.476+0000 7f42afd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:37.481 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:37.480+0000 7f42afd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:38.272 INFO:tasks.rados.rados.0.smithi064.stdout:2155: done (2 left) 2024-03-04T06:16:38.273 INFO:tasks.rados.rados.0.smithi064.stdout:2156: done (1 left) 2024-03-04T06:16:38.273 INFO:tasks.rados.rados.0.smithi064.stdout:2158: done (0 left) 2024-03-04T06:16:38.273 INFO:tasks.rados.rados.0.smithi064.stdout:2159: read oid 19 snap -1 2024-03-04T06:16:38.273 INFO:tasks.rados.rados.0.smithi064.stdout:2159: expect deleted 2024-03-04T06:16:38.273 INFO:tasks.rados.rados.0.smithi064.stdout:2160: write oid 35 current snap is 260 2024-03-04T06:16:38.273 INFO:tasks.rados.rados.0.smithi064.stdout:2160: seq_num 1013 ranges {625491=502186,1861671=713346} 2024-03-04T06:16:38.278 INFO:tasks.rados.rados.0.smithi064.stdout:2160: writing smithi06431743-35 from 625491 to 1127677 tid 1 2024-03-04T06:16:38.287 INFO:tasks.rados.rados.0.smithi064.stdout:2160: writing smithi06431743-35 from 1861671 to 2575017 tid 2 2024-03-04T06:16:38.287 INFO:tasks.rados.rados.0.smithi064.stdout:2161: delete oid 23 current snap is 260 2024-03-04T06:16:38.343 INFO:tasks.rados.rados.0.smithi064.stdout:2160: finishing write tid 1 to smithi06431743-35 2024-03-04T06:16:38.349 INFO:tasks.rados.rados.0.smithi064.stdout:2159: done (2 left) 2024-03-04T06:16:38.349 INFO:tasks.rados.rados.0.smithi064.stdout:2161: done (1 left) 2024-03-04T06:16:38.349 INFO:tasks.rados.rados.0.smithi064.stdout:2162: read oid 19 snap 247 2024-03-04T06:16:38.349 INFO:tasks.rados.rados.0.smithi064.stdout:2162: expect (ObjNum 904 snap 223 seq_num 904) 2024-03-04T06:16:38.349 INFO:tasks.rados.rados.0.smithi064.stdout:2163: read oid 11 snap -1 2024-03-04T06:16:38.349 INFO:tasks.rados.rados.0.smithi064.stdout:2163: expect (ObjNum 956 snap 240 seq_num 956) 2024-03-04T06:16:38.349 INFO:tasks.rados.rados.0.smithi064.stdout:2164: read oid 3 snap 247 2024-03-04T06:16:38.349 INFO:tasks.rados.rados.0.smithi064.stdout:2164: expect (ObjNum 964 snap 241 seq_num 964) 2024-03-04T06:16:38.349 INFO:tasks.rados.rados.0.smithi064.stdout:2165: write (excl) oid 29 current snap is 260 2024-03-04T06:16:38.350 INFO:tasks.rados.rados.0.smithi064.stdout:2165: seq_num 1015 ranges {659105=652684,1862282=717015} 2024-03-04T06:16:38.358 INFO:tasks.rados.rados.0.smithi064.stdout:2165: writing smithi06431743-29 from 659105 to 1311789 tid 1 2024-03-04T06:16:38.367 INFO:tasks.rados.rados.0.smithi064.stdout:2165: writing smithi06431743-29 from 1862282 to 2579297 tid 2 2024-03-04T06:16:38.367 INFO:tasks.rados.rados.0.smithi064.stdout:2166: snap_create 2024-03-04T06:16:38.367 INFO:tasks.rados.rados.0.smithi064.stdout:2160: finishing write tid 2 to smithi06431743-35 2024-03-04T06:16:38.374 INFO:tasks.rados.rados.0.smithi064.stdout:2162: expect (ObjNum 904 snap 223 seq_num 904) 2024-03-04T06:16:38.635 INFO:tasks.rados.rados.0.smithi064.stdout:2163: expect (ObjNum 956 snap 240 seq_num 956) 2024-03-04T06:16:38.755 INFO:tasks.rados.rados.0.smithi064.stdout:2160: finishing write tid 3 to smithi06431743-35 2024-03-04T06:16:38.755 INFO:tasks.rados.rados.0.smithi064.stdout:2160: finishing write tid 4 to smithi06431743-35 2024-03-04T06:16:38.755 INFO:tasks.rados.rados.0.smithi064.stdout:2160: oid 35 updating version 0 to 593 2024-03-04T06:16:38.755 INFO:tasks.rados.rados.0.smithi064.stdout:2160: oid 35 version 593 is already newer than 591 2024-03-04T06:16:38.755 INFO:tasks.rados.rados.0.smithi064.stdout:2160: oid 35 version 593 is already newer than 592 2024-03-04T06:16:38.756 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 593 (ObjNum 1013 snap 260 seq_num 1013) dirty exists 2024-03-04T06:16:38.756 INFO:tasks.rados.rados.0.smithi064.stdout:2160: left oid 35 (ObjNum 1013 snap 260 seq_num 1013) 2024-03-04T06:16:38.756 INFO:tasks.rados.rados.0.smithi064.stdout:2165: finishing write tid 1 to smithi06431743-29 2024-03-04T06:16:38.756 INFO:tasks.rados.rados.0.smithi064.stdout:2164: expect (ObjNum 964 snap 241 seq_num 964) 2024-03-04T06:16:38.880 INFO:tasks.rados.rados.0.smithi064.stdout:2165: finishing write tid 2 to smithi06431743-29 2024-03-04T06:16:38.880 INFO:tasks.rados.rados.0.smithi064.stdout:2165: finishing write tid 3 to smithi06431743-29 2024-03-04T06:16:38.880 INFO:tasks.rados.rados.0.smithi064.stdout:2165: finishing write tid 4 to smithi06431743-29 2024-03-04T06:16:38.880 INFO:tasks.rados.rados.0.smithi064.stdout:2165: oid 29 updating version 0 to 908 2024-03-04T06:16:38.880 INFO:tasks.rados.rados.0.smithi064.stdout:2165: oid 29 updating version 908 to 909 2024-03-04T06:16:38.880 INFO:tasks.rados.rados.0.smithi064.stdout:2165: oid 29 updating version 909 to 910 2024-03-04T06:16:38.880 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 910 (ObjNum 1015 snap 260 seq_num 1015) dirty exists 2024-03-04T06:16:38.880 INFO:tasks.rados.rados.0.smithi064.stdout:2165: left oid 29 (ObjNum 1015 snap 260 seq_num 1015) 2024-03-04T06:16:39.275 INFO:tasks.rados.rados.0.smithi064.stdout:2160: done (5 left) 2024-03-04T06:16:39.276 INFO:tasks.rados.rados.0.smithi064.stdout:2162: done (4 left) 2024-03-04T06:16:39.276 INFO:tasks.rados.rados.0.smithi064.stdout:2163: done (3 left) 2024-03-04T06:16:39.276 INFO:tasks.rados.rados.0.smithi064.stdout:2164: done (2 left) 2024-03-04T06:16:39.276 INFO:tasks.rados.rados.0.smithi064.stdout:2165: done (1 left) 2024-03-04T06:16:39.276 INFO:tasks.rados.rados.0.smithi064.stdout:2166: done (0 left) 2024-03-04T06:16:39.276 INFO:tasks.rados.rados.0.smithi064.stdout:2167: write oid 29 current snap is 261 2024-03-04T06:16:39.276 INFO:tasks.rados.rados.0.smithi064.stdout:2167: seq_num 1016 ranges {675912=727933,1920764=460673,2801290=274622} 2024-03-04T06:16:39.283 INFO:tasks.rados.rados.0.smithi064.stdout:2167: writing smithi06431743-29 from 675912 to 1403845 tid 1 2024-03-04T06:16:39.290 INFO:tasks.rados.rados.0.smithi064.stdout:2167: writing smithi06431743-29 from 1920764 to 2381437 tid 2 2024-03-04T06:16:39.294 INFO:tasks.rados.rados.0.smithi064.stdout:2167: writing smithi06431743-29 from 2801290 to 3075912 tid 3 2024-03-04T06:16:39.294 INFO:tasks.rados.rados.0.smithi064.stdout:2168: snap_remove snap 251 2024-03-04T06:16:39.546 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:39.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:39.653+0000 7f66c115d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:39.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:39.657+0000 7f66c115d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:40.279 INFO:tasks.rados.rados.0.smithi064.stdout:2168: done (1 left) 2024-03-04T06:16:40.279 INFO:tasks.rados.rados.0.smithi064.stdout:2169: snap_create 2024-03-04T06:16:40.279 INFO:tasks.rados.rados.0.smithi064.stdout:2167: finishing write tid 1 to smithi06431743-29 2024-03-04T06:16:40.279 INFO:tasks.rados.rados.0.smithi064.stdout:2167: finishing write tid 2 to smithi06431743-29 2024-03-04T06:16:40.280 INFO:tasks.rados.rados.0.smithi064.stdout:2167: finishing write tid 3 to smithi06431743-29 2024-03-04T06:16:40.280 INFO:tasks.rados.rados.0.smithi064.stdout:2167: finishing write tid 4 to smithi06431743-29 2024-03-04T06:16:40.280 INFO:tasks.rados.rados.0.smithi064.stdout:2167: finishing write tid 5 to smithi06431743-29 2024-03-04T06:16:40.280 INFO:tasks.rados.rados.0.smithi064.stdout:2167: oid 29 updating version 0 to 915 2024-03-04T06:16:40.280 INFO:tasks.rados.rados.0.smithi064.stdout:2167: oid 29 version 915 is already newer than 915 2024-03-04T06:16:40.280 INFO:tasks.rados.rados.0.smithi064.stdout:2167: oid 29 version 915 is already newer than 912 2024-03-04T06:16:40.280 INFO:tasks.rados.rados.0.smithi064.stdout:2167: oid 29 version 915 is already newer than 913 2024-03-04T06:16:40.280 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 915 (ObjNum 1016 snap 261 seq_num 1016) dirty exists 2024-03-04T06:16:40.280 INFO:tasks.rados.rados.0.smithi064.stdout:2167: left oid 29 (ObjNum 1016 snap 261 seq_num 1016) 2024-03-04T06:16:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:2167: done (1 left) 2024-03-04T06:16:41.280 INFO:tasks.rados.rados.0.smithi064.stdout:2169: done (0 left) 2024-03-04T06:16:41.281 INFO:tasks.rados.rados.0.smithi064.stdout:2170: write (excl) oid 30 current snap is 262 2024-03-04T06:16:41.281 INFO:tasks.rados.rados.0.smithi064.stdout:2170: seq_num 1017 ranges {692719=403182,1579246=720684,2828713=264006} 2024-03-04T06:16:41.286 INFO:tasks.rados.rados.0.smithi064.stdout:2170: writing smithi06431743-30 from 692719 to 1095901 tid 1 2024-03-04T06:16:41.292 INFO:tasks.rados.rados.0.smithi064.stdout:2170: writing smithi06431743-30 from 1579246 to 2299930 tid 2 2024-03-04T06:16:41.297 INFO:tasks.rados.rados.0.smithi064.stdout:2170: writing smithi06431743-30 from 2828713 to 3092719 tid 3 2024-03-04T06:16:41.298 INFO:tasks.rados.rados.0.smithi064.stdout:2171: read oid 50 snap -1 2024-03-04T06:16:41.298 INFO:tasks.rados.rados.0.smithi064.stdout:2171: expect (ObjNum 975 snap 244 seq_num 975) 2024-03-04T06:16:41.298 INFO:tasks.rados.rados.0.smithi064.stdout:2172: delete oid 46 current snap is 262 2024-03-04T06:16:41.364 INFO:tasks.rados.rados.0.smithi064.stdout:2171: expect (ObjNum 975 snap 244 seq_num 975) 2024-03-04T06:16:41.484 INFO:tasks.rados.rados.0.smithi064.stdout:2170: finishing write tid 1 to smithi06431743-30 2024-03-04T06:16:41.484 INFO:tasks.rados.rados.0.smithi064.stdout:2170: finishing write tid 2 to smithi06431743-30 2024-03-04T06:16:41.484 INFO:tasks.rados.rados.0.smithi064.stdout:2170: finishing write tid 3 to smithi06431743-30 2024-03-04T06:16:41.484 INFO:tasks.rados.rados.0.smithi064.stdout:2170: finishing write tid 4 to smithi06431743-30 2024-03-04T06:16:41.484 INFO:tasks.rados.rados.0.smithi064.stdout:2170: finishing write tid 5 to smithi06431743-30 2024-03-04T06:16:41.484 INFO:tasks.rados.rados.0.smithi064.stdout:2170: oid 30 updating version 0 to 923 2024-03-04T06:16:41.485 INFO:tasks.rados.rados.0.smithi064.stdout:2170: oid 30 version 923 is already newer than 923 2024-03-04T06:16:41.485 INFO:tasks.rados.rados.0.smithi064.stdout:2170: oid 30 version 923 is already newer than 920 2024-03-04T06:16:41.485 INFO:tasks.rados.rados.0.smithi064.stdout:2170: oid 30 version 923 is already newer than 921 2024-03-04T06:16:41.485 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 923 (ObjNum 1017 snap 262 seq_num 1017) dirty exists 2024-03-04T06:16:41.485 INFO:tasks.rados.rados.0.smithi064.stdout:2170: left oid 30 (ObjNum 1017 snap 262 seq_num 1017) 2024-03-04T06:16:41.485 INFO:tasks.rados.rados.0.smithi064.stdout:2170: done (2 left) 2024-03-04T06:16:41.485 INFO:tasks.rados.rados.0.smithi064.stdout:2171: done (1 left) 2024-03-04T06:16:41.485 INFO:tasks.rados.rados.0.smithi064.stdout:2172: done (0 left) 2024-03-04T06:16:41.485 INFO:tasks.rados.rados.0.smithi064.stdout:2173: delete oid 22 current snap is 262 2024-03-04T06:16:41.486 INFO:tasks.rados.rados.0.smithi064.stdout:2173: done (0 left) 2024-03-04T06:16:41.486 INFO:tasks.rados.rados.0.smithi064.stdout:2174: write (excl) oid 7 current snap is 262 2024-03-04T06:16:41.487 INFO:tasks.rados.rados.0.smithi064.stdout:2174: seq_num 1020 ranges {743140=745282,2154692=468011} 2024-03-04T06:16:41.494 INFO:tasks.rados.rados.0.smithi064.stdout:2174: writing smithi06431743-7 from 743140 to 1488422 tid 1 2024-03-04T06:16:41.501 INFO:tasks.rados.rados.0.smithi064.stdout:2174: writing smithi06431743-7 from 2154692 to 2622703 tid 2 2024-03-04T06:16:41.501 INFO:tasks.rados.rados.0.smithi064.stdout:2175: write oid 15 current snap is 262 2024-03-04T06:16:41.501 INFO:tasks.rados.rados.0.smithi064.stdout:2175: seq_num 1021 ranges {759947=420531,1813174=728022} 2024-03-04T06:16:41.507 INFO:tasks.rados.rados.0.smithi064.stdout:2175: writing smithi06431743-15 from 759947 to 1180478 tid 1 2024-03-04T06:16:41.516 INFO:tasks.rados.rados.0.smithi064.stdout:2175: writing smithi06431743-15 from 1813174 to 2541196 tid 2 2024-03-04T06:16:41.516 INFO:tasks.rados.rados.0.smithi064.stdout:2176: write (excl) oid 30 current snap is 262 2024-03-04T06:16:41.516 INFO:tasks.rados.rados.0.smithi064.stdout:2176: seq_num 1022 ranges {776754=495780,1871656=471680,2965828=210926} 2024-03-04T06:16:41.523 INFO:tasks.rados.rados.0.smithi064.stdout:2176: writing smithi06431743-30 from 776754 to 1272534 tid 1 2024-03-04T06:16:41.529 INFO:tasks.rados.rados.0.smithi064.stdout:2176: writing smithi06431743-30 from 1871656 to 2343336 tid 2 2024-03-04T06:16:41.534 INFO:tasks.rados.rados.0.smithi064.stdout:2176: writing smithi06431743-30 from 2965828 to 3176754 tid 3 2024-03-04T06:16:41.534 INFO:tasks.rados.rados.0.smithi064.stdout:2177: delete oid 47 current snap is 262 2024-03-04T06:16:41.534 INFO:tasks.rados.rados.0.smithi064.stdout:2174: finishing write tid 1 to smithi06431743-7 2024-03-04T06:16:41.552 INFO:tasks.rados.rados.0.smithi064.stdout:2176: finishing write tid 1 to smithi06431743-30 2024-03-04T06:16:41.559 INFO:tasks.rados.rados.0.smithi064.stdout:2174: finishing write tid 2 to smithi06431743-7 2024-03-04T06:16:41.566 INFO:tasks.rados.rados.0.smithi064.stdout:2177: done (3 left) 2024-03-04T06:16:41.567 INFO:tasks.rados.rados.0.smithi064.stdout:2178: read oid 48 snap 256 2024-03-04T06:16:41.567 INFO:tasks.rados.rados.0.smithi064.stdout:2178: expect deleted 2024-03-04T06:16:41.567 INFO:tasks.rados.rados.0.smithi064.stdout:2179: read oid 32 snap -1 2024-03-04T06:16:41.567 INFO:tasks.rados.rados.0.smithi064.stdout:2179: expect deleted 2024-03-04T06:16:41.567 INFO:tasks.rados.rados.0.smithi064.stdout:2180: write (excl) oid 27 current snap is 262 2024-03-04T06:16:41.567 INFO:tasks.rados.rados.0.smithi064.stdout:2180: seq_num 1024 ranges {410368=646278,1472267=475349,2504321=405600} 2024-03-04T06:16:41.573 INFO:tasks.rados.rados.0.smithi064.stdout:2180: writing smithi06431743-27 from 410368 to 1056646 tid 1 2024-03-04T06:16:41.578 INFO:tasks.rados.rados.0.smithi064.stdout:2180: writing smithi06431743-27 from 1472267 to 1947616 tid 2 2024-03-04T06:16:41.585 INFO:tasks.rados.rados.0.smithi064.stdout:2180: writing smithi06431743-27 from 2504321 to 2909921 tid 3 2024-03-04T06:16:41.585 INFO:tasks.rados.rados.0.smithi064.stdout:2181: delete oid 20 current snap is 262 2024-03-04T06:16:41.585 INFO:tasks.rados.rados.0.smithi064.stdout:2176: finishing write tid 2 to smithi06431743-30 2024-03-04T06:16:41.586 INFO:tasks.rados.rados.0.smithi064.stdout:2174: finishing write tid 3 to smithi06431743-7 2024-03-04T06:16:41.589 INFO:tasks.rados.rados.0.smithi064.stdout:2174: finishing write tid 4 to smithi06431743-7 2024-03-04T06:16:41.589 INFO:tasks.rados.rados.0.smithi064.stdout:2174: oid 7 updating version 0 to 364 2024-03-04T06:16:41.589 INFO:tasks.rados.rados.0.smithi064.stdout:2174: oid 7 updating version 364 to 365 2024-03-04T06:16:41.589 INFO:tasks.rados.rados.0.smithi064.stdout:2174: oid 7 updating version 365 to 366 2024-03-04T06:16:41.590 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 366 (ObjNum 1020 snap 262 seq_num 1020) dirty exists 2024-03-04T06:16:41.590 INFO:tasks.rados.rados.0.smithi064.stdout:2174: left oid 7 (ObjNum 1020 snap 262 seq_num 1020) 2024-03-04T06:16:41.596 INFO:tasks.rados.rados.0.smithi064.stdout:2175: finishing write tid 1 to smithi06431743-15 2024-03-04T06:16:41.613 INFO:tasks.rados.rados.0.smithi064.stdout:2176: finishing write tid 3 to smithi06431743-30 2024-03-04T06:16:41.614 INFO:tasks.rados.rados.0.smithi064.stdout:2180: finishing write tid 1 to smithi06431743-27 2024-03-04T06:16:41.626 INFO:tasks.rados.rados.0.smithi064.stdout:2176: finishing write tid 4 to smithi06431743-30 2024-03-04T06:16:41.627 INFO:tasks.rados.rados.0.smithi064.stdout:2174: done (6 left) 2024-03-04T06:16:41.627 INFO:tasks.rados.rados.0.smithi064.stdout:2178: done (5 left) 2024-03-04T06:16:41.628 INFO:tasks.rados.rados.0.smithi064.stdout:2179: done (4 left) 2024-03-04T06:16:41.628 INFO:tasks.rados.rados.0.smithi064.stdout:2181: done (3 left) 2024-03-04T06:16:41.628 INFO:tasks.rados.rados.0.smithi064.stdout:2182: read oid 13 snap -1 2024-03-04T06:16:41.628 INFO:tasks.rados.rados.0.smithi064.stdout:2182: expect deleted 2024-03-04T06:16:41.628 INFO:tasks.rados.rados.0.smithi064.stdout:2183: snap_create 2024-03-04T06:16:41.628 INFO:tasks.rados.rados.0.smithi064.stdout:2175: finishing write tid 2 to smithi06431743-15 2024-03-04T06:16:41.628 INFO:tasks.rados.rados.0.smithi064.stdout:2176: finishing write tid 5 to smithi06431743-30 2024-03-04T06:16:41.628 INFO:tasks.rados.rados.0.smithi064.stdout:2176: oid 30 updating version 0 to 926 2024-03-04T06:16:41.628 INFO:tasks.rados.rados.0.smithi064.stdout:2176: oid 30 version 926 is already newer than 924 2024-03-04T06:16:41.629 INFO:tasks.rados.rados.0.smithi064.stdout:2176: oid 30 version 926 is already newer than 925 2024-03-04T06:16:41.629 INFO:tasks.rados.rados.0.smithi064.stdout:2176: oid 30 updating version 926 to 927 2024-03-04T06:16:41.629 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 927 (ObjNum 1022 snap 262 seq_num 1022) dirty exists 2024-03-04T06:16:41.629 INFO:tasks.rados.rados.0.smithi064.stdout:2176: left oid 30 (ObjNum 1022 snap 262 seq_num 1022) 2024-03-04T06:16:41.634 INFO:tasks.rados.rados.0.smithi064.stdout:2180: finishing write tid 2 to smithi06431743-27 2024-03-04T06:16:41.640 INFO:tasks.rados.rados.0.smithi064.stdout:2175: finishing write tid 3 to smithi06431743-15 2024-03-04T06:16:41.642 INFO:tasks.rados.rados.0.smithi064.stdout:2175: finishing write tid 4 to smithi06431743-15 2024-03-04T06:16:41.642 INFO:tasks.rados.rados.0.smithi064.stdout:2175: oid 15 updating version 0 to 408 2024-03-04T06:16:41.643 INFO:tasks.rados.rados.0.smithi064.stdout:2175: oid 15 version 408 is already newer than 406 2024-03-04T06:16:41.643 INFO:tasks.rados.rados.0.smithi064.stdout:2175: oid 15 version 408 is already newer than 407 2024-03-04T06:16:41.643 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 408 (ObjNum 1021 snap 262 seq_num 1021) dirty exists 2024-03-04T06:16:41.643 INFO:tasks.rados.rados.0.smithi064.stdout:2175: left oid 15 (ObjNum 1021 snap 262 seq_num 1021) 2024-03-04T06:16:41.645 INFO:tasks.rados.rados.0.smithi064.stdout:2180: finishing write tid 3 to smithi06431743-27 2024-03-04T06:16:41.649 INFO:tasks.rados.rados.0.smithi064.stdout:2180: finishing write tid 4 to smithi06431743-27 2024-03-04T06:16:41.651 INFO:tasks.rados.rados.0.smithi064.stdout:2180: finishing write tid 5 to smithi06431743-27 2024-03-04T06:16:41.651 INFO:tasks.rados.rados.0.smithi064.stdout:2180: oid 27 updating version 0 to 338 2024-03-04T06:16:41.651 INFO:tasks.rados.rados.0.smithi064.stdout:2180: oid 27 version 338 is already newer than 336 2024-03-04T06:16:41.652 INFO:tasks.rados.rados.0.smithi064.stdout:2180: oid 27 version 338 is already newer than 337 2024-03-04T06:16:41.652 INFO:tasks.rados.rados.0.smithi064.stdout:2180: oid 27 updating version 338 to 339 2024-03-04T06:16:41.652 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 339 (ObjNum 1024 snap 262 seq_num 1024) dirty exists 2024-03-04T06:16:41.652 INFO:tasks.rados.rados.0.smithi064.stdout:2180: left oid 27 (ObjNum 1024 snap 262 seq_num 1024) 2024-03-04T06:16:41.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:41.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:41.830+0000 7fee0219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:41.833 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:16:41.833 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:16:41.833 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:16:41.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:16:41.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:41.835+0000 7fee0219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:41.950 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:41.945+0000 7f742da6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:41.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:41.953+0000 7f742da6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:41.970 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:41.970 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:41.970 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:41.970 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:41.970 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:41.970 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:41.970 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:41.970 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:41.971 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:41.971 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:41.971 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:41.971 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:41.974 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:41.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:16:42.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:42.116+0000 7f2d4459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:42.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:42.123+0000 7f2d4459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:42.136 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:42.137 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:42.140 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:42.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:16:42.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:42.245+0000 7fa07f19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:42.253 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:42.253+0000 7fa07f19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:42.266 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:42.266 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:42.266 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:42.266 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:42.266 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:42.267 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:42.267 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:42.267 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:42.267 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:42.267 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:42.267 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:42.267 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:42.269 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:42.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:16:42.293 INFO:tasks.rados.rados.0.smithi064.stdout:2175: done (4 left) 2024-03-04T06:16:42.293 INFO:tasks.rados.rados.0.smithi064.stdout:2176: done (3 left) 2024-03-04T06:16:42.293 INFO:tasks.rados.rados.0.smithi064.stdout:2180: done (2 left) 2024-03-04T06:16:42.293 INFO:tasks.rados.rados.0.smithi064.stdout:2182: done (1 left) 2024-03-04T06:16:42.294 INFO:tasks.rados.rados.0.smithi064.stdout:2183: done (0 left) 2024-03-04T06:16:42.294 INFO:tasks.rados.rados.0.smithi064.stdout:2184: read oid 33 snap -1 2024-03-04T06:16:42.294 INFO:tasks.rados.rados.0.smithi064.stdout:2184: expect (ObjNum 1001 snap 259 seq_num 1001) 2024-03-04T06:16:42.294 INFO:tasks.rados.rados.0.smithi064.stdout:2185: write (excl) oid 39 current snap is 263 2024-03-04T06:16:42.294 INFO:tasks.rados.rados.0.smithi064.stdout:2185: seq_num 1026 ranges {443982=796776,1989231=479018,2959167=284815} 2024-03-04T06:16:42.300 INFO:tasks.rados.rados.0.smithi064.stdout:2185: writing smithi06431743-39 from 443982 to 1240758 tid 1 2024-03-04T06:16:42.308 INFO:tasks.rados.rados.0.smithi064.stdout:2185: writing smithi06431743-39 from 1989231 to 2468249 tid 2 2024-03-04T06:16:42.314 INFO:tasks.rados.rados.0.smithi064.stdout:2185: writing smithi06431743-39 from 2959167 to 3243982 tid 3 2024-03-04T06:16:42.314 INFO:tasks.rados.rados.0.smithi064.stdout:2186: read oid 16 snap -1 2024-03-04T06:16:42.314 INFO:tasks.rados.rados.0.smithi064.stdout:2186: expect (ObjNum 996 snap 254 seq_num 996) 2024-03-04T06:16:42.314 INFO:tasks.rados.rados.0.smithi064.stdout:2187: snap_remove snap 246 2024-03-04T06:16:42.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:42.381+0000 7f704f15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:42.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:42.388+0000 7f704f15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:42.402 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:16:42.402 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:16:42.402 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:16:42.402 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:16:42.402 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:16:42.403 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:16:42.403 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:16:42.403 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:16:42.403 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:16:42.403 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:16:42.403 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:16:42.403 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:16:42.404 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:16:42.405 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:16:43.261 INFO:tasks.rados.rados.0.smithi064.stdout:2187: done (3 left) 2024-03-04T06:16:43.261 INFO:tasks.rados.rados.0.smithi064.stdout:2188: write (excl) oid 10 current snap is 263 2024-03-04T06:16:43.261 INFO:tasks.rados.rados.0.smithi064.stdout:2188: seq_num 1027 ranges {460789=588378,1764066=739029,3219296=41493} 2024-03-04T06:16:43.267 INFO:tasks.rados.rados.0.smithi064.stdout:2188: writing smithi06431743-10 from 460789 to 1049167 tid 1 2024-03-04T06:16:43.275 INFO:tasks.rados.rados.0.smithi064.stdout:2188: writing smithi06431743-10 from 1764066 to 2503095 tid 2 2024-03-04T06:16:43.279 INFO:tasks.rados.rados.0.smithi064.stdout:2188: writing smithi06431743-10 from 3219296 to 3260789 tid 3 2024-03-04T06:16:43.279 INFO:tasks.rados.rados.0.smithi064.stdout:2189: write (excl) oid 2 current snap is 263 2024-03-04T06:16:43.279 INFO:tasks.rados.rados.0.smithi064.stdout:2189: seq_num 1028 ranges {477596=663627,1706195=482687,2614013=567041} 2024-03-04T06:16:43.285 INFO:tasks.rados.rados.0.smithi064.stdout:2189: writing smithi06431743-2 from 477596 to 1141223 tid 1 2024-03-04T06:16:43.294 INFO:tasks.rados.rados.0.smithi064.stdout:2189: writing smithi06431743-2 from 1706195 to 2188882 tid 2 2024-03-04T06:16:43.302 INFO:tasks.rados.rados.0.smithi064.stdout:2189: writing smithi06431743-2 from 2614013 to 3181054 tid 3 2024-03-04T06:16:43.302 INFO:tasks.rados.rados.0.smithi064.stdout:2190: snap_remove snap 254 2024-03-04T06:16:43.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:44.028 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:44.026+0000 7f2b3b064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:44.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:44.030+0000 7f2b3b064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:44.303 INFO:tasks.rados.rados.0.smithi064.stdout:2190: done (5 left) 2024-03-04T06:16:44.303 INFO:tasks.rados.rados.0.smithi064.stdout:2191: delete oid 24 current snap is 263 2024-03-04T06:16:44.304 INFO:tasks.rados.rados.0.smithi064.stdout:2186: expect (ObjNum 996 snap 254 seq_num 996) 2024-03-04T06:16:44.484 INFO:tasks.rados.rados.0.smithi064.stdout:2184: expect (ObjNum 1001 snap 259 seq_num 1001) 2024-03-04T06:16:44.679 INFO:tasks.rados.rados.0.smithi064.stdout:2185: finishing write tid 1 to smithi06431743-39 2024-03-04T06:16:44.680 INFO:tasks.rados.rados.0.smithi064.stdout:2185: finishing write tid 2 to smithi06431743-39 2024-03-04T06:16:44.680 INFO:tasks.rados.rados.0.smithi064.stdout:2185: finishing write tid 3 to smithi06431743-39 2024-03-04T06:16:44.680 INFO:tasks.rados.rados.0.smithi064.stdout:2185: finishing write tid 4 to smithi06431743-39 2024-03-04T06:16:44.680 INFO:tasks.rados.rados.0.smithi064.stdout:2185: finishing write tid 5 to smithi06431743-39 2024-03-04T06:16:44.680 INFO:tasks.rados.rados.0.smithi064.stdout:2185: oid 39 updating version 0 to 470 2024-03-04T06:16:44.680 INFO:tasks.rados.rados.0.smithi064.stdout:2185: oid 39 version 470 is already newer than 467 2024-03-04T06:16:44.680 INFO:tasks.rados.rados.0.smithi064.stdout:2185: oid 39 version 470 is already newer than 469 2024-03-04T06:16:44.680 INFO:tasks.rados.rados.0.smithi064.stdout:2185: oid 39 version 470 is already newer than 469 2024-03-04T06:16:44.680 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 470 (ObjNum 1026 snap 263 seq_num 1026) dirty exists 2024-03-04T06:16:44.681 INFO:tasks.rados.rados.0.smithi064.stdout:2185: left oid 39 (ObjNum 1026 snap 263 seq_num 1026) 2024-03-04T06:16:44.681 INFO:tasks.rados.rados.0.smithi064.stdout:2184: done (5 left) 2024-03-04T06:16:44.681 INFO:tasks.rados.rados.0.smithi064.stdout:2185: done (4 left) 2024-03-04T06:16:44.681 INFO:tasks.rados.rados.0.smithi064.stdout:2186: done (3 left) 2024-03-04T06:16:44.681 INFO:tasks.rados.rados.0.smithi064.stdout:2191: done (2 left) 2024-03-04T06:16:44.681 INFO:tasks.rados.rados.0.smithi064.stdout:2192: snap_create 2024-03-04T06:16:44.681 INFO:tasks.rados.rados.0.smithi064.stdout:2189: finishing write tid 1 to smithi06431743-2 2024-03-04T06:16:44.681 INFO:tasks.rados.rados.0.smithi064.stdout:2189: finishing write tid 2 to smithi06431743-2 2024-03-04T06:16:44.681 INFO:tasks.rados.rados.0.smithi064.stdout:2189: finishing write tid 3 to smithi06431743-2 2024-03-04T06:16:44.681 INFO:tasks.rados.rados.0.smithi064.stdout:2188: finishing write tid 1 to smithi06431743-10 2024-03-04T06:16:44.681 INFO:tasks.rados.rados.0.smithi064.stdout:2189: finishing write tid 4 to smithi06431743-2 2024-03-04T06:16:44.682 INFO:tasks.rados.rados.0.smithi064.stdout:2189: finishing write tid 5 to smithi06431743-2 2024-03-04T06:16:44.682 INFO:tasks.rados.rados.0.smithi064.stdout:2189: oid 2 updating version 0 to 329 2024-03-04T06:16:44.682 INFO:tasks.rados.rados.0.smithi064.stdout:2189: oid 2 version 329 is already newer than 329 2024-03-04T06:16:44.682 INFO:tasks.rados.rados.0.smithi064.stdout:2189: oid 2 updating version 329 to 330 2024-03-04T06:16:44.682 INFO:tasks.rados.rados.0.smithi064.stdout:2189: oid 2 version 330 is already newer than 327 2024-03-04T06:16:44.682 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 330 (ObjNum 1028 snap 263 seq_num 1028) dirty exists 2024-03-04T06:16:44.682 INFO:tasks.rados.rados.0.smithi064.stdout:2189: left oid 2 (ObjNum 1028 snap 263 seq_num 1028) 2024-03-04T06:16:44.682 INFO:tasks.rados.rados.0.smithi064.stdout:2188: finishing write tid 2 to smithi06431743-10 2024-03-04T06:16:44.682 INFO:tasks.rados.rados.0.smithi064.stdout:2188: finishing write tid 3 to smithi06431743-10 2024-03-04T06:16:44.682 INFO:tasks.rados.rados.0.smithi064.stdout:2188: finishing write tid 4 to smithi06431743-10 2024-03-04T06:16:44.682 INFO:tasks.rados.rados.0.smithi064.stdout:2188: finishing write tid 5 to smithi06431743-10 2024-03-04T06:16:44.683 INFO:tasks.rados.rados.0.smithi064.stdout:2188: oid 10 updating version 0 to 498 2024-03-04T06:16:44.683 INFO:tasks.rados.rados.0.smithi064.stdout:2188: oid 10 updating version 498 to 500 2024-03-04T06:16:44.683 INFO:tasks.rados.rados.0.smithi064.stdout:2188: oid 10 updating version 500 to 501 2024-03-04T06:16:44.683 INFO:tasks.rados.rados.0.smithi064.stdout:2188: oid 10 version 501 is already newer than 500 2024-03-04T06:16:44.683 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 501 (ObjNum 1027 snap 263 seq_num 1027) dirty exists 2024-03-04T06:16:44.683 INFO:tasks.rados.rados.0.smithi064.stdout:2188: left oid 10 (ObjNum 1027 snap 263 seq_num 1027) 2024-03-04T06:16:45.307 INFO:tasks.rados.rados.0.smithi064.stdout:2188: done (2 left) 2024-03-04T06:16:45.307 INFO:tasks.rados.rados.0.smithi064.stdout:2189: done (1 left) 2024-03-04T06:16:45.307 INFO:tasks.rados.rados.0.smithi064.stdout:2192: done (0 left) 2024-03-04T06:16:45.307 INFO:tasks.rados.rados.0.smithi064.stdout:2193: read oid 41 snap 258 2024-03-04T06:16:45.307 INFO:tasks.rados.rados.0.smithi064.stdout:2193: expect (ObjNum 985 snap 251 seq_num 985) 2024-03-04T06:16:45.307 INFO:tasks.rados.rados.0.smithi064.stdout:2194: write oid 39 current snap is 264 2024-03-04T06:16:45.307 INFO:tasks.rados.rados.0.smithi064.stdout:2194: seq_num 1030 ranges {511210=414125,1423159=486356,2668859=589585} 2024-03-04T06:16:45.311 INFO:tasks.rados.rados.0.smithi064.stdout:2194: writing smithi06431743-39 from 511210 to 925335 tid 1 2024-03-04T06:16:45.317 INFO:tasks.rados.rados.0.smithi064.stdout:2194: writing smithi06431743-39 from 1423159 to 1909515 tid 2 2024-03-04T06:16:45.324 INFO:tasks.rados.rados.0.smithi064.stdout:2194: writing smithi06431743-39 from 2668859 to 3258444 tid 3 2024-03-04T06:16:45.324 INFO:tasks.rados.rados.0.smithi064.stdout:2195: delete oid 11 current snap is 264 2024-03-04T06:16:45.359 INFO:tasks.rados.rados.0.smithi064.stdout:2193: expect (ObjNum 985 snap 251 seq_num 985) 2024-03-04T06:16:45.528 INFO:tasks.rados.rados.0.smithi064.stdout:2194: finishing write tid 1 to smithi06431743-39 2024-03-04T06:16:45.528 INFO:tasks.rados.rados.0.smithi064.stdout:2194: finishing write tid 2 to smithi06431743-39 2024-03-04T06:16:45.528 INFO:tasks.rados.rados.0.smithi064.stdout:2194: finishing write tid 3 to smithi06431743-39 2024-03-04T06:16:45.528 INFO:tasks.rados.rados.0.smithi064.stdout:2194: finishing write tid 4 to smithi06431743-39 2024-03-04T06:16:45.528 INFO:tasks.rados.rados.0.smithi064.stdout:2194: finishing write tid 5 to smithi06431743-39 2024-03-04T06:16:45.528 INFO:tasks.rados.rados.0.smithi064.stdout:2194: oid 39 updating version 0 to 479 2024-03-04T06:16:45.528 INFO:tasks.rados.rados.0.smithi064.stdout:2194: oid 39 version 479 is already newer than 476 2024-03-04T06:16:45.528 INFO:tasks.rados.rados.0.smithi064.stdout:2194: oid 39 version 479 is already newer than 477 2024-03-04T06:16:45.528 INFO:tasks.rados.rados.0.smithi064.stdout:2194: oid 39 version 479 is already newer than 478 2024-03-04T06:16:45.529 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 479 (ObjNum 1030 snap 264 seq_num 1030) dirty exists 2024-03-04T06:16:45.529 INFO:tasks.rados.rados.0.smithi064.stdout:2194: left oid 39 (ObjNum 1030 snap 264 seq_num 1030) 2024-03-04T06:16:45.529 INFO:tasks.rados.rados.0.smithi064.stdout:2193: done (2 left) 2024-03-04T06:16:45.529 INFO:tasks.rados.rados.0.smithi064.stdout:2194: done (1 left) 2024-03-04T06:16:45.529 INFO:tasks.rados.rados.0.smithi064.stdout:2195: done (0 left) 2024-03-04T06:16:45.529 INFO:tasks.rados.rados.0.smithi064.stdout:2196: snap_remove snap 261 2024-03-04T06:16:46.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:46.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:46.212+0000 7f123846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:46.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:46.217+0000 7f123846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:46.307 INFO:tasks.rados.rados.0.smithi064.stdout:2196: done (0 left) 2024-03-04T06:16:46.307 INFO:tasks.rados.rados.0.smithi064.stdout:2197: snap_remove snap 262 2024-03-04T06:16:47.313 INFO:tasks.rados.rados.0.smithi064.stdout:2197: done (0 left) 2024-03-04T06:16:47.313 INFO:tasks.rados.rados.0.smithi064.stdout:2198: write oid 3 current snap is 264 2024-03-04T06:16:47.313 INFO:tasks.rados.rados.0.smithi064.stdout:2198: seq_num 1032 ranges {544824=564623,1823770=490025,3007352=337472} 2024-03-04T06:16:47.319 INFO:tasks.rados.rados.0.smithi064.stdout:2198: writing smithi06431743-3 from 544824 to 1109447 tid 1 2024-03-04T06:16:47.326 INFO:tasks.rados.rados.0.smithi064.stdout:2198: writing smithi06431743-3 from 1823770 to 2313795 tid 2 2024-03-04T06:16:47.332 INFO:tasks.rados.rados.0.smithi064.stdout:2198: writing smithi06431743-3 from 3007352 to 3344824 tid 3 2024-03-04T06:16:47.332 INFO:tasks.rados.rados.0.smithi064.stdout:2199: read oid 23 snap 259 2024-03-04T06:16:47.332 INFO:tasks.rados.rados.0.smithi064.stdout:2199: expect (ObjNum 923 snap 226 seq_num 923) 2024-03-04T06:16:47.332 INFO:tasks.rados.rados.0.smithi064.stdout:2200: write oid 43 current snap is 264 2024-03-04T06:16:47.332 INFO:tasks.rados.rados.0.smithi064.stdout:2200: seq_num 1033 ranges {561631=639872,1882252=633683,3034775=326856} 2024-03-04T06:16:47.338 INFO:tasks.rados.rados.0.smithi064.stdout:2200: writing smithi06431743-43 from 561631 to 1201503 tid 1 2024-03-04T06:16:47.347 INFO:tasks.rados.rados.0.smithi064.stdout:2200: writing smithi06431743-43 from 1882252 to 2515935 tid 2 2024-03-04T06:16:47.354 INFO:tasks.rados.rados.0.smithi064.stdout:2200: writing smithi06431743-43 from 3034775 to 3361631 tid 3 2024-03-04T06:16:47.354 INFO:tasks.rados.rados.0.smithi064.stdout:2201: snap_create 2024-03-04T06:16:47.391 INFO:tasks.rados.rados.0.smithi064.stdout:2200: finishing write tid 1 to smithi06431743-43 2024-03-04T06:16:47.392 INFO:tasks.rados.rados.0.smithi064.stdout:2198: finishing write tid 1 to smithi06431743-3 2024-03-04T06:16:47.405 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:16:47.405 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:16:47.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:16:47.413 INFO:tasks.rados.rados.0.smithi064.stdout:2200: finishing write tid 2 to smithi06431743-43 2024-03-04T06:16:47.413 INFO:tasks.rados.rados.0.smithi064.stdout:2198: finishing write tid 2 to smithi06431743-3 2024-03-04T06:16:47.419 INFO:tasks.rados.rados.0.smithi064.stdout:2198: finishing write tid 3 to smithi06431743-3 2024-03-04T06:16:47.421 INFO:tasks.rados.rados.0.smithi064.stdout:2200: finishing write tid 3 to smithi06431743-43 2024-03-04T06:16:47.429 INFO:tasks.rados.rados.0.smithi064.stdout:2200: finishing write tid 4 to smithi06431743-43 2024-03-04T06:16:47.429 INFO:tasks.rados.rados.0.smithi064.stdout:2198: finishing write tid 4 to smithi06431743-3 2024-03-04T06:16:47.430 INFO:tasks.rados.rados.0.smithi064.stdout:2200: finishing write tid 5 to smithi06431743-43 2024-03-04T06:16:47.430 INFO:tasks.rados.rados.0.smithi064.stdout:2200: oid 43 updating version 0 to 455 2024-03-04T06:16:47.431 INFO:tasks.rados.rados.0.smithi064.stdout:2200: oid 43 version 455 is already newer than 455 2024-03-04T06:16:47.431 INFO:tasks.rados.rados.0.smithi064.stdout:2200: oid 43 version 455 is already newer than 453 2024-03-04T06:16:47.431 INFO:tasks.rados.rados.0.smithi064.stdout:2200: oid 43 updating version 455 to 456 2024-03-04T06:16:47.431 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 456 (ObjNum 1033 snap 264 seq_num 1033) dirty exists 2024-03-04T06:16:47.431 INFO:tasks.rados.rados.0.smithi064.stdout:2200: left oid 43 (ObjNum 1033 snap 264 seq_num 1033) 2024-03-04T06:16:47.436 INFO:tasks.rados.rados.0.smithi064.stdout:2198: finishing write tid 5 to smithi06431743-3 2024-03-04T06:16:47.436 INFO:tasks.rados.rados.0.smithi064.stdout:2198: oid 3 updating version 0 to 504 2024-03-04T06:16:47.436 INFO:tasks.rados.rados.0.smithi064.stdout:2198: oid 3 version 504 is already newer than 504 2024-03-04T06:16:47.436 INFO:tasks.rados.rados.0.smithi064.stdout:2198: oid 3 version 504 is already newer than 502 2024-03-04T06:16:47.436 INFO:tasks.rados.rados.0.smithi064.stdout:2198: oid 3 updating version 504 to 505 2024-03-04T06:16:47.436 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 505 (ObjNum 1032 snap 264 seq_num 1032) dirty exists 2024-03-04T06:16:47.436 INFO:tasks.rados.rados.0.smithi064.stdout:2198: left oid 3 (ObjNum 1032 snap 264 seq_num 1032) 2024-03-04T06:16:47.451 INFO:tasks.rados.rados.0.smithi064.stdout:2199: expect (ObjNum 923 snap 226 seq_num 923) 2024-03-04T06:16:47.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:47.518+0000 7f49d1f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:47.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:47.524+0000 7f49d1f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:47.716 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:16:47.717 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:16:47.728 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":799,"stamp":"2024-03-04T06:16:47.311682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95372433,"num_objects":770,"num_object_clones":589,"num_object_copies":2310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":146,"num_read":4113,"num_read_kb":3096245,"num_write":3572,"num_write_kb":935919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7438,"ondisk_log_size":7438,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":495036,"kb_used_data":305884,"kb_used_omap":0,"kb_used_meta":189120,"kb_avail":376992324,"statfs":{"total":386547056640,"available":386040139776,"internally_reserved":0,"allocated":313225216,"data_stored":347131559,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":193658880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9815783,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":41,"num_read_kb":25288,"num_write":46,"num_write_kb":12305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.052790"},"pg_stats":[{"pgid":"3.f","version":"533'451","reported_seq":1875,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-04T06:16:44.334182+0000","last_change":"2024-03-04T06:16:44.342760+0000","last_active":"2024-03-04T06:16:44.334182+0000","last_peered":"2024-03-04T06:16:44.334182+0000","last_clean":"2024-03-04T06:16:44.334182+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:16:44.334182+0000","last_undegraded":"2024-03-04T06:16:44.334182+0000","last_fullsized":"2024-03-04T06:16:44.334182+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7866369,"num_objects":50,"num_object_clones":39,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":249,"num_read_kb":201026,"num_write":210,"num_write_kb":56814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"534'343","reported_seq":1805,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.320534+0000","last_change":"2024-03-04T06:16:45.333205+0000","last_active":"2024-03-04T06:16:45.320534+0000","last_peered":"2024-03-04T06:16:45.320534+0000","last_clean":"2024-03-04T06:16:45.320534+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:16:45.320534+0000","last_undegraded":"2024-03-04T06:16:45.320534+0000","last_fullsized":"2024-03-04T06:16:45.320534+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8757140,"num_objects":41,"num_object_clones":27,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":134,"num_read_kb":80770,"num_write":165,"num_write_kb":43312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"535'479","reported_seq":1949,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.394586+0000","last_change":"2024-03-04T06:16:44.461336+0000","last_active":"2024-03-04T06:16:45.394586+0000","last_peered":"2024-03-04T06:16:45.394586+0000","last_clean":"2024-03-04T06:16:45.394586+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:16:45.394586+0000","last_undegraded":"2024-03-04T06:16:45.394586+0000","last_fullsized":"2024-03-04T06:16:45.394586+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7160211,"num_objects":59,"num_object_clones":46,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":329,"num_read_kb":259518,"num_write":252,"num_write_kb":69591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"534'332","reported_seq":1816,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-04T06:16:44.326721+0000","last_change":"2024-03-04T06:16:44.365701+0000","last_active":"2024-03-04T06:16:44.326721+0000","last_peered":"2024-03-04T06:16:44.326721+0000","last_clean":"2024-03-04T06:16:44.326721+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:16:44.326721+0000","last_undegraded":"2024-03-04T06:16:44.326721+0000","last_fullsized":"2024-03-04T06:16:44.326721+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185372,"num_objects":35,"num_object_clones":25,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":193,"num_read_kb":149127,"num_write":169,"num_write_kb":47053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1659,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-04T06:16:44.357614+0000","last_change":"2024-03-04T06:16:44.362544+0000","last_active":"2024-03-04T06:16:44.357614+0000","last_peered":"2024-03-04T06:16:44.357614+0000","last_clean":"2024-03-04T06:16:44.357614+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:16:44.357614+0000","last_undegraded":"2024-03-04T06:16:44.357614+0000","last_fullsized":"2024-03-04T06:16:44.357614+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":58,"num_read_kb":33096,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"534'507","reported_seq":1906,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-04T06:16:44.337198+0000","last_change":"2024-03-04T06:16:44.401738+0000","last_active":"2024-03-04T06:16:44.337198+0000","last_peered":"2024-03-04T06:16:44.337198+0000","last_clean":"2024-03-04T06:16:44.337198+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:16:44.337198+0000","last_undegraded":"2024-03-04T06:16:44.337198+0000","last_fullsized":"2024-03-04T06:16:44.337198+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5950140,"num_objects":49,"num_object_clones":40,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":251,"num_read_kb":192235,"num_write":245,"num_write_kb":71955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1576,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-04T06:16:44.338740+0000","last_change":"2024-03-04T06:16:44.342338+0000","last_active":"2024-03-04T06:16:44.338740+0000","last_peered":"2024-03-04T06:16:44.338740+0000","last_clean":"2024-03-04T06:16:44.338740+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:16:44.338740+0000","last_undegraded":"2024-03-04T06:16:44.338740+0000","last_fullsized":"2024-03-04T06:16:44.338740+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"507'249","reported_seq":1762,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.321420+0000","last_change":"2024-03-04T06:16:44.436190+0000","last_active":"2024-03-04T06:16:45.321420+0000","last_peered":"2024-03-04T06:16:45.321420+0000","last_clean":"2024-03-04T06:16:45.321420+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:16:45.321420+0000","last_undegraded":"2024-03-04T06:16:45.321420+0000","last_fullsized":"2024-03-04T06:16:45.321420+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5378636,"num_objects":26,"num_object_clones":17,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":174,"num_read_kb":163220,"num_write":118,"num_write_kb":30955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1457,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-04T06:16:44.332084+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:16:44.332084+0000","last_peered":"2024-03-04T06:16:44.332084+0000","last_clean":"2024-03-04T06:16:44.332084+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:16:44.332084+0000","last_undegraded":"2024-03-04T06:16:44.332084+0000","last_fullsized":"2024-03-04T06:16:44.332084+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1333,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.328718+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:16:45.328718+0000","last_peered":"2024-03-04T06:16:45.328718+0000","last_clean":"2024-03-04T06:16:45.328718+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:16:45.328718+0000","last_undegraded":"2024-03-04T06:16:45.328718+0000","last_fullsized":"2024-03-04T06:16:45.328718+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"534'501","reported_seq":1888,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.354413+0000","last_change":"2024-03-04T06:16:44.471930+0000","last_active":"2024-03-04T06:16:45.354413+0000","last_peered":"2024-03-04T06:16:45.354413+0000","last_clean":"2024-03-04T06:16:45.354413+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:16:45.354413+0000","last_undegraded":"2024-03-04T06:16:45.354413+0000","last_fullsized":"2024-03-04T06:16:45.354413+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554895,"num_objects":52,"num_object_clones":40,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":244,"num_read_kb":188468,"num_write":229,"num_write_kb":62351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1332,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-04T06:16:44.333621+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:16:44.333621+0000","last_peered":"2024-03-04T06:16:44.333621+0000","last_clean":"2024-03-04T06:16:44.333621+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:16:44.333621+0000","last_undegraded":"2024-03-04T06:16:44.333621+0000","last_fullsized":"2024-03-04T06:16:44.333621+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"531'366","reported_seq":1786,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-04T06:16:44.350729+0000","last_change":"2024-03-04T06:16:44.364045+0000","last_active":"2024-03-04T06:16:44.350729+0000","last_peered":"2024-03-04T06:16:44.350729+0000","last_clean":"2024-03-04T06:16:44.350729+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:16:44.350729+0000","last_undegraded":"2024-03-04T06:16:44.350729+0000","last_fullsized":"2024-03-04T06:16:44.350729+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3143140,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":204,"num_read_kb":166620,"num_write":141,"num_write_kb":40250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1333,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.322053+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:16:45.322053+0000","last_peered":"2024-03-04T06:16:45.322053+0000","last_clean":"2024-03-04T06:16:45.322053+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:16:45.322053+0000","last_undegraded":"2024-03-04T06:16:45.322053+0000","last_fullsized":"2024-03-04T06:16:45.322053+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"531'841","reported_seq":2173,"reported_epoch":534,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:44.350567+0000","last_change":"2024-03-04T06:16:44.350567+0000","last_active":"2024-03-04T06:16:44.350567+0000","last_peered":"2024-03-04T06:16:44.350567+0000","last_clean":"2024-03-04T06:16:44.350567+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:16:44.350567+0000","last_undegraded":"2024-03-04T06:16:44.350567+0000","last_fullsized":"2024-03-04T06:16:44.350567+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10774211,"num_objects":88,"num_object_clones":67,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":525,"num_read_kb":421074,"num_write":412,"num_write_kb":110526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1332,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-04T06:16:44.327464+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:16:44.327464+0000","last_peered":"2024-03-04T06:16:44.327464+0000","last_clean":"2024-03-04T06:16:44.327464+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:16:44.327464+0000","last_undegraded":"2024-03-04T06:16:44.327464+0000","last_fullsized":"2024-03-04T06:16:44.327464+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"535'933","reported_seq":2148,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.322635+0000","last_change":"2024-03-04T06:16:45.344568+0000","last_active":"2024-03-04T06:16:45.322635+0000","last_peered":"2024-03-04T06:16:45.322635+0000","last_clean":"2024-03-04T06:16:45.322635+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:16:45.322635+0000","last_undegraded":"2024-03-04T06:16:45.322635+0000","last_fullsized":"2024-03-04T06:16:45.322635+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11530493,"num_objects":92,"num_object_clones":71,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":395,"num_read_kb":277931,"num_write":422,"num_write_kb":110956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"534'416","reported_seq":1890,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.330669+0000","last_change":"2024-03-04T06:16:44.471567+0000","last_active":"2024-03-04T06:16:45.330669+0000","last_peered":"2024-03-04T06:16:45.330669+0000","last_clean":"2024-03-04T06:16:45.330669+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:16:45.330669+0000","last_undegraded":"2024-03-04T06:16:45.330669+0000","last_fullsized":"2024-03-04T06:16:45.330669+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135017,"num_objects":44,"num_object_clones":32,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":293,"num_read_kb":229913,"num_write":199,"num_write_kb":53350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1359,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.328485+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:16:45.328485+0000","last_peered":"2024-03-04T06:16:45.328485+0000","last_clean":"2024-03-04T06:16:45.328485+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:16:45.328485+0000","last_undegraded":"2024-03-04T06:16:45.328485+0000","last_fullsized":"2024-03-04T06:16:45.328485+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"534'422","reported_seq":1875,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.324706+0000","last_change":"2024-03-04T06:16:45.331405+0000","last_active":"2024-03-04T06:16:45.324706+0000","last_peered":"2024-03-04T06:16:45.324706+0000","last_clean":"2024-03-04T06:16:45.324706+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:16:45.324706+0000","last_undegraded":"2024-03-04T06:16:45.324706+0000","last_fullsized":"2024-03-04T06:16:45.324706+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2739772,"num_objects":51,"num_object_clones":36,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":220,"num_read_kb":149342,"num_write":200,"num_write_kb":52704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1324,"reported_epoch":534,"state":"active+clean","last_fresh":"2024-03-04T06:16:44.350784+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:16:44.350784+0000","last_peered":"2024-03-04T06:16:44.350784+0000","last_clean":"2024-03-04T06:16:44.350784+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:16:44.350784+0000","last_undegraded":"2024-03-04T06:16:44.350784+0000","last_fullsized":"2024-03-04T06:16:44.350784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"534'605","reported_seq":1960,"reported_epoch":534,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:44.341213+0000","last_change":"2024-03-04T06:16:44.341018+0000","last_active":"2024-03-04T06:16:44.341213+0000","last_peered":"2024-03-04T06:16:44.341213+0000","last_clean":"2024-03-04T06:16:44.341213+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:16:44.341213+0000","last_undegraded":"2024-03-04T06:16:44.341213+0000","last_fullsized":"2024-03-04T06:16:44.341213+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882912,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":322,"num_read_kb":248370,"num_write":282,"num_write_kb":78197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1359,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.321324+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:16:45.321324+0000","last_peered":"2024-03-04T06:16:45.321324+0000","last_clean":"2024-03-04T06:16:45.321324+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:16:45.321324+0000","last_undegraded":"2024-03-04T06:16:45.321324+0000","last_fullsized":"2024-03-04T06:16:45.321324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"534'729","reported_seq":2074,"reported_epoch":535,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:45.317666+0000","last_change":"2024-03-04T06:16:44.323726+0000","last_active":"2024-03-04T06:16:45.317666+0000","last_peered":"2024-03-04T06:16:45.317666+0000","last_clean":"2024-03-04T06:16:45.317666+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:16:45.317666+0000","last_undegraded":"2024-03-04T06:16:45.317666+0000","last_fullsized":"2024-03-04T06:16:45.317666+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395773,"num_objects":68,"num_object_clones":59,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":419,"num_read_kb":335444,"num_write":316,"num_write_kb":86049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1333,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-04T06:16:45.328078+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:16:45.328078+0000","last_peered":"2024-03-04T06:16:45.328078+0000","last_clean":"2024-03-04T06:16:45.328078+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:16:45.328078+0000","last_undegraded":"2024-03-04T06:16:45.328078+0000","last_fullsized":"2024-03-04T06:16:45.328078+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94782065,"num_objects":768,"num_object_clones":589,"num_object_copies":2304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":146,"num_read":4010,"num_read_kb":3096154,"num_write":3435,"num_write_kb":934025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7341,"ondisk_log_size":7341,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":142624,"kb_used_data":93592,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94229216,"statfs":{"total":96636764160,"available":96490717184,"internally_reserved":0,"allocated":95838208,"data_stored":106378292,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":141932,"kb_used_data":80420,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94229908,"statfs":{"total":96636764160,"available":96491425792,"internally_reserved":0,"allocated":82350080,"data_stored":92101587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":106180,"kb_used_data":66876,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94265660,"statfs":{"total":96636764160,"available":96528035840,"internally_reserved":0,"allocated":68481024,"data_stored":75777917,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706203,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":104300,"kb_used_data":64996,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94267540,"statfs":{"total":96636764160,"available":96529960960,"internally_reserved":0,"allocated":66555904,"data_stored":72873763,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:16:48.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:48.315 INFO:tasks.rados.rados.0.smithi064.stdout:2198: done (3 left) 2024-03-04T06:16:48.315 INFO:tasks.rados.rados.0.smithi064.stdout:2199: done (2 left) 2024-03-04T06:16:48.315 INFO:tasks.rados.rados.0.smithi064.stdout:2200: done (1 left) 2024-03-04T06:16:48.315 INFO:tasks.rados.rados.0.smithi064.stdout:2201: done (0 left) 2024-03-04T06:16:48.316 INFO:tasks.rados.rados.0.smithi064.stdout:2202: read oid 35 snap 253 2024-03-04T06:16:48.316 INFO:tasks.rados.rados.0.smithi064.stdout:2202: expect deleted 2024-03-04T06:16:48.316 INFO:tasks.rados.rados.0.smithi064.stdout:2203: write oid 15 current snap is 265 2024-03-04T06:16:48.316 INFO:tasks.rados.rados.0.smithi064.stdout:2203: seq_num 1034 ranges {578438=431474,1657087=493694,2778551=599887} 2024-03-04T06:16:48.321 INFO:tasks.rados.rados.0.smithi064.stdout:2203: writing smithi06431743-15 from 578438 to 1009912 tid 1 2024-03-04T06:16:48.327 INFO:tasks.rados.rados.0.smithi064.stdout:2203: writing smithi06431743-15 from 1657087 to 2150781 tid 2 2024-03-04T06:16:48.334 INFO:tasks.rados.rados.0.smithi064.stdout:2203: writing smithi06431743-15 from 2778551 to 3378438 tid 3 2024-03-04T06:16:48.335 INFO:tasks.rados.rados.0.smithi064.stdout:2204: delete oid 30 current snap is 265 2024-03-04T06:16:48.401 INFO:tasks.rados.rados.0.smithi064.stdout:2203: finishing write tid 1 to smithi06431743-15 2024-03-04T06:16:48.409 INFO:tasks.rados.rados.0.smithi064.stdout:2202: done (2 left) 2024-03-04T06:16:48.409 INFO:tasks.rados.rados.0.smithi064.stdout:2204: done (1 left) 2024-03-04T06:16:48.410 INFO:tasks.rados.rados.0.smithi064.stdout:2205: delete oid 8 current snap is 265 2024-03-04T06:16:48.413 INFO:tasks.rados.rados.0.smithi064.stdout:2205: done (1 left) 2024-03-04T06:16:48.413 INFO:tasks.rados.rados.0.smithi064.stdout:2206: delete oid 9 current snap is 265 2024-03-04T06:16:48.415 INFO:tasks.rados.rados.0.smithi064.stdout:2203: finishing write tid 2 to smithi06431743-15 2024-03-04T06:16:48.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:48.415+0000 7f582c2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:48.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:48.420+0000 7f582c2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:48.433 INFO:tasks.rados.rados.0.smithi064.stdout:2203: finishing write tid 3 to smithi06431743-15 2024-03-04T06:16:48.438 INFO:tasks.rados.rados.0.smithi064.stdout:2206: done (1 left) 2024-03-04T06:16:48.438 INFO:tasks.rados.rados.0.smithi064.stdout:2207: write oid 16 current snap is 265 2024-03-04T06:16:48.438 INFO:tasks.rados.rados.0.smithi064.stdout:2207: seq_num 1038 ranges {645666=732470,2174662=501032,3288243=157423} 2024-03-04T06:16:48.446 INFO:tasks.rados.rados.0.smithi064.stdout:2207: writing smithi06431743-16 from 645666 to 1378136 tid 1 2024-03-04T06:16:48.453 INFO:tasks.rados.rados.0.smithi064.stdout:2207: writing smithi06431743-16 from 2174662 to 2675694 tid 2 2024-03-04T06:16:48.458 INFO:tasks.rados.rados.0.smithi064.stdout:2207: writing smithi06431743-16 from 3288243 to 3445666 tid 3 2024-03-04T06:16:48.458 INFO:tasks.rados.rados.0.smithi064.stdout:2208: delete oid 30 current snap is 265 2024-03-04T06:16:48.458 INFO:tasks.rados.rados.0.smithi064.stdout:2203: finishing write tid 4 to smithi06431743-15 2024-03-04T06:16:48.458 INFO:tasks.rados.rados.0.smithi064.stdout:2203: finishing write tid 5 to smithi06431743-15 2024-03-04T06:16:48.458 INFO:tasks.rados.rados.0.smithi064.stdout:2203: oid 15 updating version 0 to 419 2024-03-04T06:16:48.458 INFO:tasks.rados.rados.0.smithi064.stdout:2203: oid 15 updating version 419 to 421 2024-03-04T06:16:48.458 INFO:tasks.rados.rados.0.smithi064.stdout:2203: oid 15 version 421 is already newer than 418 2024-03-04T06:16:48.458 INFO:tasks.rados.rados.0.smithi064.stdout:2203: oid 15 version 421 is already newer than 420 2024-03-04T06:16:48.458 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 421 (ObjNum 1034 snap 265 seq_num 1034) dirty exists 2024-03-04T06:16:48.458 INFO:tasks.rados.rados.0.smithi064.stdout:2203: left oid 15 (ObjNum 1034 snap 265 seq_num 1034) 2024-03-04T06:16:48.465 INFO:tasks.rados.rados.0.smithi064.stdout:2203: done (2 left) 2024-03-04T06:16:48.466 INFO:tasks.rados.rados.0.smithi064.stdout:2208: done (1 left) 2024-03-04T06:16:48.466 INFO:tasks.rados.rados.0.smithi064.stdout:2209: write (excl) oid 49 current snap is 265 2024-03-04T06:16:48.466 INFO:tasks.rados.rados.0.smithi064.stdout:2209: seq_num 1040 ranges {679280=482968,1775273=788348,3110383=368897} 2024-03-04T06:16:48.472 INFO:tasks.rados.rados.0.smithi064.stdout:2209: writing smithi06431743-49 from 679280 to 1162248 tid 1 2024-03-04T06:16:48.479 INFO:tasks.rados.rados.0.smithi064.stdout:2209: writing smithi06431743-49 from 1775273 to 2563621 tid 2 2024-03-04T06:16:48.485 INFO:tasks.rados.rados.0.smithi064.stdout:2209: writing smithi06431743-49 from 3110383 to 3479280 tid 3 2024-03-04T06:16:48.485 INFO:tasks.rados.rados.0.smithi064.stdout:2210: delete oid 30 current snap is 265 2024-03-04T06:16:48.485 INFO:tasks.rados.rados.0.smithi064.stdout:2207: finishing write tid 1 to smithi06431743-16 2024-03-04T06:16:48.488 INFO:tasks.rados.rados.0.smithi064.stdout:2210: done (2 left) 2024-03-04T06:16:48.488 INFO:tasks.rados.rados.0.smithi064.stdout:2211: write (excl) oid 33 current snap is 265 2024-03-04T06:16:48.488 INFO:tasks.rados.rados.0.smithi064.stdout:2211: seq_num 1042 ranges {712894=749819,2008590=792017,3281582=231312} 2024-03-04T06:16:48.495 INFO:tasks.rados.rados.0.smithi064.stdout:2211: writing smithi06431743-33 from 712894 to 1462713 tid 1 2024-03-04T06:16:48.504 INFO:tasks.rados.rados.0.smithi064.stdout:2211: writing smithi06431743-33 from 2008590 to 2800607 tid 2 2024-03-04T06:16:48.511 INFO:tasks.rados.rados.0.smithi064.stdout:2211: writing smithi06431743-33 from 3281582 to 3512894 tid 3 2024-03-04T06:16:48.512 INFO:tasks.rados.rados.0.smithi064.stdout:2212: read oid 42 snap 256 2024-03-04T06:16:48.512 INFO:tasks.rados.rados.0.smithi064.stdout:2212: expect (ObjNum 994 snap 254 seq_num 994) 2024-03-04T06:16:48.512 INFO:tasks.rados.rados.0.smithi064.stdout:2213: write oid 13 current snap is 265 2024-03-04T06:16:48.512 INFO:tasks.rados.rados.0.smithi064.stdout:2213: seq_num 1043 ranges {729701=425068,1667072=652028,2909005=485180} 2024-03-04T06:16:48.516 INFO:tasks.rados.rados.0.smithi064.stdout:2213: writing smithi06431743-13 from 729701 to 1154769 tid 1 2024-03-04T06:16:48.523 INFO:tasks.rados.rados.0.smithi064.stdout:2213: writing smithi06431743-13 from 1667072 to 2319100 tid 2 2024-03-04T06:16:48.531 INFO:tasks.rados.rados.0.smithi064.stdout:2213: writing smithi06431743-13 from 2909005 to 3394185 tid 3 2024-03-04T06:16:48.532 INFO:tasks.rados.rados.0.smithi064.stdout:2214: read oid 9 snap -1 2024-03-04T06:16:48.532 INFO:tasks.rados.rados.0.smithi064.stdout:2214: expect deleted 2024-03-04T06:16:48.532 INFO:tasks.rados.rados.0.smithi064.stdout:2215: write (excl) oid 15 current snap is 265 2024-03-04T06:16:48.532 INFO:tasks.rados.rados.0.smithi064.stdout:2209: finishing write tid 1 to smithi06431743-49 2024-03-04T06:16:48.532 INFO:tasks.rados.rados.0.smithi064.stdout:2207: finishing write tid 2 to smithi06431743-16 2024-03-04T06:16:48.532 INFO:tasks.rados.rados.0.smithi064.stdout:2207: finishing write tid 3 to smithi06431743-16 2024-03-04T06:16:48.532 INFO:tasks.rados.rados.0.smithi064.stdout:2209: finishing write tid 2 to smithi06431743-49 2024-03-04T06:16:48.532 INFO:tasks.rados.rados.0.smithi064.stdout:2215: seq_num 1044 ranges {746508=500317,2009201=795686,3220075=326433} 2024-03-04T06:16:48.538 INFO:tasks.rados.rados.0.smithi064.stdout:2215: writing smithi06431743-15 from 746508 to 1246825 tid 1 2024-03-04T06:16:48.547 INFO:tasks.rados.rados.0.smithi064.stdout:2215: writing smithi06431743-15 from 2009201 to 2804887 tid 2 2024-03-04T06:16:48.552 INFO:tasks.rados.rados.0.smithi064.stdout:2215: writing smithi06431743-15 from 3220075 to 3546508 tid 3 2024-03-04T06:16:48.552 INFO:tasks.rados.rados.0.smithi064.stdout:2216: read oid 36 snap -1 2024-03-04T06:16:48.552 INFO:tasks.rados.rados.0.smithi064.stdout:2216: expect deleted 2024-03-04T06:16:48.552 INFO:tasks.rados.rados.0.smithi064.stdout:2217: write oid 21 current snap is 265 2024-03-04T06:16:48.552 INFO:tasks.rados.rados.0.smithi064.stdout:2217: seq_num 1045 ranges {763315=575566,2067683=655697,3247498=315817} 2024-03-04T06:16:48.559 INFO:tasks.rados.rados.0.smithi064.stdout:2217: writing smithi06431743-21 from 763315 to 1338881 tid 1 2024-03-04T06:16:48.566 INFO:tasks.rados.rados.0.smithi064.stdout:2217: writing smithi06431743-21 from 2067683 to 2723380 tid 2 2024-03-04T06:16:48.571 INFO:tasks.rados.rados.0.smithi064.stdout:2217: writing smithi06431743-21 from 3247498 to 3563315 tid 3 2024-03-04T06:16:48.571 INFO:tasks.rados.rados.0.smithi064.stdout:2218: write (excl) oid 1 current snap is 265 2024-03-04T06:16:48.571 INFO:tasks.rados.rados.0.smithi064.stdout:2212: expect (ObjNum 994 snap 254 seq_num 994) 2024-03-04T06:16:48.795 INFO:tasks.rados.rados.0.smithi064.stdout:2207: finishing write tid 4 to smithi06431743-16 2024-03-04T06:16:48.795 INFO:tasks.rados.rados.0.smithi064.stdout:2211: finishing write tid 1 to smithi06431743-33 2024-03-04T06:16:48.795 INFO:tasks.rados.rados.0.smithi064.stdout:2207: finishing write tid 5 to smithi06431743-16 2024-03-04T06:16:48.796 INFO:tasks.rados.rados.0.smithi064.stdout:2207: oid 16 updating version 0 to 427 2024-03-04T06:16:48.796 INFO:tasks.rados.rados.0.smithi064.stdout:2207: oid 16 version 427 is already newer than 426 2024-03-04T06:16:48.796 INFO:tasks.rados.rados.0.smithi064.stdout:2207: oid 16 version 427 is already newer than 424 2024-03-04T06:16:48.796 INFO:tasks.rados.rados.0.smithi064.stdout:2207: oid 16 version 427 is already newer than 425 2024-03-04T06:16:48.796 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 427 (ObjNum 1038 snap 265 seq_num 1038) dirty exists 2024-03-04T06:16:48.796 INFO:tasks.rados.rados.0.smithi064.stdout:2207: left oid 16 (ObjNum 1038 snap 265 seq_num 1038) 2024-03-04T06:16:48.796 INFO:tasks.rados.rados.0.smithi064.stdout:2213: finishing write tid 1 to smithi06431743-13 2024-03-04T06:16:48.796 INFO:tasks.rados.rados.0.smithi064.stdout:2218: seq_num 1046 ranges {780122=650815,2126165=799355} 2024-03-04T06:16:48.803 INFO:tasks.rados.rados.0.smithi064.stdout:2218: writing smithi06431743-1 from 780122 to 1430937 tid 1 2024-03-04T06:16:48.811 INFO:tasks.rados.rados.0.smithi064.stdout:2218: writing smithi06431743-1 from 2126165 to 2925520 tid 2 2024-03-04T06:16:48.811 INFO:tasks.rados.rados.0.smithi064.stdout:2207: done (9 left) 2024-03-04T06:16:48.811 INFO:tasks.rados.rados.0.smithi064.stdout:2212: done (8 left) 2024-03-04T06:16:48.811 INFO:tasks.rados.rados.0.smithi064.stdout:2219: snap_remove snap 249 2024-03-04T06:16:49.323 INFO:tasks.rados.rados.0.smithi064.stdout:2219: done (8 left) 2024-03-04T06:16:49.323 INFO:tasks.rados.rados.0.smithi064.stdout:2220: snap_remove snap 264 2024-03-04T06:16:50.322 INFO:tasks.rados.rados.0.smithi064.stdout:2220: done (8 left) 2024-03-04T06:16:50.322 INFO:tasks.rados.rados.0.smithi064.stdout:2221: write (excl) oid 4 current snap is 265 2024-03-04T06:16:50.322 INFO:tasks.rados.rados.0.smithi064.stdout:2221: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2024-03-04T06:16:50.330 INFO:tasks.rados.rados.0.smithi064.stdout:2221: writing smithi06431743-4 from 796929 to 1522993 tid 1 2024-03-04T06:16:50.337 INFO:tasks.rados.rados.0.smithi064.stdout:2221: writing smithi06431743-4 from 2184647 to 2844013 tid 2 2024-03-04T06:16:50.342 INFO:tasks.rados.rados.0.smithi064.stdout:2221: writing smithi06431743-4 from 3302344 to 3596929 tid 3 2024-03-04T06:16:50.342 INFO:tasks.rados.rados.0.smithi064.stdout:2222: read oid 30 snap 260 2024-03-04T06:16:50.342 INFO:tasks.rados.rados.0.smithi064.stdout:2222: expect deleted 2024-03-04T06:16:50.342 INFO:tasks.rados.rados.0.smithi064.stdout:2223: write (excl) oid 5 current snap is 265 2024-03-04T06:16:50.342 INFO:tasks.rados.rados.0.smithi064.stdout:2209: finishing write tid 3 to smithi06431743-49 2024-03-04T06:16:50.343 INFO:tasks.rados.rados.0.smithi064.stdout:2223: seq_num 1048 ranges {413736=401313,1326776=403024,2413414=457893,3438716=175020} 2024-03-04T06:16:50.346 INFO:tasks.rados.rados.0.smithi064.stdout:2223: writing smithi06431743-5 from 413736 to 815049 tid 1 2024-03-04T06:16:50.352 INFO:tasks.rados.rados.0.smithi064.stdout:2223: writing smithi06431743-5 from 1326776 to 1729800 tid 2 2024-03-04T06:16:50.358 INFO:tasks.rados.rados.0.smithi064.stdout:2223: writing smithi06431743-5 from 2413414 to 2871307 tid 3 2024-03-04T06:16:50.365 INFO:tasks.rados.rados.0.smithi064.stdout:2223: writing smithi06431743-5 from 3438716 to 3613736 tid 4 2024-03-04T06:16:50.365 INFO:tasks.rados.rados.0.smithi064.stdout:2224: read oid 7 snap -1 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2224: expect (ObjNum 1020 snap 262 seq_num 1020) 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2225: read oid 35 snap -1 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2225: expect (ObjNum 1013 snap 260 seq_num 1013) 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2226: read oid 3 snap -1 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2226: expect (ObjNum 1032 snap 264 seq_num 1032) 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2227: snap_create 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2213: finishing write tid 2 to smithi06431743-13 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2211: finishing write tid 2 to smithi06431743-33 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2209: finishing write tid 4 to smithi06431743-49 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2209: finishing write tid 5 to smithi06431743-49 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2209: oid 49 updating version 0 to 251 2024-03-04T06:16:50.366 INFO:tasks.rados.rados.0.smithi064.stdout:2209: oid 49 updating version 251 to 254 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:2209: oid 49 version 254 is already newer than 253 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:2209: oid 49 version 254 is already newer than 252 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 254 (ObjNum 1040 snap 265 seq_num 1040) dirty exists 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:2209: left oid 49 (ObjNum 1040 snap 265 seq_num 1040) 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:2215: finishing write tid 1 to smithi06431743-15 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:2213: finishing write tid 3 to smithi06431743-13 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:2211: finishing write tid 3 to smithi06431743-33 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:2213: finishing write tid 4 to smithi06431743-13 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:2213: finishing write tid 5 to smithi06431743-13 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:2213: oid 13 updating version 0 to 936 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:2213: oid 13 updating version 936 to 938 2024-03-04T06:16:50.367 INFO:tasks.rados.rados.0.smithi064.stdout:2213: oid 13 updating version 938 to 939 2024-03-04T06:16:50.368 INFO:tasks.rados.rados.0.smithi064.stdout:2213: oid 13 version 939 is already newer than 938 2024-03-04T06:16:50.368 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 939 (ObjNum 1043 snap 265 seq_num 1043) dirty exists 2024-03-04T06:16:50.368 INFO:tasks.rados.rados.0.smithi064.stdout:2213: left oid 13 (ObjNum 1043 snap 265 seq_num 1043) 2024-03-04T06:16:50.368 INFO:tasks.rados.rados.0.smithi064.stdout:2215: finishing write tid 2 to smithi06431743-15 2024-03-04T06:16:50.368 INFO:tasks.rados.rados.0.smithi064.stdout:2215: finishing write tid 3 to smithi06431743-15 2024-03-04T06:16:50.368 INFO:tasks.rados.rados.0.smithi064.stdout:2211: finishing write tid 4 to smithi06431743-33 2024-03-04T06:16:50.368 INFO:tasks.rados.rados.0.smithi064.stdout:2211: finishing write tid 5 to smithi06431743-33 2024-03-04T06:16:50.368 INFO:tasks.rados.rados.0.smithi064.stdout:2211: oid 33 updating version 0 to 735 2024-03-04T06:16:50.368 INFO:tasks.rados.rados.0.smithi064.stdout:2211: oid 33 updating version 735 to 736 2024-03-04T06:16:50.368 INFO:tasks.rados.rados.0.smithi064.stdout:2211: oid 33 updating version 736 to 738 2024-03-04T06:16:50.369 INFO:tasks.rados.rados.0.smithi064.stdout:2211: oid 33 version 738 is already newer than 737 2024-03-04T06:16:50.369 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 738 (ObjNum 1042 snap 265 seq_num 1042) dirty exists 2024-03-04T06:16:50.369 INFO:tasks.rados.rados.0.smithi064.stdout:2211: left oid 33 (ObjNum 1042 snap 265 seq_num 1042) 2024-03-04T06:16:50.369 INFO:tasks.rados.rados.0.smithi064.stdout:2215: finishing write tid 4 to smithi06431743-15 2024-03-04T06:16:50.369 INFO:tasks.rados.rados.0.smithi064.stdout:2215: finishing write tid 5 to smithi06431743-15 2024-03-04T06:16:50.369 INFO:tasks.rados.rados.0.smithi064.stdout:2215: oid 15 updating version 0 to 424 2024-03-04T06:16:50.369 INFO:tasks.rados.rados.0.smithi064.stdout:2215: oid 15 version 424 is already newer than 424 2024-03-04T06:16:50.369 INFO:tasks.rados.rados.0.smithi064.stdout:2215: oid 15 updating version 424 to 425 2024-03-04T06:16:50.369 INFO:tasks.rados.rados.0.smithi064.stdout:2215: oid 15 version 425 is already newer than 422 2024-03-04T06:16:50.369 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 425 (ObjNum 1044 snap 265 seq_num 1044) dirty exists 2024-03-04T06:16:50.370 INFO:tasks.rados.rados.0.smithi064.stdout:2215: left oid 15 (ObjNum 1044 snap 265 seq_num 1044) 2024-03-04T06:16:50.370 INFO:tasks.rados.rados.0.smithi064.stdout:2217: finishing write tid 1 to smithi06431743-21 2024-03-04T06:16:50.370 INFO:tasks.rados.rados.0.smithi064.stdout:2217: finishing write tid 2 to smithi06431743-21 2024-03-04T06:16:50.370 INFO:tasks.rados.rados.0.smithi064.stdout:2217: finishing write tid 3 to smithi06431743-21 2024-03-04T06:16:50.370 INFO:tasks.rados.rados.0.smithi064.stdout:2217: finishing write tid 4 to smithi06431743-21 2024-03-04T06:16:50.370 INFO:tasks.rados.rados.0.smithi064.stdout:2217: finishing write tid 5 to smithi06431743-21 2024-03-04T06:16:50.370 INFO:tasks.rados.rados.0.smithi064.stdout:2217: oid 21 updating version 0 to 430 2024-03-04T06:16:50.370 INFO:tasks.rados.rados.0.smithi064.stdout:2217: oid 21 version 430 is already newer than 427 2024-03-04T06:16:50.370 INFO:tasks.rados.rados.0.smithi064.stdout:2217: oid 21 version 430 is already newer than 429 2024-03-04T06:16:50.370 INFO:tasks.rados.rados.0.smithi064.stdout:2217: oid 21 version 430 is already newer than 428 2024-03-04T06:16:50.370 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 430 (ObjNum 1045 snap 265 seq_num 1045) dirty exists 2024-03-04T06:16:50.371 INFO:tasks.rados.rados.0.smithi064.stdout:2217: left oid 21 (ObjNum 1045 snap 265 seq_num 1045) 2024-03-04T06:16:50.371 INFO:tasks.rados.rados.0.smithi064.stdout:2218: finishing write tid 1 to smithi06431743-1 2024-03-04T06:16:50.371 INFO:tasks.rados.rados.0.smithi064.stdout:2218: finishing write tid 2 to smithi06431743-1 2024-03-04T06:16:50.371 INFO:tasks.rados.rados.0.smithi064.stdout:2218: finishing write tid 3 to smithi06431743-1 2024-03-04T06:16:50.371 INFO:tasks.rados.rados.0.smithi064.stdout:2218: finishing write tid 4 to smithi06431743-1 2024-03-04T06:16:50.371 INFO:tasks.rados.rados.0.smithi064.stdout:2218: oid 1 updating version 0 to 851 2024-03-04T06:16:50.371 INFO:tasks.rados.rados.0.smithi064.stdout:2218: oid 1 version 851 is already newer than 849 2024-03-04T06:16:50.371 INFO:tasks.rados.rados.0.smithi064.stdout:2218: oid 1 version 851 is already newer than 850 2024-03-04T06:16:50.371 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 851 (ObjNum 1046 snap 265 seq_num 1046) dirty exists 2024-03-04T06:16:50.371 INFO:tasks.rados.rados.0.smithi064.stdout:2218: left oid 1 (ObjNum 1046 snap 265 seq_num 1046) 2024-03-04T06:16:50.381 INFO:tasks.rados.rados.0.smithi064.stdout:2226: expect (ObjNum 1032 snap 264 seq_num 1032) 2024-03-04T06:16:50.514 INFO:tasks.rados.rados.0.smithi064.stdout:2224: expect (ObjNum 1020 snap 262 seq_num 1020) 2024-03-04T06:16:50.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:50.633 INFO:tasks.rados.rados.0.smithi064.stdout:2221: finishing write tid 1 to smithi06431743-4 2024-03-04T06:16:50.634 INFO:tasks.rados.rados.0.smithi064.stdout:2225: expect (ObjNum 1013 snap 260 seq_num 1013) 2024-03-04T06:16:50.634 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:50.632+0000 7fd99d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:50.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:50.636+0000 7fd99d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:50.755 INFO:tasks.rados.rados.0.smithi064.stdout:2221: finishing write tid 2 to smithi06431743-4 2024-03-04T06:16:50.755 INFO:tasks.rados.rados.0.smithi064.stdout:2221: finishing write tid 3 to smithi06431743-4 2024-03-04T06:16:50.755 INFO:tasks.rados.rados.0.smithi064.stdout:2221: finishing write tid 4 to smithi06431743-4 2024-03-04T06:16:50.755 INFO:tasks.rados.rados.0.smithi064.stdout:2221: finishing write tid 5 to smithi06431743-4 2024-03-04T06:16:50.755 INFO:tasks.rados.rados.0.smithi064.stdout:2221: oid 4 updating version 0 to 952 2024-03-04T06:16:50.755 INFO:tasks.rados.rados.0.smithi064.stdout:2221: oid 4 version 952 is already newer than 951 2024-03-04T06:16:50.755 INFO:tasks.rados.rados.0.smithi064.stdout:2221: oid 4 version 952 is already newer than 947 2024-03-04T06:16:50.755 INFO:tasks.rados.rados.0.smithi064.stdout:2221: oid 4 version 952 is already newer than 951 2024-03-04T06:16:50.755 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 952 (ObjNum 1047 snap 265 seq_num 1047) dirty exists 2024-03-04T06:16:50.755 INFO:tasks.rados.rados.0.smithi064.stdout:2221: left oid 4 (ObjNum 1047 snap 265 seq_num 1047) 2024-03-04T06:16:50.756 INFO:tasks.rados.rados.0.smithi064.stdout:2223: finishing write tid 1 to smithi06431743-5 2024-03-04T06:16:50.756 INFO:tasks.rados.rados.0.smithi064.stdout:2223: finishing write tid 2 to smithi06431743-5 2024-03-04T06:16:50.756 INFO:tasks.rados.rados.0.smithi064.stdout:2223: finishing write tid 3 to smithi06431743-5 2024-03-04T06:16:50.756 INFO:tasks.rados.rados.0.smithi064.stdout:2223: finishing write tid 4 to smithi06431743-5 2024-03-04T06:16:50.756 INFO:tasks.rados.rados.0.smithi064.stdout:2223: finishing write tid 5 to smithi06431743-5 2024-03-04T06:16:50.756 INFO:tasks.rados.rados.0.smithi064.stdout:2223: finishing write tid 6 to smithi06431743-5 2024-03-04T06:16:50.756 INFO:tasks.rados.rados.0.smithi064.stdout:2223: oid 5 updating version 0 to 958 2024-03-04T06:16:50.756 INFO:tasks.rados.rados.0.smithi064.stdout:2223: oid 5 version 958 is already newer than 958 2024-03-04T06:16:50.757 INFO:tasks.rados.rados.0.smithi064.stdout:2223: oid 5 version 958 is already newer than 958 2024-03-04T06:16:50.757 INFO:tasks.rados.rados.0.smithi064.stdout:2223: oid 5 version 958 is already newer than 958 2024-03-04T06:16:50.757 INFO:tasks.rados.rados.0.smithi064.stdout:2223: oid 5 version 958 is already newer than 954 2024-03-04T06:16:50.757 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 958 (ObjNum 1048 snap 265 seq_num 1048) dirty exists 2024-03-04T06:16:50.757 INFO:tasks.rados.rados.0.smithi064.stdout:2223: left oid 5 (ObjNum 1048 snap 265 seq_num 1048) 2024-03-04T06:16:51.327 INFO:tasks.rados.rados.0.smithi064.stdout:2209: done (14 left) 2024-03-04T06:16:51.327 INFO:tasks.rados.rados.0.smithi064.stdout:2211: done (13 left) 2024-03-04T06:16:51.328 INFO:tasks.rados.rados.0.smithi064.stdout:2213: done (12 left) 2024-03-04T06:16:51.328 INFO:tasks.rados.rados.0.smithi064.stdout:2214: done (11 left) 2024-03-04T06:16:51.328 INFO:tasks.rados.rados.0.smithi064.stdout:2215: done (10 left) 2024-03-04T06:16:51.328 INFO:tasks.rados.rados.0.smithi064.stdout:2216: done (9 left) 2024-03-04T06:16:51.328 INFO:tasks.rados.rados.0.smithi064.stdout:2217: done (8 left) 2024-03-04T06:16:51.328 INFO:tasks.rados.rados.0.smithi064.stdout:2218: done (7 left) 2024-03-04T06:16:51.328 INFO:tasks.rados.rados.0.smithi064.stdout:2221: done (6 left) 2024-03-04T06:16:51.328 INFO:tasks.rados.rados.0.smithi064.stdout:2222: done (5 left) 2024-03-04T06:16:51.328 INFO:tasks.rados.rados.0.smithi064.stdout:2223: done (4 left) 2024-03-04T06:16:51.329 INFO:tasks.rados.rados.0.smithi064.stdout:2224: done (3 left) 2024-03-04T06:16:51.329 INFO:tasks.rados.rados.0.smithi064.stdout:2225: done (2 left) 2024-03-04T06:16:51.329 INFO:tasks.rados.rados.0.smithi064.stdout:2226: done (1 left) 2024-03-04T06:16:51.329 INFO:tasks.rados.rados.0.smithi064.stdout:2227: done (0 left) 2024-03-04T06:16:51.329 INFO:tasks.rados.rados.0.smithi064.stdout:2228: write (excl) oid 15 current snap is 266 2024-03-04T06:16:51.329 INFO:tasks.rados.rados.0.smithi064.stdout:2228: seq_num 1049 ranges {430543=476562,1385258=663035,2840837=669165} 2024-03-04T06:16:51.331 INFO:tasks.rados.rados.0.smithi064.stdout:2228: writing smithi06431743-15 from 430543 to 907105 tid 1 2024-03-04T06:16:51.338 INFO:tasks.rados.rados.0.smithi064.stdout:2228: writing smithi06431743-15 from 1385258 to 2048293 tid 2 2024-03-04T06:16:51.346 INFO:tasks.rados.rados.0.smithi064.stdout:2228: writing smithi06431743-15 from 2840837 to 3510002 tid 3 2024-03-04T06:16:51.346 INFO:tasks.rados.rados.0.smithi064.stdout:2229: snap_create 2024-03-04T06:16:51.382 INFO:tasks.rados.rados.0.smithi064.stdout:2228: finishing write tid 1 to smithi06431743-15 2024-03-04T06:16:51.403 INFO:tasks.rados.rados.0.smithi064.stdout:2228: finishing write tid 2 to smithi06431743-15 2024-03-04T06:16:51.419 INFO:tasks.rados.rados.0.smithi064.stdout:2228: finishing write tid 3 to smithi06431743-15 2024-03-04T06:16:51.428 INFO:tasks.rados.rados.0.smithi064.stdout:2228: finishing write tid 4 to smithi06431743-15 2024-03-04T06:16:51.428 INFO:tasks.rados.rados.0.smithi064.stdout:2228: finishing write tid 5 to smithi06431743-15 2024-03-04T06:16:51.428 INFO:tasks.rados.rados.0.smithi064.stdout:2228: oid 15 updating version 0 to 443 2024-03-04T06:16:51.428 INFO:tasks.rados.rados.0.smithi064.stdout:2228: oid 15 version 443 is already newer than 441 2024-03-04T06:16:51.428 INFO:tasks.rados.rados.0.smithi064.stdout:2228: oid 15 version 443 is already newer than 442 2024-03-04T06:16:51.429 INFO:tasks.rados.rados.0.smithi064.stdout:2228: oid 15 version 443 is already newer than 440 2024-03-04T06:16:51.429 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 443 (ObjNum 1049 snap 266 seq_num 1049) dirty exists 2024-03-04T06:16:51.429 INFO:tasks.rados.rados.0.smithi064.stdout:2228: left oid 15 (ObjNum 1049 snap 266 seq_num 1049) 2024-03-04T06:16:52.329 INFO:tasks.rados.rados.0.smithi064.stdout:2228: done (1 left) 2024-03-04T06:16:52.329 INFO:tasks.rados.rados.0.smithi064.stdout:2229: done (0 left) 2024-03-04T06:16:52.329 INFO:tasks.rados.rados.0.smithi064.stdout:2230: snap_create 2024-03-04T06:16:52.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:52.730 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:16:52.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:16:52.833 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:52.831+0000 7f5b69b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:52.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:52.837+0000 7f5b69b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:52.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:52.869+0000 7f2fb746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:52.878 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:52.878+0000 7f2fb746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:53.260 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:16:53.265 INFO:tasks.rados.rados.0.smithi064.stdout:2230: done (0 left) 2024-03-04T06:16:53.265 INFO:tasks.rados.rados.0.smithi064.stdout:2231: snap_remove snap 253 2024-03-04T06:16:53.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:16:53.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:53.378+0000 7f7772664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:53.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:53.386+0000 7f7772664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:54.261 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:16:54.263 INFO:tasks.rados.rados.0.smithi064.stdout:2231: done (0 left) 2024-03-04T06:16:54.263 INFO:tasks.rados.rados.0.smithi064.stdout:2232: snap_remove snap 256 2024-03-04T06:16:54.276 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:16:54.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:54.387+0000 7f65672dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:54.396 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:54.396+0000 7f65672dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:54.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:55.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:55.012+0000 7fafcff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:55.017 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:55.016+0000 7fafcff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:55.261 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:16:55.266 INFO:tasks.rados.rados.0.smithi064.stdout:2232: done (0 left) 2024-03-04T06:16:55.266 INFO:tasks.rados.rados.0.smithi064.stdout:2233: write oid 37 current snap is 268 2024-03-04T06:16:55.266 INFO:tasks.rados.rados.0.smithi064.stdout:2233: seq_num 1050 ranges {447350=668164,1560093=406693,2584613=480437} 2024-03-04T06:16:55.272 INFO:tasks.rados.rados.0.smithi064.stdout:2233: writing smithi06431743-37 from 447350 to 1115514 tid 1 2024-03-04T06:16:55.276 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:16:55.277 INFO:tasks.rados.rados.0.smithi064.stdout:2233: writing smithi06431743-37 from 1560093 to 1966786 tid 2 2024-03-04T06:16:55.283 INFO:tasks.rados.rados.0.smithi064.stdout:2233: writing smithi06431743-37 from 2584613 to 3065050 tid 3 2024-03-04T06:16:55.283 INFO:tasks.rados.rados.0.smithi064.stdout:2234: delete oid 10 current snap is 268 2024-03-04T06:16:55.370 INFO:tasks.rados.rados.0.smithi064.stdout:2234: done (1 left) 2024-03-04T06:16:55.370 INFO:tasks.rados.rados.0.smithi064.stdout:2235: snap_create 2024-03-04T06:16:55.373 INFO:tasks.rados.rados.0.smithi064.stdout:2233: finishing write tid 1 to smithi06431743-37 2024-03-04T06:16:55.392 INFO:tasks.rados.rados.0.smithi064.stdout:2233: finishing write tid 2 to smithi06431743-37 2024-03-04T06:16:55.400 INFO:tasks.rados.rados.0.smithi064.stdout:2233: finishing write tid 3 to smithi06431743-37 2024-03-04T06:16:55.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:55.405+0000 7ff3bf017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:55.412 INFO:tasks.rados.rados.0.smithi064.stdout:2233: finishing write tid 4 to smithi06431743-37 2024-03-04T06:16:55.414 INFO:tasks.rados.rados.0.smithi064.stdout:2233: finishing write tid 5 to smithi06431743-37 2024-03-04T06:16:55.414 INFO:tasks.rados.rados.0.smithi064.stdout:2233: oid 37 updating version 0 to 355 2024-03-04T06:16:55.414 INFO:tasks.rados.rados.0.smithi064.stdout:2233: oid 37 version 355 is already newer than 354 2024-03-04T06:16:55.414 INFO:tasks.rados.rados.0.smithi064.stdout:2233: oid 37 updating version 355 to 356 2024-03-04T06:16:55.414 INFO:tasks.rados.rados.0.smithi064.stdout:2233: oid 37 version 356 is already newer than 353 2024-03-04T06:16:55.415 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 356 (ObjNum 1050 snap 268 seq_num 1050) dirty exists 2024-03-04T06:16:55.415 INFO:tasks.rados.rados.0.smithi064.stdout:2233: left oid 37 (ObjNum 1050 snap 268 seq_num 1050) 2024-03-04T06:16:55.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:55.413+0000 7ff3bf017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:56.341 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:16:56.344 INFO:tasks.rados.rados.0.smithi064.stdout:2233: done (1 left) 2024-03-04T06:16:56.345 INFO:tasks.rados.rados.0.smithi064.stdout:2235: done (0 left) 2024-03-04T06:16:56.345 INFO:tasks.rados.rados.0.smithi064.stdout:2236: read oid 13 snap -1 2024-03-04T06:16:56.345 INFO:tasks.rados.rados.0.smithi064.stdout:2236: expect (ObjNum 1043 snap 265 seq_num 1043) 2024-03-04T06:16:56.345 INFO:tasks.rados.rados.0.smithi064.stdout:2237: read oid 32 snap -1 2024-03-04T06:16:56.345 INFO:tasks.rados.rados.0.smithi064.stdout:2237: expect deleted 2024-03-04T06:16:56.345 INFO:tasks.rados.rados.0.smithi064.stdout:2238: snap_remove snap 265 2024-03-04T06:16:56.364 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:16:56.365 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:16:56.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:56.509+0000 7f8120e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:56.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:56.515+0000 7f8120e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:56.707 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:16:56.708 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:16:56.719 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":812,"stamp":"2024-03-04T06:16:56.341906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":100733668,"num_objects":785,"num_object_clones":603,"num_object_copies":2355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4165,"num_read_kb":3131863,"num_write":3637,"num_write_kb":955240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7635,"ondisk_log_size":7635,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":535080,"kb_used_data":357264,"kb_used_omap":0,"kb_used_meta":176640,"kb_avail":376952280,"statfs":{"total":386547056640,"available":385999134720,"internally_reserved":0,"allocated":365838336,"data_stored":415558983,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":180879360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1327951,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":5,"num_read_kb":2,"num_write":13,"num_write_kb":3700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015669"},"pg_stats":[{"pgid":"3.f","version":"545'466","reported_seq":1914,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.306674+0000","last_change":"2024-03-04T06:16:55.355423+0000","last_active":"2024-03-04T06:16:55.306674+0000","last_peered":"2024-03-04T06:16:55.306674+0000","last_clean":"2024-03-04T06:16:55.306674+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:16:55.306674+0000","last_undegraded":"2024-03-04T06:16:55.306674+0000","last_fullsized":"2024-03-04T06:16:55.306674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8942017,"num_objects":51,"num_object_clones":40,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":207696,"num_write":215,"num_write_kb":58378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"545'351","reported_seq":1838,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.298262+0000","last_change":"2024-03-04T06:16:55.298075+0000","last_active":"2024-03-04T06:16:55.298262+0000","last_peered":"2024-03-04T06:16:55.298262+0000","last_clean":"2024-03-04T06:16:55.298262+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:16:55.298262+0000","last_undegraded":"2024-03-04T06:16:55.298262+0000","last_fullsized":"2024-03-04T06:16:55.298262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8757140,"num_objects":42,"num_object_clones":28,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":134,"num_read_kb":80770,"num_write":166,"num_write_kb":43965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"539'493","reported_seq":1971,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-04T06:16:51.348652+0000","last_change":"2024-03-04T06:16:50.341246+0000","last_active":"2024-03-04T06:16:51.348652+0000","last_peered":"2024-03-04T06:16:51.348652+0000","last_clean":"2024-03-04T06:16:51.348652+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:16:51.348652+0000","last_undegraded":"2024-03-04T06:16:51.348652+0000","last_fullsized":"2024-03-04T06:16:51.348652+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":493,"log_dups_size":0,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7160211,"num_objects":59,"num_object_clones":46,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":329,"num_read_kb":259518,"num_write":252,"num_write_kb":69591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"545'340","reported_seq":1847,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.298319+0000","last_change":"2024-03-04T06:16:55.348449+0000","last_active":"2024-03-04T06:16:55.298319+0000","last_peered":"2024-03-04T06:16:55.298319+0000","last_clean":"2024-03-04T06:16:55.298319+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:16:55.298319+0000","last_undegraded":"2024-03-04T06:16:55.298319+0000","last_fullsized":"2024-03-04T06:16:55.298319+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185372,"num_objects":35,"num_object_clones":25,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":193,"num_read_kb":149127,"num_write":169,"num_write_kb":47053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1677,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-04T06:16:50.375129+0000","last_change":"2024-03-04T06:16:50.385102+0000","last_active":"2024-03-04T06:16:50.375129+0000","last_peered":"2024-03-04T06:16:50.375129+0000","last_clean":"2024-03-04T06:16:50.375129+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:16:50.375129+0000","last_undegraded":"2024-03-04T06:16:50.375129+0000","last_fullsized":"2024-03-04T06:16:50.375129+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":58,"num_read_kb":33096,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"544'523","reported_seq":1937,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.300631+0000","last_change":"2024-03-04T06:16:55.300382+0000","last_active":"2024-03-04T06:16:55.300631+0000","last_peered":"2024-03-04T06:16:55.300631+0000","last_clean":"2024-03-04T06:16:55.300631+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:16:55.300631+0000","last_undegraded":"2024-03-04T06:16:55.300631+0000","last_fullsized":"2024-03-04T06:16:55.300631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2689351,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":252,"num_read_kb":192235,"num_write":246,"num_write_kb":71955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1607,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.302079+0000","last_change":"2024-03-04T06:16:55.314933+0000","last_active":"2024-03-04T06:16:55.302079+0000","last_peered":"2024-03-04T06:16:55.302079+0000","last_clean":"2024-03-04T06:16:55.302079+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:16:55.302079+0000","last_undegraded":"2024-03-04T06:16:55.302079+0000","last_fullsized":"2024-03-04T06:16:55.302079+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"540'258","reported_seq":1789,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-04T06:16:51.347950+0000","last_change":"2024-03-04T06:16:50.392053+0000","last_active":"2024-03-04T06:16:51.347950+0000","last_peered":"2024-03-04T06:16:51.347950+0000","last_clean":"2024-03-04T06:16:51.347950+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:16:51.347950+0000","last_undegraded":"2024-03-04T06:16:51.347950+0000","last_fullsized":"2024-03-04T06:16:51.347950+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6437477,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":176,"num_read_kb":163221,"num_write":123,"num_write_kb":32558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1471,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-04T06:16:50.373604+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:16:50.373604+0000","last_peered":"2024-03-04T06:16:50.373604+0000","last_clean":"2024-03-04T06:16:50.373604+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:16:50.373604+0000","last_undegraded":"2024-03-04T06:16:50.373604+0000","last_fullsized":"2024-03-04T06:16:50.373604+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1351,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-04T06:16:51.351893+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:16:51.351893+0000","last_peered":"2024-03-04T06:16:51.351893+0000","last_clean":"2024-03-04T06:16:51.351893+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:16:51.351893+0000","last_undegraded":"2024-03-04T06:16:51.351893+0000","last_fullsized":"2024-03-04T06:16:51.351893+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"539'511","reported_seq":1921,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-04T06:16:51.353842+0000","last_change":"2024-03-04T06:16:50.343639+0000","last_active":"2024-03-04T06:16:51.353842+0000","last_peered":"2024-03-04T06:16:51.353842+0000","last_clean":"2024-03-04T06:16:51.353842+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:16:51.353842+0000","last_undegraded":"2024-03-04T06:16:51.353842+0000","last_fullsized":"2024-03-04T06:16:51.353842+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":52,"num_object_clones":40,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":257,"num_read_kb":204750,"num_write":234,"num_write_kb":63712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1357,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.302417+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:16:55.302417+0000","last_peered":"2024-03-04T06:16:55.302417+0000","last_clean":"2024-03-04T06:16:55.302417+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:16:55.302417+0000","last_undegraded":"2024-03-04T06:16:55.302417+0000","last_fullsized":"2024-03-04T06:16:55.302417+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"531'366","reported_seq":1807,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-04T06:16:50.423604+0000","last_change":"2024-03-04T06:16:50.379710+0000","last_active":"2024-03-04T06:16:50.423604+0000","last_peered":"2024-03-04T06:16:50.423604+0000","last_clean":"2024-03-04T06:16:50.423604+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:16:50.423604+0000","last_undegraded":"2024-03-04T06:16:50.423604+0000","last_fullsized":"2024-03-04T06:16:50.423604+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3143140,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":210,"num_read_kb":173945,"num_write":141,"num_write_kb":40250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1351,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-04T06:16:51.345215+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:16:51.345215+0000","last_peered":"2024-03-04T06:16:51.345215+0000","last_clean":"2024-03-04T06:16:51.345215+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:16:51.345215+0000","last_undegraded":"2024-03-04T06:16:51.345215+0000","last_fullsized":"2024-03-04T06:16:51.345215+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"540'857","reported_seq":2197,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-04T06:16:50.377434+0000","last_change":"2024-03-04T06:16:50.439194+0000","last_active":"2024-03-04T06:16:50.377434+0000","last_peered":"2024-03-04T06:16:50.377434+0000","last_clean":"2024-03-04T06:16:50.377434+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:16:50.377434+0000","last_undegraded":"2024-03-04T06:16:50.377434+0000","last_fullsized":"2024-03-04T06:16:50.377434+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":857,"log_dups_size":0,"ondisk_log_size":857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11446491,"num_objects":89,"num_object_clones":68,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":527,"num_read_kb":421075,"num_write":416,"num_write_kb":111943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1357,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.290772+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:16:55.290772+0000","last_peered":"2024-03-04T06:16:55.290772+0000","last_clean":"2024-03-04T06:16:55.290772+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:16:55.290772+0000","last_undegraded":"2024-03-04T06:16:55.290772+0000","last_fullsized":"2024-03-04T06:16:55.290772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"544'966","reported_seq":2199,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.296206+0000","last_change":"2024-03-04T06:16:55.296206+0000","last_active":"2024-03-04T06:16:55.296206+0000","last_peered":"2024-03-04T06:16:55.296206+0000","last_clean":"2024-03-04T06:16:55.296206+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:16:55.296206+0000","last_undegraded":"2024-03-04T06:16:55.296206+0000","last_fullsized":"2024-03-04T06:16:55.296206+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":966,"log_dups_size":0,"ondisk_log_size":966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13816278,"num_objects":96,"num_object_clones":74,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":401,"num_read_kb":277934,"num_write":439,"num_write_kb":115530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"540'438","reported_seq":1928,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-04T06:16:51.353942+0000","last_change":"2024-03-04T06:16:50.423570+0000","last_active":"2024-03-04T06:16:51.353942+0000","last_peered":"2024-03-04T06:16:51.353942+0000","last_clean":"2024-03-04T06:16:51.353942+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:16:51.353942+0000","last_undegraded":"2024-03-04T06:16:51.353942+0000","last_fullsized":"2024-03-04T06:16:51.353942+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7109823,"num_objects":47,"num_object_clones":35,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":298,"num_read_kb":229916,"num_write":215,"num_write_kb":58406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1385,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.311331+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:16:55.311331+0000","last_peered":"2024-03-04T06:16:55.311331+0000","last_clean":"2024-03-04T06:16:55.311331+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:16:55.311331+0000","last_undegraded":"2024-03-04T06:16:55.311331+0000","last_fullsized":"2024-03-04T06:16:55.311331+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"544'435","reported_seq":1913,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.316268+0000","last_change":"2024-03-04T06:16:55.316268+0000","last_active":"2024-03-04T06:16:55.316268+0000","last_peered":"2024-03-04T06:16:55.316268+0000","last_clean":"2024-03-04T06:16:55.316268+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:16:55.316268+0000","last_undegraded":"2024-03-04T06:16:55.316268+0000","last_fullsized":"2024-03-04T06:16:55.316268+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445666,"num_objects":52,"num_object_clones":37,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":221,"num_read_kb":149343,"num_write":205,"num_write_kb":54064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1338,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-04T06:16:50.376593+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:16:50.376593+0000","last_peered":"2024-03-04T06:16:50.376593+0000","last_clean":"2024-03-04T06:16:50.376593+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:16:50.376593+0000","last_undegraded":"2024-03-04T06:16:50.376593+0000","last_fullsized":"2024-03-04T06:16:50.376593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"539'615","reported_seq":1983,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-04T06:16:50.433040+0000","last_change":"2024-03-04T06:16:50.380623+0000","last_active":"2024-03-04T06:16:50.433040+0000","last_peered":"2024-03-04T06:16:50.433040+0000","last_clean":"2024-03-04T06:16:50.433040+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:16:50.433040+0000","last_undegraded":"2024-03-04T06:16:50.433040+0000","last_fullsized":"2024-03-04T06:16:50.433040+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882912,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":328,"num_read_kb":253701,"num_write":282,"num_write_kb":78197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1385,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.293047+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:16:55.293047+0000","last_peered":"2024-03-04T06:16:55.293047+0000","last_clean":"2024-03-04T06:16:55.293047+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:16:55.293047+0000","last_undegraded":"2024-03-04T06:16:55.293047+0000","last_fullsized":"2024-03-04T06:16:55.293047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"544'752","reported_seq":2113,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.288356+0000","last_change":"2024-03-04T06:16:55.288356+0000","last_active":"2024-03-04T06:16:55.288356+0000","last_peered":"2024-03-04T06:16:55.288356+0000","last_clean":"2024-03-04T06:16:55.288356+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:16:55.288356+0000","last_undegraded":"2024-03-04T06:16:55.288356+0000","last_fullsized":"2024-03-04T06:16:55.288356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":70,"num_object_clones":61,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":422,"num_read_kb":335445,"num_write":322,"num_write_kb":87782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1351,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-04T06:16:51.351103+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:16:51.351103+0000","last_peered":"2024-03-04T06:16:51.351103+0000","last_clean":"2024-03-04T06:16:51.351103+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:16:51.351103+0000","last_undegraded":"2024-03-04T06:16:51.351103+0000","last_fullsized":"2024-03-04T06:16:51.351103+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":100143300,"num_objects":783,"num_object_clones":603,"num_object_copies":2349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4062,"num_read_kb":3131772,"num_write":3500,"num_write_kb":953346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7538,"ondisk_log_size":7538,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":159788,"kb_used_data":110184,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94212052,"statfs":{"total":96636764160,"available":96473141248,"internally_reserved":0,"allocated":112828416,"data_stored":128230761,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":136468,"kb_used_data":91024,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94235372,"statfs":{"total":96636764160,"available":96497020928,"internally_reserved":0,"allocated":93208576,"data_stored":105664924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":124928,"kb_used_data":81464,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94246912,"statfs":{"total":96636764160,"available":96508837888,"internally_reserved":0,"allocated":83419136,"data_stored":96046901,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":113896,"kb_used_data":74592,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94257944,"statfs":{"total":96636764160,"available":96520134656,"internally_reserved":0,"allocated":76382208,"data_stored":85616397,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:16:56.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:16:56.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:56.824+0000 7feedf55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:56.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:56.830+0000 7feedf55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:57.020 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:16:57.020 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:16:57.032 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":813,"stamp":"2024-03-04T06:16:56.823007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":100817703,"num_objects":785,"num_object_clones":603,"num_object_copies":2355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4166,"num_read_kb":3131864,"num_write":3641,"num_write_kb":956542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7676,"ondisk_log_size":7676,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":539752,"kb_used_data":363080,"kb_used_omap":0,"kb_used_meta":176640,"kb_avail":376947608,"statfs":{"total":386547056640,"available":385994350592,"internally_reserved":0,"allocated":371793920,"data_stored":424497536,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":180879360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1243916,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":6,"num_read_kb":3,"num_write":17,"num_write_kb":5002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000894"},"pg_stats":[{"pgid":"3.f","version":"545'466","reported_seq":1914,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.306674+0000","last_change":"2024-03-04T06:16:55.355423+0000","last_active":"2024-03-04T06:16:55.306674+0000","last_peered":"2024-03-04T06:16:55.306674+0000","last_clean":"2024-03-04T06:16:55.306674+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:16:55.306674+0000","last_undegraded":"2024-03-04T06:16:55.306674+0000","last_fullsized":"2024-03-04T06:16:55.306674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8942017,"num_objects":51,"num_object_clones":40,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":207696,"num_write":215,"num_write_kb":58378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"545'351","reported_seq":1838,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.298262+0000","last_change":"2024-03-04T06:16:55.298075+0000","last_active":"2024-03-04T06:16:55.298262+0000","last_peered":"2024-03-04T06:16:55.298262+0000","last_clean":"2024-03-04T06:16:55.298262+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:16:55.298262+0000","last_undegraded":"2024-03-04T06:16:55.298262+0000","last_fullsized":"2024-03-04T06:16:55.298262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8757140,"num_objects":42,"num_object_clones":28,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":134,"num_read_kb":80770,"num_write":166,"num_write_kb":43965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"544'497","reported_seq":1980,"reported_epoch":546,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:56.324521+0000","last_change":"2024-03-04T06:16:56.324521+0000","last_active":"2024-03-04T06:16:56.324521+0000","last_peered":"2024-03-04T06:16:56.324521+0000","last_clean":"2024-03-04T06:16:56.324521+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:16:56.324521+0000","last_undegraded":"2024-03-04T06:16:56.324521+0000","last_fullsized":"2024-03-04T06:16:56.324521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7160211,"num_objects":59,"num_object_clones":46,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":329,"num_read_kb":259518,"num_write":252,"num_write_kb":69591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"545'340","reported_seq":1847,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.298319+0000","last_change":"2024-03-04T06:16:55.348449+0000","last_active":"2024-03-04T06:16:55.298319+0000","last_peered":"2024-03-04T06:16:55.298319+0000","last_clean":"2024-03-04T06:16:55.298319+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:16:55.298319+0000","last_undegraded":"2024-03-04T06:16:55.298319+0000","last_fullsized":"2024-03-04T06:16:55.298319+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185372,"num_objects":35,"num_object_clones":25,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":193,"num_read_kb":149127,"num_write":169,"num_write_kb":47053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1695,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.387129+0000","last_change":"2024-03-04T06:16:55.295909+0000","last_active":"2024-03-04T06:16:56.387129+0000","last_peered":"2024-03-04T06:16:56.387129+0000","last_clean":"2024-03-04T06:16:56.387129+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:16:56.387129+0000","last_undegraded":"2024-03-04T06:16:56.387129+0000","last_fullsized":"2024-03-04T06:16:56.387129+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":58,"num_read_kb":33096,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"544'523","reported_seq":1937,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.300631+0000","last_change":"2024-03-04T06:16:55.300382+0000","last_active":"2024-03-04T06:16:55.300631+0000","last_peered":"2024-03-04T06:16:55.300631+0000","last_clean":"2024-03-04T06:16:55.300631+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:16:55.300631+0000","last_undegraded":"2024-03-04T06:16:55.300631+0000","last_fullsized":"2024-03-04T06:16:55.300631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2689351,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":252,"num_read_kb":192235,"num_write":246,"num_write_kb":71955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1607,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.302079+0000","last_change":"2024-03-04T06:16:55.314933+0000","last_active":"2024-03-04T06:16:55.302079+0000","last_peered":"2024-03-04T06:16:55.302079+0000","last_clean":"2024-03-04T06:16:55.302079+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:16:55.302079+0000","last_undegraded":"2024-03-04T06:16:55.302079+0000","last_fullsized":"2024-03-04T06:16:55.302079+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"545'262","reported_seq":1799,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.328663+0000","last_change":"2024-03-04T06:16:56.362927+0000","last_active":"2024-03-04T06:16:56.328663+0000","last_peered":"2024-03-04T06:16:56.328663+0000","last_clean":"2024-03-04T06:16:56.328663+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:16:56.328663+0000","last_undegraded":"2024-03-04T06:16:56.328663+0000","last_fullsized":"2024-03-04T06:16:56.328663+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6437477,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":176,"num_read_kb":163221,"num_write":123,"num_write_kb":32558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1487,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.370840+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:16:56.370840+0000","last_peered":"2024-03-04T06:16:56.370840+0000","last_clean":"2024-03-04T06:16:56.370840+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:16:56.370840+0000","last_undegraded":"2024-03-04T06:16:56.370840+0000","last_fullsized":"2024-03-04T06:16:56.370840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1359,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.334190+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:16:56.334190+0000","last_peered":"2024-03-04T06:16:56.334190+0000","last_clean":"2024-03-04T06:16:56.334190+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:16:56.334190+0000","last_undegraded":"2024-03-04T06:16:56.334190+0000","last_fullsized":"2024-03-04T06:16:56.334190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"544'515","reported_seq":1930,"reported_epoch":546,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:56.333690+0000","last_change":"2024-03-04T06:16:56.333690+0000","last_active":"2024-03-04T06:16:56.333690+0000","last_peered":"2024-03-04T06:16:56.333690+0000","last_clean":"2024-03-04T06:16:56.333690+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:16:56.333690+0000","last_undegraded":"2024-03-04T06:16:56.333690+0000","last_fullsized":"2024-03-04T06:16:56.333690+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":52,"num_object_clones":40,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":257,"num_read_kb":204750,"num_write":234,"num_write_kb":63712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1357,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.302417+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:16:55.302417+0000","last_peered":"2024-03-04T06:16:55.302417+0000","last_clean":"2024-03-04T06:16:55.302417+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:16:55.302417+0000","last_undegraded":"2024-03-04T06:16:55.302417+0000","last_fullsized":"2024-03-04T06:16:55.302417+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"531'366","reported_seq":1825,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.383228+0000","last_change":"2024-03-04T06:16:55.294953+0000","last_active":"2024-03-04T06:16:56.383228+0000","last_peered":"2024-03-04T06:16:56.383228+0000","last_clean":"2024-03-04T06:16:56.383228+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:16:56.383228+0000","last_undegraded":"2024-03-04T06:16:56.383228+0000","last_fullsized":"2024-03-04T06:16:56.383228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3143140,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":210,"num_read_kb":173945,"num_write":141,"num_write_kb":40250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1359,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.326183+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:16:56.326183+0000","last_peered":"2024-03-04T06:16:56.326183+0000","last_clean":"2024-03-04T06:16:56.326183+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:16:56.326183+0000","last_undegraded":"2024-03-04T06:16:56.326183+0000","last_fullsized":"2024-03-04T06:16:56.326183+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"545'865","reported_seq":2215,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.380596+0000","last_change":"2024-03-04T06:16:55.354005+0000","last_active":"2024-03-04T06:16:56.380596+0000","last_peered":"2024-03-04T06:16:56.380596+0000","last_clean":"2024-03-04T06:16:56.380596+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:16:56.380596+0000","last_undegraded":"2024-03-04T06:16:56.380596+0000","last_fullsized":"2024-03-04T06:16:56.380596+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11446491,"num_objects":89,"num_object_clones":68,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":527,"num_read_kb":421075,"num_write":416,"num_write_kb":111943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1357,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.290772+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:16:55.290772+0000","last_peered":"2024-03-04T06:16:55.290772+0000","last_clean":"2024-03-04T06:16:55.290772+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:16:55.290772+0000","last_undegraded":"2024-03-04T06:16:55.290772+0000","last_fullsized":"2024-03-04T06:16:55.290772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"544'966","reported_seq":2199,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.296206+0000","last_change":"2024-03-04T06:16:55.296206+0000","last_active":"2024-03-04T06:16:55.296206+0000","last_peered":"2024-03-04T06:16:55.296206+0000","last_clean":"2024-03-04T06:16:55.296206+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:16:55.296206+0000","last_undegraded":"2024-03-04T06:16:55.296206+0000","last_fullsized":"2024-03-04T06:16:55.296206+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":966,"log_dups_size":0,"ondisk_log_size":966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13816278,"num_objects":96,"num_object_clones":74,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":401,"num_read_kb":277934,"num_write":439,"num_write_kb":115530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"544'447","reported_seq":1941,"reported_epoch":546,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:56.335805+0000","last_change":"2024-03-04T06:16:56.335805+0000","last_active":"2024-03-04T06:16:56.335805+0000","last_peered":"2024-03-04T06:16:56.335805+0000","last_clean":"2024-03-04T06:16:56.335805+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:16:56.335805+0000","last_undegraded":"2024-03-04T06:16:56.335805+0000","last_fullsized":"2024-03-04T06:16:56.335805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193858,"num_objects":47,"num_object_clones":35,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":299,"num_read_kb":229917,"num_write":219,"num_write_kb":59708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1385,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.311331+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:16:55.311331+0000","last_peered":"2024-03-04T06:16:55.311331+0000","last_clean":"2024-03-04T06:16:55.311331+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:16:55.311331+0000","last_undegraded":"2024-03-04T06:16:55.311331+0000","last_fullsized":"2024-03-04T06:16:55.311331+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"544'435","reported_seq":1913,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.316268+0000","last_change":"2024-03-04T06:16:55.316268+0000","last_active":"2024-03-04T06:16:55.316268+0000","last_peered":"2024-03-04T06:16:55.316268+0000","last_clean":"2024-03-04T06:16:55.316268+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:16:55.316268+0000","last_undegraded":"2024-03-04T06:16:55.316268+0000","last_fullsized":"2024-03-04T06:16:55.316268+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445666,"num_objects":52,"num_object_clones":37,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":221,"num_read_kb":149343,"num_write":205,"num_write_kb":54064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1354,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.385688+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:16:56.385688+0000","last_peered":"2024-03-04T06:16:56.385688+0000","last_clean":"2024-03-04T06:16:56.385688+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:16:56.385688+0000","last_undegraded":"2024-03-04T06:16:56.385688+0000","last_fullsized":"2024-03-04T06:16:56.385688+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"545'627","reported_seq":2001,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.381776+0000","last_change":"2024-03-04T06:16:55.355047+0000","last_active":"2024-03-04T06:16:56.381776+0000","last_peered":"2024-03-04T06:16:56.381776+0000","last_clean":"2024-03-04T06:16:56.381776+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:16:56.381776+0000","last_undegraded":"2024-03-04T06:16:56.381776+0000","last_fullsized":"2024-03-04T06:16:56.381776+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882912,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":328,"num_read_kb":253701,"num_write":282,"num_write_kb":78197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1385,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.293047+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:16:55.293047+0000","last_peered":"2024-03-04T06:16:55.293047+0000","last_clean":"2024-03-04T06:16:55.293047+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:16:55.293047+0000","last_undegraded":"2024-03-04T06:16:55.293047+0000","last_fullsized":"2024-03-04T06:16:55.293047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"544'752","reported_seq":2113,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.288356+0000","last_change":"2024-03-04T06:16:55.288356+0000","last_active":"2024-03-04T06:16:55.288356+0000","last_peered":"2024-03-04T06:16:55.288356+0000","last_clean":"2024-03-04T06:16:55.288356+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:16:55.288356+0000","last_undegraded":"2024-03-04T06:16:55.288356+0000","last_fullsized":"2024-03-04T06:16:55.288356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":70,"num_object_clones":61,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":422,"num_read_kb":335445,"num_write":322,"num_write_kb":87782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1359,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.335183+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:16:56.335183+0000","last_peered":"2024-03-04T06:16:56.335183+0000","last_clean":"2024-03-04T06:16:56.335183+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:16:56.335183+0000","last_undegraded":"2024-03-04T06:16:56.335183+0000","last_fullsized":"2024-03-04T06:16:56.335183+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":100227335,"num_objects":783,"num_object_clones":603,"num_object_copies":2349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4063,"num_read_kb":3131773,"num_write":3504,"num_write_kb":954648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7579,"ondisk_log_size":7579,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":161692,"kb_used_data":112660,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94210148,"statfs":{"total":96636764160,"available":96471191552,"internally_reserved":0,"allocated":115363840,"data_stored":132024912,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":139236,"kb_used_data":94364,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94232604,"statfs":{"total":96636764160,"available":96494186496,"internally_reserved":0,"allocated":96628736,"data_stored":110809326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":124928,"kb_used_data":81464,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94246912,"statfs":{"total":96636764160,"available":96508837888,"internally_reserved":0,"allocated":83419136,"data_stored":96046901,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":113896,"kb_used_data":74592,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94257944,"statfs":{"total":96636764160,"available":96520134656,"internally_reserved":0,"allocated":76382208,"data_stored":85616397,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:16:57.033 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:16:57.034 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:16:57.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:16:57.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:16:57.141 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:57.137+0000 7f4db006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:57.144 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:57.143+0000 7f4db006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:57.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:57.203+0000 7f70a411d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:57.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:57.207+0000 7f70a411d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:57.348 INFO:tasks.rados.rados.0.smithi064.stdout:2238: done (2 left) 2024-03-04T06:16:57.348 INFO:tasks.rados.rados.0.smithi064.stdout:2239: write (excl) oid 35 current snap is 269 2024-03-04T06:16:57.348 INFO:tasks.rados.rados.0.smithi064.stdout:2239: seq_num 1052 ranges {480964=418662,1560704=410362,2523106=619334} 2024-03-04T06:16:57.353 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:16:57.353 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:16:57.354 INFO:tasks.rados.rados.0.smithi064.stdout:2239: writing smithi06431743-35 from 480964 to 899626 tid 1 2024-03-04T06:16:57.358 INFO:tasks.rados.rados.0.smithi064.stdout:2239: writing smithi06431743-35 from 1560704 to 1971066 tid 2 2024-03-04T06:16:57.365 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":814,"stamp":"2024-03-04T06:16:57.346761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":100817703,"num_objects":785,"num_object_clones":603,"num_object_copies":2355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4166,"num_read_kb":3131864,"num_write":3641,"num_write_kb":956542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7676,"ondisk_log_size":7676,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":539752,"kb_used_data":363080,"kb_used_omap":0,"kb_used_meta":176640,"kb_avail":376947608,"statfs":{"total":386547056640,"available":385994350592,"internally_reserved":0,"allocated":371793920,"data_stored":424497536,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":180879360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1243916,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":6,"num_read_kb":3,"num_write":17,"num_write_kb":5002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.083885"},"pg_stats":[{"pgid":"3.f","version":"545'466","reported_seq":1914,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.306674+0000","last_change":"2024-03-04T06:16:55.355423+0000","last_active":"2024-03-04T06:16:55.306674+0000","last_peered":"2024-03-04T06:16:55.306674+0000","last_clean":"2024-03-04T06:16:55.306674+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:16:55.306674+0000","last_undegraded":"2024-03-04T06:16:55.306674+0000","last_fullsized":"2024-03-04T06:16:55.306674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8942017,"num_objects":51,"num_object_clones":40,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":207696,"num_write":215,"num_write_kb":58378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"545'351","reported_seq":1838,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.298262+0000","last_change":"2024-03-04T06:16:55.298075+0000","last_active":"2024-03-04T06:16:55.298262+0000","last_peered":"2024-03-04T06:16:55.298262+0000","last_clean":"2024-03-04T06:16:55.298262+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:16:55.298262+0000","last_undegraded":"2024-03-04T06:16:55.298262+0000","last_fullsized":"2024-03-04T06:16:55.298262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8757140,"num_objects":42,"num_object_clones":28,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":134,"num_read_kb":80770,"num_write":166,"num_write_kb":43965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"544'497","reported_seq":1980,"reported_epoch":546,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:56.324521+0000","last_change":"2024-03-04T06:16:56.324521+0000","last_active":"2024-03-04T06:16:56.324521+0000","last_peered":"2024-03-04T06:16:56.324521+0000","last_clean":"2024-03-04T06:16:56.324521+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:16:56.324521+0000","last_undegraded":"2024-03-04T06:16:56.324521+0000","last_fullsized":"2024-03-04T06:16:56.324521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7160211,"num_objects":59,"num_object_clones":46,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":329,"num_read_kb":259518,"num_write":252,"num_write_kb":69591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"545'340","reported_seq":1847,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.298319+0000","last_change":"2024-03-04T06:16:55.348449+0000","last_active":"2024-03-04T06:16:55.298319+0000","last_peered":"2024-03-04T06:16:55.298319+0000","last_clean":"2024-03-04T06:16:55.298319+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:16:55.298319+0000","last_undegraded":"2024-03-04T06:16:55.298319+0000","last_fullsized":"2024-03-04T06:16:55.298319+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":340,"log_dups_size":0,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185372,"num_objects":35,"num_object_clones":25,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":193,"num_read_kb":149127,"num_write":169,"num_write_kb":47053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1695,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.387129+0000","last_change":"2024-03-04T06:16:55.295909+0000","last_active":"2024-03-04T06:16:56.387129+0000","last_peered":"2024-03-04T06:16:56.387129+0000","last_clean":"2024-03-04T06:16:56.387129+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:16:56.387129+0000","last_undegraded":"2024-03-04T06:16:56.387129+0000","last_fullsized":"2024-03-04T06:16:56.387129+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":58,"num_read_kb":33096,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"544'523","reported_seq":1937,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.300631+0000","last_change":"2024-03-04T06:16:55.300382+0000","last_active":"2024-03-04T06:16:55.300631+0000","last_peered":"2024-03-04T06:16:55.300631+0000","last_clean":"2024-03-04T06:16:55.300631+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:16:55.300631+0000","last_undegraded":"2024-03-04T06:16:55.300631+0000","last_fullsized":"2024-03-04T06:16:55.300631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2689351,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":252,"num_read_kb":192235,"num_write":246,"num_write_kb":71955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1607,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.302079+0000","last_change":"2024-03-04T06:16:55.314933+0000","last_active":"2024-03-04T06:16:55.302079+0000","last_peered":"2024-03-04T06:16:55.302079+0000","last_clean":"2024-03-04T06:16:55.302079+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:16:55.302079+0000","last_undegraded":"2024-03-04T06:16:55.302079+0000","last_fullsized":"2024-03-04T06:16:55.302079+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"545'262","reported_seq":1799,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.328663+0000","last_change":"2024-03-04T06:16:56.362927+0000","last_active":"2024-03-04T06:16:56.328663+0000","last_peered":"2024-03-04T06:16:56.328663+0000","last_clean":"2024-03-04T06:16:56.328663+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:16:56.328663+0000","last_undegraded":"2024-03-04T06:16:56.328663+0000","last_fullsized":"2024-03-04T06:16:56.328663+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6437477,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":176,"num_read_kb":163221,"num_write":123,"num_write_kb":32558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1487,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.370840+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:16:56.370840+0000","last_peered":"2024-03-04T06:16:56.370840+0000","last_clean":"2024-03-04T06:16:56.370840+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:16:56.370840+0000","last_undegraded":"2024-03-04T06:16:56.370840+0000","last_fullsized":"2024-03-04T06:16:56.370840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1359,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.334190+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:16:56.334190+0000","last_peered":"2024-03-04T06:16:56.334190+0000","last_clean":"2024-03-04T06:16:56.334190+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:16:56.334190+0000","last_undegraded":"2024-03-04T06:16:56.334190+0000","last_fullsized":"2024-03-04T06:16:56.334190+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"544'515","reported_seq":1930,"reported_epoch":546,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:56.333690+0000","last_change":"2024-03-04T06:16:56.333690+0000","last_active":"2024-03-04T06:16:56.333690+0000","last_peered":"2024-03-04T06:16:56.333690+0000","last_clean":"2024-03-04T06:16:56.333690+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:16:56.333690+0000","last_undegraded":"2024-03-04T06:16:56.333690+0000","last_fullsized":"2024-03-04T06:16:56.333690+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":52,"num_object_clones":40,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":257,"num_read_kb":204750,"num_write":234,"num_write_kb":63712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1357,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.302417+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:16:55.302417+0000","last_peered":"2024-03-04T06:16:55.302417+0000","last_clean":"2024-03-04T06:16:55.302417+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:16:55.302417+0000","last_undegraded":"2024-03-04T06:16:55.302417+0000","last_fullsized":"2024-03-04T06:16:55.302417+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"531'366","reported_seq":1825,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.383228+0000","last_change":"2024-03-04T06:16:55.294953+0000","last_active":"2024-03-04T06:16:56.383228+0000","last_peered":"2024-03-04T06:16:56.383228+0000","last_clean":"2024-03-04T06:16:56.383228+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:16:56.383228+0000","last_undegraded":"2024-03-04T06:16:56.383228+0000","last_fullsized":"2024-03-04T06:16:56.383228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3143140,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":210,"num_read_kb":173945,"num_write":141,"num_write_kb":40250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1359,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.326183+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:16:56.326183+0000","last_peered":"2024-03-04T06:16:56.326183+0000","last_clean":"2024-03-04T06:16:56.326183+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:16:56.326183+0000","last_undegraded":"2024-03-04T06:16:56.326183+0000","last_fullsized":"2024-03-04T06:16:56.326183+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"545'865","reported_seq":2215,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.380596+0000","last_change":"2024-03-04T06:16:55.354005+0000","last_active":"2024-03-04T06:16:56.380596+0000","last_peered":"2024-03-04T06:16:56.380596+0000","last_clean":"2024-03-04T06:16:56.380596+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:16:56.380596+0000","last_undegraded":"2024-03-04T06:16:56.380596+0000","last_fullsized":"2024-03-04T06:16:56.380596+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":865,"log_dups_size":0,"ondisk_log_size":865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11446491,"num_objects":89,"num_object_clones":68,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":527,"num_read_kb":421075,"num_write":416,"num_write_kb":111943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1357,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.290772+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:16:55.290772+0000","last_peered":"2024-03-04T06:16:55.290772+0000","last_clean":"2024-03-04T06:16:55.290772+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:16:55.290772+0000","last_undegraded":"2024-03-04T06:16:55.290772+0000","last_fullsized":"2024-03-04T06:16:55.290772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"544'966","reported_seq":2199,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.296206+0000","last_change":"2024-03-04T06:16:55.296206+0000","last_active":"2024-03-04T06:16:55.296206+0000","last_peered":"2024-03-04T06:16:55.296206+0000","last_clean":"2024-03-04T06:16:55.296206+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:16:55.296206+0000","last_undegraded":"2024-03-04T06:16:55.296206+0000","last_fullsized":"2024-03-04T06:16:55.296206+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":966,"log_dups_size":0,"ondisk_log_size":966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13816278,"num_objects":96,"num_object_clones":74,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":401,"num_read_kb":277934,"num_write":439,"num_write_kb":115530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"544'447","reported_seq":1941,"reported_epoch":546,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:56.335805+0000","last_change":"2024-03-04T06:16:56.335805+0000","last_active":"2024-03-04T06:16:56.335805+0000","last_peered":"2024-03-04T06:16:56.335805+0000","last_clean":"2024-03-04T06:16:56.335805+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:16:56.335805+0000","last_undegraded":"2024-03-04T06:16:56.335805+0000","last_fullsized":"2024-03-04T06:16:56.335805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193858,"num_objects":47,"num_object_clones":35,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":299,"num_read_kb":229917,"num_write":219,"num_write_kb":59708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1385,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.311331+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:16:55.311331+0000","last_peered":"2024-03-04T06:16:55.311331+0000","last_clean":"2024-03-04T06:16:55.311331+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:16:55.311331+0000","last_undegraded":"2024-03-04T06:16:55.311331+0000","last_fullsized":"2024-03-04T06:16:55.311331+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"544'435","reported_seq":1913,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.316268+0000","last_change":"2024-03-04T06:16:55.316268+0000","last_active":"2024-03-04T06:16:55.316268+0000","last_peered":"2024-03-04T06:16:55.316268+0000","last_clean":"2024-03-04T06:16:55.316268+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:16:55.316268+0000","last_undegraded":"2024-03-04T06:16:55.316268+0000","last_fullsized":"2024-03-04T06:16:55.316268+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445666,"num_objects":52,"num_object_clones":37,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":221,"num_read_kb":149343,"num_write":205,"num_write_kb":54064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1354,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.385688+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:16:56.385688+0000","last_peered":"2024-03-04T06:16:56.385688+0000","last_clean":"2024-03-04T06:16:56.385688+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:16:56.385688+0000","last_undegraded":"2024-03-04T06:16:56.385688+0000","last_fullsized":"2024-03-04T06:16:56.385688+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"545'627","reported_seq":2001,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.381776+0000","last_change":"2024-03-04T06:16:55.355047+0000","last_active":"2024-03-04T06:16:56.381776+0000","last_peered":"2024-03-04T06:16:56.381776+0000","last_clean":"2024-03-04T06:16:56.381776+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:16:56.381776+0000","last_undegraded":"2024-03-04T06:16:56.381776+0000","last_fullsized":"2024-03-04T06:16:56.381776+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882912,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":328,"num_read_kb":253701,"num_write":282,"num_write_kb":78197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1385,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-03-04T06:16:55.293047+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:16:55.293047+0000","last_peered":"2024-03-04T06:16:55.293047+0000","last_clean":"2024-03-04T06:16:55.293047+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:16:55.293047+0000","last_undegraded":"2024-03-04T06:16:55.293047+0000","last_fullsized":"2024-03-04T06:16:55.293047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"544'752","reported_seq":2113,"reported_epoch":545,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:16:55.288356+0000","last_change":"2024-03-04T06:16:55.288356+0000","last_active":"2024-03-04T06:16:55.288356+0000","last_peered":"2024-03-04T06:16:55.288356+0000","last_clean":"2024-03-04T06:16:55.288356+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:16:55.288356+0000","last_undegraded":"2024-03-04T06:16:55.288356+0000","last_fullsized":"2024-03-04T06:16:55.288356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":70,"num_object_clones":61,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":422,"num_read_kb":335445,"num_write":322,"num_write_kb":87782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1359,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-03-04T06:16:56.335183+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:16:56.335183+0000","last_peered":"2024-03-04T06:16:56.335183+0000","last_clean":"2024-03-04T06:16:56.335183+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:16:56.335183+0000","last_undegraded":"2024-03-04T06:16:56.335183+0000","last_fullsized":"2024-03-04T06:16:56.335183+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":100227335,"num_objects":783,"num_object_clones":603,"num_object_copies":2349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4063,"num_read_kb":3131773,"num_write":3504,"num_write_kb":954648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7579,"ondisk_log_size":7579,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":161692,"kb_used_data":112660,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94210148,"statfs":{"total":96636764160,"available":96471191552,"internally_reserved":0,"allocated":115363840,"data_stored":132024912,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":139236,"kb_used_data":94364,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94232604,"statfs":{"total":96636764160,"available":96494186496,"internally_reserved":0,"allocated":96628736,"data_stored":110809326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":124928,"kb_used_data":81464,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94246912,"statfs":{"total":96636764160,"available":96508837888,"internally_reserved":0,"allocated":83419136,"data_stored":96046901,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":113896,"kb_used_data":74592,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94257944,"statfs":{"total":96636764160,"available":96520134656,"internally_reserved":0,"allocated":76382208,"data_stored":85616397,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:16:57.368 INFO:tasks.rados.rados.0.smithi064.stdout:2239: writing smithi06431743-35 from 2523106 to 3142440 tid 3 2024-03-04T06:16:57.368 INFO:tasks.rados.rados.0.smithi064.stdout:2240: read oid 6 snap -1 2024-03-04T06:16:57.368 INFO:tasks.rados.rados.0.smithi064.stdout:2240: expect deleted 2024-03-04T06:16:57.368 INFO:tasks.rados.rados.0.smithi064.stdout:2241: read oid 29 snap -1 2024-03-04T06:16:57.368 INFO:tasks.rados.rados.0.smithi064.stdout:2241: expect (ObjNum 1016 snap 261 seq_num 1016) 2024-03-04T06:16:57.368 INFO:tasks.rados.rados.0.smithi064.stdout:2242: write (excl) oid 8 current snap is 269 2024-03-04T06:16:57.368 INFO:tasks.rados.rados.0.smithi064.stdout:2242: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2024-03-04T06:16:57.371 INFO:tasks.rados.rados.0.smithi064.stdout:2242: writing smithi06431743-8 from 497771 to 991682 tid 1 2024-03-04T06:16:57.378 INFO:tasks.rados.rados.0.smithi064.stdout:2242: writing smithi06431743-8 from 1619186 to 2173206 tid 2 2024-03-04T06:16:57.384 INFO:tasks.rados.rados.0.smithi064.stdout:2242: writing smithi06431743-8 from 2950529 to 3381135 tid 3 2024-03-04T06:16:57.384 INFO:tasks.rados.rados.0.smithi064.stdout:2243: snap_remove snap 257 2024-03-04T06:16:58.350 INFO:tasks.rados.rados.0.smithi064.stdout:2243: done (6 left) 2024-03-04T06:16:58.350 INFO:tasks.rados.rados.0.smithi064.stdout:2244: read oid 7 snap -1 2024-03-04T06:16:58.350 INFO:tasks.rados.rados.0.smithi064.stdout:2244: expect (ObjNum 1020 snap 262 seq_num 1020) 2024-03-04T06:16:58.350 INFO:tasks.rados.rados.0.smithi064.stdout:2245: read oid 28 snap -1 2024-03-04T06:16:58.350 INFO:tasks.rados.rados.0.smithi064.stdout:2245: expect (ObjNum 982 snap 249 seq_num 982) 2024-03-04T06:16:58.350 INFO:tasks.rados.rados.0.smithi064.stdout:2246: read oid 4 snap -1 2024-03-04T06:16:58.350 INFO:tasks.rados.rados.0.smithi064.stdout:2246: expect (ObjNum 1047 snap 265 seq_num 1047) 2024-03-04T06:16:58.350 INFO:tasks.rados.rados.0.smithi064.stdout:2247: snap_create 2024-03-04T06:16:58.351 INFO:tasks.rados.rados.0.smithi064.stdout:2236: expect (ObjNum 1043 snap 265 seq_num 1043) 2024-03-04T06:16:58.508 INFO:tasks.rados.rados.0.smithi064.stdout:2239: finishing write tid 1 to smithi06431743-35 2024-03-04T06:16:58.508 INFO:tasks.rados.rados.0.smithi064.stdout:2239: finishing write tid 2 to smithi06431743-35 2024-03-04T06:16:58.508 INFO:tasks.rados.rados.0.smithi064.stdout:2241: expect (ObjNum 1016 snap 261 seq_num 1016) 2024-03-04T06:16:58.731 INFO:tasks.rados.rados.0.smithi064.stdout:2242: finishing write tid 1 to smithi06431743-8 2024-03-04T06:16:58.731 INFO:tasks.rados.rados.0.smithi064.stdout:2239: finishing write tid 3 to smithi06431743-35 2024-03-04T06:16:58.731 INFO:tasks.rados.rados.0.smithi064.stdout:2242: finishing write tid 2 to smithi06431743-8 2024-03-04T06:16:58.731 INFO:tasks.rados.rados.0.smithi064.stdout:2239: finishing write tid 4 to smithi06431743-35 2024-03-04T06:16:58.731 INFO:tasks.rados.rados.0.smithi064.stdout:2239: finishing write tid 5 to smithi06431743-35 2024-03-04T06:16:58.732 INFO:tasks.rados.rados.0.smithi064.stdout:2239: oid 35 updating version 0 to 632 2024-03-04T06:16:58.732 INFO:tasks.rados.rados.0.smithi064.stdout:2239: oid 35 version 632 is already newer than 630 2024-03-04T06:16:58.732 INFO:tasks.rados.rados.0.smithi064.stdout:2239: oid 35 version 632 is already newer than 631 2024-03-04T06:16:58.732 INFO:tasks.rados.rados.0.smithi064.stdout:2239: oid 35 version 632 is already newer than 629 2024-03-04T06:16:58.732 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 632 (ObjNum 1052 snap 269 seq_num 1052) dirty exists 2024-03-04T06:16:58.732 INFO:tasks.rados.rados.0.smithi064.stdout:2239: left oid 35 (ObjNum 1052 snap 269 seq_num 1052) 2024-03-04T06:16:58.732 INFO:tasks.rados.rados.0.smithi064.stdout:2242: finishing write tid 3 to smithi06431743-8 2024-03-04T06:16:58.733 INFO:tasks.rados.rados.0.smithi064.stdout:2242: finishing write tid 4 to smithi06431743-8 2024-03-04T06:16:58.733 INFO:tasks.rados.rados.0.smithi064.stdout:2242: finishing write tid 5 to smithi06431743-8 2024-03-04T06:16:58.733 INFO:tasks.rados.rados.0.smithi064.stdout:2242: oid 8 updating version 0 to 441 2024-03-04T06:16:58.733 INFO:tasks.rados.rados.0.smithi064.stdout:2242: oid 8 version 441 is already newer than 438 2024-03-04T06:16:58.733 INFO:tasks.rados.rados.0.smithi064.stdout:2242: oid 8 version 441 is already newer than 439 2024-03-04T06:16:58.733 INFO:tasks.rados.rados.0.smithi064.stdout:2242: oid 8 version 441 is already newer than 441 2024-03-04T06:16:58.733 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 441 (ObjNum 1053 snap 269 seq_num 1053) dirty exists 2024-03-04T06:16:58.733 INFO:tasks.rados.rados.0.smithi064.stdout:2242: left oid 8 (ObjNum 1053 snap 269 seq_num 1053) 2024-03-04T06:16:58.733 INFO:tasks.rados.rados.0.smithi064.stdout:2245: expect (ObjNum 982 snap 249 seq_num 982) 2024-03-04T06:16:58.927 INFO:tasks.rados.rados.0.smithi064.stdout:2244: expect (ObjNum 1020 snap 262 seq_num 1020) 2024-03-04T06:16:59.047 INFO:tasks.rados.rados.0.smithi064.stdout:2246: expect (ObjNum 1047 snap 265 seq_num 1047) 2024-03-04T06:16:59.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:16:59.353 INFO:tasks.rados.rados.0.smithi064.stdout:2236: done (9 left) 2024-03-04T06:16:59.353 INFO:tasks.rados.rados.0.smithi064.stdout:2237: done (8 left) 2024-03-04T06:16:59.353 INFO:tasks.rados.rados.0.smithi064.stdout:2239: done (7 left) 2024-03-04T06:16:59.353 INFO:tasks.rados.rados.0.smithi064.stdout:2240: done (6 left) 2024-03-04T06:16:59.353 INFO:tasks.rados.rados.0.smithi064.stdout:2241: done (5 left) 2024-03-04T06:16:59.353 INFO:tasks.rados.rados.0.smithi064.stdout:2242: done (4 left) 2024-03-04T06:16:59.354 INFO:tasks.rados.rados.0.smithi064.stdout:2244: done (3 left) 2024-03-04T06:16:59.354 INFO:tasks.rados.rados.0.smithi064.stdout:2245: done (2 left) 2024-03-04T06:16:59.354 INFO:tasks.rados.rados.0.smithi064.stdout:2246: done (1 left) 2024-03-04T06:16:59.354 INFO:tasks.rados.rados.0.smithi064.stdout:2247: done (0 left) 2024-03-04T06:16:59.354 INFO:tasks.rados.rados.0.smithi064.stdout:2248: write oid 35 current snap is 270 2024-03-04T06:16:59.354 INFO:tasks.rados.rados.0.smithi064.stdout:2248: seq_num 1054 ranges {514578=569160,1677668=414031,2577952=641878} 2024-03-04T06:16:59.359 INFO:tasks.rados.rados.0.smithi064.stdout:2248: writing smithi06431743-35 from 514578 to 1083738 tid 1 2024-03-04T06:16:59.365 INFO:tasks.rados.rados.0.smithi064.stdout:2248: writing smithi06431743-35 from 1677668 to 2091699 tid 2 2024-03-04T06:16:59.373 INFO:tasks.rados.rados.0.smithi064.stdout:2248: writing smithi06431743-35 from 2577952 to 3219830 tid 3 2024-03-04T06:16:59.373 INFO:tasks.rados.rados.0.smithi064.stdout:2249: write (excl) oid 12 current snap is 270 2024-03-04T06:16:59.373 INFO:tasks.rados.rados.0.smithi064.stdout:2249: seq_num 1055 ranges {531385=644409,1736150=557689,3005375=569503} 2024-03-04T06:16:59.379 INFO:tasks.rados.rados.0.smithi064.stdout:2249: writing smithi06431743-12 from 531385 to 1175794 tid 1 2024-03-04T06:16:59.386 INFO:tasks.rados.rados.0.smithi064.stdout:2249: writing smithi06431743-12 from 1736150 to 2293839 tid 2 2024-03-04T06:16:59.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:59.385+0000 7f9992d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:59.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:16:59.390+0000 7f9992d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:16:59.394 INFO:tasks.rados.rados.0.smithi064.stdout:2249: writing smithi06431743-12 from 3005375 to 3574878 tid 3 2024-03-04T06:16:59.394 INFO:tasks.rados.rados.0.smithi064.stdout:2250: read oid 6 snap 268 2024-03-04T06:16:59.394 INFO:tasks.rados.rados.0.smithi064.stdout:2250: expect deleted 2024-03-04T06:16:59.394 INFO:tasks.rados.rados.0.smithi064.stdout:2251: snap_create 2024-03-04T06:16:59.447 INFO:tasks.rados.rados.0.smithi064.stdout:2248: finishing write tid 1 to smithi06431743-35 2024-03-04T06:16:59.468 INFO:tasks.rados.rados.0.smithi064.stdout:2249: finishing write tid 1 to smithi06431743-12 2024-03-04T06:16:59.474 INFO:tasks.rados.rados.0.smithi064.stdout:2248: finishing write tid 2 to smithi06431743-35 2024-03-04T06:16:59.490 INFO:tasks.rados.rados.0.smithi064.stdout:2249: finishing write tid 2 to smithi06431743-12 2024-03-04T06:16:59.494 INFO:tasks.rados.rados.0.smithi064.stdout:2248: finishing write tid 3 to smithi06431743-35 2024-03-04T06:16:59.503 INFO:tasks.rados.rados.0.smithi064.stdout:2248: finishing write tid 4 to smithi06431743-35 2024-03-04T06:16:59.505 INFO:tasks.rados.rados.0.smithi064.stdout:2248: finishing write tid 5 to smithi06431743-35 2024-03-04T06:16:59.505 INFO:tasks.rados.rados.0.smithi064.stdout:2248: oid 35 updating version 0 to 643 2024-03-04T06:16:59.505 INFO:tasks.rados.rados.0.smithi064.stdout:2248: oid 35 version 643 is already newer than 642 2024-03-04T06:16:59.506 INFO:tasks.rados.rados.0.smithi064.stdout:2248: oid 35 version 643 is already newer than 641 2024-03-04T06:16:59.506 INFO:tasks.rados.rados.0.smithi064.stdout:2248: oid 35 version 643 is already newer than 640 2024-03-04T06:16:59.506 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 643 (ObjNum 1054 snap 270 seq_num 1054) dirty exists 2024-03-04T06:16:59.506 INFO:tasks.rados.rados.0.smithi064.stdout:2248: left oid 35 (ObjNum 1054 snap 270 seq_num 1054) 2024-03-04T06:16:59.509 INFO:tasks.rados.rados.0.smithi064.stdout:2249: finishing write tid 3 to smithi06431743-12 2024-03-04T06:16:59.511 INFO:tasks.rados.rados.0.smithi064.stdout:2249: finishing write tid 4 to smithi06431743-12 2024-03-04T06:16:59.513 INFO:tasks.rados.rados.0.smithi064.stdout:2249: finishing write tid 5 to smithi06431743-12 2024-03-04T06:16:59.513 INFO:tasks.rados.rados.0.smithi064.stdout:2249: oid 12 updating version 0 to 345 2024-03-04T06:16:59.513 INFO:tasks.rados.rados.0.smithi064.stdout:2249: oid 12 updating version 345 to 347 2024-03-04T06:16:59.513 INFO:tasks.rados.rados.0.smithi064.stdout:2249: oid 12 version 347 is already newer than 347 2024-03-04T06:16:59.513 INFO:tasks.rados.rados.0.smithi064.stdout:2249: oid 12 version 347 is already newer than 344 2024-03-04T06:16:59.514 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 347 (ObjNum 1055 snap 270 seq_num 1055) dirty exists 2024-03-04T06:16:59.514 INFO:tasks.rados.rados.0.smithi064.stdout:2249: left oid 12 (ObjNum 1055 snap 270 seq_num 1055) 2024-03-04T06:17:00.353 INFO:tasks.rados.rados.0.smithi064.stdout:2248: done (3 left) 2024-03-04T06:17:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2249: done (2 left) 2024-03-04T06:17:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2250: done (1 left) 2024-03-04T06:17:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2251: done (0 left) 2024-03-04T06:17:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2252: read oid 40 snap -1 2024-03-04T06:17:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2252: expect (ObjNum 1004 snap 259 seq_num 1004) 2024-03-04T06:17:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2253: read oid 43 snap -1 2024-03-04T06:17:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2253: expect (ObjNum 1033 snap 264 seq_num 1033) 2024-03-04T06:17:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2254: read oid 3 snap -1 2024-03-04T06:17:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2254: expect (ObjNum 1032 snap 264 seq_num 1032) 2024-03-04T06:17:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2255: write oid 50 current snap is 271 2024-03-04T06:17:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2255: seq_num 1056 ranges {548192=719658,1678279=417700,2516445=780775} 2024-03-04T06:17:00.360 INFO:tasks.rados.rados.0.smithi064.stdout:2255: writing smithi06431743-50 from 548192 to 1267850 tid 1 2024-03-04T06:17:00.365 INFO:tasks.rados.rados.0.smithi064.stdout:2255: writing smithi06431743-50 from 1678279 to 2095979 tid 2 2024-03-04T06:17:00.372 INFO:tasks.rados.rados.0.smithi064.stdout:2255: writing smithi06431743-50 from 2516445 to 3297220 tid 3 2024-03-04T06:17:00.372 INFO:tasks.rados.rados.0.smithi064.stdout:2256: delete oid 27 current snap is 271 2024-03-04T06:17:00.406 INFO:tasks.rados.rados.0.smithi064.stdout:2254: expect (ObjNum 1032 snap 264 seq_num 1032) 2024-03-04T06:17:00.548 INFO:tasks.rados.rados.0.smithi064.stdout:2252: expect (ObjNum 1004 snap 259 seq_num 1004) 2024-03-04T06:17:00.656 INFO:tasks.rados.rados.0.smithi064.stdout:2253: expect (ObjNum 1033 snap 264 seq_num 1033) 2024-03-04T06:17:00.850 INFO:tasks.rados.rados.0.smithi064.stdout:2255: finishing write tid 1 to smithi06431743-50 2024-03-04T06:17:00.850 INFO:tasks.rados.rados.0.smithi064.stdout:2255: finishing write tid 2 to smithi06431743-50 2024-03-04T06:17:00.850 INFO:tasks.rados.rados.0.smithi064.stdout:2255: finishing write tid 3 to smithi06431743-50 2024-03-04T06:17:00.850 INFO:tasks.rados.rados.0.smithi064.stdout:2255: finishing write tid 4 to smithi06431743-50 2024-03-04T06:17:00.850 INFO:tasks.rados.rados.0.smithi064.stdout:2255: finishing write tid 5 to smithi06431743-50 2024-03-04T06:17:00.851 INFO:tasks.rados.rados.0.smithi064.stdout:2255: oid 50 updating version 0 to 765 2024-03-04T06:17:00.851 INFO:tasks.rados.rados.0.smithi064.stdout:2255: oid 50 version 765 is already newer than 763 2024-03-04T06:17:00.851 INFO:tasks.rados.rados.0.smithi064.stdout:2255: oid 50 version 765 is already newer than 765 2024-03-04T06:17:00.851 INFO:tasks.rados.rados.0.smithi064.stdout:2255: oid 50 version 765 is already newer than 762 2024-03-04T06:17:00.851 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 765 (ObjNum 1056 snap 271 seq_num 1056) dirty exists 2024-03-04T06:17:00.851 INFO:tasks.rados.rados.0.smithi064.stdout:2255: left oid 50 (ObjNum 1056 snap 271 seq_num 1056) 2024-03-04T06:17:00.851 INFO:tasks.rados.rados.0.smithi064.stdout:2252: done (4 left) 2024-03-04T06:17:00.851 INFO:tasks.rados.rados.0.smithi064.stdout:2253: done (3 left) 2024-03-04T06:17:00.851 INFO:tasks.rados.rados.0.smithi064.stdout:2254: done (2 left) 2024-03-04T06:17:00.851 INFO:tasks.rados.rados.0.smithi064.stdout:2255: done (1 left) 2024-03-04T06:17:00.852 INFO:tasks.rados.rados.0.smithi064.stdout:2256: done (0 left) 2024-03-04T06:17:00.852 INFO:tasks.rados.rados.0.smithi064.stdout:2257: read oid 30 snap 268 2024-03-04T06:17:00.852 INFO:tasks.rados.rados.0.smithi064.stdout:2257: expect deleted 2024-03-04T06:17:00.852 INFO:tasks.rados.rados.0.smithi064.stdout:2258: read oid 29 snap 269 2024-03-04T06:17:00.852 INFO:tasks.rados.rados.0.smithi064.stdout:2258: expect (ObjNum 1016 snap 261 seq_num 1016) 2024-03-04T06:17:00.852 INFO:tasks.rados.rados.0.smithi064.stdout:2259: snap_create 2024-03-04T06:17:00.878 INFO:tasks.rados.rados.0.smithi064.stdout:2258: expect (ObjNum 1016 snap 261 seq_num 1016) 2024-03-04T06:17:01.358 INFO:tasks.rados.rados.0.smithi064.stdout:2257: done (2 left) 2024-03-04T06:17:01.358 INFO:tasks.rados.rados.0.smithi064.stdout:2258: done (1 left) 2024-03-04T06:17:01.358 INFO:tasks.rados.rados.0.smithi064.stdout:2259: done (0 left) 2024-03-04T06:17:01.358 INFO:tasks.rados.rados.0.smithi064.stdout:2260: snap_remove snap 267 2024-03-04T06:17:01.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:01.637 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:01.635+0000 7f667746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:01.640 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:01.639+0000 7f667746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:02.359 INFO:tasks.rados.rados.0.smithi064.stdout:2260: done (0 left) 2024-03-04T06:17:02.359 INFO:tasks.rados.rados.0.smithi064.stdout:2261: delete oid 47 current snap is 272 2024-03-04T06:17:02.367 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:17:02.367 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:17:02.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:17:02.390 INFO:tasks.rados.rados.0.smithi064.stdout:2261: done (0 left) 2024-03-04T06:17:02.390 INFO:tasks.rados.rados.0.smithi064.stdout:2262: delete oid 45 current snap is 272 2024-03-04T06:17:02.414 INFO:tasks.rados.rados.0.smithi064.stdout:2262: done (0 left) 2024-03-04T06:17:02.414 INFO:tasks.rados.rados.0.smithi064.stdout:2263: delete oid 23 current snap is 272 2024-03-04T06:17:02.417 INFO:tasks.rados.rados.0.smithi064.stdout:2263: done (0 left) 2024-03-04T06:17:02.417 INFO:tasks.rados.rados.0.smithi064.stdout:2264: snap_remove snap 268 2024-03-04T06:17:02.489 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:02.485+0000 7f6f42c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:02.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:02.492+0000 7f6f42c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:02.680 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:17:02.681 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:17:02.692 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":821,"stamp":"2024-03-04T06:17:02.357204+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107070204,"num_objects":788,"num_object_clones":606,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":147,"num_read":4210,"num_read_kb":3167727,"num_write":3665,"num_write_kb":963591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7784,"ondisk_log_size":7784,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":559852,"kb_used_data":383180,"kb_used_omap":0,"kb_used_meta":176640,"kb_avail":376927508,"statfs":{"total":386547056640,"available":385973768192,"internally_reserved":0,"allocated":392376320,"data_stored":449243670,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":180879360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6252501,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":44,"num_read_kb":35863,"num_write":24,"num_write_kb":7049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009838"},"pg_stats":[{"pgid":"3.f","version":"549'468","reported_seq":1926,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-04T06:16:59.416748+0000","last_change":"2024-03-04T06:16:59.449515+0000","last_active":"2024-03-04T06:16:59.416748+0000","last_peered":"2024-03-04T06:16:59.416748+0000","last_clean":"2024-03-04T06:16:59.416748+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:16:59.416748+0000","last_undegraded":"2024-03-04T06:16:59.416748+0000","last_fullsized":"2024-03-04T06:16:59.416748+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8942017,"num_objects":51,"num_object_clones":40,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":207696,"num_write":215,"num_write_kb":58378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"548'362","reported_seq":1854,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-04T06:16:58.365409+0000","last_change":"2024-03-04T06:16:58.462082+0000","last_active":"2024-03-04T06:16:58.365409+0000","last_peered":"2024-03-04T06:16:58.365409+0000","last_clean":"2024-03-04T06:16:58.365409+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:16:58.365409+0000","last_undegraded":"2024-03-04T06:16:58.365409+0000","last_fullsized":"2024-03-04T06:16:58.365409+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9799174,"num_objects":42,"num_object_clones":28,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":135,"num_read_kb":80771,"num_write":170,"num_write_kb":44833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"548'505","reported_seq":1998,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.366784+0000","last_change":"2024-03-04T06:16:58.431666+0000","last_active":"2024-03-04T06:17:00.366784+0000","last_peered":"2024-03-04T06:17:00.366784+0000","last_clean":"2024-03-04T06:17:00.366784+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:17:00.366784+0000","last_undegraded":"2024-03-04T06:17:00.366784+0000","last_fullsized":"2024-03-04T06:17:00.366784+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7160211,"num_objects":59,"num_object_clones":46,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":329,"num_read_kb":259518,"num_write":252,"num_write_kb":69591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"549'347","reported_seq":1864,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-04T06:16:59.512799+0000","last_change":"2024-03-04T06:16:59.514362+0000","last_active":"2024-03-04T06:16:59.512799+0000","last_peered":"2024-03-04T06:16:59.512799+0000","last_clean":"2024-03-04T06:16:59.512799+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:16:59.512799+0000","last_undegraded":"2024-03-04T06:16:59.512799+0000","last_fullsized":"2024-03-04T06:16:59.512799+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008981,"num_objects":36,"num_object_clones":26,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":195,"num_read_kb":149128,"num_write":174,"num_write_kb":48785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1709,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.381729+0000","last_change":"2024-03-04T06:16:58.391238+0000","last_active":"2024-03-04T06:17:00.381729+0000","last_peered":"2024-03-04T06:17:00.381729+0000","last_clean":"2024-03-04T06:17:00.381729+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:17:00.381729+0000","last_undegraded":"2024-03-04T06:17:00.381729+0000","last_fullsized":"2024-03-04T06:17:00.381729+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":58,"num_read_kb":33096,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"549'535","reported_seq":1950,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-04T06:16:59.422825+0000","last_change":"2024-03-04T06:16:59.463067+0000","last_active":"2024-03-04T06:16:59.422825+0000","last_peered":"2024-03-04T06:16:59.422825+0000","last_clean":"2024-03-04T06:16:59.422825+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:16:59.422825+0000","last_undegraded":"2024-03-04T06:16:59.422825+0000","last_fullsized":"2024-03-04T06:16:59.422825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2689351,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":252,"num_read_kb":192235,"num_write":246,"num_write_kb":71955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1619,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-04T06:16:59.422034+0000","last_change":"2024-03-04T06:16:59.430876+0000","last_active":"2024-03-04T06:16:59.422034+0000","last_peered":"2024-03-04T06:16:59.422034+0000","last_clean":"2024-03-04T06:16:59.422034+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:16:59.422034+0000","last_undegraded":"2024-03-04T06:16:59.422034+0000","last_fullsized":"2024-03-04T06:16:59.422034+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"548'264","reported_seq":1816,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.367712+0000","last_change":"2024-03-04T06:16:58.400270+0000","last_active":"2024-03-04T06:17:00.367712+0000","last_peered":"2024-03-04T06:17:00.367712+0000","last_clean":"2024-03-04T06:17:00.367712+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:17:00.367712+0000","last_undegraded":"2024-03-04T06:17:00.367712+0000","last_fullsized":"2024-03-04T06:17:00.367712+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6437477,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":176,"num_read_kb":163221,"num_write":123,"num_write_kb":32558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1499,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.359208+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:17:00.359208+0000","last_peered":"2024-03-04T06:17:00.359208+0000","last_clean":"2024-03-04T06:17:00.359208+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:17:00.359208+0000","last_undegraded":"2024-03-04T06:17:00.359208+0000","last_fullsized":"2024-03-04T06:17:00.359208+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1374,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.385053+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:17:00.385053+0000","last_peered":"2024-03-04T06:17:00.385053+0000","last_clean":"2024-03-04T06:17:00.385053+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:17:00.385053+0000","last_undegraded":"2024-03-04T06:17:00.385053+0000","last_fullsized":"2024-03-04T06:17:00.385053+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"548'519","reported_seq":1951,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.399521+0000","last_change":"2024-03-04T06:16:58.415728+0000","last_active":"2024-03-04T06:17:00.399521+0000","last_peered":"2024-03-04T06:17:00.399521+0000","last_clean":"2024-03-04T06:17:00.399521+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:17:00.399521+0000","last_undegraded":"2024-03-04T06:17:00.399521+0000","last_fullsized":"2024-03-04T06:17:00.399521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":52,"num_object_clones":40,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":263,"num_read_kb":210737,"num_write":234,"num_write_kb":63712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1367,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-04T06:16:59.417673+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:16:59.417673+0000","last_peered":"2024-03-04T06:16:59.417673+0000","last_clean":"2024-03-04T06:16:59.417673+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:16:59.417673+0000","last_undegraded":"2024-03-04T06:16:59.417673+0000","last_fullsized":"2024-03-04T06:16:59.417673+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"531'366","reported_seq":1842,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.375615+0000","last_change":"2024-03-04T06:16:58.381996+0000","last_active":"2024-03-04T06:17:00.375615+0000","last_peered":"2024-03-04T06:17:00.375615+0000","last_clean":"2024-03-04T06:17:00.375615+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:17:00.375615+0000","last_undegraded":"2024-03-04T06:17:00.375615+0000","last_fullsized":"2024-03-04T06:17:00.375615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3143140,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":216,"num_read_kb":177500,"num_write":141,"num_write_kb":40250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1374,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.364532+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:17:00.364532+0000","last_peered":"2024-03-04T06:17:00.364532+0000","last_clean":"2024-03-04T06:17:00.364532+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:17:00.364532+0000","last_undegraded":"2024-03-04T06:17:00.364532+0000","last_fullsized":"2024-03-04T06:17:00.364532+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"548'869","reported_seq":2232,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.382629+0000","last_change":"2024-03-04T06:16:58.458461+0000","last_active":"2024-03-04T06:17:00.382629+0000","last_peered":"2024-03-04T06:17:00.382629+0000","last_clean":"2024-03-04T06:17:00.382629+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:17:00.382629+0000","last_undegraded":"2024-03-04T06:17:00.382629+0000","last_fullsized":"2024-03-04T06:17:00.382629+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":869,"log_dups_size":0,"ondisk_log_size":869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11446491,"num_objects":89,"num_object_clones":68,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":533,"num_read_kb":427271,"num_write":416,"num_write_kb":111943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1367,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-04T06:16:59.412636+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:16:59.412636+0000","last_peered":"2024-03-04T06:16:59.412636+0000","last_clean":"2024-03-04T06:16:59.412636+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:16:59.412636+0000","last_undegraded":"2024-03-04T06:16:59.412636+0000","last_fullsized":"2024-03-04T06:16:59.412636+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"548'982","reported_seq":2220,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-04T06:16:58.450802+0000","last_change":"2024-03-04T06:16:58.477240+0000","last_active":"2024-03-04T06:16:58.450802+0000","last_peered":"2024-03-04T06:16:58.450802+0000","last_clean":"2024-03-04T06:16:58.450802+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:16:58.450802+0000","last_undegraded":"2024-03-04T06:16:58.450802+0000","last_fullsized":"2024-03-04T06:16:58.450802+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":982,"log_dups_size":0,"ondisk_log_size":982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13816278,"num_objects":96,"num_object_clones":74,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":419,"num_read_kb":298054,"num_write":439,"num_write_kb":115530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"548'457","reported_seq":1959,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.386098+0000","last_change":"2024-03-04T06:16:58.443422+0000","last_active":"2024-03-04T06:17:00.386098+0000","last_peered":"2024-03-04T06:17:00.386098+0000","last_clean":"2024-03-04T06:17:00.386098+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:17:00.386098+0000","last_undegraded":"2024-03-04T06:17:00.386098+0000","last_fullsized":"2024-03-04T06:17:00.386098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":457,"log_dups_size":0,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193858,"num_objects":47,"num_object_clones":35,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":299,"num_read_kb":229917,"num_write":219,"num_write_kb":59708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1394,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-04T06:16:58.376667+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:16:58.376667+0000","last_peered":"2024-03-04T06:16:58.376667+0000","last_clean":"2024-03-04T06:16:58.376667+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:16:58.376667+0000","last_undegraded":"2024-03-04T06:16:58.376667+0000","last_fullsized":"2024-03-04T06:16:58.376667+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"548'443","reported_seq":1930,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-04T06:16:58.379248+0000","last_change":"2024-03-04T06:16:58.443453+0000","last_active":"2024-03-04T06:16:58.379248+0000","last_peered":"2024-03-04T06:16:58.379248+0000","last_clean":"2024-03-04T06:16:58.379248+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:16:58.379248+0000","last_undegraded":"2024-03-04T06:16:58.379248+0000","last_fullsized":"2024-03-04T06:16:58.379248+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143437,"num_objects":52,"num_object_clones":37,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":223,"num_read_kb":149344,"num_write":210,"num_write_kb":55510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1366,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.388178+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:17:00.388178+0000","last_peered":"2024-03-04T06:17:00.388178+0000","last_clean":"2024-03-04T06:17:00.388178+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:17:00.388178+0000","last_undegraded":"2024-03-04T06:17:00.388178+0000","last_fullsized":"2024-03-04T06:17:00.388178+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"549'643","reported_seq":2025,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.374795+0000","last_change":"2024-03-04T06:16:58.471924+0000","last_active":"2024-03-04T06:17:00.374795+0000","last_peered":"2024-03-04T06:17:00.374795+0000","last_clean":"2024-03-04T06:17:00.374795+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:17:00.374795+0000","last_undegraded":"2024-03-04T06:17:00.374795+0000","last_fullsized":"2024-03-04T06:17:00.374795+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6571999,"num_objects":64,"num_object_clones":51,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":331,"num_read_kb":253703,"num_write":292,"num_write_kb":81200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1394,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-04T06:16:58.367759+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:16:58.367759+0000","last_peered":"2024-03-04T06:16:58.367759+0000","last_clean":"2024-03-04T06:16:58.367759+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:16:58.367759+0000","last_undegraded":"2024-03-04T06:16:58.367759+0000","last_fullsized":"2024-03-04T06:16:58.367759+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"548'760","reported_seq":2125,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-04T06:16:58.364523+0000","last_change":"2024-03-04T06:16:58.463379+0000","last_active":"2024-03-04T06:16:58.364523+0000","last_peered":"2024-03-04T06:16:58.364523+0000","last_clean":"2024-03-04T06:16:58.364523+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:16:58.364523+0000","last_undegraded":"2024-03-04T06:16:58.364523+0000","last_fullsized":"2024-03-04T06:16:58.364523+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":760,"log_dups_size":0,"ondisk_log_size":760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":70,"num_object_clones":61,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":422,"num_read_kb":335445,"num_write":322,"num_write_kb":87782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1374,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-04T06:17:00.379684+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:17:00.379684+0000","last_peered":"2024-03-04T06:17:00.379684+0000","last_clean":"2024-03-04T06:17:00.379684+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:17:00.379684+0000","last_undegraded":"2024-03-04T06:17:00.379684+0000","last_fullsized":"2024-03-04T06:17:00.379684+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":106479836,"num_objects":786,"num_object_clones":606,"num_object_copies":2358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":147,"num_read":4107,"num_read_kb":3167636,"num_write":3528,"num_write_kb":961697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7687,"ondisk_log_size":7687,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":168496,"kb_used_data":119464,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94203344,"statfs":{"total":96636764160,"available":96464224256,"internally_reserved":0,"allocated":122331136,"data_stored":140370334,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":142844,"kb_used_data":97972,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94228996,"statfs":{"total":96636764160,"available":96490491904,"internally_reserved":0,"allocated":100323328,"data_stored":115268755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706218,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":128344,"kb_used_data":84880,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94243496,"statfs":{"total":96636764160,"available":96505339904,"internally_reserved":0,"allocated":86917120,"data_stored":99545737,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706218,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":120168,"kb_used_data":80864,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94251672,"statfs":{"total":96636764160,"available":96513712128,"internally_reserved":0,"allocated":82804736,"data_stored":94058844,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:17:03.263 INFO:tasks.rados.rados.0.smithi064.stdout:2264: done (0 left) 2024-03-04T06:17:03.264 INFO:tasks.rados.rados.0.smithi064.stdout:2265: write (excl) oid 18 current snap is 272 2024-03-04T06:17:03.264 INFO:tasks.rados.rados.0.smithi064.stdout:2265: seq_num 1061 ranges {632227=412256,1570689=568696,2653560=753488} 2024-03-04T06:17:03.269 INFO:tasks.rados.rados.0.smithi064.stdout:2265: writing smithi06431743-18 from 632227 to 1044483 tid 1 2024-03-04T06:17:03.275 INFO:tasks.rados.rados.0.smithi064.stdout:2265: writing smithi06431743-18 from 1570689 to 2139385 tid 2 2024-03-04T06:17:03.282 INFO:tasks.rados.rados.0.smithi064.stdout:2265: writing smithi06431743-18 from 2653560 to 3407048 tid 3 2024-03-04T06:17:03.282 INFO:tasks.rados.rados.0.smithi064.stdout:2266: write (excl) oid 34 current snap is 272 2024-03-04T06:17:03.282 INFO:tasks.rados.rados.0.smithi064.stdout:2266: seq_num 1062 ranges {649034=487505,1629171=428707,2680983=564760,3798768=50266} 2024-03-04T06:17:03.288 INFO:tasks.rados.rados.0.smithi064.stdout:2266: writing smithi06431743-34 from 649034 to 1136539 tid 1 2024-03-04T06:17:03.293 INFO:tasks.rados.rados.0.smithi064.stdout:2266: writing smithi06431743-34 from 1629171 to 2057878 tid 2 2024-03-04T06:17:03.300 INFO:tasks.rados.rados.0.smithi064.stdout:2266: writing smithi06431743-34 from 2680983 to 3245743 tid 3 2024-03-04T06:17:03.303 INFO:tasks.rados.rados.0.smithi064.stdout:2266: writing smithi06431743-34 from 3798768 to 3849034 tid 4 2024-03-04T06:17:03.303 INFO:tasks.rados.rados.0.smithi064.stdout:2267: read oid 33 snap -1 2024-03-04T06:17:03.303 INFO:tasks.rados.rados.0.smithi064.stdout:2267: expect (ObjNum 1042 snap 265 seq_num 1042) 2024-03-04T06:17:03.303 INFO:tasks.rados.rados.0.smithi064.stdout:2268: snap_remove snap 260 2024-03-04T06:17:03.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:03.815 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:03.813+0000 7fae9e35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:03.818 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:03.817+0000 7fae9e35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:04.267 INFO:tasks.rados.rados.0.smithi064.stdout:2268: done (3 left) 2024-03-04T06:17:04.267 INFO:tasks.rados.rados.0.smithi064.stdout:2269: delete oid 14 current snap is 272 2024-03-04T06:17:04.268 INFO:tasks.rados.rados.0.smithi064.stdout:2267: expect (ObjNum 1042 snap 265 seq_num 1042) 2024-03-04T06:17:04.489 INFO:tasks.rados.rados.0.smithi064.stdout:2266: finishing write tid 1 to smithi06431743-34 2024-03-04T06:17:04.489 INFO:tasks.rados.rados.0.smithi064.stdout:2265: finishing write tid 1 to smithi06431743-18 2024-03-04T06:17:04.489 INFO:tasks.rados.rados.0.smithi064.stdout:2265: finishing write tid 2 to smithi06431743-18 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2266: finishing write tid 2 to smithi06431743-34 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2266: finishing write tid 3 to smithi06431743-34 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2267: done (3 left) 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2269: done (2 left) 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2270: write oid 26 current snap is 272 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2265: finishing write tid 3 to smithi06431743-18 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2266: finishing write tid 4 to smithi06431743-34 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2265: finishing write tid 4 to smithi06431743-18 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2265: finishing write tid 5 to smithi06431743-18 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2265: oid 18 updating version 0 to 373 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2265: oid 18 updating version 373 to 375 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2265: oid 18 version 375 is already newer than 372 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:2265: oid 18 version 375 is already newer than 375 2024-03-04T06:17:04.490 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 375 (ObjNum 1061 snap 272 seq_num 1061) dirty exists 2024-03-04T06:17:04.491 INFO:tasks.rados.rados.0.smithi064.stdout:2265: left oid 18 (ObjNum 1061 snap 272 seq_num 1061) 2024-03-04T06:17:04.491 INFO:tasks.rados.rados.0.smithi064.stdout:2266: finishing write tid 5 to smithi06431743-34 2024-03-04T06:17:04.491 INFO:tasks.rados.rados.0.smithi064.stdout:2266: finishing write tid 6 to smithi06431743-34 2024-03-04T06:17:04.491 INFO:tasks.rados.rados.0.smithi064.stdout:2266: oid 34 updating version 0 to 875 2024-03-04T06:17:04.491 INFO:tasks.rados.rados.0.smithi064.stdout:2266: oid 34 version 875 is already newer than 871 2024-03-04T06:17:04.491 INFO:tasks.rados.rados.0.smithi064.stdout:2266: oid 34 version 875 is already newer than 874 2024-03-04T06:17:04.491 INFO:tasks.rados.rados.0.smithi064.stdout:2266: oid 34 version 875 is already newer than 873 2024-03-04T06:17:04.491 INFO:tasks.rados.rados.0.smithi064.stdout:2266: oid 34 version 875 is already newer than 873 2024-03-04T06:17:04.491 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 875 (ObjNum 1062 snap 272 seq_num 1062) dirty exists 2024-03-04T06:17:04.491 INFO:tasks.rados.rados.0.smithi064.stdout:2266: left oid 34 (ObjNum 1062 snap 272 seq_num 1062) 2024-03-04T06:17:04.492 INFO:tasks.rados.rados.0.smithi064.stdout:2270: seq_num 1064 ranges {682648=638003,2029782=432376,3019476=587304} 2024-03-04T06:17:04.496 INFO:tasks.rados.rados.0.smithi064.stdout:2270: writing smithi06431743-26 from 682648 to 1320651 tid 1 2024-03-04T06:17:04.502 INFO:tasks.rados.rados.0.smithi064.stdout:2270: writing smithi06431743-26 from 2029782 to 2462158 tid 2 2024-03-04T06:17:04.509 INFO:tasks.rados.rados.0.smithi064.stdout:2270: writing smithi06431743-26 from 3019476 to 3606780 tid 3 2024-03-04T06:17:04.509 INFO:tasks.rados.rados.0.smithi064.stdout:2265: done (2 left) 2024-03-04T06:17:04.509 INFO:tasks.rados.rados.0.smithi064.stdout:2266: done (1 left) 2024-03-04T06:17:04.509 INFO:tasks.rados.rados.0.smithi064.stdout:2271: write oid 10 current snap is 272 2024-03-04T06:17:04.509 INFO:tasks.rados.rados.0.smithi064.stdout:2271: seq_num 1065 ranges {699455=429605,1804617=576034,3163252=514929} 2024-03-04T06:17:04.515 INFO:tasks.rados.rados.0.smithi064.stdout:2271: writing smithi06431743-10 from 699455 to 1129060 tid 1 2024-03-04T06:17:04.524 INFO:tasks.rados.rados.0.smithi064.stdout:2271: writing smithi06431743-10 from 1804617 to 2380651 tid 2 2024-03-04T06:17:04.532 INFO:tasks.rados.rados.0.smithi064.stdout:2271: writing smithi06431743-10 from 3163252 to 3678181 tid 3 2024-03-04T06:17:04.532 INFO:tasks.rados.rados.0.smithi064.stdout:2272: delete oid 42 current snap is 272 2024-03-04T06:17:04.532 INFO:tasks.rados.rados.0.smithi064.stdout:2270: finishing write tid 1 to smithi06431743-26 2024-03-04T06:17:04.542 INFO:tasks.rados.rados.0.smithi064.stdout:2270: finishing write tid 2 to smithi06431743-26 2024-03-04T06:17:04.560 INFO:tasks.rados.rados.0.smithi064.stdout:2270: finishing write tid 3 to smithi06431743-26 2024-03-04T06:17:04.568 INFO:tasks.rados.rados.0.smithi064.stdout:2270: finishing write tid 4 to smithi06431743-26 2024-03-04T06:17:04.570 INFO:tasks.rados.rados.0.smithi064.stdout:2270: finishing write tid 5 to smithi06431743-26 2024-03-04T06:17:04.570 INFO:tasks.rados.rados.0.smithi064.stdout:2270: oid 26 updating version 0 to 882 2024-03-04T06:17:04.570 INFO:tasks.rados.rados.0.smithi064.stdout:2270: oid 26 updating version 882 to 884 2024-03-04T06:17:04.570 INFO:tasks.rados.rados.0.smithi064.stdout:2270: oid 26 version 884 is already newer than 883 2024-03-04T06:17:04.570 INFO:tasks.rados.rados.0.smithi064.stdout:2270: oid 26 version 884 is already newer than 881 2024-03-04T06:17:04.570 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 884 (ObjNum 1064 snap 272 seq_num 1064) dirty exists 2024-03-04T06:17:04.570 INFO:tasks.rados.rados.0.smithi064.stdout:2270: left oid 26 (ObjNum 1064 snap 272 seq_num 1064) 2024-03-04T06:17:04.584 INFO:tasks.rados.rados.0.smithi064.stdout:2270: done (2 left) 2024-03-04T06:17:04.584 INFO:tasks.rados.rados.0.smithi064.stdout:2272: done (1 left) 2024-03-04T06:17:04.584 INFO:tasks.rados.rados.0.smithi064.stdout:2273: delete oid 43 current snap is 272 2024-03-04T06:17:04.592 INFO:tasks.rados.rados.0.smithi064.stdout:2271: finishing write tid 1 to smithi06431743-10 2024-03-04T06:17:04.608 INFO:tasks.rados.rados.0.smithi064.stdout:2271: finishing write tid 2 to smithi06431743-10 2024-03-04T06:17:04.610 INFO:tasks.rados.rados.0.smithi064.stdout:2273: done (1 left) 2024-03-04T06:17:04.610 INFO:tasks.rados.rados.0.smithi064.stdout:2274: delete oid 28 current snap is 272 2024-03-04T06:17:04.612 INFO:tasks.rados.rados.0.smithi064.stdout:2271: finishing write tid 3 to smithi06431743-10 2024-03-04T06:17:04.622 INFO:tasks.rados.rados.0.smithi064.stdout:2271: finishing write tid 4 to smithi06431743-10 2024-03-04T06:17:04.623 INFO:tasks.rados.rados.0.smithi064.stdout:2271: finishing write tid 5 to smithi06431743-10 2024-03-04T06:17:04.623 INFO:tasks.rados.rados.0.smithi064.stdout:2271: oid 10 updating version 0 to 542 2024-03-04T06:17:04.623 INFO:tasks.rados.rados.0.smithi064.stdout:2271: oid 10 updating version 542 to 543 2024-03-04T06:17:04.623 INFO:tasks.rados.rados.0.smithi064.stdout:2271: oid 10 version 543 is already newer than 540 2024-03-04T06:17:04.623 INFO:tasks.rados.rados.0.smithi064.stdout:2271: oid 10 version 543 is already newer than 542 2024-03-04T06:17:04.623 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 543 (ObjNum 1065 snap 272 seq_num 1065) dirty exists 2024-03-04T06:17:04.623 INFO:tasks.rados.rados.0.smithi064.stdout:2271: left oid 10 (ObjNum 1065 snap 272 seq_num 1065) 2024-03-04T06:17:04.630 INFO:tasks.rados.rados.0.smithi064.stdout:2271: done (1 left) 2024-03-04T06:17:04.630 INFO:tasks.rados.rados.0.smithi064.stdout:2274: done (0 left) 2024-03-04T06:17:04.630 INFO:tasks.rados.rados.0.smithi064.stdout:2275: write (excl) oid 30 current snap is 272 2024-03-04T06:17:04.630 INFO:tasks.rados.rados.0.smithi064.stdout:2275: seq_num 1069 ranges {766683=730601,1922192=583372,3156591=676370} 2024-03-04T06:17:04.638 INFO:tasks.rados.rados.0.smithi064.stdout:2275: writing smithi06431743-30 from 766683 to 1497284 tid 1 2024-03-04T06:17:04.643 INFO:tasks.rados.rados.0.smithi064.stdout:2275: writing smithi06431743-30 from 1922192 to 2505564 tid 2 2024-03-04T06:17:04.651 INFO:tasks.rados.rados.0.smithi064.stdout:2275: writing smithi06431743-30 from 3156591 to 3832961 tid 3 2024-03-04T06:17:04.651 INFO:tasks.rados.rados.0.smithi064.stdout:2276: read oid 39 snap -1 2024-03-04T06:17:04.651 INFO:tasks.rados.rados.0.smithi064.stdout:2276: expect (ObjNum 1030 snap 264 seq_num 1030) 2024-03-04T06:17:04.652 INFO:tasks.rados.rados.0.smithi064.stdout:2277: read oid 25 snap -1 2024-03-04T06:17:04.652 INFO:tasks.rados.rados.0.smithi064.stdout:2277: expect (ObjNum 943 snap 232 seq_num 943) 2024-03-04T06:17:04.652 INFO:tasks.rados.rados.0.smithi064.stdout:2278: write (excl) oid 50 current snap is 272 2024-03-04T06:17:04.652 INFO:tasks.rados.rados.0.smithi064.stdout:2278: seq_num 1070 ranges {783490=405850,1980674=727030,3184014=487642} 2024-03-04T06:17:04.658 INFO:tasks.rados.rados.0.smithi064.stdout:2278: writing smithi06431743-50 from 783490 to 1189340 tid 1 2024-03-04T06:17:04.669 INFO:tasks.rados.rados.0.smithi064.stdout:2278: writing smithi06431743-50 from 1980674 to 2707704 tid 2 2024-03-04T06:17:04.676 INFO:tasks.rados.rados.0.smithi064.stdout:2278: writing smithi06431743-50 from 3184014 to 3671656 tid 3 2024-03-04T06:17:04.676 INFO:tasks.rados.rados.0.smithi064.stdout:2279: snap_remove snap 247 2024-03-04T06:17:05.376 INFO:tasks.rados.rados.0.smithi064.stdout:2279: done (4 left) 2024-03-04T06:17:05.376 INFO:tasks.rados.rados.0.smithi064.stdout:2280: delete oid 34 current snap is 272 2024-03-04T06:17:05.376 INFO:tasks.rados.rados.0.smithi064.stdout:2275: finishing write tid 1 to smithi06431743-30 2024-03-04T06:17:05.376 INFO:tasks.rados.rados.0.smithi064.stdout:2276: expect (ObjNum 1030 snap 264 seq_num 1030) 2024-03-04T06:17:05.605 INFO:tasks.rados.rados.0.smithi064.stdout:2278: finishing write tid 1 to smithi06431743-50 2024-03-04T06:17:05.605 INFO:tasks.rados.rados.0.smithi064.stdout:2275: finishing write tid 2 to smithi06431743-30 2024-03-04T06:17:05.605 INFO:tasks.rados.rados.0.smithi064.stdout:2277: expect (ObjNum 943 snap 232 seq_num 943) 2024-03-04T06:17:05.794 INFO:tasks.rados.rados.0.smithi064.stdout:2275: finishing write tid 3 to smithi06431743-30 2024-03-04T06:17:05.794 INFO:tasks.rados.rados.0.smithi064.stdout:2278: finishing write tid 2 to smithi06431743-50 2024-03-04T06:17:05.794 INFO:tasks.rados.rados.0.smithi064.stdout:2275: finishing write tid 4 to smithi06431743-30 2024-03-04T06:17:05.794 INFO:tasks.rados.rados.0.smithi064.stdout:2275: finishing write tid 5 to smithi06431743-30 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2275: oid 30 updating version 0 to 990 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2275: oid 30 version 990 is already newer than 989 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2275: oid 30 version 990 is already newer than 987 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2275: oid 30 version 990 is already newer than 988 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 990 (ObjNum 1069 snap 272 seq_num 1069) dirty exists 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2275: left oid 30 (ObjNum 1069 snap 272 seq_num 1069) 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2278: finishing write tid 3 to smithi06431743-50 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2275: done (4 left) 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2276: done (3 left) 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2277: done (2 left) 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2280: done (1 left) 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2281: write oid 14 current snap is 272 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2278: finishing write tid 4 to smithi06431743-50 2024-03-04T06:17:05.795 INFO:tasks.rados.rados.0.smithi064.stdout:2281: seq_num 1072 ranges {417104=672701,1813991=203113} 2024-03-04T06:17:05.800 INFO:tasks.rados.rados.0.smithi064.stdout:2281: writing smithi06431743-14 from 417104 to 1089805 tid 1 2024-03-04T06:17:05.805 INFO:tasks.rados.rados.0.smithi064.stdout:2281: writing smithi06431743-14 from 1813991 to 2017104 tid 2 2024-03-04T06:17:05.805 INFO:tasks.rados.rados.0.smithi064.stdout:2282: snap_remove snap 269 2024-03-04T06:17:05.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:06.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:06.010+0000 7f42fd2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:06.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:06.015+0000 7f42fd2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:06.375 INFO:tasks.rados.rados.0.smithi064.stdout:2282: done (2 left) 2024-03-04T06:17:06.376 INFO:tasks.rados.rados.0.smithi064.stdout:2283: write (excl) oid 38 current snap is 272 2024-03-04T06:17:06.376 INFO:tasks.rados.rados.0.smithi064.stdout:2278: finishing write tid 5 to smithi06431743-50 2024-03-04T06:17:06.376 INFO:tasks.rados.rados.0.smithi064.stdout:2278: oid 50 updating version 0 to 776 2024-03-04T06:17:06.376 INFO:tasks.rados.rados.0.smithi064.stdout:2278: oid 50 updating version 776 to 778 2024-03-04T06:17:06.376 INFO:tasks.rados.rados.0.smithi064.stdout:2278: oid 50 version 778 is already newer than 775 2024-03-04T06:17:06.376 INFO:tasks.rados.rados.0.smithi064.stdout:2278: oid 50 version 778 is already newer than 778 2024-03-04T06:17:06.376 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 778 (ObjNum 1070 snap 272 seq_num 1070) dirty exists 2024-03-04T06:17:06.376 INFO:tasks.rados.rados.0.smithi064.stdout:2278: left oid 50 (ObjNum 1070 snap 272 seq_num 1070) 2024-03-04T06:17:06.376 INFO:tasks.rados.rados.0.smithi064.stdout:2283: seq_num 1073 ranges {433911=747950,1756120=277791} 2024-03-04T06:17:06.381 INFO:tasks.rados.rados.0.smithi064.stdout:2283: writing smithi06431743-38 from 433911 to 1181861 tid 1 2024-03-04T06:17:06.386 INFO:tasks.rados.rados.0.smithi064.stdout:2283: writing smithi06431743-38 from 1756120 to 2033911 tid 2 2024-03-04T06:17:06.386 INFO:tasks.rados.rados.0.smithi064.stdout:2278: done (2 left) 2024-03-04T06:17:06.387 INFO:tasks.rados.rados.0.smithi064.stdout:2284: write oid 26 current snap is 272 2024-03-04T06:17:06.387 INFO:tasks.rados.rados.0.smithi064.stdout:2281: finishing write tid 1 to smithi06431743-14 2024-03-04T06:17:06.387 INFO:tasks.rados.rados.0.smithi064.stdout:2281: finishing write tid 2 to smithi06431743-14 2024-03-04T06:17:06.387 INFO:tasks.rados.rados.0.smithi064.stdout:2281: finishing write tid 3 to smithi06431743-14 2024-03-04T06:17:06.387 INFO:tasks.rados.rados.0.smithi064.stdout:2284: seq_num 1074 ranges {450718=423199,1414602=636116} 2024-03-04T06:17:06.391 INFO:tasks.rados.rados.0.smithi064.stdout:2284: writing smithi06431743-26 from 450718 to 873917 tid 1 2024-03-04T06:17:06.398 INFO:tasks.rados.rados.0.smithi064.stdout:2284: writing smithi06431743-26 from 1414602 to 2050718 tid 2 2024-03-04T06:17:06.398 INFO:tasks.rados.rados.0.smithi064.stdout:2285: snap_remove snap 259 2024-03-04T06:17:06.398 INFO:tasks.rados.rados.0.smithi064.stdout:2281: finishing write tid 4 to smithi06431743-14 2024-03-04T06:17:06.398 INFO:tasks.rados.rados.0.smithi064.stdout:2281: oid 14 updating version 0 to 661 2024-03-04T06:17:06.398 INFO:tasks.rados.rados.0.smithi064.stdout:2281: oid 14 updating version 661 to 662 2024-03-04T06:17:06.398 INFO:tasks.rados.rados.0.smithi064.stdout:2281: oid 14 version 662 is already newer than 660 2024-03-04T06:17:06.398 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 662 (ObjNum 1072 snap 272 seq_num 1072) dirty exists 2024-03-04T06:17:06.398 INFO:tasks.rados.rados.0.smithi064.stdout:2281: left oid 14 (ObjNum 1072 snap 272 seq_num 1072) 2024-03-04T06:17:07.385 INFO:tasks.rados.rados.0.smithi064.stdout:2281: done (3 left) 2024-03-04T06:17:07.385 INFO:tasks.rados.rados.0.smithi064.stdout:2285: done (2 left) 2024-03-04T06:17:07.385 INFO:tasks.rados.rados.0.smithi064.stdout:2286: read oid 15 snap -1 2024-03-04T06:17:07.385 INFO:tasks.rados.rados.0.smithi064.stdout:2286: expect (ObjNum 1049 snap 266 seq_num 1049) 2024-03-04T06:17:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:2287: read oid 4 snap -1 2024-03-04T06:17:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:2287: expect (ObjNum 1047 snap 265 seq_num 1047) 2024-03-04T06:17:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:2288: read oid 30 snap 271 2024-03-04T06:17:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:2288: expect deleted 2024-03-04T06:17:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:2289: read oid 24 snap -1 2024-03-04T06:17:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:2289: expect deleted 2024-03-04T06:17:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:2290: read oid 9 snap -1 2024-03-04T06:17:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:2290: expect deleted 2024-03-04T06:17:07.386 INFO:tasks.rados.rados.0.smithi064.stdout:2283: finishing write tid 1 to smithi06431743-38 2024-03-04T06:17:07.388 INFO:tasks.rados.rados.0.smithi064.stdout:2291: read oid 21 snap -1 2024-03-04T06:17:07.388 INFO:tasks.rados.rados.0.smithi064.stdout:2291: expect (ObjNum 1045 snap 265 seq_num 1045) 2024-03-04T06:17:07.388 INFO:tasks.rados.rados.0.smithi064.stdout:2292: delete oid 22 current snap is 272 2024-03-04T06:17:07.388 INFO:tasks.rados.rados.0.smithi064.stdout:2283: finishing write tid 2 to smithi06431743-38 2024-03-04T06:17:07.388 INFO:tasks.rados.rados.0.smithi064.stdout:2283: finishing write tid 3 to smithi06431743-38 2024-03-04T06:17:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:2284: finishing write tid 1 to smithi06431743-26 2024-03-04T06:17:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:2283: finishing write tid 4 to smithi06431743-38 2024-03-04T06:17:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:2283: oid 38 updating version 0 to 550 2024-03-04T06:17:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:2283: oid 38 version 550 is already newer than 550 2024-03-04T06:17:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:2283: oid 38 version 550 is already newer than 548 2024-03-04T06:17:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 550 (ObjNum 1073 snap 272 seq_num 1073) dirty exists 2024-03-04T06:17:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:2283: left oid 38 (ObjNum 1073 snap 272 seq_num 1073) 2024-03-04T06:17:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:2284: finishing write tid 2 to smithi06431743-26 2024-03-04T06:17:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:2284: finishing write tid 3 to smithi06431743-26 2024-03-04T06:17:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:2284: finishing write tid 4 to smithi06431743-26 2024-03-04T06:17:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:2284: oid 26 updating version 0 to 902 2024-03-04T06:17:07.390 INFO:tasks.rados.rados.0.smithi064.stdout:2284: oid 26 version 902 is already newer than 902 2024-03-04T06:17:07.390 INFO:tasks.rados.rados.0.smithi064.stdout:2284: oid 26 version 902 is already newer than 900 2024-03-04T06:17:07.390 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 902 (ObjNum 1074 snap 272 seq_num 1074) dirty exists 2024-03-04T06:17:07.390 INFO:tasks.rados.rados.0.smithi064.stdout:2284: left oid 26 (ObjNum 1074 snap 272 seq_num 1074) 2024-03-04T06:17:07.471 INFO:tasks.rados.rados.0.smithi064.stdout:2286: expect (ObjNum 1049 snap 266 seq_num 1049) 2024-03-04T06:17:07.695 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:17:07.695 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:17:07.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:17:07.700 INFO:tasks.rados.rados.0.smithi064.stdout:2283: done (8 left) 2024-03-04T06:17:07.700 INFO:tasks.rados.rados.0.smithi064.stdout:2284: done (7 left) 2024-03-04T06:17:07.700 INFO:tasks.rados.rados.0.smithi064.stdout:2286: done (6 left) 2024-03-04T06:17:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:2289: done (5 left) 2024-03-04T06:17:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:2292: done (4 left) 2024-03-04T06:17:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:2293: read oid 50 snap -1 2024-03-04T06:17:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:2293: expect (ObjNum 1070 snap 272 seq_num 1070) 2024-03-04T06:17:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:2294: snap_create 2024-03-04T06:17:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:2291: expect (ObjNum 1045 snap 265 seq_num 1045) 2024-03-04T06:17:07.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:07.799+0000 7fb3fb66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:07.806 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:07.805+0000 7fb3fb66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:07.919 INFO:tasks.rados.rados.0.smithi064.stdout:2287: expect (ObjNum 1047 snap 265 seq_num 1047) 2024-03-04T06:17:08.002 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:17:08.002 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:17:08.014 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":829,"stamp":"2024-03-04T06:17:07.382392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103321583,"num_objects":798,"num_object_clones":616,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":150,"num_read":4263,"num_read_kb":3216951,"num_write":3711,"num_write_kb":976150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7907,"ondisk_log_size":7907,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":553072,"kb_used_data":393040,"kb_used_omap":0,"kb_used_meta":160000,"kb_avail":376934288,"statfs":{"total":386547056640,"available":385980710912,"internally_reserved":0,"allocated":402472960,"data_stored":459155649,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":163840000},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3748621,"num_objects":10,"num_object_clones":10,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":53,"num_read_kb":49224,"num_write":46,"num_write_kb":12559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.120268"},"pg_stats":[{"pgid":"3.f","version":"554'472","reported_seq":1948,"reported_epoch":554,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:04.566690+0000","last_change":"2024-03-04T06:17:04.542574+0000","last_active":"2024-03-04T06:17:04.566690+0000","last_peered":"2024-03-04T06:17:04.566690+0000","last_clean":"2024-03-04T06:17:04.566690+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:17:04.566690+0000","last_undegraded":"2024-03-04T06:17:04.566690+0000","last_fullsized":"2024-03-04T06:17:04.566690+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2874228,"num_objects":53,"num_object_clones":42,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":269,"num_read_kb":224364,"num_write":217,"num_write_kb":58378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"554'379","reported_seq":1879,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-04T06:17:04.654823+0000","last_change":"2024-03-04T06:17:04.736292+0000","last_active":"2024-03-04T06:17:04.654823+0000","last_peered":"2024-03-04T06:17:04.654823+0000","last_clean":"2024-03-04T06:17:04.654823+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:17:04.654823+0000","last_undegraded":"2024-03-04T06:17:04.654823+0000","last_fullsized":"2024-03-04T06:17:04.654823+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7479577,"num_objects":44,"num_object_clones":30,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":137,"num_read_kb":80772,"num_write":176,"num_write_kb":46528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"554'510","reported_seq":2018,"reported_epoch":555,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:05.914446+0000","last_change":"2024-03-04T06:17:04.680416+0000","last_active":"2024-03-04T06:17:05.914446+0000","last_peered":"2024-03-04T06:17:05.914446+0000","last_clean":"2024-03-04T06:17:05.914446+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:17:05.914446+0000","last_undegraded":"2024-03-04T06:17:05.914446+0000","last_fullsized":"2024-03-04T06:17:05.914446+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7160211,"num_objects":59,"num_object_clones":46,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":341,"num_read_kb":275867,"num_write":252,"num_write_kb":69591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"552'349","reported_seq":1879,"reported_epoch":554,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:04.534583+0000","last_change":"2024-03-04T06:17:04.534583+0000","last_active":"2024-03-04T06:17:04.534583+0000","last_peered":"2024-03-04T06:17:04.534583+0000","last_clean":"2024-03-04T06:17:04.534583+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:17:04.534583+0000","last_undegraded":"2024-03-04T06:17:04.534583+0000","last_fullsized":"2024-03-04T06:17:04.534583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7008981,"num_objects":36,"num_object_clones":26,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":195,"num_read_kb":149128,"num_write":174,"num_write_kb":48785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"462'167","reported_seq":1728,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-04T06:17:05.403368+0000","last_change":"2024-03-04T06:17:05.407188+0000","last_active":"2024-03-04T06:17:05.403368+0000","last_peered":"2024-03-04T06:17:05.403368+0000","last_clean":"2024-03-04T06:17:05.403368+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:17:05.403368+0000","last_undegraded":"2024-03-04T06:17:05.403368+0000","last_fullsized":"2024-03-04T06:17:05.403368+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327984,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":58,"num_read_kb":33096,"num_write":75,"num_write_kb":19962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"554'543","reported_seq":1969,"reported_epoch":554,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:04.613120+0000","last_change":"2024-03-04T06:17:04.539883+0000","last_active":"2024-03-04T06:17:04.613120+0000","last_peered":"2024-03-04T06:17:04.613120+0000","last_clean":"2024-03-04T06:17:04.613120+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:17:04.613120+0000","last_undegraded":"2024-03-04T06:17:04.613120+0000","last_fullsized":"2024-03-04T06:17:04.613120+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6588806,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":253,"num_read_kb":192236,"num_write":251,"num_write_kb":73441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1635,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-04T06:17:04.555055+0000","last_change":"2024-03-04T06:17:04.564938+0000","last_active":"2024-03-04T06:17:04.555055+0000","last_peered":"2024-03-04T06:17:04.555055+0000","last_clean":"2024-03-04T06:17:04.555055+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:17:04.555055+0000","last_undegraded":"2024-03-04T06:17:04.555055+0000","last_fullsized":"2024-03-04T06:17:04.555055+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"554'266","reported_seq":1830,"reported_epoch":555,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:05.917557+0000","last_change":"2024-03-04T06:17:04.681303+0000","last_active":"2024-03-04T06:17:05.917557+0000","last_peered":"2024-03-04T06:17:05.917557+0000","last_clean":"2024-03-04T06:17:05.917557+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:17:05.917557+0000","last_undegraded":"2024-03-04T06:17:05.917557+0000","last_fullsized":"2024-03-04T06:17:05.917557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6437477,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":176,"num_read_kb":163221,"num_write":123,"num_write_kb":32558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1514,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-04T06:17:05.385173+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:17:05.385173+0000","last_peered":"2024-03-04T06:17:05.385173+0000","last_clean":"2024-03-04T06:17:05.385173+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:17:05.385173+0000","last_undegraded":"2024-03-04T06:17:05.385173+0000","last_fullsized":"2024-03-04T06:17:05.385173+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1387,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-04T06:17:05.928375+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:17:05.928375+0000","last_peered":"2024-03-04T06:17:05.928375+0000","last_clean":"2024-03-04T06:17:05.928375+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:17:05.928375+0000","last_undegraded":"2024-03-04T06:17:05.928375+0000","last_fullsized":"2024-03-04T06:17:05.928375+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"555'525","reported_seq":1969,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-04T06:17:05.925420+0000","last_change":"2024-03-04T06:17:06.010889+0000","last_active":"2024-03-04T06:17:05.925420+0000","last_peered":"2024-03-04T06:17:05.925420+0000","last_clean":"2024-03-04T06:17:05.925420+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:17:05.925420+0000","last_undegraded":"2024-03-04T06:17:05.925420+0000","last_fullsized":"2024-03-04T06:17:05.925420+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":52,"num_object_clones":40,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":264,"num_read_kb":210737,"num_write":234,"num_write_kb":63712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1380,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-04T06:17:04.542703+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:17:04.542703+0000","last_peered":"2024-03-04T06:17:04.542703+0000","last_clean":"2024-03-04T06:17:04.542703+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:17:04.542703+0000","last_undegraded":"2024-03-04T06:17:04.542703+0000","last_fullsized":"2024-03-04T06:17:04.542703+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"531'366","reported_seq":1861,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-04T06:17:05.392307+0000","last_change":"2024-03-04T06:17:05.398791+0000","last_active":"2024-03-04T06:17:05.392307+0000","last_peered":"2024-03-04T06:17:05.392307+0000","last_clean":"2024-03-04T06:17:05.392307+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:17:05.392307+0000","last_undegraded":"2024-03-04T06:17:05.392307+0000","last_fullsized":"2024-03-04T06:17:05.392307+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3143140,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":216,"num_read_kb":177500,"num_write":141,"num_write_kb":40250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1387,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-04T06:17:05.912526+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:17:05.912526+0000","last_peered":"2024-03-04T06:17:05.912526+0000","last_clean":"2024-03-04T06:17:05.912526+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:17:05.912526+0000","last_undegraded":"2024-03-04T06:17:05.912526+0000","last_fullsized":"2024-03-04T06:17:05.912526+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"555'895","reported_seq":2264,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-04T06:17:05.426021+0000","last_change":"2024-03-04T06:17:05.468735+0000","last_active":"2024-03-04T06:17:05.426021+0000","last_peered":"2024-03-04T06:17:05.426021+0000","last_clean":"2024-03-04T06:17:05.426021+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:17:05.426021+0000","last_undegraded":"2024-03-04T06:17:05.426021+0000","last_fullsized":"2024-03-04T06:17:05.426021+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462770,"num_objects":92,"num_object_clones":71,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":538,"num_read_kb":427273,"num_write":429,"num_write_kb":115062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1380,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-04T06:17:04.531652+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:17:04.531652+0000","last_peered":"2024-03-04T06:17:04.531652+0000","last_clean":"2024-03-04T06:17:04.531652+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:17:04.531652+0000","last_undegraded":"2024-03-04T06:17:04.531652+0000","last_fullsized":"2024-03-04T06:17:04.531652+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"554'992","reported_seq":2247,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-04T06:17:04.783323+0000","last_change":"2024-03-04T06:17:04.804818+0000","last_active":"2024-03-04T06:17:04.783323+0000","last_peered":"2024-03-04T06:17:04.783323+0000","last_clean":"2024-03-04T06:17:04.783323+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:17:04.783323+0000","last_undegraded":"2024-03-04T06:17:04.783323+0000","last_fullsized":"2024-03-04T06:17:04.783323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":992,"log_dups_size":0,"ondisk_log_size":992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17782961,"num_objects":96,"num_object_clones":74,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":427,"num_read_kb":304917,"num_write":444,"num_write_kb":117475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"554'459","reported_seq":1973,"reported_epoch":555,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:05.926315+0000","last_change":"2024-03-04T06:17:04.696826+0000","last_active":"2024-03-04T06:17:05.926315+0000","last_peered":"2024-03-04T06:17:05.926315+0000","last_clean":"2024-03-04T06:17:05.926315+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:17:05.926315+0000","last_undegraded":"2024-03-04T06:17:05.926315+0000","last_fullsized":"2024-03-04T06:17:05.926315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7193858,"num_objects":47,"num_object_clones":35,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":299,"num_read_kb":229917,"num_write":219,"num_write_kb":59708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1410,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-04T06:17:04.657660+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:17:04.657660+0000","last_peered":"2024-03-04T06:17:04.657660+0000","last_clean":"2024-03-04T06:17:04.657660+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:17:04.657660+0000","last_undegraded":"2024-03-04T06:17:04.657660+0000","last_fullsized":"2024-03-04T06:17:04.657660+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"554'445","reported_seq":1949,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-04T06:17:04.661770+0000","last_change":"2024-03-04T06:17:04.707774+0000","last_active":"2024-03-04T06:17:04.661770+0000","last_peered":"2024-03-04T06:17:04.661770+0000","last_clean":"2024-03-04T06:17:04.661770+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:17:04.661770+0000","last_undegraded":"2024-03-04T06:17:04.661770+0000","last_fullsized":"2024-03-04T06:17:04.661770+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143437,"num_objects":52,"num_object_clones":37,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":223,"num_read_kb":149344,"num_write":210,"num_write_kb":55510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1381,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-04T06:17:05.401989+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:17:05.401989+0000","last_peered":"2024-03-04T06:17:05.401989+0000","last_clean":"2024-03-04T06:17:05.401989+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:17:05.401989+0000","last_undegraded":"2024-03-04T06:17:05.401989+0000","last_fullsized":"2024-03-04T06:17:05.401989+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"555'662","reported_seq":2049,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-04T06:17:05.826737+0000","last_change":"2024-03-04T06:17:05.482038+0000","last_active":"2024-03-04T06:17:05.826737+0000","last_peered":"2024-03-04T06:17:05.826737+0000","last_clean":"2024-03-04T06:17:05.826737+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:17:05.826737+0000","last_undegraded":"2024-03-04T06:17:05.826737+0000","last_fullsized":"2024-03-04T06:17:05.826737+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5731682,"num_objects":65,"num_object_clones":52,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":333,"num_read_kb":253704,"num_write":297,"num_write_kb":82056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1410,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-04T06:17:04.655771+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:17:04.655771+0000","last_peered":"2024-03-04T06:17:04.655771+0000","last_clean":"2024-03-04T06:17:04.655771+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:17:04.655771+0000","last_undegraded":"2024-03-04T06:17:04.655771+0000","last_fullsized":"2024-03-04T06:17:04.655771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"554'780","reported_seq":2157,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-04T06:17:04.796181+0000","last_change":"2024-03-04T06:17:04.810680+0000","last_active":"2024-03-04T06:17:04.796181+0000","last_peered":"2024-03-04T06:17:04.796181+0000","last_clean":"2024-03-04T06:17:04.796181+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:17:04.796181+0000","last_undegraded":"2024-03-04T06:17:04.796181+0000","last_fullsized":"2024-03-04T06:17:04.796181+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":780,"log_dups_size":0,"ondisk_log_size":780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7496384,"num_objects":72,"num_object_clones":63,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":431,"num_read_kb":344784,"num_write":332,"num_write_kb":91240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1387,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-04T06:17:05.924451+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:17:05.924451+0000","last_peered":"2024-03-04T06:17:05.924451+0000","last_clean":"2024-03-04T06:17:05.924451+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:17:05.924451+0000","last_undegraded":"2024-03-04T06:17:05.924451+0000","last_fullsized":"2024-03-04T06:17:05.924451+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102731215,"num_objects":796,"num_object_clones":616,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":150,"num_read":4160,"num_read_kb":3216860,"num_write":3574,"num_write_kb":974256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7810,"ondisk_log_size":7810,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":169504,"kb_used_data":120472,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94202336,"statfs":{"total":96636764160,"available":96463192064,"internally_reserved":0,"allocated":123363328,"data_stored":140174733,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":144456,"kb_used_data":99584,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94227384,"statfs":{"total":96636764160,"available":96488841216,"internally_reserved":0,"allocated":101974016,"data_stored":116208192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":118480,"kb_used_data":91656,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94253360,"statfs":{"total":96636764160,"available":96515440640,"internally_reserved":0,"allocated":93855744,"data_stored":108265794,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":120632,"kb_used_data":81328,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94251208,"statfs":{"total":96636764160,"available":96513236992,"internally_reserved":0,"allocated":83279872,"data_stored":94506930,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:17:08.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:08.096 INFO:tasks.rados.rados.0.smithi064.stdout:2293: expect (ObjNum 1070 snap 272 seq_num 1070) 2024-03-04T06:17:08.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:08.185+0000 7ff618d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:08.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:08.190+0000 7ff618d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:08.388 INFO:tasks.rados.rados.0.smithi064.stdout:2287: done (5 left) 2024-03-04T06:17:08.388 INFO:tasks.rados.rados.0.smithi064.stdout:2288: done (4 left) 2024-03-04T06:17:08.388 INFO:tasks.rados.rados.0.smithi064.stdout:2290: done (3 left) 2024-03-04T06:17:08.389 INFO:tasks.rados.rados.0.smithi064.stdout:2291: done (2 left) 2024-03-04T06:17:08.389 INFO:tasks.rados.rados.0.smithi064.stdout:2293: done (1 left) 2024-03-04T06:17:08.389 INFO:tasks.rados.rados.0.smithi064.stdout:2294: done (0 left) 2024-03-04T06:17:08.389 INFO:tasks.rados.rados.0.smithi064.stdout:2295: write (excl) oid 31 current snap is 273 2024-03-04T06:17:08.389 INFO:tasks.rados.rados.0.smithi064.stdout:2295: seq_num 1076 ranges {484332=573697,1531566=552766} 2024-03-04T06:17:08.393 INFO:tasks.rados.rados.0.smithi064.stdout:2295: writing smithi06431743-31 from 484332 to 1058029 tid 1 2024-03-04T06:17:08.399 INFO:tasks.rados.rados.0.smithi064.stdout:2295: writing smithi06431743-31 from 1531566 to 2084332 tid 2 2024-03-04T06:17:08.399 INFO:tasks.rados.rados.0.smithi064.stdout:2296: delete oid 14 current snap is 273 2024-03-04T06:17:08.441 INFO:tasks.rados.rados.0.smithi064.stdout:2296: done (1 left) 2024-03-04T06:17:08.441 INFO:tasks.rados.rados.0.smithi064.stdout:2297: snap_create 2024-03-04T06:17:08.449 INFO:tasks.rados.rados.0.smithi064.stdout:2295: finishing write tid 1 to smithi06431743-31 2024-03-04T06:17:08.462 INFO:tasks.rados.rados.0.smithi064.stdout:2295: finishing write tid 2 to smithi06431743-31 2024-03-04T06:17:08.468 INFO:tasks.rados.rados.0.smithi064.stdout:2295: finishing write tid 3 to smithi06431743-31 2024-03-04T06:17:08.469 INFO:tasks.rados.rados.0.smithi064.stdout:2295: finishing write tid 4 to smithi06431743-31 2024-03-04T06:17:08.469 INFO:tasks.rados.rados.0.smithi064.stdout:2295: oid 31 updating version 0 to 171 2024-03-04T06:17:08.469 INFO:tasks.rados.rados.0.smithi064.stdout:2295: oid 31 version 171 is already newer than 170 2024-03-04T06:17:08.469 INFO:tasks.rados.rados.0.smithi064.stdout:2295: oid 31 version 171 is already newer than 169 2024-03-04T06:17:08.470 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 171 (ObjNum 1076 snap 273 seq_num 1076) dirty exists 2024-03-04T06:17:08.470 INFO:tasks.rados.rados.0.smithi064.stdout:2295: left oid 31 (ObjNum 1076 snap 273 seq_num 1076) 2024-03-04T06:17:09.388 INFO:tasks.rados.rados.0.smithi064.stdout:2295: done (1 left) 2024-03-04T06:17:09.388 INFO:tasks.rados.rados.0.smithi064.stdout:2297: done (0 left) 2024-03-04T06:17:09.388 INFO:tasks.rados.rados.0.smithi064.stdout:2298: write (excl) oid 6 current snap is 274 2024-03-04T06:17:09.388 INFO:tasks.rados.rados.0.smithi064.stdout:2298: seq_num 1078 ranges {517946=724195,1932177=185769} 2024-03-04T06:17:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:2298: writing smithi06431743-6 from 517946 to 1242141 tid 1 2024-03-04T06:17:09.399 INFO:tasks.rados.rados.0.smithi064.stdout:2298: writing smithi06431743-6 from 1932177 to 2117946 tid 2 2024-03-04T06:17:09.399 INFO:tasks.rados.rados.0.smithi064.stdout:2299: delete oid 7 current snap is 274 2024-03-04T06:17:09.449 INFO:tasks.rados.rados.0.smithi064.stdout:2299: done (1 left) 2024-03-04T06:17:09.450 INFO:tasks.rados.rados.0.smithi064.stdout:2300: read oid 4 snap -1 2024-03-04T06:17:09.450 INFO:tasks.rados.rados.0.smithi064.stdout:2300: expect (ObjNum 1047 snap 265 seq_num 1047) 2024-03-04T06:17:09.450 INFO:tasks.rados.rados.0.smithi064.stdout:2301: read oid 33 snap 272 2024-03-04T06:17:09.450 INFO:tasks.rados.rados.0.smithi064.stdout:2301: expect (ObjNum 1042 snap 265 seq_num 1042) 2024-03-04T06:17:09.450 INFO:tasks.rados.rados.0.smithi064.stdout:2302: read oid 26 snap -1 2024-03-04T06:17:09.450 INFO:tasks.rados.rados.0.smithi064.stdout:2302: expect (ObjNum 1074 snap 272 seq_num 1074) 2024-03-04T06:17:09.450 INFO:tasks.rados.rados.0.smithi064.stdout:2303: write (excl) oid 43 current snap is 274 2024-03-04T06:17:09.450 INFO:tasks.rados.rados.0.smithi064.stdout:2303: seq_num 1080 ranges {551560=591046,1765494=386066} 2024-03-04T06:17:09.455 INFO:tasks.rados.rados.0.smithi064.stdout:2303: writing smithi06431743-43 from 551560 to 1142606 tid 1 2024-03-04T06:17:09.461 INFO:tasks.rados.rados.0.smithi064.stdout:2303: writing smithi06431743-43 from 1765494 to 2151560 tid 2 2024-03-04T06:17:09.461 INFO:tasks.rados.rados.0.smithi064.stdout:2304: read oid 8 snap -1 2024-03-04T06:17:09.461 INFO:tasks.rados.rados.0.smithi064.stdout:2304: expect (ObjNum 1053 snap 269 seq_num 1053) 2024-03-04T06:17:09.462 INFO:tasks.rados.rados.0.smithi064.stdout:2305: write (excl) oid 28 current snap is 274 2024-03-04T06:17:09.462 INFO:tasks.rados.rados.0.smithi064.stdout:2305: seq_num 1081 ranges {568367=666295,1707623=460744} 2024-03-04T06:17:09.468 INFO:tasks.rados.rados.0.smithi064.stdout:2305: writing smithi06431743-28 from 568367 to 1234662 tid 1 2024-03-04T06:17:09.474 INFO:tasks.rados.rados.0.smithi064.stdout:2305: writing smithi06431743-28 from 1707623 to 2168367 tid 2 2024-03-04T06:17:09.474 INFO:tasks.rados.rados.0.smithi064.stdout:2306: snap_create 2024-03-04T06:17:09.474 INFO:tasks.rados.rados.0.smithi064.stdout:2298: finishing write tid 1 to smithi06431743-6 2024-03-04T06:17:09.474 INFO:tasks.rados.rados.0.smithi064.stdout:2302: expect (ObjNum 1074 snap 272 seq_num 1074) 2024-03-04T06:17:09.572 INFO:tasks.rados.rados.0.smithi064.stdout:2300: expect (ObjNum 1047 snap 265 seq_num 1047) 2024-03-04T06:17:09.748 INFO:tasks.rados.rados.0.smithi064.stdout:2298: finishing write tid 2 to smithi06431743-6 2024-03-04T06:17:09.748 INFO:tasks.rados.rados.0.smithi064.stdout:2305: finishing write tid 1 to smithi06431743-28 2024-03-04T06:17:09.748 INFO:tasks.rados.rados.0.smithi064.stdout:2301: expect (ObjNum 1042 snap 265 seq_num 1042) 2024-03-04T06:17:09.969 INFO:tasks.rados.rados.0.smithi064.stdout:2304: expect (ObjNum 1053 snap 269 seq_num 1053) 2024-03-04T06:17:10.108 INFO:tasks.rados.rados.0.smithi064.stdout:2303: finishing write tid 1 to smithi06431743-43 2024-03-04T06:17:10.108 INFO:tasks.rados.rados.0.smithi064.stdout:2298: finishing write tid 3 to smithi06431743-6 2024-03-04T06:17:10.108 INFO:tasks.rados.rados.0.smithi064.stdout:2298: finishing write tid 4 to smithi06431743-6 2024-03-04T06:17:10.109 INFO:tasks.rados.rados.0.smithi064.stdout:2298: oid 6 updating version 0 to 468 2024-03-04T06:17:10.109 INFO:tasks.rados.rados.0.smithi064.stdout:2298: oid 6 version 468 is already newer than 466 2024-03-04T06:17:10.109 INFO:tasks.rados.rados.0.smithi064.stdout:2298: oid 6 version 468 is already newer than 467 2024-03-04T06:17:10.109 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 468 (ObjNum 1078 snap 274 seq_num 1078) dirty exists 2024-03-04T06:17:10.109 INFO:tasks.rados.rados.0.smithi064.stdout:2298: left oid 6 (ObjNum 1078 snap 274 seq_num 1078) 2024-03-04T06:17:10.109 INFO:tasks.rados.rados.0.smithi064.stdout:2305: finishing write tid 2 to smithi06431743-28 2024-03-04T06:17:10.109 INFO:tasks.rados.rados.0.smithi064.stdout:2305: finishing write tid 3 to smithi06431743-28 2024-03-04T06:17:10.109 INFO:tasks.rados.rados.0.smithi064.stdout:2305: finishing write tid 4 to smithi06431743-28 2024-03-04T06:17:10.109 INFO:tasks.rados.rados.0.smithi064.stdout:2305: oid 28 updating version 0 to 913 2024-03-04T06:17:10.109 INFO:tasks.rados.rados.0.smithi064.stdout:2305: oid 28 updating version 913 to 915 2024-03-04T06:17:10.110 INFO:tasks.rados.rados.0.smithi064.stdout:2305: oid 28 version 915 is already newer than 915 2024-03-04T06:17:10.110 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 915 (ObjNum 1081 snap 274 seq_num 1081) dirty exists 2024-03-04T06:17:10.110 INFO:tasks.rados.rados.0.smithi064.stdout:2305: left oid 28 (ObjNum 1081 snap 274 seq_num 1081) 2024-03-04T06:17:10.110 INFO:tasks.rados.rados.0.smithi064.stdout:2303: finishing write tid 2 to smithi06431743-43 2024-03-04T06:17:10.110 INFO:tasks.rados.rados.0.smithi064.stdout:2303: finishing write tid 3 to smithi06431743-43 2024-03-04T06:17:10.110 INFO:tasks.rados.rados.0.smithi064.stdout:2303: finishing write tid 4 to smithi06431743-43 2024-03-04T06:17:10.110 INFO:tasks.rados.rados.0.smithi064.stdout:2303: oid 43 updating version 0 to 489 2024-03-04T06:17:10.110 INFO:tasks.rados.rados.0.smithi064.stdout:2303: oid 43 version 489 is already newer than 489 2024-03-04T06:17:10.110 INFO:tasks.rados.rados.0.smithi064.stdout:2303: oid 43 version 489 is already newer than 487 2024-03-04T06:17:10.110 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 489 (ObjNum 1080 snap 274 seq_num 1080) dirty exists 2024-03-04T06:17:10.110 INFO:tasks.rados.rados.0.smithi064.stdout:2303: left oid 43 (ObjNum 1080 snap 274 seq_num 1080) 2024-03-04T06:17:10.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:10.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:10.366+0000 7fe91246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:10.371 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:10.370+0000 7fe91246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:10.390 INFO:tasks.rados.rados.0.smithi064.stdout:2298: done (7 left) 2024-03-04T06:17:10.390 INFO:tasks.rados.rados.0.smithi064.stdout:2300: done (6 left) 2024-03-04T06:17:10.390 INFO:tasks.rados.rados.0.smithi064.stdout:2301: done (5 left) 2024-03-04T06:17:10.390 INFO:tasks.rados.rados.0.smithi064.stdout:2302: done (4 left) 2024-03-04T06:17:10.390 INFO:tasks.rados.rados.0.smithi064.stdout:2303: done (3 left) 2024-03-04T06:17:10.391 INFO:tasks.rados.rados.0.smithi064.stdout:2304: done (2 left) 2024-03-04T06:17:10.391 INFO:tasks.rados.rados.0.smithi064.stdout:2305: done (1 left) 2024-03-04T06:17:10.391 INFO:tasks.rados.rados.0.smithi064.stdout:2306: done (0 left) 2024-03-04T06:17:10.391 INFO:tasks.rados.rados.0.smithi064.stdout:2307: write oid 14 current snap is 275 2024-03-04T06:17:10.391 INFO:tasks.rados.rados.0.smithi064.stdout:2307: seq_num 1082 ranges {585174=741544,1766105=419069} 2024-03-04T06:17:10.399 INFO:tasks.rados.rados.0.smithi064.stdout:2307: writing smithi06431743-14 from 585174 to 1326718 tid 1 2024-03-04T06:17:10.404 INFO:tasks.rados.rados.0.smithi064.stdout:2307: writing smithi06431743-14 from 1766105 to 2185174 tid 2 2024-03-04T06:17:10.405 INFO:tasks.rados.rados.0.smithi064.stdout:2308: snap_create 2024-03-04T06:17:10.472 INFO:tasks.rados.rados.0.smithi064.stdout:2307: finishing write tid 1 to smithi06431743-14 2024-03-04T06:17:10.487 INFO:tasks.rados.rados.0.smithi064.stdout:2307: finishing write tid 2 to smithi06431743-14 2024-03-04T06:17:10.493 INFO:tasks.rados.rados.0.smithi064.stdout:2307: finishing write tid 3 to smithi06431743-14 2024-03-04T06:17:10.495 INFO:tasks.rados.rados.0.smithi064.stdout:2307: finishing write tid 4 to smithi06431743-14 2024-03-04T06:17:10.495 INFO:tasks.rados.rados.0.smithi064.stdout:2307: oid 14 updating version 0 to 674 2024-03-04T06:17:10.495 INFO:tasks.rados.rados.0.smithi064.stdout:2307: oid 14 updating version 674 to 675 2024-03-04T06:17:10.495 INFO:tasks.rados.rados.0.smithi064.stdout:2307: oid 14 version 675 is already newer than 673 2024-03-04T06:17:10.495 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 675 (ObjNum 1082 snap 275 seq_num 1082) dirty exists 2024-03-04T06:17:10.495 INFO:tasks.rados.rados.0.smithi064.stdout:2307: left oid 14 (ObjNum 1082 snap 275 seq_num 1082) 2024-03-04T06:17:11.394 INFO:tasks.rados.rados.0.smithi064.stdout:2307: done (1 left) 2024-03-04T06:17:11.394 INFO:tasks.rados.rados.0.smithi064.stdout:2308: done (0 left) 2024-03-04T06:17:11.394 INFO:tasks.rados.rados.0.smithi064.stdout:2309: snap_remove snap 272 2024-03-04T06:17:12.395 INFO:tasks.rados.rados.0.smithi064.stdout:2309: done (0 left) 2024-03-04T06:17:12.395 INFO:tasks.rados.rados.0.smithi064.stdout:2310: read oid 42 snap -1 2024-03-04T06:17:12.395 INFO:tasks.rados.rados.0.smithi064.stdout:2310: expect deleted 2024-03-04T06:17:12.396 INFO:tasks.rados.rados.0.smithi064.stdout:2311: delete oid 31 current snap is 276 2024-03-04T06:17:12.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:12.469 INFO:tasks.rados.rados.0.smithi064.stdout:2310: done (1 left) 2024-03-04T06:17:12.469 INFO:tasks.rados.rados.0.smithi064.stdout:2311: done (0 left) 2024-03-04T06:17:12.469 INFO:tasks.rados.rados.0.smithi064.stdout:2312: read oid 19 snap -1 2024-03-04T06:17:12.469 INFO:tasks.rados.rados.0.smithi064.stdout:2312: expect deleted 2024-03-04T06:17:12.469 INFO:tasks.rados.rados.0.smithi064.stdout:2313: snap_create 2024-03-04T06:17:12.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:12.566+0000 7fb71e11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:12.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:12.570+0000 7fb71e11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:13.016 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:17:13.016 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:17:13.017 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:17:13.017 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:17:13.126 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:13.121+0000 7f37660ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:13.129 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:13.128+0000 7f37660ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:13.141 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:13.142 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:13.144 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:13.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:17:13.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:13.248+0000 7f0b8c514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:13.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:13.255+0000 7f0b8c514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:13.265 INFO:tasks.rados.rados.0.smithi064.stdout:2312: done (1 left) 2024-03-04T06:17:13.265 INFO:tasks.rados.rados.0.smithi064.stdout:2313: done (0 left) 2024-03-04T06:17:13.265 INFO:tasks.rados.rados.0.smithi064.stdout:2314: read oid 33 snap 274 2024-03-04T06:17:13.265 INFO:tasks.rados.rados.0.smithi064.stdout:2314: expect (ObjNum 1042 snap 265 seq_num 1042) 2024-03-04T06:17:13.265 INFO:tasks.rados.rados.0.smithi064.stdout:2315: snap_remove snap 266 2024-03-04T06:17:13.268 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:13.268 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:13.268 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:13.271 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:13.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:17:13.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:13.374+0000 7ffbc8cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:13.382 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:13.382+0000 7ffbc8cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:13.395 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:13.395 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:13.395 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:13.395 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:13.395 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:13.395 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:13.396 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:13.396 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:13.396 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:13.396 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:13.396 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:13.396 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:13.398 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:13.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:17:13.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:13.501+0000 7f0f0be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:13.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:13.508+0000 7f0f0be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:13.522 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:13.522 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:13.522 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:13.522 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:13.522 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:13.522 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:13.522 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:13.522 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:13.522 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:13.523 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:13.523 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:13.523 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:13.525 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:14.265 INFO:tasks.rados.rados.0.smithi064.stdout:2315: done (1 left) 2024-03-04T06:17:14.266 INFO:tasks.rados.rados.0.smithi064.stdout:2316: read oid 21 snap -1 2024-03-04T06:17:14.266 INFO:tasks.rados.rados.0.smithi064.stdout:2316: expect (ObjNum 1045 snap 265 seq_num 1045) 2024-03-04T06:17:14.266 INFO:tasks.rados.rados.0.smithi064.stdout:2317: write (excl) oid 35 current snap is 277 2024-03-04T06:17:14.266 INFO:tasks.rados.rados.0.smithi064.stdout:2317: seq_num 1084 ranges {618788=492042,1883069=335719} 2024-03-04T06:17:14.271 INFO:tasks.rados.rados.0.smithi064.stdout:2317: writing smithi06431743-35 from 618788 to 1110830 tid 1 2024-03-04T06:17:14.277 INFO:tasks.rados.rados.0.smithi064.stdout:2317: writing smithi06431743-35 from 1883069 to 2218788 tid 2 2024-03-04T06:17:14.278 INFO:tasks.rados.rados.0.smithi064.stdout:2318: snap_remove snap 276 2024-03-04T06:17:14.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:14.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:14.744+0000 7f6ef495d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:14.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:14.748+0000 7f6ef495d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:15.268 INFO:tasks.rados.rados.0.smithi064.stdout:2318: done (3 left) 2024-03-04T06:17:15.268 INFO:tasks.rados.rados.0.smithi064.stdout:2319: snap_create 2024-03-04T06:17:15.268 INFO:tasks.rados.rados.0.smithi064.stdout:2314: expect (ObjNum 1042 snap 265 seq_num 1042) 2024-03-04T06:17:15.500 INFO:tasks.rados.rados.0.smithi064.stdout:2316: expect (ObjNum 1045 snap 265 seq_num 1045) 2024-03-04T06:17:15.720 INFO:tasks.rados.rados.0.smithi064.stdout:2317: finishing write tid 1 to smithi06431743-35 2024-03-04T06:17:15.720 INFO:tasks.rados.rados.0.smithi064.stdout:2317: finishing write tid 2 to smithi06431743-35 2024-03-04T06:17:15.720 INFO:tasks.rados.rados.0.smithi064.stdout:2317: finishing write tid 3 to smithi06431743-35 2024-03-04T06:17:15.720 INFO:tasks.rados.rados.0.smithi064.stdout:2317: finishing write tid 4 to smithi06431743-35 2024-03-04T06:17:15.720 INFO:tasks.rados.rados.0.smithi064.stdout:2317: oid 35 updating version 0 to 685 2024-03-04T06:17:15.720 INFO:tasks.rados.rados.0.smithi064.stdout:2317: oid 35 version 685 is already newer than 685 2024-03-04T06:17:15.720 INFO:tasks.rados.rados.0.smithi064.stdout:2317: oid 35 version 685 is already newer than 683 2024-03-04T06:17:15.720 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 685 (ObjNum 1084 snap 277 seq_num 1084) dirty exists 2024-03-04T06:17:15.720 INFO:tasks.rados.rados.0.smithi064.stdout:2317: left oid 35 (ObjNum 1084 snap 277 seq_num 1084) 2024-03-04T06:17:16.270 INFO:tasks.rados.rados.0.smithi064.stdout:2314: done (3 left) 2024-03-04T06:17:16.270 INFO:tasks.rados.rados.0.smithi064.stdout:2316: done (2 left) 2024-03-04T06:17:16.270 INFO:tasks.rados.rados.0.smithi064.stdout:2317: done (1 left) 2024-03-04T06:17:16.270 INFO:tasks.rados.rados.0.smithi064.stdout:2319: done (0 left) 2024-03-04T06:17:16.270 INFO:tasks.rados.rados.0.smithi064.stdout:2320: snap_remove snap 273 2024-03-04T06:17:16.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:16.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:16.932+0000 7f1d36b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:16.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:16.936+0000 7f1d36b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:17.274 INFO:tasks.rados.rados.0.smithi064.stdout:2320: done (0 left) 2024-03-04T06:17:17.275 INFO:tasks.rados.rados.0.smithi064.stdout:2321: snap_create 2024-03-04T06:17:18.276 INFO:tasks.rados.rados.0.smithi064.stdout:2321: done (0 left) 2024-03-04T06:17:18.276 INFO:tasks.rados.rados.0.smithi064.stdout:2322: delete oid 45 current snap is 279 2024-03-04T06:17:18.327 INFO:tasks.rados.rados.0.smithi064.stdout:2322: done (0 left) 2024-03-04T06:17:18.327 INFO:tasks.rados.rados.0.smithi064.stdout:2323: read oid 1 snap 278 2024-03-04T06:17:18.327 INFO:tasks.rados.rados.0.smithi064.stdout:2323: expect (ObjNum 1046 snap 265 seq_num 1046) 2024-03-04T06:17:18.327 INFO:tasks.rados.rados.0.smithi064.stdout:2324: delete oid 28 current snap is 279 2024-03-04T06:17:18.377 INFO:tasks.rados.rados.0.smithi064.stdout:2323: expect (ObjNum 1046 snap 265 seq_num 1046) 2024-03-04T06:17:18.526 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:17:18.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:17:18.586 INFO:tasks.rados.rados.0.smithi064.stdout:2323: done (1 left) 2024-03-04T06:17:18.587 INFO:tasks.rados.rados.0.smithi064.stdout:2324: done (0 left) 2024-03-04T06:17:18.587 INFO:tasks.rados.rados.0.smithi064.stdout:2325: read oid 16 snap -1 2024-03-04T06:17:18.587 INFO:tasks.rados.rados.0.smithi064.stdout:2325: expect (ObjNum 1038 snap 265 seq_num 1038) 2024-03-04T06:17:18.587 INFO:tasks.rados.rados.0.smithi064.stdout:2326: read oid 24 snap -1 2024-03-04T06:17:18.587 INFO:tasks.rados.rados.0.smithi064.stdout:2326: expect deleted 2024-03-04T06:17:18.587 INFO:tasks.rados.rados.0.smithi064.stdout:2327: read oid 9 snap -1 2024-03-04T06:17:18.587 INFO:tasks.rados.rados.0.smithi064.stdout:2327: expect deleted 2024-03-04T06:17:18.587 INFO:tasks.rados.rados.0.smithi064.stdout:2328: read oid 3 snap 271 2024-03-04T06:17:18.587 INFO:tasks.rados.rados.0.smithi064.stdout:2328: expect (ObjNum 1032 snap 264 seq_num 1032) 2024-03-04T06:17:18.587 INFO:tasks.rados.rados.0.smithi064.stdout:2329: delete oid 40 current snap is 279 2024-03-04T06:17:18.597 INFO:tasks.rados.rados.0.smithi064.stdout:2328: expect (ObjNum 1032 snap 264 seq_num 1032) 2024-03-04T06:17:18.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:18.637+0000 7f18de46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:18.645 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:18.645+0000 7f18de46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:18.726 INFO:tasks.rados.rados.0.smithi064.stdout:2326: done (4 left) 2024-03-04T06:17:18.726 INFO:tasks.rados.rados.0.smithi064.stdout:2327: done (3 left) 2024-03-04T06:17:18.726 INFO:tasks.rados.rados.0.smithi064.stdout:2328: done (2 left) 2024-03-04T06:17:18.726 INFO:tasks.rados.rados.0.smithi064.stdout:2329: done (1 left) 2024-03-04T06:17:18.726 INFO:tasks.rados.rados.0.smithi064.stdout:2330: read oid 29 snap 270 2024-03-04T06:17:18.726 INFO:tasks.rados.rados.0.smithi064.stdout:2330: expect (ObjNum 1016 snap 261 seq_num 1016) 2024-03-04T06:17:18.726 INFO:tasks.rados.rados.0.smithi064.stdout:2331: write (excl) oid 4 current snap is 279 2024-03-04T06:17:18.726 INFO:tasks.rados.rados.0.smithi064.stdout:2331: seq_num 1088 ranges {686016=509391,1716997=569019} 2024-03-04T06:17:18.732 INFO:tasks.rados.rados.0.smithi064.stdout:2331: writing smithi06431743-4 from 686016 to 1195407 tid 1 2024-03-04T06:17:18.739 INFO:tasks.rados.rados.0.smithi064.stdout:2331: writing smithi06431743-4 from 1716997 to 2286016 tid 2 2024-03-04T06:17:18.739 INFO:tasks.rados.rados.0.smithi064.stdout:2332: read oid 24 snap -1 2024-03-04T06:17:18.739 INFO:tasks.rados.rados.0.smithi064.stdout:2332: expect deleted 2024-03-04T06:17:18.739 INFO:tasks.rados.rados.0.smithi064.stdout:2333: read oid 15 snap -1 2024-03-04T06:17:18.739 INFO:tasks.rados.rados.0.smithi064.stdout:2333: expect (ObjNum 1049 snap 266 seq_num 1049) 2024-03-04T06:17:18.739 INFO:tasks.rados.rados.0.smithi064.stdout:2334: delete oid 39 current snap is 279 2024-03-04T06:17:18.739 INFO:tasks.rados.rados.0.smithi064.stdout:2325: expect (ObjNum 1038 snap 265 seq_num 1038) 2024-03-04T06:17:18.957 INFO:tasks.rados.rados.0.smithi064.stdout:2330: expect (ObjNum 1016 snap 261 seq_num 1016) 2024-03-04T06:17:19.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:19.113 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:19.111+0000 7fd7c4224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:19.116 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:19.115+0000 7fd7c4224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:19.182 INFO:tasks.rados.rados.0.smithi064.stdout:2333: expect (ObjNum 1049 snap 266 seq_num 1049) 2024-03-04T06:17:19.277 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:17:19.290 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:17:19.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:19.392+0000 7f861546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:19.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:19.401+0000 7f861546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:19.413 INFO:tasks.rados.rados.0.smithi064.stdout:2331: finishing write tid 1 to smithi06431743-4 2024-03-04T06:17:19.413 INFO:tasks.rados.rados.0.smithi064.stdout:2331: finishing write tid 2 to smithi06431743-4 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2331: finishing write tid 3 to smithi06431743-4 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2331: finishing write tid 4 to smithi06431743-4 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2331: oid 4 updating version 0 to 1010 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2331: oid 4 version 1010 is already newer than 1008 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2331: oid 4 version 1010 is already newer than 1009 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 1010 (ObjNum 1088 snap 279 seq_num 1088) dirty exists 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2331: left oid 4 (ObjNum 1088 snap 279 seq_num 1088) 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2325: done (5 left) 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2330: done (4 left) 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2331: done (3 left) 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2332: done (2 left) 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2333: done (1 left) 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2334: done (0 left) 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2335: write (excl) oid 20 current snap is 279 2024-03-04T06:17:19.414 INFO:tasks.rados.rados.0.smithi064.stdout:2335: seq_num 1090 ranges {719630=659889,2117608=202022} 2024-03-04T06:17:19.420 INFO:tasks.rados.rados.0.smithi064.stdout:2335: writing smithi06431743-20 from 719630 to 1379519 tid 1 2024-03-04T06:17:19.425 INFO:tasks.rados.rados.0.smithi064.stdout:2335: writing smithi06431743-20 from 2117608 to 2319630 tid 2 2024-03-04T06:17:19.425 INFO:tasks.rados.rados.0.smithi064.stdout:2336: write oid 50 current snap is 279 2024-03-04T06:17:19.426 INFO:tasks.rados.rados.0.smithi064.stdout:2336: seq_num 1091 ranges {736437=735138,2176090=160347} 2024-03-04T06:17:19.434 INFO:tasks.rados.rados.0.smithi064.stdout:2336: writing smithi06431743-50 from 736437 to 1471575 tid 1 2024-03-04T06:17:19.440 INFO:tasks.rados.rados.0.smithi064.stdout:2336: writing smithi06431743-50 from 2176090 to 2336437 tid 2 2024-03-04T06:17:19.440 INFO:tasks.rados.rados.0.smithi064.stdout:2337: read oid 21 snap -1 2024-03-04T06:17:19.440 INFO:tasks.rados.rados.0.smithi064.stdout:2337: expect (ObjNum 1045 snap 265 seq_num 1045) 2024-03-04T06:17:19.440 INFO:tasks.rados.rados.0.smithi064.stdout:2338: write (excl) oid 47 current snap is 279 2024-03-04T06:17:19.440 INFO:tasks.rados.rados.0.smithi064.stdout:2338: seq_num 1092 ranges {753244=410387,1834572=518672} 2024-03-04T06:17:19.446 INFO:tasks.rados.rados.0.smithi064.stdout:2338: writing smithi06431743-47 from 753244 to 1163631 tid 1 2024-03-04T06:17:19.455 INFO:tasks.rados.rados.0.smithi064.stdout:2338: writing smithi06431743-47 from 1834572 to 2353244 tid 2 2024-03-04T06:17:19.455 INFO:tasks.rados.rados.0.smithi064.stdout:2339: snap_create 2024-03-04T06:17:19.455 INFO:tasks.rados.rados.0.smithi064.stdout:2335: finishing write tid 1 to smithi06431743-20 2024-03-04T06:17:19.472 INFO:tasks.rados.rados.0.smithi064.stdout:2337: expect (ObjNum 1045 snap 265 seq_num 1045) 2024-03-04T06:17:19.696 INFO:tasks.rados.rados.0.smithi064.stdout:2338: finishing write tid 1 to smithi06431743-47 2024-03-04T06:17:19.696 INFO:tasks.rados.rados.0.smithi064.stdout:2336: finishing write tid 1 to smithi06431743-50 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2335: finishing write tid 2 to smithi06431743-20 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2338: finishing write tid 2 to smithi06431743-47 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2335: finishing write tid 3 to smithi06431743-20 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2336: finishing write tid 2 to smithi06431743-50 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2335: finishing write tid 4 to smithi06431743-20 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2335: oid 20 updating version 0 to 692 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2335: oid 20 version 692 is already newer than 690 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2335: oid 20 version 692 is already newer than 691 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 692 (ObjNum 1090 snap 279 seq_num 1090) dirty exists 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2335: left oid 20 (ObjNum 1090 snap 279 seq_num 1090) 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2338: finishing write tid 3 to smithi06431743-47 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2338: finishing write tid 4 to smithi06431743-47 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2338: oid 47 updating version 0 to 367 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2338: oid 47 version 367 is already newer than 366 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:2338: oid 47 updating version 367 to 368 2024-03-04T06:17:19.697 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 368 (ObjNum 1092 snap 279 seq_num 1092) dirty exists 2024-03-04T06:17:19.698 INFO:tasks.rados.rados.0.smithi064.stdout:2338: left oid 47 (ObjNum 1092 snap 279 seq_num 1092) 2024-03-04T06:17:19.698 INFO:tasks.rados.rados.0.smithi064.stdout:2336: finishing write tid 3 to smithi06431743-50 2024-03-04T06:17:19.698 INFO:tasks.rados.rados.0.smithi064.stdout:2336: finishing write tid 4 to smithi06431743-50 2024-03-04T06:17:19.698 INFO:tasks.rados.rados.0.smithi064.stdout:2336: oid 50 updating version 0 to 801 2024-03-04T06:17:19.698 INFO:tasks.rados.rados.0.smithi064.stdout:2336: oid 50 updating version 801 to 802 2024-03-04T06:17:19.698 INFO:tasks.rados.rados.0.smithi064.stdout:2336: oid 50 updating version 802 to 803 2024-03-04T06:17:19.698 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 803 (ObjNum 1091 snap 279 seq_num 1091) dirty exists 2024-03-04T06:17:19.698 INFO:tasks.rados.rados.0.smithi064.stdout:2336: left oid 50 (ObjNum 1091 snap 279 seq_num 1091) 2024-03-04T06:17:20.283 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:17:20.286 INFO:tasks.rados.rados.0.smithi064.stdout:2335: done (4 left) 2024-03-04T06:17:20.286 INFO:tasks.rados.rados.0.smithi064.stdout:2336: done (3 left) 2024-03-04T06:17:20.286 INFO:tasks.rados.rados.0.smithi064.stdout:2337: done (2 left) 2024-03-04T06:17:20.286 INFO:tasks.rados.rados.0.smithi064.stdout:2338: done (1 left) 2024-03-04T06:17:20.286 INFO:tasks.rados.rados.0.smithi064.stdout:2339: done (0 left) 2024-03-04T06:17:20.286 INFO:tasks.rados.rados.0.smithi064.stdout:2340: write (excl) oid 7 current snap is 280 2024-03-04T06:17:20.287 INFO:tasks.rados.rados.0.smithi064.stdout:2340: seq_num 1093 ranges {770051=485636,1776701=511047} 2024-03-04T06:17:20.293 INFO:tasks.rados.rados.0.smithi064.stdout:2340: writing smithi06431743-7 from 770051 to 1255687 tid 1 2024-03-04T06:17:20.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:17:20.300 INFO:tasks.rados.rados.0.smithi064.stdout:2340: writing smithi06431743-7 from 1776701 to 2287748 tid 2 2024-03-04T06:17:20.300 INFO:tasks.rados.rados.0.smithi064.stdout:2341: snap_create 2024-03-04T06:17:20.356 INFO:tasks.rados.rados.0.smithi064.stdout:2340: finishing write tid 1 to smithi06431743-7 2024-03-04T06:17:20.386 INFO:tasks.rados.rados.0.smithi064.stdout:2340: finishing write tid 2 to smithi06431743-7 2024-03-04T06:17:20.390 INFO:tasks.rados.rados.0.smithi064.stdout:2340: finishing write tid 3 to smithi06431743-7 2024-03-04T06:17:20.390 INFO:tasks.rados.rados.0.smithi064.stdout:2340: finishing write tid 4 to smithi06431743-7 2024-03-04T06:17:20.391 INFO:tasks.rados.rados.0.smithi064.stdout:2340: oid 7 updating version 0 to 377 2024-03-04T06:17:20.391 INFO:tasks.rados.rados.0.smithi064.stdout:2340: oid 7 version 377 is already newer than 377 2024-03-04T06:17:20.391 INFO:tasks.rados.rados.0.smithi064.stdout:2340: oid 7 version 377 is already newer than 375 2024-03-04T06:17:20.391 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 377 (ObjNum 1093 snap 280 seq_num 1093) dirty exists 2024-03-04T06:17:20.391 INFO:tasks.rados.rados.0.smithi064.stdout:2340: left oid 7 (ObjNum 1093 snap 280 seq_num 1093) 2024-03-04T06:17:20.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:20.415+0000 7fb9c0a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:20.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:20.423+0000 7fb9c0a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:21.183 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:21.286 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:17:21.289 INFO:tasks.rados.rados.0.smithi064.stdout:2340: done (1 left) 2024-03-04T06:17:21.290 INFO:tasks.rados.rados.0.smithi064.stdout:2341: done (0 left) 2024-03-04T06:17:21.290 INFO:tasks.rados.rados.0.smithi064.stdout:2342: read oid 41 snap -1 2024-03-04T06:17:21.290 INFO:tasks.rados.rados.0.smithi064.stdout:2342: expect (ObjNum 985 snap 251 seq_num 985) 2024-03-04T06:17:21.290 INFO:tasks.rados.rados.0.smithi064.stdout:2343: write oid 8 current snap is 281 2024-03-04T06:17:21.290 INFO:tasks.rados.rados.0.smithi064.stdout:2343: seq_num 1094 ranges {786858=560885,1835183=551675} 2024-03-04T06:17:21.294 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:21.291+0000 7f537b817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:21.297 INFO:tasks.rados.rados.0.smithi064.stdout:2343: writing smithi06431743-8 from 786858 to 1347743 tid 1 2024-03-04T06:17:21.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:21.296+0000 7f537b817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:21.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:17:21.305 INFO:tasks.rados.rados.0.smithi064.stdout:2343: writing smithi06431743-8 from 1835183 to 2386858 tid 2 2024-03-04T06:17:21.305 INFO:tasks.rados.rados.0.smithi064.stdout:2344: read oid 42 snap -1 2024-03-04T06:17:21.305 INFO:tasks.rados.rados.0.smithi064.stdout:2344: expect deleted 2024-03-04T06:17:21.305 INFO:tasks.rados.rados.0.smithi064.stdout:2345: read oid 34 snap 279 2024-03-04T06:17:21.305 INFO:tasks.rados.rados.0.smithi064.stdout:2345: expect deleted 2024-03-04T06:17:21.306 INFO:tasks.rados.rados.0.smithi064.stdout:2346: read oid 31 snap -1 2024-03-04T06:17:21.306 INFO:tasks.rados.rados.0.smithi064.stdout:2346: expect deleted 2024-03-04T06:17:21.306 INFO:tasks.rados.rados.0.smithi064.stdout:2347: snap_remove snap 275 2024-03-04T06:17:21.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:21.422+0000 7ff46986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:21.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:21.429+0000 7ff46986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:22.289 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:17:22.292 INFO:tasks.rados.rados.0.smithi064.stdout:2347: done (5 left) 2024-03-04T06:17:22.292 INFO:tasks.rados.rados.0.smithi064.stdout:2348: snap_create 2024-03-04T06:17:22.292 INFO:tasks.rados.rados.0.smithi064.stdout:2342: expect (ObjNum 985 snap 251 seq_num 985) 2024-03-04T06:17:22.304 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:17:22.304 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:17:22.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:22.448+0000 7f239fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:22.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:22.455+0000 7f239fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:2343: finishing write tid 1 to smithi06431743-8 2024-03-04T06:17:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:2343: finishing write tid 2 to smithi06431743-8 2024-03-04T06:17:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:2343: finishing write tid 3 to smithi06431743-8 2024-03-04T06:17:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:2343: finishing write tid 4 to smithi06431743-8 2024-03-04T06:17:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:2343: oid 8 updating version 0 to 453 2024-03-04T06:17:22.462 INFO:tasks.rados.rados.0.smithi064.stdout:2343: oid 8 version 453 is already newer than 452 2024-03-04T06:17:22.463 INFO:tasks.rados.rados.0.smithi064.stdout:2343: oid 8 version 453 is already newer than 451 2024-03-04T06:17:22.463 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 453 (ObjNum 1094 snap 281 seq_num 1094) dirty exists 2024-03-04T06:17:22.463 INFO:tasks.rados.rados.0.smithi064.stdout:2343: left oid 8 (ObjNum 1094 snap 281 seq_num 1094) 2024-03-04T06:17:22.646 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:17:22.646 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:17:22.657 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":851,"stamp":"2024-03-04T06:17:22.290525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":97893747,"num_objects":808,"num_object_clones":626,"num_object_copies":2424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4391,"num_read_kb":3353324,"num_write":3769,"num_write_kb":988836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8132,"ondisk_log_size":8132,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":563184,"kb_used_data":380944,"kb_used_omap":0,"kb_used_meta":182208,"kb_avail":376924176,"statfs":{"total":386547056640,"available":385970356224,"internally_reserved":0,"allocated":390086656,"data_stored":428935964,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":186580992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4268846,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":50014,"num_write":23,"num_write_kb":4655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016341"},"pg_stats":[{"pgid":"3.f","version":"568'495","reported_seq":2002,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.341573+0000","last_change":"2024-03-04T06:17:18.329967+0000","last_active":"2024-03-04T06:17:20.341573+0000","last_peered":"2024-03-04T06:17:20.341573+0000","last_clean":"2024-03-04T06:17:20.341573+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:17:20.341573+0000","last_undegraded":"2024-03-04T06:17:20.341573+0000","last_fullsized":"2024-03-04T06:17:20.341573+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151560,"num_objects":54,"num_object_clones":43,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":271,"num_read_kb":224365,"num_write":222,"num_write_kb":59334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"564'397","reported_seq":1926,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.292609+0000","last_change":"2024-03-04T06:17:18.457263+0000","last_active":"2024-03-04T06:17:20.292609+0000","last_peered":"2024-03-04T06:17:20.292609+0000","last_clean":"2024-03-04T06:17:20.292609+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:17:20.292609+0000","last_undegraded":"2024-03-04T06:17:20.292609+0000","last_fullsized":"2024-03-04T06:17:20.292609+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7479577,"num_objects":44,"num_object_clones":30,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":137,"num_read_kb":80772,"num_write":176,"num_write_kb":46528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"568'521","reported_seq":2064,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.309107+0000","last_change":"2024-03-04T06:17:18.308698+0000","last_active":"2024-03-04T06:17:21.309107+0000","last_peered":"2024-03-04T06:17:21.309107+0000","last_clean":"2024-03-04T06:17:21.309107+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:17:21.309107+0000","last_undegraded":"2024-03-04T06:17:21.309107+0000","last_fullsized":"2024-03-04T06:17:21.309107+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3849001,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":342,"num_read_kb":275867,"num_write":253,"num_write_kb":69591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"568'368","reported_seq":1932,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.329855+0000","last_change":"2024-03-04T06:17:18.316518+0000","last_active":"2024-03-04T06:17:20.329855+0000","last_peered":"2024-03-04T06:17:20.329855+0000","last_clean":"2024-03-04T06:17:20.329855+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:17:20.329855+0000","last_undegraded":"2024-03-04T06:17:20.329855+0000","last_fullsized":"2024-03-04T06:17:20.329855+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9362225,"num_objects":36,"num_object_clones":26,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":197,"num_read_kb":149129,"num_write":178,"num_write_kb":49693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"567'179","reported_seq":1781,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.334654+0000","last_change":"2024-03-04T06:17:17.324329+0000","last_active":"2024-03-04T06:17:21.334654+0000","last_peered":"2024-03-04T06:17:21.334654+0000","last_clean":"2024-03-04T06:17:21.334654+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:17:21.334654+0000","last_undegraded":"2024-03-04T06:17:21.334654+0000","last_fullsized":"2024-03-04T06:17:21.334654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":60,"num_read_kb":33097,"num_write":80,"num_write_kb":21063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"564'556","reported_seq":2022,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.344699+0000","last_change":"2024-03-04T06:17:18.329195+0000","last_active":"2024-03-04T06:17:20.344699+0000","last_peered":"2024-03-04T06:17:20.344699+0000","last_clean":"2024-03-04T06:17:20.344699+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:17:20.344699+0000","last_undegraded":"2024-03-04T06:17:20.344699+0000","last_fullsized":"2024-03-04T06:17:20.344699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8622717,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":255,"num_read_kb":192237,"num_write":255,"num_write_kb":74444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1682,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.338873+0000","last_change":"2024-03-04T06:17:18.329854+0000","last_active":"2024-03-04T06:17:20.338873+0000","last_peered":"2024-03-04T06:17:20.338873+0000","last_clean":"2024-03-04T06:17:20.338873+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:17:20.338873+0000","last_undegraded":"2024-03-04T06:17:20.338873+0000","last_fullsized":"2024-03-04T06:17:20.338873+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"557'270","reported_seq":1875,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.318457+0000","last_change":"2024-03-04T06:17:18.309259+0000","last_active":"2024-03-04T06:17:21.318457+0000","last_peered":"2024-03-04T06:17:21.318457+0000","last_clean":"2024-03-04T06:17:21.318457+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:17:21.318457+0000","last_undegraded":"2024-03-04T06:17:21.318457+0000","last_fullsized":"2024-03-04T06:17:21.318457+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6437477,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":176,"num_read_kb":163221,"num_write":123,"num_write_kb":32558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1556,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.321849+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:17:21.321849+0000","last_peered":"2024-03-04T06:17:21.321849+0000","last_clean":"2024-03-04T06:17:21.321849+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:17:21.321849+0000","last_undegraded":"2024-03-04T06:17:21.321849+0000","last_fullsized":"2024-03-04T06:17:21.321849+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.328926+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:17:21.328926+0000","last_peered":"2024-03-04T06:17:21.328926+0000","last_clean":"2024-03-04T06:17:21.328926+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:17:21.328926+0000","last_undegraded":"2024-03-04T06:17:21.328926+0000","last_fullsized":"2024-03-04T06:17:21.328926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"557'527","reported_seq":2017,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.339869+0000","last_change":"2024-03-04T06:17:18.320820+0000","last_active":"2024-03-04T06:17:21.339869+0000","last_peered":"2024-03-04T06:17:21.339869+0000","last_clean":"2024-03-04T06:17:21.339869+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:17:21.339869+0000","last_undegraded":"2024-03-04T06:17:21.339869+0000","last_fullsized":"2024-03-04T06:17:21.339869+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":52,"num_object_clones":40,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":276,"num_read_kb":221534,"num_write":234,"num_write_kb":63712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1420,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.344082+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:17:20.344082+0000","last_peered":"2024-03-04T06:17:20.344082+0000","last_clean":"2024-03-04T06:17:20.344082+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:17:20.344082+0000","last_undegraded":"2024-03-04T06:17:20.344082+0000","last_fullsized":"2024-03-04T06:17:20.344082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"570'377","reported_seq":1914,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.320564+0000","last_change":"2024-03-04T06:17:17.323092+0000","last_active":"2024-03-04T06:17:21.320564+0000","last_peered":"2024-03-04T06:17:21.320564+0000","last_clean":"2024-03-04T06:17:21.320564+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:17:21.320564+0000","last_undegraded":"2024-03-04T06:17:21.320564+0000","last_fullsized":"2024-03-04T06:17:21.320564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370051,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":219,"num_read_kb":177501,"num_write":146,"num_write_kb":41225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.316794+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:17:21.316794+0000","last_peered":"2024-03-04T06:17:21.316794+0000","last_clean":"2024-03-04T06:17:21.316794+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:17:21.316794+0000","last_undegraded":"2024-03-04T06:17:21.316794+0000","last_fullsized":"2024-03-04T06:17:21.316794+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"568'923","reported_seq":2327,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.328518+0000","last_change":"2024-03-04T06:17:17.304752+0000","last_active":"2024-03-04T06:17:21.328518+0000","last_peered":"2024-03-04T06:17:21.328518+0000","last_clean":"2024-03-04T06:17:21.328518+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:17:21.328518+0000","last_undegraded":"2024-03-04T06:17:21.328518+0000","last_fullsized":"2024-03-04T06:17:21.328518+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":923,"log_dups_size":0,"ondisk_log_size":923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5630840,"num_objects":93,"num_object_clones":72,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":553,"num_read_kb":440379,"num_write":438,"num_write_kb":117199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1420,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.329171+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:17:20.329171+0000","last_peered":"2024-03-04T06:17:20.329171+0000","last_clean":"2024-03-04T06:17:20.329171+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:17:20.329171+0000","last_undegraded":"2024-03-04T06:17:20.329171+0000","last_fullsized":"2024-03-04T06:17:20.329171+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"568'1010","reported_seq":2307,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.298923+0000","last_change":"2024-03-04T06:17:18.458101+0000","last_active":"2024-03-04T06:17:20.298923+0000","last_peered":"2024-03-04T06:17:20.298923+0000","last_clean":"2024-03-04T06:17:20.298923+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:17:20.298923+0000","last_undegraded":"2024-03-04T06:17:20.298923+0000","last_fullsized":"2024-03-04T06:17:20.298923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1010,"log_dups_size":0,"ondisk_log_size":1010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16472048,"num_objects":97,"num_object_clones":75,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":447,"num_read_kb":328457,"num_write":448,"num_write_kb":118529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"559'468","reported_seq":2037,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.326062+0000","last_change":"2024-03-04T06:17:18.321202+0000","last_active":"2024-03-04T06:17:21.326062+0000","last_peered":"2024-03-04T06:17:21.326062+0000","last_clean":"2024-03-04T06:17:21.326062+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:17:21.326062+0000","last_undegraded":"2024-03-04T06:17:21.326062+0000","last_fullsized":"2024-03-04T06:17:21.326062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9311804,"num_objects":47,"num_object_clones":35,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":331,"num_read_kb":272228,"num_write":223,"num_write_kb":60598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1450,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.309012+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:17:20.309012+0000","last_peered":"2024-03-04T06:17:20.309012+0000","last_clean":"2024-03-04T06:17:20.309012+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:17:20.309012+0000","last_undegraded":"2024-03-04T06:17:20.309012+0000","last_fullsized":"2024-03-04T06:17:20.309012+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"557'449","reported_seq":2002,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.307732+0000","last_change":"2024-03-04T06:17:18.463468+0000","last_active":"2024-03-04T06:17:20.307732+0000","last_peered":"2024-03-04T06:17:20.307732+0000","last_clean":"2024-03-04T06:17:20.307732+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:17:20.307732+0000","last_undegraded":"2024-03-04T06:17:20.307732+0000","last_fullsized":"2024-03-04T06:17:20.307732+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143437,"num_objects":52,"num_object_clones":37,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":235,"num_read_kb":167519,"num_write":210,"num_write_kb":55510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.333768+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:17:21.333768+0000","last_peered":"2024-03-04T06:17:21.333768+0000","last_clean":"2024-03-04T06:17:21.333768+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:17:21.333768+0000","last_undegraded":"2024-03-04T06:17:21.333768+0000","last_fullsized":"2024-03-04T06:17:21.333768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"568'692","reported_seq":2110,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.323093+0000","last_change":"2024-03-04T06:17:17.321941+0000","last_active":"2024-03-04T06:17:21.323093+0000","last_peered":"2024-03-04T06:17:21.323093+0000","last_clean":"2024-03-04T06:17:21.323093+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:17:21.323093+0000","last_undegraded":"2024-03-04T06:17:21.323093+0000","last_fullsized":"2024-03-04T06:17:21.323093+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6723592,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":339,"num_read_kb":253707,"num_write":310,"num_write_kb":84843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1450,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.301276+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:17:20.301276+0000","last_peered":"2024-03-04T06:17:20.301276+0000","last_clean":"2024-03-04T06:17:20.301276+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:17:20.301276+0000","last_undegraded":"2024-03-04T06:17:20.301276+0000","last_fullsized":"2024-03-04T06:17:20.301276+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"569'803","reported_seq":2217,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.297716+0000","last_change":"2024-03-04T06:17:18.457995+0000","last_active":"2024-03-04T06:17:20.297716+0000","last_peered":"2024-03-04T06:17:20.297716+0000","last_clean":"2024-03-04T06:17:20.297716+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:17:20.297716+0000","last_undegraded":"2024-03-04T06:17:20.297716+0000","last_fullsized":"2024-03-04T06:17:20.297716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849331,"num_objects":73,"num_object_clones":64,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":450,"num_read_kb":373220,"num_write":336,"num_write_kb":92115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.330114+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:17:21.330114+0000","last_peered":"2024-03-04T06:17:21.330114+0000","last_clean":"2024-03-04T06:17:21.330114+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:17:21.330114+0000","last_undegraded":"2024-03-04T06:17:21.330114+0000","last_fullsized":"2024-03-04T06:17:21.330114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":97303379,"num_objects":806,"num_object_clones":626,"num_object_copies":2418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4288,"num_read_kb":3353233,"num_write":3632,"num_write_kb":986942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8035,"ondisk_log_size":8035,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166444,"kb_used_data":117412,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94205396,"statfs":{"total":96636764160,"available":96466325504,"internally_reserved":0,"allocated":120229888,"data_stored":133616821,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":141916,"kb_used_data":97044,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94229924,"statfs":{"total":96636764160,"available":96491442176,"internally_reserved":0,"allocated":99373056,"data_stored":109085667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706238,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":134732,"kb_used_data":89860,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94237108,"statfs":{"total":96636764160,"available":96498798592,"internally_reserved":0,"allocated":92016640,"data_stored":102692131,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706238,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":120092,"kb_used_data":76628,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94251748,"statfs":{"total":96636764160,"available":96513789952,"internally_reserved":0,"allocated":78467072,"data_stored":83541345,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:17:22.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:17:22.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:22.762+0000 7fcfa846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:22.769 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:22.769+0000 7fcfa846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:22.957 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:17:22.958 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:17:22.969 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":852,"stamp":"2024-03-04T06:17:22.828469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":97893747,"num_objects":808,"num_object_clones":626,"num_object_copies":2424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4391,"num_read_kb":3353324,"num_write":3769,"num_write_kb":988836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8132,"ondisk_log_size":8132,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":563184,"kb_used_data":380944,"kb_used_omap":0,"kb_used_meta":182208,"kb_avail":376924176,"statfs":{"total":386547056640,"available":385970356224,"internally_reserved":0,"allocated":390086656,"data_stored":428935964,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":186580992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4268846,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":50014,"num_write":23,"num_write_kb":4655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000704"},"pg_stats":[{"pgid":"3.f","version":"568'495","reported_seq":2002,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.341573+0000","last_change":"2024-03-04T06:17:18.329967+0000","last_active":"2024-03-04T06:17:20.341573+0000","last_peered":"2024-03-04T06:17:20.341573+0000","last_clean":"2024-03-04T06:17:20.341573+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:17:20.341573+0000","last_undegraded":"2024-03-04T06:17:20.341573+0000","last_fullsized":"2024-03-04T06:17:20.341573+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151560,"num_objects":54,"num_object_clones":43,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":271,"num_read_kb":224365,"num_write":222,"num_write_kb":59334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"564'397","reported_seq":1926,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.292609+0000","last_change":"2024-03-04T06:17:18.457263+0000","last_active":"2024-03-04T06:17:20.292609+0000","last_peered":"2024-03-04T06:17:20.292609+0000","last_clean":"2024-03-04T06:17:20.292609+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:17:20.292609+0000","last_undegraded":"2024-03-04T06:17:20.292609+0000","last_fullsized":"2024-03-04T06:17:20.292609+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7479577,"num_objects":44,"num_object_clones":30,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":137,"num_read_kb":80772,"num_write":176,"num_write_kb":46528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"568'521","reported_seq":2064,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.309107+0000","last_change":"2024-03-04T06:17:18.308698+0000","last_active":"2024-03-04T06:17:21.309107+0000","last_peered":"2024-03-04T06:17:21.309107+0000","last_clean":"2024-03-04T06:17:21.309107+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:17:21.309107+0000","last_undegraded":"2024-03-04T06:17:21.309107+0000","last_fullsized":"2024-03-04T06:17:21.309107+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3849001,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":342,"num_read_kb":275867,"num_write":253,"num_write_kb":69591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"568'368","reported_seq":1932,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.329855+0000","last_change":"2024-03-04T06:17:18.316518+0000","last_active":"2024-03-04T06:17:20.329855+0000","last_peered":"2024-03-04T06:17:20.329855+0000","last_clean":"2024-03-04T06:17:20.329855+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:17:20.329855+0000","last_undegraded":"2024-03-04T06:17:20.329855+0000","last_fullsized":"2024-03-04T06:17:20.329855+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9362225,"num_objects":36,"num_object_clones":26,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":197,"num_read_kb":149129,"num_write":178,"num_write_kb":49693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"567'179","reported_seq":1781,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.334654+0000","last_change":"2024-03-04T06:17:17.324329+0000","last_active":"2024-03-04T06:17:21.334654+0000","last_peered":"2024-03-04T06:17:21.334654+0000","last_clean":"2024-03-04T06:17:21.334654+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:17:21.334654+0000","last_undegraded":"2024-03-04T06:17:21.334654+0000","last_fullsized":"2024-03-04T06:17:21.334654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":60,"num_read_kb":33097,"num_write":80,"num_write_kb":21063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"564'556","reported_seq":2022,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.344699+0000","last_change":"2024-03-04T06:17:18.329195+0000","last_active":"2024-03-04T06:17:20.344699+0000","last_peered":"2024-03-04T06:17:20.344699+0000","last_clean":"2024-03-04T06:17:20.344699+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:17:20.344699+0000","last_undegraded":"2024-03-04T06:17:20.344699+0000","last_fullsized":"2024-03-04T06:17:20.344699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8622717,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":255,"num_read_kb":192237,"num_write":255,"num_write_kb":74444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1682,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.338873+0000","last_change":"2024-03-04T06:17:18.329854+0000","last_active":"2024-03-04T06:17:20.338873+0000","last_peered":"2024-03-04T06:17:20.338873+0000","last_clean":"2024-03-04T06:17:20.338873+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:17:20.338873+0000","last_undegraded":"2024-03-04T06:17:20.338873+0000","last_fullsized":"2024-03-04T06:17:20.338873+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"557'270","reported_seq":1875,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.318457+0000","last_change":"2024-03-04T06:17:18.309259+0000","last_active":"2024-03-04T06:17:21.318457+0000","last_peered":"2024-03-04T06:17:21.318457+0000","last_clean":"2024-03-04T06:17:21.318457+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:17:21.318457+0000","last_undegraded":"2024-03-04T06:17:21.318457+0000","last_fullsized":"2024-03-04T06:17:21.318457+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6437477,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":176,"num_read_kb":163221,"num_write":123,"num_write_kb":32558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1556,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.321849+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:17:21.321849+0000","last_peered":"2024-03-04T06:17:21.321849+0000","last_clean":"2024-03-04T06:17:21.321849+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:17:21.321849+0000","last_undegraded":"2024-03-04T06:17:21.321849+0000","last_fullsized":"2024-03-04T06:17:21.321849+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.328926+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:17:21.328926+0000","last_peered":"2024-03-04T06:17:21.328926+0000","last_clean":"2024-03-04T06:17:21.328926+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:17:21.328926+0000","last_undegraded":"2024-03-04T06:17:21.328926+0000","last_fullsized":"2024-03-04T06:17:21.328926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"557'527","reported_seq":2017,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.339869+0000","last_change":"2024-03-04T06:17:18.320820+0000","last_active":"2024-03-04T06:17:21.339869+0000","last_peered":"2024-03-04T06:17:21.339869+0000","last_clean":"2024-03-04T06:17:21.339869+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:17:21.339869+0000","last_undegraded":"2024-03-04T06:17:21.339869+0000","last_fullsized":"2024-03-04T06:17:21.339869+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":52,"num_object_clones":40,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":276,"num_read_kb":221534,"num_write":234,"num_write_kb":63712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1420,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.344082+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:17:20.344082+0000","last_peered":"2024-03-04T06:17:20.344082+0000","last_clean":"2024-03-04T06:17:20.344082+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:17:20.344082+0000","last_undegraded":"2024-03-04T06:17:20.344082+0000","last_fullsized":"2024-03-04T06:17:20.344082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"570'377","reported_seq":1914,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.320564+0000","last_change":"2024-03-04T06:17:17.323092+0000","last_active":"2024-03-04T06:17:21.320564+0000","last_peered":"2024-03-04T06:17:21.320564+0000","last_clean":"2024-03-04T06:17:21.320564+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:17:21.320564+0000","last_undegraded":"2024-03-04T06:17:21.320564+0000","last_fullsized":"2024-03-04T06:17:21.320564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370051,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":219,"num_read_kb":177501,"num_write":146,"num_write_kb":41225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.316794+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:17:21.316794+0000","last_peered":"2024-03-04T06:17:21.316794+0000","last_clean":"2024-03-04T06:17:21.316794+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:17:21.316794+0000","last_undegraded":"2024-03-04T06:17:21.316794+0000","last_fullsized":"2024-03-04T06:17:21.316794+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"568'923","reported_seq":2327,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.328518+0000","last_change":"2024-03-04T06:17:17.304752+0000","last_active":"2024-03-04T06:17:21.328518+0000","last_peered":"2024-03-04T06:17:21.328518+0000","last_clean":"2024-03-04T06:17:21.328518+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:17:21.328518+0000","last_undegraded":"2024-03-04T06:17:21.328518+0000","last_fullsized":"2024-03-04T06:17:21.328518+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":923,"log_dups_size":0,"ondisk_log_size":923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5630840,"num_objects":93,"num_object_clones":72,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":553,"num_read_kb":440379,"num_write":438,"num_write_kb":117199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1420,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.329171+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:17:20.329171+0000","last_peered":"2024-03-04T06:17:20.329171+0000","last_clean":"2024-03-04T06:17:20.329171+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:17:20.329171+0000","last_undegraded":"2024-03-04T06:17:20.329171+0000","last_fullsized":"2024-03-04T06:17:20.329171+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"568'1010","reported_seq":2307,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.298923+0000","last_change":"2024-03-04T06:17:18.458101+0000","last_active":"2024-03-04T06:17:20.298923+0000","last_peered":"2024-03-04T06:17:20.298923+0000","last_clean":"2024-03-04T06:17:20.298923+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:17:20.298923+0000","last_undegraded":"2024-03-04T06:17:20.298923+0000","last_fullsized":"2024-03-04T06:17:20.298923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1010,"log_dups_size":0,"ondisk_log_size":1010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16472048,"num_objects":97,"num_object_clones":75,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":447,"num_read_kb":328457,"num_write":448,"num_write_kb":118529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"559'468","reported_seq":2037,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.326062+0000","last_change":"2024-03-04T06:17:18.321202+0000","last_active":"2024-03-04T06:17:21.326062+0000","last_peered":"2024-03-04T06:17:21.326062+0000","last_clean":"2024-03-04T06:17:21.326062+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:17:21.326062+0000","last_undegraded":"2024-03-04T06:17:21.326062+0000","last_fullsized":"2024-03-04T06:17:21.326062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9311804,"num_objects":47,"num_object_clones":35,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":331,"num_read_kb":272228,"num_write":223,"num_write_kb":60598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1450,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.309012+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:17:20.309012+0000","last_peered":"2024-03-04T06:17:20.309012+0000","last_clean":"2024-03-04T06:17:20.309012+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:17:20.309012+0000","last_undegraded":"2024-03-04T06:17:20.309012+0000","last_fullsized":"2024-03-04T06:17:20.309012+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"557'449","reported_seq":2002,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.307732+0000","last_change":"2024-03-04T06:17:18.463468+0000","last_active":"2024-03-04T06:17:20.307732+0000","last_peered":"2024-03-04T06:17:20.307732+0000","last_clean":"2024-03-04T06:17:20.307732+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:17:20.307732+0000","last_undegraded":"2024-03-04T06:17:20.307732+0000","last_fullsized":"2024-03-04T06:17:20.307732+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143437,"num_objects":52,"num_object_clones":37,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":235,"num_read_kb":167519,"num_write":210,"num_write_kb":55510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.333768+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:17:21.333768+0000","last_peered":"2024-03-04T06:17:21.333768+0000","last_clean":"2024-03-04T06:17:21.333768+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:17:21.333768+0000","last_undegraded":"2024-03-04T06:17:21.333768+0000","last_fullsized":"2024-03-04T06:17:21.333768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"568'692","reported_seq":2110,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.323093+0000","last_change":"2024-03-04T06:17:17.321941+0000","last_active":"2024-03-04T06:17:21.323093+0000","last_peered":"2024-03-04T06:17:21.323093+0000","last_clean":"2024-03-04T06:17:21.323093+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:17:21.323093+0000","last_undegraded":"2024-03-04T06:17:21.323093+0000","last_fullsized":"2024-03-04T06:17:21.323093+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6723592,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":339,"num_read_kb":253707,"num_write":310,"num_write_kb":84843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1450,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.301276+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:17:20.301276+0000","last_peered":"2024-03-04T06:17:20.301276+0000","last_clean":"2024-03-04T06:17:20.301276+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:17:20.301276+0000","last_undegraded":"2024-03-04T06:17:20.301276+0000","last_fullsized":"2024-03-04T06:17:20.301276+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"569'803","reported_seq":2217,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.297716+0000","last_change":"2024-03-04T06:17:18.457995+0000","last_active":"2024-03-04T06:17:20.297716+0000","last_peered":"2024-03-04T06:17:20.297716+0000","last_clean":"2024-03-04T06:17:20.297716+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:17:20.297716+0000","last_undegraded":"2024-03-04T06:17:20.297716+0000","last_fullsized":"2024-03-04T06:17:20.297716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849331,"num_objects":73,"num_object_clones":64,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":450,"num_read_kb":373220,"num_write":336,"num_write_kb":92115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.330114+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:17:21.330114+0000","last_peered":"2024-03-04T06:17:21.330114+0000","last_clean":"2024-03-04T06:17:21.330114+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:17:21.330114+0000","last_undegraded":"2024-03-04T06:17:21.330114+0000","last_fullsized":"2024-03-04T06:17:21.330114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":97303379,"num_objects":806,"num_object_clones":626,"num_object_copies":2418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4288,"num_read_kb":3353233,"num_write":3632,"num_write_kb":986942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8035,"ondisk_log_size":8035,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166444,"kb_used_data":117412,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94205396,"statfs":{"total":96636764160,"available":96466325504,"internally_reserved":0,"allocated":120229888,"data_stored":133616821,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":141916,"kb_used_data":97044,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94229924,"statfs":{"total":96636764160,"available":96491442176,"internally_reserved":0,"allocated":99373056,"data_stored":109085667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706238,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":134732,"kb_used_data":89860,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94237108,"statfs":{"total":96636764160,"available":96498798592,"internally_reserved":0,"allocated":92016640,"data_stored":102692131,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706238,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":120092,"kb_used_data":76628,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94251748,"statfs":{"total":96636764160,"available":96513789952,"internally_reserved":0,"allocated":78467072,"data_stored":83541345,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:17:22.970 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:17:22.971 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:17:22.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:17:23.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:23.074+0000 7f87dc62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:23.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:23.080+0000 7f87dc62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:2342: done (5 left) 2024-03-04T06:17:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:2343: done (4 left) 2024-03-04T06:17:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:2344: done (3 left) 2024-03-04T06:17:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:2345: done (2 left) 2024-03-04T06:17:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:2346: done (1 left) 2024-03-04T06:17:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:2348: done (0 left) 2024-03-04T06:17:23.266 INFO:tasks.rados.rados.0.smithi064.stdout:2349: snap_create 2024-03-04T06:17:23.273 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:17:23.273 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:17:23.289 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":853,"stamp":"2024-03-04T06:17:23.264961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":97893747,"num_objects":808,"num_object_clones":626,"num_object_copies":2424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4391,"num_read_kb":3353324,"num_write":3769,"num_write_kb":988836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8132,"ondisk_log_size":8132,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":563184,"kb_used_data":380944,"kb_used_omap":0,"kb_used_meta":182208,"kb_avail":376924176,"statfs":{"total":386547056640,"available":385970356224,"internally_reserved":0,"allocated":390086656,"data_stored":428935964,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":186580992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4268846,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":50014,"num_write":23,"num_write_kb":4655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.987131"},"pg_stats":[{"pgid":"3.f","version":"568'495","reported_seq":2002,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.341573+0000","last_change":"2024-03-04T06:17:18.329967+0000","last_active":"2024-03-04T06:17:20.341573+0000","last_peered":"2024-03-04T06:17:20.341573+0000","last_clean":"2024-03-04T06:17:20.341573+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:17:20.341573+0000","last_undegraded":"2024-03-04T06:17:20.341573+0000","last_fullsized":"2024-03-04T06:17:20.341573+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151560,"num_objects":54,"num_object_clones":43,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":271,"num_read_kb":224365,"num_write":222,"num_write_kb":59334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"564'397","reported_seq":1926,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.292609+0000","last_change":"2024-03-04T06:17:18.457263+0000","last_active":"2024-03-04T06:17:20.292609+0000","last_peered":"2024-03-04T06:17:20.292609+0000","last_clean":"2024-03-04T06:17:20.292609+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:17:20.292609+0000","last_undegraded":"2024-03-04T06:17:20.292609+0000","last_fullsized":"2024-03-04T06:17:20.292609+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7479577,"num_objects":44,"num_object_clones":30,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":137,"num_read_kb":80772,"num_write":176,"num_write_kb":46528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"568'521","reported_seq":2064,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.309107+0000","last_change":"2024-03-04T06:17:18.308698+0000","last_active":"2024-03-04T06:17:21.309107+0000","last_peered":"2024-03-04T06:17:21.309107+0000","last_clean":"2024-03-04T06:17:21.309107+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:17:21.309107+0000","last_undegraded":"2024-03-04T06:17:21.309107+0000","last_fullsized":"2024-03-04T06:17:21.309107+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3849001,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":342,"num_read_kb":275867,"num_write":253,"num_write_kb":69591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"568'368","reported_seq":1932,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.329855+0000","last_change":"2024-03-04T06:17:18.316518+0000","last_active":"2024-03-04T06:17:20.329855+0000","last_peered":"2024-03-04T06:17:20.329855+0000","last_clean":"2024-03-04T06:17:20.329855+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:17:20.329855+0000","last_undegraded":"2024-03-04T06:17:20.329855+0000","last_fullsized":"2024-03-04T06:17:20.329855+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9362225,"num_objects":36,"num_object_clones":26,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":197,"num_read_kb":149129,"num_write":178,"num_write_kb":49693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"567'179","reported_seq":1781,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.334654+0000","last_change":"2024-03-04T06:17:17.324329+0000","last_active":"2024-03-04T06:17:21.334654+0000","last_peered":"2024-03-04T06:17:21.334654+0000","last_clean":"2024-03-04T06:17:21.334654+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:17:21.334654+0000","last_undegraded":"2024-03-04T06:17:21.334654+0000","last_fullsized":"2024-03-04T06:17:21.334654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":60,"num_read_kb":33097,"num_write":80,"num_write_kb":21063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"564'556","reported_seq":2022,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.344699+0000","last_change":"2024-03-04T06:17:18.329195+0000","last_active":"2024-03-04T06:17:20.344699+0000","last_peered":"2024-03-04T06:17:20.344699+0000","last_clean":"2024-03-04T06:17:20.344699+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:17:20.344699+0000","last_undegraded":"2024-03-04T06:17:20.344699+0000","last_fullsized":"2024-03-04T06:17:20.344699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8622717,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":255,"num_read_kb":192237,"num_write":255,"num_write_kb":74444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1682,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.338873+0000","last_change":"2024-03-04T06:17:18.329854+0000","last_active":"2024-03-04T06:17:20.338873+0000","last_peered":"2024-03-04T06:17:20.338873+0000","last_clean":"2024-03-04T06:17:20.338873+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:17:20.338873+0000","last_undegraded":"2024-03-04T06:17:20.338873+0000","last_fullsized":"2024-03-04T06:17:20.338873+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"557'270","reported_seq":1875,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.318457+0000","last_change":"2024-03-04T06:17:18.309259+0000","last_active":"2024-03-04T06:17:21.318457+0000","last_peered":"2024-03-04T06:17:21.318457+0000","last_clean":"2024-03-04T06:17:21.318457+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:17:21.318457+0000","last_undegraded":"2024-03-04T06:17:21.318457+0000","last_fullsized":"2024-03-04T06:17:21.318457+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6437477,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":176,"num_read_kb":163221,"num_write":123,"num_write_kb":32558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1556,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.321849+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:17:21.321849+0000","last_peered":"2024-03-04T06:17:21.321849+0000","last_clean":"2024-03-04T06:17:21.321849+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:17:21.321849+0000","last_undegraded":"2024-03-04T06:17:21.321849+0000","last_fullsized":"2024-03-04T06:17:21.321849+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.328926+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:17:21.328926+0000","last_peered":"2024-03-04T06:17:21.328926+0000","last_clean":"2024-03-04T06:17:21.328926+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:17:21.328926+0000","last_undegraded":"2024-03-04T06:17:21.328926+0000","last_fullsized":"2024-03-04T06:17:21.328926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"557'527","reported_seq":2017,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.339869+0000","last_change":"2024-03-04T06:17:18.320820+0000","last_active":"2024-03-04T06:17:21.339869+0000","last_peered":"2024-03-04T06:17:21.339869+0000","last_clean":"2024-03-04T06:17:21.339869+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:17:21.339869+0000","last_undegraded":"2024-03-04T06:17:21.339869+0000","last_fullsized":"2024-03-04T06:17:21.339869+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899719,"num_objects":52,"num_object_clones":40,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":276,"num_read_kb":221534,"num_write":234,"num_write_kb":63712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1420,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.344082+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:17:20.344082+0000","last_peered":"2024-03-04T06:17:20.344082+0000","last_clean":"2024-03-04T06:17:20.344082+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:17:20.344082+0000","last_undegraded":"2024-03-04T06:17:20.344082+0000","last_fullsized":"2024-03-04T06:17:20.344082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"570'377","reported_seq":1914,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.320564+0000","last_change":"2024-03-04T06:17:17.323092+0000","last_active":"2024-03-04T06:17:21.320564+0000","last_peered":"2024-03-04T06:17:21.320564+0000","last_clean":"2024-03-04T06:17:21.320564+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:17:21.320564+0000","last_undegraded":"2024-03-04T06:17:21.320564+0000","last_fullsized":"2024-03-04T06:17:21.320564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370051,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":219,"num_read_kb":177501,"num_write":146,"num_write_kb":41225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.316794+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:17:21.316794+0000","last_peered":"2024-03-04T06:17:21.316794+0000","last_clean":"2024-03-04T06:17:21.316794+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:17:21.316794+0000","last_undegraded":"2024-03-04T06:17:21.316794+0000","last_fullsized":"2024-03-04T06:17:21.316794+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"568'923","reported_seq":2327,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.328518+0000","last_change":"2024-03-04T06:17:17.304752+0000","last_active":"2024-03-04T06:17:21.328518+0000","last_peered":"2024-03-04T06:17:21.328518+0000","last_clean":"2024-03-04T06:17:21.328518+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:17:21.328518+0000","last_undegraded":"2024-03-04T06:17:21.328518+0000","last_fullsized":"2024-03-04T06:17:21.328518+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":923,"log_dups_size":0,"ondisk_log_size":923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5630840,"num_objects":93,"num_object_clones":72,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":553,"num_read_kb":440379,"num_write":438,"num_write_kb":117199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1420,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.329171+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:17:20.329171+0000","last_peered":"2024-03-04T06:17:20.329171+0000","last_clean":"2024-03-04T06:17:20.329171+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:17:20.329171+0000","last_undegraded":"2024-03-04T06:17:20.329171+0000","last_fullsized":"2024-03-04T06:17:20.329171+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"568'1010","reported_seq":2307,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.298923+0000","last_change":"2024-03-04T06:17:18.458101+0000","last_active":"2024-03-04T06:17:20.298923+0000","last_peered":"2024-03-04T06:17:20.298923+0000","last_clean":"2024-03-04T06:17:20.298923+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:17:20.298923+0000","last_undegraded":"2024-03-04T06:17:20.298923+0000","last_fullsized":"2024-03-04T06:17:20.298923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1010,"log_dups_size":0,"ondisk_log_size":1010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16472048,"num_objects":97,"num_object_clones":75,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":447,"num_read_kb":328457,"num_write":448,"num_write_kb":118529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"559'468","reported_seq":2037,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.326062+0000","last_change":"2024-03-04T06:17:18.321202+0000","last_active":"2024-03-04T06:17:21.326062+0000","last_peered":"2024-03-04T06:17:21.326062+0000","last_clean":"2024-03-04T06:17:21.326062+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:17:21.326062+0000","last_undegraded":"2024-03-04T06:17:21.326062+0000","last_fullsized":"2024-03-04T06:17:21.326062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9311804,"num_objects":47,"num_object_clones":35,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":331,"num_read_kb":272228,"num_write":223,"num_write_kb":60598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1450,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.309012+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:17:20.309012+0000","last_peered":"2024-03-04T06:17:20.309012+0000","last_clean":"2024-03-04T06:17:20.309012+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:17:20.309012+0000","last_undegraded":"2024-03-04T06:17:20.309012+0000","last_fullsized":"2024-03-04T06:17:20.309012+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"557'449","reported_seq":2002,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.307732+0000","last_change":"2024-03-04T06:17:18.463468+0000","last_active":"2024-03-04T06:17:20.307732+0000","last_peered":"2024-03-04T06:17:20.307732+0000","last_clean":"2024-03-04T06:17:20.307732+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:17:20.307732+0000","last_undegraded":"2024-03-04T06:17:20.307732+0000","last_fullsized":"2024-03-04T06:17:20.307732+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143437,"num_objects":52,"num_object_clones":37,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":235,"num_read_kb":167519,"num_write":210,"num_write_kb":55510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.333768+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:17:21.333768+0000","last_peered":"2024-03-04T06:17:21.333768+0000","last_clean":"2024-03-04T06:17:21.333768+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:17:21.333768+0000","last_undegraded":"2024-03-04T06:17:21.333768+0000","last_fullsized":"2024-03-04T06:17:21.333768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"568'692","reported_seq":2110,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.323093+0000","last_change":"2024-03-04T06:17:17.321941+0000","last_active":"2024-03-04T06:17:21.323093+0000","last_peered":"2024-03-04T06:17:21.323093+0000","last_clean":"2024-03-04T06:17:21.323093+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:17:21.323093+0000","last_undegraded":"2024-03-04T06:17:21.323093+0000","last_fullsized":"2024-03-04T06:17:21.323093+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6723592,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":339,"num_read_kb":253707,"num_write":310,"num_write_kb":84843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1450,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.301276+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:17:20.301276+0000","last_peered":"2024-03-04T06:17:20.301276+0000","last_clean":"2024-03-04T06:17:20.301276+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:17:20.301276+0000","last_undegraded":"2024-03-04T06:17:20.301276+0000","last_fullsized":"2024-03-04T06:17:20.301276+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"569'803","reported_seq":2217,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-04T06:17:20.297716+0000","last_change":"2024-03-04T06:17:18.457995+0000","last_active":"2024-03-04T06:17:20.297716+0000","last_peered":"2024-03-04T06:17:20.297716+0000","last_clean":"2024-03-04T06:17:20.297716+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:17:20.297716+0000","last_undegraded":"2024-03-04T06:17:20.297716+0000","last_fullsized":"2024-03-04T06:17:20.297716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849331,"num_objects":73,"num_object_clones":64,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":450,"num_read_kb":373220,"num_write":336,"num_write_kb":92115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1423,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-03-04T06:17:21.330114+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:17:21.330114+0000","last_peered":"2024-03-04T06:17:21.330114+0000","last_clean":"2024-03-04T06:17:21.330114+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:17:21.330114+0000","last_undegraded":"2024-03-04T06:17:21.330114+0000","last_fullsized":"2024-03-04T06:17:21.330114+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":97303379,"num_objects":806,"num_object_clones":626,"num_object_copies":2418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4288,"num_read_kb":3353233,"num_write":3632,"num_write_kb":986942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8035,"ondisk_log_size":8035,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166444,"kb_used_data":117412,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94205396,"statfs":{"total":96636764160,"available":96466325504,"internally_reserved":0,"allocated":120229888,"data_stored":133616821,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706239,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":141916,"kb_used_data":97044,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94229924,"statfs":{"total":96636764160,"available":96491442176,"internally_reserved":0,"allocated":99373056,"data_stored":109085667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706238,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":134732,"kb_used_data":89860,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94237108,"statfs":{"total":96636764160,"available":96498798592,"internally_reserved":0,"allocated":92016640,"data_stored":102692131,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706238,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":120092,"kb_used_data":76628,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94251748,"statfs":{"total":96636764160,"available":96513789952,"internally_reserved":0,"allocated":78467072,"data_stored":83541345,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:17:23.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:23.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:23.510+0000 7fdf9091d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:23.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:23.514+0000 7fdf9091d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:24.267 INFO:tasks.rados.rados.0.smithi064.stdout:2349: done (0 left) 2024-03-04T06:17:24.267 INFO:tasks.rados.rados.0.smithi064.stdout:2350: snap_create 2024-03-04T06:17:25.268 INFO:tasks.rados.rados.0.smithi064.stdout:2350: done (0 left) 2024-03-04T06:17:25.269 INFO:tasks.rados.rados.0.smithi064.stdout:2351: read oid 47 snap -1 2024-03-04T06:17:25.269 INFO:tasks.rados.rados.0.smithi064.stdout:2351: expect (ObjNum 1092 snap 279 seq_num 1092) 2024-03-04T06:17:25.269 INFO:tasks.rados.rados.0.smithi064.stdout:2352: write oid 31 current snap is 284 2024-03-04T06:17:25.269 INFO:tasks.rados.rados.0.smithi064.stdout:2352: seq_num 1095 ranges {403665=752487,1610018=514716} 2024-03-04T06:17:25.274 INFO:tasks.rados.rados.0.smithi064.stdout:2352: writing smithi06431743-31 from 403665 to 1156152 tid 1 2024-03-04T06:17:25.280 INFO:tasks.rados.rados.0.smithi064.stdout:2352: writing smithi06431743-31 from 1610018 to 2124734 tid 2 2024-03-04T06:17:25.280 INFO:tasks.rados.rados.0.smithi064.stdout:2353: snap_create 2024-03-04T06:17:25.328 INFO:tasks.rados.rados.0.smithi064.stdout:2351: expect (ObjNum 1092 snap 279 seq_num 1092) 2024-03-04T06:17:25.418 INFO:tasks.rados.rados.0.smithi064.stdout:2352: finishing write tid 1 to smithi06431743-31 2024-03-04T06:17:25.418 INFO:tasks.rados.rados.0.smithi064.stdout:2352: finishing write tid 2 to smithi06431743-31 2024-03-04T06:17:25.418 INFO:tasks.rados.rados.0.smithi064.stdout:2352: finishing write tid 3 to smithi06431743-31 2024-03-04T06:17:25.418 INFO:tasks.rados.rados.0.smithi064.stdout:2352: finishing write tid 4 to smithi06431743-31 2024-03-04T06:17:25.418 INFO:tasks.rados.rados.0.smithi064.stdout:2352: oid 31 updating version 0 to 182 2024-03-04T06:17:25.418 INFO:tasks.rados.rados.0.smithi064.stdout:2352: oid 31 updating version 182 to 183 2024-03-04T06:17:25.418 INFO:tasks.rados.rados.0.smithi064.stdout:2352: oid 31 updating version 183 to 184 2024-03-04T06:17:25.418 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 184 (ObjNum 1095 snap 284 seq_num 1095) dirty exists 2024-03-04T06:17:25.418 INFO:tasks.rados.rados.0.smithi064.stdout:2352: left oid 31 (ObjNum 1095 snap 284 seq_num 1095) 2024-03-04T06:17:25.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:25.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:25.688+0000 7f9093e0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:25.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:25.693+0000 7f9093e0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:26.271 INFO:tasks.rados.rados.0.smithi064.stdout:2351: done (2 left) 2024-03-04T06:17:26.271 INFO:tasks.rados.rados.0.smithi064.stdout:2352: done (1 left) 2024-03-04T06:17:26.271 INFO:tasks.rados.rados.0.smithi064.stdout:2353: done (0 left) 2024-03-04T06:17:26.272 INFO:tasks.rados.rados.0.smithi064.stdout:2354: snap_create 2024-03-04T06:17:27.276 INFO:tasks.rados.rados.0.smithi064.stdout:2354: done (0 left) 2024-03-04T06:17:27.276 INFO:tasks.rados.rados.0.smithi064.stdout:2355: write (excl) oid 6 current snap is 286 2024-03-04T06:17:27.276 INFO:tasks.rados.rados.0.smithi064.stdout:2355: seq_num 1096 ranges {420472=427736,1268500=658374,2347953=72519} 2024-03-04T06:17:27.280 INFO:tasks.rados.rados.0.smithi064.stdout:2355: writing smithi06431743-6 from 420472 to 848208 tid 1 2024-03-04T06:17:27.287 INFO:tasks.rados.rados.0.smithi064.stdout:2355: writing smithi06431743-6 from 1268500 to 1926874 tid 2 2024-03-04T06:17:27.291 INFO:tasks.rados.rados.0.smithi064.stdout:2355: writing smithi06431743-6 from 2347953 to 2420472 tid 3 2024-03-04T06:17:27.291 INFO:tasks.rados.rados.0.smithi064.stdout:2356: write oid 16 current snap is 286 2024-03-04T06:17:27.291 INFO:tasks.rados.rados.0.smithi064.stdout:2356: seq_num 1097 ranges {437279=502985,1610629=518385} 2024-03-04T06:17:27.297 INFO:tasks.rados.rados.0.smithi064.stdout:2356: writing smithi06431743-16 from 437279 to 940264 tid 1 2024-03-04T06:17:27.305 INFO:tasks.rados.rados.0.smithi064.stdout:2356: writing smithi06431743-16 from 1610629 to 2129014 tid 2 2024-03-04T06:17:27.305 INFO:tasks.rados.rados.0.smithi064.stdout:2357: write oid 29 current snap is 286 2024-03-04T06:17:27.305 INFO:tasks.rados.rados.0.smithi064.stdout:2357: seq_num 1098 ranges {454086=578234,1669111=662043} 2024-03-04T06:17:27.311 INFO:tasks.rados.rados.0.smithi064.stdout:2357: writing smithi06431743-29 from 454086 to 1032320 tid 1 2024-03-04T06:17:27.319 INFO:tasks.rados.rados.0.smithi064.stdout:2357: writing smithi06431743-29 from 1669111 to 2331154 tid 2 2024-03-04T06:17:27.319 INFO:tasks.rados.rados.0.smithi064.stdout:2358: read oid 27 snap -1 2024-03-04T06:17:27.319 INFO:tasks.rados.rados.0.smithi064.stdout:2358: expect deleted 2024-03-04T06:17:27.319 INFO:tasks.rados.rados.0.smithi064.stdout:2359: snap_create 2024-03-04T06:17:27.352 INFO:tasks.rados.rados.0.smithi064.stdout:2355: finishing write tid 1 to smithi06431743-6 2024-03-04T06:17:27.369 INFO:tasks.rados.rados.0.smithi064.stdout:2356: finishing write tid 1 to smithi06431743-16 2024-03-04T06:17:27.387 INFO:tasks.rados.rados.0.smithi064.stdout:2355: finishing write tid 2 to smithi06431743-6 2024-03-04T06:17:27.388 INFO:tasks.rados.rados.0.smithi064.stdout:2357: finishing write tid 1 to smithi06431743-29 2024-03-04T06:17:27.390 INFO:tasks.rados.rados.0.smithi064.stdout:2356: finishing write tid 2 to smithi06431743-16 2024-03-04T06:17:27.407 INFO:tasks.rados.rados.0.smithi064.stdout:2355: finishing write tid 3 to smithi06431743-6 2024-03-04T06:17:27.407 INFO:tasks.rados.rados.0.smithi064.stdout:2357: finishing write tid 2 to smithi06431743-29 2024-03-04T06:17:27.414 INFO:tasks.rados.rados.0.smithi064.stdout:2356: finishing write tid 3 to smithi06431743-16 2024-03-04T06:17:27.416 INFO:tasks.rados.rados.0.smithi064.stdout:2356: finishing write tid 4 to smithi06431743-16 2024-03-04T06:17:27.416 INFO:tasks.rados.rados.0.smithi064.stdout:2356: oid 16 updating version 0 to 459 2024-03-04T06:17:27.416 INFO:tasks.rados.rados.0.smithi064.stdout:2356: oid 16 version 459 is already newer than 458 2024-03-04T06:17:27.416 INFO:tasks.rados.rados.0.smithi064.stdout:2356: oid 16 version 459 is already newer than 458 2024-03-04T06:17:27.416 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 459 (ObjNum 1097 snap 286 seq_num 1097) dirty exists 2024-03-04T06:17:27.416 INFO:tasks.rados.rados.0.smithi064.stdout:2356: left oid 16 (ObjNum 1097 snap 286 seq_num 1097) 2024-03-04T06:17:27.417 INFO:tasks.rados.rados.0.smithi064.stdout:2355: finishing write tid 4 to smithi06431743-6 2024-03-04T06:17:27.418 INFO:tasks.rados.rados.0.smithi064.stdout:2355: finishing write tid 5 to smithi06431743-6 2024-03-04T06:17:27.418 INFO:tasks.rados.rados.0.smithi064.stdout:2355: oid 6 updating version 0 to 472 2024-03-04T06:17:27.418 INFO:tasks.rados.rados.0.smithi064.stdout:2355: oid 6 version 472 is already newer than 470 2024-03-04T06:17:27.418 INFO:tasks.rados.rados.0.smithi064.stdout:2355: oid 6 updating version 472 to 473 2024-03-04T06:17:27.418 INFO:tasks.rados.rados.0.smithi064.stdout:2355: oid 6 version 473 is already newer than 471 2024-03-04T06:17:27.418 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 473 (ObjNum 1096 snap 286 seq_num 1096) dirty exists 2024-03-04T06:17:27.418 INFO:tasks.rados.rados.0.smithi064.stdout:2355: left oid 6 (ObjNum 1096 snap 286 seq_num 1096) 2024-03-04T06:17:27.420 INFO:tasks.rados.rados.0.smithi064.stdout:2357: finishing write tid 3 to smithi06431743-29 2024-03-04T06:17:27.420 INFO:tasks.rados.rados.0.smithi064.stdout:2357: finishing write tid 4 to smithi06431743-29 2024-03-04T06:17:27.421 INFO:tasks.rados.rados.0.smithi064.stdout:2357: oid 29 updating version 0 to 1016 2024-03-04T06:17:27.421 INFO:tasks.rados.rados.0.smithi064.stdout:2357: oid 29 version 1016 is already newer than 1014 2024-03-04T06:17:27.421 INFO:tasks.rados.rados.0.smithi064.stdout:2357: oid 29 version 1016 is already newer than 1015 2024-03-04T06:17:27.421 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 1016 (ObjNum 1098 snap 286 seq_num 1098) dirty exists 2024-03-04T06:17:27.421 INFO:tasks.rados.rados.0.smithi064.stdout:2357: left oid 29 (ObjNum 1098 snap 286 seq_num 1098) 2024-03-04T06:17:27.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:27.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:27.870+0000 7f1fc546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:27.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:27.874+0000 7f1fc546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:28.280 INFO:tasks.rados.rados.0.smithi064.stdout:2355: done (4 left) 2024-03-04T06:17:28.280 INFO:tasks.rados.rados.0.smithi064.stdout:2356: done (3 left) 2024-03-04T06:17:28.280 INFO:tasks.rados.rados.0.smithi064.stdout:2357: done (2 left) 2024-03-04T06:17:28.280 INFO:tasks.rados.rados.0.smithi064.stdout:2358: done (1 left) 2024-03-04T06:17:28.280 INFO:tasks.rados.rados.0.smithi064.stdout:2359: done (0 left) 2024-03-04T06:17:28.280 INFO:tasks.rados.rados.0.smithi064.stdout:2360: delete oid 46 current snap is 287 2024-03-04T06:17:28.290 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:17:28.290 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:17:28.290 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:17:28.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:17:28.321 INFO:tasks.rados.rados.0.smithi064.stdout:2360: done (0 left) 2024-03-04T06:17:28.321 INFO:tasks.rados.rados.0.smithi064.stdout:2361: delete oid 25 current snap is 287 2024-03-04T06:17:28.343 INFO:tasks.rados.rados.0.smithi064.stdout:2361: done (0 left) 2024-03-04T06:17:28.343 INFO:tasks.rados.rados.0.smithi064.stdout:2362: write (excl) oid 3 current snap is 287 2024-03-04T06:17:28.343 INFO:tasks.rados.rados.0.smithi064.stdout:2362: seq_num 1101 ranges {504507=403981,1328204=409370,2252362=252145} 2024-03-04T06:17:28.348 INFO:tasks.rados.rados.0.smithi064.stdout:2362: writing smithi06431743-3 from 504507 to 908488 tid 1 2024-03-04T06:17:28.353 INFO:tasks.rados.rados.0.smithi064.stdout:2362: writing smithi06431743-3 from 1328204 to 1737574 tid 2 2024-03-04T06:17:28.358 INFO:tasks.rados.rados.0.smithi064.stdout:2362: writing smithi06431743-3 from 2252362 to 2504507 tid 3 2024-03-04T06:17:28.358 INFO:tasks.rados.rados.0.smithi064.stdout:2363: read oid 39 snap -1 2024-03-04T06:17:28.358 INFO:tasks.rados.rados.0.smithi064.stdout:2363: expect deleted 2024-03-04T06:17:28.359 INFO:tasks.rados.rados.0.smithi064.stdout:2364: read oid 27 snap -1 2024-03-04T06:17:28.359 INFO:tasks.rados.rados.0.smithi064.stdout:2364: expect deleted 2024-03-04T06:17:28.359 INFO:tasks.rados.rados.0.smithi064.stdout:2365: snap_create 2024-03-04T06:17:28.380 INFO:tasks.rados.rados.0.smithi064.stdout:2362: finishing write tid 1 to smithi06431743-3 2024-03-04T06:17:28.392 INFO:tasks.rados.rados.0.smithi064.stdout:2362: finishing write tid 2 to smithi06431743-3 2024-03-04T06:17:28.400 INFO:tasks.rados.rados.0.smithi064.stdout:2362: finishing write tid 3 to smithi06431743-3 2024-03-04T06:17:28.413 INFO:tasks.rados.rados.0.smithi064.stdout:2362: finishing write tid 4 to smithi06431743-3 2024-03-04T06:17:28.414 INFO:tasks.rados.rados.0.smithi064.stdout:2362: finishing write tid 5 to smithi06431743-3 2024-03-04T06:17:28.414 INFO:tasks.rados.rados.0.smithi064.stdout:2362: oid 3 updating version 0 to 529 2024-03-04T06:17:28.414 INFO:tasks.rados.rados.0.smithi064.stdout:2362: oid 3 updating version 529 to 532 2024-03-04T06:17:28.415 INFO:tasks.rados.rados.0.smithi064.stdout:2362: oid 3 version 532 is already newer than 530 2024-03-04T06:17:28.415 INFO:tasks.rados.rados.0.smithi064.stdout:2362: oid 3 version 532 is already newer than 531 2024-03-04T06:17:28.415 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 532 (ObjNum 1101 snap 287 seq_num 1101) dirty exists 2024-03-04T06:17:28.415 INFO:tasks.rados.rados.0.smithi064.stdout:2362: left oid 3 (ObjNum 1101 snap 287 seq_num 1101) 2024-03-04T06:17:28.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:28.411+0000 7f6ad3b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:28.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:28.418+0000 7f6ad3b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:28.431 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:28.434 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:28.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:17:28.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:28.577+0000 7f3ace064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:28.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:28.584+0000 7f3ace064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:28.597 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:28.597 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:28.597 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:28.597 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:28.597 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:28.598 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:28.598 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:28.598 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:28.598 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:28.598 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:28.598 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:28.598 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:28.600 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:28.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:17:28.708 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:28.704+0000 7f77db8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:28.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:28.711+0000 7f77db8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:28.724 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:28.724 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:28.724 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:28.724 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:28.724 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:28.724 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:28.724 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:28.724 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:28.724 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:28.725 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:28.725 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:28.725 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:28.727 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:28.727 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:28.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:17:28.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:28.830+0000 7f1e14310640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:28.838 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:28.838+0000 7f1e14310640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:28.851 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:28.851 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:28.851 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:28.851 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:28.852 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:28.852 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:28.852 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:28.852 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:28.852 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:28.852 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:28.852 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:28.852 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:28.854 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:29.284 INFO:tasks.rados.rados.0.smithi064.stdout:2362: done (3 left) 2024-03-04T06:17:29.284 INFO:tasks.rados.rados.0.smithi064.stdout:2363: done (2 left) 2024-03-04T06:17:29.284 INFO:tasks.rados.rados.0.smithi064.stdout:2364: done (1 left) 2024-03-04T06:17:29.284 INFO:tasks.rados.rados.0.smithi064.stdout:2365: done (0 left) 2024-03-04T06:17:29.285 INFO:tasks.rados.rados.0.smithi064.stdout:2366: read oid 31 snap -1 2024-03-04T06:17:29.285 INFO:tasks.rados.rados.0.smithi064.stdout:2366: expect (ObjNum 1095 snap 284 seq_num 1095) 2024-03-04T06:17:29.285 INFO:tasks.rados.rados.0.smithi064.stdout:2367: read oid 50 snap 281 2024-03-04T06:17:29.285 INFO:tasks.rados.rados.0.smithi064.stdout:2367: expect (ObjNum 1091 snap 279 seq_num 1091) 2024-03-04T06:17:29.285 INFO:tasks.rados.rados.0.smithi064.stdout:2368: write (excl) oid 27 current snap is 288 2024-03-04T06:17:29.285 INFO:tasks.rados.rados.0.smithi064.stdout:2368: seq_num 1102 ranges {521314=479230,1786686=669381} 2024-03-04T06:17:29.289 INFO:tasks.rados.rados.0.smithi064.stdout:2368: writing smithi06431743-27 from 521314 to 1000544 tid 1 2024-03-04T06:17:29.297 INFO:tasks.rados.rados.0.smithi064.stdout:2368: writing smithi06431743-27 from 1786686 to 2456067 tid 2 2024-03-04T06:17:29.297 INFO:tasks.rados.rados.0.smithi064.stdout:2369: write oid 3 current snap is 288 2024-03-04T06:17:29.297 INFO:tasks.rados.rados.0.smithi064.stdout:2369: seq_num 1103 ranges {538121=670832,1961521=413039} 2024-03-04T06:17:29.303 INFO:tasks.rados.rados.0.smithi064.stdout:2369: writing smithi06431743-3 from 538121 to 1208953 tid 1 2024-03-04T06:17:29.310 INFO:tasks.rados.rados.0.smithi064.stdout:2369: writing smithi06431743-3 from 1961521 to 2374560 tid 2 2024-03-04T06:17:29.310 INFO:tasks.rados.rados.0.smithi064.stdout:2370: write oid 10 current snap is 288 2024-03-04T06:17:29.310 INFO:tasks.rados.rados.0.smithi064.stdout:2370: seq_num 1104 ranges {554928=746081,2020003=534925} 2024-03-04T06:17:29.319 INFO:tasks.rados.rados.0.smithi064.stdout:2370: writing smithi06431743-10 from 554928 to 1301009 tid 1 2024-03-04T06:17:29.327 INFO:tasks.rados.rados.0.smithi064.stdout:2370: writing smithi06431743-10 from 2020003 to 2554928 tid 2 2024-03-04T06:17:29.328 INFO:tasks.rados.rados.0.smithi064.stdout:2371: write (excl) oid 34 current snap is 288 2024-03-04T06:17:29.328 INFO:tasks.rados.rados.0.smithi064.stdout:2371: seq_num 1105 ranges {571735=421330,1562132=416708} 2024-03-04T06:17:29.333 INFO:tasks.rados.rados.0.smithi064.stdout:2371: writing smithi06431743-34 from 571735 to 993065 tid 1 2024-03-04T06:17:29.339 INFO:tasks.rados.rados.0.smithi064.stdout:2371: writing smithi06431743-34 from 1562132 to 1978840 tid 2 2024-03-04T06:17:29.339 INFO:tasks.rados.rados.0.smithi064.stdout:2372: snap_create 2024-03-04T06:17:29.342 INFO:tasks.rados.rados.0.smithi064.stdout:2367: expect (ObjNum 1091 snap 279 seq_num 1091) 2024-03-04T06:17:29.531 INFO:tasks.rados.rados.0.smithi064.stdout:2368: finishing write tid 1 to smithi06431743-27 2024-03-04T06:17:29.531 INFO:tasks.rados.rados.0.smithi064.stdout:2369: finishing write tid 1 to smithi06431743-3 2024-03-04T06:17:29.531 INFO:tasks.rados.rados.0.smithi064.stdout:2366: expect (ObjNum 1095 snap 284 seq_num 1095) 2024-03-04T06:17:29.619 INFO:tasks.rados.rados.0.smithi064.stdout:2371: finishing write tid 1 to smithi06431743-34 2024-03-04T06:17:29.619 INFO:tasks.rados.rados.0.smithi064.stdout:2368: finishing write tid 2 to smithi06431743-27 2024-03-04T06:17:29.619 INFO:tasks.rados.rados.0.smithi064.stdout:2371: finishing write tid 2 to smithi06431743-34 2024-03-04T06:17:29.619 INFO:tasks.rados.rados.0.smithi064.stdout:2368: finishing write tid 3 to smithi06431743-27 2024-03-04T06:17:29.619 INFO:tasks.rados.rados.0.smithi064.stdout:2371: finishing write tid 3 to smithi06431743-34 2024-03-04T06:17:29.619 INFO:tasks.rados.rados.0.smithi064.stdout:2371: finishing write tid 4 to smithi06431743-34 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:2371: oid 34 updating version 0 to 926 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:2371: oid 34 updating version 926 to 928 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:2371: oid 34 version 928 is already newer than 928 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 928 (ObjNum 1105 snap 288 seq_num 1105) dirty exists 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:2371: left oid 34 (ObjNum 1105 snap 288 seq_num 1105) 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:2368: finishing write tid 4 to smithi06431743-27 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:2368: oid 27 updating version 0 to 398 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:2368: oid 27 updating version 398 to 400 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:2368: oid 27 version 400 is already newer than 399 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 400 (ObjNum 1102 snap 288 seq_num 1102) dirty exists 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:2368: left oid 27 (ObjNum 1102 snap 288 seq_num 1102) 2024-03-04T06:17:29.620 INFO:tasks.rados.rados.0.smithi064.stdout:2369: finishing write tid 2 to smithi06431743-3 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2369: finishing write tid 3 to smithi06431743-3 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2369: finishing write tid 4 to smithi06431743-3 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2369: oid 3 updating version 0 to 535 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2369: oid 3 version 535 is already newer than 534 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2369: oid 3 updating version 535 to 536 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 536 (ObjNum 1103 snap 288 seq_num 1103) dirty exists 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2369: left oid 3 (ObjNum 1103 snap 288 seq_num 1103) 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2370: finishing write tid 1 to smithi06431743-10 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2370: finishing write tid 2 to smithi06431743-10 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2370: finishing write tid 3 to smithi06431743-10 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2370: finishing write tid 4 to smithi06431743-10 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2370: oid 10 updating version 0 to 560 2024-03-04T06:17:29.621 INFO:tasks.rados.rados.0.smithi064.stdout:2370: oid 10 version 560 is already newer than 558 2024-03-04T06:17:29.622 INFO:tasks.rados.rados.0.smithi064.stdout:2370: oid 10 version 560 is already newer than 560 2024-03-04T06:17:29.622 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 560 (ObjNum 1104 snap 288 seq_num 1104) dirty exists 2024-03-04T06:17:29.622 INFO:tasks.rados.rados.0.smithi064.stdout:2370: left oid 10 (ObjNum 1104 snap 288 seq_num 1104) 2024-03-04T06:17:29.944 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:30.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:30.052+0000 7f623939d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:30.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:30.056+0000 7f623939d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:30.285 INFO:tasks.rados.rados.0.smithi064.stdout:2366: done (6 left) 2024-03-04T06:17:30.285 INFO:tasks.rados.rados.0.smithi064.stdout:2367: done (5 left) 2024-03-04T06:17:30.285 INFO:tasks.rados.rados.0.smithi064.stdout:2368: done (4 left) 2024-03-04T06:17:30.285 INFO:tasks.rados.rados.0.smithi064.stdout:2369: done (3 left) 2024-03-04T06:17:30.285 INFO:tasks.rados.rados.0.smithi064.stdout:2370: done (2 left) 2024-03-04T06:17:30.285 INFO:tasks.rados.rados.0.smithi064.stdout:2371: done (1 left) 2024-03-04T06:17:30.285 INFO:tasks.rados.rados.0.smithi064.stdout:2372: done (0 left) 2024-03-04T06:17:30.285 INFO:tasks.rados.rados.0.smithi064.stdout:2373: read oid 45 snap -1 2024-03-04T06:17:30.286 INFO:tasks.rados.rados.0.smithi064.stdout:2373: expect deleted 2024-03-04T06:17:30.286 INFO:tasks.rados.rados.0.smithi064.stdout:2374: read oid 44 snap 284 2024-03-04T06:17:30.286 INFO:tasks.rados.rados.0.smithi064.stdout:2374: expect (ObjNum 993 snap 254 seq_num 993) 2024-03-04T06:17:30.286 INFO:tasks.rados.rados.0.smithi064.stdout:2375: read oid 12 snap 284 2024-03-04T06:17:30.286 INFO:tasks.rados.rados.0.smithi064.stdout:2375: expect (ObjNum 1055 snap 270 seq_num 1055) 2024-03-04T06:17:30.286 INFO:tasks.rados.rados.0.smithi064.stdout:2376: snap_remove snap 258 2024-03-04T06:17:31.292 INFO:tasks.rados.rados.0.smithi064.stdout:2376: done (3 left) 2024-03-04T06:17:31.292 INFO:tasks.rados.rados.0.smithi064.stdout:2377: read oid 3 snap -1 2024-03-04T06:17:31.292 INFO:tasks.rados.rados.0.smithi064.stdout:2377: expect (ObjNum 1103 snap 288 seq_num 1103) 2024-03-04T06:17:31.292 INFO:tasks.rados.rados.0.smithi064.stdout:2378: write (excl) oid 24 current snap is 289 2024-03-04T06:17:31.292 INFO:tasks.rados.rados.0.smithi064.stdout:2378: seq_num 1106 ranges {588542=496579,1620614=676719} 2024-03-04T06:17:31.297 INFO:tasks.rados.rados.0.smithi064.stdout:2378: writing smithi06431743-24 from 588542 to 1085121 tid 1 2024-03-04T06:17:31.305 INFO:tasks.rados.rados.0.smithi064.stdout:2378: writing smithi06431743-24 from 1620614 to 2297333 tid 2 2024-03-04T06:17:31.305 INFO:tasks.rados.rados.0.smithi064.stdout:2379: read oid 26 snap 274 2024-03-04T06:17:31.305 INFO:tasks.rados.rados.0.smithi064.stdout:2379: expect (ObjNum 1074 snap 272 seq_num 1074) 2024-03-04T06:17:31.305 INFO:tasks.rados.rados.0.smithi064.stdout:2380: write oid 2 current snap is 289 2024-03-04T06:17:31.305 INFO:tasks.rados.rados.0.smithi064.stdout:2380: seq_num 1107 ranges {605349=571828,1679096=420377} 2024-03-04T06:17:31.310 INFO:tasks.rados.rados.0.smithi064.stdout:2380: writing smithi06431743-2 from 605349 to 1177177 tid 1 2024-03-04T06:17:31.316 INFO:tasks.rados.rados.0.smithi064.stdout:2380: writing smithi06431743-2 from 1679096 to 2099473 tid 2 2024-03-04T06:17:31.316 INFO:tasks.rados.rados.0.smithi064.stdout:2381: snap_create 2024-03-04T06:17:31.316 INFO:tasks.rados.rados.0.smithi064.stdout:2374: expect (ObjNum 993 snap 254 seq_num 993) 2024-03-04T06:17:31.506 INFO:tasks.rados.rados.0.smithi064.stdout:2375: expect (ObjNum 1055 snap 270 seq_num 1055) 2024-03-04T06:17:31.765 INFO:tasks.rados.rados.0.smithi064.stdout:2379: expect (ObjNum 1074 snap 272 seq_num 1074) 2024-03-04T06:17:31.857 INFO:tasks.rados.rados.0.smithi064.stdout:2377: expect (ObjNum 1103 snap 288 seq_num 1103) 2024-03-04T06:17:32.018 INFO:tasks.rados.rados.0.smithi064.stdout:2380: finishing write tid 1 to smithi06431743-2 2024-03-04T06:17:32.018 INFO:tasks.rados.rados.0.smithi064.stdout:2380: finishing write tid 2 to smithi06431743-2 2024-03-04T06:17:32.018 INFO:tasks.rados.rados.0.smithi064.stdout:2378: finishing write tid 1 to smithi06431743-24 2024-03-04T06:17:32.019 INFO:tasks.rados.rados.0.smithi064.stdout:2380: finishing write tid 3 to smithi06431743-2 2024-03-04T06:17:32.019 INFO:tasks.rados.rados.0.smithi064.stdout:2378: finishing write tid 2 to smithi06431743-24 2024-03-04T06:17:32.019 INFO:tasks.rados.rados.0.smithi064.stdout:2380: finishing write tid 4 to smithi06431743-2 2024-03-04T06:17:32.019 INFO:tasks.rados.rados.0.smithi064.stdout:2380: oid 2 updating version 0 to 375 2024-03-04T06:17:32.019 INFO:tasks.rados.rados.0.smithi064.stdout:2380: oid 2 version 375 is already newer than 374 2024-03-04T06:17:32.019 INFO:tasks.rados.rados.0.smithi064.stdout:2380: oid 2 updating version 375 to 376 2024-03-04T06:17:32.019 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 376 (ObjNum 1107 snap 289 seq_num 1107) dirty exists 2024-03-04T06:17:32.019 INFO:tasks.rados.rados.0.smithi064.stdout:2380: left oid 2 (ObjNum 1107 snap 289 seq_num 1107) 2024-03-04T06:17:32.019 INFO:tasks.rados.rados.0.smithi064.stdout:2378: finishing write tid 3 to smithi06431743-24 2024-03-04T06:17:32.019 INFO:tasks.rados.rados.0.smithi064.stdout:2378: finishing write tid 4 to smithi06431743-24 2024-03-04T06:17:32.020 INFO:tasks.rados.rados.0.smithi064.stdout:2378: oid 24 updating version 0 to 699 2024-03-04T06:17:32.020 INFO:tasks.rados.rados.0.smithi064.stdout:2378: oid 24 updating version 699 to 700 2024-03-04T06:17:32.020 INFO:tasks.rados.rados.0.smithi064.stdout:2378: oid 24 updating version 700 to 701 2024-03-04T06:17:32.020 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 701 (ObjNum 1106 snap 289 seq_num 1106) dirty exists 2024-03-04T06:17:32.020 INFO:tasks.rados.rados.0.smithi064.stdout:2378: left oid 24 (ObjNum 1106 snap 289 seq_num 1106) 2024-03-04T06:17:32.126 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:32.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:32.229+0000 7f1f7375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:32.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:32.234+0000 7f1f7375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:32.298 INFO:tasks.rados.rados.0.smithi064.stdout:2373: done (7 left) 2024-03-04T06:17:32.298 INFO:tasks.rados.rados.0.smithi064.stdout:2374: done (6 left) 2024-03-04T06:17:32.298 INFO:tasks.rados.rados.0.smithi064.stdout:2375: done (5 left) 2024-03-04T06:17:32.298 INFO:tasks.rados.rados.0.smithi064.stdout:2377: done (4 left) 2024-03-04T06:17:32.298 INFO:tasks.rados.rados.0.smithi064.stdout:2378: done (3 left) 2024-03-04T06:17:32.299 INFO:tasks.rados.rados.0.smithi064.stdout:2379: done (2 left) 2024-03-04T06:17:32.299 INFO:tasks.rados.rados.0.smithi064.stdout:2380: done (1 left) 2024-03-04T06:17:32.299 INFO:tasks.rados.rados.0.smithi064.stdout:2381: done (0 left) 2024-03-04T06:17:32.299 INFO:tasks.rados.rados.0.smithi064.stdout:2382: write (excl) oid 50 current snap is 290 2024-03-04T06:17:32.299 INFO:tasks.rados.rados.0.smithi064.stdout:2382: seq_num 1108 ranges {622156=647077,1737578=680388} 2024-03-04T06:17:32.304 INFO:tasks.rados.rados.0.smithi064.stdout:2382: writing smithi06431743-50 from 622156 to 1269233 tid 1 2024-03-04T06:17:32.311 INFO:tasks.rados.rados.0.smithi064.stdout:2382: writing smithi06431743-50 from 1737578 to 2417966 tid 2 2024-03-04T06:17:32.311 INFO:tasks.rados.rados.0.smithi064.stdout:2383: write (excl) oid 49 current snap is 290 2024-03-04T06:17:32.312 INFO:tasks.rados.rados.0.smithi064.stdout:2383: seq_num 1109 ranges {638963=722326,2079707=424046} 2024-03-04T06:17:32.318 INFO:tasks.rados.rados.0.smithi064.stdout:2383: writing smithi06431743-49 from 638963 to 1361289 tid 1 2024-03-04T06:17:32.325 INFO:tasks.rados.rados.0.smithi064.stdout:2383: writing smithi06431743-49 from 2079707 to 2503753 tid 2 2024-03-04T06:17:32.325 INFO:tasks.rados.rados.0.smithi064.stdout:2384: snap_remove snap 278 2024-03-04T06:17:33.270 INFO:tasks.rados.rados.0.smithi064.stdout:2384: done (2 left) 2024-03-04T06:17:33.270 INFO:tasks.rados.rados.0.smithi064.stdout:2385: snap_remove snap 282 2024-03-04T06:17:33.855 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:17:33.856 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:17:33.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:17:33.978 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:33.974+0000 7f4ee659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:33.981 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:33.980+0000 7f4ee659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:17:34.171 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:17:34.182 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":868,"stamp":"2024-03-04T06:17:33.268121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":96751630,"num_objects":816,"num_object_clones":634,"num_object_copies":2448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":146,"num_read":4442,"num_read_kb":3391515,"num_write":3812,"num_write_kb":999799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8195,"ondisk_log_size":8195,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":597612,"kb_used_data":415372,"kb_used_omap":0,"kb_used_meta":182208,"kb_avail":376889748,"statfs":{"total":386547056640,"available":385935101952,"internally_reserved":0,"allocated":425340928,"data_stored":475514184,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":186580992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2234869,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":43,"num_read_kb":32683,"num_write":35,"num_write_kb":8638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.986579"},"pg_stats":[{"pgid":"3.f","version":"572'497","reported_seq":2027,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-03-04T06:17:30.304148+0000","last_change":"2024-03-04T06:17:22.346194+0000","last_active":"2024-03-04T06:17:30.304148+0000","last_peered":"2024-03-04T06:17:30.304148+0000","last_clean":"2024-03-04T06:17:30.304148+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:17:30.304148+0000","last_undegraded":"2024-03-04T06:17:30.304148+0000","last_fullsized":"2024-03-04T06:17:30.304148+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151560,"num_objects":54,"num_object_clones":43,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":271,"num_read_kb":224365,"num_write":222,"num_write_kb":59334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"579'400","reported_seq":1952,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-03-04T06:17:29.373913+0000","last_change":"2024-03-04T06:17:25.323644+0000","last_active":"2024-03-04T06:17:29.373913+0000","last_peered":"2024-03-04T06:17:29.373913+0000","last_clean":"2024-03-04T06:17:29.373913+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:17:29.373913+0000","last_undegraded":"2024-03-04T06:17:29.373913+0000","last_fullsized":"2024-03-04T06:17:29.373913+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000891,"num_objects":44,"num_object_clones":30,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":139,"num_read_kb":80773,"num_write":180,"num_write_kb":47650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"578'525","reported_seq":2090,"reported_epoch":581,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:31.305263+0000","last_change":"2024-03-04T06:17:31.305263+0000","last_active":"2024-03-04T06:17:31.305263+0000","last_peered":"2024-03-04T06:17:31.305263+0000","last_clean":"2024-03-04T06:17:31.305263+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:17:31.305263+0000","last_undegraded":"2024-03-04T06:17:31.305263+0000","last_fullsized":"2024-03-04T06:17:31.305263+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":61,"num_object_clones":48,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":343,"num_read_kb":275867,"num_write":254,"num_write_kb":69591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"568'368","reported_seq":1963,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-03-04T06:17:30.334763+0000","last_change":"2024-03-04T06:17:22.333155+0000","last_active":"2024-03-04T06:17:30.334763+0000","last_peered":"2024-03-04T06:17:30.334763+0000","last_clean":"2024-03-04T06:17:30.334763+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:17:30.334763+0000","last_undegraded":"2024-03-04T06:17:30.334763+0000","last_fullsized":"2024-03-04T06:17:30.334763+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9362225,"num_objects":36,"num_object_clones":26,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":209,"num_read_kb":164278,"num_write":178,"num_write_kb":49693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"581'186","reported_seq":1814,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-03-04T06:17:31.324208+0000","last_change":"2024-03-04T06:17:31.383987+0000","last_active":"2024-03-04T06:17:31.324208+0000","last_peered":"2024-03-04T06:17:31.324208+0000","last_clean":"2024-03-04T06:17:31.324208+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:17:31.324208+0000","last_undegraded":"2024-03-04T06:17:31.324208+0000","last_fullsized":"2024-03-04T06:17:31.324208+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403665,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":67,"num_read_kb":40142,"num_write":84,"num_write_kb":22301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"579'560","reported_seq":2054,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-03-04T06:17:30.325663+0000","last_change":"2024-03-04T06:17:22.322275+0000","last_active":"2024-03-04T06:17:30.325663+0000","last_peered":"2024-03-04T06:17:30.325663+0000","last_clean":"2024-03-04T06:17:30.325663+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:17:30.325663+0000","last_undegraded":"2024-03-04T06:17:30.325663+0000","last_fullsized":"2024-03-04T06:17:30.325663+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7278190,"num_objects":51,"num_object_clones":42,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":262,"num_read_kb":199375,"num_write":259,"num_write_kb":75696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1707,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-03-04T06:17:30.305885+0000","last_change":"2024-03-04T06:17:22.322021+0000","last_active":"2024-03-04T06:17:30.305885+0000","last_peered":"2024-03-04T06:17:30.305885+0000","last_clean":"2024-03-04T06:17:30.305885+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:17:30.305885+0000","last_undegraded":"2024-03-04T06:17:30.305885+0000","last_fullsized":"2024-03-04T06:17:30.305885+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"557'270","reported_seq":1900,"reported_epoch":581,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:31.300602+0000","last_change":"2024-03-04T06:17:31.300602+0000","last_active":"2024-03-04T06:17:31.300602+0000","last_peered":"2024-03-04T06:17:31.300602+0000","last_clean":"2024-03-04T06:17:31.300602+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:17:31.300602+0000","last_undegraded":"2024-03-04T06:17:31.300602+0000","last_fullsized":"2024-03-04T06:17:31.300602+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6437477,"num_objects":27,"num_object_clones":18,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":176,"num_read_kb":163221,"num_write":123,"num_write_kb":32558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1580,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-03-04T06:17:31.312377+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:17:31.312377+0000","last_peered":"2024-03-04T06:17:31.312377+0000","last_clean":"2024-03-04T06:17:31.312377+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:17:31.312377+0000","last_undegraded":"2024-03-04T06:17:31.312377+0000","last_fullsized":"2024-03-04T06:17:31.312377+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1447,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-03-04T06:17:31.324935+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:17:31.324935+0000","last_peered":"2024-03-04T06:17:31.324935+0000","last_clean":"2024-03-04T06:17:31.324935+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:17:31.324935+0000","last_undegraded":"2024-03-04T06:17:31.324935+0000","last_fullsized":"2024-03-04T06:17:31.324935+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"579'536","reported_seq":2051,"reported_epoch":581,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:31.309991+0000","last_change":"2024-03-04T06:17:31.309991+0000","last_active":"2024-03-04T06:17:31.309991+0000","last_peered":"2024-03-04T06:17:31.309991+0000","last_clean":"2024-03-04T06:17:31.309991+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:17:31.309991+0000","last_undegraded":"2024-03-04T06:17:31.309991+0000","last_fullsized":"2024-03-04T06:17:31.309991+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093016,"num_objects":54,"num_object_clones":42,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":279,"num_read_kb":221536,"num_write":243,"num_write_kb":65814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1444,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-03-04T06:17:30.315535+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:17:30.315535+0000","last_peered":"2024-03-04T06:17:30.315535+0000","last_clean":"2024-03-04T06:17:30.315535+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:17:30.315535+0000","last_undegraded":"2024-03-04T06:17:30.315535+0000","last_fullsized":"2024-03-04T06:17:30.315535+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"570'377","reported_seq":1940,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-03-04T06:17:31.319266+0000","last_change":"2024-03-04T06:17:31.324169+0000","last_active":"2024-03-04T06:17:31.319266+0000","last_peered":"2024-03-04T06:17:31.319266+0000","last_clean":"2024-03-04T06:17:31.319266+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:17:31.319266+0000","last_undegraded":"2024-03-04T06:17:31.319266+0000","last_fullsized":"2024-03-04T06:17:31.319266+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370051,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":219,"num_read_kb":177501,"num_write":146,"num_write_kb":41225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1447,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-03-04T06:17:31.306387+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:17:31.306387+0000","last_peered":"2024-03-04T06:17:31.306387+0000","last_clean":"2024-03-04T06:17:31.306387+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:17:31.306387+0000","last_undegraded":"2024-03-04T06:17:31.306387+0000","last_fullsized":"2024-03-04T06:17:31.306387+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"581'932","reported_seq":2359,"reported_epoch":581,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:31.390007+0000","last_change":"2024-03-04T06:17:31.323080+0000","last_active":"2024-03-04T06:17:31.390007+0000","last_peered":"2024-03-04T06:17:31.390007+0000","last_clean":"2024-03-04T06:17:31.390007+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:17:31.390007+0000","last_undegraded":"2024-03-04T06:17:31.390007+0000","last_fullsized":"2024-03-04T06:17:31.390007+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":932,"log_dups_size":0,"ondisk_log_size":932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8202575,"num_objects":93,"num_object_clones":72,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":561,"num_read_kb":444386,"num_write":442,"num_write_kb":118018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1444,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-03-04T06:17:30.298017+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:17:30.298017+0000","last_peered":"2024-03-04T06:17:30.298017+0000","last_clean":"2024-03-04T06:17:30.298017+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:17:30.298017+0000","last_undegraded":"2024-03-04T06:17:30.298017+0000","last_fullsized":"2024-03-04T06:17:30.298017+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"577'1016","reported_seq":2333,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-03-04T06:17:29.299354+0000","last_change":"2024-03-04T06:17:25.345618+0000","last_active":"2024-03-04T06:17:29.299354+0000","last_peered":"2024-03-04T06:17:29.299354+0000","last_clean":"2024-03-04T06:17:29.299354+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:17:29.299354+0000","last_undegraded":"2024-03-04T06:17:29.299354+0000","last_fullsized":"2024-03-04T06:17:29.299354+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1016,"log_dups_size":0,"ondisk_log_size":1016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15850222,"num_objects":98,"num_object_clones":76,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":448,"num_read_kb":328458,"num_write":452,"num_write_kb":119741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"577'473","reported_seq":2067,"reported_epoch":581,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:31.310659+0000","last_change":"2024-03-04T06:17:31.310659+0000","last_active":"2024-03-04T06:17:31.310659+0000","last_peered":"2024-03-04T06:17:31.310659+0000","last_clean":"2024-03-04T06:17:31.310659+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:17:31.310659+0000","last_undegraded":"2024-03-04T06:17:31.310659+0000","last_fullsized":"2024-03-04T06:17:31.310659+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9614330,"num_objects":48,"num_object_clones":36,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":333,"num_read_kb":272229,"num_write":228,"num_write_kb":61730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1471,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-03-04T06:17:29.308375+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:17:29.308375+0000","last_peered":"2024-03-04T06:17:29.308375+0000","last_clean":"2024-03-04T06:17:29.308375+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:17:29.308375+0000","last_undegraded":"2024-03-04T06:17:29.308375+0000","last_fullsized":"2024-03-04T06:17:29.308375+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"577'459","reported_seq":2032,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-03-04T06:17:29.305991+0000","last_change":"2024-03-04T06:17:25.353755+0000","last_active":"2024-03-04T06:17:29.305991+0000","last_peered":"2024-03-04T06:17:29.305991+0000","last_clean":"2024-03-04T06:17:29.305991+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:17:29.305991+0000","last_undegraded":"2024-03-04T06:17:29.305991+0000","last_fullsized":"2024-03-04T06:17:29.305991+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4824137,"num_objects":54,"num_object_clones":39,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":237,"num_read_kb":167521,"num_write":218,"num_write_kb":57596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1447,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-03-04T06:17:31.324152+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:17:31.324152+0000","last_peered":"2024-03-04T06:17:31.324152+0000","last_clean":"2024-03-04T06:17:31.324152+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:17:31.324152+0000","last_undegraded":"2024-03-04T06:17:31.324152+0000","last_fullsized":"2024-03-04T06:17:31.324152+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"574'694","reported_seq":2135,"reported_epoch":581,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:31.320048+0000","last_change":"2024-03-04T06:17:31.320048+0000","last_active":"2024-03-04T06:17:31.320048+0000","last_peered":"2024-03-04T06:17:31.320048+0000","last_clean":"2024-03-04T06:17:31.320048+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:17:31.320048+0000","last_undegraded":"2024-03-04T06:17:31.320048+0000","last_fullsized":"2024-03-04T06:17:31.320048+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":694,"log_dups_size":0,"ondisk_log_size":694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6723592,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":339,"num_read_kb":253707,"num_write":310,"num_write_kb":84843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1471,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-03-04T06:17:29.301083+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:17:29.301083+0000","last_peered":"2024-03-04T06:17:29.301083+0000","last_clean":"2024-03-04T06:17:29.301083+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:17:29.301083+0000","last_undegraded":"2024-03-04T06:17:29.301083+0000","last_fullsized":"2024-03-04T06:17:29.301083+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"575'805","reported_seq":2242,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-03-04T06:17:29.336718+0000","last_change":"2024-03-04T06:17:25.345435+0000","last_active":"2024-03-04T06:17:29.336718+0000","last_peered":"2024-03-04T06:17:29.336718+0000","last_clean":"2024-03-04T06:17:29.336718+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:17:29.336718+0000","last_undegraded":"2024-03-04T06:17:29.336718+0000","last_fullsized":"2024-03-04T06:17:29.336718+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":805,"log_dups_size":0,"ondisk_log_size":805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849331,"num_objects":73,"num_object_clones":64,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":456,"num_read_kb":378065,"num_write":336,"num_write_kb":92115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1447,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-03-04T06:17:31.311722+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:17:31.311722+0000","last_peered":"2024-03-04T06:17:31.311722+0000","last_clean":"2024-03-04T06:17:31.311722+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:17:31.311722+0000","last_undegraded":"2024-03-04T06:17:31.311722+0000","last_fullsized":"2024-03-04T06:17:31.311722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":96161262,"num_objects":814,"num_object_clones":634,"num_object_copies":2442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":146,"num_read":4339,"num_read_kb":3391424,"num_write":3675,"num_write_kb":997905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8098,"ondisk_log_size":8098,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706249,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":176308,"kb_used_data":127276,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94195532,"statfs":{"total":96636764160,"available":96456224768,"internally_reserved":0,"allocated":130330624,"data_stored":147300903,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706249,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":148472,"kb_used_data":103600,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94223368,"statfs":{"total":96636764160,"available":96484728832,"internally_reserved":0,"allocated":106086400,"data_stored":117833401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706248,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":143732,"kb_used_data":98860,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94228108,"statfs":{"total":96636764160,"available":96489582592,"internally_reserved":0,"allocated":101232640,"data_stored":114383013,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706248,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":129100,"kb_used_data":85636,"kb_used_omap":0,"kb_used_meta":43456,"kb_avail":94242740,"statfs":{"total":96636764160,"available":96504565760,"internally_reserved":0,"allocated":87691264,"data_stored":95996867,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":44498944},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:17:34.271 INFO:tasks.rados.rados.0.smithi064.stdout:2385: done (2 left) 2024-03-04T06:17:34.271 INFO:tasks.rados.rados.0.smithi064.stdout:2386: snap_create 2024-03-04T06:17:34.271 INFO:tasks.rados.rados.0.smithi064.stdout:2383: finishing write tid 1 to smithi06431743-49 2024-03-04T06:17:34.271 INFO:tasks.rados.rados.0.smithi064.stdout:2383: finishing write tid 2 to smithi06431743-49 2024-03-04T06:17:34.271 INFO:tasks.rados.rados.0.smithi064.stdout:2382: finishing write tid 1 to smithi06431743-50 2024-03-04T06:17:34.271 INFO:tasks.rados.rados.0.smithi064.stdout:2383: finishing write tid 3 to smithi06431743-49 2024-03-04T06:17:34.271 INFO:tasks.rados.rados.0.smithi064.stdout:2383: finishing write tid 4 to smithi06431743-49 2024-03-04T06:17:34.271 INFO:tasks.rados.rados.0.smithi064.stdout:2383: oid 49 updating version 0 to 276 2024-03-04T06:17:34.272 INFO:tasks.rados.rados.0.smithi064.stdout:2383: oid 49 version 276 is already newer than 276 2024-03-04T06:17:34.272 INFO:tasks.rados.rados.0.smithi064.stdout:2383: oid 49 version 276 is already newer than 274 2024-03-04T06:17:34.272 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 276 (ObjNum 1109 snap 290 seq_num 1109) dirty exists 2024-03-04T06:17:34.272 INFO:tasks.rados.rados.0.smithi064.stdout:2383: left oid 49 (ObjNum 1109 snap 290 seq_num 1109) 2024-03-04T06:17:34.272 INFO:tasks.rados.rados.0.smithi064.stdout:2382: finishing write tid 2 to smithi06431743-50 2024-03-04T06:17:34.272 INFO:tasks.rados.rados.0.smithi064.stdout:2382: finishing write tid 3 to smithi06431743-50 2024-03-04T06:17:34.272 INFO:tasks.rados.rados.0.smithi064.stdout:2382: finishing write tid 4 to smithi06431743-50 2024-03-04T06:17:34.272 INFO:tasks.rados.rados.0.smithi064.stdout:2382: oid 50 updating version 0 to 811 2024-03-04T06:17:34.273 INFO:tasks.rados.rados.0.smithi064.stdout:2382: oid 50 updating version 811 to 812 2024-03-04T06:17:34.273 INFO:tasks.rados.rados.0.smithi064.stdout:2382: oid 50 updating version 812 to 813 2024-03-04T06:17:34.273 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 813 (ObjNum 1108 snap 290 seq_num 1108) dirty exists 2024-03-04T06:17:34.273 INFO:tasks.rados.rados.0.smithi064.stdout:2382: left oid 50 (ObjNum 1108 snap 290 seq_num 1108) 2024-03-04T06:17:34.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:34.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:34.413+0000 7fcf1e02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:34.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:34.418+0000 7fcf1e02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:35.271 INFO:tasks.rados.rados.0.smithi064.stdout:2382: done (2 left) 2024-03-04T06:17:35.271 INFO:tasks.rados.rados.0.smithi064.stdout:2383: done (1 left) 2024-03-04T06:17:35.271 INFO:tasks.rados.rados.0.smithi064.stdout:2386: done (0 left) 2024-03-04T06:17:35.271 INFO:tasks.rados.rados.0.smithi064.stdout:2387: snap_create 2024-03-04T06:17:36.273 INFO:tasks.rados.rados.0.smithi064.stdout:2387: done (0 left) 2024-03-04T06:17:36.273 INFO:tasks.rados.rados.0.smithi064.stdout:2388: read oid 39 snap 280 2024-03-04T06:17:36.273 INFO:tasks.rados.rados.0.smithi064.stdout:2388: expect deleted 2024-03-04T06:17:36.273 INFO:tasks.rados.rados.0.smithi064.stdout:2389: write (excl) oid 34 current snap is 292 2024-03-04T06:17:36.273 INFO:tasks.rados.rados.0.smithi064.stdout:2389: seq_num 1110 ranges {655770=513928,1854542=684057} 2024-03-04T06:17:36.279 INFO:tasks.rados.rados.0.smithi064.stdout:2389: writing smithi06431743-34 from 655770 to 1169698 tid 1 2024-03-04T06:17:36.289 INFO:tasks.rados.rados.0.smithi064.stdout:2389: writing smithi06431743-34 from 1854542 to 2538599 tid 2 2024-03-04T06:17:36.290 INFO:tasks.rados.rados.0.smithi064.stdout:2390: write (excl) oid 17 current snap is 292 2024-03-04T06:17:36.290 INFO:tasks.rados.rados.0.smithi064.stdout:2390: seq_num 1111 ranges {672577=589177,1913024=427715} 2024-03-04T06:17:36.297 INFO:tasks.rados.rados.0.smithi064.stdout:2390: writing smithi06431743-17 from 672577 to 1261754 tid 1 2024-03-04T06:17:36.304 INFO:tasks.rados.rados.0.smithi064.stdout:2390: writing smithi06431743-17 from 1913024 to 2340739 tid 2 2024-03-04T06:17:36.304 INFO:tasks.rados.rados.0.smithi064.stdout:2391: write oid 4 current snap is 292 2024-03-04T06:17:36.304 INFO:tasks.rados.rados.0.smithi064.stdout:2391: seq_num 1112 ranges {689384=664426,1971506=687726} 2024-03-04T06:17:36.312 INFO:tasks.rados.rados.0.smithi064.stdout:2391: writing smithi06431743-4 from 689384 to 1353810 tid 1 2024-03-04T06:17:36.322 INFO:tasks.rados.rados.0.smithi064.stdout:2391: writing smithi06431743-4 from 1971506 to 2659232 tid 2 2024-03-04T06:17:36.322 INFO:tasks.rados.rados.0.smithi064.stdout:2392: read oid 50 snap -1 2024-03-04T06:17:36.322 INFO:tasks.rados.rados.0.smithi064.stdout:2392: expect (ObjNum 1108 snap 290 seq_num 1108) 2024-03-04T06:17:36.322 INFO:tasks.rados.rados.0.smithi064.stdout:2393: snap_remove snap 286 2024-03-04T06:17:36.542 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:36.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:36.650+0000 7f3cdd4dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:36.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:36.654+0000 7f3cdd4dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2393: done (5 left) 2024-03-04T06:17:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2394: snap_create 2024-03-04T06:17:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2390: finishing write tid 1 to smithi06431743-17 2024-03-04T06:17:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2389: finishing write tid 1 to smithi06431743-34 2024-03-04T06:17:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2391: finishing write tid 1 to smithi06431743-4 2024-03-04T06:17:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2390: finishing write tid 2 to smithi06431743-17 2024-03-04T06:17:37.307 INFO:tasks.rados.rados.0.smithi064.stdout:2392: expect (ObjNum 1108 snap 290 seq_num 1108) 2024-03-04T06:17:37.500 INFO:tasks.rados.rados.0.smithi064.stdout:2391: finishing write tid 2 to smithi06431743-4 2024-03-04T06:17:37.500 INFO:tasks.rados.rados.0.smithi064.stdout:2389: finishing write tid 2 to smithi06431743-34 2024-03-04T06:17:37.500 INFO:tasks.rados.rados.0.smithi064.stdout:2390: finishing write tid 3 to smithi06431743-17 2024-03-04T06:17:37.501 INFO:tasks.rados.rados.0.smithi064.stdout:2390: finishing write tid 4 to smithi06431743-17 2024-03-04T06:17:37.501 INFO:tasks.rados.rados.0.smithi064.stdout:2390: oid 17 updating version 0 to 284 2024-03-04T06:17:37.501 INFO:tasks.rados.rados.0.smithi064.stdout:2390: oid 17 version 284 is already newer than 282 2024-03-04T06:17:37.501 INFO:tasks.rados.rados.0.smithi064.stdout:2390: oid 17 version 284 is already newer than 283 2024-03-04T06:17:37.501 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 284 (ObjNum 1111 snap 292 seq_num 1111) dirty exists 2024-03-04T06:17:37.501 INFO:tasks.rados.rados.0.smithi064.stdout:2390: left oid 17 (ObjNum 1111 snap 292 seq_num 1111) 2024-03-04T06:17:37.501 INFO:tasks.rados.rados.0.smithi064.stdout:2391: finishing write tid 3 to smithi06431743-4 2024-03-04T06:17:37.501 INFO:tasks.rados.rados.0.smithi064.stdout:2391: finishing write tid 4 to smithi06431743-4 2024-03-04T06:17:37.501 INFO:tasks.rados.rados.0.smithi064.stdout:2391: oid 4 updating version 0 to 1029 2024-03-04T06:17:37.501 INFO:tasks.rados.rados.0.smithi064.stdout:2391: oid 4 updating version 1029 to 1030 2024-03-04T06:17:37.501 INFO:tasks.rados.rados.0.smithi064.stdout:2391: oid 4 version 1030 is already newer than 1028 2024-03-04T06:17:37.502 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 1030 (ObjNum 1112 snap 292 seq_num 1112) dirty exists 2024-03-04T06:17:37.502 INFO:tasks.rados.rados.0.smithi064.stdout:2391: left oid 4 (ObjNum 1112 snap 292 seq_num 1112) 2024-03-04T06:17:37.502 INFO:tasks.rados.rados.0.smithi064.stdout:2389: finishing write tid 3 to smithi06431743-34 2024-03-04T06:17:37.502 INFO:tasks.rados.rados.0.smithi064.stdout:2389: finishing write tid 4 to smithi06431743-34 2024-03-04T06:17:37.502 INFO:tasks.rados.rados.0.smithi064.stdout:2389: oid 34 updating version 0 to 942 2024-03-04T06:17:37.502 INFO:tasks.rados.rados.0.smithi064.stdout:2389: oid 34 version 942 is already newer than 940 2024-03-04T06:17:37.502 INFO:tasks.rados.rados.0.smithi064.stdout:2389: oid 34 version 942 is already newer than 941 2024-03-04T06:17:37.502 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 942 (ObjNum 1110 snap 292 seq_num 1110) dirty exists 2024-03-04T06:17:37.502 INFO:tasks.rados.rados.0.smithi064.stdout:2389: left oid 34 (ObjNum 1110 snap 292 seq_num 1110) 2024-03-04T06:17:38.311 INFO:tasks.rados.rados.0.smithi064.stdout:2388: done (5 left) 2024-03-04T06:17:38.312 INFO:tasks.rados.rados.0.smithi064.stdout:2389: done (4 left) 2024-03-04T06:17:38.312 INFO:tasks.rados.rados.0.smithi064.stdout:2390: done (3 left) 2024-03-04T06:17:38.312 INFO:tasks.rados.rados.0.smithi064.stdout:2391: done (2 left) 2024-03-04T06:17:38.312 INFO:tasks.rados.rados.0.smithi064.stdout:2392: done (1 left) 2024-03-04T06:17:38.312 INFO:tasks.rados.rados.0.smithi064.stdout:2394: done (0 left) 2024-03-04T06:17:38.312 INFO:tasks.rados.rados.0.smithi064.stdout:2395: snap_create 2024-03-04T06:17:38.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:38.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:38.830+0000 7fad90b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:38.835 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:38.834+0000 7fad90b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:39.185 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:17:39.185 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:17:39.185 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:17:39.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:17:39.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:39.290+0000 7fd54a99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:39.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:39.297+0000 7fd54a99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:39.308 INFO:tasks.rados.rados.0.smithi064.stdout:2395: done (0 left) 2024-03-04T06:17:39.308 INFO:tasks.rados.rados.0.smithi064.stdout:2396: read oid 44 snap -1 2024-03-04T06:17:39.308 INFO:tasks.rados.rados.0.smithi064.stdout:2396: expect (ObjNum 993 snap 254 seq_num 993) 2024-03-04T06:17:39.308 INFO:tasks.rados.rados.0.smithi064.stdout:2397: write (excl) oid 39 current snap is 294 2024-03-04T06:17:39.309 INFO:tasks.rados.rados.0.smithi064.stdout:2397: seq_num 1113 ranges {706191=739675,1913635=431384} 2024-03-04T06:17:39.313 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:39.313 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:39.313 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:39.313 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:39.313 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:39.313 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:39.313 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:39.314 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:39.314 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:39.314 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:39.314 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:39.314 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:39.317 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:39.318 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:39.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:17:39.318 INFO:tasks.rados.rados.0.smithi064.stdout:2397: writing smithi06431743-39 from 706191 to 1445866 tid 1 2024-03-04T06:17:39.323 INFO:tasks.rados.rados.0.smithi064.stdout:2397: writing smithi06431743-39 from 1913635 to 2345019 tid 2 2024-03-04T06:17:39.323 INFO:tasks.rados.rados.0.smithi064.stdout:2398: read oid 45 snap 283 2024-03-04T06:17:39.323 INFO:tasks.rados.rados.0.smithi064.stdout:2398: expect deleted 2024-03-04T06:17:39.324 INFO:tasks.rados.rados.0.smithi064.stdout:2399: snap_remove snap 292 2024-03-04T06:17:39.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:39.440+0000 7f15baa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:39.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:39.447+0000 7f15baa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:39.460 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:39.460 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:39.460 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:39.460 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:39.460 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:39.461 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:39.461 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:39.461 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:39.461 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:39.461 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:39.461 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:39.461 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:39.463 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:39.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:17:39.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:39.608+0000 7f009f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:39.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:39.616+0000 7f009f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:39.630 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:39.630 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:39.630 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:39.630 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:39.630 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:39.630 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:39.630 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:39.630 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:39.631 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:39.631 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:39.631 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:39.631 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:39.633 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:39.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:17:39.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:39.737+0000 7f4330e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:39.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:39.744+0000 7f4330e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:39.758 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:39.758 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:39.758 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:39.758 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:39.758 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:39.758 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:39.758 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:39.758 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:39.759 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:39.759 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:39.759 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:39.759 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:39.761 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:39.761 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:40.323 INFO:tasks.rados.rados.0.smithi064.stdout:2399: done (3 left) 2024-03-04T06:17:40.323 INFO:tasks.rados.rados.0.smithi064.stdout:2400: read oid 9 snap 279 2024-03-04T06:17:40.323 INFO:tasks.rados.rados.0.smithi064.stdout:2400: expect deleted 2024-03-04T06:17:40.324 INFO:tasks.rados.rados.0.smithi064.stdout:2401: write oid 34 current snap is 294 2024-03-04T06:17:40.324 INFO:tasks.rados.rados.0.smithi064.stdout:2401: seq_num 1114 ranges {722998=414924,1572117=575042} 2024-03-04T06:17:40.329 INFO:tasks.rados.rados.0.smithi064.stdout:2401: writing smithi06431743-34 from 722998 to 1137922 tid 1 2024-03-04T06:17:40.334 INFO:tasks.rados.rados.0.smithi064.stdout:2401: writing smithi06431743-34 from 1572117 to 2147159 tid 2 2024-03-04T06:17:40.335 INFO:tasks.rados.rados.0.smithi064.stdout:2402: write oid 12 current snap is 294 2024-03-04T06:17:40.335 INFO:tasks.rados.rados.0.smithi064.stdout:2402: seq_num 1115 ranges {739805=490173,1630599=435053,2636284=103521} 2024-03-04T06:17:40.341 INFO:tasks.rados.rados.0.smithi064.stdout:2402: writing smithi06431743-12 from 739805 to 1229978 tid 1 2024-03-04T06:17:40.346 INFO:tasks.rados.rados.0.smithi064.stdout:2402: writing smithi06431743-12 from 1630599 to 2065652 tid 2 2024-03-04T06:17:40.350 INFO:tasks.rados.rados.0.smithi064.stdout:2402: writing smithi06431743-12 from 2636284 to 2739805 tid 3 2024-03-04T06:17:40.350 INFO:tasks.rados.rados.0.smithi064.stdout:2403: read oid 10 snap -1 2024-03-04T06:17:40.350 INFO:tasks.rados.rados.0.smithi064.stdout:2403: expect (ObjNum 1104 snap 288 seq_num 1104) 2024-03-04T06:17:40.350 INFO:tasks.rados.rados.0.smithi064.stdout:2404: read oid 30 snap -1 2024-03-04T06:17:40.350 INFO:tasks.rados.rados.0.smithi064.stdout:2404: expect (ObjNum 1069 snap 272 seq_num 1069) 2024-03-04T06:17:40.351 INFO:tasks.rados.rados.0.smithi064.stdout:2405: read oid 6 snap 277 2024-03-04T06:17:40.351 INFO:tasks.rados.rados.0.smithi064.stdout:2405: expect (ObjNum 1078 snap 274 seq_num 1078) 2024-03-04T06:17:40.351 INFO:tasks.rados.rados.0.smithi064.stdout:2406: read oid 40 snap -1 2024-03-04T06:17:40.351 INFO:tasks.rados.rados.0.smithi064.stdout:2406: expect deleted 2024-03-04T06:17:40.351 INFO:tasks.rados.rados.0.smithi064.stdout:2407: write (excl) oid 28 current snap is 294 2024-03-04T06:17:40.351 INFO:tasks.rados.rados.0.smithi064.stdout:2407: seq_num 1116 ranges {756612=565422,2089081=578711} 2024-03-04T06:17:40.357 INFO:tasks.rados.rados.0.smithi064.stdout:2407: writing smithi06431743-28 from 756612 to 1322034 tid 1 2024-03-04T06:17:40.366 INFO:tasks.rados.rados.0.smithi064.stdout:2407: writing smithi06431743-28 from 2089081 to 2667792 tid 2 2024-03-04T06:17:40.366 INFO:tasks.rados.rados.0.smithi064.stdout:2408: snap_remove snap 287 2024-03-04T06:17:40.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:41.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:41.011+0000 7f70fc42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:41.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:41.015+0000 7f70fc42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:41.334 INFO:tasks.rados.rados.0.smithi064.stdout:2408: done (11 left) 2024-03-04T06:17:41.334 INFO:tasks.rados.rados.0.smithi064.stdout:2409: read oid 29 snap -1 2024-03-04T06:17:41.334 INFO:tasks.rados.rados.0.smithi064.stdout:2409: expect (ObjNum 1098 snap 286 seq_num 1098) 2024-03-04T06:17:41.334 INFO:tasks.rados.rados.0.smithi064.stdout:2410: snap_create 2024-03-04T06:17:41.335 INFO:tasks.rados.rados.0.smithi064.stdout:2396: expect (ObjNum 993 snap 254 seq_num 993) 2024-03-04T06:17:41.496 INFO:tasks.rados.rados.0.smithi064.stdout:2397: finishing write tid 1 to smithi06431743-39 2024-03-04T06:17:41.497 INFO:tasks.rados.rados.0.smithi064.stdout:2397: finishing write tid 2 to smithi06431743-39 2024-03-04T06:17:41.497 INFO:tasks.rados.rados.0.smithi064.stdout:2397: finishing write tid 3 to smithi06431743-39 2024-03-04T06:17:41.497 INFO:tasks.rados.rados.0.smithi064.stdout:2397: finishing write tid 4 to smithi06431743-39 2024-03-04T06:17:41.497 INFO:tasks.rados.rados.0.smithi064.stdout:2397: oid 39 updating version 0 to 541 2024-03-04T06:17:41.497 INFO:tasks.rados.rados.0.smithi064.stdout:2397: oid 39 version 541 is already newer than 540 2024-03-04T06:17:41.497 INFO:tasks.rados.rados.0.smithi064.stdout:2397: oid 39 updating version 541 to 542 2024-03-04T06:17:41.497 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 542 (ObjNum 1113 snap 294 seq_num 1113) dirty exists 2024-03-04T06:17:41.497 INFO:tasks.rados.rados.0.smithi064.stdout:2397: left oid 39 (ObjNum 1113 snap 294 seq_num 1113) 2024-03-04T06:17:41.497 INFO:tasks.rados.rados.0.smithi064.stdout:2405: expect (ObjNum 1078 snap 274 seq_num 1078) 2024-03-04T06:17:41.575 INFO:tasks.rados.rados.0.smithi064.stdout:2403: expect (ObjNum 1104 snap 288 seq_num 1104) 2024-03-04T06:17:41.699 INFO:tasks.rados.rados.0.smithi064.stdout:2404: expect (ObjNum 1069 snap 272 seq_num 1069) 2024-03-04T06:17:41.845 INFO:tasks.rados.rados.0.smithi064.stdout:2402: finishing write tid 1 to smithi06431743-12 2024-03-04T06:17:41.845 INFO:tasks.rados.rados.0.smithi064.stdout:2401: finishing write tid 1 to smithi06431743-34 2024-03-04T06:17:41.845 INFO:tasks.rados.rados.0.smithi064.stdout:2402: finishing write tid 2 to smithi06431743-12 2024-03-04T06:17:41.845 INFO:tasks.rados.rados.0.smithi064.stdout:2401: finishing write tid 2 to smithi06431743-34 2024-03-04T06:17:41.845 INFO:tasks.rados.rados.0.smithi064.stdout:2402: finishing write tid 3 to smithi06431743-12 2024-03-04T06:17:41.846 INFO:tasks.rados.rados.0.smithi064.stdout:2401: finishing write tid 3 to smithi06431743-34 2024-03-04T06:17:41.846 INFO:tasks.rados.rados.0.smithi064.stdout:2402: finishing write tid 4 to smithi06431743-12 2024-03-04T06:17:41.846 INFO:tasks.rados.rados.0.smithi064.stdout:2402: finishing write tid 5 to smithi06431743-12 2024-03-04T06:17:41.846 INFO:tasks.rados.rados.0.smithi064.stdout:2402: oid 12 updating version 0 to 383 2024-03-04T06:17:41.846 INFO:tasks.rados.rados.0.smithi064.stdout:2402: oid 12 version 383 is already newer than 383 2024-03-04T06:17:41.846 INFO:tasks.rados.rados.0.smithi064.stdout:2402: oid 12 updating version 383 to 384 2024-03-04T06:17:41.846 INFO:tasks.rados.rados.0.smithi064.stdout:2402: oid 12 updating version 384 to 385 2024-03-04T06:17:41.846 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 385 (ObjNum 1115 snap 294 seq_num 1115) dirty exists 2024-03-04T06:17:41.846 INFO:tasks.rados.rados.0.smithi064.stdout:2402: left oid 12 (ObjNum 1115 snap 294 seq_num 1115) 2024-03-04T06:17:41.846 INFO:tasks.rados.rados.0.smithi064.stdout:2401: finishing write tid 4 to smithi06431743-34 2024-03-04T06:17:41.846 INFO:tasks.rados.rados.0.smithi064.stdout:2401: oid 34 updating version 0 to 946 2024-03-04T06:17:41.847 INFO:tasks.rados.rados.0.smithi064.stdout:2401: oid 34 version 946 is already newer than 944 2024-03-04T06:17:41.847 INFO:tasks.rados.rados.0.smithi064.stdout:2401: oid 34 version 946 is already newer than 945 2024-03-04T06:17:41.847 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 946 (ObjNum 1114 snap 294 seq_num 1114) dirty exists 2024-03-04T06:17:41.847 INFO:tasks.rados.rados.0.smithi064.stdout:2401: left oid 34 (ObjNum 1114 snap 294 seq_num 1114) 2024-03-04T06:17:41.847 INFO:tasks.rados.rados.0.smithi064.stdout:2407: finishing write tid 1 to smithi06431743-28 2024-03-04T06:17:41.847 INFO:tasks.rados.rados.0.smithi064.stdout:2407: finishing write tid 2 to smithi06431743-28 2024-03-04T06:17:41.847 INFO:tasks.rados.rados.0.smithi064.stdout:2407: finishing write tid 3 to smithi06431743-28 2024-03-04T06:17:41.847 INFO:tasks.rados.rados.0.smithi064.stdout:2407: finishing write tid 4 to smithi06431743-28 2024-03-04T06:17:41.847 INFO:tasks.rados.rados.0.smithi064.stdout:2407: oid 28 updating version 0 to 949 2024-03-04T06:17:41.847 INFO:tasks.rados.rados.0.smithi064.stdout:2407: oid 28 version 949 is already newer than 949 2024-03-04T06:17:41.848 INFO:tasks.rados.rados.0.smithi064.stdout:2407: oid 28 version 949 is already newer than 947 2024-03-04T06:17:41.848 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 949 (ObjNum 1116 snap 294 seq_num 1116) dirty exists 2024-03-04T06:17:41.848 INFO:tasks.rados.rados.0.smithi064.stdout:2407: left oid 28 (ObjNum 1116 snap 294 seq_num 1116) 2024-03-04T06:17:41.848 INFO:tasks.rados.rados.0.smithi064.stdout:2409: expect (ObjNum 1098 snap 286 seq_num 1098) 2024-03-04T06:17:42.333 INFO:tasks.rados.rados.0.smithi064.stdout:2396: done (12 left) 2024-03-04T06:17:42.333 INFO:tasks.rados.rados.0.smithi064.stdout:2397: done (11 left) 2024-03-04T06:17:42.334 INFO:tasks.rados.rados.0.smithi064.stdout:2398: done (10 left) 2024-03-04T06:17:42.334 INFO:tasks.rados.rados.0.smithi064.stdout:2400: done (9 left) 2024-03-04T06:17:42.334 INFO:tasks.rados.rados.0.smithi064.stdout:2401: done (8 left) 2024-03-04T06:17:42.334 INFO:tasks.rados.rados.0.smithi064.stdout:2402: done (7 left) 2024-03-04T06:17:42.334 INFO:tasks.rados.rados.0.smithi064.stdout:2403: done (6 left) 2024-03-04T06:17:42.334 INFO:tasks.rados.rados.0.smithi064.stdout:2404: done (5 left) 2024-03-04T06:17:42.334 INFO:tasks.rados.rados.0.smithi064.stdout:2405: done (4 left) 2024-03-04T06:17:42.334 INFO:tasks.rados.rados.0.smithi064.stdout:2406: done (3 left) 2024-03-04T06:17:42.334 INFO:tasks.rados.rados.0.smithi064.stdout:2407: done (2 left) 2024-03-04T06:17:42.334 INFO:tasks.rados.rados.0.smithi064.stdout:2409: done (1 left) 2024-03-04T06:17:42.334 INFO:tasks.rados.rados.0.smithi064.stdout:2410: done (0 left) 2024-03-04T06:17:42.335 INFO:tasks.rados.rados.0.smithi064.stdout:2411: snap_create 2024-03-04T06:17:43.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:43.192 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:43.190+0000 7f3cd7e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:43.195 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:43.194+0000 7f3cd7e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:43.270 INFO:tasks.rados.rados.0.smithi064.stdout:2411: done (0 left) 2024-03-04T06:17:43.271 INFO:tasks.rados.rados.0.smithi064.stdout:2412: read oid 1 snap -1 2024-03-04T06:17:43.271 INFO:tasks.rados.rados.0.smithi064.stdout:2412: expect (ObjNum 1046 snap 265 seq_num 1046) 2024-03-04T06:17:43.271 INFO:tasks.rados.rados.0.smithi064.stdout:2413: delete oid 19 current snap is 296 2024-03-04T06:17:43.313 INFO:tasks.rados.rados.0.smithi064.stdout:2413: done (1 left) 2024-03-04T06:17:43.313 INFO:tasks.rados.rados.0.smithi064.stdout:2414: read oid 50 snap 281 2024-03-04T06:17:43.313 INFO:tasks.rados.rados.0.smithi064.stdout:2414: expect (ObjNum 1091 snap 279 seq_num 1091) 2024-03-04T06:17:43.313 INFO:tasks.rados.rados.0.smithi064.stdout:2415: snap_remove snap 271 2024-03-04T06:17:44.272 INFO:tasks.rados.rados.0.smithi064.stdout:2415: done (2 left) 2024-03-04T06:17:44.272 INFO:tasks.rados.rados.0.smithi064.stdout:2416: read oid 11 snap 281 2024-03-04T06:17:44.273 INFO:tasks.rados.rados.0.smithi064.stdout:2416: expect deleted 2024-03-04T06:17:44.273 INFO:tasks.rados.rados.0.smithi064.stdout:2417: write (excl) oid 8 current snap is 296 2024-03-04T06:17:44.273 INFO:tasks.rados.rados.0.smithi064.stdout:2417: seq_num 1118 ranges {790226=432273,1806045=582380} 2024-03-04T06:17:44.278 INFO:tasks.rados.rados.0.smithi064.stdout:2417: writing smithi06431743-8 from 790226 to 1222499 tid 1 2024-03-04T06:17:44.285 INFO:tasks.rados.rados.0.smithi064.stdout:2417: writing smithi06431743-8 from 1806045 to 2388425 tid 2 2024-03-04T06:17:44.285 INFO:tasks.rados.rados.0.smithi064.stdout:2418: write oid 26 current snap is 296 2024-03-04T06:17:44.285 INFO:tasks.rados.rados.0.smithi064.stdout:2418: seq_num 1119 ranges {407033=507522,1464527=442391,2345976=461057} 2024-03-04T06:17:44.290 INFO:tasks.rados.rados.0.smithi064.stdout:2418: writing smithi06431743-26 from 407033 to 914555 tid 1 2024-03-04T06:17:44.297 INFO:tasks.rados.rados.0.smithi064.stdout:2418: writing smithi06431743-26 from 1464527 to 1906918 tid 2 2024-03-04T06:17:44.304 INFO:tasks.rados.rados.0.smithi064.stdout:2418: writing smithi06431743-26 from 2345976 to 2807033 tid 3 2024-03-04T06:17:44.304 INFO:tasks.rados.rados.0.smithi064.stdout:2419: snap_create 2024-03-04T06:17:44.304 INFO:tasks.rados.rados.0.smithi064.stdout:2412: expect (ObjNum 1046 snap 265 seq_num 1046) 2024-03-04T06:17:44.524 INFO:tasks.rados.rados.0.smithi064.stdout:2414: expect (ObjNum 1091 snap 279 seq_num 1091) 2024-03-04T06:17:44.702 INFO:tasks.rados.rados.0.smithi064.stdout:2418: finishing write tid 1 to smithi06431743-26 2024-03-04T06:17:44.702 INFO:tasks.rados.rados.0.smithi064.stdout:2417: finishing write tid 1 to smithi06431743-8 2024-03-04T06:17:44.702 INFO:tasks.rados.rados.0.smithi064.stdout:2418: finishing write tid 2 to smithi06431743-26 2024-03-04T06:17:44.702 INFO:tasks.rados.rados.0.smithi064.stdout:2417: finishing write tid 2 to smithi06431743-8 2024-03-04T06:17:44.702 INFO:tasks.rados.rados.0.smithi064.stdout:2417: finishing write tid 3 to smithi06431743-8 2024-03-04T06:17:44.702 INFO:tasks.rados.rados.0.smithi064.stdout:2418: finishing write tid 3 to smithi06431743-26 2024-03-04T06:17:44.702 INFO:tasks.rados.rados.0.smithi064.stdout:2417: finishing write tid 4 to smithi06431743-8 2024-03-04T06:17:44.702 INFO:tasks.rados.rados.0.smithi064.stdout:2417: oid 8 updating version 0 to 473 2024-03-04T06:17:44.702 INFO:tasks.rados.rados.0.smithi064.stdout:2417: oid 8 version 473 is already newer than 471 2024-03-04T06:17:44.703 INFO:tasks.rados.rados.0.smithi064.stdout:2417: oid 8 version 473 is already newer than 473 2024-03-04T06:17:44.703 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 473 (ObjNum 1118 snap 296 seq_num 1118) dirty exists 2024-03-04T06:17:44.703 INFO:tasks.rados.rados.0.smithi064.stdout:2417: left oid 8 (ObjNum 1118 snap 296 seq_num 1118) 2024-03-04T06:17:44.703 INFO:tasks.rados.rados.0.smithi064.stdout:2418: finishing write tid 4 to smithi06431743-26 2024-03-04T06:17:44.703 INFO:tasks.rados.rados.0.smithi064.stdout:2418: finishing write tid 5 to smithi06431743-26 2024-03-04T06:17:44.703 INFO:tasks.rados.rados.0.smithi064.stdout:2418: oid 26 updating version 0 to 956 2024-03-04T06:17:44.703 INFO:tasks.rados.rados.0.smithi064.stdout:2418: oid 26 updating version 956 to 958 2024-03-04T06:17:44.703 INFO:tasks.rados.rados.0.smithi064.stdout:2418: oid 26 version 958 is already newer than 955 2024-03-04T06:17:44.703 INFO:tasks.rados.rados.0.smithi064.stdout:2418: oid 26 version 958 is already newer than 957 2024-03-04T06:17:44.703 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 958 (ObjNum 1119 snap 296 seq_num 1119) dirty exists 2024-03-04T06:17:44.703 INFO:tasks.rados.rados.0.smithi064.stdout:2418: left oid 26 (ObjNum 1119 snap 296 seq_num 1119) 2024-03-04T06:17:44.762 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:17:44.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:17:44.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:44.867+0000 7f808d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:44.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:44.875+0000 7f808d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:45.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:45.272 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:17:45.275 INFO:tasks.rados.rados.0.smithi064.stdout:2412: done (5 left) 2024-03-04T06:17:45.275 INFO:tasks.rados.rados.0.smithi064.stdout:2414: done (4 left) 2024-03-04T06:17:45.275 INFO:tasks.rados.rados.0.smithi064.stdout:2416: done (3 left) 2024-03-04T06:17:45.275 INFO:tasks.rados.rados.0.smithi064.stdout:2417: done (2 left) 2024-03-04T06:17:45.276 INFO:tasks.rados.rados.0.smithi064.stdout:2418: done (1 left) 2024-03-04T06:17:45.276 INFO:tasks.rados.rados.0.smithi064.stdout:2419: done (0 left) 2024-03-04T06:17:45.276 INFO:tasks.rados.rados.0.smithi064.stdout:2420: read oid 38 snap -1 2024-03-04T06:17:45.276 INFO:tasks.rados.rados.0.smithi064.stdout:2420: expect (ObjNum 1073 snap 272 seq_num 1073) 2024-03-04T06:17:45.276 INFO:tasks.rados.rados.0.smithi064.stdout:2421: snap_create 2024-03-04T06:17:45.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:17:45.316 INFO:tasks.rados.rados.0.smithi064.stdout:2420: expect (ObjNum 1073 snap 272 seq_num 1073) 2024-03-04T06:17:45.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:45.380+0000 7f06e779d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:45.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:45.384+0000 7f06e779d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:45.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:45.423+0000 7fa49211d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:45.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:45.432+0000 7fa49211d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:46.273 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:17:46.276 INFO:tasks.rados.rados.0.smithi064.stdout:2420: done (1 left) 2024-03-04T06:17:46.276 INFO:tasks.rados.rados.0.smithi064.stdout:2421: done (0 left) 2024-03-04T06:17:46.276 INFO:tasks.rados.rados.0.smithi064.stdout:2422: read oid 16 snap -1 2024-03-04T06:17:46.276 INFO:tasks.rados.rados.0.smithi064.stdout:2422: expect (ObjNum 1097 snap 286 seq_num 1097) 2024-03-04T06:17:46.276 INFO:tasks.rados.rados.0.smithi064.stdout:2423: read oid 21 snap -1 2024-03-04T06:17:46.276 INFO:tasks.rados.rados.0.smithi064.stdout:2423: expect (ObjNum 1045 snap 265 seq_num 1045) 2024-03-04T06:17:46.276 INFO:tasks.rados.rados.0.smithi064.stdout:2424: write (excl) oid 39 current snap is 298 2024-03-04T06:17:46.277 INFO:tasks.rados.rados.0.smithi064.stdout:2424: seq_num 1120 ranges {423840=582771,1523009=586049,2657046=166794} 2024-03-04T06:17:46.281 INFO:tasks.rados.rados.0.smithi064.stdout:2424: writing smithi06431743-39 from 423840 to 1006611 tid 1 2024-03-04T06:17:46.288 INFO:tasks.rados.rados.0.smithi064.stdout:2424: writing smithi06431743-39 from 1523009 to 2109058 tid 2 2024-03-04T06:17:46.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:17:46.292 INFO:tasks.rados.rados.0.smithi064.stdout:2424: writing smithi06431743-39 from 2657046 to 2823840 tid 3 2024-03-04T06:17:46.292 INFO:tasks.rados.rados.0.smithi064.stdout:2425: write oid 5 current snap is 298 2024-03-04T06:17:46.292 INFO:tasks.rados.rados.0.smithi064.stdout:2425: seq_num 1121 ranges {440647=658020,1581491=446060,2800822=39825} 2024-03-04T06:17:46.298 INFO:tasks.rados.rados.0.smithi064.stdout:2425: writing smithi06431743-5 from 440647 to 1098667 tid 1 2024-03-04T06:17:46.304 INFO:tasks.rados.rados.0.smithi064.stdout:2425: writing smithi06431743-5 from 1581491 to 2027551 tid 2 2024-03-04T06:17:46.309 INFO:tasks.rados.rados.0.smithi064.stdout:2425: writing smithi06431743-5 from 2800822 to 2840647 tid 3 2024-03-04T06:17:46.309 INFO:tasks.rados.rados.0.smithi064.stdout:2426: write oid 32 current snap is 298 2024-03-04T06:17:46.309 INFO:tasks.rados.rados.0.smithi064.stdout:2426: seq_num 1122 ranges {457454=733269,1923620=589718} 2024-03-04T06:17:46.316 INFO:tasks.rados.rados.0.smithi064.stdout:2426: writing smithi06431743-32 from 457454 to 1190723 tid 1 2024-03-04T06:17:46.324 INFO:tasks.rados.rados.0.smithi064.stdout:2426: writing smithi06431743-32 from 1923620 to 2513338 tid 2 2024-03-04T06:17:46.324 INFO:tasks.rados.rados.0.smithi064.stdout:2427: snap_remove snap 288 2024-03-04T06:17:46.410 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:46.402+0000 7fe0f802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:46.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:46.413+0000 7fe0f802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:47.279 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:17:47.283 INFO:tasks.rados.rados.0.smithi064.stdout:2427: done (5 left) 2024-03-04T06:17:47.283 INFO:tasks.rados.rados.0.smithi064.stdout:2428: delete oid 38 current snap is 298 2024-03-04T06:17:47.283 INFO:tasks.rados.rados.0.smithi064.stdout:2423: expect (ObjNum 1045 snap 265 seq_num 1045) 2024-03-04T06:17:47.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:17:47.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:47.446+0000 7fc90299d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:47.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:47.454+0000 7fc90299d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:47.458 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:47.512 INFO:tasks.rados.rados.0.smithi064.stdout:2422: expect (ObjNum 1097 snap 286 seq_num 1097) 2024-03-04T06:17:47.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:47.563+0000 7fee3bd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:47.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:47.568+0000 7fee3bd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:47.675 INFO:tasks.rados.rados.0.smithi064.stdout:2424: finishing write tid 1 to smithi06431743-39 2024-03-04T06:17:47.675 INFO:tasks.rados.rados.0.smithi064.stdout:2425: finishing write tid 1 to smithi06431743-5 2024-03-04T06:17:47.675 INFO:tasks.rados.rados.0.smithi064.stdout:2424: finishing write tid 2 to smithi06431743-39 2024-03-04T06:17:47.675 INFO:tasks.rados.rados.0.smithi064.stdout:2425: finishing write tid 2 to smithi06431743-5 2024-03-04T06:17:47.675 INFO:tasks.rados.rados.0.smithi064.stdout:2426: finishing write tid 1 to smithi06431743-32 2024-03-04T06:17:47.675 INFO:tasks.rados.rados.0.smithi064.stdout:2425: finishing write tid 3 to smithi06431743-5 2024-03-04T06:17:47.675 INFO:tasks.rados.rados.0.smithi064.stdout:2424: finishing write tid 3 to smithi06431743-39 2024-03-04T06:17:47.675 INFO:tasks.rados.rados.0.smithi064.stdout:2424: finishing write tid 4 to smithi06431743-39 2024-03-04T06:17:47.675 INFO:tasks.rados.rados.0.smithi064.stdout:2425: finishing write tid 4 to smithi06431743-5 2024-03-04T06:17:47.675 INFO:tasks.rados.rados.0.smithi064.stdout:2422: done (5 left) 2024-03-04T06:17:47.676 INFO:tasks.rados.rados.0.smithi064.stdout:2423: done (4 left) 2024-03-04T06:17:47.676 INFO:tasks.rados.rados.0.smithi064.stdout:2428: done (3 left) 2024-03-04T06:17:47.676 INFO:tasks.rados.rados.0.smithi064.stdout:2429: write (excl) oid 21 current snap is 298 2024-03-04T06:17:47.676 INFO:tasks.rados.rados.0.smithi064.stdout:2424: finishing write tid 5 to smithi06431743-39 2024-03-04T06:17:47.676 INFO:tasks.rados.rados.0.smithi064.stdout:2424: oid 39 updating version 0 to 549 2024-03-04T06:17:47.676 INFO:tasks.rados.rados.0.smithi064.stdout:2424: oid 39 updating version 549 to 551 2024-03-04T06:17:47.676 INFO:tasks.rados.rados.0.smithi064.stdout:2424: oid 39 version 551 is already newer than 548 2024-03-04T06:17:47.676 INFO:tasks.rados.rados.0.smithi064.stdout:2424: oid 39 version 551 is already newer than 550 2024-03-04T06:17:47.676 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 551 (ObjNum 1120 snap 298 seq_num 1120) dirty exists 2024-03-04T06:17:47.676 INFO:tasks.rados.rados.0.smithi064.stdout:2424: left oid 39 (ObjNum 1120 snap 298 seq_num 1120) 2024-03-04T06:17:47.676 INFO:tasks.rados.rados.0.smithi064.stdout:2426: finishing write tid 2 to smithi06431743-32 2024-03-04T06:17:47.677 INFO:tasks.rados.rados.0.smithi064.stdout:2429: seq_num 1124 ranges {491068=483767,1640584=593387,2650385=240683} 2024-03-04T06:17:47.679 INFO:tasks.rados.rados.0.smithi064.stdout:2429: writing smithi06431743-21 from 491068 to 974835 tid 1 2024-03-04T06:17:47.686 INFO:tasks.rados.rados.0.smithi064.stdout:2429: writing smithi06431743-21 from 1640584 to 2233971 tid 2 2024-03-04T06:17:47.691 INFO:tasks.rados.rados.0.smithi064.stdout:2429: writing smithi06431743-21 from 2650385 to 2891068 tid 3 2024-03-04T06:17:47.691 INFO:tasks.rados.rados.0.smithi064.stdout:2424: done (3 left) 2024-03-04T06:17:47.691 INFO:tasks.rados.rados.0.smithi064.stdout:2430: read oid 39 snap -1 2024-03-04T06:17:47.691 INFO:tasks.rados.rados.0.smithi064.stdout:2430: expect (ObjNum 1120 snap 298 seq_num 1120) 2024-03-04T06:17:47.691 INFO:tasks.rados.rados.0.smithi064.stdout:2431: read oid 2 snap -1 2024-03-04T06:17:47.691 INFO:tasks.rados.rados.0.smithi064.stdout:2431: expect (ObjNum 1107 snap 289 seq_num 1107) 2024-03-04T06:17:47.692 INFO:tasks.rados.rados.0.smithi064.stdout:2432: delete oid 43 current snap is 298 2024-03-04T06:17:47.692 INFO:tasks.rados.rados.0.smithi064.stdout:2425: finishing write tid 5 to smithi06431743-5 2024-03-04T06:17:47.692 INFO:tasks.rados.rados.0.smithi064.stdout:2425: oid 5 updating version 0 to 1044 2024-03-04T06:17:47.692 INFO:tasks.rados.rados.0.smithi064.stdout:2425: oid 5 version 1044 is already newer than 1041 2024-03-04T06:17:47.692 INFO:tasks.rados.rados.0.smithi064.stdout:2425: oid 5 version 1044 is already newer than 1042 2024-03-04T06:17:47.692 INFO:tasks.rados.rados.0.smithi064.stdout:2425: oid 5 version 1044 is already newer than 1043 2024-03-04T06:17:47.692 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 1044 (ObjNum 1121 snap 298 seq_num 1121) dirty exists 2024-03-04T06:17:47.692 INFO:tasks.rados.rados.0.smithi064.stdout:2425: left oid 5 (ObjNum 1121 snap 298 seq_num 1121) 2024-03-04T06:17:47.692 INFO:tasks.rados.rados.0.smithi064.stdout:2426: finishing write tid 3 to smithi06431743-32 2024-03-04T06:17:47.692 INFO:tasks.rados.rados.0.smithi064.stdout:2426: finishing write tid 4 to smithi06431743-32 2024-03-04T06:17:47.693 INFO:tasks.rados.rados.0.smithi064.stdout:2426: oid 32 updating version 0 to 961 2024-03-04T06:17:47.693 INFO:tasks.rados.rados.0.smithi064.stdout:2426: oid 32 updating version 961 to 962 2024-03-04T06:17:47.693 INFO:tasks.rados.rados.0.smithi064.stdout:2426: oid 32 updating version 962 to 963 2024-03-04T06:17:47.693 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 963 (ObjNum 1122 snap 298 seq_num 1122) dirty exists 2024-03-04T06:17:47.693 INFO:tasks.rados.rados.0.smithi064.stdout:2426: left oid 32 (ObjNum 1122 snap 298 seq_num 1122) 2024-03-04T06:17:47.715 INFO:tasks.rados.rados.0.smithi064.stdout:2431: expect (ObjNum 1107 snap 289 seq_num 1107) 2024-03-04T06:17:47.850 INFO:tasks.rados.rados.0.smithi064.stdout:2430: expect (ObjNum 1120 snap 298 seq_num 1120) 2024-03-04T06:17:47.986 INFO:tasks.rados.rados.0.smithi064.stdout:2429: finishing write tid 1 to smithi06431743-21 2024-03-04T06:17:47.987 INFO:tasks.rados.rados.0.smithi064.stdout:2429: finishing write tid 2 to smithi06431743-21 2024-03-04T06:17:47.987 INFO:tasks.rados.rados.0.smithi064.stdout:2429: finishing write tid 3 to smithi06431743-21 2024-03-04T06:17:47.987 INFO:tasks.rados.rados.0.smithi064.stdout:2425: done (5 left) 2024-03-04T06:17:47.987 INFO:tasks.rados.rados.0.smithi064.stdout:2426: done (4 left) 2024-03-04T06:17:47.987 INFO:tasks.rados.rados.0.smithi064.stdout:2430: done (3 left) 2024-03-04T06:17:47.987 INFO:tasks.rados.rados.0.smithi064.stdout:2431: done (2 left) 2024-03-04T06:17:47.987 INFO:tasks.rados.rados.0.smithi064.stdout:2432: done (1 left) 2024-03-04T06:17:47.987 INFO:tasks.rados.rados.0.smithi064.stdout:2433: snap_create 2024-03-04T06:17:47.987 INFO:tasks.rados.rados.0.smithi064.stdout:2429: finishing write tid 4 to smithi06431743-21 2024-03-04T06:17:47.988 INFO:tasks.rados.rados.0.smithi064.stdout:2429: finishing write tid 5 to smithi06431743-21 2024-03-04T06:17:47.988 INFO:tasks.rados.rados.0.smithi064.stdout:2429: oid 21 updating version 0 to 485 2024-03-04T06:17:47.988 INFO:tasks.rados.rados.0.smithi064.stdout:2429: oid 21 version 485 is already newer than 483 2024-03-04T06:17:47.988 INFO:tasks.rados.rados.0.smithi064.stdout:2429: oid 21 updating version 485 to 486 2024-03-04T06:17:47.988 INFO:tasks.rados.rados.0.smithi064.stdout:2429: oid 21 version 486 is already newer than 484 2024-03-04T06:17:47.988 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 486 (ObjNum 1124 snap 298 seq_num 1124) dirty exists 2024-03-04T06:17:47.988 INFO:tasks.rados.rados.0.smithi064.stdout:2429: left oid 21 (ObjNum 1124 snap 298 seq_num 1124) 2024-03-04T06:17:48.281 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:17:48.289 INFO:tasks.rados.rados.0.smithi064.stdout:2429: done (1 left) 2024-03-04T06:17:48.289 INFO:tasks.rados.rados.0.smithi064.stdout:2433: done (0 left) 2024-03-04T06:17:48.289 INFO:tasks.rados.rados.0.smithi064.stdout:2434: write (excl) oid 15 current snap is 299 2024-03-04T06:17:48.289 INFO:tasks.rados.rados.0.smithi064.stdout:2434: seq_num 1126 ranges {524682=750618,1757548=597056} 2024-03-04T06:17:48.298 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:17:48.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:17:48.298 INFO:tasks.rados.rados.0.smithi064.stdout:2434: writing smithi06431743-15 from 524682 to 1275300 tid 1 2024-03-04T06:17:48.303 INFO:tasks.rados.rados.0.smithi064.stdout:2434: writing smithi06431743-15 from 1757548 to 2354604 tid 2 2024-03-04T06:17:48.304 INFO:tasks.rados.rados.0.smithi064.stdout:2435: delete oid 17 current snap is 299 2024-03-04T06:17:48.364 INFO:tasks.rados.rados.0.smithi064.stdout:2435: done (1 left) 2024-03-04T06:17:48.364 INFO:tasks.rados.rados.0.smithi064.stdout:2436: read oid 37 snap 277 2024-03-04T06:17:48.364 INFO:tasks.rados.rados.0.smithi064.stdout:2436: expect (ObjNum 1050 snap 268 seq_num 1050) 2024-03-04T06:17:48.365 INFO:tasks.rados.rados.0.smithi064.stdout:2437: snap_remove snap 293 2024-03-04T06:17:48.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:48.410+0000 7f14a9413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:48.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:48.418+0000 7f14a9413640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:48.609 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:17:48.610 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:17:48.622 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":890,"stamp":"2024-03-04T06:17:48.284001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":105203967,"num_objects":829,"num_object_clones":645,"num_object_copies":2487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":144,"num_read":4531,"num_read_kb":3461953,"num_write":3869,"num_write_kb":1016289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8419,"ondisk_log_size":8419,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":573512,"kb_used_data":389864,"kb_used_omap":0,"kb_used_meta":183616,"kb_avail":376913848,"statfs":{"total":386547056640,"available":385959780352,"internally_reserved":0,"allocated":399220736,"data_stored":449411258,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":188022784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1358826,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":43555,"num_write":17,"num_write_kb":5238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013786"},"pg_stats":[{"pgid":"3.f","version":"594'509","reported_seq":2069,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.301056+0000","last_change":"2024-03-04T06:17:44.435888+0000","last_active":"2024-03-04T06:17:45.301056+0000","last_peered":"2024-03-04T06:17:45.301056+0000","last_clean":"2024-03-04T06:17:45.301056+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:17:45.301056+0000","last_undegraded":"2024-03-04T06:17:45.301056+0000","last_fullsized":"2024-03-04T06:17:45.301056+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151560,"num_objects":54,"num_object_clones":43,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":271,"num_read_kb":224365,"num_write":222,"num_write_kb":59334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"594'406","reported_seq":1994,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.283938+0000","last_change":"2024-03-04T06:17:44.360573+0000","last_active":"2024-03-04T06:17:44.283938+0000","last_peered":"2024-03-04T06:17:44.283938+0000","last_clean":"2024-03-04T06:17:44.283938+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:17:44.283938+0000","last_undegraded":"2024-03-04T06:17:44.283938+0000","last_fullsized":"2024-03-04T06:17:44.283938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000891,"num_objects":44,"num_object_clones":30,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":139,"num_read_kb":80773,"num_write":180,"num_write_kb":47650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"596'548","reported_seq":2146,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.308319+0000","last_change":"2024-03-04T06:17:44.371590+0000","last_active":"2024-03-04T06:17:46.308319+0000","last_peered":"2024-03-04T06:17:46.308319+0000","last_clean":"2024-03-04T06:17:46.308319+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:17:46.308319+0000","last_undegraded":"2024-03-04T06:17:46.308319+0000","last_fullsized":"2024-03-04T06:17:46.308319+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706191,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":346,"num_read_kb":275868,"num_write":260,"num_write_kb":71879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"594'393","reported_seq":2014,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.290965+0000","last_change":"2024-03-04T06:17:44.411376+0000","last_active":"2024-03-04T06:17:45.290965+0000","last_peered":"2024-03-04T06:17:45.290965+0000","last_clean":"2024-03-04T06:17:45.290965+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:17:45.290965+0000","last_undegraded":"2024-03-04T06:17:45.290965+0000","last_fullsized":"2024-03-04T06:17:45.290965+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7698398,"num_objects":38,"num_object_clones":28,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":211,"num_read_kb":164280,"num_write":187,"num_write_kb":51669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"594'188","reported_seq":1859,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.365715+0000","last_change":"2024-03-04T06:17:44.391978+0000","last_active":"2024-03-04T06:17:46.365715+0000","last_peered":"2024-03-04T06:17:46.365715+0000","last_clean":"2024-03-04T06:17:46.365715+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:17:46.365715+0000","last_undegraded":"2024-03-04T06:17:46.365715+0000","last_fullsized":"2024-03-04T06:17:46.365715+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403665,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":67,"num_read_kb":40142,"num_write":84,"num_write_kb":22301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"592'570","reported_seq":2105,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.312217+0000","last_change":"2024-03-04T06:17:44.385509+0000","last_active":"2024-03-04T06:17:45.312217+0000","last_peered":"2024-03-04T06:17:45.312217+0000","last_clean":"2024-03-04T06:17:45.312217+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:17:45.312217+0000","last_undegraded":"2024-03-04T06:17:45.312217+0000","last_fullsized":"2024-03-04T06:17:45.312217+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7278190,"num_objects":51,"num_object_clones":42,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":280,"num_read_kb":215130,"num_write":259,"num_write_kb":75696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1749,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.297310+0000","last_change":"2024-03-04T06:17:44.386007+0000","last_active":"2024-03-04T06:17:45.297310+0000","last_peered":"2024-03-04T06:17:45.297310+0000","last_clean":"2024-03-04T06:17:45.297310+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:17:45.297310+0000","last_undegraded":"2024-03-04T06:17:45.297310+0000","last_fullsized":"2024-03-04T06:17:45.297310+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"594'296","reported_seq":1958,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.292922+0000","last_change":"2024-03-04T06:17:44.401444+0000","last_active":"2024-03-04T06:17:46.292922+0000","last_peered":"2024-03-04T06:17:46.292922+0000","last_clean":"2024-03-04T06:17:46.292922+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:17:46.292922+0000","last_undegraded":"2024-03-04T06:17:46.292922+0000","last_fullsized":"2024-03-04T06:17:46.292922+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5311540,"num_objects":29,"num_object_clones":20,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":180,"num_read_kb":163223,"num_write":131,"num_write_kb":34673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1619,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.355372+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:17:46.355372+0000","last_peered":"2024-03-04T06:17:46.355372+0000","last_clean":"2024-03-04T06:17:46.355372+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:17:46.355372+0000","last_undegraded":"2024-03-04T06:17:46.355372+0000","last_fullsized":"2024-03-04T06:17:46.355372+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1490,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.301750+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:17:46.301750+0000","last_peered":"2024-03-04T06:17:46.301750+0000","last_clean":"2024-03-04T06:17:46.301750+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:17:46.301750+0000","last_undegraded":"2024-03-04T06:17:46.301750+0000","last_fullsized":"2024-03-04T06:17:46.301750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"594'548","reported_seq":2104,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.298321+0000","last_change":"2024-03-04T06:17:44.394104+0000","last_active":"2024-03-04T06:17:46.298321+0000","last_peered":"2024-03-04T06:17:46.298321+0000","last_clean":"2024-03-04T06:17:46.298321+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:17:46.298321+0000","last_undegraded":"2024-03-04T06:17:46.298321+0000","last_fullsized":"2024-03-04T06:17:46.298321+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093016,"num_objects":54,"num_object_clones":42,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":285,"num_read_kb":227521,"num_write":243,"num_write_kb":65814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1479,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.300116+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:17:45.300116+0000","last_peered":"2024-03-04T06:17:45.300116+0000","last_clean":"2024-03-04T06:17:45.300116+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:17:45.300116+0000","last_undegraded":"2024-03-04T06:17:45.300116+0000","last_fullsized":"2024-03-04T06:17:45.300116+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"594'379","reported_seq":1985,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.361085+0000","last_change":"2024-03-04T06:17:44.392516+0000","last_active":"2024-03-04T06:17:46.361085+0000","last_peered":"2024-03-04T06:17:46.361085+0000","last_clean":"2024-03-04T06:17:46.361085+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:17:46.361085+0000","last_undegraded":"2024-03-04T06:17:46.361085+0000","last_fullsized":"2024-03-04T06:17:46.361085+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370051,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":219,"num_read_kb":177501,"num_write":146,"num_write_kb":41225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1490,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.287926+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:17:46.287926+0000","last_peered":"2024-03-04T06:17:46.287926+0000","last_clean":"2024-03-04T06:17:46.287926+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:17:46.287926+0000","last_undegraded":"2024-03-04T06:17:46.287926+0000","last_fullsized":"2024-03-04T06:17:46.287926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"594'960","reported_seq":2427,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.371285+0000","last_change":"2024-03-04T06:17:44.459636+0000","last_active":"2024-03-04T06:17:46.371285+0000","last_peered":"2024-03-04T06:17:46.371285+0000","last_clean":"2024-03-04T06:17:46.371285+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:17:46.371285+0000","last_undegraded":"2024-03-04T06:17:46.371285+0000","last_fullsized":"2024-03-04T06:17:46.371285+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":960,"log_dups_size":0,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13057488,"num_objects":97,"num_object_clones":75,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":574,"num_read_kb":452854,"num_write":460,"num_write_kb":123373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1479,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.286404+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:17:45.286404+0000","last_peered":"2024-03-04T06:17:45.286404+0000","last_clean":"2024-03-04T06:17:45.286404+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:17:45.286404+0000","last_undegraded":"2024-03-04T06:17:45.286404+0000","last_fullsized":"2024-03-04T06:17:45.286404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"594'1039","reported_seq":2385,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.285051+0000","last_change":"2024-03-04T06:17:44.391004+0000","last_active":"2024-03-04T06:17:44.285051+0000","last_peered":"2024-03-04T06:17:44.285051+0000","last_clean":"2024-03-04T06:17:44.285051+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:17:44.285051+0000","last_undegraded":"2024-03-04T06:17:44.285051+0000","last_fullsized":"2024-03-04T06:17:44.285051+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1039,"log_dups_size":0,"ondisk_log_size":1039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16253590,"num_objects":99,"num_object_clones":77,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":461,"num_read_kb":342096,"num_write":456,"num_write_kb":121062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"584'481","reported_seq":2123,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.312712+0000","last_change":"2024-03-04T06:17:44.309404+0000","last_active":"2024-03-04T06:17:46.312712+0000","last_peered":"2024-03-04T06:17:46.312712+0000","last_clean":"2024-03-04T06:17:46.312712+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:17:46.312712+0000","last_undegraded":"2024-03-04T06:17:46.312712+0000","last_fullsized":"2024-03-04T06:17:46.312712+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9614330,"num_objects":48,"num_object_clones":36,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":345,"num_read_kb":286484,"num_write":228,"num_write_kb":61730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1506,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.303474+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:17:44.303474+0000","last_peered":"2024-03-04T06:17:44.303474+0000","last_clean":"2024-03-04T06:17:44.303474+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:17:44.303474+0000","last_undegraded":"2024-03-04T06:17:44.303474+0000","last_fullsized":"2024-03-04T06:17:44.303474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"594'473","reported_seq":2078,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.436469+0000","last_change":"2024-03-04T06:17:44.439748+0000","last_active":"2024-03-04T06:17:44.436469+0000","last_peered":"2024-03-04T06:17:44.436469+0000","last_clean":"2024-03-04T06:17:44.436469+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:17:44.436469+0000","last_undegraded":"2024-03-04T06:17:44.436469+0000","last_fullsized":"2024-03-04T06:17:44.436469+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227505,"num_objects":55,"num_object_clones":40,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":239,"num_read_kb":167522,"num_write":222,"num_write_kb":58588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1486,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.369221+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:17:46.369221+0000","last_peered":"2024-03-04T06:17:46.369221+0000","last_clean":"2024-03-04T06:17:46.369221+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:17:46.369221+0000","last_undegraded":"2024-03-04T06:17:46.369221+0000","last_fullsized":"2024-03-04T06:17:46.369221+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"594'707","reported_seq":2185,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.360227+0000","last_change":"2024-03-04T06:17:44.422554+0000","last_active":"2024-03-04T06:17:46.360227+0000","last_peered":"2024-03-04T06:17:46.360227+0000","last_clean":"2024-03-04T06:17:46.360227+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:17:46.360227+0000","last_undegraded":"2024-03-04T06:17:46.360227+0000","last_fullsized":"2024-03-04T06:17:46.360227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":707,"log_dups_size":0,"ondisk_log_size":707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9312134,"num_objects":68,"num_object_clones":54,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":341,"num_read_kb":253708,"num_write":314,"num_write_kb":85989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1506,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.286029+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:17:44.286029+0000","last_peered":"2024-03-04T06:17:44.286029+0000","last_clean":"2024-03-04T06:17:44.286029+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:17:44.286029+0000","last_undegraded":"2024-03-04T06:17:44.286029+0000","last_fullsized":"2024-03-04T06:17:44.286029+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"594'825","reported_seq":2294,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.287590+0000","last_change":"2024-03-04T06:17:44.389931+0000","last_active":"2024-03-04T06:17:44.287590+0000","last_peered":"2024-03-04T06:17:44.287590+0000","last_clean":"2024-03-04T06:17:44.287590+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:17:44.287590+0000","last_undegraded":"2024-03-04T06:17:44.287590+0000","last_fullsized":"2024-03-04T06:17:44.287590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135050,"num_objects":74,"num_object_clones":65,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":470,"num_read_kb":390395,"num_write":340,"num_write_kb":93412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1490,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.300051+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:17:46.300051+0000","last_peered":"2024-03-04T06:17:46.300051+0000","last_clean":"2024-03-04T06:17:46.300051+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:17:46.300051+0000","last_undegraded":"2024-03-04T06:17:46.300051+0000","last_fullsized":"2024-03-04T06:17:46.300051+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":104613599,"num_objects":827,"num_object_clones":645,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":144,"num_read":4428,"num_read_kb":3461862,"num_write":3732,"num_write_kb":1014395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8322,"ondisk_log_size":8322,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":170596,"kb_used_data":121564,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94201244,"statfs":{"total":96636764160,"available":96462073856,"internally_reserved":0,"allocated":124481536,"data_stored":141550469,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":137468,"kb_used_data":92596,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94234372,"statfs":{"total":96636764160,"available":96495996928,"internally_reserved":0,"allocated":94818304,"data_stored":104684949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":143144,"kb_used_data":98272,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94228696,"statfs":{"total":96636764160,"available":96490184704,"internally_reserved":0,"allocated":100630528,"data_stored":115768468,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":122304,"kb_used_data":77432,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94249536,"statfs":{"total":96636764160,"available":96511524864,"internally_reserved":0,"allocated":79290368,"data_stored":87407372,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:17:48.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:17:48.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:48.726+0000 7f7d12554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:48.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:48.732+0000 7f7d12554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:48.921 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:17:48.921 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:17:48.932 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":891,"stamp":"2024-03-04T06:17:48.834013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":105203967,"num_objects":829,"num_object_clones":645,"num_object_copies":2487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":144,"num_read":4531,"num_read_kb":3461953,"num_write":3869,"num_write_kb":1016289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8419,"ondisk_log_size":8419,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":573512,"kb_used_data":389864,"kb_used_omap":0,"kb_used_meta":183616,"kb_avail":376913848,"statfs":{"total":386547056640,"available":385959780352,"internally_reserved":0,"allocated":399220736,"data_stored":449411258,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":188022784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1358826,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":43555,"num_write":17,"num_write_kb":5238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000788"},"pg_stats":[{"pgid":"3.f","version":"594'509","reported_seq":2069,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.301056+0000","last_change":"2024-03-04T06:17:44.435888+0000","last_active":"2024-03-04T06:17:45.301056+0000","last_peered":"2024-03-04T06:17:45.301056+0000","last_clean":"2024-03-04T06:17:45.301056+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:17:45.301056+0000","last_undegraded":"2024-03-04T06:17:45.301056+0000","last_fullsized":"2024-03-04T06:17:45.301056+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151560,"num_objects":54,"num_object_clones":43,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":271,"num_read_kb":224365,"num_write":222,"num_write_kb":59334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"594'406","reported_seq":1994,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.283938+0000","last_change":"2024-03-04T06:17:44.360573+0000","last_active":"2024-03-04T06:17:44.283938+0000","last_peered":"2024-03-04T06:17:44.283938+0000","last_clean":"2024-03-04T06:17:44.283938+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:17:44.283938+0000","last_undegraded":"2024-03-04T06:17:44.283938+0000","last_fullsized":"2024-03-04T06:17:44.283938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000891,"num_objects":44,"num_object_clones":30,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":139,"num_read_kb":80773,"num_write":180,"num_write_kb":47650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"596'548","reported_seq":2146,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.308319+0000","last_change":"2024-03-04T06:17:44.371590+0000","last_active":"2024-03-04T06:17:46.308319+0000","last_peered":"2024-03-04T06:17:46.308319+0000","last_clean":"2024-03-04T06:17:46.308319+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:17:46.308319+0000","last_undegraded":"2024-03-04T06:17:46.308319+0000","last_fullsized":"2024-03-04T06:17:46.308319+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706191,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":346,"num_read_kb":275868,"num_write":260,"num_write_kb":71879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"594'393","reported_seq":2014,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.290965+0000","last_change":"2024-03-04T06:17:44.411376+0000","last_active":"2024-03-04T06:17:45.290965+0000","last_peered":"2024-03-04T06:17:45.290965+0000","last_clean":"2024-03-04T06:17:45.290965+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:17:45.290965+0000","last_undegraded":"2024-03-04T06:17:45.290965+0000","last_fullsized":"2024-03-04T06:17:45.290965+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7698398,"num_objects":38,"num_object_clones":28,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":211,"num_read_kb":164280,"num_write":187,"num_write_kb":51669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"594'188","reported_seq":1859,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.365715+0000","last_change":"2024-03-04T06:17:44.391978+0000","last_active":"2024-03-04T06:17:46.365715+0000","last_peered":"2024-03-04T06:17:46.365715+0000","last_clean":"2024-03-04T06:17:46.365715+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:17:46.365715+0000","last_undegraded":"2024-03-04T06:17:46.365715+0000","last_fullsized":"2024-03-04T06:17:46.365715+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403665,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":67,"num_read_kb":40142,"num_write":84,"num_write_kb":22301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"592'570","reported_seq":2105,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.312217+0000","last_change":"2024-03-04T06:17:44.385509+0000","last_active":"2024-03-04T06:17:45.312217+0000","last_peered":"2024-03-04T06:17:45.312217+0000","last_clean":"2024-03-04T06:17:45.312217+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:17:45.312217+0000","last_undegraded":"2024-03-04T06:17:45.312217+0000","last_fullsized":"2024-03-04T06:17:45.312217+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7278190,"num_objects":51,"num_object_clones":42,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":280,"num_read_kb":215130,"num_write":259,"num_write_kb":75696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1749,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.297310+0000","last_change":"2024-03-04T06:17:44.386007+0000","last_active":"2024-03-04T06:17:45.297310+0000","last_peered":"2024-03-04T06:17:45.297310+0000","last_clean":"2024-03-04T06:17:45.297310+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:17:45.297310+0000","last_undegraded":"2024-03-04T06:17:45.297310+0000","last_fullsized":"2024-03-04T06:17:45.297310+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"594'296","reported_seq":1958,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.292922+0000","last_change":"2024-03-04T06:17:44.401444+0000","last_active":"2024-03-04T06:17:46.292922+0000","last_peered":"2024-03-04T06:17:46.292922+0000","last_clean":"2024-03-04T06:17:46.292922+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:17:46.292922+0000","last_undegraded":"2024-03-04T06:17:46.292922+0000","last_fullsized":"2024-03-04T06:17:46.292922+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5311540,"num_objects":29,"num_object_clones":20,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":180,"num_read_kb":163223,"num_write":131,"num_write_kb":34673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1619,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.355372+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:17:46.355372+0000","last_peered":"2024-03-04T06:17:46.355372+0000","last_clean":"2024-03-04T06:17:46.355372+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:17:46.355372+0000","last_undegraded":"2024-03-04T06:17:46.355372+0000","last_fullsized":"2024-03-04T06:17:46.355372+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1490,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.301750+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:17:46.301750+0000","last_peered":"2024-03-04T06:17:46.301750+0000","last_clean":"2024-03-04T06:17:46.301750+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:17:46.301750+0000","last_undegraded":"2024-03-04T06:17:46.301750+0000","last_fullsized":"2024-03-04T06:17:46.301750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"594'548","reported_seq":2104,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.298321+0000","last_change":"2024-03-04T06:17:44.394104+0000","last_active":"2024-03-04T06:17:46.298321+0000","last_peered":"2024-03-04T06:17:46.298321+0000","last_clean":"2024-03-04T06:17:46.298321+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:17:46.298321+0000","last_undegraded":"2024-03-04T06:17:46.298321+0000","last_fullsized":"2024-03-04T06:17:46.298321+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093016,"num_objects":54,"num_object_clones":42,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":285,"num_read_kb":227521,"num_write":243,"num_write_kb":65814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1479,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.300116+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:17:45.300116+0000","last_peered":"2024-03-04T06:17:45.300116+0000","last_clean":"2024-03-04T06:17:45.300116+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:17:45.300116+0000","last_undegraded":"2024-03-04T06:17:45.300116+0000","last_fullsized":"2024-03-04T06:17:45.300116+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"594'379","reported_seq":1985,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.361085+0000","last_change":"2024-03-04T06:17:44.392516+0000","last_active":"2024-03-04T06:17:46.361085+0000","last_peered":"2024-03-04T06:17:46.361085+0000","last_clean":"2024-03-04T06:17:46.361085+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:17:46.361085+0000","last_undegraded":"2024-03-04T06:17:46.361085+0000","last_fullsized":"2024-03-04T06:17:46.361085+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370051,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":219,"num_read_kb":177501,"num_write":146,"num_write_kb":41225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1490,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.287926+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:17:46.287926+0000","last_peered":"2024-03-04T06:17:46.287926+0000","last_clean":"2024-03-04T06:17:46.287926+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:17:46.287926+0000","last_undegraded":"2024-03-04T06:17:46.287926+0000","last_fullsized":"2024-03-04T06:17:46.287926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"594'960","reported_seq":2427,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.371285+0000","last_change":"2024-03-04T06:17:44.459636+0000","last_active":"2024-03-04T06:17:46.371285+0000","last_peered":"2024-03-04T06:17:46.371285+0000","last_clean":"2024-03-04T06:17:46.371285+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:17:46.371285+0000","last_undegraded":"2024-03-04T06:17:46.371285+0000","last_fullsized":"2024-03-04T06:17:46.371285+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":960,"log_dups_size":0,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13057488,"num_objects":97,"num_object_clones":75,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":574,"num_read_kb":452854,"num_write":460,"num_write_kb":123373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1479,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.286404+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:17:45.286404+0000","last_peered":"2024-03-04T06:17:45.286404+0000","last_clean":"2024-03-04T06:17:45.286404+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:17:45.286404+0000","last_undegraded":"2024-03-04T06:17:45.286404+0000","last_fullsized":"2024-03-04T06:17:45.286404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"594'1039","reported_seq":2385,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.285051+0000","last_change":"2024-03-04T06:17:44.391004+0000","last_active":"2024-03-04T06:17:44.285051+0000","last_peered":"2024-03-04T06:17:44.285051+0000","last_clean":"2024-03-04T06:17:44.285051+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:17:44.285051+0000","last_undegraded":"2024-03-04T06:17:44.285051+0000","last_fullsized":"2024-03-04T06:17:44.285051+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1039,"log_dups_size":0,"ondisk_log_size":1039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16253590,"num_objects":99,"num_object_clones":77,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":461,"num_read_kb":342096,"num_write":456,"num_write_kb":121062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"584'481","reported_seq":2123,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.312712+0000","last_change":"2024-03-04T06:17:44.309404+0000","last_active":"2024-03-04T06:17:46.312712+0000","last_peered":"2024-03-04T06:17:46.312712+0000","last_clean":"2024-03-04T06:17:46.312712+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:17:46.312712+0000","last_undegraded":"2024-03-04T06:17:46.312712+0000","last_fullsized":"2024-03-04T06:17:46.312712+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9614330,"num_objects":48,"num_object_clones":36,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":345,"num_read_kb":286484,"num_write":228,"num_write_kb":61730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1506,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.303474+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:17:44.303474+0000","last_peered":"2024-03-04T06:17:44.303474+0000","last_clean":"2024-03-04T06:17:44.303474+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:17:44.303474+0000","last_undegraded":"2024-03-04T06:17:44.303474+0000","last_fullsized":"2024-03-04T06:17:44.303474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"594'473","reported_seq":2078,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.436469+0000","last_change":"2024-03-04T06:17:44.439748+0000","last_active":"2024-03-04T06:17:44.436469+0000","last_peered":"2024-03-04T06:17:44.436469+0000","last_clean":"2024-03-04T06:17:44.436469+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:17:44.436469+0000","last_undegraded":"2024-03-04T06:17:44.436469+0000","last_fullsized":"2024-03-04T06:17:44.436469+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227505,"num_objects":55,"num_object_clones":40,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":239,"num_read_kb":167522,"num_write":222,"num_write_kb":58588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1486,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.369221+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:17:46.369221+0000","last_peered":"2024-03-04T06:17:46.369221+0000","last_clean":"2024-03-04T06:17:46.369221+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:17:46.369221+0000","last_undegraded":"2024-03-04T06:17:46.369221+0000","last_fullsized":"2024-03-04T06:17:46.369221+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"594'707","reported_seq":2185,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.360227+0000","last_change":"2024-03-04T06:17:44.422554+0000","last_active":"2024-03-04T06:17:46.360227+0000","last_peered":"2024-03-04T06:17:46.360227+0000","last_clean":"2024-03-04T06:17:46.360227+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:17:46.360227+0000","last_undegraded":"2024-03-04T06:17:46.360227+0000","last_fullsized":"2024-03-04T06:17:46.360227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":707,"log_dups_size":0,"ondisk_log_size":707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9312134,"num_objects":68,"num_object_clones":54,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":341,"num_read_kb":253708,"num_write":314,"num_write_kb":85989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1506,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.286029+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:17:44.286029+0000","last_peered":"2024-03-04T06:17:44.286029+0000","last_clean":"2024-03-04T06:17:44.286029+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:17:44.286029+0000","last_undegraded":"2024-03-04T06:17:44.286029+0000","last_fullsized":"2024-03-04T06:17:44.286029+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"594'825","reported_seq":2294,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.287590+0000","last_change":"2024-03-04T06:17:44.389931+0000","last_active":"2024-03-04T06:17:44.287590+0000","last_peered":"2024-03-04T06:17:44.287590+0000","last_clean":"2024-03-04T06:17:44.287590+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:17:44.287590+0000","last_undegraded":"2024-03-04T06:17:44.287590+0000","last_fullsized":"2024-03-04T06:17:44.287590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135050,"num_objects":74,"num_object_clones":65,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":470,"num_read_kb":390395,"num_write":340,"num_write_kb":93412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1490,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.300051+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:17:46.300051+0000","last_peered":"2024-03-04T06:17:46.300051+0000","last_clean":"2024-03-04T06:17:46.300051+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:17:46.300051+0000","last_undegraded":"2024-03-04T06:17:46.300051+0000","last_fullsized":"2024-03-04T06:17:46.300051+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":104613599,"num_objects":827,"num_object_clones":645,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":144,"num_read":4428,"num_read_kb":3461862,"num_write":3732,"num_write_kb":1014395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8322,"ondisk_log_size":8322,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":170596,"kb_used_data":121564,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94201244,"statfs":{"total":96636764160,"available":96462073856,"internally_reserved":0,"allocated":124481536,"data_stored":141550469,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":137468,"kb_used_data":92596,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94234372,"statfs":{"total":96636764160,"available":96495996928,"internally_reserved":0,"allocated":94818304,"data_stored":104684949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":143144,"kb_used_data":98272,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94228696,"statfs":{"total":96636764160,"available":96490184704,"internally_reserved":0,"allocated":100630528,"data_stored":115768468,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":122304,"kb_used_data":77432,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94249536,"statfs":{"total":96636764160,"available":96511524864,"internally_reserved":0,"allocated":79290368,"data_stored":87407372,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:17:48.934 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:17:48.934 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:17:48.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:17:49.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:49.038+0000 7f65e186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:49.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:49.044+0000 7f65e186d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:49.238 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:17:49.239 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:17:49.250 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":891,"stamp":"2024-03-04T06:17:48.834013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":105203967,"num_objects":829,"num_object_clones":645,"num_object_copies":2487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":144,"num_read":4531,"num_read_kb":3461953,"num_write":3869,"num_write_kb":1016289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8419,"ondisk_log_size":8419,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":573512,"kb_used_data":389864,"kb_used_omap":0,"kb_used_meta":183616,"kb_avail":376913848,"statfs":{"total":386547056640,"available":385959780352,"internally_reserved":0,"allocated":399220736,"data_stored":449411258,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":188022784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1358826,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":47,"num_read_kb":43555,"num_write":17,"num_write_kb":5238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000788"},"pg_stats":[{"pgid":"3.f","version":"594'509","reported_seq":2069,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.301056+0000","last_change":"2024-03-04T06:17:44.435888+0000","last_active":"2024-03-04T06:17:45.301056+0000","last_peered":"2024-03-04T06:17:45.301056+0000","last_clean":"2024-03-04T06:17:45.301056+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:17:45.301056+0000","last_undegraded":"2024-03-04T06:17:45.301056+0000","last_fullsized":"2024-03-04T06:17:45.301056+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151560,"num_objects":54,"num_object_clones":43,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":271,"num_read_kb":224365,"num_write":222,"num_write_kb":59334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"594'406","reported_seq":1994,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.283938+0000","last_change":"2024-03-04T06:17:44.360573+0000","last_active":"2024-03-04T06:17:44.283938+0000","last_peered":"2024-03-04T06:17:44.283938+0000","last_clean":"2024-03-04T06:17:44.283938+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:17:44.283938+0000","last_undegraded":"2024-03-04T06:17:44.283938+0000","last_fullsized":"2024-03-04T06:17:44.283938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000891,"num_objects":44,"num_object_clones":30,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":139,"num_read_kb":80773,"num_write":180,"num_write_kb":47650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"596'548","reported_seq":2146,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.308319+0000","last_change":"2024-03-04T06:17:44.371590+0000","last_active":"2024-03-04T06:17:46.308319+0000","last_peered":"2024-03-04T06:17:46.308319+0000","last_clean":"2024-03-04T06:17:46.308319+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:17:46.308319+0000","last_undegraded":"2024-03-04T06:17:46.308319+0000","last_fullsized":"2024-03-04T06:17:46.308319+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706191,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":346,"num_read_kb":275868,"num_write":260,"num_write_kb":71879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"594'393","reported_seq":2014,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.290965+0000","last_change":"2024-03-04T06:17:44.411376+0000","last_active":"2024-03-04T06:17:45.290965+0000","last_peered":"2024-03-04T06:17:45.290965+0000","last_clean":"2024-03-04T06:17:45.290965+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:17:45.290965+0000","last_undegraded":"2024-03-04T06:17:45.290965+0000","last_fullsized":"2024-03-04T06:17:45.290965+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7698398,"num_objects":38,"num_object_clones":28,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":211,"num_read_kb":164280,"num_write":187,"num_write_kb":51669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"594'188","reported_seq":1859,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.365715+0000","last_change":"2024-03-04T06:17:44.391978+0000","last_active":"2024-03-04T06:17:46.365715+0000","last_peered":"2024-03-04T06:17:46.365715+0000","last_clean":"2024-03-04T06:17:46.365715+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:17:46.365715+0000","last_undegraded":"2024-03-04T06:17:46.365715+0000","last_fullsized":"2024-03-04T06:17:46.365715+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403665,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":67,"num_read_kb":40142,"num_write":84,"num_write_kb":22301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"592'570","reported_seq":2105,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.312217+0000","last_change":"2024-03-04T06:17:44.385509+0000","last_active":"2024-03-04T06:17:45.312217+0000","last_peered":"2024-03-04T06:17:45.312217+0000","last_clean":"2024-03-04T06:17:45.312217+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:17:45.312217+0000","last_undegraded":"2024-03-04T06:17:45.312217+0000","last_fullsized":"2024-03-04T06:17:45.312217+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7278190,"num_objects":51,"num_object_clones":42,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":280,"num_read_kb":215130,"num_write":259,"num_write_kb":75696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1749,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.297310+0000","last_change":"2024-03-04T06:17:44.386007+0000","last_active":"2024-03-04T06:17:45.297310+0000","last_peered":"2024-03-04T06:17:45.297310+0000","last_clean":"2024-03-04T06:17:45.297310+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:17:45.297310+0000","last_undegraded":"2024-03-04T06:17:45.297310+0000","last_fullsized":"2024-03-04T06:17:45.297310+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"594'296","reported_seq":1958,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.292922+0000","last_change":"2024-03-04T06:17:44.401444+0000","last_active":"2024-03-04T06:17:46.292922+0000","last_peered":"2024-03-04T06:17:46.292922+0000","last_clean":"2024-03-04T06:17:46.292922+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:17:46.292922+0000","last_undegraded":"2024-03-04T06:17:46.292922+0000","last_fullsized":"2024-03-04T06:17:46.292922+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5311540,"num_objects":29,"num_object_clones":20,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":180,"num_read_kb":163223,"num_write":131,"num_write_kb":34673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1619,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.355372+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:17:46.355372+0000","last_peered":"2024-03-04T06:17:46.355372+0000","last_clean":"2024-03-04T06:17:46.355372+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:17:46.355372+0000","last_undegraded":"2024-03-04T06:17:46.355372+0000","last_fullsized":"2024-03-04T06:17:46.355372+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1490,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.301750+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:17:46.301750+0000","last_peered":"2024-03-04T06:17:46.301750+0000","last_clean":"2024-03-04T06:17:46.301750+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:17:46.301750+0000","last_undegraded":"2024-03-04T06:17:46.301750+0000","last_fullsized":"2024-03-04T06:17:46.301750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"594'548","reported_seq":2104,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.298321+0000","last_change":"2024-03-04T06:17:44.394104+0000","last_active":"2024-03-04T06:17:46.298321+0000","last_peered":"2024-03-04T06:17:46.298321+0000","last_clean":"2024-03-04T06:17:46.298321+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:17:46.298321+0000","last_undegraded":"2024-03-04T06:17:46.298321+0000","last_fullsized":"2024-03-04T06:17:46.298321+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093016,"num_objects":54,"num_object_clones":42,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":285,"num_read_kb":227521,"num_write":243,"num_write_kb":65814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1479,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.300116+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:17:45.300116+0000","last_peered":"2024-03-04T06:17:45.300116+0000","last_clean":"2024-03-04T06:17:45.300116+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:17:45.300116+0000","last_undegraded":"2024-03-04T06:17:45.300116+0000","last_fullsized":"2024-03-04T06:17:45.300116+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"594'379","reported_seq":1985,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.361085+0000","last_change":"2024-03-04T06:17:44.392516+0000","last_active":"2024-03-04T06:17:46.361085+0000","last_peered":"2024-03-04T06:17:46.361085+0000","last_clean":"2024-03-04T06:17:46.361085+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:17:46.361085+0000","last_undegraded":"2024-03-04T06:17:46.361085+0000","last_fullsized":"2024-03-04T06:17:46.361085+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370051,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":219,"num_read_kb":177501,"num_write":146,"num_write_kb":41225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1490,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.287926+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:17:46.287926+0000","last_peered":"2024-03-04T06:17:46.287926+0000","last_clean":"2024-03-04T06:17:46.287926+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:17:46.287926+0000","last_undegraded":"2024-03-04T06:17:46.287926+0000","last_fullsized":"2024-03-04T06:17:46.287926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"594'960","reported_seq":2427,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.371285+0000","last_change":"2024-03-04T06:17:44.459636+0000","last_active":"2024-03-04T06:17:46.371285+0000","last_peered":"2024-03-04T06:17:46.371285+0000","last_clean":"2024-03-04T06:17:46.371285+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:17:46.371285+0000","last_undegraded":"2024-03-04T06:17:46.371285+0000","last_fullsized":"2024-03-04T06:17:46.371285+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":960,"log_dups_size":0,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13057488,"num_objects":97,"num_object_clones":75,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":574,"num_read_kb":452854,"num_write":460,"num_write_kb":123373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1479,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-04T06:17:45.286404+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:17:45.286404+0000","last_peered":"2024-03-04T06:17:45.286404+0000","last_clean":"2024-03-04T06:17:45.286404+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:17:45.286404+0000","last_undegraded":"2024-03-04T06:17:45.286404+0000","last_fullsized":"2024-03-04T06:17:45.286404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"594'1039","reported_seq":2385,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.285051+0000","last_change":"2024-03-04T06:17:44.391004+0000","last_active":"2024-03-04T06:17:44.285051+0000","last_peered":"2024-03-04T06:17:44.285051+0000","last_clean":"2024-03-04T06:17:44.285051+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:17:44.285051+0000","last_undegraded":"2024-03-04T06:17:44.285051+0000","last_fullsized":"2024-03-04T06:17:44.285051+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1039,"log_dups_size":0,"ondisk_log_size":1039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16253590,"num_objects":99,"num_object_clones":77,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":461,"num_read_kb":342096,"num_write":456,"num_write_kb":121062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"584'481","reported_seq":2123,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.312712+0000","last_change":"2024-03-04T06:17:44.309404+0000","last_active":"2024-03-04T06:17:46.312712+0000","last_peered":"2024-03-04T06:17:46.312712+0000","last_clean":"2024-03-04T06:17:46.312712+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:17:46.312712+0000","last_undegraded":"2024-03-04T06:17:46.312712+0000","last_fullsized":"2024-03-04T06:17:46.312712+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9614330,"num_objects":48,"num_object_clones":36,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":345,"num_read_kb":286484,"num_write":228,"num_write_kb":61730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1506,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.303474+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:17:44.303474+0000","last_peered":"2024-03-04T06:17:44.303474+0000","last_clean":"2024-03-04T06:17:44.303474+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:17:44.303474+0000","last_undegraded":"2024-03-04T06:17:44.303474+0000","last_fullsized":"2024-03-04T06:17:44.303474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"594'473","reported_seq":2078,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.436469+0000","last_change":"2024-03-04T06:17:44.439748+0000","last_active":"2024-03-04T06:17:44.436469+0000","last_peered":"2024-03-04T06:17:44.436469+0000","last_clean":"2024-03-04T06:17:44.436469+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:17:44.436469+0000","last_undegraded":"2024-03-04T06:17:44.436469+0000","last_fullsized":"2024-03-04T06:17:44.436469+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227505,"num_objects":55,"num_object_clones":40,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":239,"num_read_kb":167522,"num_write":222,"num_write_kb":58588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1486,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.369221+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:17:46.369221+0000","last_peered":"2024-03-04T06:17:46.369221+0000","last_clean":"2024-03-04T06:17:46.369221+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:17:46.369221+0000","last_undegraded":"2024-03-04T06:17:46.369221+0000","last_fullsized":"2024-03-04T06:17:46.369221+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"594'707","reported_seq":2185,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.360227+0000","last_change":"2024-03-04T06:17:44.422554+0000","last_active":"2024-03-04T06:17:46.360227+0000","last_peered":"2024-03-04T06:17:46.360227+0000","last_clean":"2024-03-04T06:17:46.360227+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:17:46.360227+0000","last_undegraded":"2024-03-04T06:17:46.360227+0000","last_fullsized":"2024-03-04T06:17:46.360227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":707,"log_dups_size":0,"ondisk_log_size":707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9312134,"num_objects":68,"num_object_clones":54,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":341,"num_read_kb":253708,"num_write":314,"num_write_kb":85989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1506,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.286029+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:17:44.286029+0000","last_peered":"2024-03-04T06:17:44.286029+0000","last_clean":"2024-03-04T06:17:44.286029+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:17:44.286029+0000","last_undegraded":"2024-03-04T06:17:44.286029+0000","last_fullsized":"2024-03-04T06:17:44.286029+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"594'825","reported_seq":2294,"reported_epoch":594,"state":"active+clean","last_fresh":"2024-03-04T06:17:44.287590+0000","last_change":"2024-03-04T06:17:44.389931+0000","last_active":"2024-03-04T06:17:44.287590+0000","last_peered":"2024-03-04T06:17:44.287590+0000","last_clean":"2024-03-04T06:17:44.287590+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:17:44.287590+0000","last_undegraded":"2024-03-04T06:17:44.287590+0000","last_fullsized":"2024-03-04T06:17:44.287590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135050,"num_objects":74,"num_object_clones":65,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":470,"num_read_kb":390395,"num_write":340,"num_write_kb":93412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1490,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-04T06:17:46.300051+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:17:46.300051+0000","last_peered":"2024-03-04T06:17:46.300051+0000","last_clean":"2024-03-04T06:17:46.300051+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:17:46.300051+0000","last_undegraded":"2024-03-04T06:17:46.300051+0000","last_fullsized":"2024-03-04T06:17:46.300051+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":104613599,"num_objects":827,"num_object_clones":645,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":144,"num_read":4428,"num_read_kb":3461862,"num_write":3732,"num_write_kb":1014395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8322,"ondisk_log_size":8322,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":170596,"kb_used_data":121564,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94201244,"statfs":{"total":96636764160,"available":96462073856,"internally_reserved":0,"allocated":124481536,"data_stored":141550469,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706264,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":137468,"kb_used_data":92596,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94234372,"statfs":{"total":96636764160,"available":96495996928,"internally_reserved":0,"allocated":94818304,"data_stored":104684949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":143144,"kb_used_data":98272,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94228696,"statfs":{"total":96636764160,"available":96490184704,"internally_reserved":0,"allocated":100630528,"data_stored":115768468,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":122304,"kb_used_data":77432,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94249536,"statfs":{"total":96636764160,"available":96511524864,"internally_reserved":0,"allocated":79290368,"data_stored":87407372,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:17:49.291 INFO:tasks.rados.rados.0.smithi064.stdout:2437: done (2 left) 2024-03-04T06:17:49.291 INFO:tasks.rados.rados.0.smithi064.stdout:2438: write (excl) oid 41 current snap is 299 2024-03-04T06:17:49.291 INFO:tasks.rados.rados.0.smithi064.stdout:2434: finishing write tid 1 to smithi06431743-15 2024-03-04T06:17:49.292 INFO:tasks.rados.rados.0.smithi064.stdout:2438: seq_num 1128 ranges {558296=501116,1474512=600725,2760077=198219} 2024-03-04T06:17:49.297 INFO:tasks.rados.rados.0.smithi064.stdout:2438: writing smithi06431743-41 from 558296 to 1059412 tid 1 2024-03-04T06:17:49.302 INFO:tasks.rados.rados.0.smithi064.stdout:2438: writing smithi06431743-41 from 1474512 to 2075237 tid 2 2024-03-04T06:17:49.308 INFO:tasks.rados.rados.0.smithi064.stdout:2438: writing smithi06431743-41 from 2760077 to 2958296 tid 3 2024-03-04T06:17:49.308 INFO:tasks.rados.rados.0.smithi064.stdout:2439: snap_remove snap 270 2024-03-04T06:17:49.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:49.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:49.741+0000 7f554322e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:49.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:49.745+0000 7f554322e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:50.294 INFO:tasks.rados.rados.0.smithi064.stdout:2439: done (3 left) 2024-03-04T06:17:50.294 INFO:tasks.rados.rados.0.smithi064.stdout:2440: read oid 43 snap -1 2024-03-04T06:17:50.295 INFO:tasks.rados.rados.0.smithi064.stdout:2440: expect deleted 2024-03-04T06:17:50.295 INFO:tasks.rados.rados.0.smithi064.stdout:2441: snap_remove snap 284 2024-03-04T06:17:51.298 INFO:tasks.rados.rados.0.smithi064.stdout:2441: done (4 left) 2024-03-04T06:17:51.298 INFO:tasks.rados.rados.0.smithi064.stdout:2442: delete oid 1 current snap is 299 2024-03-04T06:17:51.299 INFO:tasks.rados.rados.0.smithi064.stdout:2434: finishing write tid 2 to smithi06431743-15 2024-03-04T06:17:51.299 INFO:tasks.rados.rados.0.smithi064.stdout:2434: finishing write tid 3 to smithi06431743-15 2024-03-04T06:17:51.299 INFO:tasks.rados.rados.0.smithi064.stdout:2434: finishing write tid 4 to smithi06431743-15 2024-03-04T06:17:51.299 INFO:tasks.rados.rados.0.smithi064.stdout:2434: oid 15 updating version 0 to 488 2024-03-04T06:17:51.299 INFO:tasks.rados.rados.0.smithi064.stdout:2434: oid 15 updating version 488 to 490 2024-03-04T06:17:51.299 INFO:tasks.rados.rados.0.smithi064.stdout:2434: oid 15 version 490 is already newer than 490 2024-03-04T06:17:51.299 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 490 (ObjNum 1126 snap 299 seq_num 1126) dirty exists 2024-03-04T06:17:51.299 INFO:tasks.rados.rados.0.smithi064.stdout:2434: left oid 15 (ObjNum 1126 snap 299 seq_num 1126) 2024-03-04T06:17:51.299 INFO:tasks.rados.rados.0.smithi064.stdout:2436: expect (ObjNum 1050 snap 268 seq_num 1050) 2024-03-04T06:17:51.517 INFO:tasks.rados.rados.0.smithi064.stdout:2438: finishing write tid 1 to smithi06431743-41 2024-03-04T06:17:51.517 INFO:tasks.rados.rados.0.smithi064.stdout:2438: finishing write tid 2 to smithi06431743-41 2024-03-04T06:17:51.517 INFO:tasks.rados.rados.0.smithi064.stdout:2438: finishing write tid 3 to smithi06431743-41 2024-03-04T06:17:51.517 INFO:tasks.rados.rados.0.smithi064.stdout:2438: finishing write tid 4 to smithi06431743-41 2024-03-04T06:17:51.517 INFO:tasks.rados.rados.0.smithi064.stdout:2438: finishing write tid 5 to smithi06431743-41 2024-03-04T06:17:51.517 INFO:tasks.rados.rados.0.smithi064.stdout:2438: oid 41 updating version 0 to 550 2024-03-04T06:17:51.517 INFO:tasks.rados.rados.0.smithi064.stdout:2438: oid 41 updating version 550 to 553 2024-03-04T06:17:51.517 INFO:tasks.rados.rados.0.smithi064.stdout:2438: oid 41 version 553 is already newer than 551 2024-03-04T06:17:51.517 INFO:tasks.rados.rados.0.smithi064.stdout:2438: oid 41 version 553 is already newer than 553 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 553 (ObjNum 1128 snap 299 seq_num 1128) dirty exists 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:2438: left oid 41 (ObjNum 1128 snap 299 seq_num 1128) 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:2434: done (4 left) 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:2436: done (3 left) 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:2438: done (2 left) 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:2442: done (1 left) 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:2443: read oid 42 snap 291 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:2443: expect deleted 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:2444: read oid 8 snap -1 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:2444: expect (ObjNum 1118 snap 296 seq_num 1118) 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:2445: read oid 14 snap -1 2024-03-04T06:17:51.518 INFO:tasks.rados.rados.0.smithi064.stdout:2445: expect (ObjNum 1082 snap 275 seq_num 1082) 2024-03-04T06:17:51.519 INFO:tasks.rados.rados.0.smithi064.stdout:2446: snap_create 2024-03-04T06:17:51.536 INFO:tasks.rados.rados.0.smithi064.stdout:2445: expect (ObjNum 1082 snap 275 seq_num 1082) 2024-03-04T06:17:51.622 INFO:tasks.rados.rados.0.smithi064.stdout:2444: expect (ObjNum 1118 snap 296 seq_num 1118) 2024-03-04T06:17:51.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:51.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:51.919+0000 7f036a95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:51.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:51.923+0000 7f036a95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:52.299 INFO:tasks.rados.rados.0.smithi064.stdout:2440: done (4 left) 2024-03-04T06:17:52.299 INFO:tasks.rados.rados.0.smithi064.stdout:2443: done (3 left) 2024-03-04T06:17:52.299 INFO:tasks.rados.rados.0.smithi064.stdout:2444: done (2 left) 2024-03-04T06:17:52.299 INFO:tasks.rados.rados.0.smithi064.stdout:2445: done (1 left) 2024-03-04T06:17:52.299 INFO:tasks.rados.rados.0.smithi064.stdout:2446: done (0 left) 2024-03-04T06:17:52.299 INFO:tasks.rados.rados.0.smithi064.stdout:2447: read oid 20 snap -1 2024-03-04T06:17:52.300 INFO:tasks.rados.rados.0.smithi064.stdout:2447: expect (ObjNum 1090 snap 279 seq_num 1090) 2024-03-04T06:17:52.300 INFO:tasks.rados.rados.0.smithi064.stdout:2448: write oid 31 current snap is 300 2024-03-04T06:17:52.300 INFO:tasks.rados.rados.0.smithi064.stdout:2448: seq_num 1130 ranges {591910=651614,1875123=604394} 2024-03-04T06:17:52.306 INFO:tasks.rados.rados.0.smithi064.stdout:2448: writing smithi06431743-31 from 591910 to 1243524 tid 1 2024-03-04T06:17:52.313 INFO:tasks.rados.rados.0.smithi064.stdout:2448: writing smithi06431743-31 from 1875123 to 2479517 tid 2 2024-03-04T06:17:52.314 INFO:tasks.rados.rados.0.smithi064.stdout:2449: write oid 15 current snap is 300 2024-03-04T06:17:52.314 INFO:tasks.rados.rados.0.smithi064.stdout:2449: seq_num 1131 ranges {608717=726863,1933605=748052} 2024-03-04T06:17:52.322 INFO:tasks.rados.rados.0.smithi064.stdout:2449: writing smithi06431743-15 from 608717 to 1335580 tid 1 2024-03-04T06:17:52.332 INFO:tasks.rados.rados.0.smithi064.stdout:2449: writing smithi06431743-15 from 1933605 to 2681657 tid 2 2024-03-04T06:17:52.332 INFO:tasks.rados.rados.0.smithi064.stdout:2450: write (excl) oid 28 current snap is 300 2024-03-04T06:17:52.332 INFO:tasks.rados.rados.0.smithi064.stdout:2450: seq_num 1132 ranges {625524=402112,1592087=608063,2869769=155755} 2024-03-04T06:17:52.339 INFO:tasks.rados.rados.0.smithi064.stdout:2450: writing smithi06431743-28 from 625524 to 1027636 tid 1 2024-03-04T06:17:52.347 INFO:tasks.rados.rados.0.smithi064.stdout:2450: writing smithi06431743-28 from 1592087 to 2200150 tid 2 2024-03-04T06:17:52.353 INFO:tasks.rados.rados.0.smithi064.stdout:2450: writing smithi06431743-28 from 2869769 to 3025524 tid 3 2024-03-04T06:17:52.353 INFO:tasks.rados.rados.0.smithi064.stdout:2451: read oid 37 snap -1 2024-03-04T06:17:52.353 INFO:tasks.rados.rados.0.smithi064.stdout:2451: expect (ObjNum 1050 snap 268 seq_num 1050) 2024-03-04T06:17:52.353 INFO:tasks.rados.rados.0.smithi064.stdout:2452: snap_create 2024-03-04T06:17:52.386 INFO:tasks.rados.rados.0.smithi064.stdout:2447: expect (ObjNum 1090 snap 279 seq_num 1090) 2024-03-04T06:17:52.476 INFO:tasks.rados.rados.0.smithi064.stdout:2448: finishing write tid 1 to smithi06431743-31 2024-03-04T06:17:52.476 INFO:tasks.rados.rados.0.smithi064.stdout:2450: finishing write tid 1 to smithi06431743-28 2024-03-04T06:17:52.476 INFO:tasks.rados.rados.0.smithi064.stdout:2449: finishing write tid 1 to smithi06431743-15 2024-03-04T06:17:52.476 INFO:tasks.rados.rados.0.smithi064.stdout:2450: finishing write tid 2 to smithi06431743-28 2024-03-04T06:17:52.477 INFO:tasks.rados.rados.0.smithi064.stdout:2451: expect (ObjNum 1050 snap 268 seq_num 1050) 2024-03-04T06:17:52.676 INFO:tasks.rados.rados.0.smithi064.stdout:2450: finishing write tid 3 to smithi06431743-28 2024-03-04T06:17:52.676 INFO:tasks.rados.rados.0.smithi064.stdout:2449: finishing write tid 2 to smithi06431743-15 2024-03-04T06:17:52.677 INFO:tasks.rados.rados.0.smithi064.stdout:2448: finishing write tid 2 to smithi06431743-31 2024-03-04T06:17:52.677 INFO:tasks.rados.rados.0.smithi064.stdout:2449: finishing write tid 3 to smithi06431743-15 2024-03-04T06:17:52.677 INFO:tasks.rados.rados.0.smithi064.stdout:2450: finishing write tid 4 to smithi06431743-28 2024-03-04T06:17:52.677 INFO:tasks.rados.rados.0.smithi064.stdout:2449: finishing write tid 4 to smithi06431743-15 2024-03-04T06:17:52.677 INFO:tasks.rados.rados.0.smithi064.stdout:2449: oid 15 updating version 0 to 508 2024-03-04T06:17:52.677 INFO:tasks.rados.rados.0.smithi064.stdout:2449: oid 15 version 508 is already newer than 507 2024-03-04T06:17:52.677 INFO:tasks.rados.rados.0.smithi064.stdout:2449: oid 15 version 508 is already newer than 507 2024-03-04T06:17:52.677 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 508 (ObjNum 1131 snap 300 seq_num 1131) dirty exists 2024-03-04T06:17:52.677 INFO:tasks.rados.rados.0.smithi064.stdout:2449: left oid 15 (ObjNum 1131 snap 300 seq_num 1131) 2024-03-04T06:17:52.677 INFO:tasks.rados.rados.0.smithi064.stdout:2448: finishing write tid 3 to smithi06431743-31 2024-03-04T06:17:52.678 INFO:tasks.rados.rados.0.smithi064.stdout:2450: finishing write tid 5 to smithi06431743-28 2024-03-04T06:17:52.678 INFO:tasks.rados.rados.0.smithi064.stdout:2450: oid 28 updating version 0 to 979 2024-03-04T06:17:52.678 INFO:tasks.rados.rados.0.smithi064.stdout:2450: oid 28 updating version 979 to 981 2024-03-04T06:17:52.678 INFO:tasks.rados.rados.0.smithi064.stdout:2450: oid 28 updating version 981 to 982 2024-03-04T06:17:52.678 INFO:tasks.rados.rados.0.smithi064.stdout:2450: oid 28 version 982 is already newer than 981 2024-03-04T06:17:52.678 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 982 (ObjNum 1132 snap 300 seq_num 1132) dirty exists 2024-03-04T06:17:52.678 INFO:tasks.rados.rados.0.smithi064.stdout:2450: left oid 28 (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:17:52.678 INFO:tasks.rados.rados.0.smithi064.stdout:2448: finishing write tid 4 to smithi06431743-31 2024-03-04T06:17:52.678 INFO:tasks.rados.rados.0.smithi064.stdout:2448: oid 31 updating version 0 to 194 2024-03-04T06:17:52.678 INFO:tasks.rados.rados.0.smithi064.stdout:2448: oid 31 version 194 is already newer than 193 2024-03-04T06:17:52.679 INFO:tasks.rados.rados.0.smithi064.stdout:2448: oid 31 version 194 is already newer than 192 2024-03-04T06:17:52.679 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 194 (ObjNum 1130 snap 300 seq_num 1130) dirty exists 2024-03-04T06:17:52.679 INFO:tasks.rados.rados.0.smithi064.stdout:2448: left oid 31 (ObjNum 1130 snap 300 seq_num 1130) 2024-03-04T06:17:53.304 INFO:tasks.rados.rados.0.smithi064.stdout:2447: done (5 left) 2024-03-04T06:17:53.304 INFO:tasks.rados.rados.0.smithi064.stdout:2448: done (4 left) 2024-03-04T06:17:53.305 INFO:tasks.rados.rados.0.smithi064.stdout:2449: done (3 left) 2024-03-04T06:17:53.305 INFO:tasks.rados.rados.0.smithi064.stdout:2450: done (2 left) 2024-03-04T06:17:53.305 INFO:tasks.rados.rados.0.smithi064.stdout:2451: done (1 left) 2024-03-04T06:17:53.305 INFO:tasks.rados.rados.0.smithi064.stdout:2452: done (0 left) 2024-03-04T06:17:53.305 INFO:tasks.rados.rados.0.smithi064.stdout:2453: snap_create 2024-03-04T06:17:53.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:54.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:54.103+0000 7f229e99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:54.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:54.107+0000 7f229e99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:54.253 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:17:54.253 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:17:54.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:17:54.305 INFO:tasks.rados.rados.0.smithi064.stdout:2453: done (0 left) 2024-03-04T06:17:54.305 INFO:tasks.rados.rados.0.smithi064.stdout:2454: snap_remove snap 297 2024-03-04T06:17:54.362 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:54.358+0000 7f333ea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:54.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:54.364+0000 7f333ea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:54.556 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:17:54.556 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:17:54.567 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":899,"stamp":"2024-03-04T06:17:54.303384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94802381,"num_objects":837,"num_object_clones":653,"num_object_copies":2511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4565,"num_read_kb":3488463,"num_write":3898,"num_write_kb":1022024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8510,"ondisk_log_size":8510,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":596400,"kb_used_data":412096,"kb_used_omap":0,"kb_used_meta":183616,"kb_avail":376890960,"statfs":{"total":386547056640,"available":385936343040,"internally_reserved":0,"allocated":421986304,"data_stored":486192996,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":188022784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10401586,"num_objects":8,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":4,"num_read":34,"num_read_kb":26510,"num_write":29,"num_write_kb":5735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010281"},"pg_stats":[{"pgid":"3.f","version":"597'511","reported_seq":2082,"reported_epoch":600,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:50.319358+0000","last_change":"2024-03-04T06:17:50.319358+0000","last_active":"2024-03-04T06:17:50.319358+0000","last_peered":"2024-03-04T06:17:50.319358+0000","last_clean":"2024-03-04T06:17:50.319358+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:17:50.319358+0000","last_undegraded":"2024-03-04T06:17:50.319358+0000","last_fullsized":"2024-03-04T06:17:50.319358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":44,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":271,"num_read_kb":224365,"num_write":223,"num_write_kb":59334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"594'406","reported_seq":2015,"reported_epoch":600,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:50.309366+0000","last_change":"2024-03-04T06:17:50.309366+0000","last_active":"2024-03-04T06:17:50.309366+0000","last_peered":"2024-03-04T06:17:50.309366+0000","last_clean":"2024-03-04T06:17:50.309366+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:17:50.309366+0000","last_undegraded":"2024-03-04T06:17:50.309366+0000","last_fullsized":"2024-03-04T06:17:50.309366+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000891,"num_objects":44,"num_object_clones":30,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":145,"num_read_kb":88539,"num_write":180,"num_write_kb":47650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"601'557","reported_seq":2171,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-03-04T06:17:51.319456+0000","last_change":"2024-03-04T06:17:51.366012+0000","last_active":"2024-03-04T06:17:51.319456+0000","last_peered":"2024-03-04T06:17:51.319456+0000","last_clean":"2024-03-04T06:17:51.319456+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:17:51.319456+0000","last_undegraded":"2024-03-04T06:17:51.319456+0000","last_fullsized":"2024-03-04T06:17:51.319456+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823840,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":353,"num_read_kb":282555,"num_write":263,"num_write_kb":72042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"597'397","reported_seq":2029,"reported_epoch":600,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:50.316661+0000","last_change":"2024-03-04T06:17:50.316661+0000","last_active":"2024-03-04T06:17:50.316661+0000","last_peered":"2024-03-04T06:17:50.316661+0000","last_clean":"2024-03-04T06:17:50.316661+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:17:50.316661+0000","last_undegraded":"2024-03-04T06:17:50.316661+0000","last_fullsized":"2024-03-04T06:17:50.316661+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7698398,"num_objects":38,"num_object_clones":28,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":217,"num_read_kb":169189,"num_write":187,"num_write_kb":51669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"600'190","reported_seq":1873,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-03-04T06:17:51.331794+0000","last_change":"2024-03-04T06:17:51.339312+0000","last_active":"2024-03-04T06:17:51.331794+0000","last_peered":"2024-03-04T06:17:51.331794+0000","last_clean":"2024-03-04T06:17:51.331794+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:17:51.331794+0000","last_undegraded":"2024-03-04T06:17:51.331794+0000","last_fullsized":"2024-03-04T06:17:51.331794+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403665,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":67,"num_read_kb":40142,"num_write":84,"num_write_kb":22301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"597'572","reported_seq":2119,"reported_epoch":600,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:50.326842+0000","last_change":"2024-03-04T06:17:50.326542+0000","last_active":"2024-03-04T06:17:50.326842+0000","last_peered":"2024-03-04T06:17:50.326842+0000","last_clean":"2024-03-04T06:17:50.326842+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:17:50.326842+0000","last_undegraded":"2024-03-04T06:17:50.326842+0000","last_fullsized":"2024-03-04T06:17:50.326842+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5244279,"num_objects":52,"num_object_clones":43,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":280,"num_read_kb":215130,"num_write":260,"num_write_kb":75696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1763,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-03-04T06:17:50.342027+0000","last_change":"2024-03-04T06:17:50.349876+0000","last_active":"2024-03-04T06:17:50.342027+0000","last_peered":"2024-03-04T06:17:50.342027+0000","last_clean":"2024-03-04T06:17:50.342027+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:17:50.342027+0000","last_undegraded":"2024-03-04T06:17:50.342027+0000","last_fullsized":"2024-03-04T06:17:50.342027+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"600'308","reported_seq":1977,"reported_epoch":601,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:51.320914+0000","last_change":"2024-03-04T06:17:51.320914+0000","last_active":"2024-03-04T06:17:51.320914+0000","last_peered":"2024-03-04T06:17:51.320914+0000","last_clean":"2024-03-04T06:17:51.320914+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:17:51.320914+0000","last_undegraded":"2024-03-04T06:17:51.320914+0000","last_fullsized":"2024-03-04T06:17:51.320914+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2638963,"num_objects":30,"num_object_clones":21,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":180,"num_read_kb":163223,"num_write":132,"num_write_kb":34673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1630,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-03-04T06:17:51.313285+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:17:51.313285+0000","last_peered":"2024-03-04T06:17:51.313285+0000","last_clean":"2024-03-04T06:17:51.313285+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:17:51.313285+0000","last_undegraded":"2024-03-04T06:17:51.313285+0000","last_fullsized":"2024-03-04T06:17:51.313285+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1505,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-03-04T06:17:51.330386+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:17:51.330386+0000","last_peered":"2024-03-04T06:17:51.330386+0000","last_clean":"2024-03-04T06:17:51.330386+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:17:51.330386+0000","last_undegraded":"2024-03-04T06:17:51.330386+0000","last_fullsized":"2024-03-04T06:17:51.330386+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"600'557","reported_seq":2127,"reported_epoch":601,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:51.339356+0000","last_change":"2024-03-04T06:17:51.339356+0000","last_active":"2024-03-04T06:17:51.339356+0000","last_peered":"2024-03-04T06:17:51.339356+0000","last_clean":"2024-03-04T06:17:51.339356+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:17:51.339356+0000","last_undegraded":"2024-03-04T06:17:51.339356+0000","last_fullsized":"2024-03-04T06:17:51.339356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496417,"num_objects":55,"num_object_clones":43,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":287,"num_read_kb":227522,"num_write":248,"num_write_kb":67085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1490,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-03-04T06:17:50.324037+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:17:50.324037+0000","last_peered":"2024-03-04T06:17:50.324037+0000","last_clean":"2024-03-04T06:17:50.324037+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:17:50.324037+0000","last_undegraded":"2024-03-04T06:17:50.324037+0000","last_fullsized":"2024-03-04T06:17:50.324037+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"600'381","reported_seq":1999,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-03-04T06:17:51.324915+0000","last_change":"2024-03-04T06:17:51.327388+0000","last_active":"2024-03-04T06:17:51.324915+0000","last_peered":"2024-03-04T06:17:51.324915+0000","last_clean":"2024-03-04T06:17:51.324915+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:17:51.324915+0000","last_undegraded":"2024-03-04T06:17:51.324915+0000","last_fullsized":"2024-03-04T06:17:51.324915+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370051,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":219,"num_read_kb":177501,"num_write":146,"num_write_kb":41225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1505,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-03-04T06:17:51.320210+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:17:51.320210+0000","last_peered":"2024-03-04T06:17:51.320210+0000","last_clean":"2024-03-04T06:17:51.320210+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:17:51.320210+0000","last_undegraded":"2024-03-04T06:17:51.320210+0000","last_fullsized":"2024-03-04T06:17:51.320210+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"600'971","reported_seq":2443,"reported_epoch":601,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:51.320632+0000","last_change":"2024-03-04T06:17:50.374659+0000","last_active":"2024-03-04T06:17:51.320632+0000","last_peered":"2024-03-04T06:17:51.320632+0000","last_clean":"2024-03-04T06:17:51.320632+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:17:51.320632+0000","last_undegraded":"2024-03-04T06:17:51.320632+0000","last_fullsized":"2024-03-04T06:17:51.320632+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11144097,"num_objects":98,"num_object_clones":76,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":576,"num_read_kb":452855,"num_write":464,"num_write_kb":123949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1490,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-03-04T06:17:50.311577+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:17:50.311577+0000","last_peered":"2024-03-04T06:17:50.311577+0000","last_clean":"2024-03-04T06:17:50.311577+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:17:50.311577+0000","last_undegraded":"2024-03-04T06:17:50.311577+0000","last_fullsized":"2024-03-04T06:17:50.311577+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"599'1050","reported_seq":2408,"reported_epoch":600,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:50.311957+0000","last_change":"2024-03-04T06:17:50.311957+0000","last_active":"2024-03-04T06:17:50.311957+0000","last_peered":"2024-03-04T06:17:50.311957+0000","last_clean":"2024-03-04T06:17:50.311957+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:17:50.311957+0000","last_undegraded":"2024-03-04T06:17:50.311957+0000","last_fullsized":"2024-03-04T06:17:50.311957+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15480501,"num_objects":100,"num_object_clones":78,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":462,"num_read_kb":342097,"num_write":461,"num_write_kb":122180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"600'498","reported_seq":2150,"reported_epoch":601,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:51.328559+0000","last_change":"2024-03-04T06:17:51.328559+0000","last_active":"2024-03-04T06:17:51.328559+0000","last_peered":"2024-03-04T06:17:51.328559+0000","last_clean":"2024-03-04T06:17:51.328559+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:17:51.328559+0000","last_undegraded":"2024-03-04T06:17:51.328559+0000","last_fullsized":"2024-03-04T06:17:51.328559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8236222,"num_objects":50,"num_object_clones":38,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":349,"num_read_kb":286486,"num_write":237,"num_write_kb":64337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1522,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-03-04T06:17:50.316070+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:17:50.316070+0000","last_peered":"2024-03-04T06:17:50.316070+0000","last_clean":"2024-03-04T06:17:50.316070+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:17:50.316070+0000","last_undegraded":"2024-03-04T06:17:50.316070+0000","last_fullsized":"2024-03-04T06:17:50.316070+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"599'477","reported_seq":2099,"reported_epoch":600,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:50.318828+0000","last_change":"2024-03-04T06:17:50.318828+0000","last_active":"2024-03-04T06:17:50.318828+0000","last_peered":"2024-03-04T06:17:50.318828+0000","last_clean":"2024-03-04T06:17:50.318828+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:17:50.318828+0000","last_undegraded":"2024-03-04T06:17:50.318828+0000","last_fullsized":"2024-03-04T06:17:50.318828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227505,"num_objects":55,"num_object_clones":40,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":245,"num_read_kb":174665,"num_write":222,"num_write_kb":58588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1497,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-03-04T06:17:51.322123+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:17:51.322123+0000","last_peered":"2024-03-04T06:17:51.322123+0000","last_clean":"2024-03-04T06:17:51.322123+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:17:51.322123+0000","last_undegraded":"2024-03-04T06:17:51.322123+0000","last_fullsized":"2024-03-04T06:17:51.322123+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"600'711","reported_seq":2199,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-03-04T06:17:51.323447+0000","last_change":"2024-03-04T06:17:51.326749+0000","last_active":"2024-03-04T06:17:51.323447+0000","last_peered":"2024-03-04T06:17:51.323447+0000","last_clean":"2024-03-04T06:17:51.323447+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:17:51.323447+0000","last_undegraded":"2024-03-04T06:17:51.323447+0000","last_fullsized":"2024-03-04T06:17:51.323447+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":711,"log_dups_size":0,"ondisk_log_size":711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9312134,"num_objects":68,"num_object_clones":54,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":341,"num_read_kb":253708,"num_write":314,"num_write_kb":85989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1522,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-03-04T06:17:50.304224+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:17:50.304224+0000","last_peered":"2024-03-04T06:17:50.304224+0000","last_clean":"2024-03-04T06:17:50.304224+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:17:50.304224+0000","last_undegraded":"2024-03-04T06:17:50.304224+0000","last_fullsized":"2024-03-04T06:17:50.304224+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"598'827","reported_seq":2312,"reported_epoch":600,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:17:50.308325+0000","last_change":"2024-03-04T06:17:50.308325+0000","last_active":"2024-03-04T06:17:50.308325+0000","last_peered":"2024-03-04T06:17:50.308325+0000","last_clean":"2024-03-04T06:17:50.308325+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:17:50.308325+0000","last_undegraded":"2024-03-04T06:17:50.308325+0000","last_fullsized":"2024-03-04T06:17:50.308325+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135050,"num_objects":74,"num_object_clones":65,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":470,"num_read_kb":390395,"num_write":340,"num_write_kb":93412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1505,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-03-04T06:17:51.335037+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:17:51.335037+0000","last_peered":"2024-03-04T06:17:51.335037+0000","last_clean":"2024-03-04T06:17:51.335037+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:17:51.335037+0000","last_undegraded":"2024-03-04T06:17:51.335037+0000","last_fullsized":"2024-03-04T06:17:51.335037+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94212013,"num_objects":835,"num_object_clones":653,"num_object_copies":2505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4462,"num_read_kb":3488372,"num_write":3761,"num_write_kb":1020130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8413,"ondisk_log_size":8413,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706269,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":178284,"kb_used_data":129044,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94193556,"statfs":{"total":96636764160,"available":96454201344,"internally_reserved":0,"allocated":132141056,"data_stored":153842545,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706269,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":144032,"kb_used_data":98824,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94227808,"statfs":{"total":96636764160,"available":96489275392,"internally_reserved":0,"allocated":101195776,"data_stored":114532962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":148128,"kb_used_data":103144,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94223712,"statfs":{"total":96636764160,"available":96485081088,"internally_reserved":0,"allocated":105619456,"data_stored":123999729,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":125956,"kb_used_data":81084,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94245884,"statfs":{"total":96636764160,"available":96507785216,"internally_reserved":0,"allocated":83030016,"data_stored":93817760,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:17:55.310 INFO:tasks.rados.rados.0.smithi064.stdout:2454: done (0 left) 2024-03-04T06:17:55.310 INFO:tasks.rados.rados.0.smithi064.stdout:2455: snap_create 2024-03-04T06:17:56.178 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:17:56.297 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:56.295+0000 7fe75526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:56.300 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:56.300+0000 7fe75526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:56.312 INFO:tasks.rados.rados.0.smithi064.stdout:2455: done (0 left) 2024-03-04T06:17:56.312 INFO:tasks.rados.rados.0.smithi064.stdout:2456: write (excl) oid 10 current snap is 303 2024-03-04T06:17:56.312 INFO:tasks.rados.rados.0.smithi064.stdout:2456: seq_num 1133 ranges {642331=593714,1766922=751721} 2024-03-04T06:17:56.321 INFO:tasks.rados.rados.0.smithi064.stdout:2456: writing smithi06431743-10 from 642331 to 1236045 tid 1 2024-03-04T06:17:56.330 INFO:tasks.rados.rados.0.smithi064.stdout:2456: writing smithi06431743-10 from 1766922 to 2518643 tid 2 2024-03-04T06:17:56.330 INFO:tasks.rados.rados.0.smithi064.stdout:2457: read oid 3 snap -1 2024-03-04T06:17:56.330 INFO:tasks.rados.rados.0.smithi064.stdout:2457: expect (ObjNum 1103 snap 288 seq_num 1103) 2024-03-04T06:17:56.330 INFO:tasks.rados.rados.0.smithi064.stdout:2458: snap_create 2024-03-04T06:17:56.362 INFO:tasks.rados.rados.0.smithi064.stdout:2457: expect (ObjNum 1103 snap 288 seq_num 1103) 2024-03-04T06:17:56.541 INFO:tasks.rados.rados.0.smithi064.stdout:2456: finishing write tid 1 to smithi06431743-10 2024-03-04T06:17:56.541 INFO:tasks.rados.rados.0.smithi064.stdout:2456: finishing write tid 2 to smithi06431743-10 2024-03-04T06:17:56.542 INFO:tasks.rados.rados.0.smithi064.stdout:2456: finishing write tid 3 to smithi06431743-10 2024-03-04T06:17:56.542 INFO:tasks.rados.rados.0.smithi064.stdout:2456: finishing write tid 4 to smithi06431743-10 2024-03-04T06:17:56.542 INFO:tasks.rados.rados.0.smithi064.stdout:2456: oid 10 updating version 0 to 584 2024-03-04T06:17:56.542 INFO:tasks.rados.rados.0.smithi064.stdout:2456: oid 10 version 584 is already newer than 583 2024-03-04T06:17:56.542 INFO:tasks.rados.rados.0.smithi064.stdout:2456: oid 10 version 584 is already newer than 580 2024-03-04T06:17:56.542 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 584 (ObjNum 1133 snap 303 seq_num 1133) dirty exists 2024-03-04T06:17:56.543 INFO:tasks.rados.rados.0.smithi064.stdout:2456: left oid 10 (ObjNum 1133 snap 303 seq_num 1133) 2024-03-04T06:17:57.320 INFO:tasks.rados.rados.0.smithi064.stdout:2456: done (2 left) 2024-03-04T06:17:57.320 INFO:tasks.rados.rados.0.smithi064.stdout:2457: done (1 left) 2024-03-04T06:17:57.321 INFO:tasks.rados.rados.0.smithi064.stdout:2458: done (0 left) 2024-03-04T06:17:57.321 INFO:tasks.rados.rados.0.smithi064.stdout:2459: write (excl) oid 32 current snap is 304 2024-03-04T06:17:57.321 INFO:tasks.rados.rados.0.smithi064.stdout:2459: seq_num 1134 ranges {659138=668963,2109051=611732} 2024-03-04T06:17:57.328 INFO:tasks.rados.rados.0.smithi064.stdout:2459: writing smithi06431743-32 from 659138 to 1328101 tid 1 2024-03-04T06:17:57.336 INFO:tasks.rados.rados.0.smithi064.stdout:2459: writing smithi06431743-32 from 2109051 to 2720783 tid 2 2024-03-04T06:17:57.337 INFO:tasks.rados.rados.0.smithi064.stdout:2460: read oid 4 snap 279 2024-03-04T06:17:57.337 INFO:tasks.rados.rados.0.smithi064.stdout:2460: expect (ObjNum 1088 snap 279 seq_num 1088) 2024-03-04T06:17:57.337 INFO:tasks.rados.rados.0.smithi064.stdout:2461: write (excl) oid 47 current snap is 304 2024-03-04T06:17:57.337 INFO:tasks.rados.rados.0.smithi064.stdout:2461: seq_num 1135 ranges {675945=744212,2167533=755390} 2024-03-04T06:17:57.344 INFO:tasks.rados.rados.0.smithi064.stdout:2461: writing smithi06431743-47 from 675945 to 1420157 tid 1 2024-03-04T06:17:57.354 INFO:tasks.rados.rados.0.smithi064.stdout:2461: writing smithi06431743-47 from 2167533 to 2922923 tid 2 2024-03-04T06:17:57.354 INFO:tasks.rados.rados.0.smithi064.stdout:2462: write oid 38 current snap is 304 2024-03-04T06:17:57.354 INFO:tasks.rados.rados.0.smithi064.stdout:2462: seq_num 1136 ranges {692752=419461,1826015=615401,2979461=113291} 2024-03-04T06:17:57.361 INFO:tasks.rados.rados.0.smithi064.stdout:2462: writing smithi06431743-38 from 692752 to 1112213 tid 1 2024-03-04T06:17:57.368 INFO:tasks.rados.rados.0.smithi064.stdout:2462: writing smithi06431743-38 from 1826015 to 2441416 tid 2 2024-03-04T06:17:57.373 INFO:tasks.rados.rados.0.smithi064.stdout:2462: writing smithi06431743-38 from 2979461 to 3092752 tid 3 2024-03-04T06:17:57.373 INFO:tasks.rados.rados.0.smithi064.stdout:2463: read oid 15 snap -1 2024-03-04T06:17:57.373 INFO:tasks.rados.rados.0.smithi064.stdout:2463: expect (ObjNum 1131 snap 300 seq_num 1131) 2024-03-04T06:17:57.373 INFO:tasks.rados.rados.0.smithi064.stdout:2464: read oid 42 snap -1 2024-03-04T06:17:57.373 INFO:tasks.rados.rados.0.smithi064.stdout:2464: expect deleted 2024-03-04T06:17:57.373 INFO:tasks.rados.rados.0.smithi064.stdout:2465: delete oid 29 current snap is 304 2024-03-04T06:17:57.399 INFO:tasks.rados.rados.0.smithi064.stdout:2463: expect (ObjNum 1131 snap 300 seq_num 1131) 2024-03-04T06:17:57.692 INFO:tasks.rados.rados.0.smithi064.stdout:2460: expect (ObjNum 1088 snap 279 seq_num 1088) 2024-03-04T06:17:57.800 INFO:tasks.rados.rados.0.smithi064.stdout:2459: finishing write tid 1 to smithi06431743-32 2024-03-04T06:17:57.800 INFO:tasks.rados.rados.0.smithi064.stdout:2462: finishing write tid 1 to smithi06431743-38 2024-03-04T06:17:57.800 INFO:tasks.rados.rados.0.smithi064.stdout:2461: finishing write tid 1 to smithi06431743-47 2024-03-04T06:17:57.800 INFO:tasks.rados.rados.0.smithi064.stdout:2462: finishing write tid 2 to smithi06431743-38 2024-03-04T06:17:57.800 INFO:tasks.rados.rados.0.smithi064.stdout:2459: finishing write tid 2 to smithi06431743-32 2024-03-04T06:17:57.800 INFO:tasks.rados.rados.0.smithi064.stdout:2459: finishing write tid 3 to smithi06431743-32 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2461: finishing write tid 2 to smithi06431743-47 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2460: done (6 left) 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2463: done (5 left) 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2464: done (4 left) 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2465: done (3 left) 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2466: read oid 16 snap -1 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2466: expect (ObjNum 1097 snap 286 seq_num 1097) 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2467: write oid 36 current snap is 304 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2462: finishing write tid 3 to smithi06431743-38 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2459: finishing write tid 4 to smithi06431743-32 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2459: oid 32 updating version 0 to 992 2024-03-04T06:17:57.801 INFO:tasks.rados.rados.0.smithi064.stdout:2459: oid 32 version 992 is already newer than 991 2024-03-04T06:17:57.802 INFO:tasks.rados.rados.0.smithi064.stdout:2459: oid 32 version 992 is already newer than 990 2024-03-04T06:17:57.802 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 992 (ObjNum 1134 snap 304 seq_num 1134) dirty exists 2024-03-04T06:17:57.802 INFO:tasks.rados.rados.0.smithi064.stdout:2459: left oid 32 (ObjNum 1134 snap 304 seq_num 1134) 2024-03-04T06:17:57.802 INFO:tasks.rados.rados.0.smithi064.stdout:2461: finishing write tid 3 to smithi06431743-47 2024-03-04T06:17:57.802 INFO:tasks.rados.rados.0.smithi064.stdout:2461: finishing write tid 4 to smithi06431743-47 2024-03-04T06:17:57.802 INFO:tasks.rados.rados.0.smithi064.stdout:2461: oid 47 updating version 0 to 407 2024-03-04T06:17:57.802 INFO:tasks.rados.rados.0.smithi064.stdout:2461: oid 47 updating version 407 to 408 2024-03-04T06:17:57.802 INFO:tasks.rados.rados.0.smithi064.stdout:2461: oid 47 updating version 408 to 409 2024-03-04T06:17:57.802 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 409 (ObjNum 1135 snap 304 seq_num 1135) dirty exists 2024-03-04T06:17:57.802 INFO:tasks.rados.rados.0.smithi064.stdout:2461: left oid 47 (ObjNum 1135 snap 304 seq_num 1135) 2024-03-04T06:17:57.802 INFO:tasks.rados.rados.0.smithi064.stdout:2462: finishing write tid 4 to smithi06431743-38 2024-03-04T06:17:57.803 INFO:tasks.rados.rados.0.smithi064.stdout:2467: seq_num 1138 ranges {726366=569959,1826626=502717,2801601=324765} 2024-03-04T06:17:57.806 INFO:tasks.rados.rados.0.smithi064.stdout:2467: writing smithi06431743-36 from 726366 to 1296325 tid 1 2024-03-04T06:17:57.811 INFO:tasks.rados.rados.0.smithi064.stdout:2467: writing smithi06431743-36 from 1826626 to 2329343 tid 2 2024-03-04T06:17:57.816 INFO:tasks.rados.rados.0.smithi064.stdout:2467: writing smithi06431743-36 from 2801601 to 3126366 tid 3 2024-03-04T06:17:57.816 INFO:tasks.rados.rados.0.smithi064.stdout:2459: done (4 left) 2024-03-04T06:17:57.816 INFO:tasks.rados.rados.0.smithi064.stdout:2461: done (3 left) 2024-03-04T06:17:57.816 INFO:tasks.rados.rados.0.smithi064.stdout:2468: snap_create 2024-03-04T06:17:57.817 INFO:tasks.rados.rados.0.smithi064.stdout:2462: finishing write tid 5 to smithi06431743-38 2024-03-04T06:17:57.817 INFO:tasks.rados.rados.0.smithi064.stdout:2462: oid 38 updating version 0 to 587 2024-03-04T06:17:57.817 INFO:tasks.rados.rados.0.smithi064.stdout:2462: oid 38 version 587 is already newer than 586 2024-03-04T06:17:57.817 INFO:tasks.rados.rados.0.smithi064.stdout:2462: oid 38 version 587 is already newer than 585 2024-03-04T06:17:57.817 INFO:tasks.rados.rados.0.smithi064.stdout:2462: oid 38 updating version 587 to 588 2024-03-04T06:17:57.817 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 588 (ObjNum 1136 snap 304 seq_num 1136) dirty exists 2024-03-04T06:17:57.817 INFO:tasks.rados.rados.0.smithi064.stdout:2462: left oid 38 (ObjNum 1136 snap 304 seq_num 1136) 2024-03-04T06:17:57.817 INFO:tasks.rados.rados.0.smithi064.stdout:2466: expect (ObjNum 1097 snap 286 seq_num 1097) 2024-03-04T06:17:57.977 INFO:tasks.rados.rados.0.smithi064.stdout:2467: finishing write tid 1 to smithi06431743-36 2024-03-04T06:17:57.977 INFO:tasks.rados.rados.0.smithi064.stdout:2467: finishing write tid 2 to smithi06431743-36 2024-03-04T06:17:57.977 INFO:tasks.rados.rados.0.smithi064.stdout:2467: finishing write tid 3 to smithi06431743-36 2024-03-04T06:17:57.977 INFO:tasks.rados.rados.0.smithi064.stdout:2467: finishing write tid 4 to smithi06431743-36 2024-03-04T06:17:57.977 INFO:tasks.rados.rados.0.smithi064.stdout:2467: finishing write tid 5 to smithi06431743-36 2024-03-04T06:17:57.977 INFO:tasks.rados.rados.0.smithi064.stdout:2467: oid 36 updating version 0 to 486 2024-03-04T06:17:57.977 INFO:tasks.rados.rados.0.smithi064.stdout:2467: oid 36 updating version 486 to 489 2024-03-04T06:17:57.977 INFO:tasks.rados.rados.0.smithi064.stdout:2467: oid 36 version 489 is already newer than 489 2024-03-04T06:17:57.977 INFO:tasks.rados.rados.0.smithi064.stdout:2467: oid 36 version 489 is already newer than 487 2024-03-04T06:17:57.977 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 489 (ObjNum 1138 snap 304 seq_num 1138) dirty exists 2024-03-04T06:17:57.978 INFO:tasks.rados.rados.0.smithi064.stdout:2467: left oid 36 (ObjNum 1138 snap 304 seq_num 1138) 2024-03-04T06:17:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:2462: done (3 left) 2024-03-04T06:17:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:2466: done (2 left) 2024-03-04T06:17:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:2467: done (1 left) 2024-03-04T06:17:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:2468: done (0 left) 2024-03-04T06:17:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:2469: read oid 8 snap -1 2024-03-04T06:17:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:2469: expect (ObjNum 1118 snap 296 seq_num 1118) 2024-03-04T06:17:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:2470: snap_create 2024-03-04T06:17:58.365 INFO:tasks.rados.rados.0.smithi064.stdout:2469: expect (ObjNum 1118 snap 296 seq_num 1118) 2024-03-04T06:17:58.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:17:58.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:58.555+0000 7fb70eb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:58.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:58.559+0000 7fb70eb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:59.322 INFO:tasks.rados.rados.0.smithi064.stdout:2469: done (1 left) 2024-03-04T06:17:59.322 INFO:tasks.rados.rados.0.smithi064.stdout:2470: done (0 left) 2024-03-04T06:17:59.322 INFO:tasks.rados.rados.0.smithi064.stdout:2471: delete oid 47 current snap is 306 2024-03-04T06:17:59.374 INFO:tasks.rados.rados.0.smithi064.stdout:2471: done (0 left) 2024-03-04T06:17:59.374 INFO:tasks.rados.rados.0.smithi064.stdout:2472: snap_remove snap 289 2024-03-04T06:17:59.569 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:17:59.570 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:17:59.570 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:17:59.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:17:59.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:59.688+0000 7f74d299d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:59.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:59.696+0000 7f74d299d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:59.709 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:59.709 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:59.709 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:59.709 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:59.709 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:59.709 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:59.710 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:59.710 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:59.710 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:59.710 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:59.710 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:59.710 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:59.712 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:59.713 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:59.713 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:17:59.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:59.817+0000 7f5277d10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:59.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:59.824+0000 7f5277d10640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:59.836 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:59.837 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:59.840 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:59.840 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:59.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:17:59.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:59.944+0000 7f22c295d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:59.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:17:59.951+0000 7f22c295d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:17:59.965 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:17:59.965 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:17:59.965 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:17:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:17:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:17:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:17:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:17:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:17:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:17:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:17:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:17:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:17:59.968 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:17:59.968 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:17:59.969 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:18:00.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:00.071+0000 7f8be055d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:00.080 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:00.079+0000 7f8be055d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:00.093 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:00.093 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:00.093 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:00.094 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:00.094 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:00.094 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:00.094 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:00.094 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:00.094 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:00.094 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:00.094 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:00.094 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:00.096 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:00.329 INFO:tasks.rados.rados.0.smithi064.stdout:2472: done (0 left) 2024-03-04T06:18:00.329 INFO:tasks.rados.rados.0.smithi064.stdout:2473: write (excl) oid 16 current snap is 306 2024-03-04T06:18:00.329 INFO:tasks.rados.rados.0.smithi064.stdout:2473: seq_num 1140 ranges {759980=720457,1943590=506386,2856447=303533} 2024-03-04T06:18:00.336 INFO:tasks.rados.rados.0.smithi064.stdout:2473: writing smithi06431743-16 from 759980 to 1480437 tid 1 2024-03-04T06:18:00.342 INFO:tasks.rados.rados.0.smithi064.stdout:2473: writing smithi06431743-16 from 1943590 to 2449976 tid 2 2024-03-04T06:18:00.346 INFO:tasks.rados.rados.0.smithi064.stdout:2473: writing smithi06431743-16 from 2856447 to 3159980 tid 3 2024-03-04T06:18:00.346 INFO:tasks.rados.rados.0.smithi064.stdout:2474: write oid 50 current snap is 306 2024-03-04T06:18:00.346 INFO:tasks.rados.rados.0.smithi064.stdout:2474: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2024-03-04T06:18:00.353 INFO:tasks.rados.rados.0.smithi064.stdout:2474: writing smithi06431743-50 from 776787 to 1288846 tid 1 2024-03-04T06:18:00.360 INFO:tasks.rados.rados.0.smithi064.stdout:2474: writing smithi06431743-50 from 1718425 to 2484822 tid 2 2024-03-04T06:18:00.364 INFO:tasks.rados.rados.0.smithi064.stdout:2474: writing smithi06431743-50 from 3000223 to 3176787 tid 3 2024-03-04T06:18:00.364 INFO:tasks.rados.rados.0.smithi064.stdout:2475: write oid 5 current snap is 306 2024-03-04T06:18:00.364 INFO:tasks.rados.rados.0.smithi064.stdout:2475: seq_num 1142 ranges {793594=587308,2060554=510055} 2024-03-04T06:18:00.371 INFO:tasks.rados.rados.0.smithi064.stdout:2475: writing smithi06431743-5 from 793594 to 1380902 tid 1 2024-03-04T06:18:00.378 INFO:tasks.rados.rados.0.smithi064.stdout:2475: writing smithi06431743-5 from 2060554 to 2570609 tid 2 2024-03-04T06:18:00.378 INFO:tasks.rados.rados.0.smithi064.stdout:2476: write oid 37 current snap is 306 2024-03-04T06:18:00.378 INFO:tasks.rados.rados.0.smithi064.stdout:2476: seq_num 1143 ranges {410401=662557,1719036=770066,2938716=271685} 2024-03-04T06:18:00.384 INFO:tasks.rados.rados.0.smithi064.stdout:2476: writing smithi06431743-37 from 410401 to 1072958 tid 1 2024-03-04T06:18:00.391 INFO:tasks.rados.rados.0.smithi064.stdout:2476: writing smithi06431743-37 from 1719036 to 2489102 tid 2 2024-03-04T06:18:00.396 INFO:tasks.rados.rados.0.smithi064.stdout:2476: writing smithi06431743-37 from 2938716 to 3210401 tid 3 2024-03-04T06:18:00.396 INFO:tasks.rados.rados.0.smithi064.stdout:2477: snap_create 2024-03-04T06:18:00.459 INFO:tasks.rados.rados.0.smithi064.stdout:2473: finishing write tid 1 to smithi06431743-16 2024-03-04T06:18:00.481 INFO:tasks.rados.rados.0.smithi064.stdout:2475: finishing write tid 1 to smithi06431743-5 2024-03-04T06:18:00.492 INFO:tasks.rados.rados.0.smithi064.stdout:2473: finishing write tid 2 to smithi06431743-16 2024-03-04T06:18:00.497 INFO:tasks.rados.rados.0.smithi064.stdout:2476: finishing write tid 1 to smithi06431743-37 2024-03-04T06:18:00.502 INFO:tasks.rados.rados.0.smithi064.stdout:2474: finishing write tid 1 to smithi06431743-50 2024-03-04T06:18:00.513 INFO:tasks.rados.rados.0.smithi064.stdout:2475: finishing write tid 2 to smithi06431743-5 2024-03-04T06:18:00.516 INFO:tasks.rados.rados.0.smithi064.stdout:2473: finishing write tid 3 to smithi06431743-16 2024-03-04T06:18:00.521 INFO:tasks.rados.rados.0.smithi064.stdout:2476: finishing write tid 2 to smithi06431743-37 2024-03-04T06:18:00.532 INFO:tasks.rados.rados.0.smithi064.stdout:2474: finishing write tid 2 to smithi06431743-50 2024-03-04T06:18:00.539 INFO:tasks.rados.rados.0.smithi064.stdout:2475: finishing write tid 3 to smithi06431743-5 2024-03-04T06:18:00.539 INFO:tasks.rados.rados.0.smithi064.stdout:2473: finishing write tid 4 to smithi06431743-16 2024-03-04T06:18:00.540 INFO:tasks.rados.rados.0.smithi064.stdout:2473: finishing write tid 5 to smithi06431743-16 2024-03-04T06:18:00.541 INFO:tasks.rados.rados.0.smithi064.stdout:2473: oid 16 updating version 0 to 495 2024-03-04T06:18:00.541 INFO:tasks.rados.rados.0.smithi064.stdout:2473: oid 16 version 495 is already newer than 494 2024-03-04T06:18:00.541 INFO:tasks.rados.rados.0.smithi064.stdout:2473: oid 16 updating version 495 to 496 2024-03-04T06:18:00.541 INFO:tasks.rados.rados.0.smithi064.stdout:2473: oid 16 version 496 is already newer than 493 2024-03-04T06:18:00.541 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 496 (ObjNum 1140 snap 306 seq_num 1140) dirty exists 2024-03-04T06:18:00.541 INFO:tasks.rados.rados.0.smithi064.stdout:2473: left oid 16 (ObjNum 1140 snap 306 seq_num 1140) 2024-03-04T06:18:00.541 INFO:tasks.rados.rados.0.smithi064.stdout:2476: finishing write tid 3 to smithi06431743-37 2024-03-04T06:18:00.545 INFO:tasks.rados.rados.0.smithi064.stdout:2475: finishing write tid 4 to smithi06431743-5 2024-03-04T06:18:00.545 INFO:tasks.rados.rados.0.smithi064.stdout:2475: oid 5 updating version 0 to 1072 2024-03-04T06:18:00.545 INFO:tasks.rados.rados.0.smithi064.stdout:2475: oid 5 updating version 1072 to 1073 2024-03-04T06:18:00.545 INFO:tasks.rados.rados.0.smithi064.stdout:2475: oid 5 updating version 1073 to 1074 2024-03-04T06:18:00.545 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 1074 (ObjNum 1142 snap 306 seq_num 1142) dirty exists 2024-03-04T06:18:00.545 INFO:tasks.rados.rados.0.smithi064.stdout:2475: left oid 5 (ObjNum 1142 snap 306 seq_num 1142) 2024-03-04T06:18:00.547 INFO:tasks.rados.rados.0.smithi064.stdout:2474: finishing write tid 3 to smithi06431743-50 2024-03-04T06:18:00.556 INFO:tasks.rados.rados.0.smithi064.stdout:2476: finishing write tid 4 to smithi06431743-37 2024-03-04T06:18:00.561 INFO:tasks.rados.rados.0.smithi064.stdout:2474: finishing write tid 4 to smithi06431743-50 2024-03-04T06:18:00.561 INFO:tasks.rados.rados.0.smithi064.stdout:2476: finishing write tid 5 to smithi06431743-37 2024-03-04T06:18:00.561 INFO:tasks.rados.rados.0.smithi064.stdout:2476: oid 37 updating version 0 to 415 2024-03-04T06:18:00.561 INFO:tasks.rados.rados.0.smithi064.stdout:2476: oid 37 version 415 is already newer than 412 2024-03-04T06:18:00.561 INFO:tasks.rados.rados.0.smithi064.stdout:2476: oid 37 version 415 is already newer than 414 2024-03-04T06:18:00.561 INFO:tasks.rados.rados.0.smithi064.stdout:2476: oid 37 version 415 is already newer than 413 2024-03-04T06:18:00.561 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 415 (ObjNum 1143 snap 306 seq_num 1143) dirty exists 2024-03-04T06:18:00.561 INFO:tasks.rados.rados.0.smithi064.stdout:2476: left oid 37 (ObjNum 1143 snap 306 seq_num 1143) 2024-03-04T06:18:00.562 INFO:tasks.rados.rados.0.smithi064.stdout:2474: finishing write tid 5 to smithi06431743-50 2024-03-04T06:18:00.562 INFO:tasks.rados.rados.0.smithi064.stdout:2474: oid 50 updating version 0 to 838 2024-03-04T06:18:00.562 INFO:tasks.rados.rados.0.smithi064.stdout:2474: oid 50 version 838 is already newer than 837 2024-03-04T06:18:00.563 INFO:tasks.rados.rados.0.smithi064.stdout:2474: oid 50 version 838 is already newer than 835 2024-03-04T06:18:00.563 INFO:tasks.rados.rados.0.smithi064.stdout:2474: oid 50 version 838 is already newer than 836 2024-03-04T06:18:00.563 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 838 (ObjNum 1141 snap 306 seq_num 1141) dirty exists 2024-03-04T06:18:00.563 INFO:tasks.rados.rados.0.smithi064.stdout:2474: left oid 50 (ObjNum 1141 snap 306 seq_num 1141) 2024-03-04T06:18:00.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:00.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:00.731+0000 7f7a4d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:00.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:00.735+0000 7f7a4d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:01.336 INFO:tasks.rados.rados.0.smithi064.stdout:2473: done (4 left) 2024-03-04T06:18:01.336 INFO:tasks.rados.rados.0.smithi064.stdout:2474: done (3 left) 2024-03-04T06:18:01.337 INFO:tasks.rados.rados.0.smithi064.stdout:2475: done (2 left) 2024-03-04T06:18:01.337 INFO:tasks.rados.rados.0.smithi064.stdout:2476: done (1 left) 2024-03-04T06:18:01.337 INFO:tasks.rados.rados.0.smithi064.stdout:2477: done (0 left) 2024-03-04T06:18:01.337 INFO:tasks.rados.rados.0.smithi064.stdout:2478: read oid 31 snap -1 2024-03-04T06:18:01.337 INFO:tasks.rados.rados.0.smithi064.stdout:2478: expect (ObjNum 1130 snap 300 seq_num 1130) 2024-03-04T06:18:01.337 INFO:tasks.rados.rados.0.smithi064.stdout:2479: read oid 44 snap -1 2024-03-04T06:18:01.337 INFO:tasks.rados.rados.0.smithi064.stdout:2479: expect (ObjNum 993 snap 254 seq_num 993) 2024-03-04T06:18:01.337 INFO:tasks.rados.rados.0.smithi064.stdout:2480: snap_remove snap 298 2024-03-04T06:18:02.337 INFO:tasks.rados.rados.0.smithi064.stdout:2480: done (2 left) 2024-03-04T06:18:02.337 INFO:tasks.rados.rados.0.smithi064.stdout:2481: write oid 46 current snap is 307 2024-03-04T06:18:02.337 INFO:tasks.rados.rados.0.smithi064.stdout:2481: seq_num 1144 ranges {427208=737806,1777518=513724,2966139=261069} 2024-03-04T06:18:02.343 INFO:tasks.rados.rados.0.smithi064.stdout:2481: writing smithi06431743-46 from 427208 to 1165014 tid 1 2024-03-04T06:18:02.349 INFO:tasks.rados.rados.0.smithi064.stdout:2481: writing smithi06431743-46 from 1777518 to 2291242 tid 2 2024-03-04T06:18:02.354 INFO:tasks.rados.rados.0.smithi064.stdout:2481: writing smithi06431743-46 from 2966139 to 3227208 tid 3 2024-03-04T06:18:02.355 INFO:tasks.rados.rados.0.smithi064.stdout:2482: snap_create 2024-03-04T06:18:02.355 INFO:tasks.rados.rados.0.smithi064.stdout:2478: expect (ObjNum 1130 snap 300 seq_num 1130) 2024-03-04T06:18:02.533 INFO:tasks.rados.rados.0.smithi064.stdout:2479: expect (ObjNum 993 snap 254 seq_num 993) 2024-03-04T06:18:02.696 INFO:tasks.rados.rados.0.smithi064.stdout:2481: finishing write tid 1 to smithi06431743-46 2024-03-04T06:18:02.696 INFO:tasks.rados.rados.0.smithi064.stdout:2481: finishing write tid 2 to smithi06431743-46 2024-03-04T06:18:02.696 INFO:tasks.rados.rados.0.smithi064.stdout:2481: finishing write tid 3 to smithi06431743-46 2024-03-04T06:18:02.696 INFO:tasks.rados.rados.0.smithi064.stdout:2481: finishing write tid 4 to smithi06431743-46 2024-03-04T06:18:02.696 INFO:tasks.rados.rados.0.smithi064.stdout:2481: finishing write tid 5 to smithi06431743-46 2024-03-04T06:18:02.697 INFO:tasks.rados.rados.0.smithi064.stdout:2481: oid 46 updating version 0 to 1006 2024-03-04T06:18:02.697 INFO:tasks.rados.rados.0.smithi064.stdout:2481: oid 46 version 1006 is already newer than 1005 2024-03-04T06:18:02.697 INFO:tasks.rados.rados.0.smithi064.stdout:2481: oid 46 version 1006 is already newer than 1005 2024-03-04T06:18:02.697 INFO:tasks.rados.rados.0.smithi064.stdout:2481: oid 46 version 1006 is already newer than 1003 2024-03-04T06:18:02.697 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 1006 (ObjNum 1144 snap 307 seq_num 1144) dirty exists 2024-03-04T06:18:02.697 INFO:tasks.rados.rados.0.smithi064.stdout:2481: left oid 46 (ObjNum 1144 snap 307 seq_num 1144) 2024-03-04T06:18:02.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:02.905+0000 7f4730b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:02.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:02.910+0000 7f4730b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:03.344 INFO:tasks.rados.rados.0.smithi064.stdout:2478: done (3 left) 2024-03-04T06:18:03.344 INFO:tasks.rados.rados.0.smithi064.stdout:2479: done (2 left) 2024-03-04T06:18:03.344 INFO:tasks.rados.rados.0.smithi064.stdout:2481: done (1 left) 2024-03-04T06:18:03.344 INFO:tasks.rados.rados.0.smithi064.stdout:2482: done (0 left) 2024-03-04T06:18:03.345 INFO:tasks.rados.rados.0.smithi064.stdout:2483: snap_remove snap 263 2024-03-04T06:18:04.344 INFO:tasks.rados.rados.0.smithi064.stdout:2483: done (0 left) 2024-03-04T06:18:04.345 INFO:tasks.rados.rados.0.smithi064.stdout:2484: write (excl) oid 25 current snap is 308 2024-03-04T06:18:04.345 INFO:tasks.rados.rados.0.smithi064.stdout:2484: seq_num 1145 ranges {444015=413055,1436000=773735,2709915=534100} 2024-03-04T06:18:04.349 INFO:tasks.rados.rados.0.smithi064.stdout:2484: writing smithi06431743-25 from 444015 to 857070 tid 1 2024-03-04T06:18:04.359 INFO:tasks.rados.rados.0.smithi064.stdout:2484: writing smithi06431743-25 from 1436000 to 2209735 tid 2 2024-03-04T06:18:04.366 INFO:tasks.rados.rados.0.smithi064.stdout:2484: writing smithi06431743-25 from 2709915 to 3244015 tid 3 2024-03-04T06:18:04.366 INFO:tasks.rados.rados.0.smithi064.stdout:2485: snap_remove snap 304 2024-03-04T06:18:04.977 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:05.085 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:05.083+0000 7f8a8566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:05.088 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:05.088+0000 7f8a8566d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:05.097 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:18:05.097 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:18:05.097 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:18:05.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:05.202+0000 7f871f51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:05.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:05.208+0000 7f871f51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:05.350 INFO:tasks.rados.rados.0.smithi064.stdout:2485: done (1 left) 2024-03-04T06:18:05.350 INFO:tasks.rados.rados.0.smithi064.stdout:2486: read oid 6 snap -1 2024-03-04T06:18:05.350 INFO:tasks.rados.rados.0.smithi064.stdout:2486: expect (ObjNum 1096 snap 286 seq_num 1096) 2024-03-04T06:18:05.351 INFO:tasks.rados.rados.0.smithi064.stdout:2487: read oid 9 snap -1 2024-03-04T06:18:05.351 INFO:tasks.rados.rados.0.smithi064.stdout:2487: expect deleted 2024-03-04T06:18:05.351 INFO:tasks.rados.rados.0.smithi064.stdout:2488: delete oid 3 current snap is 308 2024-03-04T06:18:05.351 INFO:tasks.rados.rados.0.smithi064.stdout:2484: finishing write tid 1 to smithi06431743-25 2024-03-04T06:18:05.351 INFO:tasks.rados.rados.0.smithi064.stdout:2484: finishing write tid 2 to smithi06431743-25 2024-03-04T06:18:05.351 INFO:tasks.rados.rados.0.smithi064.stdout:2484: finishing write tid 3 to smithi06431743-25 2024-03-04T06:18:05.351 INFO:tasks.rados.rados.0.smithi064.stdout:2484: finishing write tid 4 to smithi06431743-25 2024-03-04T06:18:05.351 INFO:tasks.rados.rados.0.smithi064.stdout:2484: finishing write tid 5 to smithi06431743-25 2024-03-04T06:18:05.351 INFO:tasks.rados.rados.0.smithi064.stdout:2484: oid 25 updating version 0 to 567 2024-03-04T06:18:05.351 INFO:tasks.rados.rados.0.smithi064.stdout:2484: oid 25 version 567 is already newer than 565 2024-03-04T06:18:05.351 INFO:tasks.rados.rados.0.smithi064.stdout:2484: oid 25 version 567 is already newer than 564 2024-03-04T06:18:05.352 INFO:tasks.rados.rados.0.smithi064.stdout:2484: oid 25 version 567 is already newer than 567 2024-03-04T06:18:05.352 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 567 (ObjNum 1145 snap 308 seq_num 1145) dirty exists 2024-03-04T06:18:05.352 INFO:tasks.rados.rados.0.smithi064.stdout:2484: left oid 25 (ObjNum 1145 snap 308 seq_num 1145) 2024-03-04T06:18:05.411 INFO:tasks.rados.rados.0.smithi064.stdout:2486: expect (ObjNum 1096 snap 286 seq_num 1096) 2024-03-04T06:18:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:18:05.415 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:18:05.426 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":916,"stamp":"2024-03-04T06:18:05.348092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":97844448,"num_objects":846,"num_object_clones":661,"num_object_copies":2538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":149,"num_read":4639,"num_read_kb":3554071,"num_write":3935,"num_write_kb":1032353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8647,"ondisk_log_size":8647,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":626416,"kb_used_data":442216,"kb_used_omap":0,"kb_used_meta":183616,"kb_avail":376860944,"statfs":{"total":386547056640,"available":385905606656,"internally_reserved":0,"allocated":452829184,"data_stored":528817482,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":188022784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":201684,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":15781,"num_write":4,"num_write_kb":1252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013522"},"pg_stats":[{"pgid":"3.f","version":"606'521","reported_seq":2109,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-03-04T06:17:59.353346+0000","last_change":"2024-03-04T06:17:56.416464+0000","last_active":"2024-03-04T06:17:59.353346+0000","last_peered":"2024-03-04T06:17:59.353346+0000","last_clean":"2024-03-04T06:17:59.353346+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:17:59.353346+0000","last_undegraded":"2024-03-04T06:17:59.353346+0000","last_fullsized":"2024-03-04T06:17:59.353346+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":44,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":271,"num_read_kb":224365,"num_write":223,"num_write_kb":59334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"600'410","reported_seq":2045,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-03-04T06:17:59.339128+0000","last_change":"2024-03-04T06:17:56.409422+0000","last_active":"2024-03-04T06:17:59.339128+0000","last_peered":"2024-03-04T06:17:59.339128+0000","last_clean":"2024-03-04T06:17:59.339128+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:17:59.339128+0000","last_undegraded":"2024-03-04T06:17:59.339128+0000","last_fullsized":"2024-03-04T06:17:59.339128+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000891,"num_objects":44,"num_object_clones":30,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":151,"num_read_kb":97765,"num_write":180,"num_write_kb":47650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"605'559","reported_seq":2198,"reported_epoch":611,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:01.356194+0000","last_change":"2024-03-04T06:18:01.356194+0000","last_active":"2024-03-04T06:18:01.356194+0000","last_peered":"2024-03-04T06:18:01.356194+0000","last_clean":"2024-03-04T06:18:01.356194+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:18:01.356194+0000","last_undegraded":"2024-03-04T06:18:01.356194+0000","last_fullsized":"2024-03-04T06:18:01.356194+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823840,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":353,"num_read_kb":282555,"num_write":263,"num_write_kb":72042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"607'409","reported_seq":2061,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-03-04T06:17:59.342861+0000","last_change":"2024-03-04T06:17:56.372611+0000","last_active":"2024-03-04T06:17:59.342861+0000","last_peered":"2024-03-04T06:17:59.342861+0000","last_clean":"2024-03-04T06:17:59.342861+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:17:59.342861+0000","last_undegraded":"2024-03-04T06:17:59.342861+0000","last_fullsized":"2024-03-04T06:17:59.342861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5345154,"num_objects":40,"num_object_clones":30,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":220,"num_read_kb":169190,"num_write":192,"num_write_kb":53134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"610'198","reported_seq":1907,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-03-04T06:18:01.384537+0000","last_change":"2024-03-04T06:18:00.402769+0000","last_active":"2024-03-04T06:18:01.384537+0000","last_peered":"2024-03-04T06:18:01.384537+0000","last_clean":"2024-03-04T06:18:01.384537+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:18:01.384537+0000","last_undegraded":"2024-03-04T06:18:01.384537+0000","last_fullsized":"2024-03-04T06:18:01.384537+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2991910,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":74,"num_read_kb":47657,"num_write":88,"num_write_kb":23529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"607'588","reported_seq":2154,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-03-04T06:17:59.351359+0000","last_change":"2024-03-04T06:17:56.463923+0000","last_active":"2024-03-04T06:17:59.351359+0000","last_peered":"2024-03-04T06:17:59.351359+0000","last_clean":"2024-03-04T06:17:59.351359+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:17:59.351359+0000","last_undegraded":"2024-03-04T06:17:59.351359+0000","last_fullsized":"2024-03-04T06:17:59.351359+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8824434,"num_objects":53,"num_object_clones":44,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":283,"num_read_kb":215132,"num_write":269,"num_write_kb":78133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1788,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-03-04T06:17:59.349886+0000","last_change":"2024-03-04T06:17:56.383958+0000","last_active":"2024-03-04T06:17:59.349886+0000","last_peered":"2024-03-04T06:17:59.349886+0000","last_clean":"2024-03-04T06:17:59.349886+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:17:59.349886+0000","last_undegraded":"2024-03-04T06:17:59.349886+0000","last_fullsized":"2024-03-04T06:17:59.349886+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"605'314","reported_seq":2005,"reported_epoch":611,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:01.367718+0000","last_change":"2024-03-04T06:18:01.367718+0000","last_active":"2024-03-04T06:18:01.367718+0000","last_peered":"2024-03-04T06:18:01.367718+0000","last_clean":"2024-03-04T06:18:01.367718+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:18:01.367718+0000","last_undegraded":"2024-03-04T06:18:01.367718+0000","last_fullsized":"2024-03-04T06:18:01.367718+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2638963,"num_objects":30,"num_object_clones":21,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":180,"num_read_kb":163223,"num_write":132,"num_write_kb":34673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1654,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-03-04T06:18:01.351071+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:18:01.351071+0000","last_peered":"2024-03-04T06:18:01.351071+0000","last_clean":"2024-03-04T06:18:01.351071+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:18:01.351071+0000","last_undegraded":"2024-03-04T06:18:01.351071+0000","last_fullsized":"2024-03-04T06:18:01.351071+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1528,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-03-04T06:18:01.356709+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:18:01.356709+0000","last_peered":"2024-03-04T06:18:01.356709+0000","last_clean":"2024-03-04T06:18:01.356709+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:18:01.356709+0000","last_undegraded":"2024-03-04T06:18:01.356709+0000","last_fullsized":"2024-03-04T06:18:01.356709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"605'563","reported_seq":2154,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-03-04T06:17:58.466739+0000","last_change":"2024-03-04T06:17:55.357041+0000","last_active":"2024-03-04T06:17:58.466739+0000","last_peered":"2024-03-04T06:17:58.466739+0000","last_clean":"2024-03-04T06:17:58.466739+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:17:58.466739+0000","last_undegraded":"2024-03-04T06:17:58.466739+0000","last_fullsized":"2024-03-04T06:17:58.466739+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496417,"num_objects":55,"num_object_clones":43,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":293,"num_read_kb":234233,"num_write":248,"num_write_kb":67085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1513,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-03-04T06:17:59.348859+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:17:59.348859+0000","last_peered":"2024-03-04T06:17:59.348859+0000","last_clean":"2024-03-04T06:17:59.348859+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:17:59.348859+0000","last_undegraded":"2024-03-04T06:17:59.348859+0000","last_fullsized":"2024-03-04T06:17:59.348859+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"600'381","reported_seq":2026,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-03-04T06:18:01.357392+0000","last_change":"2024-03-04T06:18:00.362092+0000","last_active":"2024-03-04T06:18:01.357392+0000","last_peered":"2024-03-04T06:18:01.357392+0000","last_clean":"2024-03-04T06:18:01.357392+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:18:01.357392+0000","last_undegraded":"2024-03-04T06:18:01.357392+0000","last_fullsized":"2024-03-04T06:18:01.357392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370051,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":219,"num_read_kb":177501,"num_write":146,"num_write_kb":41225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1531,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-03-04T06:18:01.357652+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:18:01.357652+0000","last_peered":"2024-03-04T06:18:01.357652+0000","last_clean":"2024-03-04T06:18:01.357652+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:18:01.357652+0000","last_undegraded":"2024-03-04T06:18:01.357652+0000","last_fullsized":"2024-03-04T06:18:01.357652+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"610'998","reported_seq":2481,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-03-04T06:18:01.361862+0000","last_change":"2024-03-04T06:18:00.501055+0000","last_active":"2024-03-04T06:18:01.361862+0000","last_peered":"2024-03-04T06:18:01.361862+0000","last_clean":"2024-03-04T06:18:01.361862+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:18:01.361862+0000","last_undegraded":"2024-03-04T06:18:01.361862+0000","last_fullsized":"2024-03-04T06:18:01.361862+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11614693,"num_objects":100,"num_object_clones":78,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":580,"num_read_kb":452857,"num_write":473,"num_write_kb":126341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1513,"reported_epoch":609,"state":"active+clean","last_fresh":"2024-03-04T06:17:59.338882+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:17:59.338882+0000","last_peered":"2024-03-04T06:17:59.338882+0000","last_clean":"2024-03-04T06:17:59.338882+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:17:59.338882+0000","last_undegraded":"2024-03-04T06:17:59.338882+0000","last_fullsized":"2024-03-04T06:17:59.338882+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"607'1066","reported_seq":2438,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-03-04T06:17:59.334697+0000","last_change":"2024-03-04T06:17:56.439797+0000","last_active":"2024-03-04T06:17:59.334697+0000","last_peered":"2024-03-04T06:17:59.334697+0000","last_clean":"2024-03-04T06:17:59.334697+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:17:59.334697+0000","last_undegraded":"2024-03-04T06:17:59.334697+0000","last_fullsized":"2024-03-04T06:17:59.334697+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1066,"log_dups_size":0,"ondisk_log_size":1066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13026415,"num_objects":101,"num_object_clones":79,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":469,"num_read_kb":347937,"num_write":462,"num_write_kb":122180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"605'512","reported_seq":2181,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-03-04T06:17:58.465791+0000","last_change":"2024-03-04T06:17:55.358395+0000","last_active":"2024-03-04T06:17:58.465791+0000","last_peered":"2024-03-04T06:17:58.465791+0000","last_clean":"2024-03-04T06:17:58.465791+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:17:58.465791+0000","last_undegraded":"2024-03-04T06:17:58.465791+0000","last_fullsized":"2024-03-04T06:17:58.465791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8320257,"num_objects":51,"num_object_clones":39,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":356,"num_read_kb":294753,"num_write":241,"num_write_kb":65778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1546,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-03-04T06:17:59.348513+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:17:59.348513+0000","last_peered":"2024-03-04T06:17:59.348513+0000","last_clean":"2024-03-04T06:17:59.348513+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:17:59.348513+0000","last_undegraded":"2024-03-04T06:17:59.348513+0000","last_fullsized":"2024-03-04T06:17:59.348513+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"607'489","reported_seq":2140,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-03-04T06:17:59.345714+0000","last_change":"2024-03-04T06:17:56.412823+0000","last_active":"2024-03-04T06:17:59.345714+0000","last_peered":"2024-03-04T06:17:59.345714+0000","last_clean":"2024-03-04T06:17:59.345714+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:17:59.345714+0000","last_undegraded":"2024-03-04T06:17:59.345714+0000","last_fullsized":"2024-03-04T06:17:59.345714+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8353871,"num_objects":56,"num_object_clones":40,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":264,"num_read_kb":192933,"num_write":227,"num_write_kb":59954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1521,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-03-04T06:18:01.361125+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:18:01.361125+0000","last_peered":"2024-03-04T06:18:01.361125+0000","last_clean":"2024-03-04T06:18:01.361125+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:18:01.361125+0000","last_undegraded":"2024-03-04T06:18:01.361125+0000","last_fullsized":"2024-03-04T06:18:01.361125+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"600'711","reported_seq":2232,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-03-04T06:18:01.356323+0000","last_change":"2024-03-04T06:18:00.361464+0000","last_active":"2024-03-04T06:18:01.356323+0000","last_peered":"2024-03-04T06:18:01.356323+0000","last_clean":"2024-03-04T06:18:01.356323+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:18:01.356323+0000","last_undegraded":"2024-03-04T06:18:01.356323+0000","last_fullsized":"2024-03-04T06:18:01.356323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":711,"log_dups_size":0,"ondisk_log_size":711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9312134,"num_objects":68,"num_object_clones":54,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":353,"num_read_kb":263484,"num_write":314,"num_write_kb":85989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1547,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-03-04T06:18:00.359487+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:18:00.359487+0000","last_peered":"2024-03-04T06:18:00.359487+0000","last_clean":"2024-03-04T06:18:00.359487+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:18:00.359487+0000","last_undegraded":"2024-03-04T06:18:00.359487+0000","last_fullsized":"2024-03-04T06:18:00.359487+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"601'831","reported_seq":2339,"reported_epoch":610,"state":"active+clean","last_fresh":"2024-03-04T06:17:59.336675+0000","last_change":"2024-03-04T06:17:56.417567+0000","last_active":"2024-03-04T06:17:59.336675+0000","last_peered":"2024-03-04T06:17:59.336675+0000","last_clean":"2024-03-04T06:17:59.336675+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:17:59.336675+0000","last_undegraded":"2024-03-04T06:17:59.336675+0000","last_fullsized":"2024-03-04T06:17:59.336675+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135050,"num_objects":74,"num_object_clones":65,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":470,"num_read_kb":390395,"num_write":340,"num_write_kb":93412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1528,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-03-04T06:18:01.363080+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:18:01.363080+0000","last_peered":"2024-03-04T06:18:01.363080+0000","last_clean":"2024-03-04T06:18:01.363080+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:18:01.363080+0000","last_undegraded":"2024-03-04T06:18:01.363080+0000","last_fullsized":"2024-03-04T06:18:01.363080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":97254080,"num_objects":844,"num_object_clones":661,"num_object_copies":2532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":149,"num_read":4536,"num_read_kb":3553980,"num_write":3798,"num_write_kb":1030459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8550,"ondisk_log_size":8550,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706279,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":188876,"kb_used_data":139740,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94182964,"statfs":{"total":96636764160,"available":96443355136,"internally_reserved":0,"allocated":143093760,"data_stored":170047629,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706279,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":150024,"kb_used_data":105152,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94221816,"statfs":{"total":96636764160,"available":96483139584,"internally_reserved":0,"allocated":107675648,"data_stored":123797536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":157156,"kb_used_data":112060,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94214684,"statfs":{"total":96636764160,"available":96475836416,"internally_reserved":0,"allocated":114749440,"data_stored":135721146,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":130360,"kb_used_data":85264,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94241480,"statfs":{"total":96636764160,"available":96503275520,"internally_reserved":0,"allocated":87310336,"data_stored":99251171,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:18:05.531 INFO:tasks.rados.rados.0.smithi064.stdout:2484: done (3 left) 2024-03-04T06:18:05.532 INFO:tasks.rados.rados.0.smithi064.stdout:2486: done (2 left) 2024-03-04T06:18:05.532 INFO:tasks.rados.rados.0.smithi064.stdout:2487: done (1 left) 2024-03-04T06:18:05.532 INFO:tasks.rados.rados.0.smithi064.stdout:2488: done (0 left) 2024-03-04T06:18:05.532 INFO:tasks.rados.rados.0.smithi064.stdout:2489: read oid 34 snap -1 2024-03-04T06:18:05.532 INFO:tasks.rados.rados.0.smithi064.stdout:2489: expect (ObjNum 1114 snap 294 seq_num 1114) 2024-03-04T06:18:05.532 INFO:tasks.rados.rados.0.smithi064.stdout:2490: snap_remove snap 281 2024-03-04T06:18:06.351 INFO:tasks.rados.rados.0.smithi064.stdout:2490: done (1 left) 2024-03-04T06:18:06.351 INFO:tasks.rados.rados.0.smithi064.stdout:2491: read oid 3 snap 291 2024-03-04T06:18:06.351 INFO:tasks.rados.rados.0.smithi064.stdout:2491: expect (ObjNum 1103 snap 288 seq_num 1103) 2024-03-04T06:18:06.351 INFO:tasks.rados.rados.0.smithi064.stdout:2492: read oid 49 snap -1 2024-03-04T06:18:06.352 INFO:tasks.rados.rados.0.smithi064.stdout:2492: expect (ObjNum 1109 snap 290 seq_num 1109) 2024-03-04T06:18:06.352 INFO:tasks.rados.rados.0.smithi064.stdout:2493: snap_create 2024-03-04T06:18:06.352 INFO:tasks.rados.rados.0.smithi064.stdout:2489: expect (ObjNum 1114 snap 294 seq_num 1114) 2024-03-04T06:18:06.559 INFO:tasks.rados.rados.0.smithi064.stdout:2491: expect (ObjNum 1103 snap 288 seq_num 1103) 2024-03-04T06:18:06.720 INFO:tasks.rados.rados.0.smithi064.stdout:2492: expect (ObjNum 1109 snap 290 seq_num 1109) 2024-03-04T06:18:07.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:07.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:07.262+0000 7f494319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:07.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:07.266+0000 7f494319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:07.357 INFO:tasks.rados.rados.0.smithi064.stdout:2489: done (3 left) 2024-03-04T06:18:07.357 INFO:tasks.rados.rados.0.smithi064.stdout:2491: done (2 left) 2024-03-04T06:18:07.357 INFO:tasks.rados.rados.0.smithi064.stdout:2492: done (1 left) 2024-03-04T06:18:07.357 INFO:tasks.rados.rados.0.smithi064.stdout:2493: done (0 left) 2024-03-04T06:18:07.357 INFO:tasks.rados.rados.0.smithi064.stdout:2494: write oid 2 current snap is 309 2024-03-04T06:18:07.357 INFO:tasks.rados.rados.0.smithi064.stdout:2494: seq_num 1147 ranges {477629=563553,1836611=777404,3048408=229221} 2024-03-04T06:18:07.363 INFO:tasks.rados.rados.0.smithi064.stdout:2494: writing smithi06431743-2 from 477629 to 1041182 tid 1 2024-03-04T06:18:07.371 INFO:tasks.rados.rados.0.smithi064.stdout:2494: writing smithi06431743-2 from 1836611 to 2614015 tid 2 2024-03-04T06:18:07.375 INFO:tasks.rados.rados.0.smithi064.stdout:2494: writing smithi06431743-2 from 3048408 to 3277629 tid 3 2024-03-04T06:18:07.375 INFO:tasks.rados.rados.0.smithi064.stdout:2495: write oid 29 current snap is 309 2024-03-04T06:18:07.376 INFO:tasks.rados.rados.0.smithi064.stdout:2495: seq_num 1148 ranges {494436=755155,2011446=521062,3075831=218605} 2024-03-04T06:18:07.382 INFO:tasks.rados.rados.0.smithi064.stdout:2495: writing smithi06431743-29 from 494436 to 1249591 tid 1 2024-03-04T06:18:07.389 INFO:tasks.rados.rados.0.smithi064.stdout:2495: writing smithi06431743-29 from 2011446 to 2532508 tid 2 2024-03-04T06:18:07.394 INFO:tasks.rados.rados.0.smithi064.stdout:2495: writing smithi06431743-29 from 3075831 to 3294436 tid 3 2024-03-04T06:18:07.395 INFO:tasks.rados.rados.0.smithi064.stdout:2496: write (excl) oid 14 current snap is 309 2024-03-04T06:18:07.395 INFO:tasks.rados.rados.0.smithi064.stdout:2496: seq_num 1149 ranges {511243=430404,1669928=664720,3103254=207989} 2024-03-04T06:18:07.401 INFO:tasks.rados.rados.0.smithi064.stdout:2496: writing smithi06431743-14 from 511243 to 941647 tid 1 2024-03-04T06:18:07.410 INFO:tasks.rados.rados.0.smithi064.stdout:2496: writing smithi06431743-14 from 1669928 to 2334648 tid 2 2024-03-04T06:18:07.416 INFO:tasks.rados.rados.0.smithi064.stdout:2496: writing smithi06431743-14 from 3103254 to 3311243 tid 3 2024-03-04T06:18:07.416 INFO:tasks.rados.rados.0.smithi064.stdout:2497: snap_remove snap 296 2024-03-04T06:18:08.364 INFO:tasks.rados.rados.0.smithi064.stdout:2497: done (3 left) 2024-03-04T06:18:08.364 INFO:tasks.rados.rados.0.smithi064.stdout:2498: snap_remove snap 280 2024-03-04T06:18:09.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:09.373 INFO:tasks.rados.rados.0.smithi064.stdout:2498: done (3 left) 2024-03-04T06:18:09.373 INFO:tasks.rados.rados.0.smithi064.stdout:2499: write oid 48 current snap is 309 2024-03-04T06:18:09.373 INFO:tasks.rados.rados.0.smithi064.stdout:2494: finishing write tid 1 to smithi06431743-2 2024-03-04T06:18:09.373 INFO:tasks.rados.rados.0.smithi064.stdout:2499: seq_num 1150 ranges {528050=505653,1612057=524731,2614324=567403} 2024-03-04T06:18:09.378 INFO:tasks.rados.rados.0.smithi064.stdout:2499: writing smithi06431743-48 from 528050 to 1033703 tid 1 2024-03-04T06:18:09.384 INFO:tasks.rados.rados.0.smithi064.stdout:2499: writing smithi06431743-48 from 1612057 to 2136788 tid 2 2024-03-04T06:18:09.391 INFO:tasks.rados.rados.0.smithi064.stdout:2499: writing smithi06431743-48 from 2614324 to 3181727 tid 3 2024-03-04T06:18:09.391 INFO:tasks.rados.rados.0.smithi064.stdout:2500: delete oid 42 current snap is 309 2024-03-04T06:18:09.391 INFO:tasks.rados.rados.0.smithi064.stdout:2495: finishing write tid 1 to smithi06431743-29 2024-03-04T06:18:09.391 INFO:tasks.rados.rados.0.smithi064.stdout:2494: finishing write tid 2 to smithi06431743-2 2024-03-04T06:18:09.392 INFO:tasks.rados.rados.0.smithi064.stdout:2496: finishing write tid 1 to smithi06431743-14 2024-03-04T06:18:09.392 INFO:tasks.rados.rados.0.smithi064.stdout:2495: finishing write tid 2 to smithi06431743-29 2024-03-04T06:18:09.392 INFO:tasks.rados.rados.0.smithi064.stdout:2494: finishing write tid 3 to smithi06431743-2 2024-03-04T06:18:09.392 INFO:tasks.rados.rados.0.smithi064.stdout:2496: finishing write tid 2 to smithi06431743-14 2024-03-04T06:18:09.392 INFO:tasks.rados.rados.0.smithi064.stdout:2495: finishing write tid 3 to smithi06431743-29 2024-03-04T06:18:09.392 INFO:tasks.rados.rados.0.smithi064.stdout:2496: finishing write tid 3 to smithi06431743-14 2024-03-04T06:18:09.392 INFO:tasks.rados.rados.0.smithi064.stdout:2496: finishing write tid 4 to smithi06431743-14 2024-03-04T06:18:09.392 INFO:tasks.rados.rados.0.smithi064.stdout:2496: finishing write tid 5 to smithi06431743-14 2024-03-04T06:18:09.392 INFO:tasks.rados.rados.0.smithi064.stdout:2496: oid 14 updating version 0 to 720 2024-03-04T06:18:09.392 INFO:tasks.rados.rados.0.smithi064.stdout:2496: oid 14 version 720 is already newer than 717 2024-03-04T06:18:09.393 INFO:tasks.rados.rados.0.smithi064.stdout:2496: oid 14 version 720 is already newer than 720 2024-03-04T06:18:09.393 INFO:tasks.rados.rados.0.smithi064.stdout:2496: oid 14 version 720 is already newer than 718 2024-03-04T06:18:09.393 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 720 (ObjNum 1149 snap 309 seq_num 1149) dirty exists 2024-03-04T06:18:09.393 INFO:tasks.rados.rados.0.smithi064.stdout:2496: left oid 14 (ObjNum 1149 snap 309 seq_num 1149) 2024-03-04T06:18:09.393 INFO:tasks.rados.rados.0.smithi064.stdout:2494: finishing write tid 4 to smithi06431743-2 2024-03-04T06:18:09.393 INFO:tasks.rados.rados.0.smithi064.stdout:2494: finishing write tid 5 to smithi06431743-2 2024-03-04T06:18:09.393 INFO:tasks.rados.rados.0.smithi064.stdout:2494: oid 2 updating version 0 to 432 2024-03-04T06:18:09.393 INFO:tasks.rados.rados.0.smithi064.stdout:2494: oid 2 version 432 is already newer than 430 2024-03-04T06:18:09.393 INFO:tasks.rados.rados.0.smithi064.stdout:2494: oid 2 version 432 is already newer than 431 2024-03-04T06:18:09.393 INFO:tasks.rados.rados.0.smithi064.stdout:2494: oid 2 version 432 is already newer than 430 2024-03-04T06:18:09.393 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 432 (ObjNum 1147 snap 309 seq_num 1147) dirty exists 2024-03-04T06:18:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:2494: left oid 2 (ObjNum 1147 snap 309 seq_num 1147) 2024-03-04T06:18:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:2495: finishing write tid 4 to smithi06431743-29 2024-03-04T06:18:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:2495: finishing write tid 5 to smithi06431743-29 2024-03-04T06:18:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:2495: oid 29 updating version 0 to 1100 2024-03-04T06:18:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:2495: oid 29 version 1100 is already newer than 1098 2024-03-04T06:18:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:2495: oid 29 version 1100 is already newer than 1099 2024-03-04T06:18:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:2495: oid 29 version 1100 is already newer than 1097 2024-03-04T06:18:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 1100 (ObjNum 1148 snap 309 seq_num 1148) dirty exists 2024-03-04T06:18:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:2495: left oid 29 (ObjNum 1148 snap 309 seq_num 1148) 2024-03-04T06:18:09.451 INFO:tasks.rados.rados.0.smithi064.stdout:2499: finishing write tid 1 to smithi06431743-48 2024-03-04T06:18:09.469 INFO:tasks.rados.rados.0.smithi064.stdout:2499: finishing write tid 2 to smithi06431743-48 2024-03-04T06:18:09.475 INFO:tasks.rados.rados.0.smithi064.stdout:2494: done (4 left) 2024-03-04T06:18:09.475 INFO:tasks.rados.rados.0.smithi064.stdout:2495: done (3 left) 2024-03-04T06:18:09.476 INFO:tasks.rados.rados.0.smithi064.stdout:2496: done (2 left) 2024-03-04T06:18:09.476 INFO:tasks.rados.rados.0.smithi064.stdout:2500: done (1 left) 2024-03-04T06:18:09.476 INFO:tasks.rados.rados.0.smithi064.stdout:2501: write (excl) oid 31 current snap is 309 2024-03-04T06:18:09.476 INFO:tasks.rados.rados.0.smithi064.stdout:2501: seq_num 1152 ranges {561664=656151,1729021=528400,2669170=692494} 2024-03-04T06:18:09.482 INFO:tasks.rados.rados.0.smithi064.stdout:2501: writing smithi06431743-31 from 561664 to 1217815 tid 1 2024-03-04T06:18:09.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:09.483+0000 7f519a114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:09.489 INFO:tasks.rados.rados.0.smithi064.stdout:2501: writing smithi06431743-31 from 1729021 to 2257421 tid 2 2024-03-04T06:18:09.491 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:09.490+0000 7f519a114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:09.497 INFO:tasks.rados.rados.0.smithi064.stdout:2501: writing smithi06431743-31 from 2669170 to 3361664 tid 3 2024-03-04T06:18:09.497 INFO:tasks.rados.rados.0.smithi064.stdout:2502: read oid 46 snap -1 2024-03-04T06:18:09.497 INFO:tasks.rados.rados.0.smithi064.stdout:2502: expect (ObjNum 1144 snap 307 seq_num 1144) 2024-03-04T06:18:09.497 INFO:tasks.rados.rados.0.smithi064.stdout:2503: read oid 10 snap 305 2024-03-04T06:18:09.497 INFO:tasks.rados.rados.0.smithi064.stdout:2503: expect (ObjNum 1133 snap 303 seq_num 1133) 2024-03-04T06:18:09.498 INFO:tasks.rados.rados.0.smithi064.stdout:2504: delete oid 47 current snap is 309 2024-03-04T06:18:09.505 INFO:tasks.rados.rados.0.smithi064.stdout:2499: finishing write tid 3 to smithi06431743-48 2024-03-04T06:18:09.507 INFO:tasks.rados.rados.0.smithi064.stdout:2499: finishing write tid 4 to smithi06431743-48 2024-03-04T06:18:09.509 INFO:tasks.rados.rados.0.smithi064.stdout:2504: done (4 left) 2024-03-04T06:18:09.509 INFO:tasks.rados.rados.0.smithi064.stdout:2505: read oid 45 snap -1 2024-03-04T06:18:09.509 INFO:tasks.rados.rados.0.smithi064.stdout:2505: expect deleted 2024-03-04T06:18:09.509 INFO:tasks.rados.rados.0.smithi064.stdout:2506: read oid 11 snap -1 2024-03-04T06:18:09.509 INFO:tasks.rados.rados.0.smithi064.stdout:2506: expect deleted 2024-03-04T06:18:09.509 INFO:tasks.rados.rados.0.smithi064.stdout:2507: write (excl) oid 4 current snap is 309 2024-03-04T06:18:09.509 INFO:tasks.rados.rados.0.smithi064.stdout:2507: seq_num 1154 ranges {595278=406649,1729632=532069,3007663=387615} 2024-03-04T06:18:09.514 INFO:tasks.rados.rados.0.smithi064.stdout:2507: writing smithi06431743-4 from 595278 to 1001927 tid 1 2024-03-04T06:18:09.521 INFO:tasks.rados.rados.0.smithi064.stdout:2507: writing smithi06431743-4 from 1729632 to 2261701 tid 2 2024-03-04T06:18:09.527 INFO:tasks.rados.rados.0.smithi064.stdout:2507: writing smithi06431743-4 from 3007663 to 3395278 tid 3 2024-03-04T06:18:09.527 INFO:tasks.rados.rados.0.smithi064.stdout:2508: snap_remove snap 307 2024-03-04T06:18:10.371 INFO:tasks.rados.rados.0.smithi064.stdout:2508: done (7 left) 2024-03-04T06:18:10.372 INFO:tasks.rados.rados.0.smithi064.stdout:2509: snap_remove snap 277 2024-03-04T06:18:10.428 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:18:10.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:18:10.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:10.544+0000 7f64cb11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:10.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:10.553+0000 7f64cb11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:11.367 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:18:11.370 INFO:tasks.rados.rados.0.smithi064.stdout:2509: done (7 left) 2024-03-04T06:18:11.370 INFO:tasks.rados.rados.0.smithi064.stdout:2510: snap_remove snap 308 2024-03-04T06:18:11.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:18:11.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:11.489+0000 7fa8f1824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:11.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:11.497+0000 7fa8f1824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:11.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:11.757 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:11.755+0000 7f29f271d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:11.760 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:11.759+0000 7f29f271d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:12.368 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:18:12.373 INFO:tasks.rados.rados.0.smithi064.stdout:2510: done (7 left) 2024-03-04T06:18:12.373 INFO:tasks.rados.rados.0.smithi064.stdout:2511: read oid 20 snap 283 2024-03-04T06:18:12.373 INFO:tasks.rados.rados.0.smithi064.stdout:2511: expect (ObjNum 1090 snap 279 seq_num 1090) 2024-03-04T06:18:12.373 INFO:tasks.rados.rados.0.smithi064.stdout:2512: write (excl) oid 14 current snap is 309 2024-03-04T06:18:12.373 INFO:tasks.rados.rados.0.smithi064.stdout:2499: finishing write tid 5 to smithi06431743-48 2024-03-04T06:18:12.373 INFO:tasks.rados.rados.0.smithi064.stdout:2499: oid 48 updating version 0 to 384 2024-03-04T06:18:12.373 INFO:tasks.rados.rados.0.smithi064.stdout:2499: oid 48 updating version 384 to 385 2024-03-04T06:18:12.374 INFO:tasks.rados.rados.0.smithi064.stdout:2499: oid 48 updating version 385 to 387 2024-03-04T06:18:12.374 INFO:tasks.rados.rados.0.smithi064.stdout:2499: oid 48 version 387 is already newer than 387 2024-03-04T06:18:12.374 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 387 (ObjNum 1150 snap 309 seq_num 1150) dirty exists 2024-03-04T06:18:12.374 INFO:tasks.rados.rados.0.smithi064.stdout:2499: left oid 48 (ObjNum 1150 snap 309 seq_num 1150) 2024-03-04T06:18:12.374 INFO:tasks.rados.rados.0.smithi064.stdout:2501: finishing write tid 1 to smithi06431743-31 2024-03-04T06:18:12.374 INFO:tasks.rados.rados.0.smithi064.stdout:2503: expect (ObjNum 1133 snap 303 seq_num 1133) 2024-03-04T06:18:12.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:18:12.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:12.519+0000 7fb832e24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:12.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:12.527+0000 7fb832e24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:12.544 INFO:tasks.rados.rados.0.smithi064.stdout:2501: finishing write tid 2 to smithi06431743-31 2024-03-04T06:18:12.544 INFO:tasks.rados.rados.0.smithi064.stdout:2502: expect (ObjNum 1144 snap 307 seq_num 1144) 2024-03-04T06:18:12.664 INFO:tasks.rados.rados.0.smithi064.stdout:2501: finishing write tid 3 to smithi06431743-31 2024-03-04T06:18:12.664 INFO:tasks.rados.rados.0.smithi064.stdout:2512: seq_num 1155 ranges {612085=481898,1788114=675727,3035086=376999} 2024-03-04T06:18:12.669 INFO:tasks.rados.rados.0.smithi064.stdout:2512: writing smithi06431743-14 from 612085 to 1093983 tid 1 2024-03-04T06:18:12.677 INFO:tasks.rados.rados.0.smithi064.stdout:2512: writing smithi06431743-14 from 1788114 to 2463841 tid 2 2024-03-04T06:18:12.684 INFO:tasks.rados.rados.0.smithi064.stdout:2512: writing smithi06431743-14 from 3035086 to 3412085 tid 3 2024-03-04T06:18:12.684 INFO:tasks.rados.rados.0.smithi064.stdout:2499: done (8 left) 2024-03-04T06:18:12.685 INFO:tasks.rados.rados.0.smithi064.stdout:2502: done (7 left) 2024-03-04T06:18:12.685 INFO:tasks.rados.rados.0.smithi064.stdout:2503: done (6 left) 2024-03-04T06:18:12.685 INFO:tasks.rados.rados.0.smithi064.stdout:2505: done (5 left) 2024-03-04T06:18:12.685 INFO:tasks.rados.rados.0.smithi064.stdout:2506: done (4 left) 2024-03-04T06:18:12.685 INFO:tasks.rados.rados.0.smithi064.stdout:2513: write (excl) oid 3 current snap is 309 2024-03-04T06:18:12.685 INFO:tasks.rados.rados.0.smithi064.stdout:2513: seq_num 1156 ranges {628892=673500,1962949=535738,3178862=250030} 2024-03-04T06:18:12.691 INFO:tasks.rados.rados.0.smithi064.stdout:2513: writing smithi06431743-3 from 628892 to 1302392 tid 1 2024-03-04T06:18:12.698 INFO:tasks.rados.rados.0.smithi064.stdout:2513: writing smithi06431743-3 from 1962949 to 2498687 tid 2 2024-03-04T06:18:12.706 INFO:tasks.rados.rados.0.smithi064.stdout:2513: writing smithi06431743-3 from 3178862 to 3428892 tid 3 2024-03-04T06:18:12.706 INFO:tasks.rados.rados.0.smithi064.stdout:2514: read oid 32 snap 290 2024-03-04T06:18:12.706 INFO:tasks.rados.rados.0.smithi064.stdout:2514: expect deleted 2024-03-04T06:18:12.707 INFO:tasks.rados.rados.0.smithi064.stdout:2515: read oid 42 snap -1 2024-03-04T06:18:12.707 INFO:tasks.rados.rados.0.smithi064.stdout:2515: expect deleted 2024-03-04T06:18:12.707 INFO:tasks.rados.rados.0.smithi064.stdout:2516: snap_remove snap 274 2024-03-04T06:18:13.383 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:18:13.389 INFO:tasks.rados.rados.0.smithi064.stdout:2516: done (7 left) 2024-03-04T06:18:13.389 INFO:tasks.rados.rados.0.smithi064.stdout:2517: write (excl) oid 11 current snap is 309 2024-03-04T06:18:13.389 INFO:tasks.rados.rados.0.smithi064.stdout:2501: finishing write tid 4 to smithi06431743-31 2024-03-04T06:18:13.389 INFO:tasks.rados.rados.0.smithi064.stdout:2517: seq_num 1157 ranges {645699=748749,2021431=679396,3206285=239414} 2024-03-04T06:18:13.396 INFO:tasks.rados.rados.0.smithi064.stdout:2517: writing smithi06431743-11 from 645699 to 1394448 tid 1 2024-03-04T06:18:13.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:18:13.403 INFO:tasks.rados.rados.0.smithi064.stdout:2517: writing smithi06431743-11 from 2021431 to 2700827 tid 2 2024-03-04T06:18:13.409 INFO:tasks.rados.rados.0.smithi064.stdout:2517: writing smithi06431743-11 from 3206285 to 3445699 tid 3 2024-03-04T06:18:13.409 INFO:tasks.rados.rados.0.smithi064.stdout:2518: snap_create 2024-03-04T06:18:13.409 INFO:tasks.rados.rados.0.smithi064.stdout:2501: finishing write tid 5 to smithi06431743-31 2024-03-04T06:18:13.409 INFO:tasks.rados.rados.0.smithi064.stdout:2501: oid 31 updating version 0 to 207 2024-03-04T06:18:13.409 INFO:tasks.rados.rados.0.smithi064.stdout:2501: oid 31 updating version 207 to 209 2024-03-04T06:18:13.409 INFO:tasks.rados.rados.0.smithi064.stdout:2501: oid 31 version 209 is already newer than 208 2024-03-04T06:18:13.409 INFO:tasks.rados.rados.0.smithi064.stdout:2501: oid 31 version 209 is already newer than 206 2024-03-04T06:18:13.409 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 209 (ObjNum 1152 snap 309 seq_num 1152) dirty exists 2024-03-04T06:18:13.410 INFO:tasks.rados.rados.0.smithi064.stdout:2501: left oid 31 (ObjNum 1152 snap 309 seq_num 1152) 2024-03-04T06:18:13.410 INFO:tasks.rados.rados.0.smithi064.stdout:2507: finishing write tid 1 to smithi06431743-4 2024-03-04T06:18:13.410 INFO:tasks.rados.rados.0.smithi064.stdout:2507: finishing write tid 2 to smithi06431743-4 2024-03-04T06:18:13.410 INFO:tasks.rados.rados.0.smithi064.stdout:2507: finishing write tid 3 to smithi06431743-4 2024-03-04T06:18:13.410 INFO:tasks.rados.rados.0.smithi064.stdout:2507: finishing write tid 4 to smithi06431743-4 2024-03-04T06:18:13.410 INFO:tasks.rados.rados.0.smithi064.stdout:2507: finishing write tid 5 to smithi06431743-4 2024-03-04T06:18:13.410 INFO:tasks.rados.rados.0.smithi064.stdout:2507: oid 4 updating version 0 to 1106 2024-03-04T06:18:13.410 INFO:tasks.rados.rados.0.smithi064.stdout:2507: oid 4 updating version 1106 to 1109 2024-03-04T06:18:13.410 INFO:tasks.rados.rados.0.smithi064.stdout:2507: oid 4 version 1109 is already newer than 1107 2024-03-04T06:18:13.410 INFO:tasks.rados.rados.0.smithi064.stdout:2507: oid 4 version 1109 is already newer than 1108 2024-03-04T06:18:13.410 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 1109 (ObjNum 1154 snap 309 seq_num 1154) dirty exists 2024-03-04T06:18:13.411 INFO:tasks.rados.rados.0.smithi064.stdout:2507: left oid 4 (ObjNum 1154 snap 309 seq_num 1154) 2024-03-04T06:18:13.411 INFO:tasks.rados.rados.0.smithi064.stdout:2511: expect (ObjNum 1090 snap 279 seq_num 1090) 2024-03-04T06:18:13.517 INFO:tasks.rados.rados.0.smithi064.stdout:2512: finishing write tid 1 to smithi06431743-14 2024-03-04T06:18:13.517 INFO:tasks.rados.rados.0.smithi064.stdout:2512: finishing write tid 2 to smithi06431743-14 2024-03-04T06:18:13.517 INFO:tasks.rados.rados.0.smithi064.stdout:2512: finishing write tid 3 to smithi06431743-14 2024-03-04T06:18:13.517 INFO:tasks.rados.rados.0.smithi064.stdout:2512: finishing write tid 4 to smithi06431743-14 2024-03-04T06:18:13.517 INFO:tasks.rados.rados.0.smithi064.stdout:2512: finishing write tid 5 to smithi06431743-14 2024-03-04T06:18:13.517 INFO:tasks.rados.rados.0.smithi064.stdout:2512: oid 14 updating version 0 to 729 2024-03-04T06:18:13.517 INFO:tasks.rados.rados.0.smithi064.stdout:2512: oid 14 version 729 is already newer than 729 2024-03-04T06:18:13.518 INFO:tasks.rados.rados.0.smithi064.stdout:2512: oid 14 version 729 is already newer than 727 2024-03-04T06:18:13.518 INFO:tasks.rados.rados.0.smithi064.stdout:2512: oid 14 updating version 729 to 730 2024-03-04T06:18:13.518 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 730 (ObjNum 1155 snap 309 seq_num 1155) dirty exists 2024-03-04T06:18:13.518 INFO:tasks.rados.rados.0.smithi064.stdout:2512: left oid 14 (ObjNum 1155 snap 309 seq_num 1155) 2024-03-04T06:18:13.518 INFO:tasks.rados.rados.0.smithi064.stdout:2513: finishing write tid 1 to smithi06431743-3 2024-03-04T06:18:13.518 INFO:tasks.rados.rados.0.smithi064.stdout:2513: finishing write tid 2 to smithi06431743-3 2024-03-04T06:18:13.518 INFO:tasks.rados.rados.0.smithi064.stdout:2513: finishing write tid 3 to smithi06431743-3 2024-03-04T06:18:13.518 INFO:tasks.rados.rados.0.smithi064.stdout:2513: finishing write tid 4 to smithi06431743-3 2024-03-04T06:18:13.518 INFO:tasks.rados.rados.0.smithi064.stdout:2513: finishing write tid 5 to smithi06431743-3 2024-03-04T06:18:13.518 INFO:tasks.rados.rados.0.smithi064.stdout:2513: oid 3 updating version 0 to 589 2024-03-04T06:18:13.518 INFO:tasks.rados.rados.0.smithi064.stdout:2513: oid 3 version 589 is already newer than 588 2024-03-04T06:18:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:2513: oid 3 updating version 589 to 591 2024-03-04T06:18:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:2513: oid 3 version 591 is already newer than 590 2024-03-04T06:18:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 591 (ObjNum 1156 snap 309 seq_num 1156) dirty exists 2024-03-04T06:18:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:2513: left oid 3 (ObjNum 1156 snap 309 seq_num 1156) 2024-03-04T06:18:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:2517: finishing write tid 1 to smithi06431743-11 2024-03-04T06:18:13.527 INFO:tasks.rados.rados.0.smithi064.stdout:2517: finishing write tid 2 to smithi06431743-11 2024-03-04T06:18:13.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:13.527+0000 7f9cc3b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:13.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:13.536+0000 7f9cc3b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:13.544 INFO:tasks.rados.rados.0.smithi064.stdout:2517: finishing write tid 3 to smithi06431743-11 2024-03-04T06:18:13.549 INFO:tasks.rados.rados.0.smithi064.stdout:2517: finishing write tid 4 to smithi06431743-11 2024-03-04T06:18:13.551 INFO:tasks.rados.rados.0.smithi064.stdout:2517: finishing write tid 5 to smithi06431743-11 2024-03-04T06:18:13.551 INFO:tasks.rados.rados.0.smithi064.stdout:2517: oid 11 updating version 0 to 587 2024-03-04T06:18:13.551 INFO:tasks.rados.rados.0.smithi064.stdout:2517: oid 11 version 587 is already newer than 584 2024-03-04T06:18:13.551 INFO:tasks.rados.rados.0.smithi064.stdout:2517: oid 11 version 587 is already newer than 586 2024-03-04T06:18:13.551 INFO:tasks.rados.rados.0.smithi064.stdout:2517: oid 11 version 587 is already newer than 585 2024-03-04T06:18:13.551 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 587 (ObjNum 1157 snap 309 seq_num 1157) dirty exists 2024-03-04T06:18:13.551 INFO:tasks.rados.rados.0.smithi064.stdout:2517: left oid 11 (ObjNum 1157 snap 309 seq_num 1157) 2024-03-04T06:18:13.829 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:13.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:13.934+0000 7fdda9754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:13.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:13.938+0000 7fdda9754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:14.387 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:18:14.389 INFO:tasks.rados.rados.0.smithi064.stdout:2501: done (8 left) 2024-03-04T06:18:14.389 INFO:tasks.rados.rados.0.smithi064.stdout:2507: done (7 left) 2024-03-04T06:18:14.389 INFO:tasks.rados.rados.0.smithi064.stdout:2511: done (6 left) 2024-03-04T06:18:14.389 INFO:tasks.rados.rados.0.smithi064.stdout:2512: done (5 left) 2024-03-04T06:18:14.389 INFO:tasks.rados.rados.0.smithi064.stdout:2513: done (4 left) 2024-03-04T06:18:14.389 INFO:tasks.rados.rados.0.smithi064.stdout:2514: done (3 left) 2024-03-04T06:18:14.390 INFO:tasks.rados.rados.0.smithi064.stdout:2515: done (2 left) 2024-03-04T06:18:14.390 INFO:tasks.rados.rados.0.smithi064.stdout:2517: done (1 left) 2024-03-04T06:18:14.390 INFO:tasks.rados.rados.0.smithi064.stdout:2518: done (0 left) 2024-03-04T06:18:14.390 INFO:tasks.rados.rados.0.smithi064.stdout:2519: write oid 40 current snap is 310 2024-03-04T06:18:14.390 INFO:tasks.rados.rados.0.smithi064.stdout:2519: seq_num 1158 ranges {662506=423998,1563560=539407,2717355=490285} 2024-03-04T06:18:14.395 INFO:tasks.rados.rados.0.smithi064.stdout:2519: writing smithi06431743-40 from 662506 to 1086504 tid 1 2024-03-04T06:18:14.400 INFO:tasks.rados.rados.0.smithi064.stdout:2519: writing smithi06431743-40 from 1563560 to 2102967 tid 2 2024-03-04T06:18:14.407 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:18:14.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:18:14.408 INFO:tasks.rados.rados.0.smithi064.stdout:2519: writing smithi06431743-40 from 2717355 to 3207640 tid 3 2024-03-04T06:18:14.408 INFO:tasks.rados.rados.0.smithi064.stdout:2520: write oid 14 current snap is 310 2024-03-04T06:18:14.408 INFO:tasks.rados.rados.0.smithi064.stdout:2520: seq_num 1159 ranges {679313=499247,1622042=683065,2744778=701557} 2024-03-04T06:18:14.414 INFO:tasks.rados.rados.0.smithi064.stdout:2520: writing smithi06431743-14 from 679313 to 1178560 tid 1 2024-03-04T06:18:14.421 INFO:tasks.rados.rados.0.smithi064.stdout:2520: writing smithi06431743-14 from 1622042 to 2305107 tid 2 2024-03-04T06:18:14.430 INFO:tasks.rados.rados.0.smithi064.stdout:2520: writing smithi06431743-14 from 2744778 to 3446335 tid 3 2024-03-04T06:18:14.430 INFO:tasks.rados.rados.0.smithi064.stdout:2521: read oid 4 snap -1 2024-03-04T06:18:14.430 INFO:tasks.rados.rados.0.smithi064.stdout:2521: expect (ObjNum 1154 snap 309 seq_num 1154) 2024-03-04T06:18:14.430 INFO:tasks.rados.rados.0.smithi064.stdout:2522: delete oid 27 current snap is 310 2024-03-04T06:18:14.504 INFO:tasks.rados.rados.0.smithi064.stdout:2521: expect (ObjNum 1154 snap 309 seq_num 1154) 2024-03-04T06:18:14.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:14.544+0000 7f0d77f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:14.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:14.555+0000 7f0d77f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:14.745 INFO:tasks.rados.rados.0.smithi064.stdout:2520: finishing write tid 1 to smithi06431743-14 2024-03-04T06:18:14.745 INFO:tasks.rados.rados.0.smithi064.stdout:2519: finishing write tid 1 to smithi06431743-40 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2519: finishing write tid 2 to smithi06431743-40 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2520: finishing write tid 2 to smithi06431743-14 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2519: finishing write tid 3 to smithi06431743-40 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2521: done (3 left) 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2522: done (2 left) 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2523: write oid 44 current snap is 310 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2519: finishing write tid 4 to smithi06431743-40 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2519: finishing write tid 5 to smithi06431743-40 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2519: oid 40 updating version 0 to 544 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2519: oid 40 updating version 544 to 546 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2519: oid 40 version 546 is already newer than 546 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2519: oid 40 version 546 is already newer than 543 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 546 (ObjNum 1158 snap 310 seq_num 1158) dirty exists 2024-03-04T06:18:14.746 INFO:tasks.rados.rados.0.smithi064.stdout:2519: left oid 40 (ObjNum 1158 snap 310 seq_num 1158) 2024-03-04T06:18:14.747 INFO:tasks.rados.rados.0.smithi064.stdout:2520: finishing write tid 3 to smithi06431743-14 2024-03-04T06:18:14.747 INFO:tasks.rados.rados.0.smithi064.stdout:2520: finishing write tid 4 to smithi06431743-14 2024-03-04T06:18:14.747 INFO:tasks.rados.rados.0.smithi064.stdout:2523: seq_num 1161 ranges {712927=649745,2139006=686734} 2024-03-04T06:18:14.752 INFO:tasks.rados.rados.0.smithi064.stdout:2523: writing smithi06431743-44 from 712927 to 1362672 tid 1 2024-03-04T06:18:14.756 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:18:14.756 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:18:14.763 INFO:tasks.rados.rados.0.smithi064.stdout:2523: writing smithi06431743-44 from 2139006 to 2825740 tid 2 2024-03-04T06:18:14.763 INFO:tasks.rados.rados.0.smithi064.stdout:2519: done (2 left) 2024-03-04T06:18:14.763 INFO:tasks.rados.rados.0.smithi064.stdout:2524: snap_remove snap 309 2024-03-04T06:18:14.771 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":929,"stamp":"2024-03-04T06:18:14.387652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112467363,"num_objects":857,"num_object_clones":670,"num_object_copies":2571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4700,"num_read_kb":3598055,"num_write":3995,"num_write_kb":1050137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8940,"ondisk_log_size":8940,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":568632,"kb_used_data":384544,"kb_used_omap":0,"kb_used_meta":183616,"kb_avail":376918728,"statfs":{"total":386547056640,"available":385964777472,"internally_reserved":0,"allocated":393773056,"data_stored":455895623,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":188022784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9496483,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":34,"num_read_kb":23767,"num_write":30,"num_write_kb":8965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017789"},"pg_stats":[{"pgid":"3.f","version":"616'531","reported_seq":2136,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.442552+0000","last_change":"2024-03-04T06:18:09.442552+0000","last_active":"2024-03-04T06:18:09.442552+0000","last_peered":"2024-03-04T06:18:09.442552+0000","last_clean":"2024-03-04T06:18:09.442552+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:18:09.442552+0000","last_undegraded":"2024-03-04T06:18:09.442552+0000","last_fullsized":"2024-03-04T06:18:09.442552+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":44,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":271,"num_read_kb":224365,"num_write":223,"num_write_kb":59334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"616'427","reported_seq":2080,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.538474+0000","last_change":"2024-03-04T06:18:09.538474+0000","last_active":"2024-03-04T06:18:09.538474+0000","last_peered":"2024-03-04T06:18:09.538474+0000","last_clean":"2024-03-04T06:18:09.538474+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:18:09.538474+0000","last_undegraded":"2024-03-04T06:18:09.538474+0000","last_fullsized":"2024-03-04T06:18:09.538474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9563942,"num_objects":45,"num_object_clones":31,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":152,"num_read_kb":97766,"num_write":185,"num_write_kb":49317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"619'575","reported_seq":2234,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.540326+0000","last_change":"2024-03-04T06:18:09.588775+0000","last_active":"2024-03-04T06:18:09.540326+0000","last_peered":"2024-03-04T06:18:09.540326+0000","last_clean":"2024-03-04T06:18:09.540326+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:18:09.540326+0000","last_undegraded":"2024-03-04T06:18:09.540326+0000","last_fullsized":"2024-03-04T06:18:09.540326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067855,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":355,"num_read_kb":282556,"num_write":268,"num_write_kb":73724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"619'438","reported_seq":2093,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.407660+0000","last_change":"2024-03-04T06:18:09.407487+0000","last_active":"2024-03-04T06:18:09.407660+0000","last_peered":"2024-03-04T06:18:09.407660+0000","last_clean":"2024-03-04T06:18:09.407660+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:18:09.407660+0000","last_undegraded":"2024-03-04T06:18:09.407660+0000","last_fullsized":"2024-03-04T06:18:09.407660+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017434,"num_objects":41,"num_object_clones":31,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":222,"num_read_kb":169191,"num_write":197,"num_write_kb":54669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"619'209","reported_seq":1940,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.550732+0000","last_change":"2024-03-04T06:18:09.404002+0000","last_active":"2024-03-04T06:18:09.550732+0000","last_peered":"2024-03-04T06:18:09.550732+0000","last_clean":"2024-03-04T06:18:09.550732+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:18:09.550732+0000","last_undegraded":"2024-03-04T06:18:09.550732+0000","last_fullsized":"2024-03-04T06:18:09.550732+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361664,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":76,"num_read_kb":47658,"num_write":93,"num_write_kb":25364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"619'604","reported_seq":2187,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.509052+0000","last_change":"2024-03-04T06:18:09.445105+0000","last_active":"2024-03-04T06:18:09.509052+0000","last_peered":"2024-03-04T06:18:09.509052+0000","last_clean":"2024-03-04T06:18:09.509052+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:18:09.509052+0000","last_undegraded":"2024-03-04T06:18:09.509052+0000","last_fullsized":"2024-03-04T06:18:09.509052+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8824434,"num_objects":53,"num_object_clones":44,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":295,"num_read_kb":228239,"num_write":269,"num_write_kb":78133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1817,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.453457+0000","last_change":"2024-03-04T06:18:09.467989+0000","last_active":"2024-03-04T06:18:09.453457+0000","last_peered":"2024-03-04T06:18:09.453457+0000","last_clean":"2024-03-04T06:18:09.453457+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:18:09.453457+0000","last_undegraded":"2024-03-04T06:18:09.453457+0000","last_fullsized":"2024-03-04T06:18:09.453457+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"619'334","reported_seq":2039,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.537323+0000","last_change":"2024-03-04T06:18:09.595728+0000","last_active":"2024-03-04T06:18:09.537323+0000","last_peered":"2024-03-04T06:18:09.537323+0000","last_clean":"2024-03-04T06:18:09.537323+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:18:09.537323+0000","last_undegraded":"2024-03-04T06:18:09.537323+0000","last_fullsized":"2024-03-04T06:18:09.537323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2638963,"num_objects":30,"num_object_clones":21,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":186,"num_read_kb":169869,"num_write":132,"num_write_kb":34673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1676,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.388840+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:18:09.388840+0000","last_peered":"2024-03-04T06:18:09.388840+0000","last_clean":"2024-03-04T06:18:09.388840+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:18:09.388840+0000","last_undegraded":"2024-03-04T06:18:09.388840+0000","last_fullsized":"2024-03-04T06:18:09.388840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1555,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.548374+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:18:09.548374+0000","last_peered":"2024-03-04T06:18:09.548374+0000","last_clean":"2024-03-04T06:18:09.548374+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:18:09.548374+0000","last_undegraded":"2024-03-04T06:18:09.548374+0000","last_fullsized":"2024-03-04T06:18:09.548374+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"619'583","reported_seq":2193,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.547768+0000","last_change":"2024-03-04T06:18:09.592974+0000","last_active":"2024-03-04T06:18:09.547768+0000","last_peered":"2024-03-04T06:18:09.547768+0000","last_clean":"2024-03-04T06:18:09.547768+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:18:09.547768+0000","last_undegraded":"2024-03-04T06:18:09.547768+0000","last_fullsized":"2024-03-04T06:18:09.547768+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2958296,"num_objects":56,"num_object_clones":44,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":300,"num_read_kb":240944,"num_write":249,"num_write_kb":67085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1535,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.436063+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:18:09.436063+0000","last_peered":"2024-03-04T06:18:09.436063+0000","last_clean":"2024-03-04T06:18:09.436063+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:18:09.436063+0000","last_undegraded":"2024-03-04T06:18:09.436063+0000","last_fullsized":"2024-03-04T06:18:09.436063+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"619'387","reported_seq":2059,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.472832+0000","last_change":"2024-03-04T06:18:09.401124+0000","last_active":"2024-03-04T06:18:09.472832+0000","last_peered":"2024-03-04T06:18:09.472832+0000","last_clean":"2024-03-04T06:18:09.472832+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:18:09.472832+0000","last_undegraded":"2024-03-04T06:18:09.472832+0000","last_fullsized":"2024-03-04T06:18:09.472832+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698101,"num_objects":37,"num_object_clones":29,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":220,"num_read_kb":177502,"num_write":151,"num_write_kb":42787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1555,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.540995+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:18:09.540995+0000","last_peered":"2024-03-04T06:18:09.540995+0000","last_clean":"2024-03-04T06:18:09.540995+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:18:09.540995+0000","last_undegraded":"2024-03-04T06:18:09.540995+0000","last_fullsized":"2024-03-04T06:18:09.540995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"619'1028","reported_seq":2520,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.524870+0000","last_change":"2024-03-04T06:18:09.436562+0000","last_active":"2024-03-04T06:18:09.524870+0000","last_peered":"2024-03-04T06:18:09.524870+0000","last_clean":"2024-03-04T06:18:09.524870+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:18:09.524870+0000","last_undegraded":"2024-03-04T06:18:09.524870+0000","last_fullsized":"2024-03-04T06:18:09.524870+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1028,"log_dups_size":0,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14841901,"num_objects":101,"num_object_clones":78,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":593,"num_read_kb":465272,"num_write":478,"num_write_kb":127819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1535,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.420617+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:18:09.420617+0000","last_peered":"2024-03-04T06:18:09.420617+0000","last_clean":"2024-03-04T06:18:09.420617+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:18:09.420617+0000","last_undegraded":"2024-03-04T06:18:09.420617+0000","last_fullsized":"2024-03-04T06:18:09.420617+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"619'1109","reported_seq":2483,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.653526+0000","last_change":"2024-03-04T06:18:09.547004+0000","last_active":"2024-03-04T06:18:09.653526+0000","last_peered":"2024-03-04T06:18:09.653526+0000","last_clean":"2024-03-04T06:18:09.653526+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:18:09.653526+0000","last_undegraded":"2024-03-04T06:18:09.653526+0000","last_fullsized":"2024-03-04T06:18:09.653526+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1109,"log_dups_size":0,"ondisk_log_size":1109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17379692,"num_objects":103,"num_object_clones":81,"num_object_copies":309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":473,"num_read_kb":347940,"num_write":476,"num_write_kb":126011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"619'538","reported_seq":2219,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.544133+0000","last_change":"2024-03-04T06:18:09.630457+0000","last_active":"2024-03-04T06:18:09.544133+0000","last_peered":"2024-03-04T06:18:09.544133+0000","last_clean":"2024-03-04T06:18:09.544133+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:18:09.544133+0000","last_undegraded":"2024-03-04T06:18:09.544133+0000","last_fullsized":"2024-03-04T06:18:09.544133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8320257,"num_objects":51,"num_object_clones":39,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":362,"num_read_kb":299847,"num_write":241,"num_write_kb":65778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1571,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.552143+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:18:09.552143+0000","last_peered":"2024-03-04T06:18:09.552143+0000","last_clean":"2024-03-04T06:18:09.552143+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:18:09.552143+0000","last_undegraded":"2024-03-04T06:18:09.552143+0000","last_fullsized":"2024-03-04T06:18:09.552143+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"616'506","reported_seq":2175,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.550840+0000","last_change":"2024-03-04T06:18:09.550840+0000","last_active":"2024-03-04T06:18:09.550840+0000","last_peered":"2024-03-04T06:18:09.550840+0000","last_clean":"2024-03-04T06:18:09.550840+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:18:09.550840+0000","last_undegraded":"2024-03-04T06:18:09.550840+0000","last_fullsized":"2024-03-04T06:18:09.550840+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9076572,"num_objects":57,"num_object_clones":41,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":266,"num_read_kb":192934,"num_write":232,"num_write_kb":61450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1543,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.399678+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:18:09.399678+0000","last_peered":"2024-03-04T06:18:09.399678+0000","last_clean":"2024-03-04T06:18:09.399678+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:18:09.399678+0000","last_undegraded":"2024-03-04T06:18:09.399678+0000","last_fullsized":"2024-03-04T06:18:09.399678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"619'724","reported_seq":2265,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.389822+0000","last_change":"2024-03-04T06:18:09.450125+0000","last_active":"2024-03-04T06:18:09.389822+0000","last_peered":"2024-03-04T06:18:09.389822+0000","last_clean":"2024-03-04T06:18:09.389822+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:18:09.389822+0000","last_undegraded":"2024-03-04T06:18:09.389822+0000","last_fullsized":"2024-03-04T06:18:09.389822+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":724,"log_dups_size":0,"ondisk_log_size":724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10438203,"num_objects":69,"num_object_clones":55,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":355,"num_read_kb":263485,"num_write":319,"num_write_kb":87264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1571,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.541218+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:18:09.541218+0000","last_peered":"2024-03-04T06:18:09.541218+0000","last_clean":"2024-03-04T06:18:09.541218+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:18:09.541218+0000","last_undegraded":"2024-03-04T06:18:09.541218+0000","last_fullsized":"2024-03-04T06:18:09.541218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"616'850","reported_seq":2374,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.543629+0000","last_change":"2024-03-04T06:18:09.543629+0000","last_active":"2024-03-04T06:18:09.543629+0000","last_peered":"2024-03-04T06:18:09.543629+0000","last_clean":"2024-03-04T06:18:09.543629+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:18:09.543629+0000","last_undegraded":"2024-03-04T06:18:09.543629+0000","last_fullsized":"2024-03-04T06:18:09.543629+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":850,"log_dups_size":0,"ondisk_log_size":850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6689681,"num_objects":75,"num_object_clones":66,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":471,"num_read_kb":390396,"num_write":345,"num_write_kb":94835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1555,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.545808+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:18:09.545808+0000","last_peered":"2024-03-04T06:18:09.545808+0000","last_clean":"2024-03-04T06:18:09.545808+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:18:09.545808+0000","last_undegraded":"2024-03-04T06:18:09.545808+0000","last_fullsized":"2024-03-04T06:18:09.545808+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":111876995,"num_objects":855,"num_object_clones":670,"num_object_copies":2565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4597,"num_read_kb":3597964,"num_write":3858,"num_write_kb":1048243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8843,"ondisk_log_size":8843,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706289,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":169868,"kb_used_data":120724,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94201972,"statfs":{"total":96636764160,"available":96462819328,"internally_reserved":0,"allocated":123621376,"data_stored":144725921,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706289,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":129828,"kb_used_data":84844,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94242012,"statfs":{"total":96636764160,"available":96503820288,"internally_reserved":0,"allocated":86880256,"data_stored":98340329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706288,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":145344,"kb_used_data":100256,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94226496,"statfs":{"total":96636764160,"available":96487931904,"internally_reserved":0,"allocated":102662144,"data_stored":121076489,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706288,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123592,"kb_used_data":78720,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94248248,"statfs":{"total":96636764160,"available":96510205952,"internally_reserved":0,"allocated":80609280,"data_stored":91752884,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:18:14.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:18:14.918 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:14.914+0000 7fec0026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:14.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:14.921+0000 7fec0026d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:15.109 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:18:15.109 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:18:15.121 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":930,"stamp":"2024-03-04T06:18:14.839645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112467363,"num_objects":857,"num_object_clones":670,"num_object_copies":2571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4700,"num_read_kb":3598055,"num_write":3995,"num_write_kb":1050137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8940,"ondisk_log_size":8940,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":568632,"kb_used_data":384544,"kb_used_omap":0,"kb_used_meta":183616,"kb_avail":376918728,"statfs":{"total":386547056640,"available":385964777472,"internally_reserved":0,"allocated":393773056,"data_stored":455895623,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":188022784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4823873,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":17794,"num_write":15,"num_write_kb":4672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000638"},"pg_stats":[{"pgid":"3.f","version":"616'531","reported_seq":2136,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.442552+0000","last_change":"2024-03-04T06:18:09.442552+0000","last_active":"2024-03-04T06:18:09.442552+0000","last_peered":"2024-03-04T06:18:09.442552+0000","last_clean":"2024-03-04T06:18:09.442552+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:18:09.442552+0000","last_undegraded":"2024-03-04T06:18:09.442552+0000","last_fullsized":"2024-03-04T06:18:09.442552+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":55,"num_object_clones":44,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":271,"num_read_kb":224365,"num_write":223,"num_write_kb":59334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"616'427","reported_seq":2080,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.538474+0000","last_change":"2024-03-04T06:18:09.538474+0000","last_active":"2024-03-04T06:18:09.538474+0000","last_peered":"2024-03-04T06:18:09.538474+0000","last_clean":"2024-03-04T06:18:09.538474+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:18:09.538474+0000","last_undegraded":"2024-03-04T06:18:09.538474+0000","last_fullsized":"2024-03-04T06:18:09.538474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9563942,"num_objects":45,"num_object_clones":31,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":152,"num_read_kb":97766,"num_write":185,"num_write_kb":49317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"619'575","reported_seq":2234,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.540326+0000","last_change":"2024-03-04T06:18:09.588775+0000","last_active":"2024-03-04T06:18:09.540326+0000","last_peered":"2024-03-04T06:18:09.540326+0000","last_clean":"2024-03-04T06:18:09.540326+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:18:09.540326+0000","last_undegraded":"2024-03-04T06:18:09.540326+0000","last_fullsized":"2024-03-04T06:18:09.540326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067855,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":355,"num_read_kb":282556,"num_write":268,"num_write_kb":73724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"619'438","reported_seq":2093,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.407660+0000","last_change":"2024-03-04T06:18:09.407487+0000","last_active":"2024-03-04T06:18:09.407660+0000","last_peered":"2024-03-04T06:18:09.407660+0000","last_clean":"2024-03-04T06:18:09.407660+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:18:09.407660+0000","last_undegraded":"2024-03-04T06:18:09.407660+0000","last_fullsized":"2024-03-04T06:18:09.407660+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017434,"num_objects":41,"num_object_clones":31,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":222,"num_read_kb":169191,"num_write":197,"num_write_kb":54669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"619'209","reported_seq":1940,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.550732+0000","last_change":"2024-03-04T06:18:09.404002+0000","last_active":"2024-03-04T06:18:09.550732+0000","last_peered":"2024-03-04T06:18:09.550732+0000","last_clean":"2024-03-04T06:18:09.550732+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:18:09.550732+0000","last_undegraded":"2024-03-04T06:18:09.550732+0000","last_fullsized":"2024-03-04T06:18:09.550732+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361664,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":76,"num_read_kb":47658,"num_write":93,"num_write_kb":25364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"619'604","reported_seq":2187,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.509052+0000","last_change":"2024-03-04T06:18:09.445105+0000","last_active":"2024-03-04T06:18:09.509052+0000","last_peered":"2024-03-04T06:18:09.509052+0000","last_clean":"2024-03-04T06:18:09.509052+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:18:09.509052+0000","last_undegraded":"2024-03-04T06:18:09.509052+0000","last_fullsized":"2024-03-04T06:18:09.509052+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8824434,"num_objects":53,"num_object_clones":44,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":295,"num_read_kb":228239,"num_write":269,"num_write_kb":78133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1817,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.453457+0000","last_change":"2024-03-04T06:18:09.467989+0000","last_active":"2024-03-04T06:18:09.453457+0000","last_peered":"2024-03-04T06:18:09.453457+0000","last_clean":"2024-03-04T06:18:09.453457+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:18:09.453457+0000","last_undegraded":"2024-03-04T06:18:09.453457+0000","last_fullsized":"2024-03-04T06:18:09.453457+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"619'334","reported_seq":2039,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.537323+0000","last_change":"2024-03-04T06:18:09.595728+0000","last_active":"2024-03-04T06:18:09.537323+0000","last_peered":"2024-03-04T06:18:09.537323+0000","last_clean":"2024-03-04T06:18:09.537323+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:18:09.537323+0000","last_undegraded":"2024-03-04T06:18:09.537323+0000","last_fullsized":"2024-03-04T06:18:09.537323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2638963,"num_objects":30,"num_object_clones":21,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":186,"num_read_kb":169869,"num_write":132,"num_write_kb":34673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1676,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.388840+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:18:09.388840+0000","last_peered":"2024-03-04T06:18:09.388840+0000","last_clean":"2024-03-04T06:18:09.388840+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:18:09.388840+0000","last_undegraded":"2024-03-04T06:18:09.388840+0000","last_fullsized":"2024-03-04T06:18:09.388840+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1555,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.548374+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:18:09.548374+0000","last_peered":"2024-03-04T06:18:09.548374+0000","last_clean":"2024-03-04T06:18:09.548374+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:18:09.548374+0000","last_undegraded":"2024-03-04T06:18:09.548374+0000","last_fullsized":"2024-03-04T06:18:09.548374+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"619'583","reported_seq":2193,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.547768+0000","last_change":"2024-03-04T06:18:09.592974+0000","last_active":"2024-03-04T06:18:09.547768+0000","last_peered":"2024-03-04T06:18:09.547768+0000","last_clean":"2024-03-04T06:18:09.547768+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:18:09.547768+0000","last_undegraded":"2024-03-04T06:18:09.547768+0000","last_fullsized":"2024-03-04T06:18:09.547768+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2958296,"num_objects":56,"num_object_clones":44,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":300,"num_read_kb":240944,"num_write":249,"num_write_kb":67085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1535,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.436063+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:18:09.436063+0000","last_peered":"2024-03-04T06:18:09.436063+0000","last_clean":"2024-03-04T06:18:09.436063+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:18:09.436063+0000","last_undegraded":"2024-03-04T06:18:09.436063+0000","last_fullsized":"2024-03-04T06:18:09.436063+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"619'387","reported_seq":2059,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.472832+0000","last_change":"2024-03-04T06:18:09.401124+0000","last_active":"2024-03-04T06:18:09.472832+0000","last_peered":"2024-03-04T06:18:09.472832+0000","last_clean":"2024-03-04T06:18:09.472832+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:18:09.472832+0000","last_undegraded":"2024-03-04T06:18:09.472832+0000","last_fullsized":"2024-03-04T06:18:09.472832+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698101,"num_objects":37,"num_object_clones":29,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":220,"num_read_kb":177502,"num_write":151,"num_write_kb":42787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1555,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.540995+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:18:09.540995+0000","last_peered":"2024-03-04T06:18:09.540995+0000","last_clean":"2024-03-04T06:18:09.540995+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:18:09.540995+0000","last_undegraded":"2024-03-04T06:18:09.540995+0000","last_fullsized":"2024-03-04T06:18:09.540995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"619'1028","reported_seq":2520,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.524870+0000","last_change":"2024-03-04T06:18:09.436562+0000","last_active":"2024-03-04T06:18:09.524870+0000","last_peered":"2024-03-04T06:18:09.524870+0000","last_clean":"2024-03-04T06:18:09.524870+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:18:09.524870+0000","last_undegraded":"2024-03-04T06:18:09.524870+0000","last_fullsized":"2024-03-04T06:18:09.524870+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1028,"log_dups_size":0,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14841901,"num_objects":101,"num_object_clones":78,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":593,"num_read_kb":465272,"num_write":478,"num_write_kb":127819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1535,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.420617+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:18:09.420617+0000","last_peered":"2024-03-04T06:18:09.420617+0000","last_clean":"2024-03-04T06:18:09.420617+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:18:09.420617+0000","last_undegraded":"2024-03-04T06:18:09.420617+0000","last_fullsized":"2024-03-04T06:18:09.420617+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"619'1109","reported_seq":2483,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.653526+0000","last_change":"2024-03-04T06:18:09.547004+0000","last_active":"2024-03-04T06:18:09.653526+0000","last_peered":"2024-03-04T06:18:09.653526+0000","last_clean":"2024-03-04T06:18:09.653526+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:18:09.653526+0000","last_undegraded":"2024-03-04T06:18:09.653526+0000","last_fullsized":"2024-03-04T06:18:09.653526+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1109,"log_dups_size":0,"ondisk_log_size":1109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17379692,"num_objects":103,"num_object_clones":81,"num_object_copies":309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":473,"num_read_kb":347940,"num_write":476,"num_write_kb":126011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"619'538","reported_seq":2219,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.544133+0000","last_change":"2024-03-04T06:18:09.630457+0000","last_active":"2024-03-04T06:18:09.544133+0000","last_peered":"2024-03-04T06:18:09.544133+0000","last_clean":"2024-03-04T06:18:09.544133+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:18:09.544133+0000","last_undegraded":"2024-03-04T06:18:09.544133+0000","last_fullsized":"2024-03-04T06:18:09.544133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8320257,"num_objects":51,"num_object_clones":39,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":362,"num_read_kb":299847,"num_write":241,"num_write_kb":65778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1571,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.552143+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:18:09.552143+0000","last_peered":"2024-03-04T06:18:09.552143+0000","last_clean":"2024-03-04T06:18:09.552143+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:18:09.552143+0000","last_undegraded":"2024-03-04T06:18:09.552143+0000","last_fullsized":"2024-03-04T06:18:09.552143+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"616'506","reported_seq":2175,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.550840+0000","last_change":"2024-03-04T06:18:09.550840+0000","last_active":"2024-03-04T06:18:09.550840+0000","last_peered":"2024-03-04T06:18:09.550840+0000","last_clean":"2024-03-04T06:18:09.550840+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:18:09.550840+0000","last_undegraded":"2024-03-04T06:18:09.550840+0000","last_fullsized":"2024-03-04T06:18:09.550840+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9076572,"num_objects":57,"num_object_clones":41,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":266,"num_read_kb":192934,"num_write":232,"num_write_kb":61450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1543,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.399678+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:18:09.399678+0000","last_peered":"2024-03-04T06:18:09.399678+0000","last_clean":"2024-03-04T06:18:09.399678+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:18:09.399678+0000","last_undegraded":"2024-03-04T06:18:09.399678+0000","last_fullsized":"2024-03-04T06:18:09.399678+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"619'724","reported_seq":2265,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.389822+0000","last_change":"2024-03-04T06:18:09.450125+0000","last_active":"2024-03-04T06:18:09.389822+0000","last_peered":"2024-03-04T06:18:09.389822+0000","last_clean":"2024-03-04T06:18:09.389822+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:18:09.389822+0000","last_undegraded":"2024-03-04T06:18:09.389822+0000","last_fullsized":"2024-03-04T06:18:09.389822+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":724,"log_dups_size":0,"ondisk_log_size":724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10438203,"num_objects":69,"num_object_clones":55,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":355,"num_read_kb":263485,"num_write":319,"num_write_kb":87264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1571,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.541218+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:18:09.541218+0000","last_peered":"2024-03-04T06:18:09.541218+0000","last_clean":"2024-03-04T06:18:09.541218+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:18:09.541218+0000","last_undegraded":"2024-03-04T06:18:09.541218+0000","last_fullsized":"2024-03-04T06:18:09.541218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"616'850","reported_seq":2374,"reported_epoch":619,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:09.543629+0000","last_change":"2024-03-04T06:18:09.543629+0000","last_active":"2024-03-04T06:18:09.543629+0000","last_peered":"2024-03-04T06:18:09.543629+0000","last_clean":"2024-03-04T06:18:09.543629+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:18:09.543629+0000","last_undegraded":"2024-03-04T06:18:09.543629+0000","last_fullsized":"2024-03-04T06:18:09.543629+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":850,"log_dups_size":0,"ondisk_log_size":850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6689681,"num_objects":75,"num_object_clones":66,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":471,"num_read_kb":390396,"num_write":345,"num_write_kb":94835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1555,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-04T06:18:09.545808+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:18:09.545808+0000","last_peered":"2024-03-04T06:18:09.545808+0000","last_clean":"2024-03-04T06:18:09.545808+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:18:09.545808+0000","last_undegraded":"2024-03-04T06:18:09.545808+0000","last_fullsized":"2024-03-04T06:18:09.545808+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":111876995,"num_objects":855,"num_object_clones":670,"num_object_copies":2565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4597,"num_read_kb":3597964,"num_write":3858,"num_write_kb":1048243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8843,"ondisk_log_size":8843,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706289,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":169868,"kb_used_data":120724,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94201972,"statfs":{"total":96636764160,"available":96462819328,"internally_reserved":0,"allocated":123621376,"data_stored":144725921,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706289,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":129828,"kb_used_data":84844,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94242012,"statfs":{"total":96636764160,"available":96503820288,"internally_reserved":0,"allocated":86880256,"data_stored":98340329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706288,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":145344,"kb_used_data":100256,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94226496,"statfs":{"total":96636764160,"available":96487931904,"internally_reserved":0,"allocated":102662144,"data_stored":121076489,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706288,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123592,"kb_used_data":78720,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94248248,"statfs":{"total":96636764160,"available":96510205952,"internally_reserved":0,"allocated":80609280,"data_stored":91752884,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:18:15.122 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:18:15.123 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:18:15.123 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:18:15.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:18:15.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:15.226+0000 7f72fa15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:15.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:15.233+0000 7f72fa15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:15.246 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:15.246 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:15.246 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:15.246 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:15.247 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:15.249 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:15.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:18:15.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:15.353+0000 7f85e6d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:15.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:15.360+0000 7f85e6d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:15.377 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:15.378 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:15.378 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:15.378 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:15.378 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:15.378 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:15.378 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:15.378 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:15.378 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:15.378 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:15.378 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:15.378 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:15.380 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:15.381 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:15.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:18:15.397 INFO:tasks.rados.rados.0.smithi064.stdout:2524: done (2 left) 2024-03-04T06:18:15.397 INFO:tasks.rados.rados.0.smithi064.stdout:2525: write oid 13 current snap is 310 2024-03-04T06:18:15.397 INFO:tasks.rados.rados.0.smithi064.stdout:2520: finishing write tid 5 to smithi06431743-14 2024-03-04T06:18:15.397 INFO:tasks.rados.rados.0.smithi064.stdout:2520: oid 14 updating version 0 to 734 2024-03-04T06:18:15.397 INFO:tasks.rados.rados.0.smithi064.stdout:2520: oid 14 updating version 734 to 737 2024-03-04T06:18:15.397 INFO:tasks.rados.rados.0.smithi064.stdout:2520: oid 14 version 737 is already newer than 737 2024-03-04T06:18:15.397 INFO:tasks.rados.rados.0.smithi064.stdout:2520: oid 14 version 737 is already newer than 735 2024-03-04T06:18:15.397 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 737 (ObjNum 1159 snap 310 seq_num 1159) dirty exists 2024-03-04T06:18:15.397 INFO:tasks.rados.rados.0.smithi064.stdout:2520: left oid 14 (ObjNum 1159 snap 310 seq_num 1159) 2024-03-04T06:18:15.397 INFO:tasks.rados.rados.0.smithi064.stdout:2525: seq_num 1162 ranges {729734=724994,2197488=430392,3227047=302687} 2024-03-04T06:18:15.404 INFO:tasks.rados.rados.0.smithi064.stdout:2525: writing smithi06431743-13 from 729734 to 1454728 tid 1 2024-03-04T06:18:15.411 INFO:tasks.rados.rados.0.smithi064.stdout:2525: writing smithi06431743-13 from 2197488 to 2627880 tid 2 2024-03-04T06:18:15.416 INFO:tasks.rados.rados.0.smithi064.stdout:2525: writing smithi06431743-13 from 3227047 to 3529734 tid 3 2024-03-04T06:18:15.416 INFO:tasks.rados.rados.0.smithi064.stdout:2520: done (2 left) 2024-03-04T06:18:15.416 INFO:tasks.rados.rados.0.smithi064.stdout:2526: read oid 33 snap -1 2024-03-04T06:18:15.416 INFO:tasks.rados.rados.0.smithi064.stdout:2526: expect (ObjNum 1042 snap 265 seq_num 1042) 2024-03-04T06:18:15.416 INFO:tasks.rados.rados.0.smithi064.stdout:2527: snap_remove snap 290 2024-03-04T06:18:15.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:15.488+0000 7f861066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:15.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:15.495+0000 7f861066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:15.508 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:15.509 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:15.509 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:15.511 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:15.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:18:15.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:15.656+0000 7fc54fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:15.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:15.663+0000 7fc54fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:15.676 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:15.677 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:15.679 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:16.006 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:16.112 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:16.110+0000 7fb09471d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:16.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:16.115+0000 7fb09471d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:16.396 INFO:tasks.rados.rados.0.smithi064.stdout:2527: done (3 left) 2024-03-04T06:18:16.397 INFO:tasks.rados.rados.0.smithi064.stdout:2528: snap_remove snap 306 2024-03-04T06:18:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:2528: done (3 left) 2024-03-04T06:18:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:2529: snap_create 2024-03-04T06:18:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:2523: finishing write tid 1 to smithi06431743-44 2024-03-04T06:18:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:2523: finishing write tid 2 to smithi06431743-44 2024-03-04T06:18:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:2523: finishing write tid 3 to smithi06431743-44 2024-03-04T06:18:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:2523: finishing write tid 4 to smithi06431743-44 2024-03-04T06:18:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:2523: oid 44 updating version 0 to 617 2024-03-04T06:18:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:2523: oid 44 version 617 is already newer than 616 2024-03-04T06:18:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:2523: oid 44 updating version 617 to 618 2024-03-04T06:18:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 618 (ObjNum 1161 snap 310 seq_num 1161) dirty exists 2024-03-04T06:18:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:2523: left oid 44 (ObjNum 1161 snap 310 seq_num 1161) 2024-03-04T06:18:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:2525: finishing write tid 1 to smithi06431743-13 2024-03-04T06:18:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:2526: expect (ObjNum 1042 snap 265 seq_num 1042) 2024-03-04T06:18:17.628 INFO:tasks.rados.rados.0.smithi064.stdout:2525: finishing write tid 2 to smithi06431743-13 2024-03-04T06:18:17.628 INFO:tasks.rados.rados.0.smithi064.stdout:2525: finishing write tid 3 to smithi06431743-13 2024-03-04T06:18:17.628 INFO:tasks.rados.rados.0.smithi064.stdout:2525: finishing write tid 4 to smithi06431743-13 2024-03-04T06:18:17.628 INFO:tasks.rados.rados.0.smithi064.stdout:2525: finishing write tid 5 to smithi06431743-13 2024-03-04T06:18:17.629 INFO:tasks.rados.rados.0.smithi064.stdout:2525: oid 13 updating version 0 to 1130 2024-03-04T06:18:17.629 INFO:tasks.rados.rados.0.smithi064.stdout:2525: oid 13 version 1130 is already newer than 1130 2024-03-04T06:18:17.629 INFO:tasks.rados.rados.0.smithi064.stdout:2525: oid 13 version 1130 is already newer than 1127 2024-03-04T06:18:17.629 INFO:tasks.rados.rados.0.smithi064.stdout:2525: oid 13 version 1130 is already newer than 1128 2024-03-04T06:18:17.629 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1130 (ObjNum 1162 snap 310 seq_num 1162) dirty exists 2024-03-04T06:18:17.629 INFO:tasks.rados.rados.0.smithi064.stdout:2525: left oid 13 (ObjNum 1162 snap 310 seq_num 1162) 2024-03-04T06:18:18.184 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:18.292 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:18.290+0000 7fc2f782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:18.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:18.294+0000 7fc2f782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:18.397 INFO:tasks.rados.rados.0.smithi064.stdout:2523: done (3 left) 2024-03-04T06:18:18.397 INFO:tasks.rados.rados.0.smithi064.stdout:2525: done (2 left) 2024-03-04T06:18:18.397 INFO:tasks.rados.rados.0.smithi064.stdout:2526: done (1 left) 2024-03-04T06:18:18.397 INFO:tasks.rados.rados.0.smithi064.stdout:2529: done (0 left) 2024-03-04T06:18:18.397 INFO:tasks.rados.rados.0.smithi064.stdout:2530: read oid 25 snap 302 2024-03-04T06:18:18.397 INFO:tasks.rados.rados.0.smithi064.stdout:2530: expect deleted 2024-03-04T06:18:18.397 INFO:tasks.rados.rados.0.smithi064.stdout:2531: read oid 9 snap -1 2024-03-04T06:18:18.397 INFO:tasks.rados.rados.0.smithi064.stdout:2531: expect deleted 2024-03-04T06:18:18.397 INFO:tasks.rados.rados.0.smithi064.stdout:2532: read oid 22 snap -1 2024-03-04T06:18:18.397 INFO:tasks.rados.rados.0.smithi064.stdout:2532: expect deleted 2024-03-04T06:18:18.398 INFO:tasks.rados.rados.0.smithi064.stdout:2533: read oid 16 snap 301 2024-03-04T06:18:18.398 INFO:tasks.rados.rados.0.smithi064.stdout:2533: expect (ObjNum 1097 snap 286 seq_num 1097) 2024-03-04T06:18:18.398 INFO:tasks.rados.rados.0.smithi064.stdout:2534: read oid 8 snap 294 2024-03-04T06:18:18.398 INFO:tasks.rados.rados.0.smithi064.stdout:2534: expect (ObjNum 1094 snap 281 seq_num 1094) 2024-03-04T06:18:18.398 INFO:tasks.rados.rados.0.smithi064.stdout:2535: delete oid 27 current snap is 311 2024-03-04T06:18:18.446 INFO:tasks.rados.rados.0.smithi064.stdout:2530: done (5 left) 2024-03-04T06:18:18.446 INFO:tasks.rados.rados.0.smithi064.stdout:2531: done (4 left) 2024-03-04T06:18:18.446 INFO:tasks.rados.rados.0.smithi064.stdout:2535: done (3 left) 2024-03-04T06:18:18.446 INFO:tasks.rados.rados.0.smithi064.stdout:2536: read oid 38 snap -1 2024-03-04T06:18:18.446 INFO:tasks.rados.rados.0.smithi064.stdout:2536: expect (ObjNum 1136 snap 304 seq_num 1136) 2024-03-04T06:18:18.446 INFO:tasks.rados.rados.0.smithi064.stdout:2537: write (excl) oid 47 current snap is 311 2024-03-04T06:18:18.446 INFO:tasks.rados.rados.0.smithi064.stdout:2537: seq_num 1164 ranges {763348=591845,1914452=434061,2881893=674270} 2024-03-04T06:18:18.453 INFO:tasks.rados.rados.0.smithi064.stdout:2537: writing smithi06431743-47 from 763348 to 1355193 tid 1 2024-03-04T06:18:18.458 INFO:tasks.rados.rados.0.smithi064.stdout:2537: writing smithi06431743-47 from 1914452 to 2348513 tid 2 2024-03-04T06:18:18.467 INFO:tasks.rados.rados.0.smithi064.stdout:2537: writing smithi06431743-47 from 2881893 to 3556163 tid 3 2024-03-04T06:18:18.467 INFO:tasks.rados.rados.0.smithi064.stdout:2538: read oid 50 snap -1 2024-03-04T06:18:18.467 INFO:tasks.rados.rados.0.smithi064.stdout:2538: expect (ObjNum 1141 snap 306 seq_num 1141) 2024-03-04T06:18:18.467 INFO:tasks.rados.rados.0.smithi064.stdout:2539: delete oid 27 current snap is 311 2024-03-04T06:18:18.467 INFO:tasks.rados.rados.0.smithi064.stdout:2533: expect (ObjNum 1097 snap 286 seq_num 1097) 2024-03-04T06:18:18.647 INFO:tasks.rados.rados.0.smithi064.stdout:2534: expect (ObjNum 1094 snap 281 seq_num 1094) 2024-03-04T06:18:18.761 INFO:tasks.rados.rados.0.smithi064.stdout:2538: expect (ObjNum 1141 snap 306 seq_num 1141) 2024-03-04T06:18:19.013 INFO:tasks.rados.rados.0.smithi064.stdout:2536: expect (ObjNum 1136 snap 304 seq_num 1136) 2024-03-04T06:18:19.131 INFO:tasks.rados.rados.0.smithi064.stdout:2537: finishing write tid 1 to smithi06431743-47 2024-03-04T06:18:19.131 INFO:tasks.rados.rados.0.smithi064.stdout:2537: finishing write tid 2 to smithi06431743-47 2024-03-04T06:18:19.131 INFO:tasks.rados.rados.0.smithi064.stdout:2537: finishing write tid 3 to smithi06431743-47 2024-03-04T06:18:19.131 INFO:tasks.rados.rados.0.smithi064.stdout:2537: finishing write tid 4 to smithi06431743-47 2024-03-04T06:18:19.131 INFO:tasks.rados.rados.0.smithi064.stdout:2537: finishing write tid 5 to smithi06431743-47 2024-03-04T06:18:19.131 INFO:tasks.rados.rados.0.smithi064.stdout:2537: oid 47 updating version 0 to 457 2024-03-04T06:18:19.131 INFO:tasks.rados.rados.0.smithi064.stdout:2537: oid 47 updating version 457 to 459 2024-03-04T06:18:19.131 INFO:tasks.rados.rados.0.smithi064.stdout:2537: oid 47 version 459 is already newer than 459 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:2537: oid 47 updating version 459 to 460 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 460 (ObjNum 1164 snap 311 seq_num 1164) dirty exists 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:2537: left oid 47 (ObjNum 1164 snap 311 seq_num 1164) 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:2532: done (6 left) 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:2533: done (5 left) 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:2534: done (4 left) 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:2536: done (3 left) 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:2537: done (2 left) 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:2538: done (1 left) 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:2539: done (0 left) 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:2540: write (excl) oid 23 current snap is 311 2024-03-04T06:18:19.132 INFO:tasks.rados.rados.0.smithi064.stdout:2540: seq_num 1166 ranges {796962=742343,2031416=437730,2936739=413167} 2024-03-04T06:18:19.139 INFO:tasks.rados.rados.0.smithi064.stdout:2540: writing smithi06431743-23 from 796962 to 1539305 tid 1 2024-03-04T06:18:19.144 INFO:tasks.rados.rados.0.smithi064.stdout:2540: writing smithi06431743-23 from 2031416 to 2469146 tid 2 2024-03-04T06:18:19.149 INFO:tasks.rados.rados.0.smithi064.stdout:2540: writing smithi06431743-23 from 2936739 to 3349906 tid 3 2024-03-04T06:18:19.149 INFO:tasks.rados.rados.0.smithi064.stdout:2541: read oid 25 snap -1 2024-03-04T06:18:19.149 INFO:tasks.rados.rados.0.smithi064.stdout:2541: expect (ObjNum 1145 snap 308 seq_num 1145) 2024-03-04T06:18:19.149 INFO:tasks.rados.rados.0.smithi064.stdout:2542: delete oid 48 current snap is 311 2024-03-04T06:18:19.163 INFO:tasks.rados.rados.0.smithi064.stdout:2540: finishing write tid 1 to smithi06431743-23 2024-03-04T06:18:19.176 INFO:tasks.rados.rados.0.smithi064.stdout:2541: expect (ObjNum 1145 snap 308 seq_num 1145) 2024-03-04T06:18:19.304 INFO:tasks.rados.rados.0.smithi064.stdout:2540: finishing write tid 2 to smithi06431743-23 2024-03-04T06:18:19.304 INFO:tasks.rados.rados.0.smithi064.stdout:2540: finishing write tid 3 to smithi06431743-23 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2540: finishing write tid 4 to smithi06431743-23 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2540: finishing write tid 5 to smithi06431743-23 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2540: oid 23 updating version 0 to 604 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2540: oid 23 version 604 is already newer than 603 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2540: oid 23 version 604 is already newer than 602 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2540: oid 23 updating version 604 to 605 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 605 (ObjNum 1166 snap 311 seq_num 1166) dirty exists 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2540: left oid 23 (ObjNum 1166 snap 311 seq_num 1166) 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2540: done (2 left) 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2541: done (1 left) 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2542: done (0 left) 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2543: read oid 45 snap -1 2024-03-04T06:18:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2543: expect deleted 2024-03-04T06:18:19.306 INFO:tasks.rados.rados.0.smithi064.stdout:2544: write oid 29 current snap is 311 2024-03-04T06:18:19.306 INFO:tasks.rados.rados.0.smithi064.stdout:2544: seq_num 1168 ranges {430576=492841,1632027=441399,2475232=435711,3365397=265179} 2024-03-04T06:18:19.311 INFO:tasks.rados.rados.0.smithi064.stdout:2544: writing smithi06431743-29 from 430576 to 923417 tid 1 2024-03-04T06:18:19.317 INFO:tasks.rados.rados.0.smithi064.stdout:2544: writing smithi06431743-29 from 1632027 to 2073426 tid 2 2024-03-04T06:18:19.321 INFO:tasks.rados.rados.0.smithi064.stdout:2544: writing smithi06431743-29 from 2475232 to 2910943 tid 3 2024-03-04T06:18:19.326 INFO:tasks.rados.rados.0.smithi064.stdout:2544: writing smithi06431743-29 from 3365397 to 3630576 tid 4 2024-03-04T06:18:19.326 INFO:tasks.rados.rados.0.smithi064.stdout:2545: read oid 18 snap 302 2024-03-04T06:18:19.326 INFO:tasks.rados.rados.0.smithi064.stdout:2545: expect (ObjNum 1061 snap 272 seq_num 1061) 2024-03-04T06:18:19.326 INFO:tasks.rados.rados.0.smithi064.stdout:2546: read oid 19 snap -1 2024-03-04T06:18:19.327 INFO:tasks.rados.rados.0.smithi064.stdout:2546: expect deleted 2024-03-04T06:18:19.327 INFO:tasks.rados.rados.0.smithi064.stdout:2547: write (excl) oid 49 current snap is 311 2024-03-04T06:18:19.327 INFO:tasks.rados.rados.0.smithi064.stdout:2547: seq_num 1169 ranges {447383=568090,1690509=701410,2902655=646983} 2024-03-04T06:18:19.331 INFO:tasks.rados.rados.0.smithi064.stdout:2547: writing smithi06431743-49 from 447383 to 1015473 tid 1 2024-03-04T06:18:19.341 INFO:tasks.rados.rados.0.smithi064.stdout:2547: writing smithi06431743-49 from 1690509 to 2391919 tid 2 2024-03-04T06:18:19.350 INFO:tasks.rados.rados.0.smithi064.stdout:2547: writing smithi06431743-49 from 2902655 to 3549638 tid 3 2024-03-04T06:18:19.350 INFO:tasks.rados.rados.0.smithi064.stdout:2548: read oid 37 snap 285 2024-03-04T06:18:19.350 INFO:tasks.rados.rados.0.smithi064.stdout:2548: expect (ObjNum 1050 snap 268 seq_num 1050) 2024-03-04T06:18:19.350 INFO:tasks.rados.rados.0.smithi064.stdout:2549: write (excl) oid 4 current snap is 311 2024-03-04T06:18:19.350 INFO:tasks.rados.rados.0.smithi064.stdout:2549: seq_num 1170 ranges {464190=643339,1748991=445068,2930078=574608} 2024-03-04T06:18:19.358 INFO:tasks.rados.rados.0.smithi064.stdout:2549: writing smithi06431743-4 from 464190 to 1107529 tid 1 2024-03-04T06:18:19.365 INFO:tasks.rados.rados.0.smithi064.stdout:2549: writing smithi06431743-4 from 1748991 to 2194059 tid 2 2024-03-04T06:18:19.372 INFO:tasks.rados.rados.0.smithi064.stdout:2549: writing smithi06431743-4 from 2930078 to 3504686 tid 3 2024-03-04T06:18:19.373 INFO:tasks.rados.rados.0.smithi064.stdout:2550: read oid 46 snap -1 2024-03-04T06:18:19.373 INFO:tasks.rados.rados.0.smithi064.stdout:2550: expect (ObjNum 1144 snap 307 seq_num 1144) 2024-03-04T06:18:19.373 INFO:tasks.rados.rados.0.smithi064.stdout:2551: snap_remove snap 303 2024-03-04T06:18:20.283 INFO:tasks.rados.rados.0.smithi064.stdout:2551: done (8 left) 2024-03-04T06:18:20.283 INFO:tasks.rados.rados.0.smithi064.stdout:2552: write oid 35 current snap is 311 2024-03-04T06:18:20.283 INFO:tasks.rados.rados.0.smithi064.stdout:2552: seq_num 1171 ranges {480997=434941,1407473=705079,2557501=785880} 2024-03-04T06:18:20.288 INFO:tasks.rados.rados.0.smithi064.stdout:2552: writing smithi06431743-35 from 480997 to 915938 tid 1 2024-03-04T06:18:20.294 INFO:tasks.rados.rados.0.smithi064.stdout:2552: writing smithi06431743-35 from 1407473 to 2112552 tid 2 2024-03-04T06:18:20.302 INFO:tasks.rados.rados.0.smithi064.stdout:2552: writing smithi06431743-35 from 2557501 to 3343381 tid 3 2024-03-04T06:18:20.303 INFO:tasks.rados.rados.0.smithi064.stdout:2543: done (8 left) 2024-03-04T06:18:20.303 INFO:tasks.rados.rados.0.smithi064.stdout:2553: read oid 39 snap -1 2024-03-04T06:18:20.303 INFO:tasks.rados.rados.0.smithi064.stdout:2553: expect (ObjNum 1120 snap 298 seq_num 1120) 2024-03-04T06:18:20.303 INFO:tasks.rados.rados.0.smithi064.stdout:2544: finishing write tid 1 to smithi06431743-29 2024-03-04T06:18:20.303 INFO:tasks.rados.rados.0.smithi064.stdout:2554: read oid 48 snap -1 2024-03-04T06:18:20.303 INFO:tasks.rados.rados.0.smithi064.stdout:2554: expect deleted 2024-03-04T06:18:20.303 INFO:tasks.rados.rados.0.smithi064.stdout:2555: read oid 3 snap 285 2024-03-04T06:18:20.303 INFO:tasks.rados.rados.0.smithi064.stdout:2555: expect (ObjNum 1032 snap 264 seq_num 1032) 2024-03-04T06:18:20.304 INFO:tasks.rados.rados.0.smithi064.stdout:2556: snap_create 2024-03-04T06:18:20.304 INFO:tasks.rados.rados.0.smithi064.stdout:2544: finishing write tid 2 to smithi06431743-29 2024-03-04T06:18:20.304 INFO:tasks.rados.rados.0.smithi064.stdout:2544: finishing write tid 3 to smithi06431743-29 2024-03-04T06:18:20.304 INFO:tasks.rados.rados.0.smithi064.stdout:2547: finishing write tid 1 to smithi06431743-49 2024-03-04T06:18:20.304 INFO:tasks.rados.rados.0.smithi064.stdout:2550: expect (ObjNum 1144 snap 307 seq_num 1144) 2024-03-04T06:18:20.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:20.453 INFO:tasks.rados.rados.0.smithi064.stdout:2545: expect (ObjNum 1061 snap 272 seq_num 1061) 2024-03-04T06:18:20.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:20.500+0000 7efe2b714640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:20.504 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:20.504+0000 7efe2b714640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:20.680 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:18:20.680 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:18:20.680 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:18:20.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:18:20.688 INFO:tasks.rados.rados.0.smithi064.stdout:2544: finishing write tid 4 to smithi06431743-29 2024-03-04T06:18:20.688 INFO:tasks.rados.rados.0.smithi064.stdout:2547: finishing write tid 2 to smithi06431743-49 2024-03-04T06:18:20.688 INFO:tasks.rados.rados.0.smithi064.stdout:2548: expect (ObjNum 1050 snap 268 seq_num 1050) 2024-03-04T06:18:20.789 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:20.784+0000 7f462ab9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:20.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:20.791+0000 7f462ab9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:20.809 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:20.811 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:20.811 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:20.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:18:20.894 INFO:tasks.rados.rados.0.smithi064.stdout:2544: finishing write tid 5 to smithi06431743-29 2024-03-04T06:18:20.894 INFO:tasks.rados.rados.0.smithi064.stdout:2544: finishing write tid 6 to smithi06431743-29 2024-03-04T06:18:20.894 INFO:tasks.rados.rados.0.smithi064.stdout:2544: oid 29 updating version 0 to 1140 2024-03-04T06:18:20.894 INFO:tasks.rados.rados.0.smithi064.stdout:2544: oid 29 version 1140 is already newer than 1138 2024-03-04T06:18:20.894 INFO:tasks.rados.rados.0.smithi064.stdout:2544: oid 29 updating version 1140 to 1142 2024-03-04T06:18:20.894 INFO:tasks.rados.rados.0.smithi064.stdout:2544: oid 29 version 1142 is already newer than 1140 2024-03-04T06:18:20.894 INFO:tasks.rados.rados.0.smithi064.stdout:2544: oid 29 version 1142 is already newer than 1141 2024-03-04T06:18:20.894 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 1142 (ObjNum 1168 snap 311 seq_num 1168) dirty exists 2024-03-04T06:18:20.894 INFO:tasks.rados.rados.0.smithi064.stdout:2544: left oid 29 (ObjNum 1168 snap 311 seq_num 1168) 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2547: finishing write tid 3 to smithi06431743-49 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2547: finishing write tid 4 to smithi06431743-49 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2547: finishing write tid 5 to smithi06431743-49 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2547: oid 49 updating version 0 to 346 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2547: oid 49 updating version 346 to 347 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2547: oid 49 updating version 347 to 348 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2547: oid 49 updating version 348 to 349 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 349 (ObjNum 1169 snap 311 seq_num 1169) dirty exists 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2547: left oid 49 (ObjNum 1169 snap 311 seq_num 1169) 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2549: finishing write tid 1 to smithi06431743-4 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2549: finishing write tid 2 to smithi06431743-4 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2549: finishing write tid 3 to smithi06431743-4 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2549: finishing write tid 4 to smithi06431743-4 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2549: finishing write tid 5 to smithi06431743-4 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2549: oid 4 updating version 0 to 1144 2024-03-04T06:18:20.895 INFO:tasks.rados.rados.0.smithi064.stdout:2549: oid 4 updating version 1144 to 1147 2024-03-04T06:18:20.896 INFO:tasks.rados.rados.0.smithi064.stdout:2549: oid 4 version 1147 is already newer than 1147 2024-03-04T06:18:20.896 INFO:tasks.rados.rados.0.smithi064.stdout:2549: oid 4 version 1147 is already newer than 1145 2024-03-04T06:18:20.896 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 1147 (ObjNum 1170 snap 311 seq_num 1170) dirty exists 2024-03-04T06:18:20.896 INFO:tasks.rados.rados.0.smithi064.stdout:2549: left oid 4 (ObjNum 1170 snap 311 seq_num 1170) 2024-03-04T06:18:20.896 INFO:tasks.rados.rados.0.smithi064.stdout:2553: expect (ObjNum 1120 snap 298 seq_num 1120) 2024-03-04T06:18:20.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:20.955+0000 7f6b6191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:20.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:20.963+0000 7f6b6191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:20.977 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:20.980 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:20.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:18:21.033 INFO:tasks.rados.rados.0.smithi064.stdout:2555: expect (ObjNum 1032 snap 264 seq_num 1032) 2024-03-04T06:18:21.128 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:21.123+0000 7f3c3eb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:21.131 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:21.131+0000 7f3c3eb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:21.144 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:21.144 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:21.144 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:21.144 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:21.144 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:21.144 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:21.144 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:21.145 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:21.145 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:21.145 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:21.145 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:21.145 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:21.147 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:21.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:18:21.159 INFO:tasks.rados.rados.0.smithi064.stdout:2552: finishing write tid 1 to smithi06431743-35 2024-03-04T06:18:21.160 INFO:tasks.rados.rados.0.smithi064.stdout:2552: finishing write tid 2 to smithi06431743-35 2024-03-04T06:18:21.160 INFO:tasks.rados.rados.0.smithi064.stdout:2552: finishing write tid 3 to smithi06431743-35 2024-03-04T06:18:21.160 INFO:tasks.rados.rados.0.smithi064.stdout:2552: finishing write tid 4 to smithi06431743-35 2024-03-04T06:18:21.160 INFO:tasks.rados.rados.0.smithi064.stdout:2552: finishing write tid 5 to smithi06431743-35 2024-03-04T06:18:21.160 INFO:tasks.rados.rados.0.smithi064.stdout:2552: oid 35 updating version 0 to 747 2024-03-04T06:18:21.160 INFO:tasks.rados.rados.0.smithi064.stdout:2552: oid 35 updating version 747 to 750 2024-03-04T06:18:21.160 INFO:tasks.rados.rados.0.smithi064.stdout:2552: oid 35 version 750 is already newer than 749 2024-03-04T06:18:21.160 INFO:tasks.rados.rados.0.smithi064.stdout:2552: oid 35 version 750 is already newer than 749 2024-03-04T06:18:21.160 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 750 (ObjNum 1171 snap 311 seq_num 1171) dirty exists 2024-03-04T06:18:21.160 INFO:tasks.rados.rados.0.smithi064.stdout:2552: left oid 35 (ObjNum 1171 snap 311 seq_num 1171) 2024-03-04T06:18:21.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:21.263+0000 7f85a826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:21.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:21.270+0000 7f85a826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:21.283 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:21.284 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:21.284 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:21.284 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:21.284 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:21.284 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:21.284 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:21.284 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:21.284 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:21.284 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:21.284 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:21.284 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:21.285 INFO:tasks.rados.rados.0.smithi064.stdout:2544: done (11 left) 2024-03-04T06:18:21.285 INFO:tasks.rados.rados.0.smithi064.stdout:2545: done (10 left) 2024-03-04T06:18:21.285 INFO:tasks.rados.rados.0.smithi064.stdout:2546: done (9 left) 2024-03-04T06:18:21.285 INFO:tasks.rados.rados.0.smithi064.stdout:2547: done (8 left) 2024-03-04T06:18:21.285 INFO:tasks.rados.rados.0.smithi064.stdout:2548: done (7 left) 2024-03-04T06:18:21.285 INFO:tasks.rados.rados.0.smithi064.stdout:2549: done (6 left) 2024-03-04T06:18:21.285 INFO:tasks.rados.rados.0.smithi064.stdout:2550: done (5 left) 2024-03-04T06:18:21.286 INFO:tasks.rados.rados.0.smithi064.stdout:2552: done (4 left) 2024-03-04T06:18:21.286 INFO:tasks.rados.rados.0.smithi064.stdout:2553: done (3 left) 2024-03-04T06:18:21.286 INFO:tasks.rados.rados.0.smithi064.stdout:2554: done (2 left) 2024-03-04T06:18:21.286 INFO:tasks.rados.rados.0.smithi064.stdout:2555: done (1 left) 2024-03-04T06:18:21.286 INFO:tasks.rados.rados.0.smithi064.stdout:2556: done (0 left) 2024-03-04T06:18:21.286 INFO:tasks.rados.rados.0.smithi064.stdout:2557: snap_create 2024-03-04T06:18:21.286 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:21.287 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:22.288 INFO:tasks.rados.rados.0.smithi064.stdout:2557: done (0 left) 2024-03-04T06:18:22.288 INFO:tasks.rados.rados.0.smithi064.stdout:2558: write (excl) oid 8 current snap is 313 2024-03-04T06:18:22.288 INFO:tasks.rados.rados.0.smithi064.stdout:2558: seq_num 1172 ranges {497804=510190,1465955=448737,2584924=597152} 2024-03-04T06:18:22.292 INFO:tasks.rados.rados.0.smithi064.stdout:2558: writing smithi06431743-8 from 497804 to 1007994 tid 1 2024-03-04T06:18:22.298 INFO:tasks.rados.rados.0.smithi064.stdout:2558: writing smithi06431743-8 from 1465955 to 1914692 tid 2 2024-03-04T06:18:22.305 INFO:tasks.rados.rados.0.smithi064.stdout:2558: writing smithi06431743-8 from 2584924 to 3182076 tid 3 2024-03-04T06:18:22.305 INFO:tasks.rados.rados.0.smithi064.stdout:2559: write oid 13 current snap is 313 2024-03-04T06:18:22.305 INFO:tasks.rados.rados.0.smithi064.stdout:2559: seq_num 1173 ranges {514611=585439,1524437=592395,2895994=408424} 2024-03-04T06:18:22.310 INFO:tasks.rados.rados.0.smithi064.stdout:2559: writing smithi06431743-13 from 514611 to 1100050 tid 1 2024-03-04T06:18:22.316 INFO:tasks.rados.rados.0.smithi064.stdout:2559: writing smithi06431743-13 from 1524437 to 2116832 tid 2 2024-03-04T06:18:22.323 INFO:tasks.rados.rados.0.smithi064.stdout:2559: writing smithi06431743-13 from 2895994 to 3304418 tid 3 2024-03-04T06:18:22.323 INFO:tasks.rados.rados.0.smithi064.stdout:2560: snap_create 2024-03-04T06:18:22.387 INFO:tasks.rados.rados.0.smithi064.stdout:2559: finishing write tid 1 to smithi06431743-13 2024-03-04T06:18:22.388 INFO:tasks.rados.rados.0.smithi064.stdout:2558: finishing write tid 1 to smithi06431743-8 2024-03-04T06:18:22.403 INFO:tasks.rados.rados.0.smithi064.stdout:2559: finishing write tid 2 to smithi06431743-13 2024-03-04T06:18:22.412 INFO:tasks.rados.rados.0.smithi064.stdout:2558: finishing write tid 2 to smithi06431743-8 2024-03-04T06:18:22.412 INFO:tasks.rados.rados.0.smithi064.stdout:2558: finishing write tid 3 to smithi06431743-8 2024-03-04T06:18:22.420 INFO:tasks.rados.rados.0.smithi064.stdout:2559: finishing write tid 3 to smithi06431743-13 2024-03-04T06:18:22.466 INFO:tasks.rados.rados.0.smithi064.stdout:2559: finishing write tid 4 to smithi06431743-13 2024-03-04T06:18:22.466 INFO:tasks.rados.rados.0.smithi064.stdout:2558: finishing write tid 4 to smithi06431743-8 2024-03-04T06:18:22.467 INFO:tasks.rados.rados.0.smithi064.stdout:2559: finishing write tid 5 to smithi06431743-13 2024-03-04T06:18:22.468 INFO:tasks.rados.rados.0.smithi064.stdout:2559: oid 13 updating version 0 to 1164 2024-03-04T06:18:22.468 INFO:tasks.rados.rados.0.smithi064.stdout:2559: oid 13 version 1164 is already newer than 1163 2024-03-04T06:18:22.468 INFO:tasks.rados.rados.0.smithi064.stdout:2559: oid 13 version 1164 is already newer than 1161 2024-03-04T06:18:22.468 INFO:tasks.rados.rados.0.smithi064.stdout:2559: oid 13 version 1164 is already newer than 1162 2024-03-04T06:18:22.468 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1164 (ObjNum 1173 snap 313 seq_num 1173) dirty exists 2024-03-04T06:18:22.468 INFO:tasks.rados.rados.0.smithi064.stdout:2559: left oid 13 (ObjNum 1173 snap 313 seq_num 1173) 2024-03-04T06:18:22.468 INFO:tasks.rados.rados.0.smithi064.stdout:2558: finishing write tid 5 to smithi06431743-8 2024-03-04T06:18:22.468 INFO:tasks.rados.rados.0.smithi064.stdout:2558: oid 8 updating version 0 to 530 2024-03-04T06:18:22.469 INFO:tasks.rados.rados.0.smithi064.stdout:2558: oid 8 version 530 is already newer than 530 2024-03-04T06:18:22.469 INFO:tasks.rados.rados.0.smithi064.stdout:2558: oid 8 updating version 530 to 531 2024-03-04T06:18:22.469 INFO:tasks.rados.rados.0.smithi064.stdout:2558: oid 8 version 531 is already newer than 528 2024-03-04T06:18:22.469 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 531 (ObjNum 1172 snap 313 seq_num 1172) dirty exists 2024-03-04T06:18:22.469 INFO:tasks.rados.rados.0.smithi064.stdout:2558: left oid 8 (ObjNum 1172 snap 313 seq_num 1172) 2024-03-04T06:18:22.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:22.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:22.682+0000 7f73a266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:22.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:22.686+0000 7f73a266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:23.292 INFO:tasks.rados.rados.0.smithi064.stdout:2558: done (2 left) 2024-03-04T06:18:23.292 INFO:tasks.rados.rados.0.smithi064.stdout:2559: done (1 left) 2024-03-04T06:18:23.292 INFO:tasks.rados.rados.0.smithi064.stdout:2560: done (0 left) 2024-03-04T06:18:23.292 INFO:tasks.rados.rados.0.smithi064.stdout:2561: write oid 16 current snap is 314 2024-03-04T06:18:23.293 INFO:tasks.rados.rados.0.smithi064.stdout:2561: seq_num 1174 ranges {531418=660688,1982919=452406,3039770=691648} 2024-03-04T06:18:23.298 INFO:tasks.rados.rados.0.smithi064.stdout:2561: writing smithi06431743-16 from 531418 to 1192106 tid 1 2024-03-04T06:18:23.305 INFO:tasks.rados.rados.0.smithi064.stdout:2561: writing smithi06431743-16 from 1982919 to 2435325 tid 2 2024-03-04T06:18:23.313 INFO:tasks.rados.rados.0.smithi064.stdout:2561: writing smithi06431743-16 from 3039770 to 3731418 tid 3 2024-03-04T06:18:23.313 INFO:tasks.rados.rados.0.smithi064.stdout:2562: delete oid 21 current snap is 314 2024-03-04T06:18:23.370 INFO:tasks.rados.rados.0.smithi064.stdout:2562: done (1 left) 2024-03-04T06:18:23.370 INFO:tasks.rados.rados.0.smithi064.stdout:2563: snap_create 2024-03-04T06:18:23.371 INFO:tasks.rados.rados.0.smithi064.stdout:2561: finishing write tid 1 to smithi06431743-16 2024-03-04T06:18:23.382 INFO:tasks.rados.rados.0.smithi064.stdout:2561: finishing write tid 2 to smithi06431743-16 2024-03-04T06:18:23.395 INFO:tasks.rados.rados.0.smithi064.stdout:2561: finishing write tid 3 to smithi06431743-16 2024-03-04T06:18:23.398 INFO:tasks.rados.rados.0.smithi064.stdout:2561: finishing write tid 4 to smithi06431743-16 2024-03-04T06:18:23.399 INFO:tasks.rados.rados.0.smithi064.stdout:2561: finishing write tid 5 to smithi06431743-16 2024-03-04T06:18:23.399 INFO:tasks.rados.rados.0.smithi064.stdout:2561: oid 16 updating version 0 to 536 2024-03-04T06:18:23.399 INFO:tasks.rados.rados.0.smithi064.stdout:2561: oid 16 version 536 is already newer than 536 2024-03-04T06:18:23.400 INFO:tasks.rados.rados.0.smithi064.stdout:2561: oid 16 version 536 is already newer than 533 2024-03-04T06:18:23.400 INFO:tasks.rados.rados.0.smithi064.stdout:2561: oid 16 version 536 is already newer than 534 2024-03-04T06:18:23.400 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 536 (ObjNum 1174 snap 314 seq_num 1174) dirty exists 2024-03-04T06:18:23.400 INFO:tasks.rados.rados.0.smithi064.stdout:2561: left oid 16 (ObjNum 1174 snap 314 seq_num 1174) 2024-03-04T06:18:24.294 INFO:tasks.rados.rados.0.smithi064.stdout:2561: done (1 left) 2024-03-04T06:18:24.295 INFO:tasks.rados.rados.0.smithi064.stdout:2563: done (0 left) 2024-03-04T06:18:24.295 INFO:tasks.rados.rados.0.smithi064.stdout:2564: read oid 44 snap 295 2024-03-04T06:18:24.295 INFO:tasks.rados.rados.0.smithi064.stdout:2564: expect (ObjNum 993 snap 254 seq_num 993) 2024-03-04T06:18:24.295 INFO:tasks.rados.rados.0.smithi064.stdout:2565: read oid 8 snap -1 2024-03-04T06:18:24.295 INFO:tasks.rados.rados.0.smithi064.stdout:2565: expect (ObjNum 1172 snap 313 seq_num 1172) 2024-03-04T06:18:24.295 INFO:tasks.rados.rados.0.smithi064.stdout:2566: delete oid 45 current snap is 315 2024-03-04T06:18:24.333 INFO:tasks.rados.rados.0.smithi064.stdout:2566: done (2 left) 2024-03-04T06:18:24.334 INFO:tasks.rados.rados.0.smithi064.stdout:2567: read oid 49 snap 300 2024-03-04T06:18:24.334 INFO:tasks.rados.rados.0.smithi064.stdout:2567: expect (ObjNum 1109 snap 290 seq_num 1109) 2024-03-04T06:18:24.334 INFO:tasks.rados.rados.0.smithi064.stdout:2568: read oid 31 snap 310 2024-03-04T06:18:24.334 INFO:tasks.rados.rados.0.smithi064.stdout:2568: expect (ObjNum 1152 snap 309 seq_num 1152) 2024-03-04T06:18:24.334 INFO:tasks.rados.rados.0.smithi064.stdout:2569: read oid 19 snap -1 2024-03-04T06:18:24.334 INFO:tasks.rados.rados.0.smithi064.stdout:2569: expect deleted 2024-03-04T06:18:24.335 INFO:tasks.rados.rados.0.smithi064.stdout:2570: write oid 20 current snap is 315 2024-03-04T06:18:24.335 INFO:tasks.rados.rados.0.smithi064.stdout:2570: seq_num 1177 ranges {581839=486435,1642012=599733,3005686=569865} 2024-03-04T06:18:24.340 INFO:tasks.rados.rados.0.smithi064.stdout:2570: writing smithi06431743-20 from 581839 to 1068274 tid 1 2024-03-04T06:18:24.350 INFO:tasks.rados.rados.0.smithi064.stdout:2570: writing smithi06431743-20 from 1642012 to 2241745 tid 2 2024-03-04T06:18:24.360 INFO:tasks.rados.rados.0.smithi064.stdout:2570: writing smithi06431743-20 from 3005686 to 3575551 tid 3 2024-03-04T06:18:24.360 INFO:tasks.rados.rados.0.smithi064.stdout:2571: snap_create 2024-03-04T06:18:24.361 INFO:tasks.rados.rados.0.smithi064.stdout:2565: expect (ObjNum 1172 snap 313 seq_num 1172) 2024-03-04T06:18:24.598 INFO:tasks.rados.rados.0.smithi064.stdout:2568: expect (ObjNum 1152 snap 309 seq_num 1152) 2024-03-04T06:18:24.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:24.798 INFO:tasks.rados.rados.0.smithi064.stdout:2567: expect (ObjNum 1109 snap 290 seq_num 1109) 2024-03-04T06:18:24.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:24.860+0000 7feaa835d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:24.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:24.864+0000 7feaa835d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:24.973 INFO:tasks.rados.rados.0.smithi064.stdout:2570: finishing write tid 1 to smithi06431743-20 2024-03-04T06:18:24.973 INFO:tasks.rados.rados.0.smithi064.stdout:2564: expect (ObjNum 993 snap 254 seq_num 993) 2024-03-04T06:18:25.136 INFO:tasks.rados.rados.0.smithi064.stdout:2570: finishing write tid 2 to smithi06431743-20 2024-03-04T06:18:25.136 INFO:tasks.rados.rados.0.smithi064.stdout:2570: finishing write tid 3 to smithi06431743-20 2024-03-04T06:18:25.136 INFO:tasks.rados.rados.0.smithi064.stdout:2570: finishing write tid 4 to smithi06431743-20 2024-03-04T06:18:25.136 INFO:tasks.rados.rados.0.smithi064.stdout:2570: finishing write tid 5 to smithi06431743-20 2024-03-04T06:18:25.136 INFO:tasks.rados.rados.0.smithi064.stdout:2570: oid 20 updating version 0 to 755 2024-03-04T06:18:25.136 INFO:tasks.rados.rados.0.smithi064.stdout:2570: oid 20 version 755 is already newer than 753 2024-03-04T06:18:25.136 INFO:tasks.rados.rados.0.smithi064.stdout:2570: oid 20 version 755 is already newer than 752 2024-03-04T06:18:25.136 INFO:tasks.rados.rados.0.smithi064.stdout:2570: oid 20 version 755 is already newer than 754 2024-03-04T06:18:25.136 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 755 (ObjNum 1177 snap 315 seq_num 1177) dirty exists 2024-03-04T06:18:25.136 INFO:tasks.rados.rados.0.smithi064.stdout:2570: left oid 20 (ObjNum 1177 snap 315 seq_num 1177) 2024-03-04T06:18:25.299 INFO:tasks.rados.rados.0.smithi064.stdout:2564: done (6 left) 2024-03-04T06:18:25.299 INFO:tasks.rados.rados.0.smithi064.stdout:2565: done (5 left) 2024-03-04T06:18:25.299 INFO:tasks.rados.rados.0.smithi064.stdout:2567: done (4 left) 2024-03-04T06:18:25.299 INFO:tasks.rados.rados.0.smithi064.stdout:2568: done (3 left) 2024-03-04T06:18:25.300 INFO:tasks.rados.rados.0.smithi064.stdout:2569: done (2 left) 2024-03-04T06:18:25.300 INFO:tasks.rados.rados.0.smithi064.stdout:2570: done (1 left) 2024-03-04T06:18:25.300 INFO:tasks.rados.rados.0.smithi064.stdout:2571: done (0 left) 2024-03-04T06:18:25.300 INFO:tasks.rados.rados.0.smithi064.stdout:2572: read oid 10 snap -1 2024-03-04T06:18:25.300 INFO:tasks.rados.rados.0.smithi064.stdout:2572: expect (ObjNum 1133 snap 303 seq_num 1133) 2024-03-04T06:18:25.300 INFO:tasks.rados.rados.0.smithi064.stdout:2573: write oid 44 current snap is 316 2024-03-04T06:18:25.300 INFO:tasks.rados.rados.0.smithi064.stdout:2573: seq_num 1178 ranges {598646=561684,1700494=459744,2633109=781137} 2024-03-04T06:18:25.304 INFO:tasks.rados.rados.0.smithi064.stdout:2573: writing smithi06431743-44 from 598646 to 1160330 tid 1 2024-03-04T06:18:25.310 INFO:tasks.rados.rados.0.smithi064.stdout:2573: writing smithi06431743-44 from 1700494 to 2160238 tid 2 2024-03-04T06:18:25.318 INFO:tasks.rados.rados.0.smithi064.stdout:2573: writing smithi06431743-44 from 2633109 to 3414246 tid 3 2024-03-04T06:18:25.318 INFO:tasks.rados.rados.0.smithi064.stdout:2574: snap_remove snap 295 2024-03-04T06:18:26.288 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:18:26.288 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:18:26.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:18:26.300 INFO:tasks.rados.rados.0.smithi064.stdout:2574: done (2 left) 2024-03-04T06:18:26.300 INFO:tasks.rados.rados.0.smithi064.stdout:2575: write oid 3 current snap is 316 2024-03-04T06:18:26.300 INFO:tasks.rados.rados.0.smithi064.stdout:2575: seq_num 1179 ranges {615453=753286,2158976=603402,3460532=354921} 2024-03-04T06:18:26.308 INFO:tasks.rados.rados.0.smithi064.stdout:2575: writing smithi06431743-3 from 615453 to 1368739 tid 1 2024-03-04T06:18:26.316 INFO:tasks.rados.rados.0.smithi064.stdout:2575: writing smithi06431743-3 from 2158976 to 2762378 tid 2 2024-03-04T06:18:26.324 INFO:tasks.rados.rados.0.smithi064.stdout:2575: writing smithi06431743-3 from 3460532 to 3815453 tid 3 2024-03-04T06:18:26.324 INFO:tasks.rados.rados.0.smithi064.stdout:2576: write oid 15 current snap is 316 2024-03-04T06:18:26.324 INFO:tasks.rados.rados.0.smithi064.stdout:2576: seq_num 1180 ranges {632260=428535,1817458=463413,2687955=520034,3709324=122936} 2024-03-04T06:18:26.330 INFO:tasks.rados.rados.0.smithi064.stdout:2576: writing smithi06431743-15 from 632260 to 1060795 tid 1 2024-03-04T06:18:26.337 INFO:tasks.rados.rados.0.smithi064.stdout:2576: writing smithi06431743-15 from 1817458 to 2280871 tid 2 2024-03-04T06:18:26.343 INFO:tasks.rados.rados.0.smithi064.stdout:2576: writing smithi06431743-15 from 2687955 to 3207989 tid 3 2024-03-04T06:18:26.348 INFO:tasks.rados.rados.0.smithi064.stdout:2576: writing smithi06431743-15 from 3709324 to 3832260 tid 4 2024-03-04T06:18:26.348 INFO:tasks.rados.rados.0.smithi064.stdout:2577: snap_create 2024-03-04T06:18:26.348 INFO:tasks.rados.rados.0.smithi064.stdout:2572: expect (ObjNum 1133 snap 303 seq_num 1133) 2024-03-04T06:18:26.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:26.442+0000 7f6329b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:26.458 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:26.457+0000 7f6329b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:26.545 INFO:tasks.rados.rados.0.smithi064.stdout:2573: finishing write tid 1 to smithi06431743-44 2024-03-04T06:18:26.545 INFO:tasks.rados.rados.0.smithi064.stdout:2573: finishing write tid 2 to smithi06431743-44 2024-03-04T06:18:26.545 INFO:tasks.rados.rados.0.smithi064.stdout:2573: finishing write tid 3 to smithi06431743-44 2024-03-04T06:18:26.545 INFO:tasks.rados.rados.0.smithi064.stdout:2573: finishing write tid 4 to smithi06431743-44 2024-03-04T06:18:26.546 INFO:tasks.rados.rados.0.smithi064.stdout:2573: finishing write tid 5 to smithi06431743-44 2024-03-04T06:18:26.546 INFO:tasks.rados.rados.0.smithi064.stdout:2573: oid 44 updating version 0 to 631 2024-03-04T06:18:26.546 INFO:tasks.rados.rados.0.smithi064.stdout:2573: oid 44 version 631 is already newer than 629 2024-03-04T06:18:26.546 INFO:tasks.rados.rados.0.smithi064.stdout:2573: oid 44 version 631 is already newer than 630 2024-03-04T06:18:26.546 INFO:tasks.rados.rados.0.smithi064.stdout:2573: oid 44 version 631 is already newer than 628 2024-03-04T06:18:26.546 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 631 (ObjNum 1178 snap 316 seq_num 1178) dirty exists 2024-03-04T06:18:26.546 INFO:tasks.rados.rados.0.smithi064.stdout:2573: left oid 44 (ObjNum 1178 snap 316 seq_num 1178) 2024-03-04T06:18:26.546 INFO:tasks.rados.rados.0.smithi064.stdout:2576: finishing write tid 1 to smithi06431743-15 2024-03-04T06:18:26.546 INFO:tasks.rados.rados.0.smithi064.stdout:2575: finishing write tid 1 to smithi06431743-3 2024-03-04T06:18:26.546 INFO:tasks.rados.rados.0.smithi064.stdout:2576: finishing write tid 2 to smithi06431743-15 2024-03-04T06:18:26.546 INFO:tasks.rados.rados.0.smithi064.stdout:2575: finishing write tid 2 to smithi06431743-3 2024-03-04T06:18:26.547 INFO:tasks.rados.rados.0.smithi064.stdout:2576: finishing write tid 3 to smithi06431743-15 2024-03-04T06:18:26.547 INFO:tasks.rados.rados.0.smithi064.stdout:2576: finishing write tid 4 to smithi06431743-15 2024-03-04T06:18:26.547 INFO:tasks.rados.rados.0.smithi064.stdout:2575: finishing write tid 3 to smithi06431743-3 2024-03-04T06:18:26.547 INFO:tasks.rados.rados.0.smithi064.stdout:2576: finishing write tid 5 to smithi06431743-15 2024-03-04T06:18:26.547 INFO:tasks.rados.rados.0.smithi064.stdout:2576: finishing write tid 6 to smithi06431743-15 2024-03-04T06:18:26.547 INFO:tasks.rados.rados.0.smithi064.stdout:2576: oid 15 updating version 0 to 564 2024-03-04T06:18:26.547 INFO:tasks.rados.rados.0.smithi064.stdout:2576: oid 15 version 564 is already newer than 561 2024-03-04T06:18:26.547 INFO:tasks.rados.rados.0.smithi064.stdout:2576: oid 15 version 564 is already newer than 562 2024-03-04T06:18:26.547 INFO:tasks.rados.rados.0.smithi064.stdout:2576: oid 15 version 564 is already newer than 563 2024-03-04T06:18:26.548 INFO:tasks.rados.rados.0.smithi064.stdout:2576: oid 15 version 564 is already newer than 560 2024-03-04T06:18:26.548 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 564 (ObjNum 1180 snap 316 seq_num 1180) dirty exists 2024-03-04T06:18:26.548 INFO:tasks.rados.rados.0.smithi064.stdout:2576: left oid 15 (ObjNum 1180 snap 316 seq_num 1180) 2024-03-04T06:18:26.548 INFO:tasks.rados.rados.0.smithi064.stdout:2575: finishing write tid 4 to smithi06431743-3 2024-03-04T06:18:26.548 INFO:tasks.rados.rados.0.smithi064.stdout:2575: finishing write tid 5 to smithi06431743-3 2024-03-04T06:18:26.548 INFO:tasks.rados.rados.0.smithi064.stdout:2575: oid 3 updating version 0 to 613 2024-03-04T06:18:26.548 INFO:tasks.rados.rados.0.smithi064.stdout:2575: oid 3 updating version 613 to 616 2024-03-04T06:18:26.548 INFO:tasks.rados.rados.0.smithi064.stdout:2575: oid 3 version 616 is already newer than 614 2024-03-04T06:18:26.549 INFO:tasks.rados.rados.0.smithi064.stdout:2575: oid 3 version 616 is already newer than 615 2024-03-04T06:18:26.549 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 616 (ObjNum 1179 snap 316 seq_num 1179) dirty exists 2024-03-04T06:18:26.549 INFO:tasks.rados.rados.0.smithi064.stdout:2575: left oid 3 (ObjNum 1179 snap 316 seq_num 1179) 2024-03-04T06:18:26.675 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:18:26.676 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:18:26.687 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":946,"stamp":"2024-03-04T06:18:26.298166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129846659,"num_objects":873,"num_object_clones":683,"num_object_copies":2619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4820,"num_read_kb":3714472,"num_write":4079,"num_write_kb":1076720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9247,"ondisk_log_size":9247,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":605396,"kb_used_data":417332,"kb_used_omap":0,"kb_used_meta":187776,"kb_avail":376881964,"statfs":{"total":386547056640,"available":385927131136,"internally_reserved":0,"allocated":427347968,"data_stored":506441368,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":192282624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1663893,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":21905,"num_write":17,"num_write_kb":5834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012967"},"pg_stats":[{"pgid":"3.f","version":"627'548","reported_seq":2184,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-04T06:18:25.329933+0000","last_change":"2024-03-04T06:18:20.409885+0000","last_active":"2024-03-04T06:18:25.329933+0000","last_peered":"2024-03-04T06:18:25.329933+0000","last_clean":"2024-03-04T06:18:25.329933+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:18:25.329933+0000","last_undegraded":"2024-03-04T06:18:25.329933+0000","last_fullsized":"2024-03-04T06:18:25.329933+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3462506,"num_objects":56,"num_object_clones":44,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":272,"num_read_kb":224366,"num_write":228,"num_write_kb":60755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"629'445","reported_seq":2137,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-04T06:18:24.305975+0000","last_change":"2024-03-04T06:18:20.376627+0000","last_active":"2024-03-04T06:18:24.305975+0000","last_peered":"2024-03-04T06:18:24.305975+0000","last_clean":"2024-03-04T06:18:24.305975+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:18:24.305975+0000","last_undegraded":"2024-03-04T06:18:24.305975+0000","last_fullsized":"2024-03-04T06:18:24.305975+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7042628,"num_objects":46,"num_object_clones":32,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":164,"num_read_kb":111777,"num_write":186,"num_write_kb":49317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"623'587","reported_seq":2277,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-03-04T06:18:20.364554+0000","last_change":"2024-03-04T06:18:20.327552+0000","last_active":"2024-03-04T06:18:20.364554+0000","last_peered":"2024-03-04T06:18:20.364554+0000","last_clean":"2024-03-04T06:18:20.364554+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:18:20.364554+0000","last_undegraded":"2024-03-04T06:18:20.364554+0000","last_fullsized":"2024-03-04T06:18:20.364554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":587,"log_dups_size":0,"ondisk_log_size":587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9513554,"num_objects":63,"num_object_clones":49,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":369,"num_read_kb":298747,"num_write":273,"num_write_kb":75354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"629'466","reported_seq":2142,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-04T06:18:25.326744+0000","last_change":"2024-03-04T06:18:20.441729+0000","last_active":"2024-03-04T06:18:25.326744+0000","last_peered":"2024-03-04T06:18:25.326744+0000","last_clean":"2024-03-04T06:18:25.326744+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:18:25.326744+0000","last_undegraded":"2024-03-04T06:18:25.326744+0000","last_fullsized":"2024-03-04T06:18:25.326744+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9580782,"num_objects":41,"num_object_clones":31,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":224,"num_read_kb":169192,"num_write":202,"num_write_kb":56330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"629'219","reported_seq":1975,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-04T06:18:21.304950+0000","last_change":"2024-03-04T06:18:20.378587+0000","last_active":"2024-03-04T06:18:21.304950+0000","last_peered":"2024-03-04T06:18:21.304950+0000","last_clean":"2024-03-04T06:18:21.304950+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:18:21.304950+0000","last_undegraded":"2024-03-04T06:18:21.304950+0000","last_fullsized":"2024-03-04T06:18:21.304950+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361664,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":76,"num_read_kb":47658,"num_write":93,"num_write_kb":25364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"634'628","reported_seq":2245,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-04T06:18:25.351381+0000","last_change":"2024-03-04T06:18:20.440045+0000","last_active":"2024-03-04T06:18:25.351381+0000","last_peered":"2024-03-04T06:18:25.351381+0000","last_clean":"2024-03-04T06:18:25.351381+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:18:25.351381+0000","last_undegraded":"2024-03-04T06:18:25.351381+0000","last_fullsized":"2024-03-04T06:18:25.351381+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9648010,"num_objects":55,"num_object_clones":46,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":314,"num_read_kb":245868,"num_write":275,"num_write_kb":80437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1858,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-04T06:18:25.336812+0000","last_change":"2024-03-04T06:18:20.410426+0000","last_active":"2024-03-04T06:18:25.336812+0000","last_peered":"2024-03-04T06:18:25.336812+0000","last_clean":"2024-03-04T06:18:25.336812+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:18:25.336812+0000","last_undegraded":"2024-03-04T06:18:25.336812+0000","last_fullsized":"2024-03-04T06:18:25.336812+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"629'351","reported_seq":2076,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-03-04T06:18:20.319192+0000","last_change":"2024-03-04T06:18:20.360308+0000","last_active":"2024-03-04T06:18:20.319192+0000","last_peered":"2024-03-04T06:18:20.319192+0000","last_clean":"2024-03-04T06:18:20.319192+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:18:20.319192+0000","last_undegraded":"2024-03-04T06:18:20.319192+0000","last_fullsized":"2024-03-04T06:18:20.319192+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3647383,"num_objects":31,"num_object_clones":22,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":188,"num_read_kb":169870,"num_write":137,"num_write_kb":36545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1703,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-04T06:18:21.298318+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:18:21.298318+0000","last_peered":"2024-03-04T06:18:21.298318+0000","last_clean":"2024-03-04T06:18:21.298318+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:18:21.298318+0000","last_undegraded":"2024-03-04T06:18:21.298318+0000","last_fullsized":"2024-03-04T06:18:21.298318+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1579,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-03-04T06:18:20.325050+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:18:20.325050+0000","last_peered":"2024-03-04T06:18:20.325050+0000","last_clean":"2024-03-04T06:18:20.325050+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:18:20.325050+0000","last_undegraded":"2024-03-04T06:18:20.325050+0000","last_fullsized":"2024-03-04T06:18:20.325050+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"629'607","reported_seq":2238,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-03-04T06:18:20.400256+0000","last_change":"2024-03-04T06:18:20.404279+0000","last_active":"2024-03-04T06:18:20.400256+0000","last_peered":"2024-03-04T06:18:20.400256+0000","last_clean":"2024-03-04T06:18:20.400256+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:18:20.400256+0000","last_undegraded":"2024-03-04T06:18:20.400256+0000","last_fullsized":"2024-03-04T06:18:20.400256+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9984150,"num_objects":57,"num_object_clones":44,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":310,"num_read_kb":246933,"num_write":259,"num_write_kb":70069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1568,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-04T06:18:25.331088+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:18:25.331088+0000","last_peered":"2024-03-04T06:18:25.331088+0000","last_clean":"2024-03-04T06:18:25.331088+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:18:25.331088+0000","last_undegraded":"2024-03-04T06:18:25.331088+0000","last_fullsized":"2024-03-04T06:18:25.331088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"628'389","reported_seq":2095,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-04T06:18:21.300413+0000","last_change":"2024-03-04T06:18:20.318263+0000","last_active":"2024-03-04T06:18:21.300413+0000","last_peered":"2024-03-04T06:18:21.300413+0000","last_clean":"2024-03-04T06:18:21.300413+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:18:21.300413+0000","last_undegraded":"2024-03-04T06:18:21.300413+0000","last_fullsized":"2024-03-04T06:18:21.300413+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370051,"num_objects":38,"num_object_clones":30,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":220,"num_read_kb":177502,"num_write":152,"num_write_kb":42787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1579,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-03-04T06:18:20.317118+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:18:20.317118+0000","last_peered":"2024-03-04T06:18:20.317118+0000","last_clean":"2024-03-04T06:18:20.317118+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:18:20.317118+0000","last_undegraded":"2024-03-04T06:18:20.317118+0000","last_fullsized":"2024-03-04T06:18:20.317118+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"629'1046","reported_seq":2558,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-04T06:18:21.307843+0000","last_change":"2024-03-04T06:18:20.378875+0000","last_active":"2024-03-04T06:18:21.307843+0000","last_peered":"2024-03-04T06:18:21.307843+0000","last_clean":"2024-03-04T06:18:21.307843+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:18:21.307843+0000","last_undegraded":"2024-03-04T06:18:21.307843+0000","last_fullsized":"2024-03-04T06:18:21.307843+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14841901,"num_objects":101,"num_object_clones":78,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":599,"num_read_kb":474728,"num_write":478,"num_write_kb":127819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1568,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-03-04T06:18:25.326140+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:18:25.326140+0000","last_peered":"2024-03-04T06:18:25.326140+0000","last_clean":"2024-03-04T06:18:25.326140+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:18:25.326140+0000","last_undegraded":"2024-03-04T06:18:25.326140+0000","last_fullsized":"2024-03-04T06:18:25.326140+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"631'1164","reported_seq":2555,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-04T06:18:24.317417+0000","last_change":"2024-03-04T06:18:20.428279+0000","last_active":"2024-03-04T06:18:24.317417+0000","last_peered":"2024-03-04T06:18:24.317417+0000","last_clean":"2024-03-04T06:18:24.317417+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:18:24.317417+0000","last_undegraded":"2024-03-04T06:18:24.317417+0000","last_fullsized":"2024-03-04T06:18:24.317417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18169654,"num_objects":107,"num_object_clones":85,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":484,"num_read_kb":355670,"num_write":497,"num_write_kb":132212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"627'552","reported_seq":2251,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-03-04T06:18:20.327452+0000","last_change":"2024-03-04T06:18:20.337420+0000","last_active":"2024-03-04T06:18:20.327452+0000","last_peered":"2024-03-04T06:18:20.327452+0000","last_clean":"2024-03-04T06:18:20.327452+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:18:20.327452+0000","last_undegraded":"2024-03-04T06:18:20.327452+0000","last_fullsized":"2024-03-04T06:18:20.327452+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8320257,"num_objects":51,"num_object_clones":39,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":362,"num_read_kb":299847,"num_write":241,"num_write_kb":65778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1609,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-04T06:18:24.321474+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:18:24.321474+0000","last_peered":"2024-03-04T06:18:24.321474+0000","last_clean":"2024-03-04T06:18:24.321474+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:18:24.321474+0000","last_undegraded":"2024-03-04T06:18:24.321474+0000","last_fullsized":"2024-03-04T06:18:24.321474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"632'536","reported_seq":2242,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-04T06:18:24.343610+0000","last_change":"2024-03-04T06:18:20.360179+0000","last_active":"2024-03-04T06:18:24.343610+0000","last_peered":"2024-03-04T06:18:24.343610+0000","last_clean":"2024-03-04T06:18:24.343610+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:18:24.343610+0000","last_undegraded":"2024-03-04T06:18:24.343610+0000","last_fullsized":"2024-03-04T06:18:24.343610+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10555588,"num_objects":59,"num_object_clones":43,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":287,"num_read_kb":215957,"num_write":242,"num_write_kb":64736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1570,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-04T06:18:21.309607+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:18:21.309607+0000","last_peered":"2024-03-04T06:18:21.309607+0000","last_clean":"2024-03-04T06:18:21.309607+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:18:21.309607+0000","last_undegraded":"2024-03-04T06:18:21.309607+0000","last_fullsized":"2024-03-04T06:18:21.309607+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"629'750","reported_seq":2318,"reported_epoch":630,"state":"active+clean","last_fresh":"2024-03-04T06:18:21.297373+0000","last_change":"2024-03-04T06:18:20.467147+0000","last_active":"2024-03-04T06:18:21.297373+0000","last_peered":"2024-03-04T06:18:21.297373+0000","last_clean":"2024-03-04T06:18:21.297373+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:18:21.297373+0000","last_undegraded":"2024-03-04T06:18:21.297373+0000","last_fullsized":"2024-03-04T06:18:21.297373+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":750,"log_dups_size":0,"ondisk_log_size":750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12068482,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":365,"num_read_kb":267438,"num_write":334,"num_write_kb":92488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1609,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-04T06:18:24.311397+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:18:24.311397+0000","last_peered":"2024-03-04T06:18:24.311397+0000","last_clean":"2024-03-04T06:18:24.311397+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:18:24.311397+0000","last_undegraded":"2024-03-04T06:18:24.311397+0000","last_fullsized":"2024-03-04T06:18:24.311397+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"629'862","reported_seq":2428,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-03-04T06:18:24.306696+0000","last_change":"2024-03-04T06:18:20.346713+0000","last_active":"2024-03-04T06:18:24.306696+0000","last_peered":"2024-03-04T06:18:24.306696+0000","last_clean":"2024-03-04T06:18:24.306696+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:18:24.306696+0000","last_undegraded":"2024-03-04T06:18:24.306696+0000","last_fullsized":"2024-03-04T06:18:24.306696+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":862,"log_dups_size":0,"ondisk_log_size":862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6689681,"num_objects":75,"num_object_clones":66,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":483,"num_read_kb":408828,"num_write":345,"num_write_kb":94835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1579,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-03-04T06:18:20.323682+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:18:20.323682+0000","last_peered":"2024-03-04T06:18:20.323682+0000","last_clean":"2024-03-04T06:18:20.323682+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:18:20.323682+0000","last_undegraded":"2024-03-04T06:18:20.323682+0000","last_fullsized":"2024-03-04T06:18:20.323682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":129256291,"num_objects":871,"num_object_clones":683,"num_object_copies":2613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":148,"num_read":4717,"num_read_kb":3714381,"num_write":3942,"num_write_kb":1074826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9150,"ondisk_log_size":9150,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706299,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":179000,"kb_used_data":125728,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94192840,"statfs":{"total":96636764160,"available":96453468160,"internally_reserved":0,"allocated":128745472,"data_stored":152110507,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706299,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":132968,"kb_used_data":88016,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94238872,"statfs":{"total":96636764160,"available":96500604928,"internally_reserved":0,"allocated":90128384,"data_stored":102040217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154428,"kb_used_data":109556,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94217412,"statfs":{"total":96636764160,"available":96478629888,"internally_reserved":0,"allocated":112185344,"data_stored":137684953,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":139000,"kb_used_data":94032,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94232840,"statfs":{"total":96636764160,"available":96494428160,"internally_reserved":0,"allocated":96288768,"data_stored":114605691,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:18:26.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:27.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:27.038+0000 7f5c9bb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:27.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:27.043+0000 7f5c9bb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:27.305 INFO:tasks.rados.rados.0.smithi064.stdout:2572: done (4 left) 2024-03-04T06:18:27.305 INFO:tasks.rados.rados.0.smithi064.stdout:2573: done (3 left) 2024-03-04T06:18:27.305 INFO:tasks.rados.rados.0.smithi064.stdout:2575: done (2 left) 2024-03-04T06:18:27.305 INFO:tasks.rados.rados.0.smithi064.stdout:2576: done (1 left) 2024-03-04T06:18:27.305 INFO:tasks.rados.rados.0.smithi064.stdout:2577: done (0 left) 2024-03-04T06:18:27.306 INFO:tasks.rados.rados.0.smithi064.stdout:2578: write oid 5 current snap is 317 2024-03-04T06:18:27.306 INFO:tasks.rados.rados.0.smithi064.stdout:2578: seq_num 1181 ranges {649067=503784,1875940=607071,3115378=731306} 2024-03-04T06:18:27.311 INFO:tasks.rados.rados.0.smithi064.stdout:2578: writing smithi06431743-5 from 649067 to 1152851 tid 1 2024-03-04T06:18:27.318 INFO:tasks.rados.rados.0.smithi064.stdout:2578: writing smithi06431743-5 from 1875940 to 2483011 tid 2 2024-03-04T06:18:27.326 INFO:tasks.rados.rados.0.smithi064.stdout:2578: writing smithi06431743-5 from 3115378 to 3846684 tid 3 2024-03-04T06:18:27.326 INFO:tasks.rados.rados.0.smithi064.stdout:2579: read oid 1 snap -1 2024-03-04T06:18:27.326 INFO:tasks.rados.rados.0.smithi064.stdout:2579: expect deleted 2024-03-04T06:18:27.326 INFO:tasks.rados.rados.0.smithi064.stdout:2580: write oid 42 current snap is 317 2024-03-04T06:18:27.327 INFO:tasks.rados.rados.0.smithi064.stdout:2580: seq_num 1182 ranges {665874=579033,1934422=467082,3142801=542578} 2024-03-04T06:18:27.333 INFO:tasks.rados.rados.0.smithi064.stdout:2580: writing smithi06431743-42 from 665874 to 1244907 tid 1 2024-03-04T06:18:27.340 INFO:tasks.rados.rados.0.smithi064.stdout:2580: writing smithi06431743-42 from 1934422 to 2401504 tid 2 2024-03-04T06:18:27.348 INFO:tasks.rados.rados.0.smithi064.stdout:2580: writing smithi06431743-42 from 3142801 to 3685379 tid 3 2024-03-04T06:18:27.348 INFO:tasks.rados.rados.0.smithi064.stdout:2581: delete oid 18 current snap is 317 2024-03-04T06:18:27.368 INFO:tasks.rados.rados.0.smithi064.stdout:2580: finishing write tid 1 to smithi06431743-42 2024-03-04T06:18:27.383 INFO:tasks.rados.rados.0.smithi064.stdout:2578: finishing write tid 1 to smithi06431743-5 2024-03-04T06:18:27.401 INFO:tasks.rados.rados.0.smithi064.stdout:2579: done (3 left) 2024-03-04T06:18:27.401 INFO:tasks.rados.rados.0.smithi064.stdout:2581: done (2 left) 2024-03-04T06:18:27.401 INFO:tasks.rados.rados.0.smithi064.stdout:2582: snap_remove snap 311 2024-03-04T06:18:28.315 INFO:tasks.rados.rados.0.smithi064.stdout:2582: done (2 left) 2024-03-04T06:18:28.315 INFO:tasks.rados.rados.0.smithi064.stdout:2583: snap_create 2024-03-04T06:18:28.315 INFO:tasks.rados.rados.0.smithi064.stdout:2580: finishing write tid 2 to smithi06431743-42 2024-03-04T06:18:28.315 INFO:tasks.rados.rados.0.smithi064.stdout:2578: finishing write tid 2 to smithi06431743-5 2024-03-04T06:18:28.316 INFO:tasks.rados.rados.0.smithi064.stdout:2580: finishing write tid 3 to smithi06431743-42 2024-03-04T06:18:28.316 INFO:tasks.rados.rados.0.smithi064.stdout:2578: finishing write tid 3 to smithi06431743-5 2024-03-04T06:18:28.316 INFO:tasks.rados.rados.0.smithi064.stdout:2580: finishing write tid 4 to smithi06431743-42 2024-03-04T06:18:28.316 INFO:tasks.rados.rados.0.smithi064.stdout:2580: finishing write tid 5 to smithi06431743-42 2024-03-04T06:18:28.316 INFO:tasks.rados.rados.0.smithi064.stdout:2580: oid 42 updating version 0 to 551 2024-03-04T06:18:28.316 INFO:tasks.rados.rados.0.smithi064.stdout:2580: oid 42 updating version 551 to 552 2024-03-04T06:18:28.316 INFO:tasks.rados.rados.0.smithi064.stdout:2580: oid 42 updating version 552 to 553 2024-03-04T06:18:28.316 INFO:tasks.rados.rados.0.smithi064.stdout:2580: oid 42 updating version 553 to 554 2024-03-04T06:18:28.316 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 554 (ObjNum 1182 snap 317 seq_num 1182) dirty exists 2024-03-04T06:18:28.317 INFO:tasks.rados.rados.0.smithi064.stdout:2580: left oid 42 (ObjNum 1182 snap 317 seq_num 1182) 2024-03-04T06:18:28.317 INFO:tasks.rados.rados.0.smithi064.stdout:2578: finishing write tid 4 to smithi06431743-5 2024-03-04T06:18:28.317 INFO:tasks.rados.rados.0.smithi064.stdout:2578: finishing write tid 5 to smithi06431743-5 2024-03-04T06:18:28.317 INFO:tasks.rados.rados.0.smithi064.stdout:2578: oid 5 updating version 0 to 1177 2024-03-04T06:18:28.317 INFO:tasks.rados.rados.0.smithi064.stdout:2578: oid 5 version 1177 is already newer than 1176 2024-03-04T06:18:28.317 INFO:tasks.rados.rados.0.smithi064.stdout:2578: oid 5 version 1177 is already newer than 1175 2024-03-04T06:18:28.317 INFO:tasks.rados.rados.0.smithi064.stdout:2578: oid 5 version 1177 is already newer than 1175 2024-03-04T06:18:28.317 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 1177 (ObjNum 1181 snap 317 seq_num 1181) dirty exists 2024-03-04T06:18:28.317 INFO:tasks.rados.rados.0.smithi064.stdout:2578: left oid 5 (ObjNum 1181 snap 317 seq_num 1181) 2024-03-04T06:18:29.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:29.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:29.228+0000 7f5517f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:29.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:29.232+0000 7f5517f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:29.311 INFO:tasks.rados.rados.0.smithi064.stdout:2578: done (2 left) 2024-03-04T06:18:29.311 INFO:tasks.rados.rados.0.smithi064.stdout:2580: done (1 left) 2024-03-04T06:18:29.311 INFO:tasks.rados.rados.0.smithi064.stdout:2583: done (0 left) 2024-03-04T06:18:29.311 INFO:tasks.rados.rados.0.smithi064.stdout:2584: delete oid 3 current snap is 318 2024-03-04T06:18:29.383 INFO:tasks.rados.rados.0.smithi064.stdout:2584: done (0 left) 2024-03-04T06:18:29.384 INFO:tasks.rados.rados.0.smithi064.stdout:2585: write (excl) oid 25 current snap is 318 2024-03-04T06:18:29.384 INFO:tasks.rados.rados.0.smithi064.stdout:2585: seq_num 1185 ranges {716295=404780,1593515=614409,2708717=492747,3640519=275776} 2024-03-04T06:18:29.389 INFO:tasks.rados.rados.0.smithi064.stdout:2585: writing smithi06431743-25 from 716295 to 1121075 tid 1 2024-03-04T06:18:29.397 INFO:tasks.rados.rados.0.smithi064.stdout:2585: writing smithi06431743-25 from 1593515 to 2207924 tid 2 2024-03-04T06:18:29.404 INFO:tasks.rados.rados.0.smithi064.stdout:2585: writing smithi06431743-25 from 2708717 to 3201464 tid 3 2024-03-04T06:18:29.410 INFO:tasks.rados.rados.0.smithi064.stdout:2585: writing smithi06431743-25 from 3640519 to 3916295 tid 4 2024-03-04T06:18:29.410 INFO:tasks.rados.rados.0.smithi064.stdout:2586: read oid 1 snap -1 2024-03-04T06:18:29.410 INFO:tasks.rados.rados.0.smithi064.stdout:2586: expect deleted 2024-03-04T06:18:29.410 INFO:tasks.rados.rados.0.smithi064.stdout:2587: delete oid 18 current snap is 318 2024-03-04T06:18:29.413 INFO:tasks.rados.rados.0.smithi064.stdout:2587: done (2 left) 2024-03-04T06:18:29.413 INFO:tasks.rados.rados.0.smithi064.stdout:2588: write (excl) oid 44 current snap is 318 2024-03-04T06:18:29.413 INFO:tasks.rados.rados.0.smithi064.stdout:2588: seq_num 1187 ranges {749909=671631,2110479=618078,3163563=515291} 2024-03-04T06:18:29.423 INFO:tasks.rados.rados.0.smithi064.stdout:2588: writing smithi06431743-44 from 749909 to 1421540 tid 1 2024-03-04T06:18:29.431 INFO:tasks.rados.rados.0.smithi064.stdout:2588: writing smithi06431743-44 from 2110479 to 2728557 tid 2 2024-03-04T06:18:29.436 INFO:tasks.rados.rados.0.smithi064.stdout:2588: writing smithi06431743-44 from 3163563 to 3678854 tid 3 2024-03-04T06:18:29.436 INFO:tasks.rados.rados.0.smithi064.stdout:2589: snap_remove snap 314 2024-03-04T06:18:30.327 INFO:tasks.rados.rados.0.smithi064.stdout:2589: done (3 left) 2024-03-04T06:18:30.327 INFO:tasks.rados.rados.0.smithi064.stdout:2590: read oid 9 snap -1 2024-03-04T06:18:30.328 INFO:tasks.rados.rados.0.smithi064.stdout:2590: expect deleted 2024-03-04T06:18:30.328 INFO:tasks.rados.rados.0.smithi064.stdout:2591: snap_remove snap 317 2024-03-04T06:18:31.304 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:31.332 INFO:tasks.rados.rados.0.smithi064.stdout:2591: done (4 left) 2024-03-04T06:18:31.332 INFO:tasks.rados.rados.0.smithi064.stdout:2592: write oid 15 current snap is 318 2024-03-04T06:18:31.332 INFO:tasks.rados.rados.0.smithi064.stdout:2585: finishing write tid 1 to smithi06431743-25 2024-03-04T06:18:31.332 INFO:tasks.rados.rados.0.smithi064.stdout:2592: seq_num 1188 ranges {766716=746880,2168961=761736,3474633=442916} 2024-03-04T06:18:31.340 INFO:tasks.rados.rados.0.smithi064.stdout:2592: writing smithi06431743-15 from 766716 to 1513596 tid 1 2024-03-04T06:18:31.348 INFO:tasks.rados.rados.0.smithi064.stdout:2592: writing smithi06431743-15 from 2168961 to 2930697 tid 2 2024-03-04T06:18:31.354 INFO:tasks.rados.rados.0.smithi064.stdout:2592: writing smithi06431743-15 from 3474633 to 3917549 tid 3 2024-03-04T06:18:31.354 INFO:tasks.rados.rados.0.smithi064.stdout:2593: delete oid 26 current snap is 318 2024-03-04T06:18:31.354 INFO:tasks.rados.rados.0.smithi064.stdout:2585: finishing write tid 2 to smithi06431743-25 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2585: finishing write tid 3 to smithi06431743-25 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2585: finishing write tid 4 to smithi06431743-25 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2585: finishing write tid 5 to smithi06431743-25 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2585: finishing write tid 6 to smithi06431743-25 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2585: oid 25 updating version 0 to 595 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2585: oid 25 updating version 595 to 596 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2585: oid 25 version 596 is already newer than 593 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2585: oid 25 version 596 is already newer than 592 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2585: oid 25 version 596 is already newer than 595 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 596 (ObjNum 1185 snap 318 seq_num 1185) dirty exists 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2585: left oid 25 (ObjNum 1185 snap 318 seq_num 1185) 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2588: finishing write tid 1 to smithi06431743-44 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2588: finishing write tid 2 to smithi06431743-44 2024-03-04T06:18:31.355 INFO:tasks.rados.rados.0.smithi064.stdout:2588: finishing write tid 3 to smithi06431743-44 2024-03-04T06:18:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:2588: finishing write tid 4 to smithi06431743-44 2024-03-04T06:18:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:2588: finishing write tid 5 to smithi06431743-44 2024-03-04T06:18:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:2588: oid 44 updating version 0 to 641 2024-03-04T06:18:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:2588: oid 44 version 641 is already newer than 639 2024-03-04T06:18:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:2588: oid 44 version 641 is already newer than 641 2024-03-04T06:18:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:2588: oid 44 updating version 641 to 644 2024-03-04T06:18:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 644 (ObjNum 1187 snap 318 seq_num 1187) dirty exists 2024-03-04T06:18:31.356 INFO:tasks.rados.rados.0.smithi064.stdout:2588: left oid 44 (ObjNum 1187 snap 318 seq_num 1187) 2024-03-04T06:18:31.421 INFO:tasks.rados.rados.0.smithi064.stdout:2592: finishing write tid 1 to smithi06431743-15 2024-03-04T06:18:31.424 INFO:tasks.rados.rados.0.smithi064.stdout:2585: done (5 left) 2024-03-04T06:18:31.424 INFO:tasks.rados.rados.0.smithi064.stdout:2586: done (4 left) 2024-03-04T06:18:31.424 INFO:tasks.rados.rados.0.smithi064.stdout:2588: done (3 left) 2024-03-04T06:18:31.424 INFO:tasks.rados.rados.0.smithi064.stdout:2590: done (2 left) 2024-03-04T06:18:31.424 INFO:tasks.rados.rados.0.smithi064.stdout:2593: done (1 left) 2024-03-04T06:18:31.425 INFO:tasks.rados.rados.0.smithi064.stdout:2594: delete oid 36 current snap is 318 2024-03-04T06:18:31.427 INFO:tasks.rados.rados.0.smithi064.stdout:2592: finishing write tid 2 to smithi06431743-15 2024-03-04T06:18:31.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:31.429+0000 7f369802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:31.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:31.434+0000 7f369802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:31.465 INFO:tasks.rados.rados.0.smithi064.stdout:2592: finishing write tid 3 to smithi06431743-15 2024-03-04T06:18:31.473 INFO:tasks.rados.rados.0.smithi064.stdout:2592: finishing write tid 4 to smithi06431743-15 2024-03-04T06:18:31.475 INFO:tasks.rados.rados.0.smithi064.stdout:2592: finishing write tid 5 to smithi06431743-15 2024-03-04T06:18:31.475 INFO:tasks.rados.rados.0.smithi064.stdout:2592: oid 15 updating version 0 to 575 2024-03-04T06:18:31.475 INFO:tasks.rados.rados.0.smithi064.stdout:2592: oid 15 version 575 is already newer than 573 2024-03-04T06:18:31.475 INFO:tasks.rados.rados.0.smithi064.stdout:2592: oid 15 version 575 is already newer than 573 2024-03-04T06:18:31.475 INFO:tasks.rados.rados.0.smithi064.stdout:2592: oid 15 version 575 is already newer than 575 2024-03-04T06:18:31.475 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 575 (ObjNum 1188 snap 318 seq_num 1188) dirty exists 2024-03-04T06:18:31.475 INFO:tasks.rados.rados.0.smithi064.stdout:2592: left oid 15 (ObjNum 1188 snap 318 seq_num 1188) 2024-03-04T06:18:31.482 INFO:tasks.rados.rados.0.smithi064.stdout:2592: done (1 left) 2024-03-04T06:18:31.482 INFO:tasks.rados.rados.0.smithi064.stdout:2594: done (0 left) 2024-03-04T06:18:31.482 INFO:tasks.rados.rados.0.smithi064.stdout:2595: read oid 34 snap -1 2024-03-04T06:18:31.482 INFO:tasks.rados.rados.0.smithi064.stdout:2595: expect (ObjNum 1114 snap 294 seq_num 1114) 2024-03-04T06:18:31.482 INFO:tasks.rados.rados.0.smithi064.stdout:2596: delete oid 11 current snap is 318 2024-03-04T06:18:31.502 INFO:tasks.rados.rados.0.smithi064.stdout:2595: expect (ObjNum 1114 snap 294 seq_num 1114) 2024-03-04T06:18:31.686 INFO:tasks.rados.rados.0.smithi064.stdout:2595: done (1 left) 2024-03-04T06:18:31.686 INFO:tasks.rados.rados.0.smithi064.stdout:2596: done (0 left) 2024-03-04T06:18:31.686 INFO:tasks.rados.rados.0.smithi064.stdout:2597: write oid 26 current snap is 318 2024-03-04T06:18:31.686 INFO:tasks.rados.rados.0.smithi064.stdout:2597: seq_num 1192 ranges {433944=647876,1486536=547408} 2024-03-04T06:18:31.689 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:18:31.689 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:18:31.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:18:31.692 INFO:tasks.rados.rados.0.smithi064.stdout:2597: writing smithi06431743-26 from 433944 to 1081820 tid 1 2024-03-04T06:18:31.697 INFO:tasks.rados.rados.0.smithi064.stdout:2597: writing smithi06431743-26 from 1486536 to 2033944 tid 2 2024-03-04T06:18:31.697 INFO:tasks.rados.rados.0.smithi064.stdout:2598: write (excl) oid 19 current snap is 318 2024-03-04T06:18:31.697 INFO:tasks.rados.rados.0.smithi064.stdout:2598: seq_num 1193 ranges {450751=723125,1945018=105733} 2024-03-04T06:18:31.706 INFO:tasks.rados.rados.0.smithi064.stdout:2598: writing smithi06431743-19 from 450751 to 1173876 tid 1 2024-03-04T06:18:31.712 INFO:tasks.rados.rados.0.smithi064.stdout:2598: writing smithi06431743-19 from 1945018 to 2050751 tid 2 2024-03-04T06:18:31.712 INFO:tasks.rados.rados.0.smithi064.stdout:2599: delete oid 8 current snap is 318 2024-03-04T06:18:31.712 INFO:tasks.rados.rados.0.smithi064.stdout:2597: finishing write tid 1 to smithi06431743-26 2024-03-04T06:18:31.731 INFO:tasks.rados.rados.0.smithi064.stdout:2598: finishing write tid 1 to smithi06431743-19 2024-03-04T06:18:31.742 INFO:tasks.rados.rados.0.smithi064.stdout:2597: finishing write tid 2 to smithi06431743-26 2024-03-04T06:18:31.742 INFO:tasks.rados.rados.0.smithi064.stdout:2598: finishing write tid 2 to smithi06431743-19 2024-03-04T06:18:31.749 INFO:tasks.rados.rados.0.smithi064.stdout:2599: done (2 left) 2024-03-04T06:18:31.749 INFO:tasks.rados.rados.0.smithi064.stdout:2600: snap_create 2024-03-04T06:18:31.751 INFO:tasks.rados.rados.0.smithi064.stdout:2597: finishing write tid 3 to smithi06431743-26 2024-03-04T06:18:31.753 INFO:tasks.rados.rados.0.smithi064.stdout:2597: finishing write tid 4 to smithi06431743-26 2024-03-04T06:18:31.753 INFO:tasks.rados.rados.0.smithi064.stdout:2597: oid 26 updating version 0 to 1057 2024-03-04T06:18:31.753 INFO:tasks.rados.rados.0.smithi064.stdout:2597: oid 26 version 1057 is already newer than 1057 2024-03-04T06:18:31.753 INFO:tasks.rados.rados.0.smithi064.stdout:2597: oid 26 version 1057 is already newer than 1055 2024-03-04T06:18:31.753 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 1057 (ObjNum 1192 snap 318 seq_num 1192) dirty exists 2024-03-04T06:18:31.753 INFO:tasks.rados.rados.0.smithi064.stdout:2597: left oid 26 (ObjNum 1192 snap 318 seq_num 1192) 2024-03-04T06:18:31.755 INFO:tasks.rados.rados.0.smithi064.stdout:2598: finishing write tid 3 to smithi06431743-19 2024-03-04T06:18:31.756 INFO:tasks.rados.rados.0.smithi064.stdout:2598: finishing write tid 4 to smithi06431743-19 2024-03-04T06:18:31.756 INFO:tasks.rados.rados.0.smithi064.stdout:2598: oid 19 updating version 0 to 1184 2024-03-04T06:18:31.756 INFO:tasks.rados.rados.0.smithi064.stdout:2598: oid 19 updating version 1184 to 1185 2024-03-04T06:18:31.756 INFO:tasks.rados.rados.0.smithi064.stdout:2598: oid 19 updating version 1185 to 1186 2024-03-04T06:18:31.757 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 1186 (ObjNum 1193 snap 318 seq_num 1193) dirty exists 2024-03-04T06:18:31.757 INFO:tasks.rados.rados.0.smithi064.stdout:2598: left oid 19 (ObjNum 1193 snap 318 seq_num 1193) 2024-03-04T06:18:31.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:31.795+0000 7fd863a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:31.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:31.801+0000 7fd863a24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:31.991 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:18:31.991 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:18:32.003 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":954,"stamp":"2024-03-04T06:18:31.330169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129543902,"num_objects":879,"num_object_clones":688,"num_object_copies":2637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":150,"num_read":4840,"num_read_kb":3731516,"num_write":4104,"num_write_kb":1084219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9318,"ondisk_log_size":9318,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":636588,"kb_used_data":444620,"kb_used_omap":0,"kb_used_meta":191936,"kb_avail":376850772,"statfs":{"total":386547056640,"available":385895190528,"internally_reserved":0,"allocated":455290880,"data_stored":547821678,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":196542464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1126102,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":7,"num_read_kb":4,"num_write":19,"num_write_kb":5880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.026345"},"pg_stats":[{"pgid":"3.f","version":"636'554","reported_seq":2201,"reported_epoch":638,"state":"active+clean","last_fresh":"2024-03-04T06:18:29.463905+0000","last_change":"2024-03-04T06:18:29.473779+0000","last_active":"2024-03-04T06:18:29.463905+0000","last_peered":"2024-03-04T06:18:29.463905+0000","last_clean":"2024-03-04T06:18:29.463905+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:18:29.463905+0000","last_undegraded":"2024-03-04T06:18:29.463905+0000","last_fullsized":"2024-03-04T06:18:29.463905+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7328380,"num_objects":57,"num_object_clones":44,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":273,"num_read_kb":224367,"num_write":233,"num_write_kb":62308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"639'455","reported_seq":2160,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-03-04T06:18:30.337343+0000","last_change":"2024-03-04T06:18:30.371557+0000","last_active":"2024-03-04T06:18:30.337343+0000","last_peered":"2024-03-04T06:18:30.337343+0000","last_clean":"2024-03-04T06:18:30.337343+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:18:30.337343+0000","last_undegraded":"2024-03-04T06:18:30.337343+0000","last_fullsized":"2024-03-04T06:18:30.337343+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3210401,"num_objects":47,"num_object_clones":33,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":166,"num_read_kb":111777,"num_write":187,"num_write_kb":49317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"633'588","reported_seq":2293,"reported_epoch":635,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:26.310471+0000","last_change":"2024-03-04T06:18:26.310471+0000","last_active":"2024-03-04T06:18:26.310471+0000","last_peered":"2024-03-04T06:18:26.310471+0000","last_clean":"2024-03-04T06:18:26.310471+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:18:26.310471+0000","last_undegraded":"2024-03-04T06:18:26.310471+0000","last_fullsized":"2024-03-04T06:18:26.310471+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9513554,"num_objects":63,"num_object_clones":49,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":369,"num_read_kb":298747,"num_write":273,"num_write_kb":75354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"635'470","reported_seq":2154,"reported_epoch":638,"state":"active+clean","last_fresh":"2024-03-04T06:18:29.442578+0000","last_change":"2024-03-04T06:18:29.450574+0000","last_active":"2024-03-04T06:18:29.442578+0000","last_peered":"2024-03-04T06:18:29.442578+0000","last_clean":"2024-03-04T06:18:29.442578+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:18:29.442578+0000","last_undegraded":"2024-03-04T06:18:29.442578+0000","last_fullsized":"2024-03-04T06:18:29.442578+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9580782,"num_objects":41,"num_object_clones":31,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":224,"num_read_kb":169192,"num_write":202,"num_write_kb":56330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"629'219","reported_seq":1993,"reported_epoch":635,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:26.395082+0000","last_change":"2024-03-04T06:18:26.395082+0000","last_active":"2024-03-04T06:18:26.395082+0000","last_peered":"2024-03-04T06:18:26.395082+0000","last_clean":"2024-03-04T06:18:26.395082+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:18:26.395082+0000","last_undegraded":"2024-03-04T06:18:26.395082+0000","last_fullsized":"2024-03-04T06:18:26.395082+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361664,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":82,"num_read_kb":57507,"num_write":93,"num_write_kb":25364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"638'644","reported_seq":2265,"reported_epoch":638,"state":"active+clean","last_fresh":"2024-03-04T06:18:29.536624+0000","last_change":"2024-03-04T06:18:29.538040+0000","last_active":"2024-03-04T06:18:29.536624+0000","last_peered":"2024-03-04T06:18:29.536624+0000","last_clean":"2024-03-04T06:18:29.536624+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:18:29.536624+0000","last_undegraded":"2024-03-04T06:18:29.536624+0000","last_fullsized":"2024-03-04T06:18:29.536624+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10084992,"num_objects":56,"num_object_clones":47,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":317,"num_read_kb":245870,"num_write":283,"num_write_kb":82964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1870,"reported_epoch":638,"state":"active+clean","last_fresh":"2024-03-04T06:18:29.458377+0000","last_change":"2024-03-04T06:18:29.470002+0000","last_active":"2024-03-04T06:18:29.458377+0000","last_peered":"2024-03-04T06:18:29.458377+0000","last_clean":"2024-03-04T06:18:29.458377+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:18:29.458377+0000","last_undegraded":"2024-03-04T06:18:29.458377+0000","last_fullsized":"2024-03-04T06:18:29.458377+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"629'351","reported_seq":2095,"reported_epoch":635,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:26.312652+0000","last_change":"2024-03-04T06:18:26.312652+0000","last_active":"2024-03-04T06:18:26.312652+0000","last_peered":"2024-03-04T06:18:26.312652+0000","last_clean":"2024-03-04T06:18:26.312652+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:18:26.312652+0000","last_undegraded":"2024-03-04T06:18:26.312652+0000","last_fullsized":"2024-03-04T06:18:26.312652+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3647383,"num_objects":31,"num_object_clones":22,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":194,"num_read_kb":177060,"num_write":137,"num_write_kb":36545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1718,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-03-04T06:18:26.376032+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:18:26.376032+0000","last_peered":"2024-03-04T06:18:26.376032+0000","last_clean":"2024-03-04T06:18:26.376032+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:18:26.376032+0000","last_undegraded":"2024-03-04T06:18:26.376032+0000","last_fullsized":"2024-03-04T06:18:26.376032+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1595,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-03-04T06:18:26.334103+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:18:26.334103+0000","last_peered":"2024-03-04T06:18:26.334103+0000","last_clean":"2024-03-04T06:18:26.334103+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:18:26.334103+0000","last_undegraded":"2024-03-04T06:18:26.334103+0000","last_fullsized":"2024-03-04T06:18:26.334103+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"629'607","reported_seq":2254,"reported_epoch":635,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:26.334067+0000","last_change":"2024-03-04T06:18:26.334067+0000","last_active":"2024-03-04T06:18:26.334067+0000","last_peered":"2024-03-04T06:18:26.334067+0000","last_clean":"2024-03-04T06:18:26.334067+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:18:26.334067+0000","last_undegraded":"2024-03-04T06:18:26.334067+0000","last_fullsized":"2024-03-04T06:18:26.334067+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9984150,"num_objects":57,"num_object_clones":44,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":310,"num_read_kb":246933,"num_write":259,"num_write_kb":70069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1578,"reported_epoch":638,"state":"active+clean","last_fresh":"2024-03-04T06:18:29.456876+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:18:29.456876+0000","last_peered":"2024-03-04T06:18:29.456876+0000","last_clean":"2024-03-04T06:18:29.456876+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:18:29.456876+0000","last_undegraded":"2024-03-04T06:18:29.456876+0000","last_fullsized":"2024-03-04T06:18:29.456876+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"628'389","reported_seq":2111,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-03-04T06:18:26.377123+0000","last_change":"2024-03-04T06:18:26.387492+0000","last_active":"2024-03-04T06:18:26.377123+0000","last_peered":"2024-03-04T06:18:26.377123+0000","last_clean":"2024-03-04T06:18:26.377123+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:18:26.377123+0000","last_undegraded":"2024-03-04T06:18:26.377123+0000","last_fullsized":"2024-03-04T06:18:26.377123+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370051,"num_objects":38,"num_object_clones":30,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":220,"num_read_kb":177502,"num_write":152,"num_write_kb":42787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1595,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-03-04T06:18:26.306602+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:18:26.306602+0000","last_peered":"2024-03-04T06:18:26.306602+0000","last_clean":"2024-03-04T06:18:26.306602+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:18:26.306602+0000","last_undegraded":"2024-03-04T06:18:26.306602+0000","last_fullsized":"2024-03-04T06:18:26.306602+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"629'1046","reported_seq":2573,"reported_epoch":635,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:26.386841+0000","last_change":"2024-03-04T06:18:26.386841+0000","last_active":"2024-03-04T06:18:26.386841+0000","last_peered":"2024-03-04T06:18:26.386841+0000","last_clean":"2024-03-04T06:18:26.386841+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:18:26.386841+0000","last_undegraded":"2024-03-04T06:18:26.386841+0000","last_fullsized":"2024-03-04T06:18:26.386841+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14841901,"num_objects":101,"num_object_clones":78,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":599,"num_read_kb":474728,"num_write":478,"num_write_kb":127819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1578,"reported_epoch":638,"state":"active+clean","last_fresh":"2024-03-04T06:18:29.443204+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:18:29.443204+0000","last_peered":"2024-03-04T06:18:29.443204+0000","last_clean":"2024-03-04T06:18:29.443204+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:18:29.443204+0000","last_undegraded":"2024-03-04T06:18:29.443204+0000","last_fullsized":"2024-03-04T06:18:29.443204+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"637'1181","reported_seq":2580,"reported_epoch":639,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:30.343275+0000","last_change":"2024-03-04T06:18:30.343275+0000","last_active":"2024-03-04T06:18:30.343275+0000","last_peered":"2024-03-04T06:18:30.343275+0000","last_clean":"2024-03-04T06:18:30.343275+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:18:30.343275+0000","last_undegraded":"2024-03-04T06:18:30.343275+0000","last_fullsized":"2024-03-04T06:18:30.343275+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1181,"log_dups_size":0,"ondisk_log_size":1181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18825127,"num_objects":108,"num_object_clones":86,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":485,"num_read_kb":355671,"num_write":502,"num_write_kb":134012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"632'554","reported_seq":2268,"reported_epoch":635,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:26.332483+0000","last_change":"2024-03-04T06:18:26.332483+0000","last_active":"2024-03-04T06:18:26.332483+0000","last_peered":"2024-03-04T06:18:26.332483+0000","last_clean":"2024-03-04T06:18:26.332483+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:18:26.332483+0000","last_undegraded":"2024-03-04T06:18:26.332483+0000","last_fullsized":"2024-03-04T06:18:26.332483+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5429189,"num_objects":52,"num_object_clones":40,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":362,"num_read_kb":299847,"num_write":242,"num_write_kb":65778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1627,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-03-04T06:18:30.350023+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:18:30.350023+0000","last_peered":"2024-03-04T06:18:30.350023+0000","last_clean":"2024-03-04T06:18:30.350023+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:18:30.350023+0000","last_undegraded":"2024-03-04T06:18:30.350023+0000","last_fullsized":"2024-03-04T06:18:30.350023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"637'544","reported_seq":2263,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-03-04T06:18:30.358301+0000","last_change":"2024-03-04T06:18:30.361371+0000","last_active":"2024-03-04T06:18:30.358301+0000","last_peered":"2024-03-04T06:18:30.358301+0000","last_clean":"2024-03-04T06:18:30.358301+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:18:30.358301+0000","last_undegraded":"2024-03-04T06:18:30.358301+0000","last_fullsized":"2024-03-04T06:18:30.358301+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10555588,"num_objects":59,"num_object_clones":43,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":287,"num_read_kb":215957,"num_write":242,"num_write_kb":64736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1585,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-03-04T06:18:26.387177+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:18:26.387177+0000","last_peered":"2024-03-04T06:18:26.387177+0000","last_clean":"2024-03-04T06:18:26.387177+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:18:26.387177+0000","last_undegraded":"2024-03-04T06:18:26.387177+0000","last_fullsized":"2024-03-04T06:18:26.387177+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"633'755","reported_seq":2338,"reported_epoch":635,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:26.379628+0000","last_change":"2024-03-04T06:18:26.379628+0000","last_active":"2024-03-04T06:18:26.379628+0000","last_peered":"2024-03-04T06:18:26.379628+0000","last_clean":"2024-03-04T06:18:26.379628+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:18:26.379628+0000","last_undegraded":"2024-03-04T06:18:26.379628+0000","last_fullsized":"2024-03-04T06:18:26.379628+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13530691,"num_objects":72,"num_object_clones":58,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":366,"num_read_kb":267439,"num_write":339,"num_write_kb":94107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1627,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-03-04T06:18:30.339748+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:18:30.339748+0000","last_peered":"2024-03-04T06:18:30.339748+0000","last_clean":"2024-03-04T06:18:30.339748+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:18:30.339748+0000","last_undegraded":"2024-03-04T06:18:30.339748+0000","last_fullsized":"2024-03-04T06:18:30.339748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"635'864","reported_seq":2449,"reported_epoch":639,"state":"active+clean","last_fresh":"2024-03-04T06:18:30.342208+0000","last_change":"2024-03-04T06:18:30.347858+0000","last_active":"2024-03-04T06:18:30.342208+0000","last_peered":"2024-03-04T06:18:30.342208+0000","last_clean":"2024-03-04T06:18:30.342208+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:18:30.342208+0000","last_undegraded":"2024-03-04T06:18:30.342208+0000","last_fullsized":"2024-03-04T06:18:30.342208+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6689681,"num_objects":75,"num_object_clones":66,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":483,"num_read_kb":408828,"num_write":345,"num_write_kb":94835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1595,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-03-04T06:18:26.333176+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:18:26.333176+0000","last_peered":"2024-03-04T06:18:26.333176+0000","last_clean":"2024-03-04T06:18:26.333176+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:18:26.333176+0000","last_undegraded":"2024-03-04T06:18:26.333176+0000","last_fullsized":"2024-03-04T06:18:26.333176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":128953534,"num_objects":877,"num_object_clones":688,"num_object_copies":2631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":150,"num_read":4737,"num_read_kb":3731425,"num_write":3967,"num_write_kb":1082325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9221,"ondisk_log_size":9221,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706304,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":185440,"kb_used_data":132248,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94186400,"statfs":{"total":96636764160,"available":96446873600,"internally_reserved":0,"allocated":135421952,"data_stored":161974908,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706304,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":140488,"kb_used_data":91456,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94231352,"statfs":{"total":96636764160,"available":96492904448,"internally_reserved":0,"allocated":93650944,"data_stored":106822442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706308,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":161344,"kb_used_data":116472,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94210496,"statfs":{"total":96636764160,"available":96471547904,"internally_reserved":0,"allocated":119267328,"data_stored":149473033,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706308,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":149316,"kb_used_data":104444,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94222524,"statfs":{"total":96636764160,"available":96483864576,"internally_reserved":0,"allocated":106950656,"data_stored":129551295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:18:32.332 INFO:tasks.rados.rados.0.smithi064.stdout:2597: done (2 left) 2024-03-04T06:18:32.332 INFO:tasks.rados.rados.0.smithi064.stdout:2598: done (1 left) 2024-03-04T06:18:32.333 INFO:tasks.rados.rados.0.smithi064.stdout:2600: done (0 left) 2024-03-04T06:18:32.333 INFO:tasks.rados.rados.0.smithi064.stdout:2601: write (excl) oid 19 current snap is 319 2024-03-04T06:18:32.333 INFO:tasks.rados.rados.0.smithi064.stdout:2601: seq_num 1195 ranges {484365=589976,1661982=422383} 2024-03-04T06:18:32.338 INFO:tasks.rados.rados.0.smithi064.stdout:2601: writing smithi06431743-19 from 484365 to 1074341 tid 1 2024-03-04T06:18:32.345 INFO:tasks.rados.rados.0.smithi064.stdout:2601: writing smithi06431743-19 from 1661982 to 2084365 tid 2 2024-03-04T06:18:32.345 INFO:tasks.rados.rados.0.smithi064.stdout:2602: snap_remove snap 316 2024-03-04T06:18:33.339 INFO:tasks.rados.rados.0.smithi064.stdout:2602: done (1 left) 2024-03-04T06:18:33.340 INFO:tasks.rados.rados.0.smithi064.stdout:2603: write oid 21 current snap is 319 2024-03-04T06:18:33.340 INFO:tasks.rados.rados.0.smithi064.stdout:2601: finishing write tid 1 to smithi06431743-19 2024-03-04T06:18:33.340 INFO:tasks.rados.rados.0.smithi064.stdout:2603: seq_num 1196 ranges {501172=665225,1720464=380708} 2024-03-04T06:18:33.345 INFO:tasks.rados.rados.0.smithi064.stdout:2603: writing smithi06431743-21 from 501172 to 1166397 tid 1 2024-03-04T06:18:33.351 INFO:tasks.rados.rados.0.smithi064.stdout:2603: writing smithi06431743-21 from 1720464 to 2101172 tid 2 2024-03-04T06:18:33.351 INFO:tasks.rados.rados.0.smithi064.stdout:2604: delete oid 47 current snap is 319 2024-03-04T06:18:33.351 INFO:tasks.rados.rados.0.smithi064.stdout:2601: finishing write tid 2 to smithi06431743-19 2024-03-04T06:18:33.351 INFO:tasks.rados.rados.0.smithi064.stdout:2601: finishing write tid 3 to smithi06431743-19 2024-03-04T06:18:33.352 INFO:tasks.rados.rados.0.smithi064.stdout:2601: finishing write tid 4 to smithi06431743-19 2024-03-04T06:18:33.352 INFO:tasks.rados.rados.0.smithi064.stdout:2601: oid 19 updating version 0 to 1190 2024-03-04T06:18:33.352 INFO:tasks.rados.rados.0.smithi064.stdout:2601: oid 19 version 1190 is already newer than 1190 2024-03-04T06:18:33.352 INFO:tasks.rados.rados.0.smithi064.stdout:2601: oid 19 version 1190 is already newer than 1188 2024-03-04T06:18:33.352 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 1190 (ObjNum 1195 snap 319 seq_num 1195) dirty exists 2024-03-04T06:18:33.352 INFO:tasks.rados.rados.0.smithi064.stdout:2601: left oid 19 (ObjNum 1195 snap 319 seq_num 1195) 2024-03-04T06:18:33.412 INFO:tasks.rados.rados.0.smithi064.stdout:2601: done (2 left) 2024-03-04T06:18:33.412 INFO:tasks.rados.rados.0.smithi064.stdout:2604: done (1 left) 2024-03-04T06:18:33.412 INFO:tasks.rados.rados.0.smithi064.stdout:2605: snap_remove snap 279 2024-03-04T06:18:33.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:33.630 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:33.628+0000 7fe4b3c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:33.633 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:33.633+0000 7fe4b3c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:34.341 INFO:tasks.rados.rados.0.smithi064.stdout:2605: done (1 left) 2024-03-04T06:18:34.341 INFO:tasks.rados.rados.0.smithi064.stdout:2606: write oid 17 current snap is 319 2024-03-04T06:18:34.341 INFO:tasks.rados.rados.0.smithi064.stdout:2603: finishing write tid 1 to smithi06431743-21 2024-03-04T06:18:34.341 INFO:tasks.rados.rados.0.smithi064.stdout:2606: seq_num 1198 ranges {534786=415723,1437428=697358} 2024-03-04T06:18:34.346 INFO:tasks.rados.rados.0.smithi064.stdout:2606: writing smithi06431743-17 from 534786 to 950509 tid 1 2024-03-04T06:18:34.353 INFO:tasks.rados.rados.0.smithi064.stdout:2606: writing smithi06431743-17 from 1437428 to 2134786 tid 2 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:2603: finishing write tid 2 to smithi06431743-21 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:2603: finishing write tid 3 to smithi06431743-21 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:2603: finishing write tid 4 to smithi06431743-21 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:2603: oid 21 updating version 0 to 578 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:2603: oid 21 updating version 578 to 580 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:2603: oid 21 version 580 is already newer than 580 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 580 (ObjNum 1196 snap 319 seq_num 1196) dirty exists 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:2603: left oid 21 (ObjNum 1196 snap 319 seq_num 1196) 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:2607: read oid 6 snap -1 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:2607: expect (ObjNum 1096 snap 286 seq_num 1096) 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:2603: done (2 left) 2024-03-04T06:18:34.354 INFO:tasks.rados.rados.0.smithi064.stdout:2608: snap_remove snap 299 2024-03-04T06:18:35.345 INFO:tasks.rados.rados.0.smithi064.stdout:2608: done (2 left) 2024-03-04T06:18:35.345 INFO:tasks.rados.rados.0.smithi064.stdout:2609: read oid 24 snap -1 2024-03-04T06:18:35.345 INFO:tasks.rados.rados.0.smithi064.stdout:2609: expect (ObjNum 1106 snap 289 seq_num 1106) 2024-03-04T06:18:35.345 INFO:tasks.rados.rados.0.smithi064.stdout:2610: snap_remove snap 285 2024-03-04T06:18:35.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:35.822 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:35.820+0000 7f0211c2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:35.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:35.824+0000 7f0211c2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:36.345 INFO:tasks.rados.rados.0.smithi064.stdout:2610: done (3 left) 2024-03-04T06:18:36.345 INFO:tasks.rados.rados.0.smithi064.stdout:2611: delete oid 23 current snap is 319 2024-03-04T06:18:36.345 INFO:tasks.rados.rados.0.smithi064.stdout:2607: expect (ObjNum 1096 snap 286 seq_num 1096) 2024-03-04T06:18:36.501 INFO:tasks.rados.rados.0.smithi064.stdout:2606: finishing write tid 1 to smithi06431743-17 2024-03-04T06:18:36.501 INFO:tasks.rados.rados.0.smithi064.stdout:2606: finishing write tid 2 to smithi06431743-17 2024-03-04T06:18:36.501 INFO:tasks.rados.rados.0.smithi064.stdout:2606: finishing write tid 3 to smithi06431743-17 2024-03-04T06:18:36.501 INFO:tasks.rados.rados.0.smithi064.stdout:2606: finishing write tid 4 to smithi06431743-17 2024-03-04T06:18:36.501 INFO:tasks.rados.rados.0.smithi064.stdout:2606: oid 17 updating version 0 to 362 2024-03-04T06:18:36.501 INFO:tasks.rados.rados.0.smithi064.stdout:2606: oid 17 version 362 is already newer than 362 2024-03-04T06:18:36.501 INFO:tasks.rados.rados.0.smithi064.stdout:2606: oid 17 version 362 is already newer than 360 2024-03-04T06:18:36.501 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 362 (ObjNum 1198 snap 319 seq_num 1198) dirty exists 2024-03-04T06:18:36.501 INFO:tasks.rados.rados.0.smithi064.stdout:2606: left oid 17 (ObjNum 1198 snap 319 seq_num 1198) 2024-03-04T06:18:36.501 INFO:tasks.rados.rados.0.smithi064.stdout:2606: done (3 left) 2024-03-04T06:18:36.502 INFO:tasks.rados.rados.0.smithi064.stdout:2607: done (2 left) 2024-03-04T06:18:36.502 INFO:tasks.rados.rados.0.smithi064.stdout:2611: done (1 left) 2024-03-04T06:18:36.502 INFO:tasks.rados.rados.0.smithi064.stdout:2612: delete oid 34 current snap is 319 2024-03-04T06:18:36.502 INFO:tasks.rados.rados.0.smithi064.stdout:2609: expect (ObjNum 1106 snap 289 seq_num 1106) 2024-03-04T06:18:36.596 INFO:tasks.rados.rados.0.smithi064.stdout:2609: done (1 left) 2024-03-04T06:18:36.596 INFO:tasks.rados.rados.0.smithi064.stdout:2612: done (0 left) 2024-03-04T06:18:36.596 INFO:tasks.rados.rados.0.smithi064.stdout:2613: read oid 4 snap 294 2024-03-04T06:18:36.597 INFO:tasks.rados.rados.0.smithi064.stdout:2613: expect (ObjNum 1112 snap 292 seq_num 1112) 2024-03-04T06:18:36.597 INFO:tasks.rados.rados.0.smithi064.stdout:2614: snap_remove snap 302 2024-03-04T06:18:37.005 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:18:37.005 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:18:37.005 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:18:37.006 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:18:37.117 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:37.112+0000 7fe50446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:37.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:37.119+0000 7fe50446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:37.132 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:37.132 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:37.132 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:37.132 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:37.132 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:37.132 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:37.132 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:37.133 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:37.133 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:37.133 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:37.133 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:37.133 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:37.135 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:37.135 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:37.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:18:37.243 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:37.238+0000 7f3cff79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:37.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:37.246+0000 7f3cff79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:37.259 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:37.259 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:37.259 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:37.259 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:37.259 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:37.259 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:37.259 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:37.259 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:37.259 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:37.260 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:37.260 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:37.260 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:37.262 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:37.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:18:37.352 INFO:tasks.rados.rados.0.smithi064.stdout:2614: done (1 left) 2024-03-04T06:18:37.352 INFO:tasks.rados.rados.0.smithi064.stdout:2615: snap_remove snap 318 2024-03-04T06:18:37.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:37.368+0000 7f78f2e0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:37.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:37.375+0000 7f78f2e0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:37.389 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:37.390 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:37.390 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:37.390 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:37.390 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:37.390 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:37.390 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:37.390 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:37.390 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:37.390 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:37.390 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:37.390 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:37.393 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:37.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:18:37.514 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:37.509+0000 7fe2eb354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:37.517 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:37.516+0000 7fe2eb354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:37.529 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:37.530 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:37.530 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:37.530 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:37.530 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:37.530 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:37.530 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:37.530 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:37.530 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:37.530 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:37.531 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:37.531 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:37.532 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:37.532 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:37.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:38.005 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:38.003+0000 7fa187c2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:38.008 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:38.007+0000 7fa187c2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:38.356 INFO:tasks.rados.rados.0.smithi064.stdout:2615: done (1 left) 2024-03-04T06:18:38.356 INFO:tasks.rados.rados.0.smithi064.stdout:2616: write oid 7 current snap is 319 2024-03-04T06:18:38.356 INFO:tasks.rados.rados.0.smithi064.stdout:2616: seq_num 1201 ranges {585207=641470,1896521=288686} 2024-03-04T06:18:38.363 INFO:tasks.rados.rados.0.smithi064.stdout:2616: writing smithi06431743-7 from 585207 to 1226677 tid 1 2024-03-04T06:18:38.369 INFO:tasks.rados.rados.0.smithi064.stdout:2616: writing smithi06431743-7 from 1896521 to 2185207 tid 2 2024-03-04T06:18:38.369 INFO:tasks.rados.rados.0.smithi064.stdout:2617: read oid 44 snap -1 2024-03-04T06:18:38.369 INFO:tasks.rados.rados.0.smithi064.stdout:2617: expect (ObjNum 1187 snap 318 seq_num 1187) 2024-03-04T06:18:38.369 INFO:tasks.rados.rados.0.smithi064.stdout:2618: snap_remove snap 291 2024-03-04T06:18:39.368 INFO:tasks.rados.rados.0.smithi064.stdout:2618: done (3 left) 2024-03-04T06:18:39.369 INFO:tasks.rados.rados.0.smithi064.stdout:2619: read oid 28 snap -1 2024-03-04T06:18:39.369 INFO:tasks.rados.rados.0.smithi064.stdout:2619: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:18:39.369 INFO:tasks.rados.rados.0.smithi064.stdout:2620: read oid 22 snap -1 2024-03-04T06:18:39.369 INFO:tasks.rados.rados.0.smithi064.stdout:2620: expect deleted 2024-03-04T06:18:39.369 INFO:tasks.rados.rados.0.smithi064.stdout:2621: delete oid 23 current snap is 319 2024-03-04T06:18:39.369 INFO:tasks.rados.rados.0.smithi064.stdout:2613: expect (ObjNum 1112 snap 292 seq_num 1112) 2024-03-04T06:18:39.555 INFO:tasks.rados.rados.0.smithi064.stdout:2617: expect (ObjNum 1187 snap 318 seq_num 1187) 2024-03-04T06:18:39.874 INFO:tasks.rados.rados.0.smithi064.stdout:2616: finishing write tid 1 to smithi06431743-7 2024-03-04T06:18:39.875 INFO:tasks.rados.rados.0.smithi064.stdout:2616: finishing write tid 2 to smithi06431743-7 2024-03-04T06:18:39.875 INFO:tasks.rados.rados.0.smithi064.stdout:2616: finishing write tid 3 to smithi06431743-7 2024-03-04T06:18:39.875 INFO:tasks.rados.rados.0.smithi064.stdout:2616: finishing write tid 4 to smithi06431743-7 2024-03-04T06:18:39.875 INFO:tasks.rados.rados.0.smithi064.stdout:2616: oid 7 updating version 0 to 392 2024-03-04T06:18:39.875 INFO:tasks.rados.rados.0.smithi064.stdout:2616: oid 7 updating version 392 to 393 2024-03-04T06:18:39.875 INFO:tasks.rados.rados.0.smithi064.stdout:2616: oid 7 version 393 is already newer than 392 2024-03-04T06:18:39.875 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 393 (ObjNum 1201 snap 319 seq_num 1201) dirty exists 2024-03-04T06:18:39.876 INFO:tasks.rados.rados.0.smithi064.stdout:2616: left oid 7 (ObjNum 1201 snap 319 seq_num 1201) 2024-03-04T06:18:39.876 INFO:tasks.rados.rados.0.smithi064.stdout:2613: done (5 left) 2024-03-04T06:18:39.876 INFO:tasks.rados.rados.0.smithi064.stdout:2616: done (4 left) 2024-03-04T06:18:39.876 INFO:tasks.rados.rados.0.smithi064.stdout:2617: done (3 left) 2024-03-04T06:18:39.876 INFO:tasks.rados.rados.0.smithi064.stdout:2621: done (2 left) 2024-03-04T06:18:39.876 INFO:tasks.rados.rados.0.smithi064.stdout:2622: read oid 39 snap -1 2024-03-04T06:18:39.876 INFO:tasks.rados.rados.0.smithi064.stdout:2622: expect (ObjNum 1120 snap 298 seq_num 1120) 2024-03-04T06:18:39.876 INFO:tasks.rados.rados.0.smithi064.stdout:2623: write (excl) oid 36 current snap is 319 2024-03-04T06:18:39.876 INFO:tasks.rados.rados.0.smithi064.stdout:2619: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:18:40.026 INFO:tasks.rados.rados.0.smithi064.stdout:2622: expect (ObjNum 1120 snap 298 seq_num 1120) 2024-03-04T06:18:40.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:40.164 INFO:tasks.rados.rados.0.smithi064.stdout:2623: seq_num 1203 ranges {618821=508321,1613485=531077} 2024-03-04T06:18:40.170 INFO:tasks.rados.rados.0.smithi064.stdout:2623: writing smithi06431743-36 from 618821 to 1127142 tid 1 2024-03-04T06:18:40.176 INFO:tasks.rados.rados.0.smithi064.stdout:2623: writing smithi06431743-36 from 1613485 to 2144562 tid 2 2024-03-04T06:18:40.176 INFO:tasks.rados.rados.0.smithi064.stdout:2619: done (3 left) 2024-03-04T06:18:40.176 INFO:tasks.rados.rados.0.smithi064.stdout:2620: done (2 left) 2024-03-04T06:18:40.176 INFO:tasks.rados.rados.0.smithi064.stdout:2622: done (1 left) 2024-03-04T06:18:40.176 INFO:tasks.rados.rados.0.smithi064.stdout:2624: read oid 43 snap -1 2024-03-04T06:18:40.176 INFO:tasks.rados.rados.0.smithi064.stdout:2624: expect deleted 2024-03-04T06:18:40.176 INFO:tasks.rados.rados.0.smithi064.stdout:2625: snap_create 2024-03-04T06:18:40.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:40.185+0000 7f39c4264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:40.188 INFO:tasks.rados.rados.0.smithi064.stdout:2623: finishing write tid 1 to smithi06431743-36 2024-03-04T06:18:40.191 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:40.190+0000 7f39c4264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:40.206 INFO:tasks.rados.rados.0.smithi064.stdout:2623: finishing write tid 2 to smithi06431743-36 2024-03-04T06:18:40.218 INFO:tasks.rados.rados.0.smithi064.stdout:2623: finishing write tid 3 to smithi06431743-36 2024-03-04T06:18:40.218 INFO:tasks.rados.rados.0.smithi064.stdout:2623: finishing write tid 4 to smithi06431743-36 2024-03-04T06:18:40.219 INFO:tasks.rados.rados.0.smithi064.stdout:2623: oid 36 updating version 0 to 575 2024-03-04T06:18:40.219 INFO:tasks.rados.rados.0.smithi064.stdout:2623: oid 36 version 575 is already newer than 573 2024-03-04T06:18:40.219 INFO:tasks.rados.rados.0.smithi064.stdout:2623: oid 36 version 575 is already newer than 574 2024-03-04T06:18:40.219 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 575 (ObjNum 1203 snap 319 seq_num 1203) dirty exists 2024-03-04T06:18:40.219 INFO:tasks.rados.rados.0.smithi064.stdout:2623: left oid 36 (ObjNum 1203 snap 319 seq_num 1203) 2024-03-04T06:18:40.370 INFO:tasks.rados.rados.0.smithi064.stdout:2623: done (2 left) 2024-03-04T06:18:40.370 INFO:tasks.rados.rados.0.smithi064.stdout:2624: done (1 left) 2024-03-04T06:18:40.370 INFO:tasks.rados.rados.0.smithi064.stdout:2625: done (0 left) 2024-03-04T06:18:40.371 INFO:tasks.rados.rados.0.smithi064.stdout:2626: delete oid 5 current snap is 320 2024-03-04T06:18:40.426 INFO:tasks.rados.rados.0.smithi064.stdout:2626: done (0 left) 2024-03-04T06:18:40.426 INFO:tasks.rados.rados.0.smithi064.stdout:2627: snap_create 2024-03-04T06:18:41.383 INFO:tasks.rados.rados.0.smithi064.stdout:2627: done (0 left) 2024-03-04T06:18:41.384 INFO:tasks.rados.rados.0.smithi064.stdout:2628: read oid 19 snap -1 2024-03-04T06:18:41.384 INFO:tasks.rados.rados.0.smithi064.stdout:2628: expect (ObjNum 1195 snap 319 seq_num 1195) 2024-03-04T06:18:41.384 INFO:tasks.rados.rados.0.smithi064.stdout:2629: read oid 17 snap -1 2024-03-04T06:18:41.384 INFO:tasks.rados.rados.0.smithi064.stdout:2629: expect (ObjNum 1198 snap 319 seq_num 1198) 2024-03-04T06:18:41.384 INFO:tasks.rados.rados.0.smithi064.stdout:2630: write (excl) oid 12 current snap is 321 2024-03-04T06:18:41.384 INFO:tasks.rados.rados.0.smithi064.stdout:2630: seq_num 1205 ranges {652435=658819,1730449=521986} 2024-03-04T06:18:41.390 INFO:tasks.rados.rados.0.smithi064.stdout:2630: writing smithi06431743-12 from 652435 to 1311254 tid 1 2024-03-04T06:18:41.397 INFO:tasks.rados.rados.0.smithi064.stdout:2630: writing smithi06431743-12 from 1730449 to 2252435 tid 2 2024-03-04T06:18:41.397 INFO:tasks.rados.rados.0.smithi064.stdout:2631: read oid 41 snap -1 2024-03-04T06:18:41.397 INFO:tasks.rados.rados.0.smithi064.stdout:2631: expect (ObjNum 1128 snap 299 seq_num 1128) 2024-03-04T06:18:41.397 INFO:tasks.rados.rados.0.smithi064.stdout:2632: read oid 11 snap 301 2024-03-04T06:18:41.398 INFO:tasks.rados.rados.0.smithi064.stdout:2632: expect deleted 2024-03-04T06:18:41.398 INFO:tasks.rados.rados.0.smithi064.stdout:2633: read oid 10 snap 300 2024-03-04T06:18:41.398 INFO:tasks.rados.rados.0.smithi064.stdout:2633: expect (ObjNum 1104 snap 288 seq_num 1104) 2024-03-04T06:18:41.398 INFO:tasks.rados.rados.0.smithi064.stdout:2634: read oid 1 snap 301 2024-03-04T06:18:41.398 INFO:tasks.rados.rados.0.smithi064.stdout:2634: expect deleted 2024-03-04T06:18:41.398 INFO:tasks.rados.rados.0.smithi064.stdout:2635: delete oid 23 current snap is 321 2024-03-04T06:18:41.433 INFO:tasks.rados.rados.0.smithi064.stdout:2633: expect (ObjNum 1104 snap 288 seq_num 1104) 2024-03-04T06:18:41.569 INFO:tasks.rados.rados.0.smithi064.stdout:2628: expect (ObjNum 1195 snap 319 seq_num 1195) 2024-03-04T06:18:41.664 INFO:tasks.rados.rados.0.smithi064.stdout:2631: expect (ObjNum 1128 snap 299 seq_num 1128) 2024-03-04T06:18:41.852 INFO:tasks.rados.rados.0.smithi064.stdout:2629: expect (ObjNum 1198 snap 319 seq_num 1198) 2024-03-04T06:18:41.930 INFO:tasks.rados.rados.0.smithi064.stdout:2630: finishing write tid 1 to smithi06431743-12 2024-03-04T06:18:41.930 INFO:tasks.rados.rados.0.smithi064.stdout:2630: finishing write tid 2 to smithi06431743-12 2024-03-04T06:18:41.930 INFO:tasks.rados.rados.0.smithi064.stdout:2630: finishing write tid 3 to smithi06431743-12 2024-03-04T06:18:41.930 INFO:tasks.rados.rados.0.smithi064.stdout:2630: finishing write tid 4 to smithi06431743-12 2024-03-04T06:18:41.930 INFO:tasks.rados.rados.0.smithi064.stdout:2630: oid 12 updating version 0 to 496 2024-03-04T06:18:41.930 INFO:tasks.rados.rados.0.smithi064.stdout:2630: oid 12 updating version 496 to 497 2024-03-04T06:18:41.930 INFO:tasks.rados.rados.0.smithi064.stdout:2630: oid 12 updating version 497 to 498 2024-03-04T06:18:41.930 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 498 (ObjNum 1205 snap 321 seq_num 1205) dirty exists 2024-03-04T06:18:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:2630: left oid 12 (ObjNum 1205 snap 321 seq_num 1205) 2024-03-04T06:18:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:2628: done (7 left) 2024-03-04T06:18:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:2629: done (6 left) 2024-03-04T06:18:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:2630: done (5 left) 2024-03-04T06:18:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:2631: done (4 left) 2024-03-04T06:18:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:2632: done (3 left) 2024-03-04T06:18:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:2633: done (2 left) 2024-03-04T06:18:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:2634: done (1 left) 2024-03-04T06:18:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:2635: done (0 left) 2024-03-04T06:18:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:2636: write oid 39 current snap is 321 2024-03-04T06:18:41.931 INFO:tasks.rados.rados.0.smithi064.stdout:2636: seq_num 1207 ranges {686049=409317,1847413=438636} 2024-03-04T06:18:41.935 INFO:tasks.rados.rados.0.smithi064.stdout:2636: writing smithi06431743-39 from 686049 to 1095366 tid 1 2024-03-04T06:18:41.942 INFO:tasks.rados.rados.0.smithi064.stdout:2636: writing smithi06431743-39 from 1847413 to 2286049 tid 2 2024-03-04T06:18:41.942 INFO:tasks.rados.rados.0.smithi064.stdout:2637: read oid 36 snap -1 2024-03-04T06:18:41.942 INFO:tasks.rados.rados.0.smithi064.stdout:2637: expect (ObjNum 1203 snap 319 seq_num 1203) 2024-03-04T06:18:41.942 INFO:tasks.rados.rados.0.smithi064.stdout:2638: snap_create 2024-03-04T06:18:41.964 INFO:tasks.rados.rados.0.smithi064.stdout:2636: finishing write tid 1 to smithi06431743-39 2024-03-04T06:18:41.965 INFO:tasks.rados.rados.0.smithi064.stdout:2637: expect (ObjNum 1203 snap 319 seq_num 1203) 2024-03-04T06:18:42.051 INFO:tasks.rados.rados.0.smithi064.stdout:2636: finishing write tid 2 to smithi06431743-39 2024-03-04T06:18:42.052 INFO:tasks.rados.rados.0.smithi064.stdout:2636: finishing write tid 3 to smithi06431743-39 2024-03-04T06:18:42.052 INFO:tasks.rados.rados.0.smithi064.stdout:2636: finishing write tid 4 to smithi06431743-39 2024-03-04T06:18:42.053 INFO:tasks.rados.rados.0.smithi064.stdout:2636: oid 39 updating version 0 to 612 2024-03-04T06:18:42.053 INFO:tasks.rados.rados.0.smithi064.stdout:2636: oid 39 updating version 612 to 614 2024-03-04T06:18:42.053 INFO:tasks.rados.rados.0.smithi064.stdout:2636: oid 39 version 614 is already newer than 614 2024-03-04T06:18:42.053 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 614 (ObjNum 1207 snap 321 seq_num 1207) dirty exists 2024-03-04T06:18:42.053 INFO:tasks.rados.rados.0.smithi064.stdout:2636: left oid 39 (ObjNum 1207 snap 321 seq_num 1207) 2024-03-04T06:18:42.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:42.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:42.368+0000 7f29bd55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:42.373 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:42.372+0000 7f29bd55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:42.379 INFO:tasks.rados.rados.0.smithi064.stdout:2636: done (2 left) 2024-03-04T06:18:42.379 INFO:tasks.rados.rados.0.smithi064.stdout:2637: done (1 left) 2024-03-04T06:18:42.379 INFO:tasks.rados.rados.0.smithi064.stdout:2638: done (0 left) 2024-03-04T06:18:42.380 INFO:tasks.rados.rados.0.smithi064.stdout:2639: read oid 40 snap -1 2024-03-04T06:18:42.380 INFO:tasks.rados.rados.0.smithi064.stdout:2639: expect (ObjNum 1158 snap 310 seq_num 1158) 2024-03-04T06:18:42.380 INFO:tasks.rados.rados.0.smithi064.stdout:2640: write (excl) oid 4 current snap is 322 2024-03-04T06:18:42.380 INFO:tasks.rados.rados.0.smithi064.stdout:2640: seq_num 1208 ranges {702856=484566,1789542=513314} 2024-03-04T06:18:42.386 INFO:tasks.rados.rados.0.smithi064.stdout:2640: writing smithi06431743-4 from 702856 to 1187422 tid 1 2024-03-04T06:18:42.392 INFO:tasks.rados.rados.0.smithi064.stdout:2640: writing smithi06431743-4 from 1789542 to 2302856 tid 2 2024-03-04T06:18:42.393 INFO:tasks.rados.rados.0.smithi064.stdout:2641: read oid 18 snap 305 2024-03-04T06:18:42.393 INFO:tasks.rados.rados.0.smithi064.stdout:2641: expect (ObjNum 1061 snap 272 seq_num 1061) 2024-03-04T06:18:42.393 INFO:tasks.rados.rados.0.smithi064.stdout:2642: snap_remove snap 313 2024-03-04T06:18:42.534 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:18:42.534 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:18:42.535 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:18:42.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:18:42.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:42.639+0000 7f415940f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:42.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:42.646+0000 7f415940f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:42.659 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:42.659 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:42.659 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:42.659 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:42.659 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:42.660 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:42.660 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:42.660 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:42.660 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:42.660 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:42.660 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:42.660 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:42.662 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:42.662 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:42.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:18:42.772 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:42.767+0000 7fb691024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:42.774 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:42.774+0000 7fb691024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:42.787 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:42.787 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:42.787 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:42.787 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:42.787 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:42.787 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:42.787 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:42.788 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:42.788 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:42.788 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:42.788 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:42.788 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:42.790 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:42.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:18:42.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:42.894+0000 7fcf1d59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:42.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:42.901+0000 7fcf1d59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:42.915 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:42.915 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:42.915 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:42.915 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:42.916 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:42.916 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:42.916 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:42.916 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:42.916 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:42.916 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:42.916 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:42.916 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:42.919 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:42.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:18:43.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:43.023+0000 7f7721b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:43.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:43.030+0000 7f7721b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:18:43.044 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:18:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:18:43.047 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:18:43.384 INFO:tasks.rados.rados.0.smithi064.stdout:2642: done (3 left) 2024-03-04T06:18:43.384 INFO:tasks.rados.rados.0.smithi064.stdout:2643: write (excl) oid 9 current snap is 322 2024-03-04T06:18:43.384 INFO:tasks.rados.rados.0.smithi064.stdout:2643: seq_num 1209 ranges {719663=676168,1964377=355286} 2024-03-04T06:18:43.391 INFO:tasks.rados.rados.0.smithi064.stdout:2643: writing smithi06431743-9 from 719663 to 1395831 tid 1 2024-03-04T06:18:43.397 INFO:tasks.rados.rados.0.smithi064.stdout:2643: writing smithi06431743-9 from 1964377 to 2319663 tid 2 2024-03-04T06:18:43.397 INFO:tasks.rados.rados.0.smithi064.stdout:2644: read oid 28 snap -1 2024-03-04T06:18:43.397 INFO:tasks.rados.rados.0.smithi064.stdout:2644: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:18:43.397 INFO:tasks.rados.rados.0.smithi064.stdout:2645: delete oid 23 current snap is 322 2024-03-04T06:18:43.397 INFO:tasks.rados.rados.0.smithi064.stdout:2639: expect (ObjNum 1158 snap 310 seq_num 1158) 2024-03-04T06:18:43.546 INFO:tasks.rados.rados.0.smithi064.stdout:2640: finishing write tid 1 to smithi06431743-4 2024-03-04T06:18:43.546 INFO:tasks.rados.rados.0.smithi064.stdout:2641: expect (ObjNum 1061 snap 272 seq_num 1061) 2024-03-04T06:18:43.776 INFO:tasks.rados.rados.0.smithi064.stdout:2640: finishing write tid 2 to smithi06431743-4 2024-03-04T06:18:43.776 INFO:tasks.rados.rados.0.smithi064.stdout:2640: finishing write tid 3 to smithi06431743-4 2024-03-04T06:18:43.776 INFO:tasks.rados.rados.0.smithi064.stdout:2640: finishing write tid 4 to smithi06431743-4 2024-03-04T06:18:43.776 INFO:tasks.rados.rados.0.smithi064.stdout:2640: oid 4 updating version 0 to 1227 2024-03-04T06:18:43.776 INFO:tasks.rados.rados.0.smithi064.stdout:2640: oid 4 updating version 1227 to 1228 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:2640: oid 4 version 1228 is already newer than 1226 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 1228 (ObjNum 1208 snap 322 seq_num 1208) dirty exists 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:2640: left oid 4 (ObjNum 1208 snap 322 seq_num 1208) 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:2639: done (5 left) 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:2640: done (4 left) 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:2641: done (3 left) 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:2645: done (2 left) 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:2646: read oid 47 snap -1 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:2646: expect deleted 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:2647: snap_create 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:2643: finishing write tid 1 to smithi06431743-9 2024-03-04T06:18:43.777 INFO:tasks.rados.rados.0.smithi064.stdout:2644: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:18:43.937 INFO:tasks.rados.rados.0.smithi064.stdout:2643: finishing write tid 2 to smithi06431743-9 2024-03-04T06:18:43.937 INFO:tasks.rados.rados.0.smithi064.stdout:2643: finishing write tid 3 to smithi06431743-9 2024-03-04T06:18:43.937 INFO:tasks.rados.rados.0.smithi064.stdout:2643: finishing write tid 4 to smithi06431743-9 2024-03-04T06:18:43.937 INFO:tasks.rados.rados.0.smithi064.stdout:2643: oid 9 updating version 0 to 875 2024-03-04T06:18:43.937 INFO:tasks.rados.rados.0.smithi064.stdout:2643: oid 9 updating version 875 to 877 2024-03-04T06:18:43.937 INFO:tasks.rados.rados.0.smithi064.stdout:2643: oid 9 version 877 is already newer than 876 2024-03-04T06:18:43.937 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 877 (ObjNum 1209 snap 322 seq_num 1209) dirty exists 2024-03-04T06:18:43.937 INFO:tasks.rados.rados.0.smithi064.stdout:2643: left oid 9 (ObjNum 1209 snap 322 seq_num 1209) 2024-03-04T06:18:44.392 INFO:tasks.rados.rados.0.smithi064.stdout:2643: done (3 left) 2024-03-04T06:18:44.392 INFO:tasks.rados.rados.0.smithi064.stdout:2644: done (2 left) 2024-03-04T06:18:44.392 INFO:tasks.rados.rados.0.smithi064.stdout:2646: done (1 left) 2024-03-04T06:18:44.392 INFO:tasks.rados.rados.0.smithi064.stdout:2647: done (0 left) 2024-03-04T06:18:44.392 INFO:tasks.rados.rados.0.smithi064.stdout:2648: delete oid 24 current snap is 323 2024-03-04T06:18:44.446 INFO:tasks.rados.rados.0.smithi064.stdout:2648: done (0 left) 2024-03-04T06:18:44.446 INFO:tasks.rados.rados.0.smithi064.stdout:2649: read oid 42 snap 294 2024-03-04T06:18:44.446 INFO:tasks.rados.rados.0.smithi064.stdout:2649: expect deleted 2024-03-04T06:18:44.447 INFO:tasks.rados.rados.0.smithi064.stdout:2650: write (excl) oid 32 current snap is 323 2024-03-04T06:18:44.447 INFO:tasks.rados.rados.0.smithi064.stdout:2650: seq_num 1212 ranges {770084=501915,2023470=346614} 2024-03-04T06:18:44.454 INFO:tasks.rados.rados.0.smithi064.stdout:2650: writing smithi06431743-32 from 770084 to 1271999 tid 1 2024-03-04T06:18:44.461 INFO:tasks.rados.rados.0.smithi064.stdout:2650: writing smithi06431743-32 from 2023470 to 2370084 tid 2 2024-03-04T06:18:44.461 INFO:tasks.rados.rados.0.smithi064.stdout:2651: read oid 15 snap -1 2024-03-04T06:18:44.461 INFO:tasks.rados.rados.0.smithi064.stdout:2651: expect (ObjNum 1188 snap 318 seq_num 1188) 2024-03-04T06:18:44.461 INFO:tasks.rados.rados.0.smithi064.stdout:2652: snap_create 2024-03-04T06:18:44.489 INFO:tasks.rados.rados.0.smithi064.stdout:2650: finishing write tid 1 to smithi06431743-32 2024-03-04T06:18:44.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:44.505 INFO:tasks.rados.rados.0.smithi064.stdout:2650: finishing write tid 2 to smithi06431743-32 2024-03-04T06:18:44.511 INFO:tasks.rados.rados.0.smithi064.stdout:2650: finishing write tid 3 to smithi06431743-32 2024-03-04T06:18:44.512 INFO:tasks.rados.rados.0.smithi064.stdout:2650: finishing write tid 4 to smithi06431743-32 2024-03-04T06:18:44.512 INFO:tasks.rados.rados.0.smithi064.stdout:2650: oid 32 updating version 0 to 1093 2024-03-04T06:18:44.512 INFO:tasks.rados.rados.0.smithi064.stdout:2650: oid 32 version 1093 is already newer than 1093 2024-03-04T06:18:44.512 INFO:tasks.rados.rados.0.smithi064.stdout:2650: oid 32 version 1093 is already newer than 1091 2024-03-04T06:18:44.512 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1093 (ObjNum 1212 snap 323 seq_num 1212) dirty exists 2024-03-04T06:18:44.512 INFO:tasks.rados.rados.0.smithi064.stdout:2650: left oid 32 (ObjNum 1212 snap 323 seq_num 1212) 2024-03-04T06:18:44.525 INFO:tasks.rados.rados.0.smithi064.stdout:2651: expect (ObjNum 1188 snap 318 seq_num 1188) 2024-03-04T06:18:44.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:44.600+0000 7faa1015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:44.604 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:44.604+0000 7faa1015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:45.398 INFO:tasks.rados.rados.0.smithi064.stdout:2649: done (3 left) 2024-03-04T06:18:45.398 INFO:tasks.rados.rados.0.smithi064.stdout:2650: done (2 left) 2024-03-04T06:18:45.399 INFO:tasks.rados.rados.0.smithi064.stdout:2651: done (1 left) 2024-03-04T06:18:45.399 INFO:tasks.rados.rados.0.smithi064.stdout:2652: done (0 left) 2024-03-04T06:18:45.399 INFO:tasks.rados.rados.0.smithi064.stdout:2653: snap_create 2024-03-04T06:18:46.402 INFO:tasks.rados.rados.0.smithi064.stdout:2653: done (0 left) 2024-03-04T06:18:46.402 INFO:tasks.rados.rados.0.smithi064.stdout:2654: delete oid 34 current snap is 325 2024-03-04T06:18:46.456 INFO:tasks.rados.rados.0.smithi064.stdout:2654: done (0 left) 2024-03-04T06:18:46.456 INFO:tasks.rados.rados.0.smithi064.stdout:2655: delete oid 1 current snap is 325 2024-03-04T06:18:46.461 INFO:tasks.rados.rados.0.smithi064.stdout:2655: done (0 left) 2024-03-04T06:18:46.461 INFO:tasks.rados.rados.0.smithi064.stdout:2656: snap_remove snap 305 2024-03-04T06:18:46.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:46.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:46.783+0000 7fdd0fa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:46.788 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:46.787+0000 7fdd0fa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:47.410 INFO:tasks.rados.rados.0.smithi064.stdout:2656: done (0 left) 2024-03-04T06:18:47.410 INFO:tasks.rados.rados.0.smithi064.stdout:2657: read oid 42 snap -1 2024-03-04T06:18:47.410 INFO:tasks.rados.rados.0.smithi064.stdout:2657: expect (ObjNum 1182 snap 317 seq_num 1182) 2024-03-04T06:18:47.410 INFO:tasks.rados.rados.0.smithi064.stdout:2658: snap_remove snap 312 2024-03-04T06:18:48.048 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:18:48.048 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:18:48.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:18:48.169 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:48.165+0000 7f92c679d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:48.172+0000 7f92c679d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:48.285 INFO:tasks.rados.rados.0.smithi064.stdout:2658: done (1 left) 2024-03-04T06:18:48.285 INFO:tasks.rados.rados.0.smithi064.stdout:2659: read oid 26 snap 324 2024-03-04T06:18:48.285 INFO:tasks.rados.rados.0.smithi064.stdout:2659: expect (ObjNum 1192 snap 318 seq_num 1192) 2024-03-04T06:18:48.285 INFO:tasks.rados.rados.0.smithi064.stdout:2660: delete oid 6 current snap is 325 2024-03-04T06:18:48.285 INFO:tasks.rados.rados.0.smithi064.stdout:2657: expect (ObjNum 1182 snap 317 seq_num 1182) 2024-03-04T06:18:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:18:48.410 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:18:48.422 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":979,"stamp":"2024-03-04T06:18:48.283733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107979828,"num_objects":901,"num_object_clones":708,"num_object_copies":2703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":156,"num_read":4969,"num_read_kb":3841520,"num_write":4184,"num_write_kb":1102850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9754,"ondisk_log_size":9754,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":678984,"kb_used_data":482856,"kb_used_omap":0,"kb_used_meta":196096,"kb_avail":376808376,"statfs":{"total":386547056640,"available":385851777024,"internally_reserved":0,"allocated":494444544,"data_stored":586280339,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":200802304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7193495,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":66,"num_read_kb":60154,"num_write":22,"num_write_kb":4797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.892538"},"pg_stats":[{"pgid":"3.f","version":"648'560","reported_seq":2255,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.437733+0000","last_change":"2024-03-04T06:18:43.808151+0000","last_active":"2024-03-04T06:18:44.437733+0000","last_peered":"2024-03-04T06:18:44.437733+0000","last_clean":"2024-03-04T06:18:44.437733+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:18:44.437733+0000","last_undegraded":"2024-03-04T06:18:44.437733+0000","last_fullsized":"2024-03-04T06:18:44.437733+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7328380,"num_objects":57,"num_object_clones":44,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":279,"num_read_kb":230589,"num_write":233,"num_write_kb":62308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"652'481","reported_seq":2212,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.416788+0000","last_change":"2024-03-04T06:18:43.469349+0000","last_active":"2024-03-04T06:18:44.416788+0000","last_peered":"2024-03-04T06:18:44.416788+0000","last_clean":"2024-03-04T06:18:44.416788+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:18:44.416788+0000","last_undegraded":"2024-03-04T06:18:44.416788+0000","last_fullsized":"2024-03-04T06:18:44.416788+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3210401,"num_objects":47,"num_object_clones":33,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":172,"num_read_kb":121244,"num_write":187,"num_write_kb":49317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"652'620","reported_seq":2363,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.485780+0000","last_change":"2024-03-04T06:18:43.504021+0000","last_active":"2024-03-04T06:18:44.485780+0000","last_peered":"2024-03-04T06:18:44.485780+0000","last_clean":"2024-03-04T06:18:44.485780+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:18:44.485780+0000","last_undegraded":"2024-03-04T06:18:44.485780+0000","last_fullsized":"2024-03-04T06:18:44.485780+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6202344,"num_objects":66,"num_object_clones":52,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":379,"num_read_kb":307023,"num_write":284,"num_write_kb":77932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"652'502","reported_seq":2210,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.427036+0000","last_change":"2024-03-04T06:18:43.807079+0000","last_active":"2024-03-04T06:18:44.427036+0000","last_peered":"2024-03-04T06:18:44.427036+0000","last_clean":"2024-03-04T06:18:44.427036+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:18:44.427036+0000","last_undegraded":"2024-03-04T06:18:44.427036+0000","last_fullsized":"2024-03-04T06:18:44.427036+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5530064,"num_objects":43,"num_object_clones":33,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":226,"num_read_kb":169193,"num_write":207,"num_write_kb":57484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"648'229","reported_seq":2053,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.414906+0000","last_change":"2024-03-04T06:18:43.455471+0000","last_active":"2024-03-04T06:18:44.414906+0000","last_peered":"2024-03-04T06:18:44.414906+0000","last_clean":"2024-03-04T06:18:44.414906+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:18:44.414906+0000","last_undegraded":"2024-03-04T06:18:44.414906+0000","last_fullsized":"2024-03-04T06:18:44.414906+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361664,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":82,"num_read_kb":57507,"num_write":93,"num_write_kb":25364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"652'672","reported_seq":2322,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.433777+0000","last_change":"2024-03-04T06:18:43.824448+0000","last_active":"2024-03-04T06:18:44.433777+0000","last_peered":"2024-03-04T06:18:44.433777+0000","last_clean":"2024-03-04T06:18:44.433777+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:18:44.433777+0000","last_undegraded":"2024-03-04T06:18:44.433777+0000","last_fullsized":"2024-03-04T06:18:44.433777+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10084992,"num_objects":56,"num_object_clones":47,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":329,"num_read_kb":261977,"num_write":283,"num_write_kb":82964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1921,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.433120+0000","last_change":"2024-03-04T06:18:43.808367+0000","last_active":"2024-03-04T06:18:44.433120+0000","last_peered":"2024-03-04T06:18:44.433120+0000","last_clean":"2024-03-04T06:18:44.433120+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:18:44.433120+0000","last_undegraded":"2024-03-04T06:18:44.433120+0000","last_fullsized":"2024-03-04T06:18:44.433120+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"648'372","reported_seq":2158,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.483805+0000","last_change":"2024-03-04T06:18:43.410883+0000","last_active":"2024-03-04T06:18:44.483805+0000","last_peered":"2024-03-04T06:18:44.483805+0000","last_clean":"2024-03-04T06:18:44.483805+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:18:44.483805+0000","last_undegraded":"2024-03-04T06:18:44.483805+0000","last_fullsized":"2024-03-04T06:18:44.483805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5782169,"num_objects":31,"num_object_clones":22,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":201,"num_read_kb":181320,"num_write":141,"num_write_kb":37633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1766,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.406993+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:18:44.406993+0000","last_peered":"2024-03-04T06:18:44.406993+0000","last_clean":"2024-03-04T06:18:44.406993+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:18:44.406993+0000","last_undegraded":"2024-03-04T06:18:44.406993+0000","last_fullsized":"2024-03-04T06:18:44.406993+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1639,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.496932+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:18:44.496932+0000","last_peered":"2024-03-04T06:18:44.496932+0000","last_clean":"2024-03-04T06:18:44.496932+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:18:44.496932+0000","last_undegraded":"2024-03-04T06:18:44.496932+0000","last_fullsized":"2024-03-04T06:18:44.496932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"652'648","reported_seq":2323,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.491167+0000","last_change":"2024-03-04T06:18:43.485910+0000","last_active":"2024-03-04T06:18:44.491167+0000","last_peered":"2024-03-04T06:18:44.491167+0000","last_clean":"2024-03-04T06:18:44.491167+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:18:44.491167+0000","last_undegraded":"2024-03-04T06:18:44.491167+0000","last_fullsized":"2024-03-04T06:18:44.491167+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2958296,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":320,"num_read_kb":254711,"num_write":266,"num_write_kb":71742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1619,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.437226+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:18:44.437226+0000","last_peered":"2024-03-04T06:18:44.437226+0000","last_clean":"2024-03-04T06:18:44.437226+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:18:44.437226+0000","last_undegraded":"2024-03-04T06:18:44.437226+0000","last_fullsized":"2024-03-04T06:18:44.437226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"652'397","reported_seq":2174,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.402506+0000","last_change":"2024-03-04T06:18:43.490299+0000","last_active":"2024-03-04T06:18:44.402506+0000","last_peered":"2024-03-04T06:18:44.402506+0000","last_clean":"2024-03-04T06:18:44.402506+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:18:44.402506+0000","last_undegraded":"2024-03-04T06:18:44.402506+0000","last_fullsized":"2024-03-04T06:18:44.402506+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185207,"num_objects":39,"num_object_clones":31,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":221,"num_read_kb":177503,"num_write":156,"num_write_kb":43696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1639,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.482458+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:18:44.482458+0000","last_peered":"2024-03-04T06:18:44.482458+0000","last_clean":"2024-03-04T06:18:44.482458+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:18:44.482458+0000","last_undegraded":"2024-03-04T06:18:44.482458+0000","last_fullsized":"2024-03-04T06:18:44.482458+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"653'1093","reported_seq":2652,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.494181+0000","last_change":"2024-03-04T06:18:43.497906+0000","last_active":"2024-03-04T06:18:44.494181+0000","last_peered":"2024-03-04T06:18:44.494181+0000","last_clean":"2024-03-04T06:18:44.494181+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:18:44.494181+0000","last_undegraded":"2024-03-04T06:18:44.494181+0000","last_fullsized":"2024-03-04T06:18:44.494181+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1093,"log_dups_size":0,"ondisk_log_size":1093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10656760,"num_objects":104,"num_object_clones":81,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":621,"num_read_kb":494980,"num_write":488,"num_write_kb":129817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1619,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.428412+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:18:44.428412+0000","last_peered":"2024-03-04T06:18:44.428412+0000","last_clean":"2024-03-04T06:18:44.428412+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:18:44.428412+0000","last_undegraded":"2024-03-04T06:18:44.428412+0000","last_fullsized":"2024-03-04T06:18:44.428412+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"652'1232","reported_seq":2649,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.410533+0000","last_change":"2024-03-04T06:18:43.490857+0000","last_active":"2024-03-04T06:18:44.410533+0000","last_peered":"2024-03-04T06:18:44.410533+0000","last_clean":"2024-03-04T06:18:44.410533+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:18:44.410533+0000","last_undegraded":"2024-03-04T06:18:44.410533+0000","last_fullsized":"2024-03-04T06:18:44.410533+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1232,"log_dups_size":0,"ondisk_log_size":1232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15699091,"num_objects":112,"num_object_clones":89,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":504,"num_read_kb":366968,"num_write":515,"num_write_kb":136789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"652'606","reported_seq":2345,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.518340+0000","last_change":"2024-03-04T06:18:43.475860+0000","last_active":"2024-03-04T06:18:44.518340+0000","last_peered":"2024-03-04T06:18:44.518340+0000","last_clean":"2024-03-04T06:18:44.518340+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:18:44.518340+0000","last_undegraded":"2024-03-04T06:18:44.518340+0000","last_fullsized":"2024-03-04T06:18:44.518340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8488360,"num_objects":54,"num_object_clones":42,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":377,"num_read_kb":317169,"num_write":257,"num_write_kb":70208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1667,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.423884+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:18:44.423884+0000","last_peered":"2024-03-04T06:18:44.423884+0000","last_clean":"2024-03-04T06:18:44.423884+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:18:44.423884+0000","last_undegraded":"2024-03-04T06:18:44.423884+0000","last_fullsized":"2024-03-04T06:18:44.423884+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"652'581","reported_seq":2321,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.425140+0000","last_change":"2024-03-04T06:18:43.506305+0000","last_active":"2024-03-04T06:18:44.425140+0000","last_peered":"2024-03-04T06:18:44.425140+0000","last_clean":"2024-03-04T06:18:44.425140+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:18:44.425140+0000","last_undegraded":"2024-03-04T06:18:44.425140+0000","last_fullsized":"2024-03-04T06:18:44.425140+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5950239,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":297,"num_read_kb":220157,"num_write":248,"num_write_kb":65752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1633,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.420113+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:18:44.420113+0000","last_peered":"2024-03-04T06:18:44.420113+0000","last_clean":"2024-03-04T06:18:44.420113+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:18:44.420113+0000","last_undegraded":"2024-03-04T06:18:44.420113+0000","last_fullsized":"2024-03-04T06:18:44.420113+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"652'787","reported_seq":2402,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.408602+0000","last_change":"2024-03-04T06:18:43.487434+0000","last_active":"2024-03-04T06:18:44.408602+0000","last_peered":"2024-03-04T06:18:44.408602+0000","last_clean":"2024-03-04T06:18:44.408602+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:18:44.408602+0000","last_undegraded":"2024-03-04T06:18:44.408602+0000","last_fullsized":"2024-03-04T06:18:44.408602+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10942149,"num_objects":73,"num_object_clones":59,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":373,"num_read_kb":272259,"num_write":340,"num_write_kb":94107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1667,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.413778+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:18:44.413778+0000","last_peered":"2024-03-04T06:18:44.413778+0000","last_clean":"2024-03-04T06:18:44.413778+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:18:44.413778+0000","last_undegraded":"2024-03-04T06:18:44.413778+0000","last_fullsized":"2024-03-04T06:18:44.413778+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"652'877","reported_seq":2502,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.409545+0000","last_change":"2024-03-04T06:18:43.421598+0000","last_active":"2024-03-04T06:18:44.409545+0000","last_peered":"2024-03-04T06:18:44.409545+0000","last_clean":"2024-03-04T06:18:44.409545+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:18:44.409545+0000","last_undegraded":"2024-03-04T06:18:44.409545+0000","last_fullsized":"2024-03-04T06:18:44.409545+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":877,"log_dups_size":0,"ondisk_log_size":877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9009344,"num_objects":76,"num_object_clones":66,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":485,"num_read_kb":408829,"num_write":349,"num_write_kb":95843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1639,"reported_epoch":653,"state":"active+clean","last_fresh":"2024-03-04T06:18:44.490235+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:18:44.490235+0000","last_peered":"2024-03-04T06:18:44.490235+0000","last_clean":"2024-03-04T06:18:44.490235+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:18:44.490235+0000","last_undegraded":"2024-03-04T06:18:44.490235+0000","last_fullsized":"2024-03-04T06:18:44.490235+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":107389460,"num_objects":899,"num_object_clones":708,"num_object_copies":2697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":156,"num_read":4866,"num_read_kb":3841429,"num_write":4047,"num_write_kb":1100956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9657,"ondisk_log_size":9657,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706324,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":201776,"kb_used_data":148584,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94170064,"statfs":{"total":96636764160,"available":96430145536,"internally_reserved":0,"allocated":152150016,"data_stored":181562035,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706324,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":152640,"kb_used_data":103608,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94219200,"statfs":{"total":96636764160,"available":96480460800,"internally_reserved":0,"allocated":106094592,"data_stored":119585120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":172608,"kb_used_data":123576,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94199232,"statfs":{"total":96636764160,"available":96460013568,"internally_reserved":0,"allocated":126541824,"data_stored":153599419,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706323,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":151960,"kb_used_data":107088,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94219880,"statfs":{"total":96636764160,"available":96481157120,"internally_reserved":0,"allocated":109658112,"data_stored":131533765,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:18:48.441 INFO:tasks.rados.rados.0.smithi064.stdout:2659: expect (ObjNum 1192 snap 318 seq_num 1192) 2024-03-04T06:18:48.515 INFO:tasks.rados.rados.0.smithi064.stdout:2657: done (2 left) 2024-03-04T06:18:48.515 INFO:tasks.rados.rados.0.smithi064.stdout:2659: done (1 left) 2024-03-04T06:18:48.515 INFO:tasks.rados.rados.0.smithi064.stdout:2660: done (0 left) 2024-03-04T06:18:48.515 INFO:tasks.rados.rados.0.smithi064.stdout:2661: write oid 45 current snap is 325 2024-03-04T06:18:48.516 INFO:tasks.rados.rados.0.smithi064.stdout:2661: seq_num 1216 ranges {437312=402911,1341045=696749} 2024-03-04T06:18:48.519 INFO:tasks.rados.rados.0.smithi064.stdout:2661: writing smithi06431743-45 from 437312 to 840223 tid 1 2024-03-04T06:18:48.527 INFO:tasks.rados.rados.0.smithi064.stdout:2661: writing smithi06431743-45 from 1341045 to 2037794 tid 2 2024-03-04T06:18:48.528 INFO:tasks.rados.rados.0.smithi064.stdout:2662: write oid 26 current snap is 325 2024-03-04T06:18:48.528 INFO:tasks.rados.rados.0.smithi064.stdout:2662: seq_num 1217 ranges {454119=594513,1515880=556760} 2024-03-04T06:18:48.534 INFO:tasks.rados.rados.0.smithi064.stdout:2662: writing smithi06431743-26 from 454119 to 1048632 tid 1 2024-03-04T06:18:48.539 INFO:tasks.rados.rados.0.smithi064.stdout:2662: writing smithi06431743-26 from 1515880 to 2072640 tid 2 2024-03-04T06:18:48.539 INFO:tasks.rados.rados.0.smithi064.stdout:2663: snap_create 2024-03-04T06:18:48.545 INFO:tasks.rados.rados.0.smithi064.stdout:2661: finishing write tid 1 to smithi06431743-45 2024-03-04T06:18:48.564 INFO:tasks.rados.rados.0.smithi064.stdout:2661: finishing write tid 2 to smithi06431743-45 2024-03-04T06:18:48.567 INFO:tasks.rados.rados.0.smithi064.stdout:2662: finishing write tid 1 to smithi06431743-26 2024-03-04T06:18:48.573 INFO:tasks.rados.rados.0.smithi064.stdout:2661: finishing write tid 3 to smithi06431743-45 2024-03-04T06:18:48.574 INFO:tasks.rados.rados.0.smithi064.stdout:2661: finishing write tid 4 to smithi06431743-45 2024-03-04T06:18:48.574 INFO:tasks.rados.rados.0.smithi064.stdout:2661: oid 45 updating version 0 to 624 2024-03-04T06:18:48.575 INFO:tasks.rados.rados.0.smithi064.stdout:2661: oid 45 version 624 is already newer than 623 2024-03-04T06:18:48.575 INFO:tasks.rados.rados.0.smithi064.stdout:2661: oid 45 updating version 624 to 625 2024-03-04T06:18:48.575 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 625 (ObjNum 1216 snap 325 seq_num 1216) dirty exists 2024-03-04T06:18:48.575 INFO:tasks.rados.rados.0.smithi064.stdout:2661: left oid 45 (ObjNum 1216 snap 325 seq_num 1216) 2024-03-04T06:18:48.583 INFO:tasks.rados.rados.0.smithi064.stdout:2662: finishing write tid 2 to smithi06431743-26 2024-03-04T06:18:48.588 INFO:tasks.rados.rados.0.smithi064.stdout:2662: finishing write tid 3 to smithi06431743-26 2024-03-04T06:18:48.589 INFO:tasks.rados.rados.0.smithi064.stdout:2662: finishing write tid 4 to smithi06431743-26 2024-03-04T06:18:48.589 INFO:tasks.rados.rados.0.smithi064.stdout:2662: oid 26 updating version 0 to 1109 2024-03-04T06:18:48.589 INFO:tasks.rados.rados.0.smithi064.stdout:2662: oid 26 version 1109 is already newer than 1109 2024-03-04T06:18:48.589 INFO:tasks.rados.rados.0.smithi064.stdout:2662: oid 26 version 1109 is already newer than 1107 2024-03-04T06:18:48.589 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 1109 (ObjNum 1217 snap 325 seq_num 1217) dirty exists 2024-03-04T06:18:48.590 INFO:tasks.rados.rados.0.smithi064.stdout:2662: left oid 26 (ObjNum 1217 snap 325 seq_num 1217) 2024-03-04T06:18:48.857 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:48.964 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:48.962+0000 7facb4e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:48.967 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:48.966+0000 7facb4e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:49.412 INFO:tasks.rados.rados.0.smithi064.stdout:2661: done (2 left) 2024-03-04T06:18:49.412 INFO:tasks.rados.rados.0.smithi064.stdout:2662: done (1 left) 2024-03-04T06:18:49.412 INFO:tasks.rados.rados.0.smithi064.stdout:2663: done (0 left) 2024-03-04T06:18:49.413 INFO:tasks.rados.rados.0.smithi064.stdout:2664: write (excl) oid 12 current snap is 326 2024-03-04T06:18:49.413 INFO:tasks.rados.rados.0.smithi064.stdout:2664: seq_num 1218 ranges {470926=669762,1574362=700418} 2024-03-04T06:18:49.419 INFO:tasks.rados.rados.0.smithi064.stdout:2664: writing smithi06431743-12 from 470926 to 1140688 tid 1 2024-03-04T06:18:49.425 INFO:tasks.rados.rados.0.smithi064.stdout:2664: writing smithi06431743-12 from 1574362 to 2274780 tid 2 2024-03-04T06:18:49.426 INFO:tasks.rados.rados.0.smithi064.stdout:2665: delete oid 29 current snap is 326 2024-03-04T06:18:49.541 INFO:tasks.rados.rados.0.smithi064.stdout:2665: done (1 left) 2024-03-04T06:18:49.541 INFO:tasks.rados.rados.0.smithi064.stdout:2666: snap_create 2024-03-04T06:18:49.543 INFO:tasks.rados.rados.0.smithi064.stdout:2664: finishing write tid 1 to smithi06431743-12 2024-03-04T06:18:49.555 INFO:tasks.rados.rados.0.smithi064.stdout:2664: finishing write tid 2 to smithi06431743-12 2024-03-04T06:18:49.562 INFO:tasks.rados.rados.0.smithi064.stdout:2664: finishing write tid 3 to smithi06431743-12 2024-03-04T06:18:49.564 INFO:tasks.rados.rados.0.smithi064.stdout:2664: finishing write tid 4 to smithi06431743-12 2024-03-04T06:18:49.564 INFO:tasks.rados.rados.0.smithi064.stdout:2664: oid 12 updating version 0 to 515 2024-03-04T06:18:49.564 INFO:tasks.rados.rados.0.smithi064.stdout:2664: oid 12 updating version 515 to 516 2024-03-04T06:18:49.564 INFO:tasks.rados.rados.0.smithi064.stdout:2664: oid 12 version 516 is already newer than 514 2024-03-04T06:18:49.564 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 516 (ObjNum 1218 snap 326 seq_num 1218) dirty exists 2024-03-04T06:18:49.564 INFO:tasks.rados.rados.0.smithi064.stdout:2664: left oid 12 (ObjNum 1218 snap 326 seq_num 1218) 2024-03-04T06:18:50.414 INFO:tasks.rados.rados.0.smithi064.stdout:2664: done (1 left) 2024-03-04T06:18:50.414 INFO:tasks.rados.rados.0.smithi064.stdout:2666: done (0 left) 2024-03-04T06:18:50.414 INFO:tasks.rados.rados.0.smithi064.stdout:2667: write (excl) oid 24 current snap is 327 2024-03-04T06:18:50.414 INFO:tasks.rados.rados.0.smithi064.stdout:2667: seq_num 1220 ranges {504540=420260,1574973=704087} 2024-03-04T06:18:50.419 INFO:tasks.rados.rados.0.smithi064.stdout:2667: writing smithi06431743-24 from 504540 to 924800 tid 1 2024-03-04T06:18:50.427 INFO:tasks.rados.rados.0.smithi064.stdout:2667: writing smithi06431743-24 from 1574973 to 2279060 tid 2 2024-03-04T06:18:50.427 INFO:tasks.rados.rados.0.smithi064.stdout:2668: snap_create 2024-03-04T06:18:50.465 INFO:tasks.rados.rados.0.smithi064.stdout:2667: finishing write tid 1 to smithi06431743-24 2024-03-04T06:18:50.480 INFO:tasks.rados.rados.0.smithi064.stdout:2667: finishing write tid 2 to smithi06431743-24 2024-03-04T06:18:50.484 INFO:tasks.rados.rados.0.smithi064.stdout:2667: finishing write tid 3 to smithi06431743-24 2024-03-04T06:18:50.485 INFO:tasks.rados.rados.0.smithi064.stdout:2667: finishing write tid 4 to smithi06431743-24 2024-03-04T06:18:50.486 INFO:tasks.rados.rados.0.smithi064.stdout:2667: oid 24 updating version 0 to 804 2024-03-04T06:18:50.486 INFO:tasks.rados.rados.0.smithi064.stdout:2667: oid 24 version 804 is already newer than 802 2024-03-04T06:18:50.486 INFO:tasks.rados.rados.0.smithi064.stdout:2667: oid 24 version 804 is already newer than 804 2024-03-04T06:18:50.486 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 804 (ObjNum 1220 snap 327 seq_num 1220) dirty exists 2024-03-04T06:18:50.486 INFO:tasks.rados.rados.0.smithi064.stdout:2667: left oid 24 (ObjNum 1220 snap 327 seq_num 1220) 2024-03-04T06:18:51.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:51.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:51.157+0000 7f9080f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:51.162 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:51.161+0000 7f9080f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:51.419 INFO:tasks.rados.rados.0.smithi064.stdout:2667: done (1 left) 2024-03-04T06:18:51.419 INFO:tasks.rados.rados.0.smithi064.stdout:2668: done (0 left) 2024-03-04T06:18:51.419 INFO:tasks.rados.rados.0.smithi064.stdout:2669: write oid 14 current snap is 328 2024-03-04T06:18:51.419 INFO:tasks.rados.rados.0.smithi064.stdout:2669: seq_num 1221 ranges {521347=495509,1633455=564098} 2024-03-04T06:18:51.424 INFO:tasks.rados.rados.0.smithi064.stdout:2669: writing smithi06431743-14 from 521347 to 1016856 tid 1 2024-03-04T06:18:51.432 INFO:tasks.rados.rados.0.smithi064.stdout:2669: writing smithi06431743-14 from 1633455 to 2197553 tid 2 2024-03-04T06:18:51.432 INFO:tasks.rados.rados.0.smithi064.stdout:2670: read oid 8 snap -1 2024-03-04T06:18:51.432 INFO:tasks.rados.rados.0.smithi064.stdout:2670: expect deleted 2024-03-04T06:18:51.432 INFO:tasks.rados.rados.0.smithi064.stdout:2671: write oid 46 current snap is 328 2024-03-04T06:18:51.432 INFO:tasks.rados.rados.0.smithi064.stdout:2671: seq_num 1222 ranges {538154=570758,1691937=707756} 2024-03-04T06:18:51.439 INFO:tasks.rados.rados.0.smithi064.stdout:2671: writing smithi06431743-46 from 538154 to 1108912 tid 1 2024-03-04T06:18:51.447 INFO:tasks.rados.rados.0.smithi064.stdout:2671: writing smithi06431743-46 from 1691937 to 2399693 tid 2 2024-03-04T06:18:51.447 INFO:tasks.rados.rados.0.smithi064.stdout:2672: write (excl) oid 30 current snap is 328 2024-03-04T06:18:51.448 INFO:tasks.rados.rados.0.smithi064.stdout:2672: seq_num 1223 ranges {554961=646007,1750419=451414} 2024-03-04T06:18:51.454 INFO:tasks.rados.rados.0.smithi064.stdout:2672: writing smithi06431743-30 from 554961 to 1200968 tid 1 2024-03-04T06:18:51.459 INFO:tasks.rados.rados.0.smithi064.stdout:2672: writing smithi06431743-30 from 1750419 to 2201833 tid 2 2024-03-04T06:18:51.460 INFO:tasks.rados.rados.0.smithi064.stdout:2673: delete oid 36 current snap is 328 2024-03-04T06:18:51.501 INFO:tasks.rados.rados.0.smithi064.stdout:2669: finishing write tid 1 to smithi06431743-14 2024-03-04T06:18:51.526 INFO:tasks.rados.rados.0.smithi064.stdout:2669: finishing write tid 2 to smithi06431743-14 2024-03-04T06:18:51.527 INFO:tasks.rados.rados.0.smithi064.stdout:2671: finishing write tid 1 to smithi06431743-46 2024-03-04T06:18:51.542 INFO:tasks.rados.rados.0.smithi064.stdout:2669: finishing write tid 3 to smithi06431743-14 2024-03-04T06:18:51.557 INFO:tasks.rados.rados.0.smithi064.stdout:2669: finishing write tid 4 to smithi06431743-14 2024-03-04T06:18:51.557 INFO:tasks.rados.rados.0.smithi064.stdout:2669: oid 14 updating version 0 to 806 2024-03-04T06:18:51.557 INFO:tasks.rados.rados.0.smithi064.stdout:2669: oid 14 updating version 806 to 808 2024-03-04T06:18:51.557 INFO:tasks.rados.rados.0.smithi064.stdout:2669: oid 14 version 808 is already newer than 808 2024-03-04T06:18:51.557 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 808 (ObjNum 1221 snap 328 seq_num 1221) dirty exists 2024-03-04T06:18:51.557 INFO:tasks.rados.rados.0.smithi064.stdout:2669: left oid 14 (ObjNum 1221 snap 328 seq_num 1221) 2024-03-04T06:18:51.560 INFO:tasks.rados.rados.0.smithi064.stdout:2671: finishing write tid 2 to smithi06431743-46 2024-03-04T06:18:51.565 INFO:tasks.rados.rados.0.smithi064.stdout:2672: finishing write tid 1 to smithi06431743-30 2024-03-04T06:18:51.575 INFO:tasks.rados.rados.0.smithi064.stdout:2669: done (4 left) 2024-03-04T06:18:51.575 INFO:tasks.rados.rados.0.smithi064.stdout:2670: done (3 left) 2024-03-04T06:18:51.575 INFO:tasks.rados.rados.0.smithi064.stdout:2673: done (2 left) 2024-03-04T06:18:51.576 INFO:tasks.rados.rados.0.smithi064.stdout:2674: snap_create 2024-03-04T06:18:51.581 INFO:tasks.rados.rados.0.smithi064.stdout:2672: finishing write tid 2 to smithi06431743-30 2024-03-04T06:18:51.585 INFO:tasks.rados.rados.0.smithi064.stdout:2671: finishing write tid 3 to smithi06431743-46 2024-03-04T06:18:51.585 INFO:tasks.rados.rados.0.smithi064.stdout:2671: finishing write tid 4 to smithi06431743-46 2024-03-04T06:18:51.585 INFO:tasks.rados.rados.0.smithi064.stdout:2671: oid 46 updating version 0 to 1113 2024-03-04T06:18:51.585 INFO:tasks.rados.rados.0.smithi064.stdout:2671: oid 46 version 1113 is already newer than 1112 2024-03-04T06:18:51.586 INFO:tasks.rados.rados.0.smithi064.stdout:2671: oid 46 version 1113 is already newer than 1111 2024-03-04T06:18:51.586 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 1113 (ObjNum 1222 snap 328 seq_num 1222) dirty exists 2024-03-04T06:18:51.586 INFO:tasks.rados.rados.0.smithi064.stdout:2671: left oid 46 (ObjNum 1222 snap 328 seq_num 1222) 2024-03-04T06:18:51.586 INFO:tasks.rados.rados.0.smithi064.stdout:2672: finishing write tid 3 to smithi06431743-30 2024-03-04T06:18:51.588 INFO:tasks.rados.rados.0.smithi064.stdout:2672: finishing write tid 4 to smithi06431743-30 2024-03-04T06:18:51.588 INFO:tasks.rados.rados.0.smithi064.stdout:2672: oid 30 updating version 0 to 1246 2024-03-04T06:18:51.588 INFO:tasks.rados.rados.0.smithi064.stdout:2672: oid 30 version 1246 is already newer than 1244 2024-03-04T06:18:51.588 INFO:tasks.rados.rados.0.smithi064.stdout:2672: oid 30 version 1246 is already newer than 1246 2024-03-04T06:18:51.589 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 1246 (ObjNum 1223 snap 328 seq_num 1223) dirty exists 2024-03-04T06:18:51.589 INFO:tasks.rados.rados.0.smithi064.stdout:2672: left oid 30 (ObjNum 1223 snap 328 seq_num 1223) 2024-03-04T06:18:52.421 INFO:tasks.rados.rados.0.smithi064.stdout:2671: done (2 left) 2024-03-04T06:18:52.421 INFO:tasks.rados.rados.0.smithi064.stdout:2672: done (1 left) 2024-03-04T06:18:52.421 INFO:tasks.rados.rados.0.smithi064.stdout:2674: done (0 left) 2024-03-04T06:18:52.422 INFO:tasks.rados.rados.0.smithi064.stdout:2675: read oid 40 snap -1 2024-03-04T06:18:52.422 INFO:tasks.rados.rados.0.smithi064.stdout:2675: expect (ObjNum 1158 snap 310 seq_num 1158) 2024-03-04T06:18:52.422 INFO:tasks.rados.rados.0.smithi064.stdout:2676: write oid 3 current snap is 329 2024-03-04T06:18:52.422 INFO:tasks.rados.rados.0.smithi064.stdout:2676: seq_num 1225 ranges {588575=512858,1867383=455083} 2024-03-04T06:18:52.427 INFO:tasks.rados.rados.0.smithi064.stdout:2676: writing smithi06431743-3 from 588575 to 1101433 tid 1 2024-03-04T06:18:52.434 INFO:tasks.rados.rados.0.smithi064.stdout:2676: writing smithi06431743-3 from 1867383 to 2322466 tid 2 2024-03-04T06:18:52.434 INFO:tasks.rados.rados.0.smithi064.stdout:2677: read oid 36 snap -1 2024-03-04T06:18:52.434 INFO:tasks.rados.rados.0.smithi064.stdout:2677: expect deleted 2024-03-04T06:18:52.435 INFO:tasks.rados.rados.0.smithi064.stdout:2678: read oid 23 snap -1 2024-03-04T06:18:52.435 INFO:tasks.rados.rados.0.smithi064.stdout:2678: expect deleted 2024-03-04T06:18:52.435 INFO:tasks.rados.rados.0.smithi064.stdout:2679: read oid 7 snap -1 2024-03-04T06:18:52.435 INFO:tasks.rados.rados.0.smithi064.stdout:2679: expect (ObjNum 1201 snap 319 seq_num 1201) 2024-03-04T06:18:52.435 INFO:tasks.rados.rados.0.smithi064.stdout:2680: snap_create 2024-03-04T06:18:52.463 INFO:tasks.rados.rados.0.smithi064.stdout:2675: expect (ObjNum 1158 snap 310 seq_num 1158) 2024-03-04T06:18:52.602 INFO:tasks.rados.rados.0.smithi064.stdout:2679: expect (ObjNum 1201 snap 319 seq_num 1201) 2024-03-04T06:18:52.713 INFO:tasks.rados.rados.0.smithi064.stdout:2676: finishing write tid 1 to smithi06431743-3 2024-03-04T06:18:52.713 INFO:tasks.rados.rados.0.smithi064.stdout:2676: finishing write tid 2 to smithi06431743-3 2024-03-04T06:18:52.713 INFO:tasks.rados.rados.0.smithi064.stdout:2676: finishing write tid 3 to smithi06431743-3 2024-03-04T06:18:52.713 INFO:tasks.rados.rados.0.smithi064.stdout:2676: finishing write tid 4 to smithi06431743-3 2024-03-04T06:18:52.713 INFO:tasks.rados.rados.0.smithi064.stdout:2676: oid 3 updating version 0 to 656 2024-03-04T06:18:52.713 INFO:tasks.rados.rados.0.smithi064.stdout:2676: oid 3 version 656 is already newer than 655 2024-03-04T06:18:52.714 INFO:tasks.rados.rados.0.smithi064.stdout:2676: oid 3 updating version 656 to 657 2024-03-04T06:18:52.714 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 657 (ObjNum 1225 snap 329 seq_num 1225) dirty exists 2024-03-04T06:18:52.714 INFO:tasks.rados.rados.0.smithi064.stdout:2676: left oid 3 (ObjNum 1225 snap 329 seq_num 1225) 2024-03-04T06:18:53.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:53.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:53.340+0000 7f45e562d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:53.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:53.344+0000 7f45e562d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:53.425 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:18:53.425 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:18:53.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:18:53.428 INFO:tasks.rados.rados.0.smithi064.stdout:2675: done (5 left) 2024-03-04T06:18:53.428 INFO:tasks.rados.rados.0.smithi064.stdout:2676: done (4 left) 2024-03-04T06:18:53.428 INFO:tasks.rados.rados.0.smithi064.stdout:2677: done (3 left) 2024-03-04T06:18:53.428 INFO:tasks.rados.rados.0.smithi064.stdout:2678: done (2 left) 2024-03-04T06:18:53.428 INFO:tasks.rados.rados.0.smithi064.stdout:2679: done (1 left) 2024-03-04T06:18:53.428 INFO:tasks.rados.rados.0.smithi064.stdout:2680: done (0 left) 2024-03-04T06:18:53.428 INFO:tasks.rados.rados.0.smithi064.stdout:2681: read oid 17 snap 325 2024-03-04T06:18:53.429 INFO:tasks.rados.rados.0.smithi064.stdout:2681: expect (ObjNum 1198 snap 319 seq_num 1198) 2024-03-04T06:18:53.429 INFO:tasks.rados.rados.0.smithi064.stdout:2682: write (excl) oid 14 current snap is 330 2024-03-04T06:18:53.429 INFO:tasks.rados.rados.0.smithi064.stdout:2682: seq_num 1226 ranges {605382=588107,1925865=679517} 2024-03-04T06:18:53.433 INFO:tasks.rados.rados.0.smithi064.stdout:2682: writing smithi06431743-14 from 605382 to 1193489 tid 1 2024-03-04T06:18:53.442 INFO:tasks.rados.rados.0.smithi064.stdout:2682: writing smithi06431743-14 from 1925865 to 2605382 tid 2 2024-03-04T06:18:53.442 INFO:tasks.rados.rados.0.smithi064.stdout:2683: read oid 7 snap -1 2024-03-04T06:18:53.442 INFO:tasks.rados.rados.0.smithi064.stdout:2683: expect (ObjNum 1201 snap 319 seq_num 1201) 2024-03-04T06:18:53.443 INFO:tasks.rados.rados.0.smithi064.stdout:2684: delete oid 33 current snap is 330 2024-03-04T06:18:53.460 INFO:tasks.rados.rados.0.smithi064.stdout:2681: expect (ObjNum 1198 snap 319 seq_num 1198) 2024-03-04T06:18:53.550 INFO:tasks.rados.rados.0.smithi064.stdout:2681: done (3 left) 2024-03-04T06:18:53.550 INFO:tasks.rados.rados.0.smithi064.stdout:2684: done (2 left) 2024-03-04T06:18:53.550 INFO:tasks.rados.rados.0.smithi064.stdout:2685: snap_remove snap 315 2024-03-04T06:18:53.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:53.549+0000 7f816e55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:53.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:53.555+0000 7f816e55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:53.743 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:18:53.744 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:18:53.756 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":987,"stamp":"2024-03-04T06:18:53.425751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":107509298,"num_objects":906,"num_object_clones":712,"num_object_copies":2718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":157,"num_read":4991,"num_read_kb":3854951,"num_write":4202,"num_write_kb":1107489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9838,"ondisk_log_size":9838,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":687660,"kb_used_data":491532,"kb_used_omap":0,"kb_used_meta":196096,"kb_avail":376799700,"statfs":{"total":386547056640,"available":385842892800,"internally_reserved":0,"allocated":503328768,"data_stored":595251043,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":200802304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-470530,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":22,"num_read_kb":13431,"num_write":18,"num_write_kb":4639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014754"},"pg_stats":[{"pgid":"3.f","version":"648'560","reported_seq":2271,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.463808+0000","last_change":"2024-03-04T06:18:49.478156+0000","last_active":"2024-03-04T06:18:49.463808+0000","last_peered":"2024-03-04T06:18:49.463808+0000","last_clean":"2024-03-04T06:18:49.463808+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:18:49.463808+0000","last_undegraded":"2024-03-04T06:18:49.463808+0000","last_fullsized":"2024-03-04T06:18:49.463808+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7328380,"num_objects":57,"num_object_clones":44,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":285,"num_read_kb":239693,"num_write":233,"num_write_kb":62308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"652'481","reported_seq":2223,"reported_epoch":658,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:49.453320+0000","last_change":"2024-03-04T06:18:49.453320+0000","last_active":"2024-03-04T06:18:49.453320+0000","last_peered":"2024-03-04T06:18:49.453320+0000","last_clean":"2024-03-04T06:18:49.453320+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:18:49.453320+0000","last_undegraded":"2024-03-04T06:18:49.453320+0000","last_fullsized":"2024-03-04T06:18:49.453320+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":481,"log_dups_size":0,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3210401,"num_objects":47,"num_object_clones":33,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":172,"num_read_kb":121244,"num_write":187,"num_write_kb":49317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"658'629","reported_seq":2380,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.459282+0000","last_change":"2024-03-04T06:18:49.459282+0000","last_active":"2024-03-04T06:18:49.459282+0000","last_peered":"2024-03-04T06:18:49.459282+0000","last_clean":"2024-03-04T06:18:49.459282+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:18:49.459282+0000","last_undegraded":"2024-03-04T06:18:49.459282+0000","last_fullsized":"2024-03-04T06:18:49.459282+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8639656,"num_objects":67,"num_object_clones":52,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":380,"num_read_kb":307024,"num_write":288,"num_write_kb":79007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"658'516","reported_seq":2227,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.563173+0000","last_change":"2024-03-04T06:18:49.565137+0000","last_active":"2024-03-04T06:18:49.563173+0000","last_peered":"2024-03-04T06:18:49.563173+0000","last_clean":"2024-03-04T06:18:49.563173+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:18:49.563173+0000","last_undegraded":"2024-03-04T06:18:49.563173+0000","last_fullsized":"2024-03-04T06:18:49.563173+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5748555,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":228,"num_read_kb":169194,"num_write":211,"num_write_kb":58824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"656'231","reported_seq":2071,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-03-04T06:18:50.442315+0000","last_change":"2024-03-04T06:18:48.316459+0000","last_active":"2024-03-04T06:18:50.442315+0000","last_peered":"2024-03-04T06:18:50.442315+0000","last_clean":"2024-03-04T06:18:50.442315+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:18:50.442315+0000","last_undegraded":"2024-03-04T06:18:50.442315+0000","last_fullsized":"2024-03-04T06:18:50.442315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361664,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":82,"num_read_kb":57507,"num_write":93,"num_write_kb":25364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"658'676","reported_seq":2335,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.461695+0000","last_change":"2024-03-04T06:18:49.520758+0000","last_active":"2024-03-04T06:18:49.461695+0000","last_peered":"2024-03-04T06:18:49.461695+0000","last_clean":"2024-03-04T06:18:49.461695+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:18:49.461695+0000","last_undegraded":"2024-03-04T06:18:49.461695+0000","last_fullsized":"2024-03-04T06:18:49.461695+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10084992,"num_objects":56,"num_object_clones":47,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":329,"num_read_kb":261977,"num_write":283,"num_write_kb":82964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1934,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.462544+0000","last_change":"2024-03-04T06:18:49.472476+0000","last_active":"2024-03-04T06:18:49.462544+0000","last_peered":"2024-03-04T06:18:49.462544+0000","last_clean":"2024-03-04T06:18:49.462544+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:18:49.462544+0000","last_undegraded":"2024-03-04T06:18:49.462544+0000","last_fullsized":"2024-03-04T06:18:49.462544+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"657'374","reported_seq":2171,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.429199+0000","last_change":"2024-03-04T06:18:49.429199+0000","last_active":"2024-03-04T06:18:49.429199+0000","last_peered":"2024-03-04T06:18:49.429199+0000","last_clean":"2024-03-04T06:18:49.429199+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:18:49.429199+0000","last_undegraded":"2024-03-04T06:18:49.429199+0000","last_fullsized":"2024-03-04T06:18:49.429199+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5782169,"num_objects":31,"num_object_clones":22,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":201,"num_read_kb":181320,"num_write":141,"num_write_kb":37633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1782,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-03-04T06:18:50.430997+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:18:50.430997+0000","last_peered":"2024-03-04T06:18:50.430997+0000","last_clean":"2024-03-04T06:18:50.430997+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:18:50.430997+0000","last_undegraded":"2024-03-04T06:18:50.430997+0000","last_fullsized":"2024-03-04T06:18:50.430997+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1650,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.433110+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:18:49.433110+0000","last_peered":"2024-03-04T06:18:49.433110+0000","last_clean":"2024-03-04T06:18:49.433110+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:18:49.433110+0000","last_undegraded":"2024-03-04T06:18:49.433110+0000","last_fullsized":"2024-03-04T06:18:49.433110+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"658'654","reported_seq":2336,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.478976+0000","last_change":"2024-03-04T06:18:49.478976+0000","last_active":"2024-03-04T06:18:49.478976+0000","last_peered":"2024-03-04T06:18:49.478976+0000","last_clean":"2024-03-04T06:18:49.478976+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:18:49.478976+0000","last_undegraded":"2024-03-04T06:18:49.478976+0000","last_fullsized":"2024-03-04T06:18:49.478976+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2958296,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":320,"num_read_kb":254711,"num_write":266,"num_write_kb":71742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1630,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.461277+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:18:49.461277+0000","last_peered":"2024-03-04T06:18:49.461277+0000","last_clean":"2024-03-04T06:18:49.461277+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:18:49.461277+0000","last_undegraded":"2024-03-04T06:18:49.461277+0000","last_fullsized":"2024-03-04T06:18:49.461277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"657'401","reported_seq":2192,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-03-04T06:18:50.434337+0000","last_change":"2024-03-04T06:18:48.353130+0000","last_active":"2024-03-04T06:18:50.434337+0000","last_peered":"2024-03-04T06:18:50.434337+0000","last_clean":"2024-03-04T06:18:50.434337+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:18:50.434337+0000","last_undegraded":"2024-03-04T06:18:50.434337+0000","last_fullsized":"2024-03-04T06:18:50.434337+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":401,"log_dups_size":0,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185207,"num_objects":39,"num_object_clones":31,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":221,"num_read_kb":177503,"num_write":156,"num_write_kb":43696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1650,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.419763+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:18:49.419763+0000","last_peered":"2024-03-04T06:18:49.419763+0000","last_clean":"2024-03-04T06:18:49.419763+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:18:49.419763+0000","last_undegraded":"2024-03-04T06:18:49.419763+0000","last_fullsized":"2024-03-04T06:18:49.419763+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"657'1109","reported_seq":2679,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-03-04T06:18:50.439635+0000","last_change":"2024-03-04T06:18:48.378908+0000","last_active":"2024-03-04T06:18:50.439635+0000","last_peered":"2024-03-04T06:18:50.439635+0000","last_clean":"2024-03-04T06:18:50.439635+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:18:50.439635+0000","last_undegraded":"2024-03-04T06:18:50.439635+0000","last_fullsized":"2024-03-04T06:18:50.439635+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1109,"log_dups_size":0,"ondisk_log_size":1109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11076935,"num_objects":105,"num_object_clones":82,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":630,"num_read_kb":499304,"num_write":492,"num_write_kb":130942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1630,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.455482+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:18:49.455482+0000","last_peered":"2024-03-04T06:18:49.455482+0000","last_clean":"2024-03-04T06:18:49.455482+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:18:49.455482+0000","last_undegraded":"2024-03-04T06:18:49.455482+0000","last_fullsized":"2024-03-04T06:18:49.455482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"658'1236","reported_seq":2661,"reported_epoch":658,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:49.457745+0000","last_change":"2024-03-04T06:18:49.457551+0000","last_active":"2024-03-04T06:18:49.457745+0000","last_peered":"2024-03-04T06:18:49.457745+0000","last_clean":"2024-03-04T06:18:49.457745+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:18:49.457745+0000","last_undegraded":"2024-03-04T06:18:49.457745+0000","last_fullsized":"2024-03-04T06:18:49.457745+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1236,"log_dups_size":0,"ondisk_log_size":1236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12068515,"num_objects":113,"num_object_clones":90,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":505,"num_read_kb":366968,"num_write":516,"num_write_kb":136789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"657'612","reported_seq":2359,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.441777+0000","last_change":"2024-03-04T06:18:49.441777+0000","last_active":"2024-03-04T06:18:49.441777+0000","last_peered":"2024-03-04T06:18:49.441777+0000","last_clean":"2024-03-04T06:18:49.441777+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:18:49.441777+0000","last_undegraded":"2024-03-04T06:18:49.441777+0000","last_fullsized":"2024-03-04T06:18:49.441777+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6067888,"num_objects":55,"num_object_clones":43,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":378,"num_read_kb":317169,"num_write":258,"num_write_kb":70208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1678,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.475960+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:18:49.475960+0000","last_peered":"2024-03-04T06:18:49.475960+0000","last_clean":"2024-03-04T06:18:49.475960+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:18:49.475960+0000","last_undegraded":"2024-03-04T06:18:49.475960+0000","last_fullsized":"2024-03-04T06:18:49.475960+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"652'581","reported_seq":2332,"reported_epoch":658,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:49.468149+0000","last_change":"2024-03-04T06:18:49.468149+0000","last_active":"2024-03-04T06:18:49.468149+0000","last_peered":"2024-03-04T06:18:49.468149+0000","last_clean":"2024-03-04T06:18:49.468149+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:18:49.468149+0000","last_undegraded":"2024-03-04T06:18:49.468149+0000","last_fullsized":"2024-03-04T06:18:49.468149+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":581,"log_dups_size":0,"ondisk_log_size":581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5950239,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":297,"num_read_kb":220157,"num_write":248,"num_write_kb":65752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1649,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-03-04T06:18:50.440896+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:18:50.440896+0000","last_peered":"2024-03-04T06:18:50.440896+0000","last_clean":"2024-03-04T06:18:50.440896+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:18:50.440896+0000","last_undegraded":"2024-03-04T06:18:50.440896+0000","last_fullsized":"2024-03-04T06:18:50.440896+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"659'804","reported_seq":2424,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-03-04T06:18:50.471614+0000","last_change":"2024-03-04T06:18:48.373604+0000","last_active":"2024-03-04T06:18:50.471614+0000","last_peered":"2024-03-04T06:18:50.471614+0000","last_clean":"2024-03-04T06:18:50.471614+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:18:50.471614+0000","last_undegraded":"2024-03-04T06:18:50.471614+0000","last_fullsized":"2024-03-04T06:18:50.471614+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":804,"log_dups_size":0,"ondisk_log_size":804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13446689,"num_objects":73,"num_object_clones":59,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":375,"num_read_kb":272260,"num_write":344,"num_write_kb":95206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1678,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.463668+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:18:49.463668+0000","last_peered":"2024-03-04T06:18:49.463668+0000","last_clean":"2024-03-04T06:18:49.463668+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:18:49.463668+0000","last_undegraded":"2024-03-04T06:18:49.463668+0000","last_fullsized":"2024-03-04T06:18:49.463668+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"652'877","reported_seq":2513,"reported_epoch":658,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:18:49.455939+0000","last_change":"2024-03-04T06:18:49.455939+0000","last_active":"2024-03-04T06:18:49.455939+0000","last_peered":"2024-03-04T06:18:49.455939+0000","last_clean":"2024-03-04T06:18:49.455939+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:18:49.455939+0000","last_undegraded":"2024-03-04T06:18:49.455939+0000","last_fullsized":"2024-03-04T06:18:49.455939+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":877,"log_dups_size":0,"ondisk_log_size":877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9009344,"num_objects":76,"num_object_clones":66,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":485,"num_read_kb":408829,"num_write":349,"num_write_kb":95843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1650,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-03-04T06:18:49.432061+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:18:49.432061+0000","last_peered":"2024-03-04T06:18:49.432061+0000","last_clean":"2024-03-04T06:18:49.432061+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:18:49.432061+0000","last_undegraded":"2024-03-04T06:18:49.432061+0000","last_fullsized":"2024-03-04T06:18:49.432061+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":106918930,"num_objects":904,"num_object_clones":712,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":157,"num_read":4888,"num_read_kb":3854860,"num_write":4065,"num_write_kb":1105595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9741,"ondisk_log_size":9741,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706329,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":205028,"kb_used_data":151836,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94166812,"statfs":{"total":96636764160,"available":96426815488,"internally_reserved":0,"allocated":155480064,"data_stored":184923303,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706329,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":156004,"kb_used_data":106972,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94215836,"statfs":{"total":96636764160,"available":96477016064,"internally_reserved":0,"allocated":109539328,"data_stored":123061724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706328,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":173664,"kb_used_data":124632,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94198176,"statfs":{"total":96636764160,"available":96458932224,"internally_reserved":0,"allocated":127623168,"data_stored":154732995,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706328,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":152964,"kb_used_data":108092,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94218876,"statfs":{"total":96636764160,"available":96480129024,"internally_reserved":0,"allocated":110686208,"data_stored":132533021,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:18:54.429 INFO:tasks.rados.rados.0.smithi064.stdout:2685: done (2 left) 2024-03-04T06:18:54.429 INFO:tasks.rados.rados.0.smithi064.stdout:2686: read oid 37 snap 321 2024-03-04T06:18:54.429 INFO:tasks.rados.rados.0.smithi064.stdout:2686: expect (ObjNum 1143 snap 306 seq_num 1143) 2024-03-04T06:18:54.429 INFO:tasks.rados.rados.0.smithi064.stdout:2687: snap_create 2024-03-04T06:18:54.429 INFO:tasks.rados.rados.0.smithi064.stdout:2682: finishing write tid 1 to smithi06431743-14 2024-03-04T06:18:54.429 INFO:tasks.rados.rados.0.smithi064.stdout:2683: expect (ObjNum 1201 snap 319 seq_num 1201) 2024-03-04T06:18:54.565 INFO:tasks.rados.rados.0.smithi064.stdout:2682: finishing write tid 2 to smithi06431743-14 2024-03-04T06:18:54.565 INFO:tasks.rados.rados.0.smithi064.stdout:2682: finishing write tid 3 to smithi06431743-14 2024-03-04T06:18:54.565 INFO:tasks.rados.rados.0.smithi064.stdout:2682: finishing write tid 4 to smithi06431743-14 2024-03-04T06:18:54.565 INFO:tasks.rados.rados.0.smithi064.stdout:2682: oid 14 updating version 0 to 810 2024-03-04T06:18:54.566 INFO:tasks.rados.rados.0.smithi064.stdout:2682: oid 14 updating version 810 to 811 2024-03-04T06:18:54.566 INFO:tasks.rados.rados.0.smithi064.stdout:2682: oid 14 updating version 811 to 812 2024-03-04T06:18:54.566 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 812 (ObjNum 1226 snap 330 seq_num 1226) dirty exists 2024-03-04T06:18:54.566 INFO:tasks.rados.rados.0.smithi064.stdout:2682: left oid 14 (ObjNum 1226 snap 330 seq_num 1226) 2024-03-04T06:18:54.566 INFO:tasks.rados.rados.0.smithi064.stdout:2686: expect (ObjNum 1143 snap 306 seq_num 1143) 2024-03-04T06:18:55.415 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:55.434 INFO:tasks.rados.rados.0.smithi064.stdout:2682: done (3 left) 2024-03-04T06:18:55.434 INFO:tasks.rados.rados.0.smithi064.stdout:2683: done (2 left) 2024-03-04T06:18:55.434 INFO:tasks.rados.rados.0.smithi064.stdout:2686: done (1 left) 2024-03-04T06:18:55.434 INFO:tasks.rados.rados.0.smithi064.stdout:2687: done (0 left) 2024-03-04T06:18:55.434 INFO:tasks.rados.rados.0.smithi064.stdout:2688: snap_create 2024-03-04T06:18:55.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:55.526+0000 7ff2eeb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:55.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:55.531+0000 7ff2eeb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:56.436 INFO:tasks.rados.rados.0.smithi064.stdout:2688: done (0 left) 2024-03-04T06:18:56.437 INFO:tasks.rados.rados.0.smithi064.stdout:2689: write oid 32 current snap is 332 2024-03-04T06:18:56.437 INFO:tasks.rados.rados.0.smithi064.stdout:2689: seq_num 1228 ranges {638996=738605,1926476=712520} 2024-03-04T06:18:56.444 INFO:tasks.rados.rados.0.smithi064.stdout:2689: writing smithi06431743-32 from 638996 to 1377601 tid 1 2024-03-04T06:18:56.453 INFO:tasks.rados.rados.0.smithi064.stdout:2689: writing smithi06431743-32 from 1926476 to 2638996 tid 2 2024-03-04T06:18:56.453 INFO:tasks.rados.rados.0.smithi064.stdout:2690: write oid 8 current snap is 332 2024-03-04T06:18:56.453 INFO:tasks.rados.rados.0.smithi064.stdout:2690: seq_num 1229 ranges {655803=413854,1584958=462421} 2024-03-04T06:18:56.459 INFO:tasks.rados.rados.0.smithi064.stdout:2690: writing smithi06431743-8 from 655803 to 1069657 tid 1 2024-03-04T06:18:56.465 INFO:tasks.rados.rados.0.smithi064.stdout:2690: writing smithi06431743-8 from 1584958 to 2047379 tid 2 2024-03-04T06:18:56.465 INFO:tasks.rados.rados.0.smithi064.stdout:2691: write (excl) oid 35 current snap is 332 2024-03-04T06:18:56.465 INFO:tasks.rados.rados.0.smithi064.stdout:2691: seq_num 1230 ranges {672610=489103,1643440=722432} 2024-03-04T06:18:56.471 INFO:tasks.rados.rados.0.smithi064.stdout:2691: writing smithi06431743-35 from 672610 to 1161713 tid 1 2024-03-04T06:18:56.478 INFO:tasks.rados.rados.0.smithi064.stdout:2691: writing smithi06431743-35 from 1643440 to 2365872 tid 2 2024-03-04T06:18:56.478 INFO:tasks.rados.rados.0.smithi064.stdout:2692: write oid 29 current snap is 332 2024-03-04T06:18:56.478 INFO:tasks.rados.rados.0.smithi064.stdout:2692: seq_num 1231 ranges {689417=564352,1701922=466090} 2024-03-04T06:18:56.485 INFO:tasks.rados.rados.0.smithi064.stdout:2692: writing smithi06431743-29 from 689417 to 1253769 tid 1 2024-03-04T06:18:56.491 INFO:tasks.rados.rados.0.smithi064.stdout:2692: writing smithi06431743-29 from 1701922 to 2168012 tid 2 2024-03-04T06:18:56.491 INFO:tasks.rados.rados.0.smithi064.stdout:2693: delete oid 22 current snap is 332 2024-03-04T06:18:56.517 INFO:tasks.rados.rados.0.smithi064.stdout:2691: finishing write tid 1 to smithi06431743-35 2024-03-04T06:18:56.531 INFO:tasks.rados.rados.0.smithi064.stdout:2693: done (4 left) 2024-03-04T06:18:56.531 INFO:tasks.rados.rados.0.smithi064.stdout:2694: read oid 2 snap -1 2024-03-04T06:18:56.531 INFO:tasks.rados.rados.0.smithi064.stdout:2694: expect (ObjNum 1147 snap 309 seq_num 1147) 2024-03-04T06:18:56.531 INFO:tasks.rados.rados.0.smithi064.stdout:2695: snap_create 2024-03-04T06:18:56.538 INFO:tasks.rados.rados.0.smithi064.stdout:2691: finishing write tid 2 to smithi06431743-35 2024-03-04T06:18:56.538 INFO:tasks.rados.rados.0.smithi064.stdout:2692: finishing write tid 1 to smithi06431743-29 2024-03-04T06:18:56.538 INFO:tasks.rados.rados.0.smithi064.stdout:2689: finishing write tid 1 to smithi06431743-32 2024-03-04T06:18:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:2690: finishing write tid 1 to smithi06431743-8 2024-03-04T06:18:56.558 INFO:tasks.rados.rados.0.smithi064.stdout:2689: finishing write tid 2 to smithi06431743-32 2024-03-04T06:18:56.561 INFO:tasks.rados.rados.0.smithi064.stdout:2692: finishing write tid 2 to smithi06431743-29 2024-03-04T06:18:56.566 INFO:tasks.rados.rados.0.smithi064.stdout:2690: finishing write tid 2 to smithi06431743-8 2024-03-04T06:18:56.568 INFO:tasks.rados.rados.0.smithi064.stdout:2691: finishing write tid 3 to smithi06431743-35 2024-03-04T06:18:56.569 INFO:tasks.rados.rados.0.smithi064.stdout:2691: finishing write tid 4 to smithi06431743-35 2024-03-04T06:18:56.569 INFO:tasks.rados.rados.0.smithi064.stdout:2691: oid 35 updating version 0 to 819 2024-03-04T06:18:56.569 INFO:tasks.rados.rados.0.smithi064.stdout:2691: oid 35 version 819 is already newer than 818 2024-03-04T06:18:56.569 INFO:tasks.rados.rados.0.smithi064.stdout:2691: oid 35 updating version 819 to 820 2024-03-04T06:18:56.569 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 820 (ObjNum 1230 snap 332 seq_num 1230) dirty exists 2024-03-04T06:18:56.570 INFO:tasks.rados.rados.0.smithi064.stdout:2691: left oid 35 (ObjNum 1230 snap 332 seq_num 1230) 2024-03-04T06:18:56.572 INFO:tasks.rados.rados.0.smithi064.stdout:2689: finishing write tid 3 to smithi06431743-32 2024-03-04T06:18:56.572 INFO:tasks.rados.rados.0.smithi064.stdout:2689: finishing write tid 4 to smithi06431743-32 2024-03-04T06:18:56.572 INFO:tasks.rados.rados.0.smithi064.stdout:2689: oid 32 updating version 0 to 1123 2024-03-04T06:18:56.572 INFO:tasks.rados.rados.0.smithi064.stdout:2689: oid 32 updating version 1123 to 1124 2024-03-04T06:18:56.572 INFO:tasks.rados.rados.0.smithi064.stdout:2689: oid 32 updating version 1124 to 1125 2024-03-04T06:18:56.573 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1125 (ObjNum 1228 snap 332 seq_num 1228) dirty exists 2024-03-04T06:18:56.573 INFO:tasks.rados.rados.0.smithi064.stdout:2689: left oid 32 (ObjNum 1228 snap 332 seq_num 1228) 2024-03-04T06:18:56.575 INFO:tasks.rados.rados.0.smithi064.stdout:2692: finishing write tid 3 to smithi06431743-29 2024-03-04T06:18:56.576 INFO:tasks.rados.rados.0.smithi064.stdout:2692: finishing write tid 4 to smithi06431743-29 2024-03-04T06:18:56.576 INFO:tasks.rados.rados.0.smithi064.stdout:2692: oid 29 updating version 0 to 1261 2024-03-04T06:18:56.576 INFO:tasks.rados.rados.0.smithi064.stdout:2692: oid 29 version 1261 is already newer than 1260 2024-03-04T06:18:56.576 INFO:tasks.rados.rados.0.smithi064.stdout:2692: oid 29 version 1261 is already newer than 1260 2024-03-04T06:18:56.577 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 1261 (ObjNum 1231 snap 332 seq_num 1231) dirty exists 2024-03-04T06:18:56.577 INFO:tasks.rados.rados.0.smithi064.stdout:2692: left oid 29 (ObjNum 1231 snap 332 seq_num 1231) 2024-03-04T06:18:56.577 INFO:tasks.rados.rados.0.smithi064.stdout:2690: finishing write tid 3 to smithi06431743-8 2024-03-04T06:18:56.578 INFO:tasks.rados.rados.0.smithi064.stdout:2690: finishing write tid 4 to smithi06431743-8 2024-03-04T06:18:56.578 INFO:tasks.rados.rados.0.smithi064.stdout:2690: oid 8 updating version 0 to 596 2024-03-04T06:18:56.578 INFO:tasks.rados.rados.0.smithi064.stdout:2690: oid 8 updating version 596 to 598 2024-03-04T06:18:56.578 INFO:tasks.rados.rados.0.smithi064.stdout:2690: oid 8 version 598 is already newer than 597 2024-03-04T06:18:56.578 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 598 (ObjNum 1229 snap 332 seq_num 1229) dirty exists 2024-03-04T06:18:56.578 INFO:tasks.rados.rados.0.smithi064.stdout:2690: left oid 8 (ObjNum 1229 snap 332 seq_num 1229) 2024-03-04T06:18:56.585 INFO:tasks.rados.rados.0.smithi064.stdout:2694: expect (ObjNum 1147 snap 309 seq_num 1147) 2024-03-04T06:18:57.441 INFO:tasks.rados.rados.0.smithi064.stdout:2689: done (5 left) 2024-03-04T06:18:57.441 INFO:tasks.rados.rados.0.smithi064.stdout:2690: done (4 left) 2024-03-04T06:18:57.441 INFO:tasks.rados.rados.0.smithi064.stdout:2691: done (3 left) 2024-03-04T06:18:57.441 INFO:tasks.rados.rados.0.smithi064.stdout:2692: done (2 left) 2024-03-04T06:18:57.441 INFO:tasks.rados.rados.0.smithi064.stdout:2694: done (1 left) 2024-03-04T06:18:57.441 INFO:tasks.rados.rados.0.smithi064.stdout:2695: done (0 left) 2024-03-04T06:18:57.441 INFO:tasks.rados.rados.0.smithi064.stdout:2696: delete oid 32 current snap is 333 2024-03-04T06:18:57.492 INFO:tasks.rados.rados.0.smithi064.stdout:2696: done (0 left) 2024-03-04T06:18:57.492 INFO:tasks.rados.rados.0.smithi064.stdout:2697: write (excl) oid 36 current snap is 333 2024-03-04T06:18:57.492 INFO:tasks.rados.rados.0.smithi064.stdout:2697: seq_num 1234 ranges {739838=506452,1877368=613417} 2024-03-04T06:18:57.499 INFO:tasks.rados.rados.0.smithi064.stdout:2697: writing smithi06431743-36 from 739838 to 1246290 tid 1 2024-03-04T06:18:57.506 INFO:tasks.rados.rados.0.smithi064.stdout:2697: writing smithi06431743-36 from 1877368 to 2490785 tid 2 2024-03-04T06:18:57.506 INFO:tasks.rados.rados.0.smithi064.stdout:2698: read oid 34 snap -1 2024-03-04T06:18:57.506 INFO:tasks.rados.rados.0.smithi064.stdout:2698: expect deleted 2024-03-04T06:18:57.506 INFO:tasks.rados.rados.0.smithi064.stdout:2699: write oid 6 current snap is 333 2024-03-04T06:18:57.506 INFO:tasks.rados.rados.0.smithi064.stdout:2699: seq_num 1235 ranges {756645=581701,1935850=473428} 2024-03-04T06:18:57.513 INFO:tasks.rados.rados.0.smithi064.stdout:2699: writing smithi06431743-6 from 756645 to 1338346 tid 1 2024-03-04T06:18:57.519 INFO:tasks.rados.rados.0.smithi064.stdout:2699: writing smithi06431743-6 from 1935850 to 2409278 tid 2 2024-03-04T06:18:57.519 INFO:tasks.rados.rados.0.smithi064.stdout:2700: delete oid 30 current snap is 333 2024-03-04T06:18:57.552 INFO:tasks.rados.rados.0.smithi064.stdout:2698: done (3 left) 2024-03-04T06:18:57.552 INFO:tasks.rados.rados.0.smithi064.stdout:2700: done (2 left) 2024-03-04T06:18:57.552 INFO:tasks.rados.rados.0.smithi064.stdout:2701: write oid 40 current snap is 333 2024-03-04T06:18:57.553 INFO:tasks.rados.rados.0.smithi064.stdout:2701: seq_num 1237 ranges {790259=732199,1936461=477097} 2024-03-04T06:18:57.560 INFO:tasks.rados.rados.0.smithi064.stdout:2701: writing smithi06431743-40 from 790259 to 1522458 tid 1 2024-03-04T06:18:57.566 INFO:tasks.rados.rados.0.smithi064.stdout:2701: writing smithi06431743-40 from 1936461 to 2413558 tid 2 2024-03-04T06:18:57.566 INFO:tasks.rados.rados.0.smithi064.stdout:2702: delete oid 15 current snap is 333 2024-03-04T06:18:57.566 INFO:tasks.rados.rados.0.smithi064.stdout:2697: finishing write tid 1 to smithi06431743-36 2024-03-04T06:18:57.566 INFO:tasks.rados.rados.0.smithi064.stdout:2699: finishing write tid 1 to smithi06431743-6 2024-03-04T06:18:57.571 INFO:tasks.rados.rados.0.smithi064.stdout:2697: finishing write tid 2 to smithi06431743-36 2024-03-04T06:18:57.572 INFO:tasks.rados.rados.0.smithi064.stdout:2697: finishing write tid 3 to smithi06431743-36 2024-03-04T06:18:57.573 INFO:tasks.rados.rados.0.smithi064.stdout:2697: finishing write tid 4 to smithi06431743-36 2024-03-04T06:18:57.573 INFO:tasks.rados.rados.0.smithi064.stdout:2697: oid 36 updating version 0 to 599 2024-03-04T06:18:57.573 INFO:tasks.rados.rados.0.smithi064.stdout:2697: oid 36 updating version 599 to 601 2024-03-04T06:18:57.574 INFO:tasks.rados.rados.0.smithi064.stdout:2697: oid 36 version 601 is already newer than 601 2024-03-04T06:18:57.574 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 601 (ObjNum 1234 snap 333 seq_num 1234) dirty exists 2024-03-04T06:18:57.574 INFO:tasks.rados.rados.0.smithi064.stdout:2697: left oid 36 (ObjNum 1234 snap 333 seq_num 1234) 2024-03-04T06:18:57.586 INFO:tasks.rados.rados.0.smithi064.stdout:2699: finishing write tid 2 to smithi06431743-6 2024-03-04T06:18:57.596 INFO:tasks.rados.rados.0.smithi064.stdout:2699: finishing write tid 3 to smithi06431743-6 2024-03-04T06:18:57.596 INFO:tasks.rados.rados.0.smithi064.stdout:2701: finishing write tid 1 to smithi06431743-40 2024-03-04T06:18:57.598 INFO:tasks.rados.rados.0.smithi064.stdout:2699: finishing write tid 4 to smithi06431743-6 2024-03-04T06:18:57.598 INFO:tasks.rados.rados.0.smithi064.stdout:2699: oid 6 updating version 0 to 617 2024-03-04T06:18:57.598 INFO:tasks.rados.rados.0.smithi064.stdout:2699: oid 6 updating version 617 to 619 2024-03-04T06:18:57.598 INFO:tasks.rados.rados.0.smithi064.stdout:2699: oid 6 version 619 is already newer than 618 2024-03-04T06:18:57.598 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 619 (ObjNum 1235 snap 333 seq_num 1235) dirty exists 2024-03-04T06:18:57.598 INFO:tasks.rados.rados.0.smithi064.stdout:2699: left oid 6 (ObjNum 1235 snap 333 seq_num 1235) 2024-03-04T06:18:57.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:18:57.621 INFO:tasks.rados.rados.0.smithi064.stdout:2701: finishing write tid 2 to smithi06431743-40 2024-03-04T06:18:57.626 INFO:tasks.rados.rados.0.smithi064.stdout:2697: done (3 left) 2024-03-04T06:18:57.626 INFO:tasks.rados.rados.0.smithi064.stdout:2699: done (2 left) 2024-03-04T06:18:57.626 INFO:tasks.rados.rados.0.smithi064.stdout:2702: done (1 left) 2024-03-04T06:18:57.626 INFO:tasks.rados.rados.0.smithi064.stdout:2703: delete oid 29 current snap is 333 2024-03-04T06:18:57.635 INFO:tasks.rados.rados.0.smithi064.stdout:2701: finishing write tid 3 to smithi06431743-40 2024-03-04T06:18:57.635 INFO:tasks.rados.rados.0.smithi064.stdout:2701: finishing write tid 4 to smithi06431743-40 2024-03-04T06:18:57.635 INFO:tasks.rados.rados.0.smithi064.stdout:2701: oid 40 updating version 0 to 564 2024-03-04T06:18:57.635 INFO:tasks.rados.rados.0.smithi064.stdout:2701: oid 40 version 564 is already newer than 563 2024-03-04T06:18:57.635 INFO:tasks.rados.rados.0.smithi064.stdout:2701: oid 40 version 564 is already newer than 562 2024-03-04T06:18:57.635 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 564 (ObjNum 1237 snap 333 seq_num 1237) dirty exists 2024-03-04T06:18:57.635 INFO:tasks.rados.rados.0.smithi064.stdout:2701: left oid 40 (ObjNum 1237 snap 333 seq_num 1237) 2024-03-04T06:18:57.643 INFO:tasks.rados.rados.0.smithi064.stdout:2701: done (1 left) 2024-03-04T06:18:57.643 INFO:tasks.rados.rados.0.smithi064.stdout:2703: done (0 left) 2024-03-04T06:18:57.643 INFO:tasks.rados.rados.0.smithi064.stdout:2704: write oid 37 current snap is 333 2024-03-04T06:18:57.643 INFO:tasks.rados.rados.0.smithi064.stdout:2704: seq_num 1240 ranges {440680=674299,1711907=624424} 2024-03-04T06:18:57.649 INFO:tasks.rados.rados.0.smithi064.stdout:2704: writing smithi06431743-37 from 440680 to 1114979 tid 1 2024-03-04T06:18:57.656 INFO:tasks.rados.rados.0.smithi064.stdout:2704: writing smithi06431743-37 from 1711907 to 2336331 tid 2 2024-03-04T06:18:57.656 INFO:tasks.rados.rados.0.smithi064.stdout:2705: delete oid 18 current snap is 333 2024-03-04T06:18:57.675 INFO:tasks.rados.rados.0.smithi064.stdout:2704: finishing write tid 1 to smithi06431743-37 2024-03-04T06:18:57.688 INFO:tasks.rados.rados.0.smithi064.stdout:2704: finishing write tid 2 to smithi06431743-37 2024-03-04T06:18:57.694 INFO:tasks.rados.rados.0.smithi064.stdout:2704: finishing write tid 3 to smithi06431743-37 2024-03-04T06:18:57.696 INFO:tasks.rados.rados.0.smithi064.stdout:2704: finishing write tid 4 to smithi06431743-37 2024-03-04T06:18:57.696 INFO:tasks.rados.rados.0.smithi064.stdout:2704: oid 37 updating version 0 to 493 2024-03-04T06:18:57.696 INFO:tasks.rados.rados.0.smithi064.stdout:2704: oid 37 version 493 is already newer than 492 2024-03-04T06:18:57.696 INFO:tasks.rados.rados.0.smithi064.stdout:2704: oid 37 version 493 is already newer than 491 2024-03-04T06:18:57.696 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 493 (ObjNum 1240 snap 333 seq_num 1240) dirty exists 2024-03-04T06:18:57.696 INFO:tasks.rados.rados.0.smithi064.stdout:2704: left oid 37 (ObjNum 1240 snap 333 seq_num 1240) 2024-03-04T06:18:57.697 INFO:tasks.rados.rados.0.smithi064.stdout:2704: done (1 left) 2024-03-04T06:18:57.697 INFO:tasks.rados.rados.0.smithi064.stdout:2705: done (0 left) 2024-03-04T06:18:57.697 INFO:tasks.rados.rados.0.smithi064.stdout:2706: write (excl) oid 43 current snap is 333 2024-03-04T06:18:57.697 INFO:tasks.rados.rados.0.smithi064.stdout:2706: seq_num 1242 ranges {474294=424797,1428871=628093,2657357=216937} 2024-03-04T06:18:57.701 INFO:tasks.rados.rados.0.smithi064.stdout:2706: writing smithi06431743-43 from 474294 to 899091 tid 1 2024-03-04T06:18:57.707 INFO:tasks.rados.rados.0.smithi064.stdout:2706: writing smithi06431743-43 from 1428871 to 2056964 tid 2 2024-03-04T06:18:57.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:57.709+0000 7f3677ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:57.713 INFO:tasks.rados.rados.0.smithi064.stdout:2706: writing smithi06431743-43 from 2657357 to 2874294 tid 3 2024-03-04T06:18:57.713 INFO:tasks.rados.rados.0.smithi064.stdout:2707: snap_create 2024-03-04T06:18:57.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:57.714+0000 7f3677ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:57.718 INFO:tasks.rados.rados.0.smithi064.stdout:2706: finishing write tid 1 to smithi06431743-43 2024-03-04T06:18:57.737 INFO:tasks.rados.rados.0.smithi064.stdout:2706: finishing write tid 2 to smithi06431743-43 2024-03-04T06:18:57.740 INFO:tasks.rados.rados.0.smithi064.stdout:2706: finishing write tid 3 to smithi06431743-43 2024-03-04T06:18:57.747 INFO:tasks.rados.rados.0.smithi064.stdout:2706: finishing write tid 4 to smithi06431743-43 2024-03-04T06:18:57.747 INFO:tasks.rados.rados.0.smithi064.stdout:2706: finishing write tid 5 to smithi06431743-43 2024-03-04T06:18:57.747 INFO:tasks.rados.rados.0.smithi064.stdout:2706: oid 43 updating version 0 to 568 2024-03-04T06:18:57.747 INFO:tasks.rados.rados.0.smithi064.stdout:2706: oid 43 version 568 is already newer than 567 2024-03-04T06:18:57.748 INFO:tasks.rados.rados.0.smithi064.stdout:2706: oid 43 version 568 is already newer than 567 2024-03-04T06:18:57.748 INFO:tasks.rados.rados.0.smithi064.stdout:2706: oid 43 version 568 is already newer than 565 2024-03-04T06:18:57.748 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 568 (ObjNum 1242 snap 333 seq_num 1242) dirty exists 2024-03-04T06:18:57.748 INFO:tasks.rados.rados.0.smithi064.stdout:2706: left oid 43 (ObjNum 1242 snap 333 seq_num 1242) 2024-03-04T06:18:58.286 INFO:tasks.rados.rados.0.smithi064.stdout:2706: done (1 left) 2024-03-04T06:18:58.286 INFO:tasks.rados.rados.0.smithi064.stdout:2707: done (0 left) 2024-03-04T06:18:58.286 INFO:tasks.rados.rados.0.smithi064.stdout:2708: snap_remove snap 283 2024-03-04T06:18:58.758 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:18:58.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:18:58.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:58.873+0000 7f7e854dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:58.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:58.881+0000 7f7e854dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:59.287 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:18:59.292 INFO:tasks.rados.rados.0.smithi064.stdout:2708: done (0 left) 2024-03-04T06:18:59.292 INFO:tasks.rados.rados.0.smithi064.stdout:2709: delete oid 5 current snap is 334 2024-03-04T06:18:59.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:18:59.381 INFO:tasks.rados.rados.0.smithi064.stdout:2709: done (0 left) 2024-03-04T06:18:59.381 INFO:tasks.rados.rados.0.smithi064.stdout:2710: snap_remove snap 324 2024-03-04T06:18:59.425 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:59.419+0000 7f5a7d6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:59.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:59.427+0000 7f5a7d6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:59.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:18:59.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:59.893+0000 7fa3a5664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:18:59.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:18:59.897+0000 7fa3a5664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:00.288 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:19:00.292 INFO:tasks.rados.rados.0.smithi064.stdout:2710: done (0 left) 2024-03-04T06:19:00.292 INFO:tasks.rados.rados.0.smithi064.stdout:2711: write (excl) oid 2 current snap is 334 2024-03-04T06:19:00.292 INFO:tasks.rados.rados.0.smithi064.stdout:2711: seq_num 1244 ranges {507908=575295,1829482=631762} 2024-03-04T06:19:00.298 INFO:tasks.rados.rados.0.smithi064.stdout:2711: writing smithi06431743-2 from 507908 to 1083203 tid 1 2024-03-04T06:19:00.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:19:00.307 INFO:tasks.rados.rados.0.smithi064.stdout:2711: writing smithi06431743-2 from 1829482 to 2461244 tid 2 2024-03-04T06:19:00.307 INFO:tasks.rados.rados.0.smithi064.stdout:2712: write oid 43 current snap is 334 2024-03-04T06:19:00.307 INFO:tasks.rados.rados.0.smithi064.stdout:2712: seq_num 1245 ranges {524715=650544,1887964=491773} 2024-03-04T06:19:00.313 INFO:tasks.rados.rados.0.smithi064.stdout:2712: writing smithi06431743-43 from 524715 to 1175259 tid 1 2024-03-04T06:19:00.320 INFO:tasks.rados.rados.0.smithi064.stdout:2712: writing smithi06431743-43 from 1887964 to 2379737 tid 2 2024-03-04T06:19:00.321 INFO:tasks.rados.rados.0.smithi064.stdout:2713: read oid 41 snap -1 2024-03-04T06:19:00.321 INFO:tasks.rados.rados.0.smithi064.stdout:2713: expect (ObjNum 1128 snap 299 seq_num 1128) 2024-03-04T06:19:00.321 INFO:tasks.rados.rados.0.smithi064.stdout:2714: snap_create 2024-03-04T06:19:00.440 INFO:tasks.rados.rados.0.smithi064.stdout:2713: expect (ObjNum 1128 snap 299 seq_num 1128) 2024-03-04T06:19:00.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:00.438+0000 7f485022d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:00.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:00.450+0000 7f485022d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:00.643 INFO:tasks.rados.rados.0.smithi064.stdout:2711: finishing write tid 1 to smithi06431743-2 2024-03-04T06:19:00.643 INFO:tasks.rados.rados.0.smithi064.stdout:2712: finishing write tid 1 to smithi06431743-43 2024-03-04T06:19:00.643 INFO:tasks.rados.rados.0.smithi064.stdout:2712: finishing write tid 2 to smithi06431743-43 2024-03-04T06:19:00.643 INFO:tasks.rados.rados.0.smithi064.stdout:2711: finishing write tid 2 to smithi06431743-2 2024-03-04T06:19:00.643 INFO:tasks.rados.rados.0.smithi064.stdout:2712: finishing write tid 3 to smithi06431743-43 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:2712: finishing write tid 4 to smithi06431743-43 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:2712: oid 43 updating version 0 to 570 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:2712: oid 43 updating version 570 to 572 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:2712: oid 43 version 572 is already newer than 571 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 572 (ObjNum 1245 snap 334 seq_num 1245) dirty exists 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:2712: left oid 43 (ObjNum 1245 snap 334 seq_num 1245) 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:2711: finishing write tid 3 to smithi06431743-2 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:2711: finishing write tid 4 to smithi06431743-2 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:2711: oid 2 updating version 0 to 528 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:2711: oid 2 version 528 is already newer than 526 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:2711: oid 2 version 528 is already newer than 527 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 528 (ObjNum 1244 snap 334 seq_num 1244) dirty exists 2024-03-04T06:19:00.644 INFO:tasks.rados.rados.0.smithi064.stdout:2711: left oid 2 (ObjNum 1244 snap 334 seq_num 1244) 2024-03-04T06:19:01.289 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:19:01.292 INFO:tasks.rados.rados.0.smithi064.stdout:2711: done (3 left) 2024-03-04T06:19:01.292 INFO:tasks.rados.rados.0.smithi064.stdout:2712: done (2 left) 2024-03-04T06:19:01.292 INFO:tasks.rados.rados.0.smithi064.stdout:2713: done (1 left) 2024-03-04T06:19:01.292 INFO:tasks.rados.rados.0.smithi064.stdout:2714: done (0 left) 2024-03-04T06:19:01.292 INFO:tasks.rados.rados.0.smithi064.stdout:2715: write (excl) oid 31 current snap is 335 2024-03-04T06:19:01.292 INFO:tasks.rados.rados.0.smithi064.stdout:2715: seq_num 1246 ranges {541522=725793,1946446=635431} 2024-03-04T06:19:01.299 INFO:tasks.rados.rados.0.smithi064.stdout:2715: writing smithi06431743-31 from 541522 to 1267315 tid 1 2024-03-04T06:19:01.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:19:01.306 INFO:tasks.rados.rados.0.smithi064.stdout:2715: writing smithi06431743-31 from 1946446 to 2581877 tid 2 2024-03-04T06:19:01.306 INFO:tasks.rados.rados.0.smithi064.stdout:2716: write (excl) oid 1 current snap is 335 2024-03-04T06:19:01.306 INFO:tasks.rados.rados.0.smithi064.stdout:2716: seq_num 1247 ranges {558329=517395,1721281=779089} 2024-03-04T06:19:01.312 INFO:tasks.rados.rados.0.smithi064.stdout:2716: writing smithi06431743-1 from 558329 to 1075724 tid 1 2024-03-04T06:19:01.320 INFO:tasks.rados.rados.0.smithi064.stdout:2716: writing smithi06431743-1 from 1721281 to 2500370 tid 2 2024-03-04T06:19:01.320 INFO:tasks.rados.rados.0.smithi064.stdout:2717: snap_create 2024-03-04T06:19:01.366 INFO:tasks.rados.rados.0.smithi064.stdout:2716: finishing write tid 1 to smithi06431743-1 2024-03-04T06:19:01.370 INFO:tasks.rados.rados.0.smithi064.stdout:2715: finishing write tid 1 to smithi06431743-31 2024-03-04T06:19:01.384 INFO:tasks.rados.rados.0.smithi064.stdout:2716: finishing write tid 2 to smithi06431743-1 2024-03-04T06:19:01.386 INFO:tasks.rados.rados.0.smithi064.stdout:2715: finishing write tid 2 to smithi06431743-31 2024-03-04T06:19:01.398 INFO:tasks.rados.rados.0.smithi064.stdout:2716: finishing write tid 3 to smithi06431743-1 2024-03-04T06:19:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:2715: finishing write tid 3 to smithi06431743-31 2024-03-04T06:19:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:2715: finishing write tid 4 to smithi06431743-31 2024-03-04T06:19:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:2715: oid 31 updating version 0 to 234 2024-03-04T06:19:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:2715: oid 31 updating version 234 to 235 2024-03-04T06:19:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:2715: oid 31 version 235 is already newer than 233 2024-03-04T06:19:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 235 (ObjNum 1246 snap 335 seq_num 1246) dirty exists 2024-03-04T06:19:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:2715: left oid 31 (ObjNum 1246 snap 335 seq_num 1246) 2024-03-04T06:19:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:2716: finishing write tid 4 to smithi06431743-1 2024-03-04T06:19:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:2716: oid 1 updating version 0 to 1138 2024-03-04T06:19:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:2716: oid 1 updating version 1138 to 1140 2024-03-04T06:19:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:2716: oid 1 version 1140 is already newer than 1139 2024-03-04T06:19:01.402 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 1140 (ObjNum 1247 snap 335 seq_num 1247) dirty exists 2024-03-04T06:19:01.402 INFO:tasks.rados.rados.0.smithi064.stdout:2716: left oid 1 (ObjNum 1247 snap 335 seq_num 1247) 2024-03-04T06:19:01.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:01.421+0000 7f5c6219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:01.430 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:01.429+0000 7f5c6219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:01.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:02.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:02.071+0000 7f08f4d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:02.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:02.075+0000 7f08f4d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:02.289 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:19:02.294 INFO:tasks.rados.rados.0.smithi064.stdout:2715: done (2 left) 2024-03-04T06:19:02.294 INFO:tasks.rados.rados.0.smithi064.stdout:2716: done (1 left) 2024-03-04T06:19:02.294 INFO:tasks.rados.rados.0.smithi064.stdout:2717: done (0 left) 2024-03-04T06:19:02.295 INFO:tasks.rados.rados.0.smithi064.stdout:2718: write oid 35 current snap is 336 2024-03-04T06:19:02.295 INFO:tasks.rados.rados.0.smithi064.stdout:2718: seq_num 1248 ranges {575136=592644,1663410=639100,2705542=269594} 2024-03-04T06:19:02.300 INFO:tasks.rados.rados.0.smithi064.stdout:2718: writing smithi06431743-35 from 575136 to 1167780 tid 1 2024-03-04T06:19:02.305 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:19:02.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:19:02.308 INFO:tasks.rados.rados.0.smithi064.stdout:2718: writing smithi06431743-35 from 1663410 to 2302510 tid 2 2024-03-04T06:19:02.312 INFO:tasks.rados.rados.0.smithi064.stdout:2718: writing smithi06431743-35 from 2705542 to 2975136 tid 3 2024-03-04T06:19:02.312 INFO:tasks.rados.rados.0.smithi064.stdout:2719: snap_remove snap 330 2024-03-04T06:19:02.438 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:02.432+0000 7f8f4a20f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:02.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:02.440+0000 7f8f4a20f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:19:02.644 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:19:02.654 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1000,"stamp":"2024-03-04T06:19:02.292651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":104014102,"num_objects":921,"num_object_clones":727,"num_object_copies":2763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":156,"num_read":5058,"num_read_kb":3900908,"num_write":4273,"num_write_kb":1125914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10027,"ondisk_log_size":10027,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":695224,"kb_used_data":499096,"kb_used_omap":0,"kb_used_meta":196096,"kb_avail":376792136,"statfs":{"total":386547056640,"available":385835147264,"internally_reserved":0,"allocated":511074304,"data_stored":601647509,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":200802304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2622651,"num_objects":9,"num_object_clones":9,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":28,"num_read_kb":16499,"num_write":49,"num_write_kb":12883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.007782"},"pg_stats":[{"pgid":"3.f","version":"666'568","reported_seq":2311,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.375208+0000","last_change":"2024-03-04T06:19:00.375208+0000","last_active":"2024-03-04T06:19:00.375208+0000","last_peered":"2024-03-04T06:19:00.375208+0000","last_clean":"2024-03-04T06:19:00.375208+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:19:00.375208+0000","last_undegraded":"2024-03-04T06:19:00.375208+0000","last_fullsized":"2024-03-04T06:19:00.375208+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9530427,"num_objects":58,"num_object_clones":45,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":294,"num_read_kb":245917,"num_write":242,"num_write_kb":64731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"668'497","reported_seq":2261,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.315195+0000","last_change":"2024-03-04T06:18:59.391827+0000","last_active":"2024-03-04T06:18:59.315195+0000","last_peered":"2024-03-04T06:18:59.315195+0000","last_clean":"2024-03-04T06:18:59.315195+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:18:59.315195+0000","last_undegraded":"2024-03-04T06:18:59.315195+0000","last_fullsized":"2024-03-04T06:18:59.315195+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2840680,"num_objects":48,"num_object_clones":34,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":180,"num_read_kb":130653,"num_write":191,"num_write_kb":50586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"663'635","reported_seq":2406,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.352838+0000","last_change":"2024-03-04T06:19:00.352838+0000","last_active":"2024-03-04T06:19:00.352838+0000","last_peered":"2024-03-04T06:19:00.352838+0000","last_clean":"2024-03-04T06:19:00.352838+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:19:00.352838+0000","last_undegraded":"2024-03-04T06:19:00.352838+0000","last_fullsized":"2024-03-04T06:19:00.352838+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8639656,"num_objects":67,"num_object_clones":52,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":380,"num_read_kb":307024,"num_write":288,"num_write_kb":79007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"663'520","reported_seq":2258,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.328266+0000","last_change":"2024-03-04T06:19:00.328266+0000","last_active":"2024-03-04T06:19:00.328266+0000","last_peered":"2024-03-04T06:19:00.328266+0000","last_clean":"2024-03-04T06:19:00.328266+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:19:00.328266+0000","last_undegraded":"2024-03-04T06:19:00.328266+0000","last_fullsized":"2024-03-04T06:19:00.328266+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5748555,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":234,"num_read_kb":178797,"num_write":211,"num_write_kb":58824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"670'234","reported_seq":2110,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.366168+0000","last_change":"2024-03-04T06:19:00.320681+0000","last_active":"2024-03-04T06:19:01.366168+0000","last_peered":"2024-03-04T06:19:01.366168+0000","last_clean":"2024-03-04T06:19:01.366168+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:19:01.366168+0000","last_undegraded":"2024-03-04T06:19:01.366168+0000","last_fullsized":"2024-03-04T06:19:01.366168+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":83,"num_read_kb":57507,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"663'678","reported_seq":2363,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.340228+0000","last_change":"2024-03-04T06:19:00.340228+0000","last_active":"2024-03-04T06:19:00.340228+0000","last_peered":"2024-03-04T06:19:00.340228+0000","last_clean":"2024-03-04T06:19:00.340228+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:19:00.340228+0000","last_undegraded":"2024-03-04T06:19:00.340228+0000","last_fullsized":"2024-03-04T06:19:00.340228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10084992,"num_objects":56,"num_object_clones":47,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":329,"num_read_kb":261977,"num_write":283,"num_write_kb":82964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1962,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.344174+0000","last_change":"2024-03-04T06:19:00.344174+0000","last_active":"2024-03-04T06:19:00.344174+0000","last_peered":"2024-03-04T06:19:00.344174+0000","last_clean":"2024-03-04T06:19:00.344174+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:19:00.344174+0000","last_undegraded":"2024-03-04T06:19:00.344174+0000","last_fullsized":"2024-03-04T06:19:00.344174+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"657'374","reported_seq":2200,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.354380+0000","last_change":"2024-03-04T06:19:00.354380+0000","last_active":"2024-03-04T06:19:00.354380+0000","last_peered":"2024-03-04T06:19:00.354380+0000","last_clean":"2024-03-04T06:19:00.354380+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:19:00.354380+0000","last_undegraded":"2024-03-04T06:19:00.354380+0000","last_fullsized":"2024-03-04T06:19:00.354380+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5782169,"num_objects":31,"num_object_clones":22,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":207,"num_read_kb":185579,"num_write":141,"num_write_kb":37633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1815,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.315549+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:19:01.315549+0000","last_peered":"2024-03-04T06:19:01.315549+0000","last_clean":"2024-03-04T06:19:01.315549+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:19:01.315549+0000","last_undegraded":"2024-03-04T06:19:01.315549+0000","last_fullsized":"2024-03-04T06:19:01.315549+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1675,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-03-04T06:19:00.371741+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:19:00.371741+0000","last_peered":"2024-03-04T06:19:00.371741+0000","last_clean":"2024-03-04T06:19:00.371741+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:19:00.371741+0000","last_undegraded":"2024-03-04T06:19:00.371741+0000","last_fullsized":"2024-03-04T06:19:00.371741+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"663'661","reported_seq":2369,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.432317+0000","last_change":"2024-03-04T06:19:00.368643+0000","last_active":"2024-03-04T06:19:00.432317+0000","last_peered":"2024-03-04T06:19:00.432317+0000","last_clean":"2024-03-04T06:19:00.432317+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:19:00.432317+0000","last_undegraded":"2024-03-04T06:19:00.432317+0000","last_fullsized":"2024-03-04T06:19:00.432317+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546871,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":327,"num_read_kb":261599,"num_write":270,"num_write_kb":72688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1657,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-03-04T06:19:00.360461+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:19:00.360461+0000","last_peered":"2024-03-04T06:19:00.360461+0000","last_clean":"2024-03-04T06:19:00.360461+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:19:00.360461+0000","last_undegraded":"2024-03-04T06:19:00.360461+0000","last_fullsized":"2024-03-04T06:19:00.360461+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"668'405","reported_seq":2234,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.318565+0000","last_change":"2024-03-04T06:19:00.313031+0000","last_active":"2024-03-04T06:19:01.318565+0000","last_peered":"2024-03-04T06:19:01.318565+0000","last_clean":"2024-03-04T06:19:01.318565+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:19:01.318565+0000","last_undegraded":"2024-03-04T06:19:01.318565+0000","last_fullsized":"2024-03-04T06:19:01.318565+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185207,"num_objects":39,"num_object_clones":31,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":233,"num_read_kb":187067,"num_write":156,"num_write_kb":43696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1675,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-03-04T06:19:00.359061+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:19:00.359061+0000","last_peered":"2024-03-04T06:19:00.359061+0000","last_clean":"2024-03-04T06:19:00.359061+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:19:00.359061+0000","last_undegraded":"2024-03-04T06:19:00.359061+0000","last_fullsized":"2024-03-04T06:19:00.359061+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"670'1139","reported_seq":2727,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.362108+0000","last_change":"2024-03-04T06:19:00.373949+0000","last_active":"2024-03-04T06:19:01.362108+0000","last_peered":"2024-03-04T06:19:01.362108+0000","last_clean":"2024-03-04T06:19:01.362108+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:19:01.362108+0000","last_undegraded":"2024-03-04T06:19:01.362108+0000","last_fullsized":"2024-03-04T06:19:01.362108+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1139,"log_dups_size":0,"ondisk_log_size":1139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10976126,"num_objects":108,"num_object_clones":85,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":634,"num_read_kb":499306,"num_write":505,"num_write_kb":134877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1657,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-03-04T06:19:00.330634+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:19:00.330634+0000","last_peered":"2024-03-04T06:19:00.330634+0000","last_clean":"2024-03-04T06:19:00.330634+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:19:00.330634+0000","last_undegraded":"2024-03-04T06:19:00.330634+0000","last_fullsized":"2024-03-04T06:19:00.330634+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"668'1275","reported_seq":2702,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.380813+0000","last_change":"2024-03-04T06:18:59.439358+0000","last_active":"2024-03-04T06:18:59.380813+0000","last_peered":"2024-03-04T06:18:59.380813+0000","last_clean":"2024-03-04T06:18:59.380813+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:18:59.380813+0000","last_undegraded":"2024-03-04T06:18:59.380813+0000","last_fullsized":"2024-03-04T06:18:59.380813+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1275,"log_dups_size":0,"ondisk_log_size":1275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101832,"num_objects":116,"num_object_clones":93,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":508,"num_read_kb":366970,"num_write":526,"num_write_kb":138869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"666'621","reported_seq":2390,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.369774+0000","last_change":"2024-03-04T06:19:00.369774+0000","last_active":"2024-03-04T06:19:00.369774+0000","last_peered":"2024-03-04T06:19:00.369774+0000","last_clean":"2024-03-04T06:19:00.369774+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:19:00.369774+0000","last_undegraded":"2024-03-04T06:19:00.369774+0000","last_fullsized":"2024-03-04T06:19:00.369774+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4857817,"num_objects":56,"num_object_clones":44,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":379,"num_read_kb":317170,"num_write":263,"num_write_kb":71240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1704,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.340479+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:18:59.340479+0000","last_peered":"2024-03-04T06:18:59.340479+0000","last_clean":"2024-03-04T06:18:59.340479+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:18:59.340479+0000","last_undegraded":"2024-03-04T06:18:59.340479+0000","last_fullsized":"2024-03-04T06:18:59.340479+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"668'605","reported_seq":2371,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.339265+0000","last_change":"2024-03-04T06:18:59.388171+0000","last_active":"2024-03-04T06:18:59.339265+0000","last_peered":"2024-03-04T06:18:59.339265+0000","last_clean":"2024-03-04T06:18:59.339265+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:18:59.339265+0000","last_undegraded":"2024-03-04T06:18:59.339265+0000","last_fullsized":"2024-03-04T06:18:59.339265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9127059,"num_objects":62,"num_object_clones":46,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":301,"num_read_kb":220159,"num_write":257,"num_write_kb":67704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1682,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.331859+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:19:01.331859+0000","last_peered":"2024-03-04T06:19:01.331859+0000","last_clean":"2024-03-04T06:19:01.331859+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:19:01.331859+0000","last_undegraded":"2024-03-04T06:19:01.331859+0000","last_fullsized":"2024-03-04T06:19:01.331859+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"669'830","reported_seq":2472,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.321213+0000","last_change":"2024-03-04T06:19:00.414536+0000","last_active":"2024-03-04T06:19:01.321213+0000","last_peered":"2024-03-04T06:19:01.321213+0000","last_clean":"2024-03-04T06:19:01.321213+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:19:01.321213+0000","last_undegraded":"2024-03-04T06:19:01.321213+0000","last_fullsized":"2024-03-04T06:19:01.321213+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11564371,"num_objects":76,"num_object_clones":62,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":380,"num_read_kb":272263,"num_write":356,"num_write_kb":98664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1704,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.317404+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:18:59.317404+0000","last_peered":"2024-03-04T06:18:59.317404+0000","last_clean":"2024-03-04T06:18:59.317404+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:18:59.317404+0000","last_undegraded":"2024-03-04T06:18:59.317404+0000","last_fullsized":"2024-03-04T06:18:59.317404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"668'888","reported_seq":2544,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.325855+0000","last_change":"2024-03-04T06:18:59.408402+0000","last_active":"2024-03-04T06:18:59.325855+0000","last_peered":"2024-03-04T06:18:59.325855+0000","last_clean":"2024-03-04T06:18:59.325855+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:18:59.325855+0000","last_undegraded":"2024-03-04T06:18:59.325855+0000","last_fullsized":"2024-03-04T06:18:59.325855+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":888,"log_dups_size":0,"ondisk_log_size":888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496450,"num_objects":77,"num_object_clones":67,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":486,"num_read_kb":408829,"num_write":350,"num_write_kb":95843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1675,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-03-04T06:19:00.367999+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:19:00.367999+0000","last_peered":"2024-03-04T06:19:00.367999+0000","last_clean":"2024-03-04T06:19:00.367999+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:19:00.367999+0000","last_undegraded":"2024-03-04T06:19:00.367999+0000","last_fullsized":"2024-03-04T06:19:00.367999+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":103423734,"num_objects":919,"num_object_clones":727,"num_object_copies":2757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":156,"num_read":4955,"num_read_kb":3900817,"num_write":4136,"num_write_kb":1124020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9930,"ondisk_log_size":9930,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":204804,"kb_used_data":151612,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94167036,"statfs":{"total":96636764160,"available":96427044864,"internally_reserved":0,"allocated":155250688,"data_stored":183369959,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":159024,"kb_used_data":109992,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94212816,"statfs":{"total":96636764160,"available":96473923584,"internally_reserved":0,"allocated":112631808,"data_stored":127242139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":174732,"kb_used_data":125700,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94197108,"statfs":{"total":96636764160,"available":96457838592,"internally_reserved":0,"allocated":128716800,"data_stored":154471457,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":156664,"kb_used_data":111792,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94215176,"statfs":{"total":96636764160,"available":96476340224,"internally_reserved":0,"allocated":114475008,"data_stored":136563954,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:19:02.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:19:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:02.797+0000 7f2db8f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:02.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:02.803+0000 7f2db8f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:02.992 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:19:02.992 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:19:03.004 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1001,"stamp":"2024-03-04T06:19:02.849446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":104014102,"num_objects":921,"num_object_clones":727,"num_object_copies":2763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":156,"num_read":5058,"num_read_kb":3900908,"num_write":4273,"num_write_kb":1125914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10027,"ondisk_log_size":10027,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":695224,"kb_used_data":499096,"kb_used_omap":0,"kb_used_meta":196096,"kb_avail":376792136,"statfs":{"total":386547056640,"available":385835147264,"internally_reserved":0,"allocated":511074304,"data_stored":601647509,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":200802304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2622651,"num_objects":9,"num_object_clones":9,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":28,"num_read_kb":16499,"num_write":49,"num_write_kb":12883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000905"},"pg_stats":[{"pgid":"3.f","version":"666'568","reported_seq":2311,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.375208+0000","last_change":"2024-03-04T06:19:00.375208+0000","last_active":"2024-03-04T06:19:00.375208+0000","last_peered":"2024-03-04T06:19:00.375208+0000","last_clean":"2024-03-04T06:19:00.375208+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:19:00.375208+0000","last_undegraded":"2024-03-04T06:19:00.375208+0000","last_fullsized":"2024-03-04T06:19:00.375208+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9530427,"num_objects":58,"num_object_clones":45,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":294,"num_read_kb":245917,"num_write":242,"num_write_kb":64731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"668'497","reported_seq":2261,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.315195+0000","last_change":"2024-03-04T06:18:59.391827+0000","last_active":"2024-03-04T06:18:59.315195+0000","last_peered":"2024-03-04T06:18:59.315195+0000","last_clean":"2024-03-04T06:18:59.315195+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:18:59.315195+0000","last_undegraded":"2024-03-04T06:18:59.315195+0000","last_fullsized":"2024-03-04T06:18:59.315195+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2840680,"num_objects":48,"num_object_clones":34,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":180,"num_read_kb":130653,"num_write":191,"num_write_kb":50586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"663'635","reported_seq":2406,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.352838+0000","last_change":"2024-03-04T06:19:00.352838+0000","last_active":"2024-03-04T06:19:00.352838+0000","last_peered":"2024-03-04T06:19:00.352838+0000","last_clean":"2024-03-04T06:19:00.352838+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:19:00.352838+0000","last_undegraded":"2024-03-04T06:19:00.352838+0000","last_fullsized":"2024-03-04T06:19:00.352838+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8639656,"num_objects":67,"num_object_clones":52,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":380,"num_read_kb":307024,"num_write":288,"num_write_kb":79007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"663'520","reported_seq":2258,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.328266+0000","last_change":"2024-03-04T06:19:00.328266+0000","last_active":"2024-03-04T06:19:00.328266+0000","last_peered":"2024-03-04T06:19:00.328266+0000","last_clean":"2024-03-04T06:19:00.328266+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:19:00.328266+0000","last_undegraded":"2024-03-04T06:19:00.328266+0000","last_fullsized":"2024-03-04T06:19:00.328266+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":520,"log_dups_size":0,"ondisk_log_size":520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5748555,"num_objects":44,"num_object_clones":34,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":234,"num_read_kb":178797,"num_write":211,"num_write_kb":58824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"670'234","reported_seq":2110,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.366168+0000","last_change":"2024-03-04T06:19:00.320681+0000","last_active":"2024-03-04T06:19:01.366168+0000","last_peered":"2024-03-04T06:19:01.366168+0000","last_clean":"2024-03-04T06:19:01.366168+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:19:01.366168+0000","last_undegraded":"2024-03-04T06:19:01.366168+0000","last_fullsized":"2024-03-04T06:19:01.366168+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":83,"num_read_kb":57507,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"663'678","reported_seq":2363,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.340228+0000","last_change":"2024-03-04T06:19:00.340228+0000","last_active":"2024-03-04T06:19:00.340228+0000","last_peered":"2024-03-04T06:19:00.340228+0000","last_clean":"2024-03-04T06:19:00.340228+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:19:00.340228+0000","last_undegraded":"2024-03-04T06:19:00.340228+0000","last_fullsized":"2024-03-04T06:19:00.340228+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10084992,"num_objects":56,"num_object_clones":47,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":329,"num_read_kb":261977,"num_write":283,"num_write_kb":82964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1962,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.344174+0000","last_change":"2024-03-04T06:19:00.344174+0000","last_active":"2024-03-04T06:19:00.344174+0000","last_peered":"2024-03-04T06:19:00.344174+0000","last_clean":"2024-03-04T06:19:00.344174+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:19:00.344174+0000","last_undegraded":"2024-03-04T06:19:00.344174+0000","last_fullsized":"2024-03-04T06:19:00.344174+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"657'374","reported_seq":2200,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.354380+0000","last_change":"2024-03-04T06:19:00.354380+0000","last_active":"2024-03-04T06:19:00.354380+0000","last_peered":"2024-03-04T06:19:00.354380+0000","last_clean":"2024-03-04T06:19:00.354380+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:19:00.354380+0000","last_undegraded":"2024-03-04T06:19:00.354380+0000","last_fullsized":"2024-03-04T06:19:00.354380+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5782169,"num_objects":31,"num_object_clones":22,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":207,"num_read_kb":185579,"num_write":141,"num_write_kb":37633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1815,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.315549+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:19:01.315549+0000","last_peered":"2024-03-04T06:19:01.315549+0000","last_clean":"2024-03-04T06:19:01.315549+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:19:01.315549+0000","last_undegraded":"2024-03-04T06:19:01.315549+0000","last_fullsized":"2024-03-04T06:19:01.315549+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1675,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-03-04T06:19:00.371741+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:19:00.371741+0000","last_peered":"2024-03-04T06:19:00.371741+0000","last_clean":"2024-03-04T06:19:00.371741+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:19:00.371741+0000","last_undegraded":"2024-03-04T06:19:00.371741+0000","last_fullsized":"2024-03-04T06:19:00.371741+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"663'661","reported_seq":2369,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.432317+0000","last_change":"2024-03-04T06:19:00.368643+0000","last_active":"2024-03-04T06:19:00.432317+0000","last_peered":"2024-03-04T06:19:00.432317+0000","last_clean":"2024-03-04T06:19:00.432317+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:19:00.432317+0000","last_undegraded":"2024-03-04T06:19:00.432317+0000","last_fullsized":"2024-03-04T06:19:00.432317+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546871,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":327,"num_read_kb":261599,"num_write":270,"num_write_kb":72688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1657,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-03-04T06:19:00.360461+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:19:00.360461+0000","last_peered":"2024-03-04T06:19:00.360461+0000","last_clean":"2024-03-04T06:19:00.360461+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:19:00.360461+0000","last_undegraded":"2024-03-04T06:19:00.360461+0000","last_fullsized":"2024-03-04T06:19:00.360461+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"668'405","reported_seq":2234,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.318565+0000","last_change":"2024-03-04T06:19:00.313031+0000","last_active":"2024-03-04T06:19:01.318565+0000","last_peered":"2024-03-04T06:19:01.318565+0000","last_clean":"2024-03-04T06:19:01.318565+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:19:01.318565+0000","last_undegraded":"2024-03-04T06:19:01.318565+0000","last_fullsized":"2024-03-04T06:19:01.318565+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185207,"num_objects":39,"num_object_clones":31,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":233,"num_read_kb":187067,"num_write":156,"num_write_kb":43696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1675,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-03-04T06:19:00.359061+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:19:00.359061+0000","last_peered":"2024-03-04T06:19:00.359061+0000","last_clean":"2024-03-04T06:19:00.359061+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:19:00.359061+0000","last_undegraded":"2024-03-04T06:19:00.359061+0000","last_fullsized":"2024-03-04T06:19:00.359061+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"670'1139","reported_seq":2727,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.362108+0000","last_change":"2024-03-04T06:19:00.373949+0000","last_active":"2024-03-04T06:19:01.362108+0000","last_peered":"2024-03-04T06:19:01.362108+0000","last_clean":"2024-03-04T06:19:01.362108+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:19:01.362108+0000","last_undegraded":"2024-03-04T06:19:01.362108+0000","last_fullsized":"2024-03-04T06:19:01.362108+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1139,"log_dups_size":0,"ondisk_log_size":1139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10976126,"num_objects":108,"num_object_clones":85,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":634,"num_read_kb":499306,"num_write":505,"num_write_kb":134877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1657,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-03-04T06:19:00.330634+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:19:00.330634+0000","last_peered":"2024-03-04T06:19:00.330634+0000","last_clean":"2024-03-04T06:19:00.330634+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:19:00.330634+0000","last_undegraded":"2024-03-04T06:19:00.330634+0000","last_fullsized":"2024-03-04T06:19:00.330634+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"668'1275","reported_seq":2702,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.380813+0000","last_change":"2024-03-04T06:18:59.439358+0000","last_active":"2024-03-04T06:18:59.380813+0000","last_peered":"2024-03-04T06:18:59.380813+0000","last_clean":"2024-03-04T06:18:59.380813+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:18:59.380813+0000","last_undegraded":"2024-03-04T06:18:59.380813+0000","last_fullsized":"2024-03-04T06:18:59.380813+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1275,"log_dups_size":0,"ondisk_log_size":1275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101832,"num_objects":116,"num_object_clones":93,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":508,"num_read_kb":366970,"num_write":526,"num_write_kb":138869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"666'621","reported_seq":2390,"reported_epoch":669,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:00.369774+0000","last_change":"2024-03-04T06:19:00.369774+0000","last_active":"2024-03-04T06:19:00.369774+0000","last_peered":"2024-03-04T06:19:00.369774+0000","last_clean":"2024-03-04T06:19:00.369774+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:19:00.369774+0000","last_undegraded":"2024-03-04T06:19:00.369774+0000","last_fullsized":"2024-03-04T06:19:00.369774+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4857817,"num_objects":56,"num_object_clones":44,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":379,"num_read_kb":317170,"num_write":263,"num_write_kb":71240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1704,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.340479+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:18:59.340479+0000","last_peered":"2024-03-04T06:18:59.340479+0000","last_clean":"2024-03-04T06:18:59.340479+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:18:59.340479+0000","last_undegraded":"2024-03-04T06:18:59.340479+0000","last_fullsized":"2024-03-04T06:18:59.340479+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"668'605","reported_seq":2371,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.339265+0000","last_change":"2024-03-04T06:18:59.388171+0000","last_active":"2024-03-04T06:18:59.339265+0000","last_peered":"2024-03-04T06:18:59.339265+0000","last_clean":"2024-03-04T06:18:59.339265+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:18:59.339265+0000","last_undegraded":"2024-03-04T06:18:59.339265+0000","last_fullsized":"2024-03-04T06:18:59.339265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9127059,"num_objects":62,"num_object_clones":46,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":301,"num_read_kb":220159,"num_write":257,"num_write_kb":67704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1682,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.331859+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:19:01.331859+0000","last_peered":"2024-03-04T06:19:01.331859+0000","last_clean":"2024-03-04T06:19:01.331859+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:19:01.331859+0000","last_undegraded":"2024-03-04T06:19:01.331859+0000","last_fullsized":"2024-03-04T06:19:01.331859+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"669'830","reported_seq":2472,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-03-04T06:19:01.321213+0000","last_change":"2024-03-04T06:19:00.414536+0000","last_active":"2024-03-04T06:19:01.321213+0000","last_peered":"2024-03-04T06:19:01.321213+0000","last_clean":"2024-03-04T06:19:01.321213+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:19:01.321213+0000","last_undegraded":"2024-03-04T06:19:01.321213+0000","last_fullsized":"2024-03-04T06:19:01.321213+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":830,"log_dups_size":0,"ondisk_log_size":830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11564371,"num_objects":76,"num_object_clones":62,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":380,"num_read_kb":272263,"num_write":356,"num_write_kb":98664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1704,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.317404+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:18:59.317404+0000","last_peered":"2024-03-04T06:18:59.317404+0000","last_clean":"2024-03-04T06:18:59.317404+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:18:59.317404+0000","last_undegraded":"2024-03-04T06:18:59.317404+0000","last_fullsized":"2024-03-04T06:18:59.317404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"668'888","reported_seq":2544,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-03-04T06:18:59.325855+0000","last_change":"2024-03-04T06:18:59.408402+0000","last_active":"2024-03-04T06:18:59.325855+0000","last_peered":"2024-03-04T06:18:59.325855+0000","last_clean":"2024-03-04T06:18:59.325855+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:18:59.325855+0000","last_undegraded":"2024-03-04T06:18:59.325855+0000","last_fullsized":"2024-03-04T06:18:59.325855+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":888,"log_dups_size":0,"ondisk_log_size":888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496450,"num_objects":77,"num_object_clones":67,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":486,"num_read_kb":408829,"num_write":350,"num_write_kb":95843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1675,"reported_epoch":669,"state":"active+clean","last_fresh":"2024-03-04T06:19:00.367999+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:19:00.367999+0000","last_peered":"2024-03-04T06:19:00.367999+0000","last_clean":"2024-03-04T06:19:00.367999+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:19:00.367999+0000","last_undegraded":"2024-03-04T06:19:00.367999+0000","last_fullsized":"2024-03-04T06:19:00.367999+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":103423734,"num_objects":919,"num_object_clones":727,"num_object_copies":2757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":156,"num_read":4955,"num_read_kb":3900817,"num_write":4136,"num_write_kb":1124020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9930,"ondisk_log_size":9930,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":204804,"kb_used_data":151612,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94167036,"statfs":{"total":96636764160,"available":96427044864,"internally_reserved":0,"allocated":155250688,"data_stored":183369959,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":159024,"kb_used_data":109992,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94212816,"statfs":{"total":96636764160,"available":96473923584,"internally_reserved":0,"allocated":112631808,"data_stored":127242139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":174732,"kb_used_data":125700,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94197108,"statfs":{"total":96636764160,"available":96457838592,"internally_reserved":0,"allocated":128716800,"data_stored":154471457,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706338,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":156664,"kb_used_data":111792,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94215176,"statfs":{"total":96636764160,"available":96476340224,"internally_reserved":0,"allocated":114475008,"data_stored":136563954,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:19:03.006 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:19:03.006 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:19:03.006 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:19:03.006 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:19:03.115 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:03.110+0000 7f49f2e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:03.118 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:03.117+0000 7f49f2e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:03.130 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:03.131 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:03.133 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:03.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:19:03.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:03.236+0000 7fcabc06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:03.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:03.243+0000 7fcabc06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:03.256 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:03.256 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:03.256 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:03.257 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:03.257 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:03.257 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:03.257 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:03.257 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:03.257 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:03.257 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:03.257 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:03.257 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:03.259 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:03.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:19:03.292 INFO:tasks.rados.rados.0.smithi064.stdout:2719: done (1 left) 2024-03-04T06:19:03.292 INFO:tasks.rados.rados.0.smithi064.stdout:2720: read oid 11 snap -1 2024-03-04T06:19:03.292 INFO:tasks.rados.rados.0.smithi064.stdout:2720: expect deleted 2024-03-04T06:19:03.293 INFO:tasks.rados.rados.0.smithi064.stdout:2721: snap_remove snap 301 2024-03-04T06:19:03.385 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:03.380+0000 7f20bd02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:03.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:03.387+0000 7f20bd02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:03.402 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:03.402 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:03.402 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:03.402 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:03.404 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:03.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:19:03.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:03.548+0000 7fa5de42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:03.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:03.555+0000 7fa5de42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:03.569 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:03.569 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:03.569 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:03.569 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:03.569 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:03.569 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:03.569 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:03.570 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:03.570 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:03.570 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:03.570 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:03.570 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:03.572 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:04.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:04.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:04.256+0000 7f9474a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:04.261 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:04.261+0000 7f9474a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:04.297 INFO:tasks.rados.rados.0.smithi064.stdout:2721: done (2 left) 2024-03-04T06:19:04.297 INFO:tasks.rados.rados.0.smithi064.stdout:2722: write oid 9 current snap is 336 2024-03-04T06:19:04.297 INFO:tasks.rados.rados.0.smithi064.stdout:2718: finishing write tid 1 to smithi06431743-35 2024-03-04T06:19:04.297 INFO:tasks.rados.rados.0.smithi064.stdout:2722: seq_num 1249 ranges {591943=667893,1721892=782758} 2024-03-04T06:19:04.303 INFO:tasks.rados.rados.0.smithi064.stdout:2722: writing smithi06431743-9 from 591943 to 1259836 tid 1 2024-03-04T06:19:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:2722: writing smithi06431743-9 from 1721892 to 2504650 tid 2 2024-03-04T06:19:04.310 INFO:tasks.rados.rados.0.smithi064.stdout:2723: write oid 21 current snap is 336 2024-03-04T06:19:04.311 INFO:tasks.rados.rados.0.smithi064.stdout:2723: seq_num 1250 ranges {608750=743142,1780374=642769} 2024-03-04T06:19:04.317 INFO:tasks.rados.rados.0.smithi064.stdout:2723: writing smithi06431743-21 from 608750 to 1351892 tid 1 2024-03-04T06:19:04.324 INFO:tasks.rados.rados.0.smithi064.stdout:2723: writing smithi06431743-21 from 1780374 to 2423143 tid 2 2024-03-04T06:19:04.325 INFO:tasks.rados.rados.0.smithi064.stdout:2724: read oid 7 snap 310 2024-03-04T06:19:04.325 INFO:tasks.rados.rados.0.smithi064.stdout:2724: expect (ObjNum 1093 snap 280 seq_num 1093) 2024-03-04T06:19:04.325 INFO:tasks.rados.rados.0.smithi064.stdout:2725: read oid 33 snap -1 2024-03-04T06:19:04.325 INFO:tasks.rados.rados.0.smithi064.stdout:2725: expect deleted 2024-03-04T06:19:04.325 INFO:tasks.rados.rados.0.smithi064.stdout:2726: snap_create 2024-03-04T06:19:04.325 INFO:tasks.rados.rados.0.smithi064.stdout:2718: finishing write tid 2 to smithi06431743-35 2024-03-04T06:19:04.325 INFO:tasks.rados.rados.0.smithi064.stdout:2718: finishing write tid 3 to smithi06431743-35 2024-03-04T06:19:04.325 INFO:tasks.rados.rados.0.smithi064.stdout:2718: finishing write tid 4 to smithi06431743-35 2024-03-04T06:19:04.325 INFO:tasks.rados.rados.0.smithi064.stdout:2718: finishing write tid 5 to smithi06431743-35 2024-03-04T06:19:04.325 INFO:tasks.rados.rados.0.smithi064.stdout:2718: oid 35 updating version 0 to 835 2024-03-04T06:19:04.326 INFO:tasks.rados.rados.0.smithi064.stdout:2718: oid 35 version 835 is already newer than 835 2024-03-04T06:19:04.326 INFO:tasks.rados.rados.0.smithi064.stdout:2718: oid 35 version 835 is already newer than 833 2024-03-04T06:19:04.326 INFO:tasks.rados.rados.0.smithi064.stdout:2718: oid 35 version 835 is already newer than 832 2024-03-04T06:19:04.326 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 835 (ObjNum 1248 snap 336 seq_num 1248) dirty exists 2024-03-04T06:19:04.326 INFO:tasks.rados.rados.0.smithi064.stdout:2718: left oid 35 (ObjNum 1248 snap 336 seq_num 1248) 2024-03-04T06:19:04.460 INFO:tasks.rados.rados.0.smithi064.stdout:2722: finishing write tid 1 to smithi06431743-9 2024-03-04T06:19:04.475 INFO:tasks.rados.rados.0.smithi064.stdout:2724: expect (ObjNum 1093 snap 280 seq_num 1093) 2024-03-04T06:19:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:2722: finishing write tid 2 to smithi06431743-9 2024-03-04T06:19:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:2723: finishing write tid 1 to smithi06431743-21 2024-03-04T06:19:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:2722: finishing write tid 3 to smithi06431743-9 2024-03-04T06:19:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:2722: finishing write tid 4 to smithi06431743-9 2024-03-04T06:19:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:2722: oid 9 updating version 0 to 898 2024-03-04T06:19:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:2722: oid 9 version 898 is already newer than 897 2024-03-04T06:19:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:2722: oid 9 version 898 is already newer than 897 2024-03-04T06:19:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 898 (ObjNum 1249 snap 336 seq_num 1249) dirty exists 2024-03-04T06:19:04.576 INFO:tasks.rados.rados.0.smithi064.stdout:2722: left oid 9 (ObjNum 1249 snap 336 seq_num 1249) 2024-03-04T06:19:04.577 INFO:tasks.rados.rados.0.smithi064.stdout:2723: finishing write tid 2 to smithi06431743-21 2024-03-04T06:19:04.577 INFO:tasks.rados.rados.0.smithi064.stdout:2723: finishing write tid 3 to smithi06431743-21 2024-03-04T06:19:04.577 INFO:tasks.rados.rados.0.smithi064.stdout:2723: finishing write tid 4 to smithi06431743-21 2024-03-04T06:19:04.577 INFO:tasks.rados.rados.0.smithi064.stdout:2723: oid 21 updating version 0 to 641 2024-03-04T06:19:04.577 INFO:tasks.rados.rados.0.smithi064.stdout:2723: oid 21 version 641 is already newer than 639 2024-03-04T06:19:04.577 INFO:tasks.rados.rados.0.smithi064.stdout:2723: oid 21 version 641 is already newer than 641 2024-03-04T06:19:04.577 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 641 (ObjNum 1250 snap 336 seq_num 1250) dirty exists 2024-03-04T06:19:04.577 INFO:tasks.rados.rados.0.smithi064.stdout:2723: left oid 21 (ObjNum 1250 snap 336 seq_num 1250) 2024-03-04T06:19:05.308 INFO:tasks.rados.rados.0.smithi064.stdout:2718: done (6 left) 2024-03-04T06:19:05.309 INFO:tasks.rados.rados.0.smithi064.stdout:2720: done (5 left) 2024-03-04T06:19:05.309 INFO:tasks.rados.rados.0.smithi064.stdout:2722: done (4 left) 2024-03-04T06:19:05.309 INFO:tasks.rados.rados.0.smithi064.stdout:2723: done (3 left) 2024-03-04T06:19:05.309 INFO:tasks.rados.rados.0.smithi064.stdout:2724: done (2 left) 2024-03-04T06:19:05.309 INFO:tasks.rados.rados.0.smithi064.stdout:2725: done (1 left) 2024-03-04T06:19:05.309 INFO:tasks.rados.rados.0.smithi064.stdout:2726: done (0 left) 2024-03-04T06:19:05.309 INFO:tasks.rados.rados.0.smithi064.stdout:2727: read oid 48 snap -1 2024-03-04T06:19:05.309 INFO:tasks.rados.rados.0.smithi064.stdout:2727: expect deleted 2024-03-04T06:19:05.309 INFO:tasks.rados.rados.0.smithi064.stdout:2728: write oid 17 current snap is 337 2024-03-04T06:19:05.309 INFO:tasks.rados.rados.0.smithi064.stdout:2728: seq_num 1251 ranges {625557=418391,1838856=786427} 2024-03-04T06:19:05.313 INFO:tasks.rados.rados.0.smithi064.stdout:2728: writing smithi06431743-17 from 625557 to 1043948 tid 1 2024-03-04T06:19:05.322 INFO:tasks.rados.rados.0.smithi064.stdout:2728: writing smithi06431743-17 from 1838856 to 2625283 tid 2 2024-03-04T06:19:05.322 INFO:tasks.rados.rados.0.smithi064.stdout:2729: snap_remove snap 329 2024-03-04T06:19:06.311 INFO:tasks.rados.rados.0.smithi064.stdout:2729: done (2 left) 2024-03-04T06:19:06.311 INFO:tasks.rados.rados.0.smithi064.stdout:2730: write oid 47 current snap is 337 2024-03-04T06:19:06.312 INFO:tasks.rados.rados.0.smithi064.stdout:2730: seq_num 1252 ranges {642364=493640,1897338=646438} 2024-03-04T06:19:06.317 INFO:tasks.rados.rados.0.smithi064.stdout:2730: writing smithi06431743-47 from 642364 to 1136004 tid 1 2024-03-04T06:19:06.325 INFO:tasks.rados.rados.0.smithi064.stdout:2730: writing smithi06431743-47 from 1897338 to 2543776 tid 2 2024-03-04T06:19:06.325 INFO:tasks.rados.rados.0.smithi064.stdout:2731: snap_remove snap 319 2024-03-04T06:19:06.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:06.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:06.545+0000 7fa63bf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:06.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:06.549+0000 7fa63bf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:07.321 INFO:tasks.rados.rados.0.smithi064.stdout:2731: done (3 left) 2024-03-04T06:19:07.322 INFO:tasks.rados.rados.0.smithi064.stdout:2732: write oid 12 current snap is 337 2024-03-04T06:19:07.322 INFO:tasks.rados.rados.0.smithi064.stdout:2732: seq_num 1253 ranges {659171=568889,1839467=790096} 2024-03-04T06:19:07.327 INFO:tasks.rados.rados.0.smithi064.stdout:2732: writing smithi06431743-12 from 659171 to 1228060 tid 1 2024-03-04T06:19:07.336 INFO:tasks.rados.rados.0.smithi064.stdout:2732: writing smithi06431743-12 from 1839467 to 2629563 tid 2 2024-03-04T06:19:07.336 INFO:tasks.rados.rados.0.smithi064.stdout:2733: read oid 3 snap 334 2024-03-04T06:19:07.336 INFO:tasks.rados.rados.0.smithi064.stdout:2733: expect (ObjNum 1225 snap 329 seq_num 1225) 2024-03-04T06:19:07.336 INFO:tasks.rados.rados.0.smithi064.stdout:2734: snap_remove snap 322 2024-03-04T06:19:08.292 INFO:tasks.rados.rados.0.smithi064.stdout:2734: done (5 left) 2024-03-04T06:19:08.292 INFO:tasks.rados.rados.0.smithi064.stdout:2735: delete oid 9 current snap is 337 2024-03-04T06:19:08.293 INFO:tasks.rados.rados.0.smithi064.stdout:2728: finishing write tid 1 to smithi06431743-17 2024-03-04T06:19:08.293 INFO:tasks.rados.rados.0.smithi064.stdout:2728: finishing write tid 2 to smithi06431743-17 2024-03-04T06:19:08.293 INFO:tasks.rados.rados.0.smithi064.stdout:2728: finishing write tid 3 to smithi06431743-17 2024-03-04T06:19:08.293 INFO:tasks.rados.rados.0.smithi064.stdout:2728: finishing write tid 4 to smithi06431743-17 2024-03-04T06:19:08.293 INFO:tasks.rados.rados.0.smithi064.stdout:2728: oid 17 updating version 0 to 383 2024-03-04T06:19:08.293 INFO:tasks.rados.rados.0.smithi064.stdout:2728: oid 17 updating version 383 to 384 2024-03-04T06:19:08.293 INFO:tasks.rados.rados.0.smithi064.stdout:2728: oid 17 version 384 is already newer than 382 2024-03-04T06:19:08.293 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 384 (ObjNum 1251 snap 337 seq_num 1251) dirty exists 2024-03-04T06:19:08.293 INFO:tasks.rados.rados.0.smithi064.stdout:2728: left oid 17 (ObjNum 1251 snap 337 seq_num 1251) 2024-03-04T06:19:08.293 INFO:tasks.rados.rados.0.smithi064.stdout:2730: finishing write tid 1 to smithi06431743-47 2024-03-04T06:19:08.293 INFO:tasks.rados.rados.0.smithi064.stdout:2730: finishing write tid 2 to smithi06431743-47 2024-03-04T06:19:08.294 INFO:tasks.rados.rados.0.smithi064.stdout:2730: finishing write tid 3 to smithi06431743-47 2024-03-04T06:19:08.294 INFO:tasks.rados.rados.0.smithi064.stdout:2730: finishing write tid 4 to smithi06431743-47 2024-03-04T06:19:08.294 INFO:tasks.rados.rados.0.smithi064.stdout:2730: oid 47 updating version 0 to 544 2024-03-04T06:19:08.294 INFO:tasks.rados.rados.0.smithi064.stdout:2730: oid 47 version 544 is already newer than 543 2024-03-04T06:19:08.294 INFO:tasks.rados.rados.0.smithi064.stdout:2730: oid 47 updating version 544 to 545 2024-03-04T06:19:08.294 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 545 (ObjNum 1252 snap 337 seq_num 1252) dirty exists 2024-03-04T06:19:08.294 INFO:tasks.rados.rados.0.smithi064.stdout:2730: left oid 47 (ObjNum 1252 snap 337 seq_num 1252) 2024-03-04T06:19:08.294 INFO:tasks.rados.rados.0.smithi064.stdout:2733: expect (ObjNum 1225 snap 329 seq_num 1225) 2024-03-04T06:19:08.407 INFO:tasks.rados.rados.0.smithi064.stdout:2732: finishing write tid 1 to smithi06431743-12 2024-03-04T06:19:08.407 INFO:tasks.rados.rados.0.smithi064.stdout:2732: finishing write tid 2 to smithi06431743-12 2024-03-04T06:19:08.408 INFO:tasks.rados.rados.0.smithi064.stdout:2732: finishing write tid 3 to smithi06431743-12 2024-03-04T06:19:08.408 INFO:tasks.rados.rados.0.smithi064.stdout:2732: finishing write tid 4 to smithi06431743-12 2024-03-04T06:19:08.408 INFO:tasks.rados.rados.0.smithi064.stdout:2732: oid 12 updating version 0 to 553 2024-03-04T06:19:08.408 INFO:tasks.rados.rados.0.smithi064.stdout:2732: oid 12 version 553 is already newer than 553 2024-03-04T06:19:08.408 INFO:tasks.rados.rados.0.smithi064.stdout:2732: oid 12 version 553 is already newer than 551 2024-03-04T06:19:08.408 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 553 (ObjNum 1253 snap 337 seq_num 1253) dirty exists 2024-03-04T06:19:08.408 INFO:tasks.rados.rados.0.smithi064.stdout:2732: left oid 12 (ObjNum 1253 snap 337 seq_num 1253) 2024-03-04T06:19:08.421 INFO:tasks.rados.rados.0.smithi064.stdout:2727: done (5 left) 2024-03-04T06:19:08.422 INFO:tasks.rados.rados.0.smithi064.stdout:2728: done (4 left) 2024-03-04T06:19:08.422 INFO:tasks.rados.rados.0.smithi064.stdout:2730: done (3 left) 2024-03-04T06:19:08.422 INFO:tasks.rados.rados.0.smithi064.stdout:2732: done (2 left) 2024-03-04T06:19:08.422 INFO:tasks.rados.rados.0.smithi064.stdout:2733: done (1 left) 2024-03-04T06:19:08.422 INFO:tasks.rados.rados.0.smithi064.stdout:2735: done (0 left) 2024-03-04T06:19:08.422 INFO:tasks.rados.rados.0.smithi064.stdout:2736: snap_create 2024-03-04T06:19:08.574 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:19:08.574 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:19:08.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:19:08.619 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:08.684 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:08.680+0000 7fafa76d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:08.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:08.687+0000 7fafa76d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:08.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:08.724+0000 7fa724a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:08.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:08.729+0000 7fa724a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:08.888 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:19:08.889 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:19:08.900 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1010,"stamp":"2024-03-04T06:19:08.850499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":106467957,"num_objects":927,"num_object_clones":733,"num_object_copies":2781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":156,"num_read":5073,"num_read_kb":3906520,"num_write":4298,"num_write_kb":1133627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10150,"ondisk_log_size":10150,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":706516,"kb_used_data":510388,"kb_used_omap":0,"kb_used_meta":196096,"kb_avail":376780844,"statfs":{"total":386547056640,"available":385823584256,"internally_reserved":0,"allocated":522637312,"data_stored":613651654,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":200802304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2453855,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":5612,"num_write":25,"num_write_kb":7713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000706"},"pg_stats":[{"pgid":"3.f","version":"672'578","reported_seq":2325,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-03-04T06:19:03.326845+0000","last_change":"2024-03-04T06:19:03.344006+0000","last_active":"2024-03-04T06:19:03.326845+0000","last_peered":"2024-03-04T06:19:03.326845+0000","last_clean":"2024-03-04T06:19:03.326845+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:19:03.326845+0000","last_undegraded":"2024-03-04T06:19:03.326845+0000","last_fullsized":"2024-03-04T06:19:03.326845+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":578,"log_dups_size":0,"ondisk_log_size":578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9580848,"num_objects":59,"num_object_clones":46,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":295,"num_read_kb":245918,"num_write":246,"num_write_kb":65848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"669'499","reported_seq":2275,"reported_epoch":673,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:04.331789+0000","last_change":"2024-03-04T06:19:04.331789+0000","last_active":"2024-03-04T06:19:04.331789+0000","last_peered":"2024-03-04T06:19:04.331789+0000","last_clean":"2024-03-04T06:19:04.331789+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:19:04.331789+0000","last_undegraded":"2024-03-04T06:19:04.331789+0000","last_fullsized":"2024-03-04T06:19:04.331789+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2840680,"num_objects":48,"num_object_clones":34,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":180,"num_read_kb":130653,"num_write":191,"num_write_kb":50586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"673'639","reported_seq":2423,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-03-04T06:19:05.327008+0000","last_change":"2024-03-04T06:19:04.421636+0000","last_active":"2024-03-04T06:19:05.327008+0000","last_peered":"2024-03-04T06:19:05.327008+0000","last_clean":"2024-03-04T06:19:05.327008+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:19:05.327008+0000","last_undegraded":"2024-03-04T06:19:05.327008+0000","last_fullsized":"2024-03-04T06:19:05.327008+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8639656,"num_objects":67,"num_object_clones":52,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":380,"num_read_kb":307024,"num_write":288,"num_write_kb":79007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"672'534","reported_seq":2272,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-03-04T06:19:03.338164+0000","last_change":"2024-03-04T06:19:03.338164+0000","last_active":"2024-03-04T06:19:03.338164+0000","last_peered":"2024-03-04T06:19:03.338164+0000","last_clean":"2024-03-04T06:19:03.338164+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:19:03.338164+0000","last_undegraded":"2024-03-04T06:19:03.338164+0000","last_fullsized":"2024-03-04T06:19:03.338164+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":534,"log_dups_size":0,"ondisk_log_size":534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5378834,"num_objects":45,"num_object_clones":35,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":236,"num_read_kb":178798,"num_write":215,"num_write_kb":60003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"675'241","reported_seq":2126,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-03-04T06:19:06.333392+0000","last_change":"2024-03-04T06:19:06.366000+0000","last_active":"2024-03-04T06:19:06.333392+0000","last_peered":"2024-03-04T06:19:06.333392+0000","last_clean":"2024-03-04T06:19:06.333392+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:19:06.333392+0000","last_undegraded":"2024-03-04T06:19:06.333392+0000","last_fullsized":"2024-03-04T06:19:06.333392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":84,"num_read_kb":57508,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"671'684","reported_seq":2373,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-03-04T06:19:03.337105+0000","last_change":"2024-03-04T06:19:03.337105+0000","last_active":"2024-03-04T06:19:03.337105+0000","last_peered":"2024-03-04T06:19:03.337105+0000","last_clean":"2024-03-04T06:19:03.337105+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:19:03.337105+0000","last_undegraded":"2024-03-04T06:19:03.337105+0000","last_fullsized":"2024-03-04T06:19:03.337105+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10084992,"num_objects":56,"num_object_clones":47,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":329,"num_read_kb":261977,"num_write":283,"num_write_kb":82964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1972,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-03-04T06:19:03.327651+0000","last_change":"2024-03-04T06:19:03.335034+0000","last_active":"2024-03-04T06:19:03.327651+0000","last_peered":"2024-03-04T06:19:03.327651+0000","last_clean":"2024-03-04T06:19:03.327651+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:19:03.327651+0000","last_undegraded":"2024-03-04T06:19:03.327651+0000","last_fullsized":"2024-03-04T06:19:03.327651+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"674'384","reported_seq":2221,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-03-04T06:19:05.381854+0000","last_change":"2024-03-04T06:19:04.386629+0000","last_active":"2024-03-04T06:19:05.381854+0000","last_peered":"2024-03-04T06:19:05.381854+0000","last_clean":"2024-03-04T06:19:05.381854+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:19:05.381854+0000","last_undegraded":"2024-03-04T06:19:05.381854+0000","last_fullsized":"2024-03-04T06:19:05.381854+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672940,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":208,"num_read_kb":185580,"num_write":145,"num_write_kb":38810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1828,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-03-04T06:19:06.322961+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:19:06.322961+0000","last_peered":"2024-03-04T06:19:06.322961+0000","last_clean":"2024-03-04T06:19:06.322961+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:19:06.322961+0000","last_undegraded":"2024-03-04T06:19:06.322961+0000","last_fullsized":"2024-03-04T06:19:06.322961+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1688,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-03-04T06:19:05.335746+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:19:05.335746+0000","last_peered":"2024-03-04T06:19:05.335746+0000","last_clean":"2024-03-04T06:19:05.335746+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:19:05.335746+0000","last_undegraded":"2024-03-04T06:19:05.335746+0000","last_fullsized":"2024-03-04T06:19:05.335746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"673'667","reported_seq":2386,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-03-04T06:19:05.337307+0000","last_change":"2024-03-04T06:19:04.431043+0000","last_active":"2024-03-04T06:19:05.337307+0000","last_peered":"2024-03-04T06:19:05.337307+0000","last_clean":"2024-03-04T06:19:05.337307+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:19:05.337307+0000","last_undegraded":"2024-03-04T06:19:05.337307+0000","last_fullsized":"2024-03-04T06:19:05.337307+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546871,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":327,"num_read_kb":261599,"num_write":270,"num_write_kb":72688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1664,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-03-04T06:19:03.328779+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:19:03.328779+0000","last_peered":"2024-03-04T06:19:03.328779+0000","last_clean":"2024-03-04T06:19:03.328779+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:19:03.328779+0000","last_undegraded":"2024-03-04T06:19:03.328779+0000","last_fullsized":"2024-03-04T06:19:03.328779+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"673'407","reported_seq":2252,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-03-04T06:19:06.326590+0000","last_change":"2024-03-04T06:19:06.334310+0000","last_active":"2024-03-04T06:19:06.326590+0000","last_peered":"2024-03-04T06:19:06.326590+0000","last_clean":"2024-03-04T06:19:06.326590+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:19:06.326590+0000","last_undegraded":"2024-03-04T06:19:06.326590+0000","last_fullsized":"2024-03-04T06:19:06.326590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185207,"num_objects":39,"num_object_clones":31,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":239,"num_read_kb":192671,"num_write":156,"num_write_kb":43696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1688,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-03-04T06:19:05.325570+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:19:05.325570+0000","last_peered":"2024-03-04T06:19:05.325570+0000","last_clean":"2024-03-04T06:19:05.325570+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:19:05.325570+0000","last_undegraded":"2024-03-04T06:19:05.325570+0000","last_fullsized":"2024-03-04T06:19:05.325570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"675'1148","reported_seq":2743,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-03-04T06:19:06.334320+0000","last_change":"2024-03-04T06:19:06.365675+0000","last_active":"2024-03-04T06:19:06.334320+0000","last_peered":"2024-03-04T06:19:06.334320+0000","last_clean":"2024-03-04T06:19:06.334320+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:19:06.334320+0000","last_undegraded":"2024-03-04T06:19:06.334320+0000","last_fullsized":"2024-03-04T06:19:06.334320+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1148,"log_dups_size":0,"ondisk_log_size":1148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10976126,"num_objects":108,"num_object_clones":85,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":635,"num_read_kb":499307,"num_write":505,"num_write_kb":134877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1664,"reported_epoch":672,"state":"active+clean","last_fresh":"2024-03-04T06:19:03.301442+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:19:03.301442+0000","last_peered":"2024-03-04T06:19:03.301442+0000","last_clean":"2024-03-04T06:19:03.301442+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:19:03.301442+0000","last_undegraded":"2024-03-04T06:19:03.301442+0000","last_fullsized":"2024-03-04T06:19:03.301442+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"669'1279","reported_seq":2716,"reported_epoch":673,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:04.322728+0000","last_change":"2024-03-04T06:19:04.322728+0000","last_active":"2024-03-04T06:19:04.322728+0000","last_peered":"2024-03-04T06:19:04.322728+0000","last_clean":"2024-03-04T06:19:04.322728+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:19:04.322728+0000","last_undegraded":"2024-03-04T06:19:04.322728+0000","last_fullsized":"2024-03-04T06:19:04.322728+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1279,"log_dups_size":0,"ondisk_log_size":1279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101832,"num_objects":116,"num_object_clones":93,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":508,"num_read_kb":366970,"num_write":526,"num_write_kb":138869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"673'643","reported_seq":2411,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-03-04T06:19:05.337158+0000","last_change":"2024-03-04T06:19:04.534016+0000","last_active":"2024-03-04T06:19:05.337158+0000","last_peered":"2024-03-04T06:19:05.337158+0000","last_clean":"2024-03-04T06:19:05.337158+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:19:05.337158+0000","last_undegraded":"2024-03-04T06:19:05.337158+0000","last_fullsized":"2024-03-04T06:19:05.337158+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5765395,"num_objects":57,"num_object_clones":45,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":380,"num_read_kb":317171,"num_write":267,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1717,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-04T06:19:04.337803+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:19:04.337803+0000","last_peered":"2024-03-04T06:19:04.337803+0000","last_clean":"2024-03-04T06:19:04.337803+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:19:04.337803+0000","last_undegraded":"2024-03-04T06:19:04.337803+0000","last_fullsized":"2024-03-04T06:19:04.337803+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"669'607","reported_seq":2385,"reported_epoch":673,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:04.338635+0000","last_change":"2024-03-04T06:19:04.338635+0000","last_active":"2024-03-04T06:19:04.338635+0000","last_peered":"2024-03-04T06:19:04.338635+0000","last_clean":"2024-03-04T06:19:04.338635+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:19:04.338635+0000","last_undegraded":"2024-03-04T06:19:04.338635+0000","last_fullsized":"2024-03-04T06:19:04.338635+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9127059,"num_objects":62,"num_object_clones":46,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":301,"num_read_kb":220159,"num_write":257,"num_write_kb":67704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1695,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-03-04T06:19:06.328482+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:19:06.328482+0000","last_peered":"2024-03-04T06:19:06.328482+0000","last_clean":"2024-03-04T06:19:06.328482+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:19:06.328482+0000","last_undegraded":"2024-03-04T06:19:06.328482+0000","last_fullsized":"2024-03-04T06:19:06.328482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"675'845","reported_seq":2492,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-03-04T06:19:06.324562+0000","last_change":"2024-03-04T06:19:06.363658+0000","last_active":"2024-03-04T06:19:06.324562+0000","last_peered":"2024-03-04T06:19:06.324562+0000","last_clean":"2024-03-04T06:19:06.324562+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:19:06.324562+0000","last_undegraded":"2024-03-04T06:19:06.324562+0000","last_fullsized":"2024-03-04T06:19:06.324562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":845,"log_dups_size":0,"ondisk_log_size":845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11866897,"num_objects":77,"num_object_clones":63,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":381,"num_read_kb":272264,"num_write":361,"num_write_kb":100132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1717,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-04T06:19:04.329026+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:19:04.329026+0000","last_peered":"2024-03-04T06:19:04.329026+0000","last_clean":"2024-03-04T06:19:04.329026+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:19:04.329026+0000","last_undegraded":"2024-03-04T06:19:04.329026+0000","last_fullsized":"2024-03-04T06:19:04.329026+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"673'898","reported_seq":2562,"reported_epoch":673,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:04.474546+0000","last_change":"2024-03-04T06:19:04.324480+0000","last_active":"2024-03-04T06:19:04.474546+0000","last_peered":"2024-03-04T06:19:04.474546+0000","last_clean":"2024-03-04T06:19:04.474546+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:19:04.474546+0000","last_undegraded":"2024-03-04T06:19:04.474546+0000","last_fullsized":"2024-03-04T06:19:04.474546+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":898,"log_dups_size":0,"ondisk_log_size":898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168730,"num_objects":78,"num_object_clones":68,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":487,"num_read_kb":408830,"num_write":354,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1688,"reported_epoch":674,"state":"active+clean","last_fresh":"2024-03-04T06:19:05.334238+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:19:05.334238+0000","last_peered":"2024-03-04T06:19:05.334238+0000","last_clean":"2024-03-04T06:19:05.334238+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:19:05.334238+0000","last_undegraded":"2024-03-04T06:19:05.334238+0000","last_fullsized":"2024-03-04T06:19:05.334238+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":105877589,"num_objects":925,"num_object_clones":733,"num_object_copies":2775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":156,"num_read":4970,"num_read_kb":3906429,"num_write":4161,"num_write_kb":1131733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10053,"ondisk_log_size":10053,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706344,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":212580,"kb_used_data":159388,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94159260,"statfs":{"total":96636764160,"available":96419082240,"internally_reserved":0,"allocated":163213312,"data_stored":192749841,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706344,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":165464,"kb_used_data":116432,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94206376,"statfs":{"total":96636764160,"available":96467329024,"internally_reserved":0,"allocated":119226368,"data_stored":134514160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706343,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":174268,"kb_used_data":125236,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94197572,"statfs":{"total":96636764160,"available":96458313728,"internally_reserved":0,"allocated":128241664,"data_stored":151434316,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706343,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154204,"kb_used_data":109332,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94217636,"statfs":{"total":96636764160,"available":96478859264,"internally_reserved":0,"allocated":111955968,"data_stored":134953337,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:19:09.320 INFO:tasks.rados.rados.0.smithi064.stdout:2736: done (0 left) 2024-03-04T06:19:09.321 INFO:tasks.rados.rados.0.smithi064.stdout:2737: delete oid 8 current snap is 338 2024-03-04T06:19:09.377 INFO:tasks.rados.rados.0.smithi064.stdout:2737: done (0 left) 2024-03-04T06:19:09.377 INFO:tasks.rados.rados.0.smithi064.stdout:2738: write (excl) oid 45 current snap is 338 2024-03-04T06:19:09.377 INFO:tasks.rados.rados.0.smithi064.stdout:2738: seq_num 1256 ranges {709592=510989,1731266=653776,3041279=68313} 2024-03-04T06:19:09.383 INFO:tasks.rados.rados.0.smithi064.stdout:2738: writing smithi06431743-45 from 709592 to 1220581 tid 1 2024-03-04T06:19:09.390 INFO:tasks.rados.rados.0.smithi064.stdout:2738: writing smithi06431743-45 from 1731266 to 2385042 tid 2 2024-03-04T06:19:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:2738: writing smithi06431743-45 from 3041279 to 3109592 tid 3 2024-03-04T06:19:09.394 INFO:tasks.rados.rados.0.smithi064.stdout:2739: snap_create 2024-03-04T06:19:09.443 INFO:tasks.rados.rados.0.smithi064.stdout:2738: finishing write tid 1 to smithi06431743-45 2024-03-04T06:19:09.457 INFO:tasks.rados.rados.0.smithi064.stdout:2738: finishing write tid 2 to smithi06431743-45 2024-03-04T06:19:09.462 INFO:tasks.rados.rados.0.smithi064.stdout:2738: finishing write tid 3 to smithi06431743-45 2024-03-04T06:19:09.469 INFO:tasks.rados.rados.0.smithi064.stdout:2738: finishing write tid 4 to smithi06431743-45 2024-03-04T06:19:09.470 INFO:tasks.rados.rados.0.smithi064.stdout:2738: finishing write tid 5 to smithi06431743-45 2024-03-04T06:19:09.470 INFO:tasks.rados.rados.0.smithi064.stdout:2738: oid 45 updating version 0 to 644 2024-03-04T06:19:09.470 INFO:tasks.rados.rados.0.smithi064.stdout:2738: oid 45 version 644 is already newer than 643 2024-03-04T06:19:09.470 INFO:tasks.rados.rados.0.smithi064.stdout:2738: oid 45 updating version 644 to 645 2024-03-04T06:19:09.470 INFO:tasks.rados.rados.0.smithi064.stdout:2738: oid 45 updating version 645 to 646 2024-03-04T06:19:09.470 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 646 (ObjNum 1256 snap 338 seq_num 1256) dirty exists 2024-03-04T06:19:09.470 INFO:tasks.rados.rados.0.smithi064.stdout:2738: left oid 45 (ObjNum 1256 snap 338 seq_num 1256) 2024-03-04T06:19:10.323 INFO:tasks.rados.rados.0.smithi064.stdout:2738: done (1 left) 2024-03-04T06:19:10.323 INFO:tasks.rados.rados.0.smithi064.stdout:2739: done (0 left) 2024-03-04T06:19:10.323 INFO:tasks.rados.rados.0.smithi064.stdout:2740: delete oid 40 current snap is 339 2024-03-04T06:19:10.387 INFO:tasks.rados.rados.0.smithi064.stdout:2740: done (0 left) 2024-03-04T06:19:10.387 INFO:tasks.rados.rados.0.smithi064.stdout:2741: write oid 46 current snap is 339 2024-03-04T06:19:10.387 INFO:tasks.rados.rados.0.smithi064.stdout:2741: seq_num 1258 ranges {743206=661487,2131877=541092} 2024-03-04T06:19:10.392 INFO:tasks.rados.rados.0.smithi064.stdout:2741: writing smithi06431743-46 from 743206 to 1404693 tid 1 2024-03-04T06:19:10.399 INFO:tasks.rados.rados.0.smithi064.stdout:2741: writing smithi06431743-46 from 2131877 to 2672969 tid 2 2024-03-04T06:19:10.399 INFO:tasks.rados.rados.0.smithi064.stdout:2742: snap_create 2024-03-04T06:19:10.465 INFO:tasks.rados.rados.0.smithi064.stdout:2741: finishing write tid 1 to smithi06431743-46 2024-03-04T06:19:10.485 INFO:tasks.rados.rados.0.smithi064.stdout:2741: finishing write tid 2 to smithi06431743-46 2024-03-04T06:19:10.501 INFO:tasks.rados.rados.0.smithi064.stdout:2741: finishing write tid 3 to smithi06431743-46 2024-03-04T06:19:10.502 INFO:tasks.rados.rados.0.smithi064.stdout:2741: finishing write tid 4 to smithi06431743-46 2024-03-04T06:19:10.502 INFO:tasks.rados.rados.0.smithi064.stdout:2741: oid 46 updating version 0 to 1162 2024-03-04T06:19:10.502 INFO:tasks.rados.rados.0.smithi064.stdout:2741: oid 46 updating version 1162 to 1163 2024-03-04T06:19:10.502 INFO:tasks.rados.rados.0.smithi064.stdout:2741: oid 46 updating version 1163 to 1164 2024-03-04T06:19:10.502 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 1164 (ObjNum 1258 snap 339 seq_num 1258) dirty exists 2024-03-04T06:19:10.502 INFO:tasks.rados.rados.0.smithi064.stdout:2741: left oid 46 (ObjNum 1258 snap 339 seq_num 1258) 2024-03-04T06:19:10.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:10.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:10.911+0000 7f188499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:10.916 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:10.916+0000 7f188499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:11.327 INFO:tasks.rados.rados.0.smithi064.stdout:2741: done (1 left) 2024-03-04T06:19:11.327 INFO:tasks.rados.rados.0.smithi064.stdout:2742: done (0 left) 2024-03-04T06:19:11.327 INFO:tasks.rados.rados.0.smithi064.stdout:2743: delete oid 11 current snap is 340 2024-03-04T06:19:11.351 INFO:tasks.rados.rados.0.smithi064.stdout:2743: done (0 left) 2024-03-04T06:19:11.352 INFO:tasks.rados.rados.0.smithi064.stdout:2744: snap_remove snap 336 2024-03-04T06:19:12.331 INFO:tasks.rados.rados.0.smithi064.stdout:2744: done (0 left) 2024-03-04T06:19:12.331 INFO:tasks.rados.rados.0.smithi064.stdout:2745: snap_create 2024-03-04T06:19:12.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:13.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:13.102+0000 7ff62459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:13.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:13.107+0000 7ff62459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:13.340 INFO:tasks.rados.rados.0.smithi064.stdout:2745: done (0 left) 2024-03-04T06:19:13.340 INFO:tasks.rados.rados.0.smithi064.stdout:2746: snap_remove snap 327 2024-03-04T06:19:13.902 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:19:13.903 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:19:13.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:19:14.041 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:14.037+0000 7f6dc539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:14.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:14.043+0000 7f6dc539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:14.233 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:19:14.233 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:19:14.245 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1017,"stamp":"2024-03-04T06:19:13.339021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":102937893,"num_objects":933,"num_object_clones":739,"num_object_copies":2799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":158,"num_read":5086,"num_read_kb":3912526,"num_write":4318,"num_write_kb":1138451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10274,"ondisk_log_size":10274,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":719948,"kb_used_data":519660,"kb_used_omap":0,"kb_used_meta":200256,"kb_avail":376767412,"statfs":{"total":386547056640,"available":385809829888,"internally_reserved":0,"allocated":532131840,"data_stored":625177572,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":205062144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3530064,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":13,"num_read_kb":6006,"num_write":20,"num_write_kb":4824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019443"},"pg_stats":[{"pgid":"3.f","version":"677'584","reported_seq":2347,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.364383+0000","last_change":"2024-03-04T06:19:08.419184+0000","last_active":"2024-03-04T06:19:10.364383+0000","last_peered":"2024-03-04T06:19:10.364383+0000","last_clean":"2024-03-04T06:19:10.364383+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:19:10.364383+0000","last_undegraded":"2024-03-04T06:19:10.364383+0000","last_fullsized":"2024-03-04T06:19:10.364383+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790589,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":295,"num_read_kb":245918,"num_write":247,"num_write_kb":65848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"677'509","reported_seq":2290,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-03-04T06:19:09.331145+0000","last_change":"2024-03-04T06:19:08.457020+0000","last_active":"2024-03-04T06:19:09.331145+0000","last_peered":"2024-03-04T06:19:09.331145+0000","last_clean":"2024-03-04T06:19:09.331145+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:19:09.331145+0000","last_undegraded":"2024-03-04T06:19:09.331145+0000","last_fullsized":"2024-03-04T06:19:09.331145+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2840680,"num_objects":48,"num_object_clones":34,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":180,"num_read_kb":130653,"num_write":191,"num_write_kb":50586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"678'646","reported_seq":2448,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-03-04T06:19:11.340741+0000","last_change":"2024-03-04T06:19:08.410097+0000","last_active":"2024-03-04T06:19:11.340741+0000","last_peered":"2024-03-04T06:19:11.340741+0000","last_clean":"2024-03-04T06:19:11.340741+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:19:11.340741+0000","last_undegraded":"2024-03-04T06:19:11.340741+0000","last_fullsized":"2024-03-04T06:19:11.340741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9311936,"num_objects":68,"num_object_clones":53,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":383,"num_read_kb":307025,"num_write":293,"num_write_kb":80213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"677'557","reported_seq":2301,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.350778+0000","last_change":"2024-03-04T06:19:08.398304+0000","last_active":"2024-03-04T06:19:10.350778+0000","last_peered":"2024-03-04T06:19:10.350778+0000","last_clean":"2024-03-04T06:19:10.350778+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:19:10.350778+0000","last_undegraded":"2024-03-04T06:19:10.350778+0000","last_fullsized":"2024-03-04T06:19:10.350778+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9009443,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":238,"num_read_kb":178800,"num_write":223,"num_write_kb":62446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"677'245","reported_seq":2141,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.422942+0000","last_change":"2024-03-04T06:19:08.464256+0000","last_active":"2024-03-04T06:19:10.422942+0000","last_peered":"2024-03-04T06:19:10.422942+0000","last_clean":"2024-03-04T06:19:10.422942+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:19:10.422942+0000","last_undegraded":"2024-03-04T06:19:10.422942+0000","last_fullsized":"2024-03-04T06:19:10.422942+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":84,"num_read_kb":57508,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"674'688","reported_seq":2394,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.361379+0000","last_change":"2024-03-04T06:19:08.369023+0000","last_active":"2024-03-04T06:19:10.361379+0000","last_peered":"2024-03-04T06:19:10.361379+0000","last_clean":"2024-03-04T06:19:10.361379+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:19:10.361379+0000","last_undegraded":"2024-03-04T06:19:10.361379+0000","last_fullsized":"2024-03-04T06:19:10.361379+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10084992,"num_objects":56,"num_object_clones":47,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":329,"num_read_kb":261977,"num_write":283,"num_write_kb":82964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1993,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.362101+0000","last_change":"2024-03-04T06:19:08.362246+0000","last_active":"2024-03-04T06:19:10.362101+0000","last_peered":"2024-03-04T06:19:10.362101+0000","last_clean":"2024-03-04T06:19:10.362101+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:19:10.362101+0000","last_undegraded":"2024-03-04T06:19:10.362101+0000","last_fullsized":"2024-03-04T06:19:10.362101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"677'388","reported_seq":2240,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-03-04T06:19:11.341634+0000","last_change":"2024-03-04T06:19:09.364243+0000","last_active":"2024-03-04T06:19:11.341634+0000","last_peered":"2024-03-04T06:19:11.341634+0000","last_clean":"2024-03-04T06:19:11.341634+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:19:11.341634+0000","last_undegraded":"2024-03-04T06:19:11.341634+0000","last_fullsized":"2024-03-04T06:19:11.341634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672940,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":208,"num_read_kb":185580,"num_write":145,"num_write_kb":38810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1840,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.415792+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:19:10.415792+0000","last_peered":"2024-03-04T06:19:10.415792+0000","last_clean":"2024-03-04T06:19:10.415792+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:19:10.415792+0000","last_undegraded":"2024-03-04T06:19:10.415792+0000","last_fullsized":"2024-03-04T06:19:10.415792+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1704,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-03-04T06:19:11.349449+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:19:11.349449+0000","last_peered":"2024-03-04T06:19:11.349449+0000","last_clean":"2024-03-04T06:19:11.349449+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:19:11.349449+0000","last_undegraded":"2024-03-04T06:19:11.349449+0000","last_fullsized":"2024-03-04T06:19:11.349449+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"673'667","reported_seq":2408,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-03-04T06:19:11.345705+0000","last_change":"2024-03-04T06:19:08.409981+0000","last_active":"2024-03-04T06:19:11.345705+0000","last_peered":"2024-03-04T06:19:11.345705+0000","last_clean":"2024-03-04T06:19:11.345705+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:19:11.345705+0000","last_undegraded":"2024-03-04T06:19:11.345705+0000","last_fullsized":"2024-03-04T06:19:11.345705+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546871,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":333,"num_read_kb":267601,"num_write":270,"num_write_kb":72688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1681,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.364487+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:19:10.364487+0000","last_peered":"2024-03-04T06:19:10.364487+0000","last_clean":"2024-03-04T06:19:10.364487+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:19:10.364487+0000","last_undegraded":"2024-03-04T06:19:10.364487+0000","last_fullsized":"2024-03-04T06:19:10.364487+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"673'407","reported_seq":2267,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.411411+0000","last_change":"2024-03-04T06:19:08.452433+0000","last_active":"2024-03-04T06:19:10.411411+0000","last_peered":"2024-03-04T06:19:10.411411+0000","last_clean":"2024-03-04T06:19:10.411411+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:19:10.411411+0000","last_undegraded":"2024-03-04T06:19:10.411411+0000","last_fullsized":"2024-03-04T06:19:10.411411+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185207,"num_objects":39,"num_object_clones":31,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":239,"num_read_kb":192671,"num_write":156,"num_write_kb":43696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1704,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-03-04T06:19:11.339791+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:19:11.339791+0000","last_peered":"2024-03-04T06:19:11.339791+0000","last_clean":"2024-03-04T06:19:11.339791+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:19:11.339791+0000","last_undegraded":"2024-03-04T06:19:11.339791+0000","last_fullsized":"2024-03-04T06:19:11.339791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"679'1164","reported_seq":2762,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.486167+0000","last_change":"2024-03-04T06:19:08.501329+0000","last_active":"2024-03-04T06:19:10.486167+0000","last_peered":"2024-03-04T06:19:10.486167+0000","last_clean":"2024-03-04T06:19:10.486167+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:19:10.486167+0000","last_undegraded":"2024-03-04T06:19:10.486167+0000","last_fullsized":"2024-03-04T06:19:10.486167+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11581178,"num_objects":109,"num_object_clones":86,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":636,"num_read_kb":499308,"num_write":509,"num_write_kb":136052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1681,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.339899+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:19:10.339899+0000","last_peered":"2024-03-04T06:19:10.339899+0000","last_clean":"2024-03-04T06:19:10.339899+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:19:10.339899+0000","last_undegraded":"2024-03-04T06:19:10.339899+0000","last_fullsized":"2024-03-04T06:19:10.339899+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"677'1295","reported_seq":2728,"reported_epoch":678,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:09.333402+0000","last_change":"2024-03-04T06:19:08.315210+0000","last_active":"2024-03-04T06:19:09.333402+0000","last_peered":"2024-03-04T06:19:09.333402+0000","last_clean":"2024-03-04T06:19:09.333402+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:19:09.333402+0000","last_undegraded":"2024-03-04T06:19:09.333402+0000","last_fullsized":"2024-03-04T06:19:09.333402+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101832,"num_objects":116,"num_object_clones":93,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":508,"num_read_kb":366970,"num_write":526,"num_write_kb":138869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"677'653","reported_seq":2429,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-03-04T06:19:11.348177+0000","last_change":"2024-03-04T06:19:09.414733+0000","last_active":"2024-03-04T06:19:11.348177+0000","last_peered":"2024-03-04T06:19:11.348177+0000","last_clean":"2024-03-04T06:19:11.348177+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:19:11.348177+0000","last_undegraded":"2024-03-04T06:19:11.348177+0000","last_fullsized":"2024-03-04T06:19:11.348177+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":653,"log_dups_size":0,"ondisk_log_size":653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5765395,"num_objects":57,"num_object_clones":45,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":380,"num_read_kb":317171,"num_write":267,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1728,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-03-04T06:19:09.353004+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:19:09.353004+0000","last_peered":"2024-03-04T06:19:09.353004+0000","last_clean":"2024-03-04T06:19:09.353004+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:19:09.353004+0000","last_undegraded":"2024-03-04T06:19:09.353004+0000","last_fullsized":"2024-03-04T06:19:09.353004+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"677'613","reported_seq":2401,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-03-04T06:19:09.360467+0000","last_change":"2024-03-04T06:19:09.360467+0000","last_active":"2024-03-04T06:19:09.360467+0000","last_peered":"2024-03-04T06:19:09.360467+0000","last_clean":"2024-03-04T06:19:09.360467+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:19:09.360467+0000","last_undegraded":"2024-03-04T06:19:09.360467+0000","last_fullsized":"2024-03-04T06:19:09.360467+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6471256,"num_objects":63,"num_object_clones":47,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":301,"num_read_kb":220159,"num_write":258,"num_write_kb":67704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1707,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.421173+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:19:10.421173+0000","last_peered":"2024-03-04T06:19:10.421173+0000","last_clean":"2024-03-04T06:19:10.421173+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:19:10.421173+0000","last_undegraded":"2024-03-04T06:19:10.421173+0000","last_fullsized":"2024-03-04T06:19:10.421173+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"677'859","reported_seq":2507,"reported_epoch":679,"state":"active+clean","last_fresh":"2024-03-04T06:19:10.413074+0000","last_change":"2024-03-04T06:19:08.501121+0000","last_active":"2024-03-04T06:19:10.413074+0000","last_peered":"2024-03-04T06:19:10.413074+0000","last_clean":"2024-03-04T06:19:10.413074+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:19:10.413074+0000","last_undegraded":"2024-03-04T06:19:10.413074+0000","last_fullsized":"2024-03-04T06:19:10.413074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11866897,"num_objects":77,"num_object_clones":63,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":381,"num_read_kb":272264,"num_write":361,"num_write_kb":100132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1728,"reported_epoch":678,"state":"active+clean","last_fresh":"2024-03-04T06:19:09.337756+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:19:09.337756+0000","last_peered":"2024-03-04T06:19:09.337756+0000","last_clean":"2024-03-04T06:19:09.337756+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:19:09.337756+0000","last_undegraded":"2024-03-04T06:19:09.337756+0000","last_fullsized":"2024-03-04T06:19:09.337756+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"677'902","reported_seq":2576,"reported_epoch":678,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:09.327909+0000","last_change":"2024-03-04T06:19:08.327763+0000","last_active":"2024-03-04T06:19:09.327909+0000","last_peered":"2024-03-04T06:19:09.327909+0000","last_clean":"2024-03-04T06:19:09.327909+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:19:09.327909+0000","last_undegraded":"2024-03-04T06:19:09.327909+0000","last_fullsized":"2024-03-04T06:19:09.327909+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3176787,"num_objects":79,"num_object_clones":69,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":488,"num_read_kb":408830,"num_write":355,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1704,"reported_epoch":680,"state":"active+clean","last_fresh":"2024-03-04T06:19:11.346960+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:19:11.346960+0000","last_peered":"2024-03-04T06:19:11.346960+0000","last_clean":"2024-03-04T06:19:11.346960+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:19:11.346960+0000","last_undegraded":"2024-03-04T06:19:11.346960+0000","last_fullsized":"2024-03-04T06:19:11.346960+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102347525,"num_objects":931,"num_object_clones":739,"num_object_copies":2793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":158,"num_read":4983,"num_read_kb":3912435,"num_write":4181,"num_write_kb":1136557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10177,"ondisk_log_size":10177,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706349,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":215656,"kb_used_data":162464,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94156184,"statfs":{"total":96636764160,"available":96415932416,"internally_reserved":0,"allocated":166363136,"data_stored":196672764,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706349,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":167900,"kb_used_data":118868,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94203940,"statfs":{"total":96636764160,"available":96464834560,"internally_reserved":0,"allocated":121720832,"data_stored":138386897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706348,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":176132,"kb_used_data":127100,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94195708,"statfs":{"total":96636764160,"available":96456404992,"internally_reserved":0,"allocated":130150400,"data_stored":153200144,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706348,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":160260,"kb_used_data":111228,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94211580,"statfs":{"total":96636764160,"available":96472657920,"internally_reserved":0,"allocated":113897472,"data_stored":136917767,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:19:14.339 INFO:tasks.rados.rados.0.smithi064.stdout:2746: done (0 left) 2024-03-04T06:19:14.339 INFO:tasks.rados.rados.0.smithi064.stdout:2747: write (excl) oid 11 current snap is 341 2024-03-04T06:19:14.339 INFO:tasks.rados.rados.0.smithi064.stdout:2747: seq_num 1260 ranges {776820=411985,1848841=544761,2918265=258555} 2024-03-04T06:19:14.346 INFO:tasks.rados.rados.0.smithi064.stdout:2747: writing smithi06431743-11 from 776820 to 1188805 tid 1 2024-03-04T06:19:14.352 INFO:tasks.rados.rados.0.smithi064.stdout:2747: writing smithi06431743-11 from 1848841 to 2393602 tid 2 2024-03-04T06:19:14.357 INFO:tasks.rados.rados.0.smithi064.stdout:2747: writing smithi06431743-11 from 2918265 to 3176820 tid 3 2024-03-04T06:19:14.357 INFO:tasks.rados.rados.0.smithi064.stdout:2748: snap_remove snap 300 2024-03-04T06:19:15.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:15.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:15.285+0000 7f058e39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:15.290 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:15.290+0000 7f058e39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:15.347 INFO:tasks.rados.rados.0.smithi064.stdout:2748: done (1 left) 2024-03-04T06:19:15.348 INFO:tasks.rados.rados.0.smithi064.stdout:2749: read oid 2 snap 331 2024-03-04T06:19:15.348 INFO:tasks.rados.rados.0.smithi064.stdout:2749: expect (ObjNum 1147 snap 309 seq_num 1147) 2024-03-04T06:19:15.348 INFO:tasks.rados.rados.0.smithi064.stdout:2750: write (excl) oid 18 current snap is 341 2024-03-04T06:19:15.348 INFO:tasks.rados.rados.0.smithi064.stdout:2747: finishing write tid 1 to smithi06431743-11 2024-03-04T06:19:15.348 INFO:tasks.rados.rados.0.smithi064.stdout:2750: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2024-03-04T06:19:15.354 INFO:tasks.rados.rados.0.smithi064.stdout:2750: writing smithi06431743-18 from 793627 to 1280861 tid 1 2024-03-04T06:19:15.360 INFO:tasks.rados.rados.0.smithi064.stdout:2750: writing smithi06431743-18 from 1790970 to 2195742 tid 2 2024-03-04T06:19:15.366 INFO:tasks.rados.rados.0.smithi064.stdout:2750: writing smithi06431743-18 from 2829335 to 3193627 tid 3 2024-03-04T06:19:15.366 INFO:tasks.rados.rados.0.smithi064.stdout:2751: snap_remove snap 340 2024-03-04T06:19:16.355 INFO:tasks.rados.rados.0.smithi064.stdout:2751: done (3 left) 2024-03-04T06:19:16.355 INFO:tasks.rados.rados.0.smithi064.stdout:2752: read oid 39 snap -1 2024-03-04T06:19:16.355 INFO:tasks.rados.rados.0.smithi064.stdout:2752: expect (ObjNum 1207 snap 321 seq_num 1207) 2024-03-04T06:19:16.355 INFO:tasks.rados.rados.0.smithi064.stdout:2753: write (excl) oid 16 current snap is 341 2024-03-04T06:19:16.355 INFO:tasks.rados.rados.0.smithi064.stdout:2747: finishing write tid 2 to smithi06431743-11 2024-03-04T06:19:16.355 INFO:tasks.rados.rados.0.smithi064.stdout:2753: seq_num 1262 ranges {410434=678836,1565805=548430,2573111=637323} 2024-03-04T06:19:16.360 INFO:tasks.rados.rados.0.smithi064.stdout:2753: writing smithi06431743-16 from 410434 to 1089270 tid 1 2024-03-04T06:19:16.366 INFO:tasks.rados.rados.0.smithi064.stdout:2753: writing smithi06431743-16 from 1565805 to 2114235 tid 2 2024-03-04T06:19:16.373 INFO:tasks.rados.rados.0.smithi064.stdout:2753: writing smithi06431743-16 from 2573111 to 3210434 tid 3 2024-03-04T06:19:16.373 INFO:tasks.rados.rados.0.smithi064.stdout:2754: read oid 21 snap -1 2024-03-04T06:19:16.373 INFO:tasks.rados.rados.0.smithi064.stdout:2754: expect (ObjNum 1250 snap 336 seq_num 1250) 2024-03-04T06:19:16.373 INFO:tasks.rados.rados.0.smithi064.stdout:2755: write (excl) oid 36 current snap is 341 2024-03-04T06:19:16.373 INFO:tasks.rados.rados.0.smithi064.stdout:2747: finishing write tid 3 to smithi06431743-11 2024-03-04T06:19:16.374 INFO:tasks.rados.rados.0.smithi064.stdout:2747: finishing write tid 4 to smithi06431743-11 2024-03-04T06:19:16.374 INFO:tasks.rados.rados.0.smithi064.stdout:2747: finishing write tid 5 to smithi06431743-11 2024-03-04T06:19:16.374 INFO:tasks.rados.rados.0.smithi064.stdout:2747: oid 11 updating version 0 to 652 2024-03-04T06:19:16.374 INFO:tasks.rados.rados.0.smithi064.stdout:2747: oid 11 version 652 is already newer than 651 2024-03-04T06:19:16.374 INFO:tasks.rados.rados.0.smithi064.stdout:2747: oid 11 updating version 652 to 654 2024-03-04T06:19:16.374 INFO:tasks.rados.rados.0.smithi064.stdout:2747: oid 11 version 654 is already newer than 654 2024-03-04T06:19:16.374 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 654 (ObjNum 1260 snap 341 seq_num 1260) dirty exists 2024-03-04T06:19:16.374 INFO:tasks.rados.rados.0.smithi064.stdout:2747: left oid 11 (ObjNum 1260 snap 341 seq_num 1260) 2024-03-04T06:19:16.374 INFO:tasks.rados.rados.0.smithi064.stdout:2755: seq_num 1263 ranges {427241=754085,1624287=408441,2600534=549964} 2024-03-04T06:19:16.378 INFO:tasks.rados.rados.0.smithi064.stdout:2755: writing smithi06431743-36 from 427241 to 1181326 tid 1 2024-03-04T06:19:16.383 INFO:tasks.rados.rados.0.smithi064.stdout:2755: writing smithi06431743-36 from 1624287 to 2032728 tid 2 2024-03-04T06:19:16.390 INFO:tasks.rados.rados.0.smithi064.stdout:2755: writing smithi06431743-36 from 2600534 to 3150498 tid 3 2024-03-04T06:19:16.390 INFO:tasks.rados.rados.0.smithi064.stdout:2747: done (6 left) 2024-03-04T06:19:16.390 INFO:tasks.rados.rados.0.smithi064.stdout:2756: write oid 8 current snap is 341 2024-03-04T06:19:16.390 INFO:tasks.rados.rados.0.smithi064.stdout:2756: seq_num 1264 ranges {444048=429334,1282769=552099,2627957=616091} 2024-03-04T06:19:16.394 INFO:tasks.rados.rados.0.smithi064.stdout:2756: writing smithi06431743-8 from 444048 to 873382 tid 1 2024-03-04T06:19:16.400 INFO:tasks.rados.rados.0.smithi064.stdout:2756: writing smithi06431743-8 from 1282769 to 1834868 tid 2 2024-03-04T06:19:16.411 INFO:tasks.rados.rados.0.smithi064.stdout:2756: writing smithi06431743-8 from 2627957 to 3244048 tid 3 2024-03-04T06:19:16.411 INFO:tasks.rados.rados.0.smithi064.stdout:2757: read oid 12 snap -1 2024-03-04T06:19:16.411 INFO:tasks.rados.rados.0.smithi064.stdout:2757: expect (ObjNum 1253 snap 337 seq_num 1253) 2024-03-04T06:19:16.411 INFO:tasks.rados.rados.0.smithi064.stdout:2758: write oid 10 current snap is 341 2024-03-04T06:19:16.411 INFO:tasks.rados.rados.0.smithi064.stdout:2758: seq_num 1265 ranges {460855=504583,1624898=412110,2539027=572508} 2024-03-04T06:19:16.418 INFO:tasks.rados.rados.0.smithi064.stdout:2758: writing smithi06431743-10 from 460855 to 965438 tid 1 2024-03-04T06:19:16.425 INFO:tasks.rados.rados.0.smithi064.stdout:2758: writing smithi06431743-10 from 1624898 to 2037008 tid 2 2024-03-04T06:19:16.431 INFO:tasks.rados.rados.0.smithi064.stdout:2758: writing smithi06431743-10 from 2539027 to 3111535 tid 3 2024-03-04T06:19:16.431 INFO:tasks.rados.rados.0.smithi064.stdout:2759: snap_create 2024-03-04T06:19:16.431 INFO:tasks.rados.rados.0.smithi064.stdout:2749: expect (ObjNum 1147 snap 309 seq_num 1147) 2024-03-04T06:19:16.619 INFO:tasks.rados.rados.0.smithi064.stdout:2750: finishing write tid 1 to smithi06431743-18 2024-03-04T06:19:16.620 INFO:tasks.rados.rados.0.smithi064.stdout:2750: finishing write tid 2 to smithi06431743-18 2024-03-04T06:19:16.620 INFO:tasks.rados.rados.0.smithi064.stdout:2750: finishing write tid 3 to smithi06431743-18 2024-03-04T06:19:16.620 INFO:tasks.rados.rados.0.smithi064.stdout:2750: finishing write tid 4 to smithi06431743-18 2024-03-04T06:19:16.620 INFO:tasks.rados.rados.0.smithi064.stdout:2750: finishing write tid 5 to smithi06431743-18 2024-03-04T06:19:16.620 INFO:tasks.rados.rados.0.smithi064.stdout:2750: oid 18 updating version 0 to 517 2024-03-04T06:19:16.620 INFO:tasks.rados.rados.0.smithi064.stdout:2750: oid 18 version 517 is already newer than 516 2024-03-04T06:19:16.620 INFO:tasks.rados.rados.0.smithi064.stdout:2750: oid 18 updating version 517 to 518 2024-03-04T06:19:16.620 INFO:tasks.rados.rados.0.smithi064.stdout:2750: oid 18 updating version 518 to 519 2024-03-04T06:19:16.620 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 519 (ObjNum 1261 snap 341 seq_num 1261) dirty exists 2024-03-04T06:19:16.620 INFO:tasks.rados.rados.0.smithi064.stdout:2750: left oid 18 (ObjNum 1261 snap 341 seq_num 1261) 2024-03-04T06:19:16.620 INFO:tasks.rados.rados.0.smithi064.stdout:2752: expect (ObjNum 1207 snap 321 seq_num 1207) 2024-03-04T06:19:16.752 INFO:tasks.rados.rados.0.smithi064.stdout:2754: expect (ObjNum 1250 snap 336 seq_num 1250) 2024-03-04T06:19:16.901 INFO:tasks.rados.rados.0.smithi064.stdout:2753: finishing write tid 1 to smithi06431743-16 2024-03-04T06:19:16.901 INFO:tasks.rados.rados.0.smithi064.stdout:2757: expect (ObjNum 1253 snap 337 seq_num 1253) 2024-03-04T06:19:17.105 INFO:tasks.rados.rados.0.smithi064.stdout:2758: finishing write tid 1 to smithi06431743-10 2024-03-04T06:19:17.105 INFO:tasks.rados.rados.0.smithi064.stdout:2753: finishing write tid 2 to smithi06431743-16 2024-03-04T06:19:17.105 INFO:tasks.rados.rados.0.smithi064.stdout:2758: finishing write tid 2 to smithi06431743-10 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2758: finishing write tid 3 to smithi06431743-10 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2753: finishing write tid 3 to smithi06431743-16 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2758: finishing write tid 4 to smithi06431743-10 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2758: finishing write tid 5 to smithi06431743-10 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2758: oid 10 updating version 0 to 697 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2758: oid 10 version 697 is already newer than 697 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2758: oid 10 version 697 is already newer than 695 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2758: oid 10 version 697 is already newer than 695 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 697 (ObjNum 1265 snap 341 seq_num 1265) dirty exists 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2758: left oid 10 (ObjNum 1265 snap 341 seq_num 1265) 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2753: finishing write tid 4 to smithi06431743-16 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2753: finishing write tid 5 to smithi06431743-16 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2753: oid 16 updating version 0 to 625 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2753: oid 16 updating version 625 to 628 2024-03-04T06:19:17.106 INFO:tasks.rados.rados.0.smithi064.stdout:2753: oid 16 version 628 is already newer than 626 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2753: oid 16 version 628 is already newer than 627 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 628 (ObjNum 1262 snap 341 seq_num 1262) dirty exists 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2753: left oid 16 (ObjNum 1262 snap 341 seq_num 1262) 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2755: finishing write tid 1 to smithi06431743-36 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2755: finishing write tid 2 to smithi06431743-36 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2755: finishing write tid 3 to smithi06431743-36 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2755: finishing write tid 4 to smithi06431743-36 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2755: finishing write tid 5 to smithi06431743-36 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2755: oid 36 updating version 0 to 630 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2755: oid 36 updating version 630 to 633 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2755: oid 36 version 633 is already newer than 633 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2755: oid 36 version 633 is already newer than 633 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 633 (ObjNum 1263 snap 341 seq_num 1263) dirty exists 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2755: left oid 36 (ObjNum 1263 snap 341 seq_num 1263) 2024-03-04T06:19:17.107 INFO:tasks.rados.rados.0.smithi064.stdout:2756: finishing write tid 1 to smithi06431743-8 2024-03-04T06:19:17.108 INFO:tasks.rados.rados.0.smithi064.stdout:2756: finishing write tid 2 to smithi06431743-8 2024-03-04T06:19:17.108 INFO:tasks.rados.rados.0.smithi064.stdout:2756: finishing write tid 3 to smithi06431743-8 2024-03-04T06:19:17.108 INFO:tasks.rados.rados.0.smithi064.stdout:2756: finishing write tid 4 to smithi06431743-8 2024-03-04T06:19:17.108 INFO:tasks.rados.rados.0.smithi064.stdout:2756: finishing write tid 5 to smithi06431743-8 2024-03-04T06:19:17.108 INFO:tasks.rados.rados.0.smithi064.stdout:2756: oid 8 updating version 0 to 637 2024-03-04T06:19:17.108 INFO:tasks.rados.rados.0.smithi064.stdout:2756: oid 8 version 637 is already newer than 637 2024-03-04T06:19:17.108 INFO:tasks.rados.rados.0.smithi064.stdout:2756: oid 8 version 637 is already newer than 637 2024-03-04T06:19:17.108 INFO:tasks.rados.rados.0.smithi064.stdout:2756: oid 8 version 637 is already newer than 637 2024-03-04T06:19:17.108 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 637 (ObjNum 1264 snap 341 seq_num 1264) dirty exists 2024-03-04T06:19:17.108 INFO:tasks.rados.rados.0.smithi064.stdout:2756: left oid 8 (ObjNum 1264 snap 341 seq_num 1264) 2024-03-04T06:19:17.357 INFO:tasks.rados.rados.0.smithi064.stdout:2749: done (9 left) 2024-03-04T06:19:17.357 INFO:tasks.rados.rados.0.smithi064.stdout:2750: done (8 left) 2024-03-04T06:19:17.357 INFO:tasks.rados.rados.0.smithi064.stdout:2752: done (7 left) 2024-03-04T06:19:17.357 INFO:tasks.rados.rados.0.smithi064.stdout:2753: done (6 left) 2024-03-04T06:19:17.358 INFO:tasks.rados.rados.0.smithi064.stdout:2754: done (5 left) 2024-03-04T06:19:17.358 INFO:tasks.rados.rados.0.smithi064.stdout:2755: done (4 left) 2024-03-04T06:19:17.358 INFO:tasks.rados.rados.0.smithi064.stdout:2756: done (3 left) 2024-03-04T06:19:17.358 INFO:tasks.rados.rados.0.smithi064.stdout:2757: done (2 left) 2024-03-04T06:19:17.358 INFO:tasks.rados.rados.0.smithi064.stdout:2758: done (1 left) 2024-03-04T06:19:17.358 INFO:tasks.rados.rados.0.smithi064.stdout:2759: done (0 left) 2024-03-04T06:19:17.358 INFO:tasks.rados.rados.0.smithi064.stdout:2760: snap_create 2024-03-04T06:19:17.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:17.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:17.465+0000 7f46ff39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:17.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:17.469+0000 7f46ff39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:18.289 INFO:tasks.rados.rados.0.smithi064.stdout:2760: done (0 left) 2024-03-04T06:19:18.290 INFO:tasks.rados.rados.0.smithi064.stdout:2761: snap_create 2024-03-04T06:19:19.248 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:19:19.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:19:19.291 INFO:tasks.rados.rados.0.smithi064.stdout:2761: done (0 left) 2024-03-04T06:19:19.291 INFO:tasks.rados.rados.0.smithi064.stdout:2762: read oid 11 snap 339 2024-03-04T06:19:19.291 INFO:tasks.rados.rados.0.smithi064.stdout:2762: expect deleted 2024-03-04T06:19:19.291 INFO:tasks.rados.rados.0.smithi064.stdout:2763: write oid 45 current snap is 344 2024-03-04T06:19:19.291 INFO:tasks.rados.rados.0.smithi064.stdout:2763: seq_num 1266 ranges {477662=579832,1683380=555768,2966450=311212} 2024-03-04T06:19:19.297 INFO:tasks.rados.rados.0.smithi064.stdout:2763: writing smithi06431743-45 from 477662 to 1057494 tid 1 2024-03-04T06:19:19.304 INFO:tasks.rados.rados.0.smithi064.stdout:2763: writing smithi06431743-45 from 1683380 to 2239148 tid 2 2024-03-04T06:19:19.309 INFO:tasks.rados.rados.0.smithi064.stdout:2763: writing smithi06431743-45 from 2966450 to 3277662 tid 3 2024-03-04T06:19:19.309 INFO:tasks.rados.rados.0.smithi064.stdout:2764: delete oid 32 current snap is 344 2024-03-04T06:19:19.346 INFO:tasks.rados.rados.0.smithi064.stdout:2762: done (2 left) 2024-03-04T06:19:19.346 INFO:tasks.rados.rados.0.smithi064.stdout:2764: done (1 left) 2024-03-04T06:19:19.346 INFO:tasks.rados.rados.0.smithi064.stdout:2765: delete oid 32 current snap is 344 2024-03-04T06:19:19.351 INFO:tasks.rados.rados.0.smithi064.stdout:2765: done (1 left) 2024-03-04T06:19:19.351 INFO:tasks.rados.rados.0.smithi064.stdout:2766: write (excl) oid 38 current snap is 344 2024-03-04T06:19:19.351 INFO:tasks.rados.rados.0.smithi064.stdout:2766: seq_num 1269 ranges {528083=405579,1342473=419448,2248719=733949} 2024-03-04T06:19:19.358 INFO:tasks.rados.rados.0.smithi064.stdout:2766: writing smithi06431743-38 from 528083 to 933662 tid 1 2024-03-04T06:19:19.365 INFO:tasks.rados.rados.0.smithi064.stdout:2766: writing smithi06431743-38 from 1342473 to 1761921 tid 2 2024-03-04T06:19:19.374 INFO:tasks.rados.rados.0.smithi064.stdout:2766: writing smithi06431743-38 from 2248719 to 2982668 tid 3 2024-03-04T06:19:19.374 INFO:tasks.rados.rados.0.smithi064.stdout:2767: read oid 7 snap -1 2024-03-04T06:19:19.374 INFO:tasks.rados.rados.0.smithi064.stdout:2767: expect (ObjNum 1201 snap 319 seq_num 1201) 2024-03-04T06:19:19.374 INFO:tasks.rados.rados.0.smithi064.stdout:2768: delete oid 48 current snap is 344 2024-03-04T06:19:19.374 INFO:tasks.rados.rados.0.smithi064.stdout:2763: finishing write tid 1 to smithi06431743-45 2024-03-04T06:19:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:19.381+0000 7ff3fb95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:19.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:19.393+0000 7ff3fb95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:19.397 INFO:tasks.rados.rados.0.smithi064.stdout:2767: expect (ObjNum 1201 snap 319 seq_num 1201) 2024-03-04T06:19:19.514 INFO:tasks.rados.rados.0.smithi064.stdout:2767: done (3 left) 2024-03-04T06:19:19.514 INFO:tasks.rados.rados.0.smithi064.stdout:2768: done (2 left) 2024-03-04T06:19:19.514 INFO:tasks.rados.rados.0.smithi064.stdout:2769: read oid 16 snap -1 2024-03-04T06:19:19.514 INFO:tasks.rados.rados.0.smithi064.stdout:2769: expect (ObjNum 1262 snap 341 seq_num 1262) 2024-03-04T06:19:19.514 INFO:tasks.rados.rados.0.smithi064.stdout:2770: snap_create 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2763: finishing write tid 2 to smithi06431743-45 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2763: finishing write tid 3 to smithi06431743-45 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2763: finishing write tid 4 to smithi06431743-45 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2763: finishing write tid 5 to smithi06431743-45 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2763: oid 45 updating version 0 to 661 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2763: oid 45 version 661 is already newer than 660 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2763: oid 45 version 661 is already newer than 658 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2763: oid 45 version 661 is already newer than 660 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 661 (ObjNum 1266 snap 344 seq_num 1266) dirty exists 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2763: left oid 45 (ObjNum 1266 snap 344 seq_num 1266) 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2766: finishing write tid 1 to smithi06431743-38 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2766: finishing write tid 2 to smithi06431743-38 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2766: finishing write tid 3 to smithi06431743-38 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2766: finishing write tid 4 to smithi06431743-38 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2766: finishing write tid 5 to smithi06431743-38 2024-03-04T06:19:19.515 INFO:tasks.rados.rados.0.smithi064.stdout:2766: oid 38 updating version 0 to 701 2024-03-04T06:19:19.516 INFO:tasks.rados.rados.0.smithi064.stdout:2766: oid 38 updating version 701 to 702 2024-03-04T06:19:19.516 INFO:tasks.rados.rados.0.smithi064.stdout:2766: oid 38 version 702 is already newer than 701 2024-03-04T06:19:19.516 INFO:tasks.rados.rados.0.smithi064.stdout:2766: oid 38 version 702 is already newer than 699 2024-03-04T06:19:19.516 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 702 (ObjNum 1269 snap 344 seq_num 1269) dirty exists 2024-03-04T06:19:19.516 INFO:tasks.rados.rados.0.smithi064.stdout:2766: left oid 38 (ObjNum 1269 snap 344 seq_num 1269) 2024-03-04T06:19:19.531 INFO:tasks.rados.rados.0.smithi064.stdout:2769: expect (ObjNum 1262 snap 341 seq_num 1262) 2024-03-04T06:19:19.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:19.646 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:19.644+0000 7fc01551d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:19.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:19.649+0000 7fc01551d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:20.359 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:19:20.362 INFO:tasks.rados.rados.0.smithi064.stdout:2763: done (3 left) 2024-03-04T06:19:20.362 INFO:tasks.rados.rados.0.smithi064.stdout:2766: done (2 left) 2024-03-04T06:19:20.362 INFO:tasks.rados.rados.0.smithi064.stdout:2769: done (1 left) 2024-03-04T06:19:20.362 INFO:tasks.rados.rados.0.smithi064.stdout:2770: done (0 left) 2024-03-04T06:19:20.363 INFO:tasks.rados.rados.0.smithi064.stdout:2771: write (excl) oid 43 current snap is 345 2024-03-04T06:19:20.363 INFO:tasks.rados.rados.0.smithi064.stdout:2771: seq_num 1271 ranges {561697=672430,1975790=706764,3103565=258132} 2024-03-04T06:19:20.369 INFO:tasks.rados.rados.0.smithi064.stdout:2771: writing smithi06431743-43 from 561697 to 1234127 tid 1 2024-03-04T06:19:20.377 INFO:tasks.rados.rados.0.smithi064.stdout:2771: writing smithi06431743-43 from 1975790 to 2682554 tid 2 2024-03-04T06:19:20.378 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:19:20.382 INFO:tasks.rados.rados.0.smithi064.stdout:2771: writing smithi06431743-43 from 3103565 to 3361697 tid 3 2024-03-04T06:19:20.382 INFO:tasks.rados.rados.0.smithi064.stdout:2772: read oid 21 snap -1 2024-03-04T06:19:20.382 INFO:tasks.rados.rados.0.smithi064.stdout:2772: expect (ObjNum 1250 snap 336 seq_num 1250) 2024-03-04T06:19:20.382 INFO:tasks.rados.rados.0.smithi064.stdout:2773: snap_create 2024-03-04T06:19:20.419 INFO:tasks.rados.rados.0.smithi064.stdout:2772: expect (ObjNum 1250 snap 336 seq_num 1250) 2024-03-04T06:19:20.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:20.516+0000 7fc1cbd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:20.524 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:20.524+0000 7fc1cbd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:20.584 INFO:tasks.rados.rados.0.smithi064.stdout:2771: finishing write tid 1 to smithi06431743-43 2024-03-04T06:19:20.584 INFO:tasks.rados.rados.0.smithi064.stdout:2771: finishing write tid 2 to smithi06431743-43 2024-03-04T06:19:20.584 INFO:tasks.rados.rados.0.smithi064.stdout:2771: finishing write tid 3 to smithi06431743-43 2024-03-04T06:19:20.584 INFO:tasks.rados.rados.0.smithi064.stdout:2771: finishing write tid 4 to smithi06431743-43 2024-03-04T06:19:20.584 INFO:tasks.rados.rados.0.smithi064.stdout:2771: finishing write tid 5 to smithi06431743-43 2024-03-04T06:19:20.585 INFO:tasks.rados.rados.0.smithi064.stdout:2771: oid 43 updating version 0 to 593 2024-03-04T06:19:20.585 INFO:tasks.rados.rados.0.smithi064.stdout:2771: oid 43 updating version 593 to 594 2024-03-04T06:19:20.585 INFO:tasks.rados.rados.0.smithi064.stdout:2771: oid 43 updating version 594 to 595 2024-03-04T06:19:20.585 INFO:tasks.rados.rados.0.smithi064.stdout:2771: oid 43 version 595 is already newer than 592 2024-03-04T06:19:20.585 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 595 (ObjNum 1271 snap 345 seq_num 1271) dirty exists 2024-03-04T06:19:20.585 INFO:tasks.rados.rados.0.smithi064.stdout:2771: left oid 43 (ObjNum 1271 snap 345 seq_num 1271) 2024-03-04T06:19:21.362 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:19:21.364 INFO:tasks.rados.rados.0.smithi064.stdout:2771: done (2 left) 2024-03-04T06:19:21.364 INFO:tasks.rados.rados.0.smithi064.stdout:2772: done (1 left) 2024-03-04T06:19:21.365 INFO:tasks.rados.rados.0.smithi064.stdout:2773: done (0 left) 2024-03-04T06:19:21.365 INFO:tasks.rados.rados.0.smithi064.stdout:2774: write (excl) oid 8 current snap is 346 2024-03-04T06:19:21.365 INFO:tasks.rados.rados.0.smithi064.stdout:2774: seq_num 1272 ranges {578504=747679,2034272=566775,3130988=247516} 2024-03-04T06:19:21.372 INFO:tasks.rados.rados.0.smithi064.stdout:2774: writing smithi06431743-8 from 578504 to 1326183 tid 1 2024-03-04T06:19:21.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:19:21.379 INFO:tasks.rados.rados.0.smithi064.stdout:2774: writing smithi06431743-8 from 2034272 to 2601047 tid 2 2024-03-04T06:19:21.383 INFO:tasks.rados.rados.0.smithi064.stdout:2774: writing smithi06431743-8 from 3130988 to 3378504 tid 3 2024-03-04T06:19:21.383 INFO:tasks.rados.rados.0.smithi064.stdout:2775: delete oid 25 current snap is 346 2024-03-04T06:19:21.422 INFO:tasks.rados.rados.0.smithi064.stdout:2774: finishing write tid 1 to smithi06431743-8 2024-03-04T06:19:21.441 INFO:tasks.rados.rados.0.smithi064.stdout:2775: done (1 left) 2024-03-04T06:19:21.441 INFO:tasks.rados.rados.0.smithi064.stdout:2776: snap_remove snap 320 2024-03-04T06:19:21.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:21.490+0000 7fb795f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:21.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:21.498+0000 7fb795f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:21.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:21.836 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:21.834+0000 7fdcfd35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:21.839 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:21.838+0000 7fdcfd35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:22.362 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:19:22.365 INFO:tasks.rados.rados.0.smithi064.stdout:2776: done (1 left) 2024-03-04T06:19:22.365 INFO:tasks.rados.rados.0.smithi064.stdout:2777: read oid 45 snap -1 2024-03-04T06:19:22.365 INFO:tasks.rados.rados.0.smithi064.stdout:2777: expect (ObjNum 1266 snap 344 seq_num 1266) 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2778: read oid 5 snap -1 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2778: expect deleted 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2779: delete oid 37 current snap is 346 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2774: finishing write tid 2 to smithi06431743-8 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2774: finishing write tid 3 to smithi06431743-8 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2774: finishing write tid 4 to smithi06431743-8 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2774: finishing write tid 5 to smithi06431743-8 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2774: oid 8 updating version 0 to 640 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2774: oid 8 updating version 640 to 642 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2774: oid 8 version 642 is already newer than 642 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2774: oid 8 version 642 is already newer than 639 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 642 (ObjNum 1272 snap 346 seq_num 1272) dirty exists 2024-03-04T06:19:22.366 INFO:tasks.rados.rados.0.smithi064.stdout:2774: left oid 8 (ObjNum 1272 snap 346 seq_num 1272) 2024-03-04T06:19:22.378 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:19:22.431 INFO:tasks.rados.rados.0.smithi064.stdout:2777: expect (ObjNum 1266 snap 344 seq_num 1266) 2024-03-04T06:19:22.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:22.488+0000 7f1c38d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:22.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:22.496+0000 7f1c38d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:22.632 INFO:tasks.rados.rados.0.smithi064.stdout:2774: done (3 left) 2024-03-04T06:19:22.632 INFO:tasks.rados.rados.0.smithi064.stdout:2777: done (2 left) 2024-03-04T06:19:22.633 INFO:tasks.rados.rados.0.smithi064.stdout:2778: done (1 left) 2024-03-04T06:19:22.633 INFO:tasks.rados.rados.0.smithi064.stdout:2779: done (0 left) 2024-03-04T06:19:22.633 INFO:tasks.rados.rados.0.smithi064.stdout:2780: write (excl) oid 32 current snap is 346 2024-03-04T06:19:22.633 INFO:tasks.rados.rados.0.smithi064.stdout:2780: seq_num 1275 ranges {628925=573426,1693365=714102,3096904=332021} 2024-03-04T06:19:22.639 INFO:tasks.rados.rados.0.smithi064.stdout:2780: writing smithi06431743-32 from 628925 to 1202351 tid 1 2024-03-04T06:19:22.645 INFO:tasks.rados.rados.0.smithi064.stdout:2780: writing smithi06431743-32 from 1693365 to 2407467 tid 2 2024-03-04T06:19:22.651 INFO:tasks.rados.rados.0.smithi064.stdout:2780: writing smithi06431743-32 from 3096904 to 3428925 tid 3 2024-03-04T06:19:22.652 INFO:tasks.rados.rados.0.smithi064.stdout:2781: read oid 22 snap -1 2024-03-04T06:19:22.652 INFO:tasks.rados.rados.0.smithi064.stdout:2781: expect deleted 2024-03-04T06:19:22.652 INFO:tasks.rados.rados.0.smithi064.stdout:2782: read oid 28 snap 326 2024-03-04T06:19:22.652 INFO:tasks.rados.rados.0.smithi064.stdout:2782: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:19:22.652 INFO:tasks.rados.rados.0.smithi064.stdout:2783: write (excl) oid 24 current snap is 346 2024-03-04T06:19:22.652 INFO:tasks.rados.rados.0.smithi064.stdout:2783: seq_num 1276 ranges {645732=648675,1751847=574113,3124327=321405} 2024-03-04T06:19:22.658 INFO:tasks.rados.rados.0.smithi064.stdout:2783: writing smithi06431743-24 from 645732 to 1294407 tid 1 2024-03-04T06:19:22.664 INFO:tasks.rados.rados.0.smithi064.stdout:2783: writing smithi06431743-24 from 1751847 to 2325960 tid 2 2024-03-04T06:19:22.670 INFO:tasks.rados.rados.0.smithi064.stdout:2783: writing smithi06431743-24 from 3124327 to 3445732 tid 3 2024-03-04T06:19:22.670 INFO:tasks.rados.rados.0.smithi064.stdout:2784: write oid 4 current snap is 346 2024-03-04T06:19:22.670 INFO:tasks.rados.rados.0.smithi064.stdout:2784: seq_num 1277 ranges {662539=723924,2093976=717771,3435397=27142} 2024-03-04T06:19:22.677 INFO:tasks.rados.rados.0.smithi064.stdout:2784: writing smithi06431743-4 from 662539 to 1386463 tid 1 2024-03-04T06:19:22.687 INFO:tasks.rados.rados.0.smithi064.stdout:2784: writing smithi06431743-4 from 2093976 to 2811747 tid 2 2024-03-04T06:19:22.691 INFO:tasks.rados.rados.0.smithi064.stdout:2784: writing smithi06431743-4 from 3435397 to 3462539 tid 3 2024-03-04T06:19:22.691 INFO:tasks.rados.rados.0.smithi064.stdout:2785: snap_remove snap 294 2024-03-04T06:19:23.368 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:19:23.373 INFO:tasks.rados.rados.0.smithi064.stdout:2785: done (5 left) 2024-03-04T06:19:23.373 INFO:tasks.rados.rados.0.smithi064.stdout:2786: read oid 41 snap -1 2024-03-04T06:19:23.373 INFO:tasks.rados.rados.0.smithi064.stdout:2786: expect (ObjNum 1128 snap 299 seq_num 1128) 2024-03-04T06:19:23.373 INFO:tasks.rados.rados.0.smithi064.stdout:2787: snap_remove snap 339 2024-03-04T06:19:23.384 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:19:23.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:19:23.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:23.514+0000 7fb22986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:23.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:23.520+0000 7fb22986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:23.711 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:19:23.712 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:19:23.723 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1032,"stamp":"2024-03-04T06:19:23.371226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":113140732,"num_objects":938,"num_object_clones":744,"num_object_copies":2814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":155,"num_read":5143,"num_read_kb":3956875,"num_write":4358,"num_write_kb":1150319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10435,"ondisk_log_size":10435,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":749668,"kb_used_data":549380,"kb_used_omap":0,"kb_used_meta":200256,"kb_avail":376737692,"statfs":{"total":386547056640,"available":385779396608,"internally_reserved":0,"allocated":562565120,"data_stored":658086038,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":205062144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7026019,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":55,"num_read_kb":44348,"num_write":35,"num_write_kb":10680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.082031"},"pg_stats":[{"pgid":"3.f","version":"683'590","reported_seq":2374,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.379631+0000","last_change":"2024-03-04T06:19:16.474865+0000","last_active":"2024-03-04T06:19:19.379631+0000","last_peered":"2024-03-04T06:19:19.379631+0000","last_clean":"2024-03-04T06:19:19.379631+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:19:19.379631+0000","last_undegraded":"2024-03-04T06:19:19.379631+0000","last_fullsized":"2024-03-04T06:19:19.379631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790589,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":295,"num_read_kb":245918,"num_write":247,"num_write_kb":65848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"684'521","reported_seq":2326,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.299399+0000","last_change":"2024-03-04T06:19:16.390157+0000","last_active":"2024-03-04T06:19:19.299399+0000","last_peered":"2024-03-04T06:19:19.299399+0000","last_clean":"2024-03-04T06:19:19.299399+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:19:19.299399+0000","last_undegraded":"2024-03-04T06:19:19.299399+0000","last_fullsized":"2024-03-04T06:19:19.299399+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6034307,"num_objects":48,"num_object_clones":34,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":182,"num_read_kb":130654,"num_write":196,"num_write_kb":51814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"688'661","reported_seq":2486,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.393203+0000","last_change":"2024-03-04T06:19:18.313757+0000","last_active":"2024-03-04T06:19:20.393203+0000","last_peered":"2024-03-04T06:19:20.393203+0000","last_clean":"2024-03-04T06:19:20.393203+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:19:20.393203+0000","last_undegraded":"2024-03-04T06:19:20.393203+0000","last_fullsized":"2024-03-04T06:19:20.393203+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12656826,"num_objects":69,"num_object_clones":54,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":392,"num_read_kb":312748,"num_write":303,"num_write_kb":82815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"684'569","reported_seq":2334,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.377991+0000","last_change":"2024-03-04T06:19:16.459484+0000","last_active":"2024-03-04T06:19:19.377991+0000","last_peered":"2024-03-04T06:19:19.377991+0000","last_clean":"2024-03-04T06:19:19.377991+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:19:19.377991+0000","last_undegraded":"2024-03-04T06:19:19.377991+0000","last_fullsized":"2024-03-04T06:19:19.377991+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9009443,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":250,"num_read_kb":193639,"num_write":223,"num_write_kb":62446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"685'249","reported_seq":2169,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.428150+0000","last_change":"2024-03-04T06:19:18.487380+0000","last_active":"2024-03-04T06:19:20.428150+0000","last_peered":"2024-03-04T06:19:20.428150+0000","last_clean":"2024-03-04T06:19:20.428150+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:19:20.428150+0000","last_undegraded":"2024-03-04T06:19:20.428150+0000","last_fullsized":"2024-03-04T06:19:20.428150+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":84,"num_read_kb":57508,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"688'702","reported_seq":2431,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.450261+0000","last_change":"2024-03-04T06:19:16.474636+0000","last_active":"2024-03-04T06:19:19.450261+0000","last_peered":"2024-03-04T06:19:19.450261+0000","last_clean":"2024-03-04T06:19:19.450261+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:19:19.450261+0000","last_undegraded":"2024-03-04T06:19:19.450261+0000","last_fullsized":"2024-03-04T06:19:19.450261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10538847,"num_objects":58,"num_object_clones":49,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":332,"num_read_kb":261979,"num_write":293,"num_write_kb":85944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2020,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.383840+0000","last_change":"2024-03-04T06:19:16.474135+0000","last_active":"2024-03-04T06:19:19.383840+0000","last_peered":"2024-03-04T06:19:19.383840+0000","last_clean":"2024-03-04T06:19:19.383840+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:19:19.383840+0000","last_undegraded":"2024-03-04T06:19:19.383840+0000","last_fullsized":"2024-03-04T06:19:19.383840+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"685'394","reported_seq":2265,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.394176+0000","last_change":"2024-03-04T06:19:18.313231+0000","last_active":"2024-03-04T06:19:20.394176+0000","last_peered":"2024-03-04T06:19:20.394176+0000","last_clean":"2024-03-04T06:19:20.394176+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:19:20.394176+0000","last_undegraded":"2024-03-04T06:19:20.394176+0000","last_fullsized":"2024-03-04T06:19:20.394176+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672940,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":208,"num_read_kb":185580,"num_write":145,"num_write_kb":38810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1864,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.412609+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:19:20.412609+0000","last_peered":"2024-03-04T06:19:20.412609+0000","last_clean":"2024-03-04T06:19:20.412609+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:19:20.412609+0000","last_undegraded":"2024-03-04T06:19:20.412609+0000","last_fullsized":"2024-03-04T06:19:20.412609+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1725,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.402232+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:19:20.402232+0000","last_peered":"2024-03-04T06:19:20.402232+0000","last_clean":"2024-03-04T06:19:20.402232+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:19:20.402232+0000","last_undegraded":"2024-03-04T06:19:20.402232+0000","last_fullsized":"2024-03-04T06:19:20.402232+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"685'669","reported_seq":2433,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.397654+0000","last_change":"2024-03-04T06:19:18.321172+0000","last_active":"2024-03-04T06:19:20.397654+0000","last_peered":"2024-03-04T06:19:20.397654+0000","last_clean":"2024-03-04T06:19:20.397654+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:19:20.397654+0000","last_undegraded":"2024-03-04T06:19:20.397654+0000","last_fullsized":"2024-03-04T06:19:20.397654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546871,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":333,"num_read_kb":267601,"num_write":270,"num_write_kb":72688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1704,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.384870+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:19:19.384870+0000","last_peered":"2024-03-04T06:19:19.384870+0000","last_clean":"2024-03-04T06:19:19.384870+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:19:19.384870+0000","last_undegraded":"2024-03-04T06:19:19.384870+0000","last_fullsized":"2024-03-04T06:19:19.384870+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"685'409","reported_seq":2299,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.411682+0000","last_change":"2024-03-04T06:19:18.476479+0000","last_active":"2024-03-04T06:19:20.411682+0000","last_peered":"2024-03-04T06:19:20.411682+0000","last_clean":"2024-03-04T06:19:20.411682+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:19:20.411682+0000","last_undegraded":"2024-03-04T06:19:20.411682+0000","last_fullsized":"2024-03-04T06:19:20.411682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185207,"num_objects":39,"num_object_clones":31,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":246,"num_read_kb":196913,"num_write":156,"num_write_kb":43696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1725,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.381715+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:19:20.381715+0000","last_peered":"2024-03-04T06:19:20.381715+0000","last_clean":"2024-03-04T06:19:20.381715+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:19:20.381715+0000","last_undegraded":"2024-03-04T06:19:20.381715+0000","last_fullsized":"2024-03-04T06:19:20.381715+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"687'1176","reported_seq":2792,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.430861+0000","last_change":"2024-03-04T06:19:18.487839+0000","last_active":"2024-03-04T06:19:20.430861+0000","last_peered":"2024-03-04T06:19:20.430861+0000","last_clean":"2024-03-04T06:19:20.430861+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:19:20.430861+0000","last_undegraded":"2024-03-04T06:19:20.430861+0000","last_fullsized":"2024-03-04T06:19:20.430861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1176,"log_dups_size":0,"ondisk_log_size":1176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11581178,"num_objects":109,"num_object_clones":86,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":637,"num_read_kb":499308,"num_write":509,"num_write_kb":136052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1705,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.375338+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:19:19.375338+0000","last_peered":"2024-03-04T06:19:19.375338+0000","last_clean":"2024-03-04T06:19:19.375338+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:19:19.375338+0000","last_undegraded":"2024-03-04T06:19:19.375338+0000","last_fullsized":"2024-03-04T06:19:19.375338+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"684'1313","reported_seq":2762,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.303236+0000","last_change":"2024-03-04T06:19:16.390303+0000","last_active":"2024-03-04T06:19:19.303236+0000","last_peered":"2024-03-04T06:19:19.303236+0000","last_clean":"2024-03-04T06:19:19.303236+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:19:19.303236+0000","last_undegraded":"2024-03-04T06:19:19.303236+0000","last_fullsized":"2024-03-04T06:19:19.303236+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101832,"num_objects":116,"num_object_clones":93,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":508,"num_read_kb":366970,"num_write":526,"num_write_kb":138869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"687'663","reported_seq":2461,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.417274+0000","last_change":"2024-03-04T06:19:18.321091+0000","last_active":"2024-03-04T06:19:20.417274+0000","last_peered":"2024-03-04T06:19:20.417274+0000","last_clean":"2024-03-04T06:19:20.417274+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:19:20.417274+0000","last_undegraded":"2024-03-04T06:19:20.417274+0000","last_fullsized":"2024-03-04T06:19:20.417274+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5765395,"num_objects":57,"num_object_clones":45,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":392,"num_read_kb":329117,"num_write":267,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1754,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.313483+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:19:19.313483+0000","last_peered":"2024-03-04T06:19:19.313483+0000","last_clean":"2024-03-04T06:19:19.313483+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:19:19.313483+0000","last_undegraded":"2024-03-04T06:19:19.313483+0000","last_fullsized":"2024-03-04T06:19:19.313483+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"685'637","reported_seq":2450,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.524867+0000","last_change":"2024-03-04T06:19:16.395295+0000","last_active":"2024-03-04T06:19:19.524867+0000","last_peered":"2024-03-04T06:19:19.524867+0000","last_clean":"2024-03-04T06:19:19.524867+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:19:19.524867+0000","last_undegraded":"2024-03-04T06:19:19.524867+0000","last_fullsized":"2024-03-04T06:19:19.524867+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9681723,"num_objects":65,"num_object_clones":49,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":312,"num_read_kb":227755,"num_write":273,"num_write_kb":72762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1731,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.422178+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:19:20.422178+0000","last_peered":"2024-03-04T06:19:20.422178+0000","last_clean":"2024-03-04T06:19:20.422178+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:19:20.422178+0000","last_undegraded":"2024-03-04T06:19:20.422178+0000","last_fullsized":"2024-03-04T06:19:20.422178+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"687'871","reported_seq":2535,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.414817+0000","last_change":"2024-03-04T06:19:18.476287+0000","last_active":"2024-03-04T06:19:20.414817+0000","last_peered":"2024-03-04T06:19:20.414817+0000","last_clean":"2024-03-04T06:19:20.414817+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:19:20.414817+0000","last_undegraded":"2024-03-04T06:19:20.414817+0000","last_fullsized":"2024-03-04T06:19:20.414817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11866897,"num_objects":77,"num_object_clones":63,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":381,"num_read_kb":272264,"num_write":361,"num_write_kb":100132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1754,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.301656+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:19:19.301656+0000","last_peered":"2024-03-04T06:19:19.301656+0000","last_clean":"2024-03-04T06:19:19.301656+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:19:19.301656+0000","last_undegraded":"2024-03-04T06:19:19.301656+0000","last_fullsized":"2024-03-04T06:19:19.301656+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"684'914","reported_seq":2609,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.298409+0000","last_change":"2024-03-04T06:19:16.392536+0000","last_active":"2024-03-04T06:19:19.298409+0000","last_peered":"2024-03-04T06:19:19.298409+0000","last_clean":"2024-03-04T06:19:19.298409+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:19:19.298409+0000","last_undegraded":"2024-03-04T06:19:19.298409+0000","last_fullsized":"2024-03-04T06:19:19.298409+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3176787,"num_objects":79,"num_object_clones":69,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":488,"num_read_kb":408830,"num_write":355,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1725,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.396230+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:19:20.396230+0000","last_peered":"2024-03-04T06:19:20.396230+0000","last_clean":"2024-03-04T06:19:20.396230+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:19:20.396230+0000","last_undegraded":"2024-03-04T06:19:20.396230+0000","last_fullsized":"2024-03-04T06:19:20.396230+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":112550364,"num_objects":936,"num_object_clones":744,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":155,"num_read":5040,"num_read_kb":3956784,"num_write":4221,"num_write_kb":1148425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10338,"ondisk_log_size":10338,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706359,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":222672,"kb_used_data":169480,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94149168,"statfs":{"total":96636764160,"available":96408748032,"internally_reserved":0,"allocated":173547520,"data_stored":203199736,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706359,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":174120,"kb_used_data":125088,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94197720,"statfs":{"total":96636764160,"available":96458465280,"internally_reserved":0,"allocated":128090112,"data_stored":147185317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706358,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":183096,"kb_used_data":134064,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94188744,"statfs":{"total":96636764160,"available":96449273856,"internally_reserved":0,"allocated":137281536,"data_stored":160465532,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706358,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":169780,"kb_used_data":120748,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94202060,"statfs":{"total":96636764160,"available":96462909440,"internally_reserved":0,"allocated":123645952,"data_stored":147235453,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:19:23.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:19:23.831 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:23.828+0000 7ff5cdb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:23.834 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:23.834+0000 7ff5cdb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:23.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:24.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:24.019+0000 7f639af9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:24.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:24.023+0000 7f639af9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:24.025 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:19:24.025 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:19:24.038 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1032,"stamp":"2024-03-04T06:19:23.371226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":113140732,"num_objects":938,"num_object_clones":744,"num_object_copies":2814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":155,"num_read":5143,"num_read_kb":3956875,"num_write":4358,"num_write_kb":1150319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10435,"ondisk_log_size":10435,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":749668,"kb_used_data":549380,"kb_used_omap":0,"kb_used_meta":200256,"kb_avail":376737692,"statfs":{"total":386547056640,"available":385779396608,"internally_reserved":0,"allocated":562565120,"data_stored":658086038,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":205062144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7026019,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":55,"num_read_kb":44348,"num_write":35,"num_write_kb":10680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.082031"},"pg_stats":[{"pgid":"3.f","version":"683'590","reported_seq":2374,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.379631+0000","last_change":"2024-03-04T06:19:16.474865+0000","last_active":"2024-03-04T06:19:19.379631+0000","last_peered":"2024-03-04T06:19:19.379631+0000","last_clean":"2024-03-04T06:19:19.379631+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:19:19.379631+0000","last_undegraded":"2024-03-04T06:19:19.379631+0000","last_fullsized":"2024-03-04T06:19:19.379631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790589,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":295,"num_read_kb":245918,"num_write":247,"num_write_kb":65848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"684'521","reported_seq":2326,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.299399+0000","last_change":"2024-03-04T06:19:16.390157+0000","last_active":"2024-03-04T06:19:19.299399+0000","last_peered":"2024-03-04T06:19:19.299399+0000","last_clean":"2024-03-04T06:19:19.299399+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:19:19.299399+0000","last_undegraded":"2024-03-04T06:19:19.299399+0000","last_fullsized":"2024-03-04T06:19:19.299399+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6034307,"num_objects":48,"num_object_clones":34,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":182,"num_read_kb":130654,"num_write":196,"num_write_kb":51814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"688'661","reported_seq":2486,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.393203+0000","last_change":"2024-03-04T06:19:18.313757+0000","last_active":"2024-03-04T06:19:20.393203+0000","last_peered":"2024-03-04T06:19:20.393203+0000","last_clean":"2024-03-04T06:19:20.393203+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:19:20.393203+0000","last_undegraded":"2024-03-04T06:19:20.393203+0000","last_fullsized":"2024-03-04T06:19:20.393203+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12656826,"num_objects":69,"num_object_clones":54,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":392,"num_read_kb":312748,"num_write":303,"num_write_kb":82815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"684'569","reported_seq":2334,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.377991+0000","last_change":"2024-03-04T06:19:16.459484+0000","last_active":"2024-03-04T06:19:19.377991+0000","last_peered":"2024-03-04T06:19:19.377991+0000","last_clean":"2024-03-04T06:19:19.377991+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:19:19.377991+0000","last_undegraded":"2024-03-04T06:19:19.377991+0000","last_fullsized":"2024-03-04T06:19:19.377991+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9009443,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":250,"num_read_kb":193639,"num_write":223,"num_write_kb":62446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"685'249","reported_seq":2169,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.428150+0000","last_change":"2024-03-04T06:19:18.487380+0000","last_active":"2024-03-04T06:19:20.428150+0000","last_peered":"2024-03-04T06:19:20.428150+0000","last_clean":"2024-03-04T06:19:20.428150+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:19:20.428150+0000","last_undegraded":"2024-03-04T06:19:20.428150+0000","last_fullsized":"2024-03-04T06:19:20.428150+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":84,"num_read_kb":57508,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"688'702","reported_seq":2431,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.450261+0000","last_change":"2024-03-04T06:19:16.474636+0000","last_active":"2024-03-04T06:19:19.450261+0000","last_peered":"2024-03-04T06:19:19.450261+0000","last_clean":"2024-03-04T06:19:19.450261+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:19:19.450261+0000","last_undegraded":"2024-03-04T06:19:19.450261+0000","last_fullsized":"2024-03-04T06:19:19.450261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10538847,"num_objects":58,"num_object_clones":49,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":332,"num_read_kb":261979,"num_write":293,"num_write_kb":85944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2020,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.383840+0000","last_change":"2024-03-04T06:19:16.474135+0000","last_active":"2024-03-04T06:19:19.383840+0000","last_peered":"2024-03-04T06:19:19.383840+0000","last_clean":"2024-03-04T06:19:19.383840+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:19:19.383840+0000","last_undegraded":"2024-03-04T06:19:19.383840+0000","last_fullsized":"2024-03-04T06:19:19.383840+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"685'394","reported_seq":2265,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.394176+0000","last_change":"2024-03-04T06:19:18.313231+0000","last_active":"2024-03-04T06:19:20.394176+0000","last_peered":"2024-03-04T06:19:20.394176+0000","last_clean":"2024-03-04T06:19:20.394176+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:19:20.394176+0000","last_undegraded":"2024-03-04T06:19:20.394176+0000","last_fullsized":"2024-03-04T06:19:20.394176+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672940,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":208,"num_read_kb":185580,"num_write":145,"num_write_kb":38810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1864,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.412609+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:19:20.412609+0000","last_peered":"2024-03-04T06:19:20.412609+0000","last_clean":"2024-03-04T06:19:20.412609+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:19:20.412609+0000","last_undegraded":"2024-03-04T06:19:20.412609+0000","last_fullsized":"2024-03-04T06:19:20.412609+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1725,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.402232+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:19:20.402232+0000","last_peered":"2024-03-04T06:19:20.402232+0000","last_clean":"2024-03-04T06:19:20.402232+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:19:20.402232+0000","last_undegraded":"2024-03-04T06:19:20.402232+0000","last_fullsized":"2024-03-04T06:19:20.402232+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"685'669","reported_seq":2433,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.397654+0000","last_change":"2024-03-04T06:19:18.321172+0000","last_active":"2024-03-04T06:19:20.397654+0000","last_peered":"2024-03-04T06:19:20.397654+0000","last_clean":"2024-03-04T06:19:20.397654+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:19:20.397654+0000","last_undegraded":"2024-03-04T06:19:20.397654+0000","last_fullsized":"2024-03-04T06:19:20.397654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546871,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":333,"num_read_kb":267601,"num_write":270,"num_write_kb":72688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1704,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.384870+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:19:19.384870+0000","last_peered":"2024-03-04T06:19:19.384870+0000","last_clean":"2024-03-04T06:19:19.384870+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:19:19.384870+0000","last_undegraded":"2024-03-04T06:19:19.384870+0000","last_fullsized":"2024-03-04T06:19:19.384870+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"685'409","reported_seq":2299,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.411682+0000","last_change":"2024-03-04T06:19:18.476479+0000","last_active":"2024-03-04T06:19:20.411682+0000","last_peered":"2024-03-04T06:19:20.411682+0000","last_clean":"2024-03-04T06:19:20.411682+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:19:20.411682+0000","last_undegraded":"2024-03-04T06:19:20.411682+0000","last_fullsized":"2024-03-04T06:19:20.411682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185207,"num_objects":39,"num_object_clones":31,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":246,"num_read_kb":196913,"num_write":156,"num_write_kb":43696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1725,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.381715+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:19:20.381715+0000","last_peered":"2024-03-04T06:19:20.381715+0000","last_clean":"2024-03-04T06:19:20.381715+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:19:20.381715+0000","last_undegraded":"2024-03-04T06:19:20.381715+0000","last_fullsized":"2024-03-04T06:19:20.381715+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"687'1176","reported_seq":2792,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.430861+0000","last_change":"2024-03-04T06:19:18.487839+0000","last_active":"2024-03-04T06:19:20.430861+0000","last_peered":"2024-03-04T06:19:20.430861+0000","last_clean":"2024-03-04T06:19:20.430861+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:19:20.430861+0000","last_undegraded":"2024-03-04T06:19:20.430861+0000","last_fullsized":"2024-03-04T06:19:20.430861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1176,"log_dups_size":0,"ondisk_log_size":1176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11581178,"num_objects":109,"num_object_clones":86,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":637,"num_read_kb":499308,"num_write":509,"num_write_kb":136052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1705,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.375338+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:19:19.375338+0000","last_peered":"2024-03-04T06:19:19.375338+0000","last_clean":"2024-03-04T06:19:19.375338+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:19:19.375338+0000","last_undegraded":"2024-03-04T06:19:19.375338+0000","last_fullsized":"2024-03-04T06:19:19.375338+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"684'1313","reported_seq":2762,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.303236+0000","last_change":"2024-03-04T06:19:16.390303+0000","last_active":"2024-03-04T06:19:19.303236+0000","last_peered":"2024-03-04T06:19:19.303236+0000","last_clean":"2024-03-04T06:19:19.303236+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:19:19.303236+0000","last_undegraded":"2024-03-04T06:19:19.303236+0000","last_fullsized":"2024-03-04T06:19:19.303236+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101832,"num_objects":116,"num_object_clones":93,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":508,"num_read_kb":366970,"num_write":526,"num_write_kb":138869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"687'663","reported_seq":2461,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.417274+0000","last_change":"2024-03-04T06:19:18.321091+0000","last_active":"2024-03-04T06:19:20.417274+0000","last_peered":"2024-03-04T06:19:20.417274+0000","last_clean":"2024-03-04T06:19:20.417274+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:19:20.417274+0000","last_undegraded":"2024-03-04T06:19:20.417274+0000","last_fullsized":"2024-03-04T06:19:20.417274+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5765395,"num_objects":57,"num_object_clones":45,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":392,"num_read_kb":329117,"num_write":267,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1754,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.313483+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:19:19.313483+0000","last_peered":"2024-03-04T06:19:19.313483+0000","last_clean":"2024-03-04T06:19:19.313483+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:19:19.313483+0000","last_undegraded":"2024-03-04T06:19:19.313483+0000","last_fullsized":"2024-03-04T06:19:19.313483+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"685'637","reported_seq":2450,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.524867+0000","last_change":"2024-03-04T06:19:16.395295+0000","last_active":"2024-03-04T06:19:19.524867+0000","last_peered":"2024-03-04T06:19:19.524867+0000","last_clean":"2024-03-04T06:19:19.524867+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:19:19.524867+0000","last_undegraded":"2024-03-04T06:19:19.524867+0000","last_fullsized":"2024-03-04T06:19:19.524867+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9681723,"num_objects":65,"num_object_clones":49,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":312,"num_read_kb":227755,"num_write":273,"num_write_kb":72762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1731,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.422178+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:19:20.422178+0000","last_peered":"2024-03-04T06:19:20.422178+0000","last_clean":"2024-03-04T06:19:20.422178+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:19:20.422178+0000","last_undegraded":"2024-03-04T06:19:20.422178+0000","last_fullsized":"2024-03-04T06:19:20.422178+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"687'871","reported_seq":2535,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.414817+0000","last_change":"2024-03-04T06:19:18.476287+0000","last_active":"2024-03-04T06:19:20.414817+0000","last_peered":"2024-03-04T06:19:20.414817+0000","last_clean":"2024-03-04T06:19:20.414817+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:19:20.414817+0000","last_undegraded":"2024-03-04T06:19:20.414817+0000","last_fullsized":"2024-03-04T06:19:20.414817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11866897,"num_objects":77,"num_object_clones":63,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":381,"num_read_kb":272264,"num_write":361,"num_write_kb":100132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1754,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.301656+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:19:19.301656+0000","last_peered":"2024-03-04T06:19:19.301656+0000","last_clean":"2024-03-04T06:19:19.301656+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:19:19.301656+0000","last_undegraded":"2024-03-04T06:19:19.301656+0000","last_fullsized":"2024-03-04T06:19:19.301656+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"684'914","reported_seq":2609,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.298409+0000","last_change":"2024-03-04T06:19:16.392536+0000","last_active":"2024-03-04T06:19:19.298409+0000","last_peered":"2024-03-04T06:19:19.298409+0000","last_clean":"2024-03-04T06:19:19.298409+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:19:19.298409+0000","last_undegraded":"2024-03-04T06:19:19.298409+0000","last_fullsized":"2024-03-04T06:19:19.298409+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3176787,"num_objects":79,"num_object_clones":69,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":488,"num_read_kb":408830,"num_write":355,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1725,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.396230+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:19:20.396230+0000","last_peered":"2024-03-04T06:19:20.396230+0000","last_clean":"2024-03-04T06:19:20.396230+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:19:20.396230+0000","last_undegraded":"2024-03-04T06:19:20.396230+0000","last_fullsized":"2024-03-04T06:19:20.396230+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":112550364,"num_objects":936,"num_object_clones":744,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":155,"num_read":5040,"num_read_kb":3956784,"num_write":4221,"num_write_kb":1148425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10338,"ondisk_log_size":10338,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706359,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":222672,"kb_used_data":169480,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94149168,"statfs":{"total":96636764160,"available":96408748032,"internally_reserved":0,"allocated":173547520,"data_stored":203199736,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706359,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":174120,"kb_used_data":125088,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94197720,"statfs":{"total":96636764160,"available":96458465280,"internally_reserved":0,"allocated":128090112,"data_stored":147185317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706358,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":183096,"kb_used_data":134064,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94188744,"statfs":{"total":96636764160,"available":96449273856,"internally_reserved":0,"allocated":137281536,"data_stored":160465532,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706358,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":169780,"kb_used_data":120748,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94202060,"statfs":{"total":96636764160,"available":96462909440,"internally_reserved":0,"allocated":123645952,"data_stored":147235453,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:19:24.039 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:19:24.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:19:24.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:19:24.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:24.145+0000 7f7b1a064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:24.152 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:24.152+0000 7f7b1a064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:24.342 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:19:24.342 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:19:24.354 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1032,"stamp":"2024-03-04T06:19:23.371226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":113140732,"num_objects":938,"num_object_clones":744,"num_object_copies":2814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":155,"num_read":5143,"num_read_kb":3956875,"num_write":4358,"num_write_kb":1150319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10435,"ondisk_log_size":10435,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":749668,"kb_used_data":549380,"kb_used_omap":0,"kb_used_meta":200256,"kb_avail":376737692,"statfs":{"total":386547056640,"available":385779396608,"internally_reserved":0,"allocated":562565120,"data_stored":658086038,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":205062144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7026019,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":55,"num_read_kb":44348,"num_write":35,"num_write_kb":10680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.082031"},"pg_stats":[{"pgid":"3.f","version":"683'590","reported_seq":2374,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.379631+0000","last_change":"2024-03-04T06:19:16.474865+0000","last_active":"2024-03-04T06:19:19.379631+0000","last_peered":"2024-03-04T06:19:19.379631+0000","last_clean":"2024-03-04T06:19:19.379631+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:19:19.379631+0000","last_undegraded":"2024-03-04T06:19:19.379631+0000","last_fullsized":"2024-03-04T06:19:19.379631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790589,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":295,"num_read_kb":245918,"num_write":247,"num_write_kb":65848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"684'521","reported_seq":2326,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.299399+0000","last_change":"2024-03-04T06:19:16.390157+0000","last_active":"2024-03-04T06:19:19.299399+0000","last_peered":"2024-03-04T06:19:19.299399+0000","last_clean":"2024-03-04T06:19:19.299399+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:19:19.299399+0000","last_undegraded":"2024-03-04T06:19:19.299399+0000","last_fullsized":"2024-03-04T06:19:19.299399+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6034307,"num_objects":48,"num_object_clones":34,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":182,"num_read_kb":130654,"num_write":196,"num_write_kb":51814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"688'661","reported_seq":2486,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.393203+0000","last_change":"2024-03-04T06:19:18.313757+0000","last_active":"2024-03-04T06:19:20.393203+0000","last_peered":"2024-03-04T06:19:20.393203+0000","last_clean":"2024-03-04T06:19:20.393203+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:19:20.393203+0000","last_undegraded":"2024-03-04T06:19:20.393203+0000","last_fullsized":"2024-03-04T06:19:20.393203+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":661,"log_dups_size":0,"ondisk_log_size":661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12656826,"num_objects":69,"num_object_clones":54,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":392,"num_read_kb":312748,"num_write":303,"num_write_kb":82815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"684'569","reported_seq":2334,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.377991+0000","last_change":"2024-03-04T06:19:16.459484+0000","last_active":"2024-03-04T06:19:19.377991+0000","last_peered":"2024-03-04T06:19:19.377991+0000","last_clean":"2024-03-04T06:19:19.377991+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:19:19.377991+0000","last_undegraded":"2024-03-04T06:19:19.377991+0000","last_fullsized":"2024-03-04T06:19:19.377991+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9009443,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":250,"num_read_kb":193639,"num_write":223,"num_write_kb":62446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"685'249","reported_seq":2169,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.428150+0000","last_change":"2024-03-04T06:19:18.487380+0000","last_active":"2024-03-04T06:19:20.428150+0000","last_peered":"2024-03-04T06:19:20.428150+0000","last_clean":"2024-03-04T06:19:20.428150+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:19:20.428150+0000","last_undegraded":"2024-03-04T06:19:20.428150+0000","last_fullsized":"2024-03-04T06:19:20.428150+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":84,"num_read_kb":57508,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"688'702","reported_seq":2431,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.450261+0000","last_change":"2024-03-04T06:19:16.474636+0000","last_active":"2024-03-04T06:19:19.450261+0000","last_peered":"2024-03-04T06:19:19.450261+0000","last_clean":"2024-03-04T06:19:19.450261+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:19:19.450261+0000","last_undegraded":"2024-03-04T06:19:19.450261+0000","last_fullsized":"2024-03-04T06:19:19.450261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10538847,"num_objects":58,"num_object_clones":49,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":332,"num_read_kb":261979,"num_write":293,"num_write_kb":85944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2020,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.383840+0000","last_change":"2024-03-04T06:19:16.474135+0000","last_active":"2024-03-04T06:19:19.383840+0000","last_peered":"2024-03-04T06:19:19.383840+0000","last_clean":"2024-03-04T06:19:19.383840+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:19:19.383840+0000","last_undegraded":"2024-03-04T06:19:19.383840+0000","last_fullsized":"2024-03-04T06:19:19.383840+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"685'394","reported_seq":2265,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.394176+0000","last_change":"2024-03-04T06:19:18.313231+0000","last_active":"2024-03-04T06:19:20.394176+0000","last_peered":"2024-03-04T06:19:20.394176+0000","last_clean":"2024-03-04T06:19:20.394176+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:19:20.394176+0000","last_undegraded":"2024-03-04T06:19:20.394176+0000","last_fullsized":"2024-03-04T06:19:20.394176+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672940,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":208,"num_read_kb":185580,"num_write":145,"num_write_kb":38810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1864,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.412609+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:19:20.412609+0000","last_peered":"2024-03-04T06:19:20.412609+0000","last_clean":"2024-03-04T06:19:20.412609+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:19:20.412609+0000","last_undegraded":"2024-03-04T06:19:20.412609+0000","last_fullsized":"2024-03-04T06:19:20.412609+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1725,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.402232+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:19:20.402232+0000","last_peered":"2024-03-04T06:19:20.402232+0000","last_clean":"2024-03-04T06:19:20.402232+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:19:20.402232+0000","last_undegraded":"2024-03-04T06:19:20.402232+0000","last_fullsized":"2024-03-04T06:19:20.402232+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"685'669","reported_seq":2433,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.397654+0000","last_change":"2024-03-04T06:19:18.321172+0000","last_active":"2024-03-04T06:19:20.397654+0000","last_peered":"2024-03-04T06:19:20.397654+0000","last_clean":"2024-03-04T06:19:20.397654+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:19:20.397654+0000","last_undegraded":"2024-03-04T06:19:20.397654+0000","last_fullsized":"2024-03-04T06:19:20.397654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":669,"log_dups_size":0,"ondisk_log_size":669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5546871,"num_objects":60,"num_object_clones":47,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":333,"num_read_kb":267601,"num_write":270,"num_write_kb":72688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1704,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.384870+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:19:19.384870+0000","last_peered":"2024-03-04T06:19:19.384870+0000","last_clean":"2024-03-04T06:19:19.384870+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:19:19.384870+0000","last_undegraded":"2024-03-04T06:19:19.384870+0000","last_fullsized":"2024-03-04T06:19:19.384870+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"685'409","reported_seq":2299,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.411682+0000","last_change":"2024-03-04T06:19:18.476479+0000","last_active":"2024-03-04T06:19:20.411682+0000","last_peered":"2024-03-04T06:19:20.411682+0000","last_clean":"2024-03-04T06:19:20.411682+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:19:20.411682+0000","last_undegraded":"2024-03-04T06:19:20.411682+0000","last_fullsized":"2024-03-04T06:19:20.411682+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185207,"num_objects":39,"num_object_clones":31,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":246,"num_read_kb":196913,"num_write":156,"num_write_kb":43696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1725,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.381715+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:19:20.381715+0000","last_peered":"2024-03-04T06:19:20.381715+0000","last_clean":"2024-03-04T06:19:20.381715+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:19:20.381715+0000","last_undegraded":"2024-03-04T06:19:20.381715+0000","last_fullsized":"2024-03-04T06:19:20.381715+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"687'1176","reported_seq":2792,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.430861+0000","last_change":"2024-03-04T06:19:18.487839+0000","last_active":"2024-03-04T06:19:20.430861+0000","last_peered":"2024-03-04T06:19:20.430861+0000","last_clean":"2024-03-04T06:19:20.430861+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:19:20.430861+0000","last_undegraded":"2024-03-04T06:19:20.430861+0000","last_fullsized":"2024-03-04T06:19:20.430861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1176,"log_dups_size":0,"ondisk_log_size":1176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11581178,"num_objects":109,"num_object_clones":86,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":637,"num_read_kb":499308,"num_write":509,"num_write_kb":136052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1705,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.375338+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:19:19.375338+0000","last_peered":"2024-03-04T06:19:19.375338+0000","last_clean":"2024-03-04T06:19:19.375338+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:19:19.375338+0000","last_undegraded":"2024-03-04T06:19:19.375338+0000","last_fullsized":"2024-03-04T06:19:19.375338+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"684'1313","reported_seq":2762,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.303236+0000","last_change":"2024-03-04T06:19:16.390303+0000","last_active":"2024-03-04T06:19:19.303236+0000","last_peered":"2024-03-04T06:19:19.303236+0000","last_clean":"2024-03-04T06:19:19.303236+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:19:19.303236+0000","last_undegraded":"2024-03-04T06:19:19.303236+0000","last_fullsized":"2024-03-04T06:19:19.303236+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8101832,"num_objects":116,"num_object_clones":93,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":508,"num_read_kb":366970,"num_write":526,"num_write_kb":138869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"687'663","reported_seq":2461,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.417274+0000","last_change":"2024-03-04T06:19:18.321091+0000","last_active":"2024-03-04T06:19:20.417274+0000","last_peered":"2024-03-04T06:19:20.417274+0000","last_clean":"2024-03-04T06:19:20.417274+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:19:20.417274+0000","last_undegraded":"2024-03-04T06:19:20.417274+0000","last_fullsized":"2024-03-04T06:19:20.417274+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5765395,"num_objects":57,"num_object_clones":45,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":392,"num_read_kb":329117,"num_write":267,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1754,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.313483+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:19:19.313483+0000","last_peered":"2024-03-04T06:19:19.313483+0000","last_clean":"2024-03-04T06:19:19.313483+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:19:19.313483+0000","last_undegraded":"2024-03-04T06:19:19.313483+0000","last_fullsized":"2024-03-04T06:19:19.313483+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"685'637","reported_seq":2450,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.524867+0000","last_change":"2024-03-04T06:19:16.395295+0000","last_active":"2024-03-04T06:19:19.524867+0000","last_peered":"2024-03-04T06:19:19.524867+0000","last_clean":"2024-03-04T06:19:19.524867+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:19:19.524867+0000","last_undegraded":"2024-03-04T06:19:19.524867+0000","last_fullsized":"2024-03-04T06:19:19.524867+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9681723,"num_objects":65,"num_object_clones":49,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":312,"num_read_kb":227755,"num_write":273,"num_write_kb":72762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1731,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.422178+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:19:20.422178+0000","last_peered":"2024-03-04T06:19:20.422178+0000","last_clean":"2024-03-04T06:19:20.422178+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:19:20.422178+0000","last_undegraded":"2024-03-04T06:19:20.422178+0000","last_fullsized":"2024-03-04T06:19:20.422178+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"687'871","reported_seq":2535,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.414817+0000","last_change":"2024-03-04T06:19:18.476287+0000","last_active":"2024-03-04T06:19:20.414817+0000","last_peered":"2024-03-04T06:19:20.414817+0000","last_clean":"2024-03-04T06:19:20.414817+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:19:20.414817+0000","last_undegraded":"2024-03-04T06:19:20.414817+0000","last_fullsized":"2024-03-04T06:19:20.414817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11866897,"num_objects":77,"num_object_clones":63,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":381,"num_read_kb":272264,"num_write":361,"num_write_kb":100132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1754,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.301656+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:19:19.301656+0000","last_peered":"2024-03-04T06:19:19.301656+0000","last_clean":"2024-03-04T06:19:19.301656+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:19:19.301656+0000","last_undegraded":"2024-03-04T06:19:19.301656+0000","last_fullsized":"2024-03-04T06:19:19.301656+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"684'914","reported_seq":2609,"reported_epoch":688,"state":"active+clean","last_fresh":"2024-03-04T06:19:19.298409+0000","last_change":"2024-03-04T06:19:16.392536+0000","last_active":"2024-03-04T06:19:19.298409+0000","last_peered":"2024-03-04T06:19:19.298409+0000","last_clean":"2024-03-04T06:19:19.298409+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:19:19.298409+0000","last_undegraded":"2024-03-04T06:19:19.298409+0000","last_fullsized":"2024-03-04T06:19:19.298409+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3176787,"num_objects":79,"num_object_clones":69,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":488,"num_read_kb":408830,"num_write":355,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1725,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-04T06:19:20.396230+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:19:20.396230+0000","last_peered":"2024-03-04T06:19:20.396230+0000","last_clean":"2024-03-04T06:19:20.396230+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:19:20.396230+0000","last_undegraded":"2024-03-04T06:19:20.396230+0000","last_fullsized":"2024-03-04T06:19:20.396230+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":112550364,"num_objects":936,"num_object_clones":744,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":155,"num_read":5040,"num_read_kb":3956784,"num_write":4221,"num_write_kb":1148425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10338,"ondisk_log_size":10338,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706359,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":222672,"kb_used_data":169480,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94149168,"statfs":{"total":96636764160,"available":96408748032,"internally_reserved":0,"allocated":173547520,"data_stored":203199736,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706359,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":174120,"kb_used_data":125088,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94197720,"statfs":{"total":96636764160,"available":96458465280,"internally_reserved":0,"allocated":128090112,"data_stored":147185317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706358,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":183096,"kb_used_data":134064,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94188744,"statfs":{"total":96636764160,"available":96449273856,"internally_reserved":0,"allocated":137281536,"data_stored":160465532,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706358,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":169780,"kb_used_data":120748,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94202060,"statfs":{"total":96636764160,"available":96462909440,"internally_reserved":0,"allocated":123645952,"data_stored":147235453,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:19:24.372 INFO:tasks.rados.rados.0.smithi064.stdout:2787: done (6 left) 2024-03-04T06:19:24.372 INFO:tasks.rados.rados.0.smithi064.stdout:2788: write (excl) oid 25 current snap is 346 2024-03-04T06:19:24.372 INFO:tasks.rados.rados.0.smithi064.stdout:2788: seq_num 1278 ranges {679346=515526,1868811=577782,3179173=300173} 2024-03-04T06:19:24.378 INFO:tasks.rados.rados.0.smithi064.stdout:2788: writing smithi06431743-25 from 679346 to 1194872 tid 1 2024-03-04T06:19:24.385 INFO:tasks.rados.rados.0.smithi064.stdout:2788: writing smithi06431743-25 from 1868811 to 2446593 tid 2 2024-03-04T06:19:24.393 INFO:tasks.rados.rados.0.smithi064.stdout:2788: writing smithi06431743-25 from 3179173 to 3479346 tid 3 2024-03-04T06:19:24.393 INFO:tasks.rados.rados.0.smithi064.stdout:2789: delete oid 1 current snap is 346 2024-03-04T06:19:24.393 INFO:tasks.rados.rados.0.smithi064.stdout:2784: finishing write tid 1 to smithi06431743-4 2024-03-04T06:19:24.393 INFO:tasks.rados.rados.0.smithi064.stdout:2783: finishing write tid 1 to smithi06431743-24 2024-03-04T06:19:24.393 INFO:tasks.rados.rados.0.smithi064.stdout:2780: finishing write tid 1 to smithi06431743-32 2024-03-04T06:19:24.393 INFO:tasks.rados.rados.0.smithi064.stdout:2783: finishing write tid 2 to smithi06431743-24 2024-03-04T06:19:24.393 INFO:tasks.rados.rados.0.smithi064.stdout:2784: finishing write tid 2 to smithi06431743-4 2024-03-04T06:19:24.393 INFO:tasks.rados.rados.0.smithi064.stdout:2780: finishing write tid 2 to smithi06431743-32 2024-03-04T06:19:24.393 INFO:tasks.rados.rados.0.smithi064.stdout:2783: finishing write tid 3 to smithi06431743-24 2024-03-04T06:19:24.393 INFO:tasks.rados.rados.0.smithi064.stdout:2784: finishing write tid 3 to smithi06431743-4 2024-03-04T06:19:24.393 INFO:tasks.rados.rados.0.smithi064.stdout:2780: finishing write tid 3 to smithi06431743-32 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2783: finishing write tid 4 to smithi06431743-24 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2783: finishing write tid 5 to smithi06431743-24 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2783: oid 24 updating version 0 to 880 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2783: oid 24 version 880 is already newer than 878 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2783: oid 24 version 880 is already newer than 880 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2783: oid 24 version 880 is already newer than 877 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 880 (ObjNum 1276 snap 346 seq_num 1276) dirty exists 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2783: left oid 24 (ObjNum 1276 snap 346 seq_num 1276) 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2784: finishing write tid 4 to smithi06431743-4 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2784: finishing write tid 5 to smithi06431743-4 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2784: oid 4 updating version 0 to 1323 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2784: oid 4 version 1323 is already newer than 1323 2024-03-04T06:19:24.394 INFO:tasks.rados.rados.0.smithi064.stdout:2784: oid 4 updating version 1323 to 1324 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:2784: oid 4 version 1324 is already newer than 1321 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 1324 (ObjNum 1277 snap 346 seq_num 1277) dirty exists 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:2784: left oid 4 (ObjNum 1277 snap 346 seq_num 1277) 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:2780: finishing write tid 4 to smithi06431743-32 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:2780: finishing write tid 5 to smithi06431743-32 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:2780: oid 32 updating version 0 to 1184 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:2780: oid 32 version 1184 is already newer than 1183 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:2780: oid 32 version 1184 is already newer than 1181 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:2780: oid 32 version 1184 is already newer than 1183 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1184 (ObjNum 1275 snap 346 seq_num 1275) dirty exists 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:2780: left oid 32 (ObjNum 1275 snap 346 seq_num 1275) 2024-03-04T06:19:24.395 INFO:tasks.rados.rados.0.smithi064.stdout:2782: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:19:24.583 INFO:tasks.rados.rados.0.smithi064.stdout:2786: expect (ObjNum 1128 snap 299 seq_num 1128) 2024-03-04T06:19:24.771 INFO:tasks.rados.rados.0.smithi064.stdout:2788: finishing write tid 1 to smithi06431743-25 2024-03-04T06:19:24.771 INFO:tasks.rados.rados.0.smithi064.stdout:2788: finishing write tid 2 to smithi06431743-25 2024-03-04T06:19:24.771 INFO:tasks.rados.rados.0.smithi064.stdout:2788: finishing write tid 3 to smithi06431743-25 2024-03-04T06:19:24.771 INFO:tasks.rados.rados.0.smithi064.stdout:2788: finishing write tid 4 to smithi06431743-25 2024-03-04T06:19:24.771 INFO:tasks.rados.rados.0.smithi064.stdout:2788: finishing write tid 5 to smithi06431743-25 2024-03-04T06:19:24.772 INFO:tasks.rados.rados.0.smithi064.stdout:2788: oid 25 updating version 0 to 676 2024-03-04T06:19:24.772 INFO:tasks.rados.rados.0.smithi064.stdout:2788: oid 25 version 676 is already newer than 675 2024-03-04T06:19:24.772 INFO:tasks.rados.rados.0.smithi064.stdout:2788: oid 25 updating version 676 to 677 2024-03-04T06:19:24.772 INFO:tasks.rados.rados.0.smithi064.stdout:2788: oid 25 version 677 is already newer than 674 2024-03-04T06:19:24.772 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 677 (ObjNum 1278 snap 346 seq_num 1278) dirty exists 2024-03-04T06:19:24.772 INFO:tasks.rados.rados.0.smithi064.stdout:2788: left oid 25 (ObjNum 1278 snap 346 seq_num 1278) 2024-03-04T06:19:24.772 INFO:tasks.rados.rados.0.smithi064.stdout:2780: done (7 left) 2024-03-04T06:19:24.772 INFO:tasks.rados.rados.0.smithi064.stdout:2781: done (6 left) 2024-03-04T06:19:24.772 INFO:tasks.rados.rados.0.smithi064.stdout:2782: done (5 left) 2024-03-04T06:19:24.772 INFO:tasks.rados.rados.0.smithi064.stdout:2783: done (4 left) 2024-03-04T06:19:24.772 INFO:tasks.rados.rados.0.smithi064.stdout:2784: done (3 left) 2024-03-04T06:19:24.773 INFO:tasks.rados.rados.0.smithi064.stdout:2786: done (2 left) 2024-03-04T06:19:24.773 INFO:tasks.rados.rados.0.smithi064.stdout:2788: done (1 left) 2024-03-04T06:19:24.773 INFO:tasks.rados.rados.0.smithi064.stdout:2789: done (0 left) 2024-03-04T06:19:24.773 INFO:tasks.rados.rados.0.smithi064.stdout:2790: delete oid 41 current snap is 346 2024-03-04T06:19:24.792 INFO:tasks.rados.rados.0.smithi064.stdout:2790: done (0 left) 2024-03-04T06:19:24.792 INFO:tasks.rados.rados.0.smithi064.stdout:2791: read oid 10 snap -1 2024-03-04T06:19:24.792 INFO:tasks.rados.rados.0.smithi064.stdout:2791: expect (ObjNum 1265 snap 341 seq_num 1265) 2024-03-04T06:19:24.792 INFO:tasks.rados.rados.0.smithi064.stdout:2792: snap_remove snap 335 2024-03-04T06:19:25.377 INFO:tasks.rados.rados.0.smithi064.stdout:2792: done (1 left) 2024-03-04T06:19:25.378 INFO:tasks.rados.rados.0.smithi064.stdout:2793: snap_create 2024-03-04T06:19:25.378 INFO:tasks.rados.rados.0.smithi064.stdout:2791: expect (ObjNum 1265 snap 341 seq_num 1265) 2024-03-04T06:19:26.095 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:26.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:26.204+0000 7fdd03f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:26.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:26.208+0000 7fdd03f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:26.379 INFO:tasks.rados.rados.0.smithi064.stdout:2791: done (1 left) 2024-03-04T06:19:26.379 INFO:tasks.rados.rados.0.smithi064.stdout:2793: done (0 left) 2024-03-04T06:19:26.379 INFO:tasks.rados.rados.0.smithi064.stdout:2794: snap_create 2024-03-04T06:19:27.379 INFO:tasks.rados.rados.0.smithi064.stdout:2794: done (0 left) 2024-03-04T06:19:27.380 INFO:tasks.rados.rados.0.smithi064.stdout:2795: snap_remove snap 332 2024-03-04T06:19:28.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:28.292 INFO:tasks.rados.rados.0.smithi064.stdout:2795: done (0 left) 2024-03-04T06:19:28.292 INFO:tasks.rados.rados.0.smithi064.stdout:2796: snap_remove snap 310 2024-03-04T06:19:28.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:28.397+0000 7f8c0a75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:28.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:28.402+0000 7f8c0a75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:29.296 INFO:tasks.rados.rados.0.smithi064.stdout:2796: done (0 left) 2024-03-04T06:19:29.296 INFO:tasks.rados.rados.0.smithi064.stdout:2797: snap_remove snap 334 2024-03-04T06:19:29.355 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:19:29.356 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:19:29.356 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:19:29.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:19:29.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:29.460+0000 7fee88217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:29.468 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:29.468+0000 7fee88217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:29.480 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:29.480 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:29.481 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:29.481 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:29.481 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:29.481 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:29.481 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:29.481 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:29.481 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:29.481 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:29.481 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:29.481 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:29.483 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:29.484 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:29.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:19:29.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:29.588+0000 7f5044c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:29.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:29.595+0000 7f5044c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:29.608 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:29.609 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:29.609 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:29.609 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:29.609 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:29.609 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:29.609 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:29.609 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:29.609 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:29.609 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:29.612 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:29.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:19:29.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:29.715+0000 7f2e3339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:29.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:29.722+0000 7f2e3339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:29.736 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:29.736 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:29.736 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:29.736 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:29.736 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:29.736 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:29.737 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:29.737 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:29.737 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:29.737 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:29.737 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:29.737 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:29.739 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:29.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:19:29.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:29.842+0000 7fd8ddf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:29.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:29.849+0000 7fd8ddf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:29.865 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:29.865 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:29.865 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:29.866 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:29.867 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:30.297 INFO:tasks.rados.rados.0.smithi064.stdout:2797: done (0 left) 2024-03-04T06:19:30.297 INFO:tasks.rados.rados.0.smithi064.stdout:2798: snap_create 2024-03-04T06:19:30.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:30.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:30.576+0000 7f0ff9606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:30.581 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:30.580+0000 7f0ff9606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:31.302 INFO:tasks.rados.rados.0.smithi064.stdout:2798: done (0 left) 2024-03-04T06:19:31.302 INFO:tasks.rados.rados.0.smithi064.stdout:2799: write oid 2 current snap is 349 2024-03-04T06:19:31.303 INFO:tasks.rados.rados.0.smithi064.stdout:2799: seq_num 1281 ranges {729767=741273,1927904=725109,3145089=384678} 2024-03-04T06:19:31.310 INFO:tasks.rados.rados.0.smithi064.stdout:2799: writing smithi06431743-2 from 729767 to 1471040 tid 1 2024-03-04T06:19:31.317 INFO:tasks.rados.rados.0.smithi064.stdout:2799: writing smithi06431743-2 from 1927904 to 2653013 tid 2 2024-03-04T06:19:31.323 INFO:tasks.rados.rados.0.smithi064.stdout:2799: writing smithi06431743-2 from 3145089 to 3529767 tid 3 2024-03-04T06:19:31.323 INFO:tasks.rados.rados.0.smithi064.stdout:2800: write (excl) oid 32 current snap is 349 2024-03-04T06:19:31.323 INFO:tasks.rados.rados.0.smithi064.stdout:2800: seq_num 1282 ranges {746574=416522,1586386=468767,2772512=629544} 2024-03-04T06:19:31.329 INFO:tasks.rados.rados.0.smithi064.stdout:2800: writing smithi06431743-32 from 746574 to 1163096 tid 1 2024-03-04T06:19:31.334 INFO:tasks.rados.rados.0.smithi064.stdout:2800: writing smithi06431743-32 from 1586386 to 2055153 tid 2 2024-03-04T06:19:31.342 INFO:tasks.rados.rados.0.smithi064.stdout:2800: writing smithi06431743-32 from 2772512 to 3402056 tid 3 2024-03-04T06:19:31.342 INFO:tasks.rados.rados.0.smithi064.stdout:2801: write (excl) oid 8 current snap is 349 2024-03-04T06:19:31.342 INFO:tasks.rados.rados.0.smithi064.stdout:2801: seq_num 1283 ranges {763381=491771,2044868=728778,3199935=363446} 2024-03-04T06:19:31.349 INFO:tasks.rados.rados.0.smithi064.stdout:2801: writing smithi06431743-8 from 763381 to 1255152 tid 1 2024-03-04T06:19:31.360 INFO:tasks.rados.rados.0.smithi064.stdout:2801: writing smithi06431743-8 from 2044868 to 2773646 tid 2 2024-03-04T06:19:31.366 INFO:tasks.rados.rados.0.smithi064.stdout:2801: writing smithi06431743-8 from 3199935 to 3563381 tid 3 2024-03-04T06:19:31.366 INFO:tasks.rados.rados.0.smithi064.stdout:2802: write (excl) oid 46 current snap is 349 2024-03-04T06:19:31.366 INFO:tasks.rados.rados.0.smithi064.stdout:2802: seq_num 1284 ranges {780188=567020,2103350=472436,3227358=352830} 2024-03-04T06:19:31.375 INFO:tasks.rados.rados.0.smithi064.stdout:2802: writing smithi06431743-46 from 780188 to 1347208 tid 1 2024-03-04T06:19:31.383 INFO:tasks.rados.rados.0.smithi064.stdout:2802: writing smithi06431743-46 from 2103350 to 2575786 tid 2 2024-03-04T06:19:31.390 INFO:tasks.rados.rados.0.smithi064.stdout:2802: writing smithi06431743-46 from 3227358 to 3580188 tid 3 2024-03-04T06:19:31.390 INFO:tasks.rados.rados.0.smithi064.stdout:2803: read oid 33 snap 331 2024-03-04T06:19:31.391 INFO:tasks.rados.rados.0.smithi064.stdout:2803: expect deleted 2024-03-04T06:19:31.391 INFO:tasks.rados.rados.0.smithi064.stdout:2804: read oid 39 snap -1 2024-03-04T06:19:31.391 INFO:tasks.rados.rados.0.smithi064.stdout:2804: expect (ObjNum 1207 snap 321 seq_num 1207) 2024-03-04T06:19:31.391 INFO:tasks.rados.rados.0.smithi064.stdout:2805: read oid 28 snap -1 2024-03-04T06:19:31.391 INFO:tasks.rados.rados.0.smithi064.stdout:2805: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:19:31.391 INFO:tasks.rados.rados.0.smithi064.stdout:2806: snap_create 2024-03-04T06:19:31.431 INFO:tasks.rados.rados.0.smithi064.stdout:2799: finishing write tid 1 to smithi06431743-2 2024-03-04T06:19:31.436 INFO:tasks.rados.rados.0.smithi064.stdout:2804: expect (ObjNum 1207 snap 321 seq_num 1207) 2024-03-04T06:19:31.569 INFO:tasks.rados.rados.0.smithi064.stdout:2801: finishing write tid 1 to smithi06431743-8 2024-03-04T06:19:31.570 INFO:tasks.rados.rados.0.smithi064.stdout:2799: finishing write tid 2 to smithi06431743-2 2024-03-04T06:19:31.570 INFO:tasks.rados.rados.0.smithi064.stdout:2800: finishing write tid 1 to smithi06431743-32 2024-03-04T06:19:31.570 INFO:tasks.rados.rados.0.smithi064.stdout:2801: finishing write tid 2 to smithi06431743-8 2024-03-04T06:19:31.570 INFO:tasks.rados.rados.0.smithi064.stdout:2799: finishing write tid 3 to smithi06431743-2 2024-03-04T06:19:31.570 INFO:tasks.rados.rados.0.smithi064.stdout:2801: finishing write tid 3 to smithi06431743-8 2024-03-04T06:19:31.570 INFO:tasks.rados.rados.0.smithi064.stdout:2799: finishing write tid 4 to smithi06431743-2 2024-03-04T06:19:31.570 INFO:tasks.rados.rados.0.smithi064.stdout:2800: finishing write tid 2 to smithi06431743-32 2024-03-04T06:19:31.570 INFO:tasks.rados.rados.0.smithi064.stdout:2799: finishing write tid 5 to smithi06431743-2 2024-03-04T06:19:31.570 INFO:tasks.rados.rados.0.smithi064.stdout:2799: oid 2 updating version 0 to 590 2024-03-04T06:19:31.570 INFO:tasks.rados.rados.0.smithi064.stdout:2799: oid 2 version 590 is already newer than 590 2024-03-04T06:19:31.570 INFO:tasks.rados.rados.0.smithi064.stdout:2799: oid 2 version 590 is already newer than 588 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:2799: oid 2 version 590 is already newer than 587 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 590 (ObjNum 1281 snap 349 seq_num 1281) dirty exists 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:2799: left oid 2 (ObjNum 1281 snap 349 seq_num 1281) 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:2801: finishing write tid 4 to smithi06431743-8 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:2801: finishing write tid 5 to smithi06431743-8 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:2801: oid 8 updating version 0 to 679 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:2801: oid 8 updating version 679 to 681 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:2801: oid 8 version 681 is already newer than 681 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:2801: oid 8 version 681 is already newer than 678 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 681 (ObjNum 1283 snap 349 seq_num 1283) dirty exists 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:2801: left oid 8 (ObjNum 1283 snap 349 seq_num 1283) 2024-03-04T06:19:31.571 INFO:tasks.rados.rados.0.smithi064.stdout:2800: finishing write tid 3 to smithi06431743-32 2024-03-04T06:19:31.572 INFO:tasks.rados.rados.0.smithi064.stdout:2800: finishing write tid 4 to smithi06431743-32 2024-03-04T06:19:31.572 INFO:tasks.rados.rados.0.smithi064.stdout:2800: finishing write tid 5 to smithi06431743-32 2024-03-04T06:19:31.572 INFO:tasks.rados.rados.0.smithi064.stdout:2800: oid 32 updating version 0 to 1213 2024-03-04T06:19:31.572 INFO:tasks.rados.rados.0.smithi064.stdout:2800: oid 32 version 1213 is already newer than 1210 2024-03-04T06:19:31.572 INFO:tasks.rados.rados.0.smithi064.stdout:2800: oid 32 version 1213 is already newer than 1213 2024-03-04T06:19:31.572 INFO:tasks.rados.rados.0.smithi064.stdout:2800: oid 32 version 1213 is already newer than 1213 2024-03-04T06:19:31.572 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1213 (ObjNum 1282 snap 349 seq_num 1282) dirty exists 2024-03-04T06:19:31.572 INFO:tasks.rados.rados.0.smithi064.stdout:2800: left oid 32 (ObjNum 1282 snap 349 seq_num 1282) 2024-03-04T06:19:31.572 INFO:tasks.rados.rados.0.smithi064.stdout:2802: finishing write tid 1 to smithi06431743-46 2024-03-04T06:19:31.572 INFO:tasks.rados.rados.0.smithi064.stdout:2802: finishing write tid 2 to smithi06431743-46 2024-03-04T06:19:31.573 INFO:tasks.rados.rados.0.smithi064.stdout:2802: finishing write tid 3 to smithi06431743-46 2024-03-04T06:19:31.573 INFO:tasks.rados.rados.0.smithi064.stdout:2802: finishing write tid 4 to smithi06431743-46 2024-03-04T06:19:31.573 INFO:tasks.rados.rados.0.smithi064.stdout:2802: finishing write tid 5 to smithi06431743-46 2024-03-04T06:19:31.573 INFO:tasks.rados.rados.0.smithi064.stdout:2802: oid 46 updating version 0 to 1218 2024-03-04T06:19:31.574 INFO:tasks.rados.rados.0.smithi064.stdout:2802: oid 46 version 1218 is already newer than 1215 2024-03-04T06:19:31.574 INFO:tasks.rados.rados.0.smithi064.stdout:2802: oid 46 version 1218 is already newer than 1218 2024-03-04T06:19:31.574 INFO:tasks.rados.rados.0.smithi064.stdout:2802: oid 46 version 1218 is already newer than 1218 2024-03-04T06:19:31.574 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 1218 (ObjNum 1284 snap 349 seq_num 1284) dirty exists 2024-03-04T06:19:31.574 INFO:tasks.rados.rados.0.smithi064.stdout:2802: left oid 46 (ObjNum 1284 snap 349 seq_num 1284) 2024-03-04T06:19:31.584 INFO:tasks.rados.rados.0.smithi064.stdout:2805: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:19:32.305 INFO:tasks.rados.rados.0.smithi064.stdout:2799: done (7 left) 2024-03-04T06:19:32.305 INFO:tasks.rados.rados.0.smithi064.stdout:2800: done (6 left) 2024-03-04T06:19:32.305 INFO:tasks.rados.rados.0.smithi064.stdout:2801: done (5 left) 2024-03-04T06:19:32.305 INFO:tasks.rados.rados.0.smithi064.stdout:2802: done (4 left) 2024-03-04T06:19:32.305 INFO:tasks.rados.rados.0.smithi064.stdout:2803: done (3 left) 2024-03-04T06:19:32.306 INFO:tasks.rados.rados.0.smithi064.stdout:2804: done (2 left) 2024-03-04T06:19:32.306 INFO:tasks.rados.rados.0.smithi064.stdout:2805: done (1 left) 2024-03-04T06:19:32.306 INFO:tasks.rados.rados.0.smithi064.stdout:2806: done (0 left) 2024-03-04T06:19:32.306 INFO:tasks.rados.rados.0.smithi064.stdout:2807: read oid 7 snap -1 2024-03-04T06:19:32.306 INFO:tasks.rados.rados.0.smithi064.stdout:2807: expect (ObjNum 1201 snap 319 seq_num 1201) 2024-03-04T06:19:32.306 INFO:tasks.rados.rados.0.smithi064.stdout:2808: snap_create 2024-03-04T06:19:32.350 INFO:tasks.rados.rados.0.smithi064.stdout:2807: expect (ObjNum 1201 snap 319 seq_num 1201) 2024-03-04T06:19:32.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:32.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:32.753+0000 7feb6586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:32.758 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:32.757+0000 7feb6586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:33.311 INFO:tasks.rados.rados.0.smithi064.stdout:2807: done (1 left) 2024-03-04T06:19:33.312 INFO:tasks.rados.rados.0.smithi064.stdout:2808: done (0 left) 2024-03-04T06:19:33.312 INFO:tasks.rados.rados.0.smithi064.stdout:2809: snap_create 2024-03-04T06:19:34.313 INFO:tasks.rados.rados.0.smithi064.stdout:2809: done (0 left) 2024-03-04T06:19:34.313 INFO:tasks.rados.rados.0.smithi064.stdout:2810: write (excl) oid 19 current snap is 352 2024-03-04T06:19:34.313 INFO:tasks.rados.rados.0.smithi064.stdout:2810: seq_num 1285 ranges {796995=758622,2161832=732447} 2024-03-04T06:19:34.321 INFO:tasks.rados.rados.0.smithi064.stdout:2810: writing smithi06431743-19 from 796995 to 1555617 tid 1 2024-03-04T06:19:34.328 INFO:tasks.rados.rados.0.smithi064.stdout:2810: writing smithi06431743-19 from 2161832 to 2894279 tid 2 2024-03-04T06:19:34.329 INFO:tasks.rados.rados.0.smithi064.stdout:2811: write (excl) oid 35 current snap is 352 2024-03-04T06:19:34.329 INFO:tasks.rados.rados.0.smithi064.stdout:2811: seq_num 1286 ranges {413802=433871,1420314=476105,2482204=790985} 2024-03-04T06:19:34.333 INFO:tasks.rados.rados.0.smithi064.stdout:2811: writing smithi06431743-35 from 413802 to 847673 tid 1 2024-03-04T06:19:34.339 INFO:tasks.rados.rados.0.smithi064.stdout:2811: writing smithi06431743-35 from 1420314 to 1896419 tid 2 2024-03-04T06:19:34.346 INFO:tasks.rados.rados.0.smithi064.stdout:2811: writing smithi06431743-35 from 2482204 to 3273189 tid 3 2024-03-04T06:19:34.347 INFO:tasks.rados.rados.0.smithi064.stdout:2812: snap_create 2024-03-04T06:19:34.396 INFO:tasks.rados.rados.0.smithi064.stdout:2810: finishing write tid 1 to smithi06431743-19 2024-03-04T06:19:34.414 INFO:tasks.rados.rados.0.smithi064.stdout:2811: finishing write tid 1 to smithi06431743-35 2024-03-04T06:19:34.417 INFO:tasks.rados.rados.0.smithi064.stdout:2810: finishing write tid 2 to smithi06431743-19 2024-03-04T06:19:34.422 INFO:tasks.rados.rados.0.smithi064.stdout:2810: finishing write tid 3 to smithi06431743-19 2024-03-04T06:19:34.423 INFO:tasks.rados.rados.0.smithi064.stdout:2810: finishing write tid 4 to smithi06431743-19 2024-03-04T06:19:34.423 INFO:tasks.rados.rados.0.smithi064.stdout:2810: oid 19 updating version 0 to 1358 2024-03-04T06:19:34.423 INFO:tasks.rados.rados.0.smithi064.stdout:2810: oid 19 updating version 1358 to 1360 2024-03-04T06:19:34.424 INFO:tasks.rados.rados.0.smithi064.stdout:2810: oid 19 version 1360 is already newer than 1360 2024-03-04T06:19:34.424 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 1360 (ObjNum 1285 snap 352 seq_num 1285) dirty exists 2024-03-04T06:19:34.424 INFO:tasks.rados.rados.0.smithi064.stdout:2810: left oid 19 (ObjNum 1285 snap 352 seq_num 1285) 2024-03-04T06:19:34.432 INFO:tasks.rados.rados.0.smithi064.stdout:2811: finishing write tid 2 to smithi06431743-35 2024-03-04T06:19:34.451 INFO:tasks.rados.rados.0.smithi064.stdout:2811: finishing write tid 3 to smithi06431743-35 2024-03-04T06:19:34.488 INFO:tasks.rados.rados.0.smithi064.stdout:2811: finishing write tid 4 to smithi06431743-35 2024-03-04T06:19:34.489 INFO:tasks.rados.rados.0.smithi064.stdout:2811: finishing write tid 5 to smithi06431743-35 2024-03-04T06:19:34.489 INFO:tasks.rados.rados.0.smithi064.stdout:2811: oid 35 updating version 0 to 910 2024-03-04T06:19:34.489 INFO:tasks.rados.rados.0.smithi064.stdout:2811: oid 35 version 910 is already newer than 909 2024-03-04T06:19:34.489 INFO:tasks.rados.rados.0.smithi064.stdout:2811: oid 35 updating version 910 to 911 2024-03-04T06:19:34.489 INFO:tasks.rados.rados.0.smithi064.stdout:2811: oid 35 version 911 is already newer than 908 2024-03-04T06:19:34.489 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 911 (ObjNum 1286 snap 352 seq_num 1286) dirty exists 2024-03-04T06:19:34.489 INFO:tasks.rados.rados.0.smithi064.stdout:2811: left oid 35 (ObjNum 1286 snap 352 seq_num 1286) 2024-03-04T06:19:34.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:34.867 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:19:34.868 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:19:34.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:19:34.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:34.936+0000 7f2b4846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:34.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:34.941+0000 7f2b4846d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:34.986 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:34.982+0000 7f0aaccdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:34.989 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:34.989+0000 7f0aaccdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:35.182 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:19:35.182 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:19:35.194 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1049,"stamp":"2024-03-04T06:19:34.856207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":110047716,"num_objects":946,"num_object_clones":752,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":157,"num_read":5178,"num_read_kb":3986689,"num_write":4392,"num_write_kb":1159337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10721,"ondisk_log_size":10721,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":620852,"kb_used_data":420564,"kb_used_omap":0,"kb_used_meta":200256,"kb_avail":376866508,"statfs":{"total":386547056640,"available":385911304192,"internally_reserved":0,"allocated":430657536,"data_stored":491380202,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":205062144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000744"},"pg_stats":[{"pgid":"3.f","version":"695'603","reported_seq":2405,"reported_epoch":698,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:29.923655+0000","last_change":"2024-03-04T06:19:29.923655+0000","last_active":"2024-03-04T06:19:29.923655+0000","last_peered":"2024-03-04T06:19:29.923655+0000","last_clean":"2024-03-04T06:19:29.923655+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:19:29.923655+0000","last_undegraded":"2024-03-04T06:19:29.923655+0000","last_fullsized":"2024-03-04T06:19:29.923655+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7227571,"num_objects":61,"num_object_clones":48,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":297,"num_read_kb":245919,"num_write":252,"num_write_kb":67449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"699'543","reported_seq":2367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-03-04T06:19:30.303874+0000","last_change":"2024-03-04T06:19:30.336042+0000","last_active":"2024-03-04T06:19:30.303874+0000","last_peered":"2024-03-04T06:19:30.303874+0000","last_clean":"2024-03-04T06:19:30.303874+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:19:30.303874+0000","last_undegraded":"2024-03-04T06:19:30.303874+0000","last_fullsized":"2024-03-04T06:19:30.303874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3193627,"num_objects":49,"num_object_clones":35,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":182,"num_read_kb":130654,"num_write":197,"num_write_kb":51814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"698'685","reported_seq":2526,"reported_epoch":700,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:31.318211+0000","last_change":"2024-03-04T06:19:29.314038+0000","last_active":"2024-03-04T06:19:31.318211+0000","last_peered":"2024-03-04T06:19:31.318211+0000","last_clean":"2024-03-04T06:19:31.318211+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:19:31.318211+0000","last_undegraded":"2024-03-04T06:19:31.318211+0000","last_fullsized":"2024-03-04T06:19:31.318211+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12219877,"num_objects":70,"num_object_clones":55,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":400,"num_read_kb":321240,"num_write":309,"num_write_kb":84178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"695'579","reported_seq":2360,"reported_epoch":698,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:29.911232+0000","last_change":"2024-03-04T06:19:29.911232+0000","last_active":"2024-03-04T06:19:29.911232+0000","last_peered":"2024-03-04T06:19:29.911232+0000","last_clean":"2024-03-04T06:19:29.911232+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:19:29.911232+0000","last_undegraded":"2024-03-04T06:19:29.911232+0000","last_fullsized":"2024-03-04T06:19:29.911232+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9009443,"num_objects":46,"num_object_clones":36,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":250,"num_read_kb":193639,"num_write":223,"num_write_kb":62446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"698'255","reported_seq":2199,"reported_epoch":700,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:31.383026+0000","last_change":"2024-03-04T06:19:29.962736+0000","last_active":"2024-03-04T06:19:31.383026+0000","last_peered":"2024-03-04T06:19:31.383026+0000","last_clean":"2024-03-04T06:19:31.383026+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:19:31.383026+0000","last_undegraded":"2024-03-04T06:19:31.383026+0000","last_fullsized":"2024-03-04T06:19:31.383026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":84,"num_read_kb":57508,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"695'712","reported_seq":2460,"reported_epoch":698,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:29.926852+0000","last_change":"2024-03-04T06:19:29.926852+0000","last_active":"2024-03-04T06:19:29.926852+0000","last_peered":"2024-03-04T06:19:29.926852+0000","last_clean":"2024-03-04T06:19:29.926852+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:19:29.926852+0000","last_undegraded":"2024-03-04T06:19:29.926852+0000","last_fullsized":"2024-03-04T06:19:29.926852+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10538847,"num_objects":58,"num_object_clones":49,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":338,"num_read_kb":268798,"num_write":293,"num_write_kb":85944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2048,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-03-04T06:19:29.932526+0000","last_change":"2024-03-04T06:19:29.940251+0000","last_active":"2024-03-04T06:19:29.932526+0000","last_peered":"2024-03-04T06:19:29.932526+0000","last_clean":"2024-03-04T06:19:29.932526+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:19:29.932526+0000","last_undegraded":"2024-03-04T06:19:29.932526+0000","last_fullsized":"2024-03-04T06:19:29.932526+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"700'406","reported_seq":2298,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-03-04T06:19:31.352513+0000","last_change":"2024-03-04T06:19:31.358466+0000","last_active":"2024-03-04T06:19:31.352513+0000","last_peered":"2024-03-04T06:19:31.352513+0000","last_clean":"2024-03-04T06:19:31.352513+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:19:31.352513+0000","last_undegraded":"2024-03-04T06:19:31.352513+0000","last_fullsized":"2024-03-04T06:19:31.352513+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672940,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":208,"num_read_kb":185580,"num_write":145,"num_write_kb":38810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1889,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-03-04T06:19:31.371367+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:19:31.371367+0000","last_peered":"2024-03-04T06:19:31.371367+0000","last_clean":"2024-03-04T06:19:31.371367+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:19:31.371367+0000","last_undegraded":"2024-03-04T06:19:31.371367+0000","last_fullsized":"2024-03-04T06:19:31.371367+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1752,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-03-04T06:19:31.340593+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:19:31.340593+0000","last_peered":"2024-03-04T06:19:31.340593+0000","last_clean":"2024-03-04T06:19:31.340593+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:19:31.340593+0000","last_undegraded":"2024-03-04T06:19:31.340593+0000","last_fullsized":"2024-03-04T06:19:31.340593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"700'683","reported_seq":2470,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-03-04T06:19:31.334290+0000","last_change":"2024-03-04T06:19:31.383995+0000","last_active":"2024-03-04T06:19:31.334290+0000","last_peered":"2024-03-04T06:19:31.334290+0000","last_clean":"2024-03-04T06:19:31.334290+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:19:31.334290+0000","last_undegraded":"2024-03-04T06:19:31.334290+0000","last_fullsized":"2024-03-04T06:19:31.334290+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588575,"num_objects":61,"num_object_clones":48,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":339,"num_read_kb":275378,"num_write":271,"num_write_kb":72688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1726,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-03-04T06:19:29.921363+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:19:29.921363+0000","last_peered":"2024-03-04T06:19:29.921363+0000","last_clean":"2024-03-04T06:19:29.921363+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:19:29.921363+0000","last_undegraded":"2024-03-04T06:19:29.921363+0000","last_fullsized":"2024-03-04T06:19:29.921363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"698'415","reported_seq":2329,"reported_epoch":700,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:31.360152+0000","last_change":"2024-03-04T06:19:29.936438+0000","last_active":"2024-03-04T06:19:31.360152+0000","last_peered":"2024-03-04T06:19:31.360152+0000","last_clean":"2024-03-04T06:19:31.360152+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:19:31.360152+0000","last_undegraded":"2024-03-04T06:19:31.360152+0000","last_fullsized":"2024-03-04T06:19:31.360152+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185207,"num_objects":39,"num_object_clones":31,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":246,"num_read_kb":196913,"num_write":156,"num_write_kb":43696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1752,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-03-04T06:19:31.321964+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:19:31.321964+0000","last_peered":"2024-03-04T06:19:31.321964+0000","last_clean":"2024-03-04T06:19:31.321964+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:19:31.321964+0000","last_undegraded":"2024-03-04T06:19:31.321964+0000","last_fullsized":"2024-03-04T06:19:31.321964+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"698'1204","reported_seq":2830,"reported_epoch":700,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:31.384174+0000","last_change":"2024-03-04T06:19:29.955493+0000","last_active":"2024-03-04T06:19:31.384174+0000","last_peered":"2024-03-04T06:19:31.384174+0000","last_clean":"2024-03-04T06:19:31.384174+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:19:31.384174+0000","last_undegraded":"2024-03-04T06:19:31.384174+0000","last_fullsized":"2024-03-04T06:19:31.384174+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12051774,"num_objects":110,"num_object_clones":87,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":645,"num_read_kb":506030,"num_write":515,"num_write_kb":137635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1726,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-03-04T06:19:29.906460+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:19:29.906460+0000","last_peered":"2024-03-04T06:19:29.906460+0000","last_clean":"2024-03-04T06:19:29.906460+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:19:29.906460+0000","last_undegraded":"2024-03-04T06:19:29.906460+0000","last_fullsized":"2024-03-04T06:19:29.906460+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"699'1356","reported_seq":2807,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-03-04T06:19:30.305977+0000","last_change":"2024-03-04T06:19:30.338430+0000","last_active":"2024-03-04T06:19:30.305977+0000","last_peered":"2024-03-04T06:19:30.305977+0000","last_clean":"2024-03-04T06:19:30.305977+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:19:30.305977+0000","last_undegraded":"2024-03-04T06:19:30.305977+0000","last_fullsized":"2024-03-04T06:19:30.305977+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1356,"log_dups_size":0,"ondisk_log_size":1356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9261515,"num_objects":117,"num_object_clones":94,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":509,"num_read_kb":366971,"num_write":531,"num_write_kb":140304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"698'681","reported_seq":2492,"reported_epoch":700,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:31.339199+0000","last_change":"2024-03-04T06:19:29.336299+0000","last_active":"2024-03-04T06:19:31.339199+0000","last_peered":"2024-03-04T06:19:31.339199+0000","last_clean":"2024-03-04T06:19:31.339199+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:19:31.339199+0000","last_undegraded":"2024-03-04T06:19:31.339199+0000","last_fullsized":"2024-03-04T06:19:31.339199+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":681,"log_dups_size":0,"ondisk_log_size":681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5765395,"num_objects":57,"num_object_clones":45,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":392,"num_read_kb":329117,"num_write":267,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1787,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-03-04T06:19:30.314129+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:19:30.314129+0000","last_peered":"2024-03-04T06:19:30.314129+0000","last_clean":"2024-03-04T06:19:30.314129+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:19:30.314129+0000","last_undegraded":"2024-03-04T06:19:30.314129+0000","last_fullsized":"2024-03-04T06:19:30.314129+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"699'676","reported_seq":2495,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-03-04T06:19:30.312187+0000","last_change":"2024-03-04T06:19:30.350007+0000","last_active":"2024-03-04T06:19:30.312187+0000","last_peered":"2024-03-04T06:19:30.312187+0000","last_clean":"2024-03-04T06:19:30.312187+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:19:30.312187+0000","last_undegraded":"2024-03-04T06:19:30.312187+0000","last_fullsized":"2024-03-04T06:19:30.312187+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9816179,"num_objects":66,"num_object_clones":50,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":314,"num_read_kb":227756,"num_write":278,"num_write_kb":74289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1756,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-03-04T06:19:31.379345+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:19:31.379345+0000","last_peered":"2024-03-04T06:19:31.379345+0000","last_clean":"2024-03-04T06:19:31.379345+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:19:31.379345+0000","last_undegraded":"2024-03-04T06:19:31.379345+0000","last_fullsized":"2024-03-04T06:19:31.379345+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"698'898","reported_seq":2569,"reported_epoch":700,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:31.374468+0000","last_change":"2024-03-04T06:19:29.941640+0000","last_active":"2024-03-04T06:19:31.374468+0000","last_peered":"2024-03-04T06:19:31.374468+0000","last_clean":"2024-03-04T06:19:31.374468+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:19:31.374468+0000","last_undegraded":"2024-03-04T06:19:31.374468+0000","last_fullsized":"2024-03-04T06:19:31.374468+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":898,"log_dups_size":0,"ondisk_log_size":898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12808089,"num_objects":78,"num_object_clones":64,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":383,"num_read_kb":272265,"num_write":366,"num_write_kb":101641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1787,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-03-04T06:19:30.306596+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:19:30.306596+0000","last_peered":"2024-03-04T06:19:30.306596+0000","last_clean":"2024-03-04T06:19:30.306596+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:19:30.306596+0000","last_undegraded":"2024-03-04T06:19:30.306596+0000","last_fullsized":"2024-03-04T06:19:30.306596+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"699'928","reported_seq":2649,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-03-04T06:19:30.305213+0000","last_change":"2024-03-04T06:19:30.332220+0000","last_active":"2024-03-04T06:19:30.305213+0000","last_peered":"2024-03-04T06:19:30.305213+0000","last_clean":"2024-03-04T06:19:30.305213+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:19:30.305213+0000","last_undegraded":"2024-03-04T06:19:30.305213+0000","last_fullsized":"2024-03-04T06:19:30.305213+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":928,"log_dups_size":0,"ondisk_log_size":928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3176787,"num_objects":79,"num_object_clones":69,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":488,"num_read_kb":408830,"num_write":355,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1752,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-03-04T06:19:31.327725+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:19:31.327725+0000","last_peered":"2024-03-04T06:19:31.327725+0000","last_clean":"2024-03-04T06:19:31.327725+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:19:31.327725+0000","last_undegraded":"2024-03-04T06:19:31.327725+0000","last_fullsized":"2024-03-04T06:19:31.327725+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":109457348,"num_objects":944,"num_object_clones":752,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":157,"num_read":5075,"num_read_kb":3986598,"num_write":4255,"num_write_kb":1157443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10624,"ondisk_log_size":10624,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706369,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":172880,"kb_used_data":119688,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94198960,"statfs":{"total":96636764160,"available":96459735040,"internally_reserved":0,"allocated":122560512,"data_stored":138316209,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706369,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":140300,"kb_used_data":91268,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94231540,"statfs":{"total":96636764160,"available":96493096960,"internally_reserved":0,"allocated":93458432,"data_stored":104152805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154544,"kb_used_data":105512,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94217296,"statfs":{"total":96636764160,"available":96478511104,"internally_reserved":0,"allocated":108044288,"data_stored":122747045,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153128,"kb_used_data":104096,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94218712,"statfs":{"total":96636764160,"available":96479961088,"internally_reserved":0,"allocated":106594304,"data_stored":126164143,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:19:35.316 INFO:tasks.rados.rados.0.smithi064.stdout:2810: done (2 left) 2024-03-04T06:19:35.316 INFO:tasks.rados.rados.0.smithi064.stdout:2811: done (1 left) 2024-03-04T06:19:35.316 INFO:tasks.rados.rados.0.smithi064.stdout:2812: done (0 left) 2024-03-04T06:19:35.316 INFO:tasks.rados.rados.0.smithi064.stdout:2813: read oid 50 snap -1 2024-03-04T06:19:35.316 INFO:tasks.rados.rados.0.smithi064.stdout:2813: expect (ObjNum 1141 snap 306 seq_num 1141) 2024-03-04T06:19:35.316 INFO:tasks.rados.rados.0.smithi064.stdout:2814: read oid 24 snap -1 2024-03-04T06:19:35.316 INFO:tasks.rados.rados.0.smithi064.stdout:2814: expect (ObjNum 1276 snap 346 seq_num 1276) 2024-03-04T06:19:35.317 INFO:tasks.rados.rados.0.smithi064.stdout:2815: delete oid 46 current snap is 353 2024-03-04T06:19:35.371 INFO:tasks.rados.rados.0.smithi064.stdout:2813: expect (ObjNum 1141 snap 306 seq_num 1141) 2024-03-04T06:19:35.622 INFO:tasks.rados.rados.0.smithi064.stdout:2814: expect (ObjNum 1276 snap 346 seq_num 1276) 2024-03-04T06:19:35.788 INFO:tasks.rados.rados.0.smithi064.stdout:2813: done (2 left) 2024-03-04T06:19:35.788 INFO:tasks.rados.rados.0.smithi064.stdout:2814: done (1 left) 2024-03-04T06:19:35.788 INFO:tasks.rados.rados.0.smithi064.stdout:2815: done (0 left) 2024-03-04T06:19:35.788 INFO:tasks.rados.rados.0.smithi064.stdout:2816: write (excl) oid 42 current snap is 353 2024-03-04T06:19:35.788 INFO:tasks.rados.rados.0.smithi064.stdout:2816: seq_num 1288 ranges {447416=584369,1537278=479774,2537050=529882,3524784=122632} 2024-03-04T06:19:35.793 INFO:tasks.rados.rados.0.smithi064.stdout:2816: writing smithi06431743-42 from 447416 to 1031785 tid 1 2024-03-04T06:19:35.799 INFO:tasks.rados.rados.0.smithi064.stdout:2816: writing smithi06431743-42 from 1537278 to 2017052 tid 2 2024-03-04T06:19:35.804 INFO:tasks.rados.rados.0.smithi064.stdout:2816: writing smithi06431743-42 from 2537050 to 3066932 tid 3 2024-03-04T06:19:35.808 INFO:tasks.rados.rados.0.smithi064.stdout:2816: writing smithi06431743-42 from 3524784 to 3647416 tid 4 2024-03-04T06:19:35.808 INFO:tasks.rados.rados.0.smithi064.stdout:2817: delete oid 29 current snap is 353 2024-03-04T06:19:35.811 INFO:tasks.rados.rados.0.smithi064.stdout:2817: done (1 left) 2024-03-04T06:19:35.811 INFO:tasks.rados.rados.0.smithi064.stdout:2818: snap_create 2024-03-04T06:19:35.844 INFO:tasks.rados.rados.0.smithi064.stdout:2816: finishing write tid 1 to smithi06431743-42 2024-03-04T06:19:35.864 INFO:tasks.rados.rados.0.smithi064.stdout:2816: finishing write tid 2 to smithi06431743-42 2024-03-04T06:19:35.873 INFO:tasks.rados.rados.0.smithi064.stdout:2816: finishing write tid 3 to smithi06431743-42 2024-03-04T06:19:35.878 INFO:tasks.rados.rados.0.smithi064.stdout:2816: finishing write tid 4 to smithi06431743-42 2024-03-04T06:19:35.886 INFO:tasks.rados.rados.0.smithi064.stdout:2816: finishing write tid 5 to smithi06431743-42 2024-03-04T06:19:35.886 INFO:tasks.rados.rados.0.smithi064.stdout:2816: finishing write tid 6 to smithi06431743-42 2024-03-04T06:19:35.886 INFO:tasks.rados.rados.0.smithi064.stdout:2816: oid 42 updating version 0 to 615 2024-03-04T06:19:35.886 INFO:tasks.rados.rados.0.smithi064.stdout:2816: oid 42 version 615 is already newer than 614 2024-03-04T06:19:35.887 INFO:tasks.rados.rados.0.smithi064.stdout:2816: oid 42 version 615 is already newer than 612 2024-03-04T06:19:35.887 INFO:tasks.rados.rados.0.smithi064.stdout:2816: oid 42 version 615 is already newer than 614 2024-03-04T06:19:35.887 INFO:tasks.rados.rados.0.smithi064.stdout:2816: oid 42 version 615 is already newer than 611 2024-03-04T06:19:35.887 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 615 (ObjNum 1288 snap 353 seq_num 1288) dirty exists 2024-03-04T06:19:35.887 INFO:tasks.rados.rados.0.smithi064.stdout:2816: left oid 42 (ObjNum 1288 snap 353 seq_num 1288) 2024-03-04T06:19:36.316 INFO:tasks.rados.rados.0.smithi064.stdout:2816: done (1 left) 2024-03-04T06:19:36.316 INFO:tasks.rados.rados.0.smithi064.stdout:2818: done (0 left) 2024-03-04T06:19:36.316 INFO:tasks.rados.rados.0.smithi064.stdout:2819: snap_remove snap 326 2024-03-04T06:19:37.017 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:37.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:37.134+0000 7f625279d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:37.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:37.138+0000 7f625279d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:37.321 INFO:tasks.rados.rados.0.smithi064.stdout:2819: done (0 left) 2024-03-04T06:19:37.322 INFO:tasks.rados.rados.0.smithi064.stdout:2820: read oid 41 snap -1 2024-03-04T06:19:37.322 INFO:tasks.rados.rados.0.smithi064.stdout:2820: expect deleted 2024-03-04T06:19:37.322 INFO:tasks.rados.rados.0.smithi064.stdout:2821: read oid 50 snap 349 2024-03-04T06:19:37.322 INFO:tasks.rados.rados.0.smithi064.stdout:2821: expect (ObjNum 1141 snap 306 seq_num 1141) 2024-03-04T06:19:37.322 INFO:tasks.rados.rados.0.smithi064.stdout:2822: write oid 44 current snap is 354 2024-03-04T06:19:37.322 INFO:tasks.rados.rados.0.smithi064.stdout:2822: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2024-03-04T06:19:37.327 INFO:tasks.rados.rados.0.smithi064.stdout:2822: writing smithi06431743-44 from 481030 to 1215897 tid 1 2024-03-04T06:19:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:2822: writing smithi06431743-44 from 1937889 to 2421332 tid 2 2024-03-04T06:19:37.340 INFO:tasks.rados.rados.0.smithi064.stdout:2822: writing smithi06431743-44 from 2875543 to 3427969 tid 3 2024-03-04T06:19:37.340 INFO:tasks.rados.rados.0.smithi064.stdout:2823: snap_remove snap 342 2024-03-04T06:19:38.294 INFO:tasks.rados.rados.0.smithi064.stdout:2823: done (3 left) 2024-03-04T06:19:38.294 INFO:tasks.rados.rados.0.smithi064.stdout:2824: write oid 23 current snap is 354 2024-03-04T06:19:38.294 INFO:tasks.rados.rados.0.smithi064.stdout:2824: seq_num 1291 ranges {497837=410116,1596371=743454,2902966=763698} 2024-03-04T06:19:38.298 INFO:tasks.rados.rados.0.smithi064.stdout:2824: writing smithi06431743-23 from 497837 to 907953 tid 1 2024-03-04T06:19:38.306 INFO:tasks.rados.rados.0.smithi064.stdout:2824: writing smithi06431743-23 from 1596371 to 2339825 tid 2 2024-03-04T06:19:38.314 INFO:tasks.rados.rados.0.smithi064.stdout:2824: writing smithi06431743-23 from 2902966 to 3666664 tid 3 2024-03-04T06:19:38.314 INFO:tasks.rados.rados.0.smithi064.stdout:2825: read oid 22 snap -1 2024-03-04T06:19:38.314 INFO:tasks.rados.rados.0.smithi064.stdout:2825: expect deleted 2024-03-04T06:19:38.314 INFO:tasks.rados.rados.0.smithi064.stdout:2826: write (excl) oid 48 current snap is 354 2024-03-04T06:19:38.314 INFO:tasks.rados.rados.0.smithi064.stdout:2826: seq_num 1292 ranges {514644=485365,1654853=487112,2930389=574970} 2024-03-04T06:19:38.319 INFO:tasks.rados.rados.0.smithi064.stdout:2826: writing smithi06431743-48 from 514644 to 1000009 tid 1 2024-03-04T06:19:38.325 INFO:tasks.rados.rados.0.smithi064.stdout:2826: writing smithi06431743-48 from 1654853 to 2141965 tid 2 2024-03-04T06:19:38.335 INFO:tasks.rados.rados.0.smithi064.stdout:2826: writing smithi06431743-48 from 2930389 to 3505359 tid 3 2024-03-04T06:19:38.335 INFO:tasks.rados.rados.0.smithi064.stdout:2827: snap_create 2024-03-04T06:19:38.335 INFO:tasks.rados.rados.0.smithi064.stdout:2821: expect (ObjNum 1141 snap 306 seq_num 1141) 2024-03-04T06:19:38.615 INFO:tasks.rados.rados.0.smithi064.stdout:2822: finishing write tid 1 to smithi06431743-44 2024-03-04T06:19:38.615 INFO:tasks.rados.rados.0.smithi064.stdout:2822: finishing write tid 2 to smithi06431743-44 2024-03-04T06:19:38.615 INFO:tasks.rados.rados.0.smithi064.stdout:2822: finishing write tid 3 to smithi06431743-44 2024-03-04T06:19:38.615 INFO:tasks.rados.rados.0.smithi064.stdout:2822: finishing write tid 4 to smithi06431743-44 2024-03-04T06:19:38.615 INFO:tasks.rados.rados.0.smithi064.stdout:2822: finishing write tid 5 to smithi06431743-44 2024-03-04T06:19:38.615 INFO:tasks.rados.rados.0.smithi064.stdout:2822: oid 44 updating version 0 to 730 2024-03-04T06:19:38.615 INFO:tasks.rados.rados.0.smithi064.stdout:2822: oid 44 version 730 is already newer than 725 2024-03-04T06:19:38.615 INFO:tasks.rados.rados.0.smithi064.stdout:2822: oid 44 version 730 is already newer than 724 2024-03-04T06:19:38.615 INFO:tasks.rados.rados.0.smithi064.stdout:2822: oid 44 updating version 730 to 731 2024-03-04T06:19:38.615 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 731 (ObjNum 1290 snap 354 seq_num 1290) dirty exists 2024-03-04T06:19:38.615 INFO:tasks.rados.rados.0.smithi064.stdout:2822: left oid 44 (ObjNum 1290 snap 354 seq_num 1290) 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2824: finishing write tid 1 to smithi06431743-23 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2826: finishing write tid 1 to smithi06431743-48 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2824: finishing write tid 2 to smithi06431743-23 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2826: finishing write tid 2 to smithi06431743-48 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2826: finishing write tid 3 to smithi06431743-48 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2826: finishing write tid 4 to smithi06431743-48 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2826: finishing write tid 5 to smithi06431743-48 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2826: oid 48 updating version 0 to 419 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2826: oid 48 version 419 is already newer than 419 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2826: oid 48 version 419 is already newer than 417 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2826: oid 48 version 419 is already newer than 416 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 419 (ObjNum 1292 snap 354 seq_num 1292) dirty exists 2024-03-04T06:19:38.616 INFO:tasks.rados.rados.0.smithi064.stdout:2826: left oid 48 (ObjNum 1292 snap 354 seq_num 1292) 2024-03-04T06:19:38.617 INFO:tasks.rados.rados.0.smithi064.stdout:2824: finishing write tid 3 to smithi06431743-23 2024-03-04T06:19:38.617 INFO:tasks.rados.rados.0.smithi064.stdout:2824: finishing write tid 4 to smithi06431743-23 2024-03-04T06:19:38.617 INFO:tasks.rados.rados.0.smithi064.stdout:2824: finishing write tid 5 to smithi06431743-23 2024-03-04T06:19:38.617 INFO:tasks.rados.rados.0.smithi064.stdout:2824: oid 23 updating version 0 to 689 2024-03-04T06:19:38.617 INFO:tasks.rados.rados.0.smithi064.stdout:2824: oid 23 version 689 is already newer than 688 2024-03-04T06:19:38.617 INFO:tasks.rados.rados.0.smithi064.stdout:2824: oid 23 updating version 689 to 691 2024-03-04T06:19:38.617 INFO:tasks.rados.rados.0.smithi064.stdout:2824: oid 23 version 691 is already newer than 690 2024-03-04T06:19:38.617 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 691 (ObjNum 1291 snap 354 seq_num 1291) dirty exists 2024-03-04T06:19:38.617 INFO:tasks.rados.rados.0.smithi064.stdout:2824: left oid 23 (ObjNum 1291 snap 354 seq_num 1291) 2024-03-04T06:19:39.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:39.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:39.318+0000 7f5ab1c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:39.323 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:39.323+0000 7f5ab1c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:39.324 INFO:tasks.rados.rados.0.smithi064.stdout:2820: done (6 left) 2024-03-04T06:19:39.325 INFO:tasks.rados.rados.0.smithi064.stdout:2821: done (5 left) 2024-03-04T06:19:39.325 INFO:tasks.rados.rados.0.smithi064.stdout:2822: done (4 left) 2024-03-04T06:19:39.325 INFO:tasks.rados.rados.0.smithi064.stdout:2824: done (3 left) 2024-03-04T06:19:39.325 INFO:tasks.rados.rados.0.smithi064.stdout:2825: done (2 left) 2024-03-04T06:19:39.325 INFO:tasks.rados.rados.0.smithi064.stdout:2826: done (1 left) 2024-03-04T06:19:39.325 INFO:tasks.rados.rados.0.smithi064.stdout:2827: done (0 left) 2024-03-04T06:19:39.325 INFO:tasks.rados.rados.0.smithi064.stdout:2828: read oid 28 snap 352 2024-03-04T06:19:39.325 INFO:tasks.rados.rados.0.smithi064.stdout:2828: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:19:39.325 INFO:tasks.rados.rados.0.smithi064.stdout:2829: write oid 7 current snap is 355 2024-03-04T06:19:39.325 INFO:tasks.rados.rados.0.smithi064.stdout:2829: seq_num 1293 ranges {531451=676967,1713335=747123,2957812=502595} 2024-03-04T06:19:39.331 INFO:tasks.rados.rados.0.smithi064.stdout:2829: writing smithi06431743-7 from 531451 to 1208418 tid 1 2024-03-04T06:19:39.338 INFO:tasks.rados.rados.0.smithi064.stdout:2829: writing smithi06431743-7 from 1713335 to 2460458 tid 2 2024-03-04T06:19:39.344 INFO:tasks.rados.rados.0.smithi064.stdout:2829: writing smithi06431743-7 from 2957812 to 3460407 tid 3 2024-03-04T06:19:39.344 INFO:tasks.rados.rados.0.smithi064.stdout:2830: read oid 44 snap -1 2024-03-04T06:19:39.344 INFO:tasks.rados.rados.0.smithi064.stdout:2830: expect (ObjNum 1290 snap 354 seq_num 1290) 2024-03-04T06:19:39.344 INFO:tasks.rados.rados.0.smithi064.stdout:2831: read oid 31 snap 325 2024-03-04T06:19:39.344 INFO:tasks.rados.rados.0.smithi064.stdout:2831: expect (ObjNum 1152 snap 309 seq_num 1152) 2024-03-04T06:19:39.345 INFO:tasks.rados.rados.0.smithi064.stdout:2832: write (excl) oid 30 current snap is 355 2024-03-04T06:19:39.345 INFO:tasks.rados.rados.0.smithi064.stdout:2832: seq_num 1294 ranges {548258=752216,1771817=490781,2985235=713867} 2024-03-04T06:19:39.352 INFO:tasks.rados.rados.0.smithi064.stdout:2832: writing smithi06431743-30 from 548258 to 1300474 tid 1 2024-03-04T06:19:39.358 INFO:tasks.rados.rados.0.smithi064.stdout:2832: writing smithi06431743-30 from 1771817 to 2262598 tid 2 2024-03-04T06:19:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:2832: writing smithi06431743-30 from 2985235 to 3699102 tid 3 2024-03-04T06:19:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:2833: read oid 36 snap -1 2024-03-04T06:19:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:2833: expect (ObjNum 1263 snap 341 seq_num 1263) 2024-03-04T06:19:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:2834: write oid 13 current snap is 355 2024-03-04T06:19:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:2834: seq_num 1295 ranges {565065=427465,1430299=634439,2496305=525139,3472104=292961} 2024-03-04T06:19:39.372 INFO:tasks.rados.rados.0.smithi064.stdout:2834: writing smithi06431743-13 from 565065 to 992530 tid 1 2024-03-04T06:19:39.380 INFO:tasks.rados.rados.0.smithi064.stdout:2834: writing smithi06431743-13 from 1430299 to 2064738 tid 2 2024-03-04T06:19:39.387 INFO:tasks.rados.rados.0.smithi064.stdout:2834: writing smithi06431743-13 from 2496305 to 3021444 tid 3 2024-03-04T06:19:39.392 INFO:tasks.rados.rados.0.smithi064.stdout:2834: writing smithi06431743-13 from 3472104 to 3765065 tid 4 2024-03-04T06:19:39.393 INFO:tasks.rados.rados.0.smithi064.stdout:2835: write oid 37 current snap is 355 2024-03-04T06:19:39.393 INFO:tasks.rados.rados.0.smithi064.stdout:2835: seq_num 1296 ranges {581872=502714,1488781=494450,2640081=736411} 2024-03-04T06:19:39.398 INFO:tasks.rados.rados.0.smithi064.stdout:2835: writing smithi06431743-37 from 581872 to 1084586 tid 1 2024-03-04T06:19:39.403 INFO:tasks.rados.rados.0.smithi064.stdout:2835: writing smithi06431743-37 from 1488781 to 1983231 tid 2 2024-03-04T06:19:39.413 INFO:tasks.rados.rados.0.smithi064.stdout:2835: writing smithi06431743-37 from 2640081 to 3376492 tid 3 2024-03-04T06:19:39.413 INFO:tasks.rados.rados.0.smithi064.stdout:2836: write (excl) oid 29 current snap is 355 2024-03-04T06:19:39.414 INFO:tasks.rados.rados.0.smithi064.stdout:2836: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2024-03-04T06:19:39.422 INFO:tasks.rados.rados.0.smithi064.stdout:2836: writing smithi06431743-29 from 598679 to 1176642 tid 1 2024-03-04T06:19:39.430 INFO:tasks.rados.rados.0.smithi064.stdout:2836: writing smithi06431743-29 from 1947263 to 2585371 tid 2 2024-03-04T06:19:39.437 INFO:tasks.rados.rados.0.smithi064.stdout:2836: writing smithi06431743-29 from 3351151 to 3798679 tid 3 2024-03-04T06:19:39.437 INFO:tasks.rados.rados.0.smithi064.stdout:2837: write oid 35 current snap is 355 2024-03-04T06:19:39.437 INFO:tasks.rados.rados.0.smithi064.stdout:2828: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:19:39.593 INFO:tasks.rados.rados.0.smithi064.stdout:2831: expect (ObjNum 1152 snap 309 seq_num 1152) 2024-03-04T06:19:39.791 INFO:tasks.rados.rados.0.smithi064.stdout:2830: expect (ObjNum 1290 snap 354 seq_num 1290) 2024-03-04T06:19:40.142 INFO:tasks.rados.rados.0.smithi064.stdout:2829: finishing write tid 1 to smithi06431743-7 2024-03-04T06:19:40.142 INFO:tasks.rados.rados.0.smithi064.stdout:2837: seq_num 1298 ranges {615486=653212,1889392=498119,2978574=475308} 2024-03-04T06:19:40.149 INFO:tasks.rados.rados.0.smithi064.stdout:2837: writing smithi06431743-35 from 615486 to 1268698 tid 1 2024-03-04T06:19:40.155 INFO:tasks.rados.rados.0.smithi064.stdout:2837: writing smithi06431743-35 from 1889392 to 2387511 tid 2 2024-03-04T06:19:40.162 INFO:tasks.rados.rados.0.smithi064.stdout:2837: writing smithi06431743-35 from 2978574 to 3453882 tid 3 2024-03-04T06:19:40.162 INFO:tasks.rados.rados.0.smithi064.stdout:2828: done (9 left) 2024-03-04T06:19:40.162 INFO:tasks.rados.rados.0.smithi064.stdout:2830: done (8 left) 2024-03-04T06:19:40.162 INFO:tasks.rados.rados.0.smithi064.stdout:2831: done (7 left) 2024-03-04T06:19:40.163 INFO:tasks.rados.rados.0.smithi064.stdout:2838: snap_remove snap 337 2024-03-04T06:19:40.196 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:19:40.196 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:19:40.196 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:19:40.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:19:40.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:40.304+0000 7fc613c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:40.311+0000 7fc613c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:40.323 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:40.323 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:40.324 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:40.324 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:40.324 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:40.324 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:40.324 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:40.324 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:40.324 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:40.324 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:40.324 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:40.324 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:40.326 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:40.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:19:40.328 INFO:tasks.rados.rados.0.smithi064.stdout:2838: done (7 left) 2024-03-04T06:19:40.328 INFO:tasks.rados.rados.0.smithi064.stdout:2839: snap_create 2024-03-04T06:19:40.328 INFO:tasks.rados.rados.0.smithi064.stdout:2833: expect (ObjNum 1263 snap 341 seq_num 1263) 2024-03-04T06:19:40.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:40.439+0000 7f1299864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:40.447 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:40.447+0000 7f1299864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:40.459 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:40.460 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:40.460 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:40.460 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:40.460 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:40.460 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:40.460 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:40.460 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:40.460 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:40.460 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:40.460 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:40.460 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:40.463 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:40.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:19:40.521 INFO:tasks.rados.rados.0.smithi064.stdout:2832: finishing write tid 1 to smithi06431743-30 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2829: finishing write tid 2 to smithi06431743-7 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2835: finishing write tid 1 to smithi06431743-37 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2829: finishing write tid 3 to smithi06431743-7 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2832: finishing write tid 2 to smithi06431743-30 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2835: finishing write tid 2 to smithi06431743-37 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2829: finishing write tid 4 to smithi06431743-7 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2829: finishing write tid 5 to smithi06431743-7 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2829: oid 7 updating version 0 to 422 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2829: oid 7 updating version 422 to 424 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2829: oid 7 version 424 is already newer than 423 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2829: oid 7 version 424 is already newer than 421 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 424 (ObjNum 1293 snap 355 seq_num 1293) dirty exists 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2829: left oid 7 (ObjNum 1293 snap 355 seq_num 1293) 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2832: finishing write tid 3 to smithi06431743-30 2024-03-04T06:19:40.522 INFO:tasks.rados.rados.0.smithi064.stdout:2832: finishing write tid 4 to smithi06431743-30 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2832: finishing write tid 5 to smithi06431743-30 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2832: oid 30 updating version 0 to 1372 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2832: oid 30 version 1372 is already newer than 1371 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2832: oid 30 updating version 1372 to 1374 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2832: oid 30 version 1374 is already newer than 1374 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 1374 (ObjNum 1294 snap 355 seq_num 1294) dirty exists 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2832: left oid 30 (ObjNum 1294 snap 355 seq_num 1294) 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2835: finishing write tid 3 to smithi06431743-37 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2835: finishing write tid 4 to smithi06431743-37 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2835: finishing write tid 5 to smithi06431743-37 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2835: oid 37 updating version 0 to 551 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2835: oid 37 version 551 is already newer than 549 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2835: oid 37 version 551 is already newer than 551 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2835: oid 37 version 551 is already newer than 548 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 551 (ObjNum 1296 snap 355 seq_num 1296) dirty exists 2024-03-04T06:19:40.523 INFO:tasks.rados.rados.0.smithi064.stdout:2835: left oid 37 (ObjNum 1296 snap 355 seq_num 1296) 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: finishing write tid 1 to smithi06431743-13 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: finishing write tid 2 to smithi06431743-13 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: finishing write tid 3 to smithi06431743-13 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: finishing write tid 4 to smithi06431743-13 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: finishing write tid 5 to smithi06431743-13 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: finishing write tid 6 to smithi06431743-13 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: oid 13 updating version 0 to 1376 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: oid 13 updating version 1376 to 1378 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: oid 13 updating version 1378 to 1380 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: oid 13 version 1380 is already newer than 1377 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: oid 13 version 1380 is already newer than 1380 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1380 (ObjNum 1295 snap 355 seq_num 1295) dirty exists 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2834: left oid 13 (ObjNum 1295 snap 355 seq_num 1295) 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2836: finishing write tid 1 to smithi06431743-29 2024-03-04T06:19:40.524 INFO:tasks.rados.rados.0.smithi064.stdout:2836: finishing write tid 2 to smithi06431743-29 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2836: finishing write tid 3 to smithi06431743-29 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2836: finishing write tid 4 to smithi06431743-29 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2836: finishing write tid 5 to smithi06431743-29 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2836: oid 29 updating version 0 to 1381 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2836: oid 29 updating version 1381 to 1383 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2836: oid 29 updating version 1383 to 1384 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2836: oid 29 version 1384 is already newer than 1382 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 1384 (ObjNum 1297 snap 355 seq_num 1297) dirty exists 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2836: left oid 29 (ObjNum 1297 snap 355 seq_num 1297) 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2837: finishing write tid 1 to smithi06431743-35 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2837: finishing write tid 2 to smithi06431743-35 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2837: finishing write tid 3 to smithi06431743-35 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2837: finishing write tid 4 to smithi06431743-35 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2837: finishing write tid 5 to smithi06431743-35 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2837: oid 35 updating version 0 to 924 2024-03-04T06:19:40.525 INFO:tasks.rados.rados.0.smithi064.stdout:2837: oid 35 version 924 is already newer than 921 2024-03-04T06:19:40.526 INFO:tasks.rados.rados.0.smithi064.stdout:2837: oid 35 version 924 is already newer than 923 2024-03-04T06:19:40.526 INFO:tasks.rados.rados.0.smithi064.stdout:2837: oid 35 version 924 is already newer than 922 2024-03-04T06:19:40.526 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 924 (ObjNum 1298 snap 355 seq_num 1298) dirty exists 2024-03-04T06:19:40.526 INFO:tasks.rados.rados.0.smithi064.stdout:2837: left oid 35 (ObjNum 1298 snap 355 seq_num 1298) 2024-03-04T06:19:40.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:40.568+0000 7f8be942d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:40.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:40.575+0000 7f8be942d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:40.588 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:40.588 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:40.588 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:40.588 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:40.588 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:40.588 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:40.588 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:40.588 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:40.588 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:40.589 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:40.589 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:40.589 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:40.591 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:40.591 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:40.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:19:40.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:40.694+0000 7f24aa19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:40.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:40.701+0000 7f24aa19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:40.720 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:40.723 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:41.330 INFO:tasks.rados.rados.0.smithi064.stdout:2829: done (7 left) 2024-03-04T06:19:41.331 INFO:tasks.rados.rados.0.smithi064.stdout:2832: done (6 left) 2024-03-04T06:19:41.331 INFO:tasks.rados.rados.0.smithi064.stdout:2833: done (5 left) 2024-03-04T06:19:41.331 INFO:tasks.rados.rados.0.smithi064.stdout:2834: done (4 left) 2024-03-04T06:19:41.331 INFO:tasks.rados.rados.0.smithi064.stdout:2835: done (3 left) 2024-03-04T06:19:41.331 INFO:tasks.rados.rados.0.smithi064.stdout:2836: done (2 left) 2024-03-04T06:19:41.331 INFO:tasks.rados.rados.0.smithi064.stdout:2837: done (1 left) 2024-03-04T06:19:41.331 INFO:tasks.rados.rados.0.smithi064.stdout:2839: done (0 left) 2024-03-04T06:19:41.331 INFO:tasks.rados.rados.0.smithi064.stdout:2840: write oid 36 current snap is 356 2024-03-04T06:19:41.331 INFO:tasks.rados.rados.0.smithi064.stdout:2840: seq_num 1299 ranges {632293=728461,1947874=641777,3005997=686580} 2024-03-04T06:19:41.337 INFO:tasks.rados.rados.0.smithi064.stdout:2840: writing smithi06431743-36 from 632293 to 1360754 tid 1 2024-03-04T06:19:41.344 INFO:tasks.rados.rados.0.smithi064.stdout:2840: writing smithi06431743-36 from 1947874 to 2589651 tid 2 2024-03-04T06:19:41.350 INFO:tasks.rados.rados.0.smithi064.stdout:2840: writing smithi06431743-36 from 3005997 to 3692577 tid 3 2024-03-04T06:19:41.351 INFO:tasks.rados.rados.0.smithi064.stdout:2841: snap_create 2024-03-04T06:19:41.433 INFO:tasks.rados.rados.0.smithi064.stdout:2840: finishing write tid 1 to smithi06431743-36 2024-03-04T06:19:41.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:41.447 INFO:tasks.rados.rados.0.smithi064.stdout:2840: finishing write tid 2 to smithi06431743-36 2024-03-04T06:19:41.459 INFO:tasks.rados.rados.0.smithi064.stdout:2840: finishing write tid 3 to smithi06431743-36 2024-03-04T06:19:41.467 INFO:tasks.rados.rados.0.smithi064.stdout:2840: finishing write tid 4 to smithi06431743-36 2024-03-04T06:19:41.468 INFO:tasks.rados.rados.0.smithi064.stdout:2840: finishing write tid 5 to smithi06431743-36 2024-03-04T06:19:41.468 INFO:tasks.rados.rados.0.smithi064.stdout:2840: oid 36 updating version 0 to 696 2024-03-04T06:19:41.468 INFO:tasks.rados.rados.0.smithi064.stdout:2840: oid 36 version 696 is already newer than 695 2024-03-04T06:19:41.468 INFO:tasks.rados.rados.0.smithi064.stdout:2840: oid 36 version 696 is already newer than 695 2024-03-04T06:19:41.469 INFO:tasks.rados.rados.0.smithi064.stdout:2840: oid 36 version 696 is already newer than 693 2024-03-04T06:19:41.469 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 696 (ObjNum 1299 snap 356 seq_num 1299) dirty exists 2024-03-04T06:19:41.469 INFO:tasks.rados.rados.0.smithi064.stdout:2840: left oid 36 (ObjNum 1299 snap 356 seq_num 1299) 2024-03-04T06:19:41.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:41.547+0000 7f306d95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:41.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:41.551+0000 7f306d95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:42.332 INFO:tasks.rados.rados.0.smithi064.stdout:2840: done (1 left) 2024-03-04T06:19:42.332 INFO:tasks.rados.rados.0.smithi064.stdout:2841: done (0 left) 2024-03-04T06:19:42.332 INFO:tasks.rados.rados.0.smithi064.stdout:2842: snap_remove snap 353 2024-03-04T06:19:43.332 INFO:tasks.rados.rados.0.smithi064.stdout:2842: done (0 left) 2024-03-04T06:19:43.333 INFO:tasks.rados.rados.0.smithi064.stdout:2843: snap_remove snap 349 2024-03-04T06:19:43.618 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:43.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:43.734+0000 7f2c772ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:43.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:43.739+0000 7f2c772ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:44.336 INFO:tasks.rados.rados.0.smithi064.stdout:2843: done (0 left) 2024-03-04T06:19:44.336 INFO:tasks.rados.rados.0.smithi064.stdout:2844: read oid 2 snap -1 2024-03-04T06:19:44.336 INFO:tasks.rados.rados.0.smithi064.stdout:2844: expect (ObjNum 1281 snap 349 seq_num 1281) 2024-03-04T06:19:44.337 INFO:tasks.rados.rados.0.smithi064.stdout:2845: delete oid 12 current snap is 357 2024-03-04T06:19:44.408 INFO:tasks.rados.rados.0.smithi064.stdout:2844: expect (ObjNum 1281 snap 349 seq_num 1281) 2024-03-04T06:19:44.643 INFO:tasks.rados.rados.0.smithi064.stdout:2844: done (1 left) 2024-03-04T06:19:44.643 INFO:tasks.rados.rados.0.smithi064.stdout:2845: done (0 left) 2024-03-04T06:19:44.643 INFO:tasks.rados.rados.0.smithi064.stdout:2846: snap_remove snap 323 2024-03-04T06:19:45.347 INFO:tasks.rados.rados.0.smithi064.stdout:2846: done (0 left) 2024-03-04T06:19:45.348 INFO:tasks.rados.rados.0.smithi064.stdout:2847: snap_create 2024-03-04T06:19:45.725 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:19:45.725 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:19:45.725 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:19:45.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:19:45.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:45.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:45.837+0000 7fcef446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:45.846 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:45.844+0000 7fcef446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:45.858 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:45.858 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:45.858 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:45.858 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:45.858 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:45.859 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:45.859 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:45.859 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:45.859 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:45.859 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:45.859 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:45.859 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:45.862 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:45.862 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:19:45.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:45.916+0000 7efd8762d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:45.922 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:45.921+0000 7efd8762d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:46.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:46.038+0000 7f25b139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:46.049 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:46.047+0000 7f25b139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:46.063 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:46.063 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:46.063 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:46.064 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:46.064 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:46.064 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:46.064 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:46.064 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:46.064 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:46.064 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:46.064 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:46.064 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:46.066 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:46.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:19:46.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:46.209+0000 7f225539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:46.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:46.216+0000 7f225539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:46.231 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:46.232 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:46.233 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:46.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:19:46.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:46.336+0000 7fbf5f46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:46.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:46.343+0000 7fbf5f46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:46.348 INFO:tasks.rados.rados.0.smithi064.stdout:2847: done (0 left) 2024-03-04T06:19:46.348 INFO:tasks.rados.rados.0.smithi064.stdout:2848: write (excl) oid 12 current snap is 358 2024-03-04T06:19:46.348 INFO:tasks.rados.rados.0.smithi064.stdout:2848: seq_num 1301 ranges {665907=595312,1781191=645446,3177196=688711} 2024-03-04T06:19:46.355 INFO:tasks.rados.rados.0.smithi064.stdout:2848: writing smithi06431743-12 from 665907 to 1261219 tid 1 2024-03-04T06:19:46.361 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:46.361 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:46.361 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:46.362 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:46.362 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:46.362 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:46.362 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:46.362 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:46.362 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:46.362 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:46.362 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:46.362 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:46.362 INFO:tasks.rados.rados.0.smithi064.stdout:2848: writing smithi06431743-12 from 1781191 to 2426637 tid 2 2024-03-04T06:19:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:46.364 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:46.369 INFO:tasks.rados.rados.0.smithi064.stdout:2848: writing smithi06431743-12 from 3177196 to 3865907 tid 3 2024-03-04T06:19:46.369 INFO:tasks.rados.rados.0.smithi064.stdout:2849: snap_remove snap 344 2024-03-04T06:19:47.356 INFO:tasks.rados.rados.0.smithi064.stdout:2849: done (1 left) 2024-03-04T06:19:47.356 INFO:tasks.rados.rados.0.smithi064.stdout:2850: snap_remove snap 352 2024-03-04T06:19:48.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:48.172 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:48.170+0000 7f22aef1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:48.175 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:48.174+0000 7f22aef1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:48.296 INFO:tasks.rados.rados.0.smithi064.stdout:2850: done (1 left) 2024-03-04T06:19:48.296 INFO:tasks.rados.rados.0.smithi064.stdout:2851: snap_remove snap 338 2024-03-04T06:19:49.296 INFO:tasks.rados.rados.0.smithi064.stdout:2851: done (1 left) 2024-03-04T06:19:49.296 INFO:tasks.rados.rados.0.smithi064.stdout:2852: snap_remove snap 357 2024-03-04T06:19:50.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:50.297 INFO:tasks.rados.rados.0.smithi064.stdout:2852: done (1 left) 2024-03-04T06:19:50.297 INFO:tasks.rados.rados.0.smithi064.stdout:2853: write (excl) oid 30 current snap is 358 2024-03-04T06:19:50.297 INFO:tasks.rados.rados.0.smithi064.stdout:2848: finishing write tid 1 to smithi06431743-12 2024-03-04T06:19:50.297 INFO:tasks.rados.rados.0.smithi064.stdout:2853: seq_num 1302 ranges {682714=670561,2123320=505457,3088266=636749} 2024-03-04T06:19:50.304 INFO:tasks.rados.rados.0.smithi064.stdout:2853: writing smithi06431743-30 from 682714 to 1353275 tid 1 2024-03-04T06:19:50.312 INFO:tasks.rados.rados.0.smithi064.stdout:2853: writing smithi06431743-30 from 2123320 to 2628777 tid 2 2024-03-04T06:19:50.318 INFO:tasks.rados.rados.0.smithi064.stdout:2853: writing smithi06431743-30 from 3088266 to 3725015 tid 3 2024-03-04T06:19:50.318 INFO:tasks.rados.rados.0.smithi064.stdout:2854: delete oid 11 current snap is 358 2024-03-04T06:19:50.318 INFO:tasks.rados.rados.0.smithi064.stdout:2848: finishing write tid 2 to smithi06431743-12 2024-03-04T06:19:50.318 INFO:tasks.rados.rados.0.smithi064.stdout:2848: finishing write tid 3 to smithi06431743-12 2024-03-04T06:19:50.318 INFO:tasks.rados.rados.0.smithi064.stdout:2848: finishing write tid 4 to smithi06431743-12 2024-03-04T06:19:50.318 INFO:tasks.rados.rados.0.smithi064.stdout:2848: finishing write tid 5 to smithi06431743-12 2024-03-04T06:19:50.319 INFO:tasks.rados.rados.0.smithi064.stdout:2848: oid 12 updating version 0 to 608 2024-03-04T06:19:50.319 INFO:tasks.rados.rados.0.smithi064.stdout:2848: oid 12 version 608 is already newer than 607 2024-03-04T06:19:50.319 INFO:tasks.rados.rados.0.smithi064.stdout:2848: oid 12 version 608 is already newer than 605 2024-03-04T06:19:50.319 INFO:tasks.rados.rados.0.smithi064.stdout:2848: oid 12 version 608 is already newer than 607 2024-03-04T06:19:50.319 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 608 (ObjNum 1301 snap 358 seq_num 1301) dirty exists 2024-03-04T06:19:50.319 INFO:tasks.rados.rados.0.smithi064.stdout:2848: left oid 12 (ObjNum 1301 snap 358 seq_num 1301) 2024-03-04T06:19:50.393 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:50.391+0000 7fba2066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:50.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:50.398+0000 7fba2066d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:50.411 INFO:tasks.rados.rados.0.smithi064.stdout:2848: done (2 left) 2024-03-04T06:19:50.411 INFO:tasks.rados.rados.0.smithi064.stdout:2854: done (1 left) 2024-03-04T06:19:50.412 INFO:tasks.rados.rados.0.smithi064.stdout:2855: write (excl) oid 32 current snap is 358 2024-03-04T06:19:50.412 INFO:tasks.rados.rados.0.smithi064.stdout:2855: seq_num 1304 ranges {716328=421059,1840284=509126,3143112=659293} 2024-03-04T06:19:50.417 INFO:tasks.rados.rados.0.smithi064.stdout:2855: writing smithi06431743-32 from 716328 to 1137387 tid 1 2024-03-04T06:19:50.424 INFO:tasks.rados.rados.0.smithi064.stdout:2855: writing smithi06431743-32 from 1840284 to 2349410 tid 2 2024-03-04T06:19:50.433 INFO:tasks.rados.rados.0.smithi064.stdout:2855: writing smithi06431743-32 from 3143112 to 3802405 tid 3 2024-03-04T06:19:50.433 INFO:tasks.rados.rados.0.smithi064.stdout:2856: read oid 49 snap -1 2024-03-04T06:19:50.433 INFO:tasks.rados.rados.0.smithi064.stdout:2856: expect (ObjNum 1169 snap 311 seq_num 1169) 2024-03-04T06:19:50.433 INFO:tasks.rados.rados.0.smithi064.stdout:2857: delete oid 50 current snap is 358 2024-03-04T06:19:50.434 INFO:tasks.rados.rados.0.smithi064.stdout:2853: finishing write tid 1 to smithi06431743-30 2024-03-04T06:19:50.465 INFO:tasks.rados.rados.0.smithi064.stdout:2855: finishing write tid 1 to smithi06431743-32 2024-03-04T06:19:50.466 INFO:tasks.rados.rados.0.smithi064.stdout:2853: finishing write tid 2 to smithi06431743-30 2024-03-04T06:19:50.476 INFO:tasks.rados.rados.0.smithi064.stdout:2857: done (3 left) 2024-03-04T06:19:50.476 INFO:tasks.rados.rados.0.smithi064.stdout:2858: snap_remove snap 351 2024-03-04T06:19:51.302 INFO:tasks.rados.rados.0.smithi064.stdout:2858: done (3 left) 2024-03-04T06:19:51.302 INFO:tasks.rados.rados.0.smithi064.stdout:2859: read oid 9 snap -1 2024-03-04T06:19:51.302 INFO:tasks.rados.rados.0.smithi064.stdout:2859: expect deleted 2024-03-04T06:19:51.302 INFO:tasks.rados.rados.0.smithi064.stdout:2860: snap_remove snap 354 2024-03-04T06:19:51.366 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:19:51.366 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:19:51.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:19:51.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:51.471+0000 7fc13c86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:51.478 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:51.477+0000 7fc13c86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:19:51.668 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:19:51.679 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1074,"stamp":"2024-03-04T06:19:51.300998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131444314,"num_objects":963,"num_object_clones":767,"num_object_copies":2889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":155,"num_read":5273,"num_read_kb":4074281,"num_write":4486,"num_write_kb":1190545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11098,"ondisk_log_size":11098,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":666460,"kb_used_data":462012,"kb_used_omap":0,"kb_used_meta":204416,"kb_avail":376820900,"statfs":{"total":386547056640,"available":385864601600,"internally_reserved":0,"allocated":473100288,"data_stored":550810713,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":209321984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3865907,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":2541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.945769"},"pg_stats":[{"pgid":"3.f","version":"717'639","reported_seq":2470,"reported_epoch":718,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:50.363532+0000","last_change":"2024-03-04T06:19:48.459204+0000","last_active":"2024-03-04T06:19:50.363532+0000","last_peered":"2024-03-04T06:19:50.363532+0000","last_clean":"2024-03-04T06:19:50.363532+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:19:50.363532+0000","last_undegraded":"2024-03-04T06:19:50.363532+0000","last_fullsized":"2024-03-04T06:19:50.363532+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":639,"log_dups_size":0,"ondisk_log_size":639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7009113,"num_objects":62,"num_object_clones":49,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":299,"num_read_kb":245920,"num_write":258,"num_write_kb":69127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"717'557","reported_seq":2430,"reported_epoch":719,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:50.328282+0000","last_change":"2024-03-04T06:19:48.444128+0000","last_active":"2024-03-04T06:19:50.328282+0000","last_peered":"2024-03-04T06:19:50.328282+0000","last_clean":"2024-03-04T06:19:50.328282+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:19:50.328282+0000","last_undegraded":"2024-03-04T06:19:50.328282+0000","last_fullsized":"2024-03-04T06:19:50.328282+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6975499,"num_objects":49,"num_object_clones":35,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":183,"num_read_kb":130655,"num_write":202,"num_write_kb":53508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"714'705","reported_seq":2570,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.359577+0000","last_change":"2024-03-04T06:19:45.394299+0000","last_active":"2024-03-04T06:19:45.359577+0000","last_peered":"2024-03-04T06:19:45.359577+0000","last_clean":"2024-03-04T06:19:45.359577+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:19:45.359577+0000","last_undegraded":"2024-03-04T06:19:45.359577+0000","last_fullsized":"2024-03-04T06:19:45.359577+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12219877,"num_objects":70,"num_object_clones":55,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":406,"num_read_kb":327450,"num_write":309,"num_write_kb":84178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"717'612","reported_seq":2433,"reported_epoch":718,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:50.344977+0000","last_change":"2024-03-04T06:19:48.450439+0000","last_active":"2024-03-04T06:19:50.344977+0000","last_peered":"2024-03-04T06:19:50.344977+0000","last_clean":"2024-03-04T06:19:50.344977+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:19:50.344977+0000","last_undegraded":"2024-03-04T06:19:50.344977+0000","last_fullsized":"2024-03-04T06:19:50.344977+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10438038,"num_objects":48,"num_object_clones":38,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":259,"num_read_kb":202834,"num_write":234,"num_write_kb":66141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"714'261","reported_seq":2250,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.937361+0000","last_change":"2024-03-04T06:19:46.029180+0000","last_active":"2024-03-04T06:19:45.937361+0000","last_peered":"2024-03-04T06:19:45.937361+0000","last_clean":"2024-03-04T06:19:45.937361+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:19:45.937361+0000","last_undegraded":"2024-03-04T06:19:45.937361+0000","last_fullsized":"2024-03-04T06:19:45.937361+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":90,"num_read_kb":65483,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"717'753","reported_seq":2527,"reported_epoch":718,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:50.354925+0000","last_change":"2024-03-04T06:19:48.461405+0000","last_active":"2024-03-04T06:19:50.354925+0000","last_peered":"2024-03-04T06:19:50.354925+0000","last_clean":"2024-03-04T06:19:50.354925+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:19:50.354925+0000","last_undegraded":"2024-03-04T06:19:50.354925+0000","last_fullsized":"2024-03-04T06:19:50.354925+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10269968,"num_objects":59,"num_object_clones":50,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":6,"num_read":345,"num_read_kb":278960,"num_write":298,"num_write_kb":87675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2108,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-03-04T06:19:50.361137+0000","last_change":"2024-03-04T06:19:50.373782+0000","last_active":"2024-03-04T06:19:50.361137+0000","last_peered":"2024-03-04T06:19:50.361137+0000","last_clean":"2024-03-04T06:19:50.361137+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:19:50.361137+0000","last_undegraded":"2024-03-04T06:19:50.361137+0000","last_fullsized":"2024-03-04T06:19:50.361137+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"714'410","reported_seq":2337,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.360773+0000","last_change":"2024-03-04T06:19:45.393102+0000","last_active":"2024-03-04T06:19:45.360773+0000","last_peered":"2024-03-04T06:19:45.360773+0000","last_clean":"2024-03-04T06:19:45.360773+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:19:45.360773+0000","last_undegraded":"2024-03-04T06:19:45.360773+0000","last_fullsized":"2024-03-04T06:19:45.360773+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672940,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":208,"num_read_kb":185580,"num_write":145,"num_write_kb":38810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1929,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.925796+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:19:45.925796+0000","last_peered":"2024-03-04T06:19:45.925796+0000","last_clean":"2024-03-04T06:19:45.925796+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:19:45.925796+0000","last_undegraded":"2024-03-04T06:19:45.925796+0000","last_fullsized":"2024-03-04T06:19:45.925796+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1784,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.366768+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:19:45.366768+0000","last_peered":"2024-03-04T06:19:45.366768+0000","last_clean":"2024-03-04T06:19:45.366768+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:19:45.366768+0000","last_undegraded":"2024-03-04T06:19:45.366768+0000","last_fullsized":"2024-03-04T06:19:45.366768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"714'695","reported_seq":2514,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.366179+0000","last_change":"2024-03-04T06:19:45.394535+0000","last_active":"2024-03-04T06:19:45.366179+0000","last_peered":"2024-03-04T06:19:45.366179+0000","last_clean":"2024-03-04T06:19:45.366179+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:19:45.366179+0000","last_undegraded":"2024-03-04T06:19:45.366179+0000","last_fullsized":"2024-03-04T06:19:45.366179+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6286412,"num_objects":62,"num_object_clones":48,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":340,"num_read_kb":275379,"num_write":276,"num_write_kb":74562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1776,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-03-04T06:19:50.354107+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:19:50.354107+0000","last_peered":"2024-03-04T06:19:50.354107+0000","last_clean":"2024-03-04T06:19:50.354107+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:19:50.354107+0000","last_undegraded":"2024-03-04T06:19:50.354107+0000","last_fullsized":"2024-03-04T06:19:50.354107+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"714'432","reported_seq":2390,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.929219+0000","last_change":"2024-03-04T06:19:46.041031+0000","last_active":"2024-03-04T06:19:45.929219+0000","last_peered":"2024-03-04T06:19:45.929219+0000","last_clean":"2024-03-04T06:19:45.929219+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:19:45.929219+0000","last_undegraded":"2024-03-04T06:19:45.929219+0000","last_fullsized":"2024-03-04T06:19:45.929219+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7446095,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":255,"num_read_kb":203317,"num_write":166,"num_write_kb":47091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1784,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.355413+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:19:45.355413+0000","last_peered":"2024-03-04T06:19:45.355413+0000","last_clean":"2024-03-04T06:19:45.355413+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:19:45.355413+0000","last_undegraded":"2024-03-04T06:19:45.355413+0000","last_fullsized":"2024-03-04T06:19:45.355413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"714'1244","reported_seq":2896,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.930410+0000","last_change":"2024-03-04T06:19:46.038742+0000","last_active":"2024-03-04T06:19:45.930410+0000","last_peered":"2024-03-04T06:19:45.930410+0000","last_clean":"2024-03-04T06:19:45.930410+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:19:45.930410+0000","last_undegraded":"2024-03-04T06:19:45.930410+0000","last_fullsized":"2024-03-04T06:19:45.930410+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1244,"log_dups_size":0,"ondisk_log_size":1244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9026217,"num_objects":113,"num_object_clones":90,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":661,"num_read_kb":522690,"num_write":526,"num_write_kb":140476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1776,"reported_epoch":718,"state":"active+clean","last_fresh":"2024-03-04T06:19:50.348128+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:19:50.348128+0000","last_peered":"2024-03-04T06:19:50.348128+0000","last_clean":"2024-03-04T06:19:50.348128+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:19:50.348128+0000","last_undegraded":"2024-03-04T06:19:50.348128+0000","last_fullsized":"2024-03-04T06:19:50.348128+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"719'1414","reported_seq":2887,"reported_epoch":719,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:50.334916+0000","last_change":"2024-03-04T06:19:48.448319+0000","last_active":"2024-03-04T06:19:50.334916+0000","last_peered":"2024-03-04T06:19:50.334916+0000","last_clean":"2024-03-04T06:19:50.334916+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:19:50.334916+0000","last_undegraded":"2024-03-04T06:19:50.334916+0000","last_fullsized":"2024-03-04T06:19:50.334916+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1414,"log_dups_size":0,"ondisk_log_size":1414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18371536,"num_objects":120,"num_object_clones":97,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":517,"num_read_kb":366975,"num_write":552,"num_write_kb":147794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"714'697","reported_seq":2533,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.368597+0000","last_change":"2024-03-04T06:19:45.407406+0000","last_active":"2024-03-04T06:19:45.368597+0000","last_peered":"2024-03-04T06:19:45.368597+0000","last_clean":"2024-03-04T06:19:45.368597+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:19:45.368597+0000","last_undegraded":"2024-03-04T06:19:45.368597+0000","last_fullsized":"2024-03-04T06:19:45.368597+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":697,"log_dups_size":0,"ondisk_log_size":697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5765395,"num_objects":57,"num_object_clones":45,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":392,"num_read_kb":329117,"num_write":267,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1839,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-04T06:19:50.340905+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:19:50.340905+0000","last_peered":"2024-03-04T06:19:50.340905+0000","last_clean":"2024-03-04T06:19:50.340905+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:19:50.340905+0000","last_undegraded":"2024-03-04T06:19:50.340905+0000","last_fullsized":"2024-03-04T06:19:50.340905+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"717'710","reported_seq":2566,"reported_epoch":719,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:19:50.341643+0000","last_change":"2024-03-04T06:19:48.475598+0000","last_active":"2024-03-04T06:19:50.341643+0000","last_peered":"2024-03-04T06:19:50.341643+0000","last_clean":"2024-03-04T06:19:50.341643+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:19:50.341643+0000","last_undegraded":"2024-03-04T06:19:50.341643+0000","last_fullsized":"2024-03-04T06:19:50.341643+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":710,"log_dups_size":0,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10606108,"num_objects":68,"num_object_clones":52,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":323,"num_read_kb":236246,"num_write":288,"num_write_kb":77847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1796,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.930603+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:19:45.930603+0000","last_peered":"2024-03-04T06:19:45.930603+0000","last_clean":"2024-03-04T06:19:45.930603+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:19:45.930603+0000","last_undegraded":"2024-03-04T06:19:45.930603+0000","last_fullsized":"2024-03-04T06:19:45.930603+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"714'936","reported_seq":2631,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.920386+0000","last_change":"2024-03-04T06:19:46.046863+0000","last_active":"2024-03-04T06:19:45.920386+0000","last_peered":"2024-03-04T06:19:45.920386+0000","last_clean":"2024-03-04T06:19:45.920386+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:19:45.920386+0000","last_undegraded":"2024-03-04T06:19:45.920386+0000","last_fullsized":"2024-03-04T06:19:45.920386+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13648439,"num_objects":80,"num_object_clones":66,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":392,"num_read_kb":280816,"num_write":376,"num_write_kb":104893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1839,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-04T06:19:50.332746+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:19:50.332746+0000","last_peered":"2024-03-04T06:19:50.332746+0000","last_clean":"2024-03-04T06:19:50.332746+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:19:50.332746+0000","last_undegraded":"2024-03-04T06:19:50.332746+0000","last_fullsized":"2024-03-04T06:19:50.332746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"714'936","reported_seq":2717,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-04T06:19:50.342739+0000","last_change":"2024-03-04T06:19:50.363458+0000","last_active":"2024-03-04T06:19:50.342739+0000","last_peered":"2024-03-04T06:19:50.342739+0000","last_clean":"2024-03-04T06:19:50.342739+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:19:50.342739+0000","last_undegraded":"2024-03-04T06:19:50.342739+0000","last_fullsized":"2024-03-04T06:19:50.342739+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3176787,"num_objects":79,"num_object_clones":69,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":500,"num_read_kb":422768,"num_write":355,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1784,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-04T06:19:45.366903+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:19:45.366903+0000","last_peered":"2024-03-04T06:19:45.366903+0000","last_clean":"2024-03-04T06:19:45.366903+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:19:45.366903+0000","last_undegraded":"2024-03-04T06:19:45.366903+0000","last_fullsized":"2024-03-04T06:19:45.366903+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":130853946,"num_objects":961,"num_object_clones":767,"num_object_copies":2883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":155,"num_read":5170,"num_read_kb":4074190,"num_write":4349,"num_write_kb":1188651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11001,"ondisk_log_size":11001,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706384,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":199328,"kb_used_data":141976,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94172512,"statfs":{"total":96636764160,"available":96432652288,"internally_reserved":0,"allocated":145383424,"data_stored":168787605,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706384,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":150232,"kb_used_data":101200,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94221608,"statfs":{"total":96636764160,"available":96482926592,"internally_reserved":0,"allocated":103628800,"data_stored":118186925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706388,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":155256,"kb_used_data":106224,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94216584,"statfs":{"total":96636764160,"available":96477782016,"internally_reserved":0,"allocated":108773376,"data_stored":126292738,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706388,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":161644,"kb_used_data":112612,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94210196,"statfs":{"total":96636764160,"available":96471240704,"internally_reserved":0,"allocated":115314688,"data_stored":137543445,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:19:52.304 INFO:tasks.rados.rados.0.smithi064.stdout:2860: done (4 left) 2024-03-04T06:19:52.305 INFO:tasks.rados.rados.0.smithi064.stdout:2861: write (excl) oid 19 current snap is 358 2024-03-04T06:19:52.305 INFO:tasks.rados.rados.0.smithi064.stdout:2861: seq_num 1306 ranges {749942=571557,1840895=796442,3365252=584690} 2024-03-04T06:19:52.313 INFO:tasks.rados.rados.0.smithi064.stdout:2861: writing smithi06431743-19 from 749942 to 1321499 tid 1 2024-03-04T06:19:52.323 INFO:tasks.rados.rados.0.smithi064.stdout:2861: writing smithi06431743-19 from 1840895 to 2637337 tid 2 2024-03-04T06:19:52.332 INFO:tasks.rados.rados.0.smithi064.stdout:2861: writing smithi06431743-19 from 3365252 to 3949942 tid 3 2024-03-04T06:19:52.332 INFO:tasks.rados.rados.0.smithi064.stdout:2862: snap_remove snap 333 2024-03-04T06:19:52.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:52.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:52.653+0000 7f6d25f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:52.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:52.657+0000 7f6d25f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:53.311 INFO:tasks.rados.rados.0.smithi064.stdout:2862: done (5 left) 2024-03-04T06:19:53.311 INFO:tasks.rados.rados.0.smithi064.stdout:2863: read oid 1 snap -1 2024-03-04T06:19:53.311 INFO:tasks.rados.rados.0.smithi064.stdout:2863: expect deleted 2024-03-04T06:19:53.311 INFO:tasks.rados.rados.0.smithi064.stdout:2864: snap_create 2024-03-04T06:19:53.311 INFO:tasks.rados.rados.0.smithi064.stdout:2853: finishing write tid 3 to smithi06431743-30 2024-03-04T06:19:53.312 INFO:tasks.rados.rados.0.smithi064.stdout:2856: expect (ObjNum 1169 snap 311 seq_num 1169) 2024-03-04T06:19:53.589 INFO:tasks.rados.rados.0.smithi064.stdout:2855: finishing write tid 2 to smithi06431743-32 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2853: finishing write tid 4 to smithi06431743-30 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2853: finishing write tid 5 to smithi06431743-30 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2853: oid 30 updating version 0 to 1419 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2853: oid 30 version 1419 is already newer than 1418 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2853: oid 30 version 1419 is already newer than 1417 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2853: oid 30 version 1419 is already newer than 1416 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 1419 (ObjNum 1302 snap 358 seq_num 1302) dirty exists 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2853: left oid 30 (ObjNum 1302 snap 358 seq_num 1302) 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2855: finishing write tid 3 to smithi06431743-32 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2855: finishing write tid 4 to smithi06431743-32 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2855: finishing write tid 5 to smithi06431743-32 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2855: oid 32 updating version 0 to 1259 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2855: oid 32 version 1259 is already newer than 1257 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2855: oid 32 version 1259 is already newer than 1259 2024-03-04T06:19:53.590 INFO:tasks.rados.rados.0.smithi064.stdout:2855: oid 32 version 1259 is already newer than 1256 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1259 (ObjNum 1304 snap 358 seq_num 1304) dirty exists 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:2855: left oid 32 (ObjNum 1304 snap 358 seq_num 1304) 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:2861: finishing write tid 1 to smithi06431743-19 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:2861: finishing write tid 2 to smithi06431743-19 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:2861: finishing write tid 3 to smithi06431743-19 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:2861: finishing write tid 4 to smithi06431743-19 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:2861: finishing write tid 5 to smithi06431743-19 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:2861: oid 19 updating version 0 to 1431 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:2861: oid 19 updating version 1431 to 1432 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:2861: oid 19 version 1432 is already newer than 1430 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:2861: oid 19 version 1432 is already newer than 1429 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 1432 (ObjNum 1306 snap 358 seq_num 1306) dirty exists 2024-03-04T06:19:53.591 INFO:tasks.rados.rados.0.smithi064.stdout:2861: left oid 19 (ObjNum 1306 snap 358 seq_num 1306) 2024-03-04T06:19:54.314 INFO:tasks.rados.rados.0.smithi064.stdout:2853: done (6 left) 2024-03-04T06:19:54.315 INFO:tasks.rados.rados.0.smithi064.stdout:2855: done (5 left) 2024-03-04T06:19:54.315 INFO:tasks.rados.rados.0.smithi064.stdout:2856: done (4 left) 2024-03-04T06:19:54.315 INFO:tasks.rados.rados.0.smithi064.stdout:2859: done (3 left) 2024-03-04T06:19:54.315 INFO:tasks.rados.rados.0.smithi064.stdout:2861: done (2 left) 2024-03-04T06:19:54.315 INFO:tasks.rados.rados.0.smithi064.stdout:2863: done (1 left) 2024-03-04T06:19:54.315 INFO:tasks.rados.rados.0.smithi064.stdout:2864: done (0 left) 2024-03-04T06:19:54.315 INFO:tasks.rados.rados.0.smithi064.stdout:2865: read oid 26 snap 321 2024-03-04T06:19:54.315 INFO:tasks.rados.rados.0.smithi064.stdout:2865: expect (ObjNum 1192 snap 318 seq_num 1192) 2024-03-04T06:19:54.315 INFO:tasks.rados.rados.0.smithi064.stdout:2866: snap_remove snap 356 2024-03-04T06:19:54.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:54.837 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:54.835+0000 7f6e5631d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:54.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:54.839+0000 7f6e5631d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:55.324 INFO:tasks.rados.rados.0.smithi064.stdout:2866: done (1 left) 2024-03-04T06:19:55.325 INFO:tasks.rados.rados.0.smithi064.stdout:2867: snap_remove snap 343 2024-03-04T06:19:56.325 INFO:tasks.rados.rados.0.smithi064.stdout:2867: done (1 left) 2024-03-04T06:19:56.325 INFO:tasks.rados.rados.0.smithi064.stdout:2868: write oid 35 current snap is 359 2024-03-04T06:19:56.325 INFO:tasks.rados.rados.0.smithi064.stdout:2868: seq_num 1307 ranges {766749=646806,1899377=656453,3109028=609462} 2024-03-04T06:19:56.333 INFO:tasks.rados.rados.0.smithi064.stdout:2868: writing smithi06431743-35 from 766749 to 1413555 tid 1 2024-03-04T06:19:56.340 INFO:tasks.rados.rados.0.smithi064.stdout:2868: writing smithi06431743-35 from 1899377 to 2555830 tid 2 2024-03-04T06:19:56.346 INFO:tasks.rados.rados.0.smithi064.stdout:2868: writing smithi06431743-35 from 3109028 to 3718490 tid 3 2024-03-04T06:19:56.346 INFO:tasks.rados.rados.0.smithi064.stdout:2869: write oid 42 current snap is 359 2024-03-04T06:19:56.346 INFO:tasks.rados.rados.0.smithi064.stdout:2869: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2024-03-04T06:19:56.353 INFO:tasks.rados.rados.0.smithi064.stdout:2869: writing smithi06431743-42 from 783556 to 1221964 tid 1 2024-03-04T06:19:56.358 INFO:tasks.rados.rados.0.smithi064.stdout:2869: writing smithi06431743-42 from 1674212 to 2074323 tid 2 2024-03-04T06:19:56.364 INFO:tasks.rados.rados.0.smithi064.stdout:2869: writing smithi06431743-42 from 2736451 to 3157185 tid 3 2024-03-04T06:19:56.368 INFO:tasks.rados.rados.0.smithi064.stdout:2869: writing smithi06431743-42 from 3882270 to 3983556 tid 4 2024-03-04T06:19:56.369 INFO:tasks.rados.rados.0.smithi064.stdout:2870: read oid 39 snap -1 2024-03-04T06:19:56.369 INFO:tasks.rados.rados.0.smithi064.stdout:2870: expect (ObjNum 1207 snap 321 seq_num 1207) 2024-03-04T06:19:56.369 INFO:tasks.rados.rados.0.smithi064.stdout:2871: snap_create 2024-03-04T06:19:56.369 INFO:tasks.rados.rados.0.smithi064.stdout:2865: expect (ObjNum 1192 snap 318 seq_num 1192) 2024-03-04T06:19:56.483 INFO:tasks.rados.rados.0.smithi064.stdout:2870: expect (ObjNum 1207 snap 321 seq_num 1207) 2024-03-04T06:19:56.629 INFO:tasks.rados.rados.0.smithi064.stdout:2868: finishing write tid 1 to smithi06431743-35 2024-03-04T06:19:56.629 INFO:tasks.rados.rados.0.smithi064.stdout:2869: finishing write tid 1 to smithi06431743-42 2024-03-04T06:19:56.629 INFO:tasks.rados.rados.0.smithi064.stdout:2868: finishing write tid 2 to smithi06431743-35 2024-03-04T06:19:56.629 INFO:tasks.rados.rados.0.smithi064.stdout:2869: finishing write tid 2 to smithi06431743-42 2024-03-04T06:19:56.629 INFO:tasks.rados.rados.0.smithi064.stdout:2869: finishing write tid 3 to smithi06431743-42 2024-03-04T06:19:56.629 INFO:tasks.rados.rados.0.smithi064.stdout:2868: finishing write tid 3 to smithi06431743-35 2024-03-04T06:19:56.630 INFO:tasks.rados.rados.0.smithi064.stdout:2869: finishing write tid 4 to smithi06431743-42 2024-03-04T06:19:56.630 INFO:tasks.rados.rados.0.smithi064.stdout:2869: finishing write tid 5 to smithi06431743-42 2024-03-04T06:19:56.630 INFO:tasks.rados.rados.0.smithi064.stdout:2869: finishing write tid 6 to smithi06431743-42 2024-03-04T06:19:56.630 INFO:tasks.rados.rados.0.smithi064.stdout:2869: oid 42 updating version 0 to 655 2024-03-04T06:19:56.630 INFO:tasks.rados.rados.0.smithi064.stdout:2869: oid 42 version 655 is already newer than 654 2024-03-04T06:19:56.630 INFO:tasks.rados.rados.0.smithi064.stdout:2869: oid 42 updating version 655 to 657 2024-03-04T06:19:56.630 INFO:tasks.rados.rados.0.smithi064.stdout:2869: oid 42 version 657 is already newer than 653 2024-03-04T06:19:56.630 INFO:tasks.rados.rados.0.smithi064.stdout:2869: oid 42 version 657 is already newer than 656 2024-03-04T06:19:56.630 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 657 (ObjNum 1308 snap 359 seq_num 1308) dirty exists 2024-03-04T06:19:56.630 INFO:tasks.rados.rados.0.smithi064.stdout:2869: left oid 42 (ObjNum 1308 snap 359 seq_num 1308) 2024-03-04T06:19:56.631 INFO:tasks.rados.rados.0.smithi064.stdout:2868: finishing write tid 4 to smithi06431743-35 2024-03-04T06:19:56.631 INFO:tasks.rados.rados.0.smithi064.stdout:2868: finishing write tid 5 to smithi06431743-35 2024-03-04T06:19:56.631 INFO:tasks.rados.rados.0.smithi064.stdout:2868: oid 35 updating version 0 to 960 2024-03-04T06:19:56.631 INFO:tasks.rados.rados.0.smithi064.stdout:2868: oid 35 updating version 960 to 963 2024-03-04T06:19:56.631 INFO:tasks.rados.rados.0.smithi064.stdout:2868: oid 35 version 963 is already newer than 962 2024-03-04T06:19:56.631 INFO:tasks.rados.rados.0.smithi064.stdout:2868: oid 35 version 963 is already newer than 962 2024-03-04T06:19:56.631 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 963 (ObjNum 1307 snap 359 seq_num 1307) dirty exists 2024-03-04T06:19:56.631 INFO:tasks.rados.rados.0.smithi064.stdout:2868: left oid 35 (ObjNum 1307 snap 359 seq_num 1307) 2024-03-04T06:19:56.682 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:19:56.682 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:19:56.682 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:19:56.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:19:56.791 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:56.786+0000 7f1fecd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:56.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:56.793+0000 7f1fecd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:56.806 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:56.806 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:56.806 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:56.806 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:56.807 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:56.809 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:56.810 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:56.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:19:56.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:19:56.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:56.914+0000 7fc50c51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:56.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:56.922+0000 7fc50c51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:56.937 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:56.937 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:56.937 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:56.937 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:56.937 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:56.937 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:56.938 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:56.938 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:56.938 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:56.938 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:56.938 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:56.938 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:56.941 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:56.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:19:57.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:57.014+0000 7f71f0a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:57.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:57.019+0000 7f71f0a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:57.100 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:57.095+0000 7efd1159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:57.103 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:57.102+0000 7efd1159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:57.115 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:57.115 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:57.116 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:57.117 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:57.117 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:57.118 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:57.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:19:57.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:57.262+0000 7f04b851d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:57.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:57.269+0000 7f04b851d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:57.282 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:19:57.282 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:19:57.282 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:19:57.282 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:19:57.283 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:19:57.283 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:19:57.283 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:19:57.283 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:19:57.283 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:19:57.283 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:19:57.283 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:19:57.283 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:19:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:19:57.285 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:19:57.332 INFO:tasks.rados.rados.0.smithi064.stdout:2865: done (4 left) 2024-03-04T06:19:57.332 INFO:tasks.rados.rados.0.smithi064.stdout:2868: done (3 left) 2024-03-04T06:19:57.332 INFO:tasks.rados.rados.0.smithi064.stdout:2869: done (2 left) 2024-03-04T06:19:57.333 INFO:tasks.rados.rados.0.smithi064.stdout:2870: done (1 left) 2024-03-04T06:19:57.333 INFO:tasks.rados.rados.0.smithi064.stdout:2871: done (0 left) 2024-03-04T06:19:57.333 INFO:tasks.rados.rados.0.smithi064.stdout:2872: snap_remove snap 359 2024-03-04T06:19:58.300 INFO:tasks.rados.rados.0.smithi064.stdout:2872: done (0 left) 2024-03-04T06:19:58.300 INFO:tasks.rados.rados.0.smithi064.stdout:2873: write oid 8 current snap is 360 2024-03-04T06:19:58.300 INFO:tasks.rados.rados.0.smithi064.stdout:2873: seq_num 1309 ranges {400363=513657,1332694=660122} 2024-03-04T06:19:58.305 INFO:tasks.rados.rados.0.smithi064.stdout:2873: writing smithi06431743-8 from 400363 to 914020 tid 1 2024-03-04T06:19:58.312 INFO:tasks.rados.rados.0.smithi064.stdout:2873: writing smithi06431743-8 from 1332694 to 1992816 tid 2 2024-03-04T06:19:58.312 INFO:tasks.rados.rados.0.smithi064.stdout:2874: read oid 25 snap 325 2024-03-04T06:19:58.312 INFO:tasks.rados.rados.0.smithi064.stdout:2874: expect (ObjNum 1185 snap 318 seq_num 1185) 2024-03-04T06:19:58.312 INFO:tasks.rados.rados.0.smithi064.stdout:2875: read oid 15 snap -1 2024-03-04T06:19:58.312 INFO:tasks.rados.rados.0.smithi064.stdout:2875: expect deleted 2024-03-04T06:19:58.312 INFO:tasks.rados.rados.0.smithi064.stdout:2876: write (excl) oid 34 current snap is 360 2024-03-04T06:19:58.312 INFO:tasks.rados.rados.0.smithi064.stdout:2876: seq_num 1310 ranges {417170=588906,1674823=342347} 2024-03-04T06:19:58.318 INFO:tasks.rados.rados.0.smithi064.stdout:2876: writing smithi06431743-34 from 417170 to 1006076 tid 1 2024-03-04T06:19:58.325 INFO:tasks.rados.rados.0.smithi064.stdout:2876: writing smithi06431743-34 from 1674823 to 2017170 tid 2 2024-03-04T06:19:58.326 INFO:tasks.rados.rados.0.smithi064.stdout:2877: read oid 13 snap -1 2024-03-04T06:19:58.326 INFO:tasks.rados.rados.0.smithi064.stdout:2877: expect (ObjNum 1295 snap 355 seq_num 1295) 2024-03-04T06:19:58.326 INFO:tasks.rados.rados.0.smithi064.stdout:2878: snap_remove snap 321 2024-03-04T06:19:59.095 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:19:59.214 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:59.212+0000 7fd45bf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:59.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:19:59.216+0000 7fd45bf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:19:59.303 INFO:tasks.rados.rados.0.smithi064.stdout:2878: done (5 left) 2024-03-04T06:19:59.303 INFO:tasks.rados.rados.0.smithi064.stdout:2879: delete oid 44 current snap is 360 2024-03-04T06:19:59.303 INFO:tasks.rados.rados.0.smithi064.stdout:2874: expect (ObjNum 1185 snap 318 seq_num 1185) 2024-03-04T06:19:59.507 INFO:tasks.rados.rados.0.smithi064.stdout:2877: expect (ObjNum 1295 snap 355 seq_num 1295) 2024-03-04T06:19:59.780 INFO:tasks.rados.rados.0.smithi064.stdout:2873: finishing write tid 1 to smithi06431743-8 2024-03-04T06:19:59.780 INFO:tasks.rados.rados.0.smithi064.stdout:2876: finishing write tid 1 to smithi06431743-34 2024-03-04T06:19:59.780 INFO:tasks.rados.rados.0.smithi064.stdout:2873: finishing write tid 2 to smithi06431743-8 2024-03-04T06:19:59.780 INFO:tasks.rados.rados.0.smithi064.stdout:2874: done (5 left) 2024-03-04T06:19:59.781 INFO:tasks.rados.rados.0.smithi064.stdout:2875: done (4 left) 2024-03-04T06:19:59.781 INFO:tasks.rados.rados.0.smithi064.stdout:2877: done (3 left) 2024-03-04T06:19:59.781 INFO:tasks.rados.rados.0.smithi064.stdout:2879: done (2 left) 2024-03-04T06:19:59.781 INFO:tasks.rados.rados.0.smithi064.stdout:2880: write (excl) oid 24 current snap is 360 2024-03-04T06:19:59.781 INFO:tasks.rados.rados.0.smithi064.stdout:2876: finishing write tid 2 to smithi06431743-34 2024-03-04T06:19:59.781 INFO:tasks.rados.rados.0.smithi064.stdout:2880: seq_num 1312 ranges {450784=739404,1791787=258997} 2024-03-04T06:19:59.786 INFO:tasks.rados.rados.0.smithi064.stdout:2880: writing smithi06431743-24 from 450784 to 1190188 tid 1 2024-03-04T06:19:59.792 INFO:tasks.rados.rados.0.smithi064.stdout:2880: writing smithi06431743-24 from 1791787 to 2050784 tid 2 2024-03-04T06:19:59.792 INFO:tasks.rados.rados.0.smithi064.stdout:2881: read oid 48 snap -1 2024-03-04T06:19:59.792 INFO:tasks.rados.rados.0.smithi064.stdout:2881: expect (ObjNum 1292 snap 354 seq_num 1292) 2024-03-04T06:19:59.793 INFO:tasks.rados.rados.0.smithi064.stdout:2882: delete oid 36 current snap is 360 2024-03-04T06:19:59.793 INFO:tasks.rados.rados.0.smithi064.stdout:2873: finishing write tid 3 to smithi06431743-8 2024-03-04T06:19:59.793 INFO:tasks.rados.rados.0.smithi064.stdout:2873: finishing write tid 4 to smithi06431743-8 2024-03-04T06:19:59.793 INFO:tasks.rados.rados.0.smithi064.stdout:2873: oid 8 updating version 0 to 732 2024-03-04T06:19:59.793 INFO:tasks.rados.rados.0.smithi064.stdout:2873: oid 8 version 732 is already newer than 731 2024-03-04T06:19:59.793 INFO:tasks.rados.rados.0.smithi064.stdout:2873: oid 8 version 732 is already newer than 731 2024-03-04T06:19:59.793 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 732 (ObjNum 1309 snap 360 seq_num 1309) dirty exists 2024-03-04T06:19:59.793 INFO:tasks.rados.rados.0.smithi064.stdout:2873: left oid 8 (ObjNum 1309 snap 360 seq_num 1309) 2024-03-04T06:19:59.793 INFO:tasks.rados.rados.0.smithi064.stdout:2876: finishing write tid 3 to smithi06431743-34 2024-03-04T06:19:59.793 INFO:tasks.rados.rados.0.smithi064.stdout:2876: finishing write tid 4 to smithi06431743-34 2024-03-04T06:19:59.793 INFO:tasks.rados.rados.0.smithi064.stdout:2876: oid 34 updating version 0 to 1274 2024-03-04T06:19:59.794 INFO:tasks.rados.rados.0.smithi064.stdout:2876: oid 34 updating version 1274 to 1276 2024-03-04T06:19:59.794 INFO:tasks.rados.rados.0.smithi064.stdout:2876: oid 34 version 1276 is already newer than 1275 2024-03-04T06:19:59.794 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 1276 (ObjNum 1310 snap 360 seq_num 1310) dirty exists 2024-03-04T06:19:59.794 INFO:tasks.rados.rados.0.smithi064.stdout:2876: left oid 34 (ObjNum 1310 snap 360 seq_num 1310) 2024-03-04T06:19:59.923 INFO:tasks.rados.rados.0.smithi064.stdout:2880: finishing write tid 1 to smithi06431743-24 2024-03-04T06:19:59.931 INFO:tasks.rados.rados.0.smithi064.stdout:2881: expect (ObjNum 1292 snap 354 seq_num 1292) 2024-03-04T06:20:00.077 INFO:tasks.rados.rados.0.smithi064.stdout:2880: finishing write tid 2 to smithi06431743-24 2024-03-04T06:20:00.077 INFO:tasks.rados.rados.0.smithi064.stdout:2880: finishing write tid 3 to smithi06431743-24 2024-03-04T06:20:00.077 INFO:tasks.rados.rados.0.smithi064.stdout:2880: finishing write tid 4 to smithi06431743-24 2024-03-04T06:20:00.077 INFO:tasks.rados.rados.0.smithi064.stdout:2880: oid 24 updating version 0 to 971 2024-03-04T06:20:00.077 INFO:tasks.rados.rados.0.smithi064.stdout:2880: oid 24 version 971 is already newer than 971 2024-03-04T06:20:00.077 INFO:tasks.rados.rados.0.smithi064.stdout:2880: oid 24 version 971 is already newer than 969 2024-03-04T06:20:00.078 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 971 (ObjNum 1312 snap 360 seq_num 1312) dirty exists 2024-03-04T06:20:00.078 INFO:tasks.rados.rados.0.smithi064.stdout:2880: left oid 24 (ObjNum 1312 snap 360 seq_num 1312) 2024-03-04T06:20:00.078 INFO:tasks.rados.rados.0.smithi064.stdout:2873: done (4 left) 2024-03-04T06:20:00.078 INFO:tasks.rados.rados.0.smithi064.stdout:2876: done (3 left) 2024-03-04T06:20:00.078 INFO:tasks.rados.rados.0.smithi064.stdout:2880: done (2 left) 2024-03-04T06:20:00.078 INFO:tasks.rados.rados.0.smithi064.stdout:2881: done (1 left) 2024-03-04T06:20:00.078 INFO:tasks.rados.rados.0.smithi064.stdout:2882: done (0 left) 2024-03-04T06:20:00.078 INFO:tasks.rados.rados.0.smithi064.stdout:2883: snap_create 2024-03-04T06:20:00.343 INFO:tasks.rados.rados.0.smithi064.stdout:2883: done (0 left) 2024-03-04T06:20:00.343 INFO:tasks.rados.rados.0.smithi064.stdout:2884: write oid 11 current snap is 361 2024-03-04T06:20:00.343 INFO:tasks.rados.rados.0.smithi064.stdout:2884: seq_num 1314 ranges {484398=489902,1392398=411118} 2024-03-04T06:20:00.348 INFO:tasks.rados.rados.0.smithi064.stdout:2884: writing smithi06431743-11 from 484398 to 974300 tid 1 2024-03-04T06:20:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2884: writing smithi06431743-11 from 1392398 to 1803516 tid 2 2024-03-04T06:20:00.354 INFO:tasks.rados.rados.0.smithi064.stdout:2885: delete oid 18 current snap is 361 2024-03-04T06:20:00.407 INFO:tasks.rados.rados.0.smithi064.stdout:2884: finishing write tid 1 to smithi06431743-11 2024-03-04T06:20:00.423 INFO:tasks.rados.rados.0.smithi064.stdout:2885: done (1 left) 2024-03-04T06:20:00.423 INFO:tasks.rados.rados.0.smithi064.stdout:2886: snap_remove snap 347 2024-03-04T06:20:01.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:01.357 INFO:tasks.rados.rados.0.smithi064.stdout:2886: done (1 left) 2024-03-04T06:20:01.357 INFO:tasks.rados.rados.0.smithi064.stdout:2887: snap_remove snap 346 2024-03-04T06:20:01.418 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:01.415+0000 7f2e3539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:01.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:01.420+0000 7f2e3539d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:02.287 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:20:02.287 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:20:02.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:20:02.357 INFO:tasks.rados.rados.0.smithi064.stdout:2887: done (1 left) 2024-03-04T06:20:02.357 INFO:tasks.rados.rados.0.smithi064.stdout:2888: read oid 39 snap -1 2024-03-04T06:20:02.357 INFO:tasks.rados.rados.0.smithi064.stdout:2888: expect (ObjNum 1207 snap 321 seq_num 1207) 2024-03-04T06:20:02.357 INFO:tasks.rados.rados.0.smithi064.stdout:2889: read oid 1 snap -1 2024-03-04T06:20:02.357 INFO:tasks.rados.rados.0.smithi064.stdout:2889: expect deleted 2024-03-04T06:20:02.357 INFO:tasks.rados.rados.0.smithi064.stdout:2890: snap_create 2024-03-04T06:20:02.357 INFO:tasks.rados.rados.0.smithi064.stdout:2884: finishing write tid 2 to smithi06431743-11 2024-03-04T06:20:02.357 INFO:tasks.rados.rados.0.smithi064.stdout:2884: finishing write tid 3 to smithi06431743-11 2024-03-04T06:20:02.358 INFO:tasks.rados.rados.0.smithi064.stdout:2884: finishing write tid 4 to smithi06431743-11 2024-03-04T06:20:02.358 INFO:tasks.rados.rados.0.smithi064.stdout:2884: oid 11 updating version 0 to 729 2024-03-04T06:20:02.358 INFO:tasks.rados.rados.0.smithi064.stdout:2884: oid 11 version 729 is already newer than 728 2024-03-04T06:20:02.358 INFO:tasks.rados.rados.0.smithi064.stdout:2884: oid 11 updating version 729 to 730 2024-03-04T06:20:02.358 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 730 (ObjNum 1314 snap 361 seq_num 1314) dirty exists 2024-03-04T06:20:02.358 INFO:tasks.rados.rados.0.smithi064.stdout:2884: left oid 11 (ObjNum 1314 snap 361 seq_num 1314) 2024-03-04T06:20:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:02.409+0000 7fdabbd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:02.421 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:02.420+0000 7fdabbd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:02.430 INFO:tasks.rados.rados.0.smithi064.stdout:2888: expect (ObjNum 1207 snap 321 seq_num 1207) 2024-03-04T06:20:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:20:02.620 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:20:02.632 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1090,"stamp":"2024-03-04T06:20:02.354043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120065546,"num_objects":974,"num_object_clones":777,"num_object_copies":2922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":158,"num_read":5324,"num_read_kb":4121968,"num_write":4531,"num_write_kb":1202238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11404,"ondisk_log_size":11404,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":677680,"kb_used_data":473232,"kb_used_omap":0,"kb_used_meta":204416,"kb_avail":376809680,"statfs":{"total":386547056640,"available":385853112320,"internally_reserved":0,"allocated":484589568,"data_stored":564683221,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":209321984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5882318,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":33,"num_read_kb":32752,"num_write":29,"num_write_kb":7115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.058047"},"pg_stats":[{"pgid":"3.f","version":"728'665","reported_seq":2511,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-03-04T06:19:59.348940+0000","last_change":"2024-03-04T06:19:59.385539+0000","last_active":"2024-03-04T06:19:59.348940+0000","last_peered":"2024-03-04T06:19:59.348940+0000","last_clean":"2024-03-04T06:19:59.348940+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:19:59.348940+0000","last_undegraded":"2024-03-04T06:19:59.348940+0000","last_fullsized":"2024-03-04T06:19:59.348940+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":665,"log_dups_size":0,"ondisk_log_size":665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7345253,"num_objects":63,"num_object_clones":50,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":300,"num_read_kb":245921,"num_write":264,"num_write_kb":70457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"728'565","reported_seq":2467,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.372613+0000","last_change":"2024-03-04T06:19:59.877425+0000","last_active":"2024-03-04T06:20:00.372613+0000","last_peered":"2024-03-04T06:20:00.372613+0000","last_clean":"2024-03-04T06:20:00.372613+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:20:00.372613+0000","last_undegraded":"2024-03-04T06:20:00.372613+0000","last_fullsized":"2024-03-04T06:20:00.372613+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6975499,"num_objects":49,"num_object_clones":35,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":183,"num_read_kb":130655,"num_write":202,"num_write_kb":53508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"729'730","reported_seq":2623,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.436080+0000","last_change":"2024-03-04T06:20:00.436080+0000","last_active":"2024-03-04T06:20:00.436080+0000","last_peered":"2024-03-04T06:20:00.436080+0000","last_clean":"2024-03-04T06:20:00.436080+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:20:00.436080+0000","last_undegraded":"2024-03-04T06:20:00.436080+0000","last_fullsized":"2024-03-04T06:20:00.436080+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":730,"log_dups_size":0,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11127455,"num_objects":71,"num_object_clones":56,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":420,"num_read_kb":341904,"num_write":314,"num_write_kb":85059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"728'632","reported_seq":2468,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-03-04T06:19:59.335087+0000","last_change":"2024-03-04T06:19:59.363691+0000","last_active":"2024-03-04T06:19:59.335087+0000","last_peered":"2024-03-04T06:19:59.335087+0000","last_clean":"2024-03-04T06:19:59.335087+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:19:59.335087+0000","last_undegraded":"2024-03-04T06:19:59.335087+0000","last_fullsized":"2024-03-04T06:19:59.335087+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10438038,"num_objects":48,"num_object_clones":38,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":259,"num_read_kb":202834,"num_write":234,"num_write_kb":66141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"728'265","reported_seq":2304,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-04T06:20:01.375064+0000","last_change":"2024-03-04T06:20:01.384046+0000","last_active":"2024-03-04T06:20:01.375064+0000","last_peered":"2024-03-04T06:20:01.375064+0000","last_clean":"2024-03-04T06:20:01.375064+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:20:01.375064+0000","last_undegraded":"2024-03-04T06:20:01.375064+0000","last_fullsized":"2024-03-04T06:20:01.375064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":90,"num_read_kb":65483,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"728'777","reported_seq":2563,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-03-04T06:19:59.371209+0000","last_change":"2024-03-04T06:19:59.407051+0000","last_active":"2024-03-04T06:19:59.371209+0000","last_peered":"2024-03-04T06:19:59.371209+0000","last_clean":"2024-03-04T06:19:59.371209+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:19:59.371209+0000","last_undegraded":"2024-03-04T06:19:59.371209+0000","last_fullsized":"2024-03-04T06:19:59.371209+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6588938,"num_objects":60,"num_object_clones":51,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":346,"num_read_kb":278960,"num_write":299,"num_write_kb":87675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2140,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-03-04T06:19:59.347009+0000","last_change":"2024-03-04T06:19:59.355298+0000","last_active":"2024-03-04T06:19:59.347009+0000","last_peered":"2024-03-04T06:19:59.347009+0000","last_clean":"2024-03-04T06:19:59.347009+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:19:59.347009+0000","last_undegraded":"2024-03-04T06:19:59.347009+0000","last_fullsized":"2024-03-04T06:19:59.347009+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"729'414","reported_seq":2382,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.369335+0000","last_change":"2024-03-04T06:20:00.404301+0000","last_active":"2024-03-04T06:20:00.369335+0000","last_peered":"2024-03-04T06:20:00.369335+0000","last_clean":"2024-03-04T06:20:00.369335+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:20:00.369335+0000","last_undegraded":"2024-03-04T06:20:00.369335+0000","last_fullsized":"2024-03-04T06:20:00.369335+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672940,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":214,"num_read_kb":195370,"num_write":145,"num_write_kb":38810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1971,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-04T06:20:01.373451+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:20:01.373451+0000","last_peered":"2024-03-04T06:20:01.373451+0000","last_clean":"2024-03-04T06:20:01.373451+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:20:01.373451+0000","last_undegraded":"2024-03-04T06:20:01.373451+0000","last_fullsized":"2024-03-04T06:20:01.373451+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1815,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.380962+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:20:00.380962+0000","last_peered":"2024-03-04T06:20:00.380962+0000","last_clean":"2024-03-04T06:20:00.380962+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:20:00.380962+0000","last_undegraded":"2024-03-04T06:20:00.380962+0000","last_fullsized":"2024-03-04T06:20:00.380962+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"729'705","reported_seq":2556,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.376770+0000","last_change":"2024-03-04T06:20:00.406373+0000","last_active":"2024-03-04T06:20:00.376770+0000","last_peered":"2024-03-04T06:20:00.376770+0000","last_clean":"2024-03-04T06:20:00.376770+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:20:00.376770+0000","last_undegraded":"2024-03-04T06:20:00.376770+0000","last_fullsized":"2024-03-04T06:20:00.376770+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6286412,"num_objects":62,"num_object_clones":48,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":340,"num_read_kb":275379,"num_write":276,"num_write_kb":74562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1800,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-03-04T06:19:59.358822+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:19:59.358822+0000","last_peered":"2024-03-04T06:19:59.358822+0000","last_clean":"2024-03-04T06:19:59.358822+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:19:59.358822+0000","last_undegraded":"2024-03-04T06:19:59.358822+0000","last_fullsized":"2024-03-04T06:19:59.358822+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"728'446","reported_seq":2446,"reported_epoch":730,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:01.360895+0000","last_change":"2024-03-04T06:20:01.360895+0000","last_active":"2024-03-04T06:20:01.360895+0000","last_peered":"2024-03-04T06:20:01.360895+0000","last_clean":"2024-03-04T06:20:01.360895+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:20:01.360895+0000","last_undegraded":"2024-03-04T06:20:01.360895+0000","last_fullsized":"2024-03-04T06:20:01.360895+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7446095,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":261,"num_read_kb":212085,"num_write":166,"num_write_kb":47091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1815,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.364584+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:20:00.364584+0000","last_peered":"2024-03-04T06:20:00.364584+0000","last_clean":"2024-03-04T06:20:00.364584+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:20:00.364584+0000","last_undegraded":"2024-03-04T06:20:00.364584+0000","last_fullsized":"2024-03-04T06:20:00.364584+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"728'1282","reported_seq":2961,"reported_epoch":730,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:01.381221+0000","last_change":"2024-03-04T06:20:01.381221+0000","last_active":"2024-03-04T06:20:01.381221+0000","last_peered":"2024-03-04T06:20:01.381221+0000","last_clean":"2024-03-04T06:20:01.381221+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:20:01.381221+0000","last_undegraded":"2024-03-04T06:20:01.381221+0000","last_fullsized":"2024-03-04T06:20:01.381221+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11413141,"num_objects":115,"num_object_clones":91,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":671,"num_read_kb":527834,"num_write":535,"num_write_kb":142941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1800,"reported_epoch":728,"state":"active+clean","last_fresh":"2024-03-04T06:19:59.340397+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:19:59.340397+0000","last_peered":"2024-03-04T06:19:59.340397+0000","last_clean":"2024-03-04T06:19:59.340397+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:19:59.340397+0000","last_undegraded":"2024-03-04T06:19:59.340397+0000","last_fullsized":"2024-03-04T06:19:59.340397+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"728'1458","reported_seq":2935,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.369308+0000","last_change":"2024-03-04T06:19:59.981724+0000","last_active":"2024-03-04T06:20:00.369308+0000","last_peered":"2024-03-04T06:20:00.369308+0000","last_clean":"2024-03-04T06:20:00.369308+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:20:00.369308+0000","last_undegraded":"2024-03-04T06:20:00.369308+0000","last_fullsized":"2024-03-04T06:20:00.369308+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18858939,"num_objects":121,"num_object_clones":98,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":526,"num_read_kb":376502,"num_write":561,"num_write_kb":150818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"729'705","reported_seq":2575,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.380173+0000","last_change":"2024-03-04T06:20:00.432256+0000","last_active":"2024-03-04T06:20:00.380173+0000","last_peered":"2024-03-04T06:20:00.380173+0000","last_clean":"2024-03-04T06:20:00.380173+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:20:00.380173+0000","last_undegraded":"2024-03-04T06:20:00.380173+0000","last_fullsized":"2024-03-04T06:20:00.380173+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5765395,"num_objects":57,"num_object_clones":45,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":392,"num_read_kb":329117,"num_write":267,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1865,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.372373+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:20:00.372373+0000","last_peered":"2024-03-04T06:20:00.372373+0000","last_clean":"2024-03-04T06:20:00.372373+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:20:00.372373+0000","last_undegraded":"2024-03-04T06:20:00.372373+0000","last_fullsized":"2024-03-04T06:20:00.372373+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"728'738","reported_seq":2608,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.378978+0000","last_change":"2024-03-04T06:19:59.927679+0000","last_active":"2024-03-04T06:20:00.378978+0000","last_peered":"2024-03-04T06:20:00.378978+0000","last_clean":"2024-03-04T06:20:00.378978+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:20:00.378978+0000","last_undegraded":"2024-03-04T06:20:00.378978+0000","last_fullsized":"2024-03-04T06:20:00.378978+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":738,"log_dups_size":0,"ondisk_log_size":738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210797,"num_objects":70,"num_object_clones":54,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":324,"num_read_kb":236247,"num_write":293,"num_write_kb":78994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1838,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-04T06:20:01.378107+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:20:01.378107+0000","last_peered":"2024-03-04T06:20:01.378107+0000","last_clean":"2024-03-04T06:20:01.378107+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:20:01.378107+0000","last_undegraded":"2024-03-04T06:20:01.378107+0000","last_fullsized":"2024-03-04T06:20:01.378107+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"728'973","reported_seq":2693,"reported_epoch":730,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:01.362973+0000","last_change":"2024-03-04T06:20:01.362973+0000","last_active":"2024-03-04T06:20:01.362973+0000","last_peered":"2024-03-04T06:20:01.362973+0000","last_clean":"2024-03-04T06:20:01.362973+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:20:01.362973+0000","last_undegraded":"2024-03-04T06:20:01.362973+0000","last_fullsized":"2024-03-04T06:20:01.362973+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12404754,"num_objects":82,"num_object_clones":68,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":395,"num_read_kb":280818,"num_write":385,"num_write_kb":107739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1865,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.367788+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:20:00.367788+0000","last_peered":"2024-03-04T06:20:00.367788+0000","last_clean":"2024-03-04T06:20:00.367788+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:20:00.367788+0000","last_undegraded":"2024-03-04T06:20:00.367788+0000","last_fullsized":"2024-03-04T06:20:00.367788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"728'952","reported_seq":2751,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.370902+0000","last_change":"2024-03-04T06:19:59.902942+0000","last_active":"2024-03-04T06:20:00.370902+0000","last_peered":"2024-03-04T06:20:00.370902+0000","last_clean":"2024-03-04T06:20:00.370902+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:20:00.370902+0000","last_undegraded":"2024-03-04T06:20:00.370902+0000","last_fullsized":"2024-03-04T06:20:00.370902+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":952,"log_dups_size":0,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":80,"num_object_clones":70,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":500,"num_read_kb":422768,"num_write":356,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1815,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-04T06:20:00.375851+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:20:00.375851+0000","last_peered":"2024-03-04T06:20:00.375851+0000","last_clean":"2024-03-04T06:20:00.375851+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:20:00.375851+0000","last_undegraded":"2024-03-04T06:20:00.375851+0000","last_fullsized":"2024-03-04T06:20:00.375851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":119475178,"num_objects":972,"num_object_clones":777,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":158,"num_read":5221,"num_read_kb":4121877,"num_write":4394,"num_write_kb":1200344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11307,"ondisk_log_size":11307,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706399,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":201972,"kb_used_data":144620,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94169868,"statfs":{"total":96636764160,"available":96429944832,"internally_reserved":0,"allocated":148090880,"data_stored":171791160,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706399,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":149460,"kb_used_data":100428,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94222380,"statfs":{"total":96636764160,"available":96483717120,"internally_reserved":0,"allocated":102838272,"data_stored":116912933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706398,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":161228,"kb_used_data":112196,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94210612,"statfs":{"total":96636764160,"available":96471666688,"internally_reserved":0,"allocated":114888704,"data_stored":134606264,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706398,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":165020,"kb_used_data":115988,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94206820,"statfs":{"total":96636764160,"available":96467783680,"internally_reserved":0,"allocated":118771712,"data_stored":141372864,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:20:03.355 INFO:tasks.rados.rados.0.smithi064.stdout:2884: done (3 left) 2024-03-04T06:20:03.355 INFO:tasks.rados.rados.0.smithi064.stdout:2888: done (2 left) 2024-03-04T06:20:03.355 INFO:tasks.rados.rados.0.smithi064.stdout:2889: done (1 left) 2024-03-04T06:20:03.355 INFO:tasks.rados.rados.0.smithi064.stdout:2890: done (0 left) 2024-03-04T06:20:03.355 INFO:tasks.rados.rados.0.smithi064.stdout:2891: read oid 34 snap -1 2024-03-04T06:20:03.355 INFO:tasks.rados.rados.0.smithi064.stdout:2891: expect (ObjNum 1310 snap 360 seq_num 1310) 2024-03-04T06:20:03.355 INFO:tasks.rados.rados.0.smithi064.stdout:2892: delete oid 38 current snap is 362 2024-03-04T06:20:03.410 INFO:tasks.rados.rados.0.smithi064.stdout:2892: done (1 left) 2024-03-04T06:20:03.410 INFO:tasks.rados.rados.0.smithi064.stdout:2893: snap_remove snap 341 2024-03-04T06:20:03.488 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:03.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:03.600+0000 7f74e8a0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:03.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:03.605+0000 7f74e8a0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:04.366 INFO:tasks.rados.rados.0.smithi064.stdout:2893: done (1 left) 2024-03-04T06:20:04.366 INFO:tasks.rados.rados.0.smithi064.stdout:2894: snap_create 2024-03-04T06:20:04.366 INFO:tasks.rados.rados.0.smithi064.stdout:2891: expect (ObjNum 1310 snap 360 seq_num 1310) 2024-03-04T06:20:05.370 INFO:tasks.rados.rados.0.smithi064.stdout:2891: done (1 left) 2024-03-04T06:20:05.370 INFO:tasks.rados.rados.0.smithi064.stdout:2894: done (0 left) 2024-03-04T06:20:05.370 INFO:tasks.rados.rados.0.smithi064.stdout:2895: snap_create 2024-03-04T06:20:05.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:05.792 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:05.790+0000 7efebc4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:05.795 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:05.795+0000 7efebc4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:06.369 INFO:tasks.rados.rados.0.smithi064.stdout:2895: done (0 left) 2024-03-04T06:20:06.370 INFO:tasks.rados.rados.0.smithi064.stdout:2896: read oid 14 snap -1 2024-03-04T06:20:06.370 INFO:tasks.rados.rados.0.smithi064.stdout:2896: expect (ObjNum 1226 snap 330 seq_num 1226) 2024-03-04T06:20:06.370 INFO:tasks.rados.rados.0.smithi064.stdout:2897: write oid 23 current snap is 364 2024-03-04T06:20:06.370 INFO:tasks.rados.rados.0.smithi064.stdout:2897: seq_num 1317 ranges {534819=432002,1684197=450622} 2024-03-04T06:20:06.375 INFO:tasks.rados.rados.0.smithi064.stdout:2897: writing smithi06431743-23 from 534819 to 966821 tid 1 2024-03-04T06:20:06.381 INFO:tasks.rados.rados.0.smithi064.stdout:2897: writing smithi06431743-23 from 1684197 to 2134819 tid 2 2024-03-04T06:20:06.381 INFO:tasks.rados.rados.0.smithi064.stdout:2898: snap_remove snap 331 2024-03-04T06:20:07.381 INFO:tasks.rados.rados.0.smithi064.stdout:2898: done (2 left) 2024-03-04T06:20:07.381 INFO:tasks.rados.rados.0.smithi064.stdout:2899: snap_create 2024-03-04T06:20:07.381 INFO:tasks.rados.rados.0.smithi064.stdout:2896: expect (ObjNum 1226 snap 330 seq_num 1226) 2024-03-04T06:20:07.570 INFO:tasks.rados.rados.0.smithi064.stdout:2897: finishing write tid 1 to smithi06431743-23 2024-03-04T06:20:07.570 INFO:tasks.rados.rados.0.smithi064.stdout:2897: finishing write tid 2 to smithi06431743-23 2024-03-04T06:20:07.570 INFO:tasks.rados.rados.0.smithi064.stdout:2897: finishing write tid 3 to smithi06431743-23 2024-03-04T06:20:07.570 INFO:tasks.rados.rados.0.smithi064.stdout:2897: finishing write tid 4 to smithi06431743-23 2024-03-04T06:20:07.570 INFO:tasks.rados.rados.0.smithi064.stdout:2897: oid 23 updating version 0 to 711 2024-03-04T06:20:07.570 INFO:tasks.rados.rados.0.smithi064.stdout:2897: oid 23 version 711 is already newer than 710 2024-03-04T06:20:07.570 INFO:tasks.rados.rados.0.smithi064.stdout:2897: oid 23 version 711 is already newer than 710 2024-03-04T06:20:07.570 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 711 (ObjNum 1317 snap 364 seq_num 1317) dirty exists 2024-03-04T06:20:07.571 INFO:tasks.rados.rados.0.smithi064.stdout:2897: left oid 23 (ObjNum 1317 snap 364 seq_num 1317) 2024-03-04T06:20:07.633 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:20:07.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:20:07.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:07.738+0000 7f1bcfd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:07.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:07.746+0000 7f1bcfd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:07.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:07.972 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:07.971+0000 7fd2f6b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:07.975 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:07.975+0000 7fd2f6b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:08.296 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:20:08.302 INFO:tasks.rados.rados.0.smithi064.stdout:2896: done (2 left) 2024-03-04T06:20:08.302 INFO:tasks.rados.rados.0.smithi064.stdout:2897: done (1 left) 2024-03-04T06:20:08.302 INFO:tasks.rados.rados.0.smithi064.stdout:2899: done (0 left) 2024-03-04T06:20:08.302 INFO:tasks.rados.rados.0.smithi064.stdout:2900: read oid 13 snap -1 2024-03-04T06:20:08.302 INFO:tasks.rados.rados.0.smithi064.stdout:2900: expect (ObjNum 1295 snap 355 seq_num 1295) 2024-03-04T06:20:08.302 INFO:tasks.rados.rados.0.smithi064.stdout:2901: write (excl) oid 43 current snap is 365 2024-03-04T06:20:08.302 INFO:tasks.rados.rados.0.smithi064.stdout:2901: seq_num 1318 ranges {551626=507251,1626326=418456} 2024-03-04T06:20:08.309 INFO:tasks.rados.rados.0.smithi064.stdout:2901: writing smithi06431743-43 from 551626 to 1058877 tid 1 2024-03-04T06:20:08.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:20:08.316 INFO:tasks.rados.rados.0.smithi064.stdout:2901: writing smithi06431743-43 from 1626326 to 2044782 tid 2 2024-03-04T06:20:08.316 INFO:tasks.rados.rados.0.smithi064.stdout:2902: snap_remove snap 363 2024-03-04T06:20:08.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:08.456+0000 7fa5a5f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:08.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:08.467+0000 7fa5a5f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:09.303 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:20:09.306 INFO:tasks.rados.rados.0.smithi064.stdout:2902: done (2 left) 2024-03-04T06:20:09.306 INFO:tasks.rados.rados.0.smithi064.stdout:2903: snap_create 2024-03-04T06:20:09.307 INFO:tasks.rados.rados.0.smithi064.stdout:2900: expect (ObjNum 1295 snap 355 seq_num 1295) 2024-03-04T06:20:09.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:20:09.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:09.467+0000 7f51ac417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:09.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:09.475+0000 7f51ac417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:09.583 INFO:tasks.rados.rados.0.smithi064.stdout:2901: finishing write tid 1 to smithi06431743-43 2024-03-04T06:20:09.583 INFO:tasks.rados.rados.0.smithi064.stdout:2901: finishing write tid 2 to smithi06431743-43 2024-03-04T06:20:09.583 INFO:tasks.rados.rados.0.smithi064.stdout:2901: finishing write tid 3 to smithi06431743-43 2024-03-04T06:20:09.583 INFO:tasks.rados.rados.0.smithi064.stdout:2901: finishing write tid 4 to smithi06431743-43 2024-03-04T06:20:09.583 INFO:tasks.rados.rados.0.smithi064.stdout:2901: oid 43 updating version 0 to 680 2024-03-04T06:20:09.583 INFO:tasks.rados.rados.0.smithi064.stdout:2901: oid 43 updating version 680 to 681 2024-03-04T06:20:09.583 INFO:tasks.rados.rados.0.smithi064.stdout:2901: oid 43 version 681 is already newer than 675 2024-03-04T06:20:09.583 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 681 (ObjNum 1318 snap 365 seq_num 1318) dirty exists 2024-03-04T06:20:09.583 INFO:tasks.rados.rados.0.smithi064.stdout:2901: left oid 43 (ObjNum 1318 snap 365 seq_num 1318) 2024-03-04T06:20:10.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:10.150 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:10.148+0000 7f873439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:10.153 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:10.152+0000 7f873439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:10.303 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:20:10.306 INFO:tasks.rados.rados.0.smithi064.stdout:2900: done (2 left) 2024-03-04T06:20:10.306 INFO:tasks.rados.rados.0.smithi064.stdout:2901: done (1 left) 2024-03-04T06:20:10.306 INFO:tasks.rados.rados.0.smithi064.stdout:2903: done (0 left) 2024-03-04T06:20:10.306 INFO:tasks.rados.rados.0.smithi064.stdout:2904: write oid 18 current snap is 366 2024-03-04T06:20:10.306 INFO:tasks.rados.rados.0.smithi064.stdout:2904: seq_num 1319 ranges {568433=582500,1684808=483625} 2024-03-04T06:20:10.312 INFO:tasks.rados.rados.0.smithi064.stdout:2904: writing smithi06431743-18 from 568433 to 1150933 tid 1 2024-03-04T06:20:10.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:20:10.319 INFO:tasks.rados.rados.0.smithi064.stdout:2904: writing smithi06431743-18 from 1684808 to 2168433 tid 2 2024-03-04T06:20:10.319 INFO:tasks.rados.rados.0.smithi064.stdout:2905: write oid 49 current snap is 366 2024-03-04T06:20:10.319 INFO:tasks.rados.rados.0.smithi064.stdout:2905: seq_num 1320 ranges {585240=657749,1743290=422125} 2024-03-04T06:20:10.327 INFO:tasks.rados.rados.0.smithi064.stdout:2905: writing smithi06431743-49 from 585240 to 1242989 tid 1 2024-03-04T06:20:10.333 INFO:tasks.rados.rados.0.smithi064.stdout:2905: writing smithi06431743-49 from 1743290 to 2165415 tid 2 2024-03-04T06:20:10.333 INFO:tasks.rados.rados.0.smithi064.stdout:2906: write (excl) oid 39 current snap is 366 2024-03-04T06:20:10.334 INFO:tasks.rados.rados.0.smithi064.stdout:2906: seq_num 1321 ranges {602047=732998,1801772=400275} 2024-03-04T06:20:10.341 INFO:tasks.rados.rados.0.smithi064.stdout:2906: writing smithi06431743-39 from 602047 to 1335045 tid 1 2024-03-04T06:20:10.346 INFO:tasks.rados.rados.0.smithi064.stdout:2906: writing smithi06431743-39 from 1801772 to 2202047 tid 2 2024-03-04T06:20:10.346 INFO:tasks.rados.rados.0.smithi064.stdout:2907: delete oid 6 current snap is 366 2024-03-04T06:20:10.386 INFO:tasks.rados.rados.0.smithi064.stdout:2905: finishing write tid 1 to smithi06431743-49 2024-03-04T06:20:10.396 INFO:tasks.rados.rados.0.smithi064.stdout:2906: finishing write tid 1 to smithi06431743-39 2024-03-04T06:20:10.412 INFO:tasks.rados.rados.0.smithi064.stdout:2905: finishing write tid 2 to smithi06431743-49 2024-03-04T06:20:10.413 INFO:tasks.rados.rados.0.smithi064.stdout:2904: finishing write tid 1 to smithi06431743-18 2024-03-04T06:20:10.424 INFO:tasks.rados.rados.0.smithi064.stdout:2907: done (3 left) 2024-03-04T06:20:10.424 INFO:tasks.rados.rados.0.smithi064.stdout:2908: read oid 3 snap 365 2024-03-04T06:20:10.424 INFO:tasks.rados.rados.0.smithi064.stdout:2908: expect (ObjNum 1225 snap 329 seq_num 1225) 2024-03-04T06:20:10.424 INFO:tasks.rados.rados.0.smithi064.stdout:2909: snap_create 2024-03-04T06:20:10.425 INFO:tasks.rados.rados.0.smithi064.stdout:2904: finishing write tid 2 to smithi06431743-18 2024-03-04T06:20:10.426 INFO:tasks.rados.rados.0.smithi064.stdout:2906: finishing write tid 2 to smithi06431743-39 2024-03-04T06:20:10.435 INFO:tasks.rados.rados.0.smithi064.stdout:2905: finishing write tid 3 to smithi06431743-49 2024-03-04T06:20:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:2905: finishing write tid 4 to smithi06431743-49 2024-03-04T06:20:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:2905: oid 49 updating version 0 to 419 2024-03-04T06:20:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:2905: oid 49 updating version 419 to 420 2024-03-04T06:20:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:2905: oid 49 version 420 is already newer than 419 2024-03-04T06:20:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 420 (ObjNum 1320 snap 366 seq_num 1320) dirty exists 2024-03-04T06:20:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:2905: left oid 49 (ObjNum 1320 snap 366 seq_num 1320) 2024-03-04T06:20:10.439 INFO:tasks.rados.rados.0.smithi064.stdout:2904: finishing write tid 3 to smithi06431743-18 2024-03-04T06:20:10.440 INFO:tasks.rados.rados.0.smithi064.stdout:2904: finishing write tid 4 to smithi06431743-18 2024-03-04T06:20:10.440 INFO:tasks.rados.rados.0.smithi064.stdout:2904: oid 18 updating version 0 to 579 2024-03-04T06:20:10.440 INFO:tasks.rados.rados.0.smithi064.stdout:2904: oid 18 version 579 is already newer than 579 2024-03-04T06:20:10.440 INFO:tasks.rados.rados.0.smithi064.stdout:2904: oid 18 updating version 579 to 580 2024-03-04T06:20:10.440 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 580 (ObjNum 1319 snap 366 seq_num 1319) dirty exists 2024-03-04T06:20:10.440 INFO:tasks.rados.rados.0.smithi064.stdout:2904: left oid 18 (ObjNum 1319 snap 366 seq_num 1319) 2024-03-04T06:20:10.441 INFO:tasks.rados.rados.0.smithi064.stdout:2906: finishing write tid 3 to smithi06431743-39 2024-03-04T06:20:10.442 INFO:tasks.rados.rados.0.smithi064.stdout:2906: finishing write tid 4 to smithi06431743-39 2024-03-04T06:20:10.442 INFO:tasks.rados.rados.0.smithi064.stdout:2906: oid 39 updating version 0 to 747 2024-03-04T06:20:10.443 INFO:tasks.rados.rados.0.smithi064.stdout:2906: oid 39 version 747 is already newer than 746 2024-03-04T06:20:10.443 INFO:tasks.rados.rados.0.smithi064.stdout:2906: oid 39 updating version 747 to 748 2024-03-04T06:20:10.443 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 748 (ObjNum 1321 snap 366 seq_num 1321) dirty exists 2024-03-04T06:20:10.443 INFO:tasks.rados.rados.0.smithi064.stdout:2906: left oid 39 (ObjNum 1321 snap 366 seq_num 1321) 2024-03-04T06:20:10.443 INFO:tasks.rados.rados.0.smithi064.stdout:2908: expect (ObjNum 1225 snap 329 seq_num 1225) 2024-03-04T06:20:10.446 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:10.440+0000 7fc93ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:10.450 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:10.449+0000 7fc93ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:11.380 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:20:11.383 INFO:tasks.rados.rados.0.smithi064.stdout:2904: done (4 left) 2024-03-04T06:20:11.383 INFO:tasks.rados.rados.0.smithi064.stdout:2905: done (3 left) 2024-03-04T06:20:11.383 INFO:tasks.rados.rados.0.smithi064.stdout:2906: done (2 left) 2024-03-04T06:20:11.383 INFO:tasks.rados.rados.0.smithi064.stdout:2908: done (1 left) 2024-03-04T06:20:11.384 INFO:tasks.rados.rados.0.smithi064.stdout:2909: done (0 left) 2024-03-04T06:20:11.384 INFO:tasks.rados.rados.0.smithi064.stdout:2910: write (excl) oid 2 current snap is 367 2024-03-04T06:20:11.384 INFO:tasks.rados.rados.0.smithi064.stdout:2910: seq_num 1323 ranges {635661=599849,1918736=316925} 2024-03-04T06:20:11.390 INFO:tasks.rados.rados.0.smithi064.stdout:2910: writing smithi06431743-2 from 635661 to 1235510 tid 1 2024-03-04T06:20:11.397 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:20:11.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:20:11.398 INFO:tasks.rados.rados.0.smithi064.stdout:2910: writing smithi06431743-2 from 1918736 to 2235661 tid 2 2024-03-04T06:20:11.398 INFO:tasks.rados.rados.0.smithi064.stdout:2911: read oid 28 snap -1 2024-03-04T06:20:11.398 INFO:tasks.rados.rados.0.smithi064.stdout:2911: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:20:11.398 INFO:tasks.rados.rados.0.smithi064.stdout:2912: write oid 16 current snap is 367 2024-03-04T06:20:11.398 INFO:tasks.rados.rados.0.smithi064.stdout:2912: seq_num 1324 ranges {652468=675098,1977218=275250} 2024-03-04T06:20:11.406 INFO:tasks.rados.rados.0.smithi064.stdout:2912: writing smithi06431743-16 from 652468 to 1327566 tid 1 2024-03-04T06:20:11.411 INFO:tasks.rados.rados.0.smithi064.stdout:2912: writing smithi06431743-16 from 1977218 to 2252468 tid 2 2024-03-04T06:20:11.411 INFO:tasks.rados.rados.0.smithi064.stdout:2913: read oid 20 snap -1 2024-03-04T06:20:11.412 INFO:tasks.rados.rados.0.smithi064.stdout:2913: expect (ObjNum 1177 snap 315 seq_num 1177) 2024-03-04T06:20:11.412 INFO:tasks.rados.rados.0.smithi064.stdout:2914: snap_create 2024-03-04T06:20:11.493 INFO:tasks.rados.rados.0.smithi064.stdout:2910: finishing write tid 1 to smithi06431743-2 2024-03-04T06:20:11.505 INFO:tasks.rados.rados.0.smithi064.stdout:2911: expect (ObjNum 1132 snap 300 seq_num 1132) 2024-03-04T06:20:11.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:11.525+0000 7f257a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:11.533 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:11.532+0000 7f257a59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:11.663 INFO:tasks.rados.rados.0.smithi064.stdout:2913: expect (ObjNum 1177 snap 315 seq_num 1177) 2024-03-04T06:20:11.725 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:20:11.725 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:20:11.737 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1104,"stamp":"2024-03-04T06:20:11.381136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114502198,"num_objects":977,"num_object_clones":780,"num_object_copies":2931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":159,"num_read":5345,"num_read_kb":4143614,"num_write":4539,"num_write_kb":1204185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11547,"ondisk_log_size":11547,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":668672,"kb_used_data":464224,"kb_used_omap":0,"kb_used_meta":204416,"kb_avail":376818688,"statfs":{"total":386547056640,"available":385862336512,"internally_reserved":0,"allocated":475365376,"data_stored":551060034,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":209321984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":958362,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":8,"num_read_kb":9526,"num_write":6,"num_write_kb":1947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.004136"},"pg_stats":[{"pgid":"3.f","version":"738'683","reported_seq":2544,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.973490+0000","last_change":"2024-03-04T06:20:09.990769+0000","last_active":"2024-03-04T06:20:09.973490+0000","last_peered":"2024-03-04T06:20:09.973490+0000","last_clean":"2024-03-04T06:20:09.973490+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:20:09.973490+0000","last_undegraded":"2024-03-04T06:20:09.973490+0000","last_fullsized":"2024-03-04T06:20:09.973490+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135182,"num_objects":64,"num_object_clones":51,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":302,"num_read_kb":245922,"num_write":268,"num_write_kb":71362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"739'578","reported_seq":2501,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.348702+0000","last_change":"2024-03-04T06:20:09.939440+0000","last_active":"2024-03-04T06:20:10.348702+0000","last_peered":"2024-03-04T06:20:10.348702+0000","last_clean":"2024-03-04T06:20:10.348702+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:20:10.348702+0000","last_undegraded":"2024-03-04T06:20:10.348702+0000","last_fullsized":"2024-03-04T06:20:10.348702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":578,"log_dups_size":0,"ondisk_log_size":578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5950305,"num_objects":50,"num_object_clones":36,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":183,"num_read_kb":130655,"num_write":205,"num_write_kb":54550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"731'734","reported_seq":2635,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.436835+0000","last_change":"2024-03-04T06:20:02.436835+0000","last_active":"2024-03-04T06:20:02.436835+0000","last_peered":"2024-03-04T06:20:02.436835+0000","last_clean":"2024-03-04T06:20:02.436835+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:20:02.436835+0000","last_undegraded":"2024-03-04T06:20:02.436835+0000","last_fullsized":"2024-03-04T06:20:02.436835+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11127455,"num_objects":71,"num_object_clones":56,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":426,"num_read_kb":348114,"num_write":314,"num_write_kb":85059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"737'648","reported_seq":2497,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.962884+0000","last_change":"2024-03-04T06:20:09.971653+0000","last_active":"2024-03-04T06:20:09.962884+0000","last_peered":"2024-03-04T06:20:09.962884+0000","last_clean":"2024-03-04T06:20:09.962884+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:20:09.962884+0000","last_undegraded":"2024-03-04T06:20:09.962884+0000","last_fullsized":"2024-03-04T06:20:09.962884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10438038,"num_objects":48,"num_object_clones":38,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":259,"num_read_kb":202834,"num_write":234,"num_write_kb":66141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"728'265","reported_seq":2316,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:04.466926+0000","last_change":"2024-03-04T06:20:04.466926+0000","last_active":"2024-03-04T06:20:04.466926+0000","last_peered":"2024-03-04T06:20:04.466926+0000","last_clean":"2024-03-04T06:20:04.466926+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:20:04.466926+0000","last_undegraded":"2024-03-04T06:20:04.466926+0000","last_fullsized":"2024-03-04T06:20:04.466926+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":90,"num_read_kb":65483,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"737'793","reported_seq":2593,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.969637+0000","last_change":"2024-03-04T06:20:09.978981+0000","last_active":"2024-03-04T06:20:09.969637+0000","last_peered":"2024-03-04T06:20:09.969637+0000","last_clean":"2024-03-04T06:20:09.969637+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:20:09.969637+0000","last_undegraded":"2024-03-04T06:20:09.969637+0000","last_fullsized":"2024-03-04T06:20:09.969637+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":793,"log_dups_size":0,"ondisk_log_size":793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260855,"num_objects":61,"num_object_clones":52,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":347,"num_read_kb":278960,"num_write":300,"num_write_kb":87675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2169,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.971458+0000","last_change":"2024-03-04T06:20:09.978390+0000","last_active":"2024-03-04T06:20:09.971458+0000","last_peered":"2024-03-04T06:20:09.971458+0000","last_clean":"2024-03-04T06:20:09.971458+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:20:09.971458+0000","last_undegraded":"2024-03-04T06:20:09.971458+0000","last_fullsized":"2024-03-04T06:20:09.971458+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"729'414","reported_seq":2391,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.381941+0000","last_change":"2024-03-04T06:20:02.381941+0000","last_active":"2024-03-04T06:20:02.381941+0000","last_peered":"2024-03-04T06:20:02.381941+0000","last_clean":"2024-03-04T06:20:02.381941+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:20:02.381941+0000","last_undegraded":"2024-03-04T06:20:02.381941+0000","last_fullsized":"2024-03-04T06:20:02.381941+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672940,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":214,"num_read_kb":195370,"num_write":145,"num_write_kb":38810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1984,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:06.390293+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:20:06.390293+0000","last_peered":"2024-03-04T06:20:06.390293+0000","last_clean":"2024-03-04T06:20:06.390293+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:20:06.390293+0000","last_undegraded":"2024-03-04T06:20:06.390293+0000","last_fullsized":"2024-03-04T06:20:06.390293+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1821,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.392801+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:20:02.392801+0000","last_peered":"2024-03-04T06:20:02.392801+0000","last_clean":"2024-03-04T06:20:02.392801+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:20:02.392801+0000","last_undegraded":"2024-03-04T06:20:02.392801+0000","last_fullsized":"2024-03-04T06:20:02.392801+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"729'705","reported_seq":2565,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.399671+0000","last_change":"2024-03-04T06:20:02.399671+0000","last_active":"2024-03-04T06:20:02.399671+0000","last_peered":"2024-03-04T06:20:02.399671+0000","last_clean":"2024-03-04T06:20:02.399671+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:20:02.399671+0000","last_undegraded":"2024-03-04T06:20:02.399671+0000","last_fullsized":"2024-03-04T06:20:02.399671+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6286412,"num_objects":62,"num_object_clones":48,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":340,"num_read_kb":275379,"num_write":276,"num_write_kb":74562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1824,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.974283+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:20:09.974283+0000","last_peered":"2024-03-04T06:20:09.974283+0000","last_clean":"2024-03-04T06:20:09.974283+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:20:09.974283+0000","last_undegraded":"2024-03-04T06:20:09.974283+0000","last_fullsized":"2024-03-04T06:20:09.974283+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"733'452","reported_seq":2462,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:06.397845+0000","last_change":"2024-03-04T06:20:04.480635+0000","last_active":"2024-03-04T06:20:06.397845+0000","last_peered":"2024-03-04T06:20:06.397845+0000","last_clean":"2024-03-04T06:20:06.397845+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:20:06.397845+0000","last_undegraded":"2024-03-04T06:20:06.397845+0000","last_fullsized":"2024-03-04T06:20:06.397845+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7446095,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":261,"num_read_kb":212085,"num_write":166,"num_write_kb":47091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1821,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.369030+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:20:02.369030+0000","last_peered":"2024-03-04T06:20:02.369030+0000","last_clean":"2024-03-04T06:20:02.369030+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:20:02.369030+0000","last_undegraded":"2024-03-04T06:20:02.369030+0000","last_fullsized":"2024-03-04T06:20:02.369030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"733'1294","reported_seq":2977,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:04.488959+0000","last_change":"2024-03-04T06:20:04.488959+0000","last_active":"2024-03-04T06:20:04.488959+0000","last_peered":"2024-03-04T06:20:04.488959+0000","last_clean":"2024-03-04T06:20:04.488959+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:20:04.488959+0000","last_undegraded":"2024-03-04T06:20:04.488959+0000","last_fullsized":"2024-03-04T06:20:04.488959+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1294,"log_dups_size":0,"ondisk_log_size":1294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11413141,"num_objects":115,"num_object_clones":91,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":677,"num_read_kb":533744,"num_write":535,"num_write_kb":142941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1824,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.965851+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:20:09.965851+0000","last_peered":"2024-03-04T06:20:09.965851+0000","last_clean":"2024-03-04T06:20:09.965851+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:20:09.965851+0000","last_undegraded":"2024-03-04T06:20:09.965851+0000","last_fullsized":"2024-03-04T06:20:09.965851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"737'1480","reported_seq":2970,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.327238+0000","last_change":"2024-03-04T06:20:09.938152+0000","last_active":"2024-03-04T06:20:10.327238+0000","last_peered":"2024-03-04T06:20:10.327238+0000","last_clean":"2024-03-04T06:20:10.327238+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:20:10.327238+0000","last_undegraded":"2024-03-04T06:20:10.327238+0000","last_fullsized":"2024-03-04T06:20:10.327238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18858939,"num_objects":121,"num_object_clones":98,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":532,"num_read_kb":386027,"num_write":561,"num_write_kb":150818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"729'705","reported_seq":2583,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.383168+0000","last_change":"2024-03-04T06:20:02.399657+0000","last_active":"2024-03-04T06:20:02.383168+0000","last_peered":"2024-03-04T06:20:02.383168+0000","last_clean":"2024-03-04T06:20:02.383168+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:20:02.383168+0000","last_undegraded":"2024-03-04T06:20:02.383168+0000","last_fullsized":"2024-03-04T06:20:02.383168+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5765395,"num_objects":57,"num_object_clones":45,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":392,"num_read_kb":329117,"num_write":267,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1891,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.338755+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:20:10.338755+0000","last_peered":"2024-03-04T06:20:10.338755+0000","last_clean":"2024-03-04T06:20:10.338755+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:20:10.338755+0000","last_undegraded":"2024-03-04T06:20:10.338755+0000","last_fullsized":"2024-03-04T06:20:10.338755+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"737'752","reported_seq":2639,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.341679+0000","last_change":"2024-03-04T06:20:09.954608+0000","last_active":"2024-03-04T06:20:10.341679+0000","last_peered":"2024-03-04T06:20:10.341679+0000","last_clean":"2024-03-04T06:20:10.341679+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:20:10.341679+0000","last_undegraded":"2024-03-04T06:20:10.341679+0000","last_fullsized":"2024-03-04T06:20:10.341679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210797,"num_objects":70,"num_object_clones":54,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":324,"num_read_kb":236247,"num_write":293,"num_write_kb":78994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1848,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:04.461436+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:20:04.461436+0000","last_peered":"2024-03-04T06:20:04.461436+0000","last_clean":"2024-03-04T06:20:04.461436+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:20:04.461436+0000","last_undegraded":"2024-03-04T06:20:04.461436+0000","last_fullsized":"2024-03-04T06:20:04.461436+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"733'985","reported_seq":2708,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:06.386931+0000","last_change":"2024-03-04T06:20:04.485556+0000","last_active":"2024-03-04T06:20:06.386931+0000","last_peered":"2024-03-04T06:20:06.386931+0000","last_clean":"2024-03-04T06:20:06.386931+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:20:06.386931+0000","last_undegraded":"2024-03-04T06:20:06.386931+0000","last_fullsized":"2024-03-04T06:20:06.386931+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":985,"log_dups_size":0,"ondisk_log_size":985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12404754,"num_objects":82,"num_object_clones":68,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":395,"num_read_kb":280818,"num_write":385,"num_write_kb":107739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1891,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.327945+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:20:10.327945+0000","last_peered":"2024-03-04T06:20:10.327945+0000","last_clean":"2024-03-04T06:20:10.327945+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:20:10.327945+0000","last_undegraded":"2024-03-04T06:20:10.327945+0000","last_fullsized":"2024-03-04T06:20:10.327945+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"737'962","reported_seq":2782,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.329477+0000","last_change":"2024-03-04T06:20:09.939940+0000","last_active":"2024-03-04T06:20:10.329477+0000","last_peered":"2024-03-04T06:20:10.329477+0000","last_clean":"2024-03-04T06:20:10.329477+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:20:10.329477+0000","last_undegraded":"2024-03-04T06:20:10.329477+0000","last_fullsized":"2024-03-04T06:20:10.329477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":80,"num_object_clones":70,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":500,"num_read_kb":422768,"num_write":356,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1822,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.393602+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:20:02.393602+0000","last_peered":"2024-03-04T06:20:02.393602+0000","last_clean":"2024-03-04T06:20:02.393602+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:20:02.393602+0000","last_undegraded":"2024-03-04T06:20:02.393602+0000","last_fullsized":"2024-03-04T06:20:02.393602+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":113911830,"num_objects":975,"num_object_clones":780,"num_object_copies":2925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":159,"num_read":5242,"num_read_kb":4143523,"num_write":4402,"num_write_kb":1202291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11450,"ondisk_log_size":11450,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706404,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":200992,"kb_used_data":143640,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94170848,"statfs":{"total":96636764160,"available":96430948352,"internally_reserved":0,"allocated":147087360,"data_stored":169937540,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706404,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":148464,"kb_used_data":99432,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94223376,"statfs":{"total":96636764160,"available":96484737024,"internally_reserved":0,"allocated":101818368,"data_stored":115051775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706408,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":155440,"kb_used_data":106408,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94216400,"statfs":{"total":96636764160,"available":96477593600,"internally_reserved":0,"allocated":108961792,"data_stored":127466827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706408,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":163776,"kb_used_data":114744,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94208064,"statfs":{"total":96636764160,"available":96469057536,"internally_reserved":0,"allocated":117497856,"data_stored":138603892,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:20:11.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:20:11.858 INFO:tasks.rados.rados.0.smithi064.stdout:2912: finishing write tid 1 to smithi06431743-16 2024-03-04T06:20:11.858 INFO:tasks.rados.rados.0.smithi064.stdout:2910: finishing write tid 2 to smithi06431743-2 2024-03-04T06:20:11.858 INFO:tasks.rados.rados.0.smithi064.stdout:2910: finishing write tid 3 to smithi06431743-2 2024-03-04T06:20:11.858 INFO:tasks.rados.rados.0.smithi064.stdout:2910: finishing write tid 4 to smithi06431743-2 2024-03-04T06:20:11.858 INFO:tasks.rados.rados.0.smithi064.stdout:2910: oid 2 updating version 0 to 652 2024-03-04T06:20:11.858 INFO:tasks.rados.rados.0.smithi064.stdout:2910: oid 2 version 652 is already newer than 652 2024-03-04T06:20:11.858 INFO:tasks.rados.rados.0.smithi064.stdout:2910: oid 2 version 652 is already newer than 650 2024-03-04T06:20:11.858 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 652 (ObjNum 1323 snap 367 seq_num 1323) dirty exists 2024-03-04T06:20:11.858 INFO:tasks.rados.rados.0.smithi064.stdout:2910: left oid 2 (ObjNum 1323 snap 367 seq_num 1323) 2024-03-04T06:20:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:2912: finishing write tid 2 to smithi06431743-16 2024-03-04T06:20:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:2912: finishing write tid 3 to smithi06431743-16 2024-03-04T06:20:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:2912: finishing write tid 4 to smithi06431743-16 2024-03-04T06:20:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:2912: oid 16 updating version 0 to 755 2024-03-04T06:20:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:2912: oid 16 version 755 is already newer than 754 2024-03-04T06:20:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:2912: oid 16 updating version 755 to 756 2024-03-04T06:20:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 756 (ObjNum 1324 snap 367 seq_num 1324) dirty exists 2024-03-04T06:20:11.859 INFO:tasks.rados.rados.0.smithi064.stdout:2912: left oid 16 (ObjNum 1324 snap 367 seq_num 1324) 2024-03-04T06:20:11.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:11.881+0000 7f1b9db9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:11.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:11.888+0000 7f1b9db9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:12.077 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:20:12.077 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:20:12.089 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1104,"stamp":"2024-03-04T06:20:11.381136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114502198,"num_objects":977,"num_object_clones":780,"num_object_copies":2931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":159,"num_read":5345,"num_read_kb":4143614,"num_write":4539,"num_write_kb":1204185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11547,"ondisk_log_size":11547,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":668672,"kb_used_data":464224,"kb_used_omap":0,"kb_used_meta":204416,"kb_avail":376818688,"statfs":{"total":386547056640,"available":385862336512,"internally_reserved":0,"allocated":475365376,"data_stored":551060034,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":209321984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":958362,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":8,"num_read_kb":9526,"num_write":6,"num_write_kb":1947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.004136"},"pg_stats":[{"pgid":"3.f","version":"738'683","reported_seq":2544,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.973490+0000","last_change":"2024-03-04T06:20:09.990769+0000","last_active":"2024-03-04T06:20:09.973490+0000","last_peered":"2024-03-04T06:20:09.973490+0000","last_clean":"2024-03-04T06:20:09.973490+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:20:09.973490+0000","last_undegraded":"2024-03-04T06:20:09.973490+0000","last_fullsized":"2024-03-04T06:20:09.973490+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135182,"num_objects":64,"num_object_clones":51,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":302,"num_read_kb":245922,"num_write":268,"num_write_kb":71362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"739'578","reported_seq":2501,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.348702+0000","last_change":"2024-03-04T06:20:09.939440+0000","last_active":"2024-03-04T06:20:10.348702+0000","last_peered":"2024-03-04T06:20:10.348702+0000","last_clean":"2024-03-04T06:20:10.348702+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:20:10.348702+0000","last_undegraded":"2024-03-04T06:20:10.348702+0000","last_fullsized":"2024-03-04T06:20:10.348702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":578,"log_dups_size":0,"ondisk_log_size":578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5950305,"num_objects":50,"num_object_clones":36,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":183,"num_read_kb":130655,"num_write":205,"num_write_kb":54550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"731'734","reported_seq":2635,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.436835+0000","last_change":"2024-03-04T06:20:02.436835+0000","last_active":"2024-03-04T06:20:02.436835+0000","last_peered":"2024-03-04T06:20:02.436835+0000","last_clean":"2024-03-04T06:20:02.436835+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:20:02.436835+0000","last_undegraded":"2024-03-04T06:20:02.436835+0000","last_fullsized":"2024-03-04T06:20:02.436835+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11127455,"num_objects":71,"num_object_clones":56,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":426,"num_read_kb":348114,"num_write":314,"num_write_kb":85059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"737'648","reported_seq":2497,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.962884+0000","last_change":"2024-03-04T06:20:09.971653+0000","last_active":"2024-03-04T06:20:09.962884+0000","last_peered":"2024-03-04T06:20:09.962884+0000","last_clean":"2024-03-04T06:20:09.962884+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:20:09.962884+0000","last_undegraded":"2024-03-04T06:20:09.962884+0000","last_fullsized":"2024-03-04T06:20:09.962884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10438038,"num_objects":48,"num_object_clones":38,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":259,"num_read_kb":202834,"num_write":234,"num_write_kb":66141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"728'265","reported_seq":2316,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:04.466926+0000","last_change":"2024-03-04T06:20:04.466926+0000","last_active":"2024-03-04T06:20:04.466926+0000","last_peered":"2024-03-04T06:20:04.466926+0000","last_clean":"2024-03-04T06:20:04.466926+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:20:04.466926+0000","last_undegraded":"2024-03-04T06:20:04.466926+0000","last_fullsized":"2024-03-04T06:20:04.466926+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":90,"num_read_kb":65483,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"737'793","reported_seq":2593,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.969637+0000","last_change":"2024-03-04T06:20:09.978981+0000","last_active":"2024-03-04T06:20:09.969637+0000","last_peered":"2024-03-04T06:20:09.969637+0000","last_clean":"2024-03-04T06:20:09.969637+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:20:09.969637+0000","last_undegraded":"2024-03-04T06:20:09.969637+0000","last_fullsized":"2024-03-04T06:20:09.969637+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":793,"log_dups_size":0,"ondisk_log_size":793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260855,"num_objects":61,"num_object_clones":52,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":347,"num_read_kb":278960,"num_write":300,"num_write_kb":87675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2169,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.971458+0000","last_change":"2024-03-04T06:20:09.978390+0000","last_active":"2024-03-04T06:20:09.971458+0000","last_peered":"2024-03-04T06:20:09.971458+0000","last_clean":"2024-03-04T06:20:09.971458+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:20:09.971458+0000","last_undegraded":"2024-03-04T06:20:09.971458+0000","last_fullsized":"2024-03-04T06:20:09.971458+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"729'414","reported_seq":2391,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.381941+0000","last_change":"2024-03-04T06:20:02.381941+0000","last_active":"2024-03-04T06:20:02.381941+0000","last_peered":"2024-03-04T06:20:02.381941+0000","last_clean":"2024-03-04T06:20:02.381941+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:20:02.381941+0000","last_undegraded":"2024-03-04T06:20:02.381941+0000","last_fullsized":"2024-03-04T06:20:02.381941+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6672940,"num_objects":32,"num_object_clones":23,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":214,"num_read_kb":195370,"num_write":145,"num_write_kb":38810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1984,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:06.390293+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:20:06.390293+0000","last_peered":"2024-03-04T06:20:06.390293+0000","last_clean":"2024-03-04T06:20:06.390293+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:20:06.390293+0000","last_undegraded":"2024-03-04T06:20:06.390293+0000","last_fullsized":"2024-03-04T06:20:06.390293+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1821,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.392801+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:20:02.392801+0000","last_peered":"2024-03-04T06:20:02.392801+0000","last_clean":"2024-03-04T06:20:02.392801+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:20:02.392801+0000","last_undegraded":"2024-03-04T06:20:02.392801+0000","last_fullsized":"2024-03-04T06:20:02.392801+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"729'705","reported_seq":2565,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.399671+0000","last_change":"2024-03-04T06:20:02.399671+0000","last_active":"2024-03-04T06:20:02.399671+0000","last_peered":"2024-03-04T06:20:02.399671+0000","last_clean":"2024-03-04T06:20:02.399671+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:20:02.399671+0000","last_undegraded":"2024-03-04T06:20:02.399671+0000","last_fullsized":"2024-03-04T06:20:02.399671+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6286412,"num_objects":62,"num_object_clones":48,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":340,"num_read_kb":275379,"num_write":276,"num_write_kb":74562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1824,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.974283+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:20:09.974283+0000","last_peered":"2024-03-04T06:20:09.974283+0000","last_clean":"2024-03-04T06:20:09.974283+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:20:09.974283+0000","last_undegraded":"2024-03-04T06:20:09.974283+0000","last_fullsized":"2024-03-04T06:20:09.974283+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"733'452","reported_seq":2462,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:06.397845+0000","last_change":"2024-03-04T06:20:04.480635+0000","last_active":"2024-03-04T06:20:06.397845+0000","last_peered":"2024-03-04T06:20:06.397845+0000","last_clean":"2024-03-04T06:20:06.397845+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:20:06.397845+0000","last_undegraded":"2024-03-04T06:20:06.397845+0000","last_fullsized":"2024-03-04T06:20:06.397845+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7446095,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":261,"num_read_kb":212085,"num_write":166,"num_write_kb":47091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1821,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.369030+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:20:02.369030+0000","last_peered":"2024-03-04T06:20:02.369030+0000","last_clean":"2024-03-04T06:20:02.369030+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:20:02.369030+0000","last_undegraded":"2024-03-04T06:20:02.369030+0000","last_fullsized":"2024-03-04T06:20:02.369030+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"733'1294","reported_seq":2977,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:04.488959+0000","last_change":"2024-03-04T06:20:04.488959+0000","last_active":"2024-03-04T06:20:04.488959+0000","last_peered":"2024-03-04T06:20:04.488959+0000","last_clean":"2024-03-04T06:20:04.488959+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:20:04.488959+0000","last_undegraded":"2024-03-04T06:20:04.488959+0000","last_fullsized":"2024-03-04T06:20:04.488959+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1294,"log_dups_size":0,"ondisk_log_size":1294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11413141,"num_objects":115,"num_object_clones":91,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":677,"num_read_kb":533744,"num_write":535,"num_write_kb":142941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1824,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.965851+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:20:09.965851+0000","last_peered":"2024-03-04T06:20:09.965851+0000","last_clean":"2024-03-04T06:20:09.965851+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:20:09.965851+0000","last_undegraded":"2024-03-04T06:20:09.965851+0000","last_fullsized":"2024-03-04T06:20:09.965851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"737'1480","reported_seq":2970,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.327238+0000","last_change":"2024-03-04T06:20:09.938152+0000","last_active":"2024-03-04T06:20:10.327238+0000","last_peered":"2024-03-04T06:20:10.327238+0000","last_clean":"2024-03-04T06:20:10.327238+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:20:10.327238+0000","last_undegraded":"2024-03-04T06:20:10.327238+0000","last_fullsized":"2024-03-04T06:20:10.327238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18858939,"num_objects":121,"num_object_clones":98,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":532,"num_read_kb":386027,"num_write":561,"num_write_kb":150818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"729'705","reported_seq":2583,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.383168+0000","last_change":"2024-03-04T06:20:02.399657+0000","last_active":"2024-03-04T06:20:02.383168+0000","last_peered":"2024-03-04T06:20:02.383168+0000","last_clean":"2024-03-04T06:20:02.383168+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:20:02.383168+0000","last_undegraded":"2024-03-04T06:20:02.383168+0000","last_fullsized":"2024-03-04T06:20:02.383168+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5765395,"num_objects":57,"num_object_clones":45,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":392,"num_read_kb":329117,"num_write":267,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1891,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.338755+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:20:10.338755+0000","last_peered":"2024-03-04T06:20:10.338755+0000","last_clean":"2024-03-04T06:20:10.338755+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:20:10.338755+0000","last_undegraded":"2024-03-04T06:20:10.338755+0000","last_fullsized":"2024-03-04T06:20:10.338755+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"737'752","reported_seq":2639,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.341679+0000","last_change":"2024-03-04T06:20:09.954608+0000","last_active":"2024-03-04T06:20:10.341679+0000","last_peered":"2024-03-04T06:20:10.341679+0000","last_clean":"2024-03-04T06:20:10.341679+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:20:10.341679+0000","last_undegraded":"2024-03-04T06:20:10.341679+0000","last_fullsized":"2024-03-04T06:20:10.341679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210797,"num_objects":70,"num_object_clones":54,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":324,"num_read_kb":236247,"num_write":293,"num_write_kb":78994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1848,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:04.461436+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:20:04.461436+0000","last_peered":"2024-03-04T06:20:04.461436+0000","last_clean":"2024-03-04T06:20:04.461436+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:20:04.461436+0000","last_undegraded":"2024-03-04T06:20:04.461436+0000","last_fullsized":"2024-03-04T06:20:04.461436+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"733'985","reported_seq":2708,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:06.386931+0000","last_change":"2024-03-04T06:20:04.485556+0000","last_active":"2024-03-04T06:20:06.386931+0000","last_peered":"2024-03-04T06:20:06.386931+0000","last_clean":"2024-03-04T06:20:06.386931+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:20:06.386931+0000","last_undegraded":"2024-03-04T06:20:06.386931+0000","last_fullsized":"2024-03-04T06:20:06.386931+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":985,"log_dups_size":0,"ondisk_log_size":985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12404754,"num_objects":82,"num_object_clones":68,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":395,"num_read_kb":280818,"num_write":385,"num_write_kb":107739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1891,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.327945+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:20:10.327945+0000","last_peered":"2024-03-04T06:20:10.327945+0000","last_clean":"2024-03-04T06:20:10.327945+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:20:10.327945+0000","last_undegraded":"2024-03-04T06:20:10.327945+0000","last_fullsized":"2024-03-04T06:20:10.327945+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"737'962","reported_seq":2782,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.329477+0000","last_change":"2024-03-04T06:20:09.939940+0000","last_active":"2024-03-04T06:20:10.329477+0000","last_peered":"2024-03-04T06:20:10.329477+0000","last_clean":"2024-03-04T06:20:10.329477+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:20:10.329477+0000","last_undegraded":"2024-03-04T06:20:10.329477+0000","last_fullsized":"2024-03-04T06:20:10.329477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":80,"num_object_clones":70,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":500,"num_read_kb":422768,"num_write":356,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1822,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-04T06:20:02.393602+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:20:02.393602+0000","last_peered":"2024-03-04T06:20:02.393602+0000","last_clean":"2024-03-04T06:20:02.393602+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:20:02.393602+0000","last_undegraded":"2024-03-04T06:20:02.393602+0000","last_fullsized":"2024-03-04T06:20:02.393602+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":113911830,"num_objects":975,"num_object_clones":780,"num_object_copies":2925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":159,"num_read":5242,"num_read_kb":4143523,"num_write":4402,"num_write_kb":1202291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11450,"ondisk_log_size":11450,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706404,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":200992,"kb_used_data":143640,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94170848,"statfs":{"total":96636764160,"available":96430948352,"internally_reserved":0,"allocated":147087360,"data_stored":169937540,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706404,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":148464,"kb_used_data":99432,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94223376,"statfs":{"total":96636764160,"available":96484737024,"internally_reserved":0,"allocated":101818368,"data_stored":115051775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706408,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":155440,"kb_used_data":106408,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94216400,"statfs":{"total":96636764160,"available":96477593600,"internally_reserved":0,"allocated":108961792,"data_stored":127466827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706408,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":163776,"kb_used_data":114744,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94208064,"statfs":{"total":96636764160,"available":96469057536,"internally_reserved":0,"allocated":117497856,"data_stored":138603892,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:20:12.090 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:20:12.090 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:20:12.091 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:20:12.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:12.194+0000 7f4d130ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:12.201 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:12.200+0000 7f4d130ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:12.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:12.325 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:12.323+0000 7f046fa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:12.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:12.328+0000 7f046fa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:12.383 INFO:tasks.rados.rados.0.smithi064.stdout:2910: done (4 left) 2024-03-04T06:20:12.383 INFO:tasks.rados.rados.0.smithi064.stdout:2911: done (3 left) 2024-03-04T06:20:12.383 INFO:tasks.rados.rados.0.smithi064.stdout:2912: done (2 left) 2024-03-04T06:20:12.383 INFO:tasks.rados.rados.0.smithi064.stdout:2913: done (1 left) 2024-03-04T06:20:12.383 INFO:tasks.rados.rados.0.smithi064.stdout:2914: done (0 left) 2024-03-04T06:20:12.383 INFO:tasks.rados.rados.0.smithi064.stdout:2915: delete oid 17 current snap is 368 2024-03-04T06:20:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:20:12.413 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:20:12.428 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1105,"stamp":"2024-03-04T06:20:12.381658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108636390,"num_objects":981,"num_object_clones":784,"num_object_copies":2943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":160,"num_read":5361,"num_read_kb":4155678,"num_write":4552,"num_write_kb":1207211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11593,"ondisk_log_size":11593,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":676984,"kb_used_data":472536,"kb_used_omap":0,"kb_used_meta":204416,"kb_avail":376810376,"statfs":{"total":386547056640,"available":385853825024,"internally_reserved":0,"allocated":483876864,"data_stored":561240813,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":209321984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4907446,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":21590,"num_write":19,"num_write_kb":4973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.081822"},"pg_stats":[{"pgid":"3.f","version":"738'683","reported_seq":2544,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.973490+0000","last_change":"2024-03-04T06:20:09.990769+0000","last_active":"2024-03-04T06:20:09.973490+0000","last_peered":"2024-03-04T06:20:09.973490+0000","last_clean":"2024-03-04T06:20:09.973490+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:20:09.973490+0000","last_undegraded":"2024-03-04T06:20:09.973490+0000","last_fullsized":"2024-03-04T06:20:09.973490+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135182,"num_objects":64,"num_object_clones":51,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":302,"num_read_kb":245922,"num_write":268,"num_write_kb":71362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"739'578","reported_seq":2501,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.348702+0000","last_change":"2024-03-04T06:20:09.939440+0000","last_active":"2024-03-04T06:20:10.348702+0000","last_peered":"2024-03-04T06:20:10.348702+0000","last_clean":"2024-03-04T06:20:10.348702+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:20:10.348702+0000","last_undegraded":"2024-03-04T06:20:10.348702+0000","last_fullsized":"2024-03-04T06:20:10.348702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":578,"log_dups_size":0,"ondisk_log_size":578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5950305,"num_objects":50,"num_object_clones":36,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":183,"num_read_kb":130655,"num_write":205,"num_write_kb":54550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"739'748","reported_seq":2659,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.426963+0000","last_change":"2024-03-04T06:20:09.329889+0000","last_active":"2024-03-04T06:20:10.426963+0000","last_peered":"2024-03-04T06:20:10.426963+0000","last_clean":"2024-03-04T06:20:10.426963+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:20:10.426963+0000","last_undegraded":"2024-03-04T06:20:10.426963+0000","last_fullsized":"2024-03-04T06:20:10.426963+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11043453,"num_objects":72,"num_object_clones":57,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":428,"num_read_kb":348115,"num_write":318,"num_write_kb":86166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"737'648","reported_seq":2497,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.962884+0000","last_change":"2024-03-04T06:20:09.971653+0000","last_active":"2024-03-04T06:20:09.962884+0000","last_peered":"2024-03-04T06:20:09.962884+0000","last_clean":"2024-03-04T06:20:09.962884+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:20:09.962884+0000","last_undegraded":"2024-03-04T06:20:09.962884+0000","last_fullsized":"2024-03-04T06:20:09.962884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10438038,"num_objects":48,"num_object_clones":38,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":259,"num_read_kb":202834,"num_write":234,"num_write_kb":66141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"737'267","reported_seq":2328,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.941885+0000","last_change":"2024-03-04T06:20:09.945549+0000","last_active":"2024-03-04T06:20:09.941885+0000","last_peered":"2024-03-04T06:20:09.941885+0000","last_clean":"2024-03-04T06:20:09.941885+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:20:09.941885+0000","last_undegraded":"2024-03-04T06:20:09.941885+0000","last_fullsized":"2024-03-04T06:20:09.941885+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941522,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":90,"num_read_kb":65483,"num_write":97,"num_write_kb":26694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"737'793","reported_seq":2593,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.969637+0000","last_change":"2024-03-04T06:20:09.978981+0000","last_active":"2024-03-04T06:20:09.969637+0000","last_peered":"2024-03-04T06:20:09.969637+0000","last_clean":"2024-03-04T06:20:09.969637+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:20:09.969637+0000","last_undegraded":"2024-03-04T06:20:09.969637+0000","last_fullsized":"2024-03-04T06:20:09.969637+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":793,"log_dups_size":0,"ondisk_log_size":793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260855,"num_objects":61,"num_object_clones":52,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":347,"num_read_kb":278960,"num_write":300,"num_write_kb":87675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2169,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.971458+0000","last_change":"2024-03-04T06:20:09.978390+0000","last_active":"2024-03-04T06:20:09.971458+0000","last_peered":"2024-03-04T06:20:09.971458+0000","last_clean":"2024-03-04T06:20:09.971458+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:20:09.971458+0000","last_undegraded":"2024-03-04T06:20:09.971458+0000","last_fullsized":"2024-03-04T06:20:09.971458+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"739'420","reported_seq":2415,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.414572+0000","last_change":"2024-03-04T06:20:09.329412+0000","last_active":"2024-03-04T06:20:10.414572+0000","last_peered":"2024-03-04T06:20:10.414572+0000","last_clean":"2024-03-04T06:20:10.414572+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:20:10.414572+0000","last_undegraded":"2024-03-04T06:20:10.414572+0000","last_fullsized":"2024-03-04T06:20:10.414572+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210797,"num_objects":33,"num_object_clones":24,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":215,"num_read_kb":195371,"num_write":149,"num_write_kb":39866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1991,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.927193+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:20:09.927193+0000","last_peered":"2024-03-04T06:20:09.927193+0000","last_clean":"2024-03-04T06:20:09.927193+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:20:09.927193+0000","last_undegraded":"2024-03-04T06:20:09.927193+0000","last_fullsized":"2024-03-04T06:20:09.927193+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1839,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.334449+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:20:10.334449+0000","last_peered":"2024-03-04T06:20:10.334449+0000","last_clean":"2024-03-04T06:20:10.334449+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:20:10.334449+0000","last_undegraded":"2024-03-04T06:20:10.334449+0000","last_fullsized":"2024-03-04T06:20:10.334449+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"738'715","reported_seq":2592,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.438157+0000","last_change":"2024-03-04T06:20:09.346849+0000","last_active":"2024-03-04T06:20:10.438157+0000","last_peered":"2024-03-04T06:20:10.438157+0000","last_clean":"2024-03-04T06:20:10.438157+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:20:10.438157+0000","last_undegraded":"2024-03-04T06:20:10.438157+0000","last_fullsized":"2024-03-04T06:20:10.438157+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":715,"log_dups_size":0,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4723394,"num_objects":63,"num_object_clones":49,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":347,"num_read_kb":281382,"num_write":280,"num_write_kb":75425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1824,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.974283+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:20:09.974283+0000","last_peered":"2024-03-04T06:20:09.974283+0000","last_clean":"2024-03-04T06:20:09.974283+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:20:09.974283+0000","last_undegraded":"2024-03-04T06:20:09.974283+0000","last_fullsized":"2024-03-04T06:20:09.974283+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"737'454","reported_seq":2471,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.920485+0000","last_change":"2024-03-04T06:20:09.941613+0000","last_active":"2024-03-04T06:20:09.920485+0000","last_peered":"2024-03-04T06:20:09.920485+0000","last_clean":"2024-03-04T06:20:09.920485+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:20:09.920485+0000","last_undegraded":"2024-03-04T06:20:09.920485+0000","last_fullsized":"2024-03-04T06:20:09.920485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7446095,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":261,"num_read_kb":212085,"num_write":166,"num_write_kb":47091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1839,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.312835+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:20:10.312835+0000","last_peered":"2024-03-04T06:20:10.312835+0000","last_clean":"2024-03-04T06:20:10.312835+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:20:10.312835+0000","last_undegraded":"2024-03-04T06:20:10.312835+0000","last_fullsized":"2024-03-04T06:20:10.312835+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"737'1298","reported_seq":2989,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.931123+0000","last_change":"2024-03-04T06:20:09.944388+0000","last_active":"2024-03-04T06:20:09.931123+0000","last_peered":"2024-03-04T06:20:09.931123+0000","last_clean":"2024-03-04T06:20:09.931123+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:20:09.931123+0000","last_undegraded":"2024-03-04T06:20:09.931123+0000","last_fullsized":"2024-03-04T06:20:09.931123+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1298,"log_dups_size":0,"ondisk_log_size":1298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11413141,"num_objects":115,"num_object_clones":91,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":677,"num_read_kb":533744,"num_write":535,"num_write_kb":142941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1824,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.965851+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:20:09.965851+0000","last_peered":"2024-03-04T06:20:09.965851+0000","last_clean":"2024-03-04T06:20:09.965851+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:20:09.965851+0000","last_undegraded":"2024-03-04T06:20:09.965851+0000","last_fullsized":"2024-03-04T06:20:09.965851+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"737'1480","reported_seq":2970,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.327238+0000","last_change":"2024-03-04T06:20:09.938152+0000","last_active":"2024-03-04T06:20:10.327238+0000","last_peered":"2024-03-04T06:20:10.327238+0000","last_clean":"2024-03-04T06:20:10.327238+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:20:10.327238+0000","last_undegraded":"2024-03-04T06:20:10.327238+0000","last_fullsized":"2024-03-04T06:20:10.327238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18858939,"num_objects":121,"num_object_clones":98,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":532,"num_read_kb":386027,"num_write":561,"num_write_kb":150818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"736'709","reported_seq":2605,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.331338+0000","last_change":"2024-03-04T06:20:09.336464+0000","last_active":"2024-03-04T06:20:10.331338+0000","last_peered":"2024-03-04T06:20:10.331338+0000","last_clean":"2024-03-04T06:20:10.331338+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:20:10.331338+0000","last_undegraded":"2024-03-04T06:20:10.331338+0000","last_fullsized":"2024-03-04T06:20:10.331338+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3008750,"num_objects":58,"num_object_clones":46,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":392,"num_read_kb":329117,"num_write":268,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1891,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.338755+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:20:10.338755+0000","last_peered":"2024-03-04T06:20:10.338755+0000","last_clean":"2024-03-04T06:20:10.338755+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:20:10.338755+0000","last_undegraded":"2024-03-04T06:20:10.338755+0000","last_fullsized":"2024-03-04T06:20:10.338755+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"737'752","reported_seq":2639,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.341679+0000","last_change":"2024-03-04T06:20:09.954608+0000","last_active":"2024-03-04T06:20:10.341679+0000","last_peered":"2024-03-04T06:20:10.341679+0000","last_clean":"2024-03-04T06:20:10.341679+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:20:10.341679+0000","last_undegraded":"2024-03-04T06:20:10.341679+0000","last_fullsized":"2024-03-04T06:20:10.341679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5210797,"num_objects":70,"num_object_clones":54,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":324,"num_read_kb":236247,"num_write":293,"num_write_kb":78994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1858,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.934239+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:20:09.934239+0000","last_peered":"2024-03-04T06:20:09.934239+0000","last_clean":"2024-03-04T06:20:09.934239+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:20:09.934239+0000","last_undegraded":"2024-03-04T06:20:09.934239+0000","last_fullsized":"2024-03-04T06:20:09.934239+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"737'989","reported_seq":2721,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-04T06:20:09.935321+0000","last_change":"2024-03-04T06:20:09.940999+0000","last_active":"2024-03-04T06:20:09.935321+0000","last_peered":"2024-03-04T06:20:09.935321+0000","last_clean":"2024-03-04T06:20:09.935321+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:20:09.935321+0000","last_undegraded":"2024-03-04T06:20:09.935321+0000","last_fullsized":"2024-03-04T06:20:09.935321+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":989,"log_dups_size":0,"ondisk_log_size":989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12404754,"num_objects":82,"num_object_clones":68,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":401,"num_read_kb":286877,"num_write":385,"num_write_kb":107739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1891,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.327945+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:20:10.327945+0000","last_peered":"2024-03-04T06:20:10.327945+0000","last_clean":"2024-03-04T06:20:10.327945+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:20:10.327945+0000","last_undegraded":"2024-03-04T06:20:10.327945+0000","last_fullsized":"2024-03-04T06:20:10.327945+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"737'962","reported_seq":2782,"reported_epoch":739,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.329477+0000","last_change":"2024-03-04T06:20:09.939940+0000","last_active":"2024-03-04T06:20:10.329477+0000","last_peered":"2024-03-04T06:20:10.329477+0000","last_clean":"2024-03-04T06:20:10.329477+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:20:10.329477+0000","last_undegraded":"2024-03-04T06:20:10.329477+0000","last_fullsized":"2024-03-04T06:20:10.329477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":80,"num_object_clones":70,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":500,"num_read_kb":422768,"num_write":356,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1839,"reported_epoch":740,"state":"active+clean","last_fresh":"2024-03-04T06:20:10.325968+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:20:10.325968+0000","last_peered":"2024-03-04T06:20:10.325968+0000","last_clean":"2024-03-04T06:20:10.325968+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:20:10.325968+0000","last_undegraded":"2024-03-04T06:20:10.325968+0000","last_fullsized":"2024-03-04T06:20:10.325968+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":108046022,"num_objects":979,"num_object_clones":784,"num_object_copies":2937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":160,"num_read":5258,"num_read_kb":4155587,"num_write":4415,"num_write_kb":1205317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11496,"ondisk_log_size":11496,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706409,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":205136,"kb_used_data":147784,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94166704,"statfs":{"total":96636764160,"available":96426704896,"internally_reserved":0,"allocated":151330816,"data_stored":175022290,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706409,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":152632,"kb_used_data":103600,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94219208,"statfs":{"total":96636764160,"available":96480468992,"internally_reserved":0,"allocated":106086400,"data_stored":120147804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706408,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":155440,"kb_used_data":106408,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94216400,"statfs":{"total":96636764160,"available":96477593600,"internally_reserved":0,"allocated":108961792,"data_stored":127466827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706408,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":163776,"kb_used_data":114744,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94208064,"statfs":{"total":96636764160,"available":96469057536,"internally_reserved":0,"allocated":117497856,"data_stored":138603892,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:20:12.434 INFO:tasks.rados.rados.0.smithi064.stdout:2915: done (0 left) 2024-03-04T06:20:12.434 INFO:tasks.rados.rados.0.smithi064.stdout:2916: write oid 10 current snap is 368 2024-03-04T06:20:12.434 INFO:tasks.rados.rados.0.smithi064.stdout:2916: seq_num 1326 ranges {686082=425596,1577829=433132} 2024-03-04T06:20:12.440 INFO:tasks.rados.rados.0.smithi064.stdout:2916: writing smithi06431743-10 from 686082 to 1111678 tid 1 2024-03-04T06:20:12.445 INFO:tasks.rados.rados.0.smithi064.stdout:2916: writing smithi06431743-10 from 1577829 to 2010961 tid 2 2024-03-04T06:20:12.445 INFO:tasks.rados.rados.0.smithi064.stdout:2917: write (excl) oid 19 current snap is 368 2024-03-04T06:20:12.445 INFO:tasks.rados.rados.0.smithi064.stdout:2917: seq_num 1327 ranges {702889=500845,1636311=576790} 2024-03-04T06:20:12.454 INFO:tasks.rados.rados.0.smithi064.stdout:2917: writing smithi06431743-19 from 702889 to 1203734 tid 1 2024-03-04T06:20:12.461 INFO:tasks.rados.rados.0.smithi064.stdout:2917: writing smithi06431743-19 from 1636311 to 2213101 tid 2 2024-03-04T06:20:12.461 INFO:tasks.rados.rados.0.smithi064.stdout:2918: delete oid 21 current snap is 368 2024-03-04T06:20:12.467 INFO:tasks.rados.rados.0.smithi064.stdout:2916: finishing write tid 1 to smithi06431743-10 2024-03-04T06:20:12.499 INFO:tasks.rados.rados.0.smithi064.stdout:2916: finishing write tid 2 to smithi06431743-10 2024-03-04T06:20:12.502 INFO:tasks.rados.rados.0.smithi064.stdout:2918: done (2 left) 2024-03-04T06:20:12.502 INFO:tasks.rados.rados.0.smithi064.stdout:2919: delete oid 44 current snap is 368 2024-03-04T06:20:12.509 INFO:tasks.rados.rados.0.smithi064.stdout:2916: finishing write tid 3 to smithi06431743-10 2024-03-04T06:20:12.510 INFO:tasks.rados.rados.0.smithi064.stdout:2916: finishing write tid 4 to smithi06431743-10 2024-03-04T06:20:12.510 INFO:tasks.rados.rados.0.smithi064.stdout:2916: oid 10 updating version 0 to 795 2024-03-04T06:20:12.510 INFO:tasks.rados.rados.0.smithi064.stdout:2916: oid 10 updating version 795 to 797 2024-03-04T06:20:12.510 INFO:tasks.rados.rados.0.smithi064.stdout:2916: oid 10 version 797 is already newer than 796 2024-03-04T06:20:12.510 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 797 (ObjNum 1326 snap 368 seq_num 1326) dirty exists 2024-03-04T06:20:12.510 INFO:tasks.rados.rados.0.smithi064.stdout:2916: left oid 10 (ObjNum 1326 snap 368 seq_num 1326) 2024-03-04T06:20:12.515 INFO:tasks.rados.rados.0.smithi064.stdout:2916: done (2 left) 2024-03-04T06:20:12.515 INFO:tasks.rados.rados.0.smithi064.stdout:2919: done (1 left) 2024-03-04T06:20:12.515 INFO:tasks.rados.rados.0.smithi064.stdout:2920: write (excl) oid 39 current snap is 368 2024-03-04T06:20:12.515 INFO:tasks.rados.rados.0.smithi064.stdout:2920: seq_num 1330 ranges {753310=726592,2095404=257906} 2024-03-04T06:20:12.523 INFO:tasks.rados.rados.0.smithi064.stdout:2920: writing smithi06431743-39 from 753310 to 1479902 tid 1 2024-03-04T06:20:12.528 INFO:tasks.rados.rados.0.smithi064.stdout:2920: writing smithi06431743-39 from 2095404 to 2353310 tid 2 2024-03-04T06:20:12.528 INFO:tasks.rados.rados.0.smithi064.stdout:2921: read oid 14 snap 365 2024-03-04T06:20:12.528 INFO:tasks.rados.rados.0.smithi064.stdout:2921: expect (ObjNum 1226 snap 330 seq_num 1226) 2024-03-04T06:20:12.528 INFO:tasks.rados.rados.0.smithi064.stdout:2922: write oid 3 current snap is 368 2024-03-04T06:20:12.528 INFO:tasks.rados.rados.0.smithi064.stdout:2922: seq_num 1331 ranges {770117=518194,1870239=499878} 2024-03-04T06:20:12.535 INFO:tasks.rados.rados.0.smithi064.stdout:2922: writing smithi06431743-3 from 770117 to 1288311 tid 1 2024-03-04T06:20:12.541 INFO:tasks.rados.rados.0.smithi064.stdout:2922: writing smithi06431743-3 from 1870239 to 2370117 tid 2 2024-03-04T06:20:12.541 INFO:tasks.rados.rados.0.smithi064.stdout:2923: read oid 40 snap -1 2024-03-04T06:20:12.541 INFO:tasks.rados.rados.0.smithi064.stdout:2923: expect deleted 2024-03-04T06:20:12.541 INFO:tasks.rados.rados.0.smithi064.stdout:2924: write (excl) oid 23 current snap is 368 2024-03-04T06:20:12.541 INFO:tasks.rados.rados.0.smithi064.stdout:2917: finishing write tid 1 to smithi06431743-19 2024-03-04T06:20:12.541 INFO:tasks.rados.rados.0.smithi064.stdout:2924: seq_num 1332 ranges {786924=593443,1928721=458203} 2024-03-04T06:20:12.550 INFO:tasks.rados.rados.0.smithi064.stdout:2924: writing smithi06431743-23 from 786924 to 1380367 tid 1 2024-03-04T06:20:12.557 INFO:tasks.rados.rados.0.smithi064.stdout:2924: writing smithi06431743-23 from 1928721 to 2386924 tid 2 2024-03-04T06:20:12.557 INFO:tasks.rados.rados.0.smithi064.stdout:2925: write oid 49 current snap is 368 2024-03-04T06:20:12.557 INFO:tasks.rados.rados.0.smithi064.stdout:2925: seq_num 1333 ranges {403731=668692,1587203=587797} 2024-03-04T06:20:12.564 INFO:tasks.rados.rados.0.smithi064.stdout:2925: writing smithi06431743-49 from 403731 to 1072423 tid 1 2024-03-04T06:20:12.572 INFO:tasks.rados.rados.0.smithi064.stdout:2925: writing smithi06431743-49 from 1587203 to 2175000 tid 2 2024-03-04T06:20:12.572 INFO:tasks.rados.rados.0.smithi064.stdout:2926: write (excl) oid 46 current snap is 368 2024-03-04T06:20:12.572 INFO:tasks.rados.rados.0.smithi064.stdout:2926: seq_num 1334 ranges {420538=743941,1929332=491206} 2024-03-04T06:20:12.580 INFO:tasks.rados.rados.0.smithi064.stdout:2926: writing smithi06431743-46 from 420538 to 1164479 tid 1 2024-03-04T06:20:12.589 INFO:tasks.rados.rados.0.smithi064.stdout:2926: writing smithi06431743-46 from 1929332 to 2420538 tid 2 2024-03-04T06:20:12.589 INFO:tasks.rados.rados.0.smithi064.stdout:2927: read oid 45 snap -1 2024-03-04T06:20:12.589 INFO:tasks.rados.rados.0.smithi064.stdout:2927: expect (ObjNum 1266 snap 344 seq_num 1266) 2024-03-04T06:20:12.589 INFO:tasks.rados.rados.0.smithi064.stdout:2928: snap_create 2024-03-04T06:20:12.589 INFO:tasks.rados.rados.0.smithi064.stdout:2917: finishing write tid 2 to smithi06431743-19 2024-03-04T06:20:12.590 INFO:tasks.rados.rados.0.smithi064.stdout:2917: finishing write tid 3 to smithi06431743-19 2024-03-04T06:20:12.590 INFO:tasks.rados.rados.0.smithi064.stdout:2917: finishing write tid 4 to smithi06431743-19 2024-03-04T06:20:12.590 INFO:tasks.rados.rados.0.smithi064.stdout:2917: oid 19 updating version 0 to 1482 2024-03-04T06:20:12.590 INFO:tasks.rados.rados.0.smithi064.stdout:2917: oid 19 updating version 1482 to 1484 2024-03-04T06:20:12.590 INFO:tasks.rados.rados.0.smithi064.stdout:2917: oid 19 version 1484 is already newer than 1484 2024-03-04T06:20:12.590 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 1484 (ObjNum 1327 snap 368 seq_num 1327) dirty exists 2024-03-04T06:20:12.590 INFO:tasks.rados.rados.0.smithi064.stdout:2917: left oid 19 (ObjNum 1327 snap 368 seq_num 1327) 2024-03-04T06:20:12.590 INFO:tasks.rados.rados.0.smithi064.stdout:2921: expect (ObjNum 1226 snap 330 seq_num 1226) 2024-03-04T06:20:12.778 INFO:tasks.rados.rados.0.smithi064.stdout:2920: finishing write tid 1 to smithi06431743-39 2024-03-04T06:20:12.778 INFO:tasks.rados.rados.0.smithi064.stdout:2922: finishing write tid 1 to smithi06431743-3 2024-03-04T06:20:12.778 INFO:tasks.rados.rados.0.smithi064.stdout:2926: finishing write tid 1 to smithi06431743-46 2024-03-04T06:20:12.778 INFO:tasks.rados.rados.0.smithi064.stdout:2920: finishing write tid 2 to smithi06431743-39 2024-03-04T06:20:12.778 INFO:tasks.rados.rados.0.smithi064.stdout:2925: finishing write tid 1 to smithi06431743-49 2024-03-04T06:20:12.778 INFO:tasks.rados.rados.0.smithi064.stdout:2922: finishing write tid 2 to smithi06431743-3 2024-03-04T06:20:12.778 INFO:tasks.rados.rados.0.smithi064.stdout:2920: finishing write tid 3 to smithi06431743-39 2024-03-04T06:20:12.778 INFO:tasks.rados.rados.0.smithi064.stdout:2926: finishing write tid 2 to smithi06431743-46 2024-03-04T06:20:12.778 INFO:tasks.rados.rados.0.smithi064.stdout:2920: finishing write tid 4 to smithi06431743-39 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2920: oid 39 updating version 0 to 752 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2920: oid 39 version 752 is already newer than 751 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2920: oid 39 version 752 is already newer than 750 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 752 (ObjNum 1330 snap 368 seq_num 1330) dirty exists 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2920: left oid 39 (ObjNum 1330 snap 368 seq_num 1330) 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2925: finishing write tid 2 to smithi06431743-49 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2926: finishing write tid 3 to smithi06431743-46 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2926: finishing write tid 4 to smithi06431743-46 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2926: oid 46 updating version 0 to 1301 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2926: oid 46 version 1301 is already newer than 1299 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2926: oid 46 version 1301 is already newer than 1301 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 1301 (ObjNum 1334 snap 368 seq_num 1334) dirty exists 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2926: left oid 46 (ObjNum 1334 snap 368 seq_num 1334) 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2922: finishing write tid 3 to smithi06431743-3 2024-03-04T06:20:12.779 INFO:tasks.rados.rados.0.smithi064.stdout:2922: finishing write tid 4 to smithi06431743-3 2024-03-04T06:20:12.780 INFO:tasks.rados.rados.0.smithi064.stdout:2922: oid 3 updating version 0 to 719 2024-03-04T06:20:12.780 INFO:tasks.rados.rados.0.smithi064.stdout:2922: oid 3 version 719 is already newer than 717 2024-03-04T06:20:12.780 INFO:tasks.rados.rados.0.smithi064.stdout:2922: oid 3 version 719 is already newer than 718 2024-03-04T06:20:12.780 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 719 (ObjNum 1331 snap 368 seq_num 1331) dirty exists 2024-03-04T06:20:12.780 INFO:tasks.rados.rados.0.smithi064.stdout:2922: left oid 3 (ObjNum 1331 snap 368 seq_num 1331) 2024-03-04T06:20:12.780 INFO:tasks.rados.rados.0.smithi064.stdout:2925: finishing write tid 3 to smithi06431743-49 2024-03-04T06:20:12.780 INFO:tasks.rados.rados.0.smithi064.stdout:2927: expect (ObjNum 1266 snap 344 seq_num 1266) 2024-03-04T06:20:12.972 INFO:tasks.rados.rados.0.smithi064.stdout:2925: finishing write tid 4 to smithi06431743-49 2024-03-04T06:20:12.972 INFO:tasks.rados.rados.0.smithi064.stdout:2925: oid 49 updating version 0 to 424 2024-03-04T06:20:12.972 INFO:tasks.rados.rados.0.smithi064.stdout:2925: oid 49 updating version 424 to 425 2024-03-04T06:20:12.972 INFO:tasks.rados.rados.0.smithi064.stdout:2925: oid 49 updating version 425 to 426 2024-03-04T06:20:12.972 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 426 (ObjNum 1333 snap 368 seq_num 1333) dirty exists 2024-03-04T06:20:12.972 INFO:tasks.rados.rados.0.smithi064.stdout:2925: left oid 49 (ObjNum 1333 snap 368 seq_num 1333) 2024-03-04T06:20:12.973 INFO:tasks.rados.rados.0.smithi064.stdout:2924: finishing write tid 1 to smithi06431743-23 2024-03-04T06:20:12.973 INFO:tasks.rados.rados.0.smithi064.stdout:2924: finishing write tid 2 to smithi06431743-23 2024-03-04T06:20:12.973 INFO:tasks.rados.rados.0.smithi064.stdout:2924: finishing write tid 3 to smithi06431743-23 2024-03-04T06:20:12.973 INFO:tasks.rados.rados.0.smithi064.stdout:2924: finishing write tid 4 to smithi06431743-23 2024-03-04T06:20:12.973 INFO:tasks.rados.rados.0.smithi064.stdout:2924: oid 23 updating version 0 to 721 2024-03-04T06:20:12.973 INFO:tasks.rados.rados.0.smithi064.stdout:2924: oid 23 updating version 721 to 723 2024-03-04T06:20:12.973 INFO:tasks.rados.rados.0.smithi064.stdout:2924: oid 23 version 723 is already newer than 723 2024-03-04T06:20:12.973 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 723 (ObjNum 1332 snap 368 seq_num 1332) dirty exists 2024-03-04T06:20:12.973 INFO:tasks.rados.rados.0.smithi064.stdout:2924: left oid 23 (ObjNum 1332 snap 368 seq_num 1332) 2024-03-04T06:20:13.393 INFO:tasks.rados.rados.0.smithi064.stdout:2917: done (9 left) 2024-03-04T06:20:13.393 INFO:tasks.rados.rados.0.smithi064.stdout:2920: done (8 left) 2024-03-04T06:20:13.393 INFO:tasks.rados.rados.0.smithi064.stdout:2921: done (7 left) 2024-03-04T06:20:13.393 INFO:tasks.rados.rados.0.smithi064.stdout:2922: done (6 left) 2024-03-04T06:20:13.393 INFO:tasks.rados.rados.0.smithi064.stdout:2923: done (5 left) 2024-03-04T06:20:13.393 INFO:tasks.rados.rados.0.smithi064.stdout:2924: done (4 left) 2024-03-04T06:20:13.394 INFO:tasks.rados.rados.0.smithi064.stdout:2925: done (3 left) 2024-03-04T06:20:13.394 INFO:tasks.rados.rados.0.smithi064.stdout:2926: done (2 left) 2024-03-04T06:20:13.394 INFO:tasks.rados.rados.0.smithi064.stdout:2927: done (1 left) 2024-03-04T06:20:13.394 INFO:tasks.rados.rados.0.smithi064.stdout:2928: done (0 left) 2024-03-04T06:20:13.394 INFO:tasks.rados.rados.0.smithi064.stdout:2929: snap_remove snap 367 2024-03-04T06:20:14.392 INFO:tasks.rados.rados.0.smithi064.stdout:2929: done (0 left) 2024-03-04T06:20:14.392 INFO:tasks.rados.rados.0.smithi064.stdout:2930: snap_create 2024-03-04T06:20:14.411 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:14.541 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:14.539+0000 7fc58f75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:14.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:14.543+0000 7fc58f75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:15.394 INFO:tasks.rados.rados.0.smithi064.stdout:2930: done (0 left) 2024-03-04T06:20:15.394 INFO:tasks.rados.rados.0.smithi064.stdout:2931: read oid 50 snap -1 2024-03-04T06:20:15.394 INFO:tasks.rados.rados.0.smithi064.stdout:2931: expect deleted 2024-03-04T06:20:15.394 INFO:tasks.rados.rados.0.smithi064.stdout:2932: write (excl) oid 2 current snap is 370 2024-03-04T06:20:15.394 INFO:tasks.rados.rados.0.smithi064.stdout:2932: seq_num 1335 ranges {437345=419190,1587814=591466} 2024-03-04T06:20:15.399 INFO:tasks.rados.rados.0.smithi064.stdout:2932: writing smithi06431743-2 from 437345 to 856535 tid 1 2024-03-04T06:20:15.406 INFO:tasks.rados.rados.0.smithi064.stdout:2932: writing smithi06431743-2 from 1587814 to 2179280 tid 2 2024-03-04T06:20:15.407 INFO:tasks.rados.rados.0.smithi064.stdout:2933: read oid 35 snap -1 2024-03-04T06:20:15.407 INFO:tasks.rados.rados.0.smithi064.stdout:2933: expect (ObjNum 1307 snap 359 seq_num 1307) 2024-03-04T06:20:15.407 INFO:tasks.rados.rados.0.smithi064.stdout:2934: delete oid 34 current snap is 370 2024-03-04T06:20:15.459 INFO:tasks.rados.rados.0.smithi064.stdout:2932: finishing write tid 1 to smithi06431743-2 2024-03-04T06:20:15.478 INFO:tasks.rados.rados.0.smithi064.stdout:2931: done (3 left) 2024-03-04T06:20:15.478 INFO:tasks.rados.rados.0.smithi064.stdout:2934: done (2 left) 2024-03-04T06:20:15.478 INFO:tasks.rados.rados.0.smithi064.stdout:2935: read oid 18 snap 362 2024-03-04T06:20:15.478 INFO:tasks.rados.rados.0.smithi064.stdout:2935: expect deleted 2024-03-04T06:20:15.479 INFO:tasks.rados.rados.0.smithi064.stdout:2936: write oid 38 current snap is 370 2024-03-04T06:20:15.479 INFO:tasks.rados.rados.0.smithi064.stdout:2936: seq_num 1337 ranges {470959=569688,1704778=595135} 2024-03-04T06:20:15.484 INFO:tasks.rados.rados.0.smithi064.stdout:2936: writing smithi06431743-38 from 470959 to 1040647 tid 1 2024-03-04T06:20:15.491 INFO:tasks.rados.rados.0.smithi064.stdout:2936: writing smithi06431743-38 from 1704778 to 2299913 tid 2 2024-03-04T06:20:15.491 INFO:tasks.rados.rados.0.smithi064.stdout:2937: write oid 28 current snap is 370 2024-03-04T06:20:15.491 INFO:tasks.rados.rados.0.smithi064.stdout:2937: seq_num 1338 ranges {487766=644937,1646907=738793} 2024-03-04T06:20:15.498 INFO:tasks.rados.rados.0.smithi064.stdout:2937: writing smithi06431743-28 from 487766 to 1132703 tid 1 2024-03-04T06:20:15.507 INFO:tasks.rados.rados.0.smithi064.stdout:2937: writing smithi06431743-28 from 1646907 to 2385700 tid 2 2024-03-04T06:20:15.507 INFO:tasks.rados.rados.0.smithi064.stdout:2938: write (excl) oid 27 current snap is 370 2024-03-04T06:20:15.507 INFO:tasks.rados.rados.0.smithi064.stdout:2938: seq_num 1339 ranges {504573=436539,1421742=598804,2437505=67068} 2024-03-04T06:20:15.513 INFO:tasks.rados.rados.0.smithi064.stdout:2938: writing smithi06431743-27 from 504573 to 941112 tid 1 2024-03-04T06:20:15.521 INFO:tasks.rados.rados.0.smithi064.stdout:2938: writing smithi06431743-27 from 1421742 to 2020546 tid 2 2024-03-04T06:20:15.524 INFO:tasks.rados.rados.0.smithi064.stdout:2938: writing smithi06431743-27 from 2437505 to 2504573 tid 3 2024-03-04T06:20:15.524 INFO:tasks.rados.rados.0.smithi064.stdout:2939: read oid 4 snap -1 2024-03-04T06:20:15.524 INFO:tasks.rados.rados.0.smithi064.stdout:2939: expect (ObjNum 1277 snap 346 seq_num 1277) 2024-03-04T06:20:15.524 INFO:tasks.rados.rados.0.smithi064.stdout:2940: snap_remove snap 366 2024-03-04T06:20:16.405 INFO:tasks.rados.rados.0.smithi064.stdout:2940: done (7 left) 2024-03-04T06:20:16.405 INFO:tasks.rados.rados.0.smithi064.stdout:2941: delete oid 9 current snap is 370 2024-03-04T06:20:16.405 INFO:tasks.rados.rados.0.smithi064.stdout:2933: expect (ObjNum 1307 snap 359 seq_num 1307) 2024-03-04T06:20:16.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:16.692 INFO:tasks.rados.rados.0.smithi064.stdout:2932: finishing write tid 2 to smithi06431743-2 2024-03-04T06:20:16.692 INFO:tasks.rados.rados.0.smithi064.stdout:2932: finishing write tid 3 to smithi06431743-2 2024-03-04T06:20:16.692 INFO:tasks.rados.rados.0.smithi064.stdout:2932: finishing write tid 4 to smithi06431743-2 2024-03-04T06:20:16.692 INFO:tasks.rados.rados.0.smithi064.stdout:2932: oid 2 updating version 0 to 654 2024-03-04T06:20:16.692 INFO:tasks.rados.rados.0.smithi064.stdout:2932: oid 2 updating version 654 to 656 2024-03-04T06:20:16.692 INFO:tasks.rados.rados.0.smithi064.stdout:2932: oid 2 version 656 is already newer than 655 2024-03-04T06:20:16.692 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 656 (ObjNum 1335 snap 370 seq_num 1335) dirty exists 2024-03-04T06:20:16.692 INFO:tasks.rados.rados.0.smithi064.stdout:2932: left oid 2 (ObjNum 1335 snap 370 seq_num 1335) 2024-03-04T06:20:16.693 INFO:tasks.rados.rados.0.smithi064.stdout:2936: finishing write tid 1 to smithi06431743-38 2024-03-04T06:20:16.693 INFO:tasks.rados.rados.0.smithi064.stdout:2936: finishing write tid 2 to smithi06431743-38 2024-03-04T06:20:16.693 INFO:tasks.rados.rados.0.smithi064.stdout:2938: finishing write tid 1 to smithi06431743-27 2024-03-04T06:20:16.693 INFO:tasks.rados.rados.0.smithi064.stdout:2937: finishing write tid 1 to smithi06431743-28 2024-03-04T06:20:16.693 INFO:tasks.rados.rados.0.smithi064.stdout:2932: done (7 left) 2024-03-04T06:20:16.693 INFO:tasks.rados.rados.0.smithi064.stdout:2933: done (6 left) 2024-03-04T06:20:16.693 INFO:tasks.rados.rados.0.smithi064.stdout:2935: done (5 left) 2024-03-04T06:20:16.693 INFO:tasks.rados.rados.0.smithi064.stdout:2941: done (4 left) 2024-03-04T06:20:16.693 INFO:tasks.rados.rados.0.smithi064.stdout:2942: read oid 6 snap -1 2024-03-04T06:20:16.693 INFO:tasks.rados.rados.0.smithi064.stdout:2942: expect deleted 2024-03-04T06:20:16.694 INFO:tasks.rados.rados.0.smithi064.stdout:2936: finishing write tid 3 to smithi06431743-38 2024-03-04T06:20:16.694 INFO:tasks.rados.rados.0.smithi064.stdout:2936: finishing write tid 4 to smithi06431743-38 2024-03-04T06:20:16.694 INFO:tasks.rados.rados.0.smithi064.stdout:2936: oid 38 updating version 0 to 800 2024-03-04T06:20:16.694 INFO:tasks.rados.rados.0.smithi064.stdout:2936: oid 38 updating version 800 to 802 2024-03-04T06:20:16.694 INFO:tasks.rados.rados.0.smithi064.stdout:2936: oid 38 version 802 is already newer than 801 2024-03-04T06:20:16.694 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 802 (ObjNum 1337 snap 370 seq_num 1337) dirty exists 2024-03-04T06:20:16.694 INFO:tasks.rados.rados.0.smithi064.stdout:2936: left oid 38 (ObjNum 1337 snap 370 seq_num 1337) 2024-03-04T06:20:16.694 INFO:tasks.rados.rados.0.smithi064.stdout:2938: finishing write tid 2 to smithi06431743-27 2024-03-04T06:20:16.694 INFO:tasks.rados.rados.0.smithi064.stdout:2939: expect (ObjNum 1277 snap 346 seq_num 1277) 2024-03-04T06:20:16.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:16.714+0000 7f445966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:16.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:16.718+0000 7f445966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:16.971 INFO:tasks.rados.rados.0.smithi064.stdout:2938: finishing write tid 3 to smithi06431743-27 2024-03-04T06:20:16.972 INFO:tasks.rados.rados.0.smithi064.stdout:2937: finishing write tid 2 to smithi06431743-28 2024-03-04T06:20:16.972 INFO:tasks.rados.rados.0.smithi064.stdout:2938: finishing write tid 4 to smithi06431743-27 2024-03-04T06:20:16.972 INFO:tasks.rados.rados.0.smithi064.stdout:2937: finishing write tid 3 to smithi06431743-28 2024-03-04T06:20:16.972 INFO:tasks.rados.rados.0.smithi064.stdout:2938: finishing write tid 5 to smithi06431743-27 2024-03-04T06:20:16.972 INFO:tasks.rados.rados.0.smithi064.stdout:2938: oid 27 updating version 0 to 584 2024-03-04T06:20:16.972 INFO:tasks.rados.rados.0.smithi064.stdout:2938: oid 27 version 584 is already newer than 581 2024-03-04T06:20:16.972 INFO:tasks.rados.rados.0.smithi064.stdout:2938: oid 27 version 584 is already newer than 583 2024-03-04T06:20:16.972 INFO:tasks.rados.rados.0.smithi064.stdout:2938: oid 27 version 584 is already newer than 583 2024-03-04T06:20:16.973 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 584 (ObjNum 1339 snap 370 seq_num 1339) dirty exists 2024-03-04T06:20:16.973 INFO:tasks.rados.rados.0.smithi064.stdout:2938: left oid 27 (ObjNum 1339 snap 370 seq_num 1339) 2024-03-04T06:20:16.973 INFO:tasks.rados.rados.0.smithi064.stdout:2937: finishing write tid 4 to smithi06431743-28 2024-03-04T06:20:16.973 INFO:tasks.rados.rados.0.smithi064.stdout:2937: oid 28 updating version 0 to 1305 2024-03-04T06:20:16.973 INFO:tasks.rados.rados.0.smithi064.stdout:2937: oid 28 updating version 1305 to 1307 2024-03-04T06:20:16.973 INFO:tasks.rados.rados.0.smithi064.stdout:2937: oid 28 version 1307 is already newer than 1307 2024-03-04T06:20:16.973 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 1307 (ObjNum 1338 snap 370 seq_num 1338) dirty exists 2024-03-04T06:20:16.973 INFO:tasks.rados.rados.0.smithi064.stdout:2937: left oid 28 (ObjNum 1338 snap 370 seq_num 1338) 2024-03-04T06:20:16.973 INFO:tasks.rados.rados.0.smithi064.stdout:2936: done (4 left) 2024-03-04T06:20:16.973 INFO:tasks.rados.rados.0.smithi064.stdout:2937: done (3 left) 2024-03-04T06:20:16.973 INFO:tasks.rados.rados.0.smithi064.stdout:2938: done (2 left) 2024-03-04T06:20:16.974 INFO:tasks.rados.rados.0.smithi064.stdout:2939: done (1 left) 2024-03-04T06:20:16.974 INFO:tasks.rados.rados.0.smithi064.stdout:2943: write (excl) oid 13 current snap is 370 2024-03-04T06:20:16.974 INFO:tasks.rados.rados.0.smithi064.stdout:2943: seq_num 1341 ranges {538187=587037,1538706=602473} 2024-03-04T06:20:16.978 INFO:tasks.rados.rados.0.smithi064.stdout:2943: writing smithi06431743-13 from 538187 to 1125224 tid 1 2024-03-04T06:20:16.986 INFO:tasks.rados.rados.0.smithi064.stdout:2943: writing smithi06431743-13 from 1538706 to 2141179 tid 2 2024-03-04T06:20:16.986 INFO:tasks.rados.rados.0.smithi064.stdout:2944: write oid 42 current snap is 370 2024-03-04T06:20:16.987 INFO:tasks.rados.rados.0.smithi064.stdout:2944: seq_num 1342 ranges {554994=662286,1997188=557806} 2024-03-04T06:20:16.995 INFO:tasks.rados.rados.0.smithi064.stdout:2944: writing smithi06431743-42 from 554994 to 1217280 tid 1 2024-03-04T06:20:17.004 INFO:tasks.rados.rados.0.smithi064.stdout:2944: writing smithi06431743-42 from 1997188 to 2554994 tid 2 2024-03-04T06:20:17.004 INFO:tasks.rados.rados.0.smithi064.stdout:2945: snap_create 2024-03-04T06:20:17.029 INFO:tasks.rados.rados.0.smithi064.stdout:2943: finishing write tid 1 to smithi06431743-13 2024-03-04T06:20:17.048 INFO:tasks.rados.rados.0.smithi064.stdout:2944: finishing write tid 1 to smithi06431743-42 2024-03-04T06:20:17.062 INFO:tasks.rados.rados.0.smithi064.stdout:2943: finishing write tid 2 to smithi06431743-13 2024-03-04T06:20:17.073 INFO:tasks.rados.rados.0.smithi064.stdout:2944: finishing write tid 2 to smithi06431743-42 2024-03-04T06:20:17.077 INFO:tasks.rados.rados.0.smithi064.stdout:2943: finishing write tid 3 to smithi06431743-13 2024-03-04T06:20:17.078 INFO:tasks.rados.rados.0.smithi064.stdout:2943: finishing write tid 4 to smithi06431743-13 2024-03-04T06:20:17.078 INFO:tasks.rados.rados.0.smithi064.stdout:2943: oid 13 updating version 0 to 1491 2024-03-04T06:20:17.078 INFO:tasks.rados.rados.0.smithi064.stdout:2943: oid 13 version 1491 is already newer than 1490 2024-03-04T06:20:17.078 INFO:tasks.rados.rados.0.smithi064.stdout:2943: oid 13 updating version 1491 to 1492 2024-03-04T06:20:17.078 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1492 (ObjNum 1341 snap 370 seq_num 1341) dirty exists 2024-03-04T06:20:17.078 INFO:tasks.rados.rados.0.smithi064.stdout:2943: left oid 13 (ObjNum 1341 snap 370 seq_num 1341) 2024-03-04T06:20:17.083 INFO:tasks.rados.rados.0.smithi064.stdout:2944: finishing write tid 3 to smithi06431743-42 2024-03-04T06:20:17.084 INFO:tasks.rados.rados.0.smithi064.stdout:2944: finishing write tid 4 to smithi06431743-42 2024-03-04T06:20:17.084 INFO:tasks.rados.rados.0.smithi064.stdout:2944: oid 42 updating version 0 to 685 2024-03-04T06:20:17.084 INFO:tasks.rados.rados.0.smithi064.stdout:2944: oid 42 updating version 685 to 686 2024-03-04T06:20:17.084 INFO:tasks.rados.rados.0.smithi064.stdout:2944: oid 42 updating version 686 to 687 2024-03-04T06:20:17.084 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 687 (ObjNum 1342 snap 370 seq_num 1342) dirty exists 2024-03-04T06:20:17.084 INFO:tasks.rados.rados.0.smithi064.stdout:2944: left oid 42 (ObjNum 1342 snap 370 seq_num 1342) 2024-03-04T06:20:17.407 INFO:tasks.rados.rados.0.smithi064.stdout:2942: done (3 left) 2024-03-04T06:20:17.408 INFO:tasks.rados.rados.0.smithi064.stdout:2943: done (2 left) 2024-03-04T06:20:17.408 INFO:tasks.rados.rados.0.smithi064.stdout:2944: done (1 left) 2024-03-04T06:20:17.408 INFO:tasks.rados.rados.0.smithi064.stdout:2945: done (0 left) 2024-03-04T06:20:17.408 INFO:tasks.rados.rados.0.smithi064.stdout:2946: read oid 28 snap -1 2024-03-04T06:20:17.408 INFO:tasks.rados.rados.0.smithi064.stdout:2946: expect (ObjNum 1338 snap 370 seq_num 1338) 2024-03-04T06:20:17.408 INFO:tasks.rados.rados.0.smithi064.stdout:2947: write oid 39 current snap is 371 2024-03-04T06:20:17.408 INFO:tasks.rados.rados.0.smithi064.stdout:2947: seq_num 1343 ranges {571801=737535,1939317=489789} 2024-03-04T06:20:17.414 INFO:tasks.rados.rados.0.smithi064.stdout:2947: writing smithi06431743-39 from 571801 to 1309336 tid 1 2024-03-04T06:20:17.422 INFO:tasks.rados.rados.0.smithi064.stdout:2947: writing smithi06431743-39 from 1939317 to 2429106 tid 2 2024-03-04T06:20:17.422 INFO:tasks.rados.rados.0.smithi064.stdout:2948: snap_remove snap 358 2024-03-04T06:20:17.431 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:20:17.431 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:20:17.431 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:20:17.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:20:17.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:17.539+0000 7fb00d79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:17.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:17.546+0000 7fb00d79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:20:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:20:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:20:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:20:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:20:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:20:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:20:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:20:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:20:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:20:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:20:17.560 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:20:17.562 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:20:17.562 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:20:17.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:20:17.672 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:17.667+0000 7f38f9017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:17.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:17.674+0000 7f38f9017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:17.689 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:20:17.689 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:20:17.689 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:20:17.690 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:20:17.690 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:20:17.690 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:20:17.690 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:20:17.690 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:20:17.690 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:20:17.690 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:20:17.690 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:20:17.690 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:20:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:20:17.693 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:20:17.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:20:17.801 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:17.796+0000 7f485d91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:17.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:17.803+0000 7f485d91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:17.817 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:20:17.817 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:20:17.817 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:20:17.818 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:20:17.818 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:20:17.818 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:20:17.818 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:20:17.818 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:20:17.818 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:20:17.818 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:20:17.818 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:20:17.818 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:20:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:20:17.820 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:20:17.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:20:17.934 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:17.929+0000 7fd944f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:17.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:17.937+0000 7fd944f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:17.951 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:20:17.951 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:20:17.951 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:20:17.951 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:20:17.951 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:20:17.951 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:20:17.952 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:20:17.952 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:20:17.952 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:20:17.952 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:20:17.952 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:20:17.952 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:20:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:20:17.955 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:20:18.300 INFO:tasks.rados.rados.0.smithi064.stdout:2948: done (2 left) 2024-03-04T06:20:18.301 INFO:tasks.rados.rados.0.smithi064.stdout:2949: snap_create 2024-03-04T06:20:18.301 INFO:tasks.rados.rados.0.smithi064.stdout:2947: finishing write tid 1 to smithi06431743-39 2024-03-04T06:20:18.301 INFO:tasks.rados.rados.0.smithi064.stdout:2946: expect (ObjNum 1338 snap 370 seq_num 1338) 2024-03-04T06:20:18.464 INFO:tasks.rados.rados.0.smithi064.stdout:2947: finishing write tid 2 to smithi06431743-39 2024-03-04T06:20:18.464 INFO:tasks.rados.rados.0.smithi064.stdout:2947: finishing write tid 3 to smithi06431743-39 2024-03-04T06:20:18.464 INFO:tasks.rados.rados.0.smithi064.stdout:2947: finishing write tid 4 to smithi06431743-39 2024-03-04T06:20:18.464 INFO:tasks.rados.rados.0.smithi064.stdout:2947: oid 39 updating version 0 to 758 2024-03-04T06:20:18.464 INFO:tasks.rados.rados.0.smithi064.stdout:2947: oid 39 updating version 758 to 759 2024-03-04T06:20:18.465 INFO:tasks.rados.rados.0.smithi064.stdout:2947: oid 39 updating version 759 to 760 2024-03-04T06:20:18.465 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 760 (ObjNum 1343 snap 371 seq_num 1343) dirty exists 2024-03-04T06:20:18.465 INFO:tasks.rados.rados.0.smithi064.stdout:2947: left oid 39 (ObjNum 1343 snap 371 seq_num 1343) 2024-03-04T06:20:18.788 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:18.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:18.893+0000 7fb624d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:18.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:18.897+0000 7fb624d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:19.304 INFO:tasks.rados.rados.0.smithi064.stdout:2946: done (2 left) 2024-03-04T06:20:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2947: done (1 left) 2024-03-04T06:20:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2949: done (0 left) 2024-03-04T06:20:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2950: read oid 38 snap 364 2024-03-04T06:20:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2950: expect deleted 2024-03-04T06:20:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2951: read oid 50 snap 368 2024-03-04T06:20:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2951: expect deleted 2024-03-04T06:20:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2952: read oid 49 snap 365 2024-03-04T06:20:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2952: expect (ObjNum 1169 snap 311 seq_num 1169) 2024-03-04T06:20:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2953: write (excl) oid 45 current snap is 372 2024-03-04T06:20:19.305 INFO:tasks.rados.rados.0.smithi064.stdout:2953: seq_num 1344 ranges {588608=412784,1597799=749800} 2024-03-04T06:20:19.310 INFO:tasks.rados.rados.0.smithi064.stdout:2953: writing smithi06431743-45 from 588608 to 1001392 tid 1 2024-03-04T06:20:19.320 INFO:tasks.rados.rados.0.smithi064.stdout:2953: writing smithi06431743-45 from 1597799 to 2347599 tid 2 2024-03-04T06:20:19.320 INFO:tasks.rados.rados.0.smithi064.stdout:2954: read oid 13 snap -1 2024-03-04T06:20:19.321 INFO:tasks.rados.rados.0.smithi064.stdout:2954: expect (ObjNum 1341 snap 370 seq_num 1341) 2024-03-04T06:20:19.321 INFO:tasks.rados.rados.0.smithi064.stdout:2955: write oid 2 current snap is 372 2024-03-04T06:20:19.321 INFO:tasks.rados.rados.0.smithi064.stdout:2955: seq_num 1345 ranges {605415=488033,1656281=493458} 2024-03-04T06:20:19.328 INFO:tasks.rados.rados.0.smithi064.stdout:2955: writing smithi06431743-2 from 605415 to 1093448 tid 1 2024-03-04T06:20:19.334 INFO:tasks.rados.rados.0.smithi064.stdout:2955: writing smithi06431743-2 from 1656281 to 2149739 tid 2 2024-03-04T06:20:19.334 INFO:tasks.rados.rados.0.smithi064.stdout:2956: write oid 14 current snap is 372 2024-03-04T06:20:19.334 INFO:tasks.rados.rados.0.smithi064.stdout:2956: seq_num 1346 ranges {622222=679635,1831116=753469} 2024-03-04T06:20:19.342 INFO:tasks.rados.rados.0.smithi064.stdout:2956: writing smithi06431743-14 from 622222 to 1301857 tid 1 2024-03-04T06:20:19.351 INFO:tasks.rados.rados.0.smithi064.stdout:2956: writing smithi06431743-14 from 1831116 to 2584585 tid 2 2024-03-04T06:20:19.351 INFO:tasks.rados.rados.0.smithi064.stdout:2957: delete oid 27 current snap is 372 2024-03-04T06:20:19.397 INFO:tasks.rados.rados.0.smithi064.stdout:2952: expect (ObjNum 1169 snap 311 seq_num 1169) 2024-03-04T06:20:19.674 INFO:tasks.rados.rados.0.smithi064.stdout:2954: expect (ObjNum 1341 snap 370 seq_num 1341) 2024-03-04T06:20:19.888 INFO:tasks.rados.rados.0.smithi064.stdout:2953: finishing write tid 1 to smithi06431743-45 2024-03-04T06:20:19.888 INFO:tasks.rados.rados.0.smithi064.stdout:2955: finishing write tid 1 to smithi06431743-2 2024-03-04T06:20:19.888 INFO:tasks.rados.rados.0.smithi064.stdout:2955: finishing write tid 2 to smithi06431743-2 2024-03-04T06:20:19.888 INFO:tasks.rados.rados.0.smithi064.stdout:2953: finishing write tid 2 to smithi06431743-45 2024-03-04T06:20:19.888 INFO:tasks.rados.rados.0.smithi064.stdout:2955: finishing write tid 3 to smithi06431743-2 2024-03-04T06:20:19.888 INFO:tasks.rados.rados.0.smithi064.stdout:2956: finishing write tid 1 to smithi06431743-14 2024-03-04T06:20:19.888 INFO:tasks.rados.rados.0.smithi064.stdout:2955: finishing write tid 4 to smithi06431743-2 2024-03-04T06:20:19.888 INFO:tasks.rados.rados.0.smithi064.stdout:2955: oid 2 updating version 0 to 664 2024-03-04T06:20:19.889 INFO:tasks.rados.rados.0.smithi064.stdout:2955: oid 2 version 664 is already newer than 662 2024-03-04T06:20:19.889 INFO:tasks.rados.rados.0.smithi064.stdout:2955: oid 2 version 664 is already newer than 664 2024-03-04T06:20:19.889 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 664 (ObjNum 1345 snap 372 seq_num 1345) dirty exists 2024-03-04T06:20:19.889 INFO:tasks.rados.rados.0.smithi064.stdout:2955: left oid 2 (ObjNum 1345 snap 372 seq_num 1345) 2024-03-04T06:20:19.889 INFO:tasks.rados.rados.0.smithi064.stdout:2956: finishing write tid 2 to smithi06431743-14 2024-03-04T06:20:19.889 INFO:tasks.rados.rados.0.smithi064.stdout:2950: done (7 left) 2024-03-04T06:20:19.889 INFO:tasks.rados.rados.0.smithi064.stdout:2951: done (6 left) 2024-03-04T06:20:19.889 INFO:tasks.rados.rados.0.smithi064.stdout:2952: done (5 left) 2024-03-04T06:20:19.889 INFO:tasks.rados.rados.0.smithi064.stdout:2954: done (4 left) 2024-03-04T06:20:19.889 INFO:tasks.rados.rados.0.smithi064.stdout:2955: done (3 left) 2024-03-04T06:20:19.890 INFO:tasks.rados.rados.0.smithi064.stdout:2957: done (2 left) 2024-03-04T06:20:19.890 INFO:tasks.rados.rados.0.smithi064.stdout:2958: read oid 11 snap -1 2024-03-04T06:20:19.890 INFO:tasks.rados.rados.0.smithi064.stdout:2958: expect (ObjNum 1314 snap 361 seq_num 1314) 2024-03-04T06:20:19.890 INFO:tasks.rados.rados.0.smithi064.stdout:2959: read oid 42 snap -1 2024-03-04T06:20:19.890 INFO:tasks.rados.rados.0.smithi064.stdout:2959: expect (ObjNum 1342 snap 370 seq_num 1342) 2024-03-04T06:20:19.890 INFO:tasks.rados.rados.0.smithi064.stdout:2960: delete oid 43 current snap is 372 2024-03-04T06:20:19.890 INFO:tasks.rados.rados.0.smithi064.stdout:2953: finishing write tid 3 to smithi06431743-45 2024-03-04T06:20:19.890 INFO:tasks.rados.rados.0.smithi064.stdout:2953: finishing write tid 4 to smithi06431743-45 2024-03-04T06:20:19.890 INFO:tasks.rados.rados.0.smithi064.stdout:2953: oid 45 updating version 0 to 764 2024-03-04T06:20:19.890 INFO:tasks.rados.rados.0.smithi064.stdout:2953: oid 45 updating version 764 to 765 2024-03-04T06:20:19.891 INFO:tasks.rados.rados.0.smithi064.stdout:2953: oid 45 updating version 765 to 766 2024-03-04T06:20:19.891 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 766 (ObjNum 1344 snap 372 seq_num 1344) dirty exists 2024-03-04T06:20:19.891 INFO:tasks.rados.rados.0.smithi064.stdout:2953: left oid 45 (ObjNum 1344 snap 372 seq_num 1344) 2024-03-04T06:20:19.891 INFO:tasks.rados.rados.0.smithi064.stdout:2956: finishing write tid 3 to smithi06431743-14 2024-03-04T06:20:19.891 INFO:tasks.rados.rados.0.smithi064.stdout:2956: finishing write tid 4 to smithi06431743-14 2024-03-04T06:20:19.891 INFO:tasks.rados.rados.0.smithi064.stdout:2956: oid 14 updating version 0 to 997 2024-03-04T06:20:19.891 INFO:tasks.rados.rados.0.smithi064.stdout:2956: oid 14 version 997 is already newer than 997 2024-03-04T06:20:19.891 INFO:tasks.rados.rados.0.smithi064.stdout:2956: oid 14 version 997 is already newer than 997 2024-03-04T06:20:19.891 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 997 (ObjNum 1346 snap 372 seq_num 1346) dirty exists 2024-03-04T06:20:19.891 INFO:tasks.rados.rados.0.smithi064.stdout:2956: left oid 14 (ObjNum 1346 snap 372 seq_num 1346) 2024-03-04T06:20:19.901 INFO:tasks.rados.rados.0.smithi064.stdout:2958: expect (ObjNum 1314 snap 361 seq_num 1314) 2024-03-04T06:20:19.980 INFO:tasks.rados.rados.0.smithi064.stdout:2953: done (4 left) 2024-03-04T06:20:19.980 INFO:tasks.rados.rados.0.smithi064.stdout:2956: done (3 left) 2024-03-04T06:20:19.980 INFO:tasks.rados.rados.0.smithi064.stdout:2958: done (2 left) 2024-03-04T06:20:19.980 INFO:tasks.rados.rados.0.smithi064.stdout:2960: done (1 left) 2024-03-04T06:20:19.980 INFO:tasks.rados.rados.0.smithi064.stdout:2961: snap_remove snap 325 2024-03-04T06:20:20.415 INFO:tasks.rados.rados.0.smithi064.stdout:2961: done (1 left) 2024-03-04T06:20:20.415 INFO:tasks.rados.rados.0.smithi064.stdout:2962: write oid 31 current snap is 372 2024-03-04T06:20:20.416 INFO:tasks.rados.rados.0.smithi064.stdout:2959: expect (ObjNum 1342 snap 370 seq_num 1342) 2024-03-04T06:20:20.604 INFO:tasks.rados.rados.0.smithi064.stdout:2962: seq_num 1349 ranges {672643=505382,1890209=500796} 2024-03-04T06:20:20.611 INFO:tasks.rados.rados.0.smithi064.stdout:2962: writing smithi06431743-31 from 672643 to 1178025 tid 1 2024-03-04T06:20:20.617 INFO:tasks.rados.rados.0.smithi064.stdout:2962: writing smithi06431743-31 from 1890209 to 2391005 tid 2 2024-03-04T06:20:20.617 INFO:tasks.rados.rados.0.smithi064.stdout:2959: done (1 left) 2024-03-04T06:20:20.617 INFO:tasks.rados.rados.0.smithi064.stdout:2963: snap_create 2024-03-04T06:20:20.632 INFO:tasks.rados.rados.0.smithi064.stdout:2962: finishing write tid 1 to smithi06431743-31 2024-03-04T06:20:20.644 INFO:tasks.rados.rados.0.smithi064.stdout:2962: finishing write tid 2 to smithi06431743-31 2024-03-04T06:20:20.649 INFO:tasks.rados.rados.0.smithi064.stdout:2962: finishing write tid 3 to smithi06431743-31 2024-03-04T06:20:20.650 INFO:tasks.rados.rados.0.smithi064.stdout:2962: finishing write tid 4 to smithi06431743-31 2024-03-04T06:20:20.650 INFO:tasks.rados.rados.0.smithi064.stdout:2962: oid 31 updating version 0 to 273 2024-03-04T06:20:20.650 INFO:tasks.rados.rados.0.smithi064.stdout:2962: oid 31 version 273 is already newer than 271 2024-03-04T06:20:20.650 INFO:tasks.rados.rados.0.smithi064.stdout:2962: oid 31 version 273 is already newer than 272 2024-03-04T06:20:20.650 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 273 (ObjNum 1349 snap 372 seq_num 1349) dirty exists 2024-03-04T06:20:20.650 INFO:tasks.rados.rados.0.smithi064.stdout:2962: left oid 31 (ObjNum 1349 snap 372 seq_num 1349) 2024-03-04T06:20:20.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:21.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:21.073+0000 7f32dd62e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:21.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:21.077+0000 7f32dd62e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:21.426 INFO:tasks.rados.rados.0.smithi064.stdout:2962: done (1 left) 2024-03-04T06:20:21.426 INFO:tasks.rados.rados.0.smithi064.stdout:2963: done (0 left) 2024-03-04T06:20:21.426 INFO:tasks.rados.rados.0.smithi064.stdout:2964: snap_create 2024-03-04T06:20:22.426 INFO:tasks.rados.rados.0.smithi064.stdout:2964: done (0 left) 2024-03-04T06:20:22.426 INFO:tasks.rados.rados.0.smithi064.stdout:2965: read oid 7 snap -1 2024-03-04T06:20:22.426 INFO:tasks.rados.rados.0.smithi064.stdout:2965: expect (ObjNum 1293 snap 355 seq_num 1293) 2024-03-04T06:20:22.426 INFO:tasks.rados.rados.0.smithi064.stdout:2966: snap_create 2024-03-04T06:20:22.483 INFO:tasks.rados.rados.0.smithi064.stdout:2965: expect (ObjNum 1293 snap 355 seq_num 1293) 2024-03-04T06:20:22.956 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:20:22.956 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:20:22.957 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:20:23.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:23.061+0000 7ff1a7d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:23.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:23.067+0000 7ff1a7d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:23.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:23.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:23.249+0000 7f36a4e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:23.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:23.253+0000 7f36a4e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:23.259 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:20:23.259 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:20:23.271 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1121,"stamp":"2024-03-04T06:20:22.866013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95308241,"num_objects":1004,"num_object_clones":806,"num_object_copies":3012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":163,"num_read":5459,"num_read_kb":4233445,"num_write":4640,"num_write_kb":1228943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11781,"ondisk_log_size":11781,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":725276,"kb_used_data":516668,"kb_used_omap":0,"kb_used_meta":208576,"kb_avail":376762084,"statfs":{"total":386547056640,"available":385804374016,"internally_reserved":0,"allocated":529068032,"data_stored":626926808,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":213581824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2688955,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":51,"num_read_kb":39129,"num_write":43,"num_write_kb":11240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000709"},"pg_stats":[{"pgid":"3.f","version":"748'691","reported_seq":2581,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.901278+0000","last_change":"2024-03-04T06:20:19.410572+0000","last_active":"2024-03-04T06:20:19.901278+0000","last_peered":"2024-03-04T06:20:19.901278+0000","last_clean":"2024-03-04T06:20:19.901278+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:20:19.901278+0000","last_undegraded":"2024-03-04T06:20:19.901278+0000","last_fullsized":"2024-03-04T06:20:19.901278+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":691,"log_dups_size":0,"ondisk_log_size":691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554994,"num_objects":66,"num_object_clones":53,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":310,"num_read_kb":251965,"num_write":273,"num_write_kb":72554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"748'588","reported_seq":2537,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.381464+0000","last_change":"2024-03-04T06:20:19.408598+0000","last_active":"2024-03-04T06:20:19.381464+0000","last_peered":"2024-03-04T06:20:19.381464+0000","last_clean":"2024-03-04T06:20:19.381464+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:20:19.381464+0000","last_undegraded":"2024-03-04T06:20:19.381464+0000","last_fullsized":"2024-03-04T06:20:19.381464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5950305,"num_objects":52,"num_object_clones":37,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":186,"num_read_kb":130657,"num_write":213,"num_write_kb":55628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"748'766","reported_seq":2705,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.898602+0000","last_change":"2024-03-04T06:20:18.348513+0000","last_active":"2024-03-04T06:20:19.898602+0000","last_peered":"2024-03-04T06:20:19.898602+0000","last_clean":"2024-03-04T06:20:19.898602+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:20:19.898602+0000","last_undegraded":"2024-03-04T06:20:19.898602+0000","last_fullsized":"2024-03-04T06:20:19.898602+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10724153,"num_objects":75,"num_object_clones":60,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":445,"num_read_kb":359293,"num_write":330,"num_write_kb":89465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"748'664","reported_seq":2538,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.497637+0000","last_change":"2024-03-04T06:20:19.499395+0000","last_active":"2024-03-04T06:20:19.497637+0000","last_peered":"2024-03-04T06:20:19.497637+0000","last_clean":"2024-03-04T06:20:19.497637+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:20:19.497637+0000","last_undegraded":"2024-03-04T06:20:19.497637+0000","last_fullsized":"2024-03-04T06:20:19.497637+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":664,"log_dups_size":0,"ondisk_log_size":664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9513686,"num_objects":51,"num_object_clones":41,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":264,"num_read_kb":202837,"num_write":246,"num_write_kb":68984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"749'273","reported_seq":2363,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-03-04T06:20:20.642673+0000","last_change":"2024-03-04T06:20:20.514434+0000","last_active":"2024-03-04T06:20:20.642673+0000","last_peered":"2024-03-04T06:20:20.642673+0000","last_clean":"2024-03-04T06:20:20.642673+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:20:20.642673+0000","last_undegraded":"2024-03-04T06:20:20.642673+0000","last_fullsized":"2024-03-04T06:20:20.642673+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672643,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":65484,"num_write":101,"num_write_kb":27678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"748'810","reported_seq":2631,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.335790+0000","last_change":"2024-03-04T06:20:19.449074+0000","last_active":"2024-03-04T06:20:19.335790+0000","last_peered":"2024-03-04T06:20:19.335790+0000","last_clean":"2024-03-04T06:20:19.335790+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:20:19.335790+0000","last_undegraded":"2024-03-04T06:20:19.335790+0000","last_fullsized":"2024-03-04T06:20:19.335790+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4757041,"num_objects":62,"num_object_clones":53,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":350,"num_read_kb":278962,"num_write":308,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2198,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.333055+0000","last_change":"2024-03-04T06:20:19.342542+0000","last_active":"2024-03-04T06:20:19.333055+0000","last_peered":"2024-03-04T06:20:19.333055+0000","last_clean":"2024-03-04T06:20:19.333055+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:20:19.333055+0000","last_undegraded":"2024-03-04T06:20:19.333055+0000","last_fullsized":"2024-03-04T06:20:19.333055+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"747'438","reported_seq":2451,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.391903+0000","last_change":"2024-03-04T06:20:18.345364+0000","last_active":"2024-03-04T06:20:19.391903+0000","last_peered":"2024-03-04T06:20:19.391903+0000","last_clean":"2024-03-04T06:20:19.391903+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:20:19.391903+0000","last_undegraded":"2024-03-04T06:20:19.391903+0000","last_fullsized":"2024-03-04T06:20:19.391903+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403731,"num_objects":35,"num_object_clones":26,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":222,"num_read_kb":204266,"num_write":154,"num_write_kb":41095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2018,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-03-04T06:20:20.427427+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:20:20.427427+0000","last_peered":"2024-03-04T06:20:20.427427+0000","last_clean":"2024-03-04T06:20:20.427427+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:20:20.427427+0000","last_undegraded":"2024-03-04T06:20:20.427427+0000","last_fullsized":"2024-03-04T06:20:20.427427+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1864,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.346667+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:20:19.346667+0000","last_peered":"2024-03-04T06:20:19.346667+0000","last_clean":"2024-03-04T06:20:19.346667+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:20:19.346667+0000","last_undegraded":"2024-03-04T06:20:19.346667+0000","last_fullsized":"2024-03-04T06:20:19.346667+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"747'735","reported_seq":2628,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.349675+0000","last_change":"2024-03-04T06:20:18.357346+0000","last_active":"2024-03-04T06:20:19.349675+0000","last_peered":"2024-03-04T06:20:19.349675+0000","last_clean":"2024-03-04T06:20:19.349675+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:20:19.349675+0000","last_undegraded":"2024-03-04T06:20:19.349675+0000","last_fullsized":"2024-03-04T06:20:19.349675+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":735,"log_dups_size":0,"ondisk_log_size":735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4757041,"num_objects":65,"num_object_clones":51,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":350,"num_read_kb":281384,"num_write":288,"num_write_kb":77449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1850,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.335110+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:20:19.335110+0000","last_peered":"2024-03-04T06:20:19.335110+0000","last_clean":"2024-03-04T06:20:19.335110+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:20:19.335110+0000","last_undegraded":"2024-03-04T06:20:19.335110+0000","last_fullsized":"2024-03-04T06:20:19.335110+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"749'456","reported_seq":2502,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-03-04T06:20:20.429025+0000","last_change":"2024-03-04T06:20:20.483331+0000","last_active":"2024-03-04T06:20:20.429025+0000","last_peered":"2024-03-04T06:20:20.429025+0000","last_clean":"2024-03-04T06:20:20.429025+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:20:20.429025+0000","last_undegraded":"2024-03-04T06:20:20.429025+0000","last_fullsized":"2024-03-04T06:20:20.429025+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":456,"log_dups_size":0,"ondisk_log_size":456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7446095,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":261,"num_read_kb":212085,"num_write":166,"num_write_kb":47091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1864,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.326152+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:20:19.326152+0000","last_peered":"2024-03-04T06:20:19.326152+0000","last_clean":"2024-03-04T06:20:19.326152+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:20:19.326152+0000","last_undegraded":"2024-03-04T06:20:19.326152+0000","last_fullsized":"2024-03-04T06:20:19.326152+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"749'1319","reported_seq":3035,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-03-04T06:20:20.435291+0000","last_change":"2024-03-04T06:20:20.553980+0000","last_active":"2024-03-04T06:20:20.435291+0000","last_peered":"2024-03-04T06:20:20.435291+0000","last_clean":"2024-03-04T06:20:20.435291+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:20:20.435291+0000","last_undegraded":"2024-03-04T06:20:20.435291+0000","last_fullsized":"2024-03-04T06:20:20.435291+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1319,"log_dups_size":0,"ondisk_log_size":1319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11278751,"num_objects":117,"num_object_clones":93,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":692,"num_read_kb":548421,"num_write":544,"num_write_kb":145500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1850,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.323993+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:20:19.323993+0000","last_peered":"2024-03-04T06:20:19.323993+0000","last_clean":"2024-03-04T06:20:19.323993+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:20:19.323993+0000","last_undegraded":"2024-03-04T06:20:19.323993+0000","last_fullsized":"2024-03-04T06:20:19.323993+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"748'1496","reported_seq":3012,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.417001+0000","last_change":"2024-03-04T06:20:19.435169+0000","last_active":"2024-03-04T06:20:19.417001+0000","last_peered":"2024-03-04T06:20:19.417001+0000","last_clean":"2024-03-04T06:20:19.417001+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:20:19.417001+0000","last_undegraded":"2024-03-04T06:20:19.417001+0000","last_fullsized":"2024-03-04T06:20:19.417001+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1496,"log_dups_size":0,"ondisk_log_size":1496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15985008,"num_objects":123,"num_object_clones":100,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":548,"num_read_kb":400606,"num_write":569,"num_write_kb":153035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"747'721","reported_seq":2634,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.348783+0000","last_change":"2024-03-04T06:20:18.357600+0000","last_active":"2024-03-04T06:20:19.348783+0000","last_peered":"2024-03-04T06:20:19.348783+0000","last_clean":"2024-03-04T06:20:19.348783+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:20:19.348783+0000","last_undegraded":"2024-03-04T06:20:19.348783+0000","last_fullsized":"2024-03-04T06:20:19.348783+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":59,"num_object_clones":47,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":392,"num_read_kb":329117,"num_write":269,"num_write_kb":72594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1916,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.332365+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:20:19.332365+0000","last_peered":"2024-03-04T06:20:19.332365+0000","last_clean":"2024-03-04T06:20:19.332365+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:20:19.332365+0000","last_undegraded":"2024-03-04T06:20:19.332365+0000","last_fullsized":"2024-03-04T06:20:19.332365+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"748'764","reported_seq":2671,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.337169+0000","last_change":"2024-03-04T06:20:19.455640+0000","last_active":"2024-03-04T06:20:19.337169+0000","last_peered":"2024-03-04T06:20:19.337169+0000","last_clean":"2024-03-04T06:20:19.337169+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:20:19.337169+0000","last_undegraded":"2024-03-04T06:20:19.337169+0000","last_fullsized":"2024-03-04T06:20:19.337169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4252831,"num_objects":71,"num_object_clones":55,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":325,"num_read_kb":236248,"num_write":297,"num_write_kb":79923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1885,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-03-04T06:20:20.436500+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:20:20.436500+0000","last_peered":"2024-03-04T06:20:20.436500+0000","last_clean":"2024-03-04T06:20:20.436500+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:20:20.436500+0000","last_undegraded":"2024-03-04T06:20:20.436500+0000","last_fullsized":"2024-03-04T06:20:20.436500+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"749'1001","reported_seq":2765,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-03-04T06:20:20.423117+0000","last_change":"2024-03-04T06:20:20.498089+0000","last_active":"2024-03-04T06:20:20.423117+0000","last_peered":"2024-03-04T06:20:20.423117+0000","last_clean":"2024-03-04T06:20:20.423117+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:20:20.423117+0000","last_undegraded":"2024-03-04T06:20:20.423117+0000","last_fullsized":"2024-03-04T06:20:20.423117+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12421594,"num_objects":83,"num_object_clones":69,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":420,"num_read_kb":309261,"num_write":389,"num_write_kb":109139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1916,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.320212+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:20:19.320212+0000","last_peered":"2024-03-04T06:20:19.320212+0000","last_clean":"2024-03-04T06:20:19.320212+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:20:19.320212+0000","last_undegraded":"2024-03-04T06:20:19.320212+0000","last_fullsized":"2024-03-04T06:20:19.320212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"737'962","reported_seq":2811,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.319253+0000","last_change":"2024-03-04T06:20:19.341514+0000","last_active":"2024-03-04T06:20:19.319253+0000","last_peered":"2024-03-04T06:20:19.319253+0000","last_clean":"2024-03-04T06:20:19.319253+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:20:19.319253+0000","last_undegraded":"2024-03-04T06:20:19.319253+0000","last_fullsized":"2024-03-04T06:20:19.319253+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":80,"num_object_clones":70,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":500,"num_read_kb":422768,"num_write":356,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1864,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-03-04T06:20:19.346480+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:20:19.346480+0000","last_peered":"2024-03-04T06:20:19.346480+0000","last_clean":"2024-03-04T06:20:19.346480+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:20:19.346480+0000","last_undegraded":"2024-03-04T06:20:19.346480+0000","last_fullsized":"2024-03-04T06:20:19.346480+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94717873,"num_objects":1002,"num_object_clones":806,"num_object_copies":3006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":163,"num_read":5356,"num_read_kb":4233354,"num_write":4503,"num_write_kb":1227049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11684,"ondisk_log_size":11684,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706419,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":212260,"kb_used_data":154908,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94159580,"statfs":{"total":96636764160,"available":96419409920,"internally_reserved":0,"allocated":158625792,"data_stored":187807148,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706419,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":167112,"kb_used_data":113920,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94204728,"statfs":{"total":96636764160,"available":96465641472,"internally_reserved":0,"allocated":116654080,"data_stored":133571953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706418,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":167632,"kb_used_data":118600,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94204208,"statfs":{"total":96636764160,"available":96465108992,"internally_reserved":0,"allocated":121446400,"data_stored":144549988,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706418,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":178272,"kb_used_data":129240,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94193568,"statfs":{"total":96636764160,"available":96454213632,"internally_reserved":0,"allocated":132341760,"data_stored":160997719,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:20:23.425 INFO:tasks.rados.rados.0.smithi064.stdout:2965: done (1 left) 2024-03-04T06:20:23.425 INFO:tasks.rados.rados.0.smithi064.stdout:2966: done (0 left) 2024-03-04T06:20:23.425 INFO:tasks.rados.rados.0.smithi064.stdout:2967: write (excl) oid 12 current snap is 375 2024-03-04T06:20:23.426 INFO:tasks.rados.rados.0.smithi064.stdout:2967: seq_num 1350 ranges {689450=580631,1948691=740759} 2024-03-04T06:20:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:2967: writing smithi06431743-12 from 689450 to 1270081 tid 1 2024-03-04T06:20:23.445 INFO:tasks.rados.rados.0.smithi064.stdout:2967: writing smithi06431743-12 from 1948691 to 2689450 tid 2 2024-03-04T06:20:23.445 INFO:tasks.rados.rados.0.smithi064.stdout:2968: read oid 28 snap -1 2024-03-04T06:20:23.445 INFO:tasks.rados.rados.0.smithi064.stdout:2968: expect (ObjNum 1338 snap 370 seq_num 1338) 2024-03-04T06:20:23.445 INFO:tasks.rados.rados.0.smithi064.stdout:2969: snap_create 2024-03-04T06:20:23.500 INFO:tasks.rados.rados.0.smithi064.stdout:2967: finishing write tid 1 to smithi06431743-12 2024-03-04T06:20:23.511 INFO:tasks.rados.rados.0.smithi064.stdout:2968: expect (ObjNum 1338 snap 370 seq_num 1338) 2024-03-04T06:20:23.667 INFO:tasks.rados.rados.0.smithi064.stdout:2967: finishing write tid 2 to smithi06431743-12 2024-03-04T06:20:23.667 INFO:tasks.rados.rados.0.smithi064.stdout:2967: finishing write tid 3 to smithi06431743-12 2024-03-04T06:20:23.667 INFO:tasks.rados.rados.0.smithi064.stdout:2967: finishing write tid 4 to smithi06431743-12 2024-03-04T06:20:23.667 INFO:tasks.rados.rados.0.smithi064.stdout:2967: oid 12 updating version 0 to 672 2024-03-04T06:20:23.667 INFO:tasks.rados.rados.0.smithi064.stdout:2967: oid 12 version 672 is already newer than 672 2024-03-04T06:20:23.667 INFO:tasks.rados.rados.0.smithi064.stdout:2967: oid 12 version 672 is already newer than 670 2024-03-04T06:20:23.667 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 672 (ObjNum 1350 snap 375 seq_num 1350) dirty exists 2024-03-04T06:20:23.667 INFO:tasks.rados.rados.0.smithi064.stdout:2967: left oid 12 (ObjNum 1350 snap 375 seq_num 1350) 2024-03-04T06:20:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:2967: done (2 left) 2024-03-04T06:20:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:2968: done (1 left) 2024-03-04T06:20:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:2969: done (0 left) 2024-03-04T06:20:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:2970: write (excl) oid 41 current snap is 376 2024-03-04T06:20:24.434 INFO:tasks.rados.rados.0.smithi064.stdout:2970: seq_num 1351 ranges {706257=655880,1890820=504465} 2024-03-04T06:20:24.441 INFO:tasks.rados.rados.0.smithi064.stdout:2970: writing smithi06431743-41 from 706257 to 1362137 tid 1 2024-03-04T06:20:24.448 INFO:tasks.rados.rados.0.smithi064.stdout:2970: writing smithi06431743-41 from 1890820 to 2395285 tid 2 2024-03-04T06:20:24.448 INFO:tasks.rados.rados.0.smithi064.stdout:2971: snap_remove snap 368 2024-03-04T06:20:25.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:25.436 INFO:tasks.rados.rados.0.smithi064.stdout:2971: done (1 left) 2024-03-04T06:20:25.437 INFO:tasks.rados.rados.0.smithi064.stdout:2972: delete oid 17 current snap is 376 2024-03-04T06:20:25.437 INFO:tasks.rados.rados.0.smithi064.stdout:2970: finishing write tid 1 to smithi06431743-41 2024-03-04T06:20:25.437 INFO:tasks.rados.rados.0.smithi064.stdout:2970: finishing write tid 2 to smithi06431743-41 2024-03-04T06:20:25.437 INFO:tasks.rados.rados.0.smithi064.stdout:2970: finishing write tid 3 to smithi06431743-41 2024-03-04T06:20:25.437 INFO:tasks.rados.rados.0.smithi064.stdout:2970: finishing write tid 4 to smithi06431743-41 2024-03-04T06:20:25.437 INFO:tasks.rados.rados.0.smithi064.stdout:2970: oid 41 updating version 0 to 740 2024-03-04T06:20:25.437 INFO:tasks.rados.rados.0.smithi064.stdout:2970: oid 41 version 740 is already newer than 740 2024-03-04T06:20:25.437 INFO:tasks.rados.rados.0.smithi064.stdout:2970: oid 41 version 740 is already newer than 738 2024-03-04T06:20:25.437 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 740 (ObjNum 1351 snap 376 seq_num 1351) dirty exists 2024-03-04T06:20:25.437 INFO:tasks.rados.rados.0.smithi064.stdout:2970: left oid 41 (ObjNum 1351 snap 376 seq_num 1351) 2024-03-04T06:20:25.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:25.442+0000 7f152240f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:25.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:25.447+0000 7f152240f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:25.472 INFO:tasks.rados.rados.0.smithi064.stdout:2970: done (1 left) 2024-03-04T06:20:25.472 INFO:tasks.rados.rados.0.smithi064.stdout:2972: done (0 left) 2024-03-04T06:20:25.472 INFO:tasks.rados.rados.0.smithi064.stdout:2973: read oid 9 snap -1 2024-03-04T06:20:25.472 INFO:tasks.rados.rados.0.smithi064.stdout:2973: expect deleted 2024-03-04T06:20:25.472 INFO:tasks.rados.rados.0.smithi064.stdout:2974: snap_create 2024-03-04T06:20:26.439 INFO:tasks.rados.rados.0.smithi064.stdout:2973: done (1 left) 2024-03-04T06:20:26.439 INFO:tasks.rados.rados.0.smithi064.stdout:2974: done (0 left) 2024-03-04T06:20:26.439 INFO:tasks.rados.rados.0.smithi064.stdout:2975: write oid 7 current snap is 377 2024-03-04T06:20:26.439 INFO:tasks.rados.rados.0.smithi064.stdout:2975: seq_num 1353 ranges {739871=406378,1607784=508134} 2024-03-04T06:20:26.445 INFO:tasks.rados.rados.0.smithi064.stdout:2975: writing smithi06431743-7 from 739871 to 1146249 tid 1 2024-03-04T06:20:26.450 INFO:tasks.rados.rados.0.smithi064.stdout:2975: writing smithi06431743-7 from 1607784 to 2115918 tid 2 2024-03-04T06:20:26.451 INFO:tasks.rados.rados.0.smithi064.stdout:2976: snap_remove snap 348 2024-03-04T06:20:27.443 INFO:tasks.rados.rados.0.smithi064.stdout:2976: done (1 left) 2024-03-04T06:20:27.443 INFO:tasks.rados.rados.0.smithi064.stdout:2977: write oid 15 current snap is 377 2024-03-04T06:20:27.443 INFO:tasks.rados.rados.0.smithi064.stdout:2975: finishing write tid 1 to smithi06431743-7 2024-03-04T06:20:27.444 INFO:tasks.rados.rados.0.smithi064.stdout:2975: finishing write tid 2 to smithi06431743-7 2024-03-04T06:20:27.444 INFO:tasks.rados.rados.0.smithi064.stdout:2975: finishing write tid 3 to smithi06431743-7 2024-03-04T06:20:27.444 INFO:tasks.rados.rados.0.smithi064.stdout:2977: seq_num 1354 ranges {756678=597980,1782619=768145} 2024-03-04T06:20:27.450 INFO:tasks.rados.rados.0.smithi064.stdout:2977: writing smithi06431743-15 from 756678 to 1354658 tid 1 2024-03-04T06:20:27.458 INFO:tasks.rados.rados.0.smithi064.stdout:2977: writing smithi06431743-15 from 1782619 to 2550764 tid 2 2024-03-04T06:20:27.458 INFO:tasks.rados.rados.0.smithi064.stdout:2978: write (excl) oid 36 current snap is 377 2024-03-04T06:20:27.458 INFO:tasks.rados.rados.0.smithi064.stdout:2975: finishing write tid 4 to smithi06431743-7 2024-03-04T06:20:27.458 INFO:tasks.rados.rados.0.smithi064.stdout:2975: oid 7 updating version 0 to 460 2024-03-04T06:20:27.458 INFO:tasks.rados.rados.0.smithi064.stdout:2975: oid 7 version 460 is already newer than 458 2024-03-04T06:20:27.458 INFO:tasks.rados.rados.0.smithi064.stdout:2975: oid 7 version 460 is already newer than 460 2024-03-04T06:20:27.458 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 460 (ObjNum 1353 snap 377 seq_num 1353) dirty exists 2024-03-04T06:20:27.458 INFO:tasks.rados.rados.0.smithi064.stdout:2975: left oid 7 (ObjNum 1353 snap 377 seq_num 1353) 2024-03-04T06:20:27.458 INFO:tasks.rados.rados.0.smithi064.stdout:2978: seq_num 1355 ranges {773485=673229,2124748=511803} 2024-03-04T06:20:27.467 INFO:tasks.rados.rados.0.smithi064.stdout:2978: writing smithi06431743-36 from 773485 to 1446714 tid 1 2024-03-04T06:20:27.475 INFO:tasks.rados.rados.0.smithi064.stdout:2978: writing smithi06431743-36 from 2124748 to 2636551 tid 2 2024-03-04T06:20:27.475 INFO:tasks.rados.rados.0.smithi064.stdout:2975: done (2 left) 2024-03-04T06:20:27.475 INFO:tasks.rados.rados.0.smithi064.stdout:2979: snap_remove snap 373 2024-03-04T06:20:27.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:27.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:27.650+0000 7f3626ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:27.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:27.654+0000 7f3626ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:28.274 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:20:28.274 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:20:28.274 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:20:28.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:20:28.303 INFO:tasks.rados.rados.0.smithi064.stdout:2979: done (2 left) 2024-03-04T06:20:28.303 INFO:tasks.rados.rados.0.smithi064.stdout:2980: write oid 2 current snap is 377 2024-03-04T06:20:28.303 INFO:tasks.rados.rados.0.smithi064.stdout:2978: finishing write tid 1 to smithi06431743-36 2024-03-04T06:20:28.303 INFO:tasks.rados.rados.0.smithi064.stdout:2980: seq_num 1356 ranges {790292=748478,2183230=607062} 2024-03-04T06:20:28.311 INFO:tasks.rados.rados.0.smithi064.stdout:2980: writing smithi06431743-2 from 790292 to 1538770 tid 1 2024-03-04T06:20:28.319 INFO:tasks.rados.rados.0.smithi064.stdout:2980: writing smithi06431743-2 from 2183230 to 2790292 tid 2 2024-03-04T06:20:28.319 INFO:tasks.rados.rados.0.smithi064.stdout:2981: snap_create 2024-03-04T06:20:28.319 INFO:tasks.rados.rados.0.smithi064.stdout:2977: finishing write tid 1 to smithi06431743-15 2024-03-04T06:20:28.319 INFO:tasks.rados.rados.0.smithi064.stdout:2978: finishing write tid 2 to smithi06431743-36 2024-03-04T06:20:28.319 INFO:tasks.rados.rados.0.smithi064.stdout:2977: finishing write tid 2 to smithi06431743-15 2024-03-04T06:20:28.319 INFO:tasks.rados.rados.0.smithi064.stdout:2978: finishing write tid 3 to smithi06431743-36 2024-03-04T06:20:28.319 INFO:tasks.rados.rados.0.smithi064.stdout:2978: finishing write tid 4 to smithi06431743-36 2024-03-04T06:20:28.319 INFO:tasks.rados.rados.0.smithi064.stdout:2978: oid 36 updating version 0 to 775 2024-03-04T06:20:28.319 INFO:tasks.rados.rados.0.smithi064.stdout:2978: oid 36 version 775 is already newer than 773 2024-03-04T06:20:28.319 INFO:tasks.rados.rados.0.smithi064.stdout:2978: oid 36 version 775 is already newer than 774 2024-03-04T06:20:28.319 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 775 (ObjNum 1355 snap 377 seq_num 1355) dirty exists 2024-03-04T06:20:28.320 INFO:tasks.rados.rados.0.smithi064.stdout:2978: left oid 36 (ObjNum 1355 snap 377 seq_num 1355) 2024-03-04T06:20:28.320 INFO:tasks.rados.rados.0.smithi064.stdout:2977: finishing write tid 3 to smithi06431743-15 2024-03-04T06:20:28.320 INFO:tasks.rados.rados.0.smithi064.stdout:2977: finishing write tid 4 to smithi06431743-15 2024-03-04T06:20:28.320 INFO:tasks.rados.rados.0.smithi064.stdout:2977: oid 15 updating version 0 to 730 2024-03-04T06:20:28.320 INFO:tasks.rados.rados.0.smithi064.stdout:2977: oid 15 updating version 730 to 731 2024-03-04T06:20:28.320 INFO:tasks.rados.rados.0.smithi064.stdout:2977: oid 15 updating version 731 to 732 2024-03-04T06:20:28.320 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 732 (ObjNum 1354 snap 377 seq_num 1354) dirty exists 2024-03-04T06:20:28.320 INFO:tasks.rados.rados.0.smithi064.stdout:2977: left oid 15 (ObjNum 1354 snap 377 seq_num 1354) 2024-03-04T06:20:28.405 INFO:tasks.rados.rados.0.smithi064.stdout:2980: finishing write tid 1 to smithi06431743-2 2024-03-04T06:20:28.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:28.414+0000 7f02ec55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:28.421 INFO:tasks.rados.rados.0.smithi064.stdout:2980: finishing write tid 2 to smithi06431743-2 2024-03-04T06:20:28.422 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:28.422+0000 7f02ec55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:28.430 INFO:tasks.rados.rados.0.smithi064.stdout:2980: finishing write tid 3 to smithi06431743-2 2024-03-04T06:20:28.431 INFO:tasks.rados.rados.0.smithi064.stdout:2980: finishing write tid 4 to smithi06431743-2 2024-03-04T06:20:28.431 INFO:tasks.rados.rados.0.smithi064.stdout:2980: oid 2 updating version 0 to 682 2024-03-04T06:20:28.431 INFO:tasks.rados.rados.0.smithi064.stdout:2980: oid 2 updating version 682 to 683 2024-03-04T06:20:28.431 INFO:tasks.rados.rados.0.smithi064.stdout:2980: oid 2 updating version 683 to 684 2024-03-04T06:20:28.432 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 684 (ObjNum 1356 snap 377 seq_num 1356) dirty exists 2024-03-04T06:20:28.432 INFO:tasks.rados.rados.0.smithi064.stdout:2980: left oid 2 (ObjNum 1356 snap 377 seq_num 1356) 2024-03-04T06:20:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:20:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:20:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:20:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:20:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:20:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:20:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:20:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:20:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:20:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:20:28.436 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:20:28.436 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:20:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:20:28.438 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:20:28.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:20:28.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:28.583+0000 7fc9db46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:28.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:28.591+0000 7fc9db46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:28.603 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:20:28.603 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:20:28.603 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:20:28.603 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:20:28.604 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:20:28.604 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:20:28.604 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:20:28.604 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:20:28.604 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:20:28.604 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:20:28.604 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:20:28.604 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:20:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:20:28.607 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:20:28.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:20:28.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:28.710+0000 7fd872264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:28.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:28.717+0000 7fd872264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:20:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:20:28.733 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:20:28.733 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:20:28.733 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:20:28.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:28.836+0000 7f7804354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:28.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:28.844+0000 7f7804354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:28.856 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:20:28.856 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:20:28.856 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:20:28.856 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:20:28.856 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:20:28.856 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:20:28.856 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:20:28.857 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:20:28.857 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:20:28.857 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:20:28.857 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:20:28.857 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:20:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:20:28.859 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:20:29.307 INFO:tasks.rados.rados.0.smithi064.stdout:2977: done (3 left) 2024-03-04T06:20:29.307 INFO:tasks.rados.rados.0.smithi064.stdout:2978: done (2 left) 2024-03-04T06:20:29.307 INFO:tasks.rados.rados.0.smithi064.stdout:2980: done (1 left) 2024-03-04T06:20:29.307 INFO:tasks.rados.rados.0.smithi064.stdout:2981: done (0 left) 2024-03-04T06:20:29.307 INFO:tasks.rados.rados.0.smithi064.stdout:2982: snap_remove snap 370 2024-03-04T06:20:29.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:29.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:29.827+0000 7f57b135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:29.832 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:29.831+0000 7f57b135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:30.308 INFO:tasks.rados.rados.0.smithi064.stdout:2982: done (0 left) 2024-03-04T06:20:30.308 INFO:tasks.rados.rados.0.smithi064.stdout:2983: delete oid 42 current snap is 378 2024-03-04T06:20:30.379 INFO:tasks.rados.rados.0.smithi064.stdout:2983: done (0 left) 2024-03-04T06:20:30.379 INFO:tasks.rados.rados.0.smithi064.stdout:2984: read oid 40 snap -1 2024-03-04T06:20:30.379 INFO:tasks.rados.rados.0.smithi064.stdout:2984: expect deleted 2024-03-04T06:20:30.379 INFO:tasks.rados.rados.0.smithi064.stdout:2985: write oid 45 current snap is 378 2024-03-04T06:20:30.379 INFO:tasks.rados.rados.0.smithi064.stdout:2985: seq_num 1358 ranges {423906=498976,1500194=659130,2609483=214423} 2024-03-04T06:20:30.383 INFO:tasks.rados.rados.0.smithi064.stdout:2985: writing smithi06431743-45 from 423906 to 922882 tid 1 2024-03-04T06:20:30.390 INFO:tasks.rados.rados.0.smithi064.stdout:2985: writing smithi06431743-45 from 1500194 to 2159324 tid 2 2024-03-04T06:20:30.394 INFO:tasks.rados.rados.0.smithi064.stdout:2985: writing smithi06431743-45 from 2609483 to 2823906 tid 3 2024-03-04T06:20:30.394 INFO:tasks.rados.rados.0.smithi064.stdout:2986: read oid 19 snap -1 2024-03-04T06:20:30.394 INFO:tasks.rados.rados.0.smithi064.stdout:2986: expect (ObjNum 1327 snap 368 seq_num 1327) 2024-03-04T06:20:30.394 INFO:tasks.rados.rados.0.smithi064.stdout:2987: write (excl) oid 42 current snap is 378 2024-03-04T06:20:30.394 INFO:tasks.rados.rados.0.smithi064.stdout:2987: seq_num 1359 ranges {440713=574225,1442323=519141,2636906=203807} 2024-03-04T06:20:30.399 INFO:tasks.rados.rados.0.smithi064.stdout:2987: writing smithi06431743-42 from 440713 to 1014938 tid 1 2024-03-04T06:20:30.405 INFO:tasks.rados.rados.0.smithi064.stdout:2987: writing smithi06431743-42 from 1442323 to 1961464 tid 2 2024-03-04T06:20:30.410 INFO:tasks.rados.rados.0.smithi064.stdout:2987: writing smithi06431743-42 from 2636906 to 2840713 tid 3 2024-03-04T06:20:30.410 INFO:tasks.rados.rados.0.smithi064.stdout:2988: write (excl) oid 8 current snap is 378 2024-03-04T06:20:30.410 INFO:tasks.rados.rados.0.smithi064.stdout:2988: seq_num 1360 ranges {457520=649474,1900805=662799} 2024-03-04T06:20:30.416 INFO:tasks.rados.rados.0.smithi064.stdout:2988: writing smithi06431743-8 from 457520 to 1106994 tid 1 2024-03-04T06:20:30.424 INFO:tasks.rados.rados.0.smithi064.stdout:2988: writing smithi06431743-8 from 1900805 to 2563604 tid 2 2024-03-04T06:20:30.424 INFO:tasks.rados.rados.0.smithi064.stdout:2989: write (excl) oid 14 current snap is 378 2024-03-04T06:20:30.424 INFO:tasks.rados.rados.0.smithi064.stdout:2989: seq_num 1361 ranges {474327=441076,1675640=522810,2808105=66222} 2024-03-04T06:20:30.430 INFO:tasks.rados.rados.0.smithi064.stdout:2989: writing smithi06431743-14 from 474327 to 915403 tid 1 2024-03-04T06:20:30.437 INFO:tasks.rados.rados.0.smithi064.stdout:2989: writing smithi06431743-14 from 1675640 to 2198450 tid 2 2024-03-04T06:20:30.441 INFO:tasks.rados.rados.0.smithi064.stdout:2989: writing smithi06431743-14 from 2808105 to 2874327 tid 3 2024-03-04T06:20:30.441 INFO:tasks.rados.rados.0.smithi064.stdout:2990: read oid 43 snap -1 2024-03-04T06:20:30.441 INFO:tasks.rados.rados.0.smithi064.stdout:2990: expect deleted 2024-03-04T06:20:30.442 INFO:tasks.rados.rados.0.smithi064.stdout:2991: snap_create 2024-03-04T06:20:30.442 INFO:tasks.rados.rados.0.smithi064.stdout:2985: finishing write tid 1 to smithi06431743-45 2024-03-04T06:20:30.442 INFO:tasks.rados.rados.0.smithi064.stdout:2987: finishing write tid 1 to smithi06431743-42 2024-03-04T06:20:30.442 INFO:tasks.rados.rados.0.smithi064.stdout:2985: finishing write tid 2 to smithi06431743-45 2024-03-04T06:20:30.444 INFO:tasks.rados.rados.0.smithi064.stdout:2985: finishing write tid 3 to smithi06431743-45 2024-03-04T06:20:30.448 INFO:tasks.rados.rados.0.smithi064.stdout:2987: finishing write tid 2 to smithi06431743-42 2024-03-04T06:20:30.468 INFO:tasks.rados.rados.0.smithi064.stdout:2985: finishing write tid 4 to smithi06431743-45 2024-03-04T06:20:30.469 INFO:tasks.rados.rados.0.smithi064.stdout:2985: finishing write tid 5 to smithi06431743-45 2024-03-04T06:20:30.469 INFO:tasks.rados.rados.0.smithi064.stdout:2985: oid 45 updating version 0 to 790 2024-03-04T06:20:30.469 INFO:tasks.rados.rados.0.smithi064.stdout:2985: oid 45 version 790 is already newer than 788 2024-03-04T06:20:30.470 INFO:tasks.rados.rados.0.smithi064.stdout:2985: oid 45 version 790 is already newer than 790 2024-03-04T06:20:30.470 INFO:tasks.rados.rados.0.smithi064.stdout:2985: oid 45 updating version 790 to 791 2024-03-04T06:20:30.470 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 791 (ObjNum 1358 snap 378 seq_num 1358) dirty exists 2024-03-04T06:20:30.470 INFO:tasks.rados.rados.0.smithi064.stdout:2985: left oid 45 (ObjNum 1358 snap 378 seq_num 1358) 2024-03-04T06:20:30.486 INFO:tasks.rados.rados.0.smithi064.stdout:2986: expect (ObjNum 1327 snap 368 seq_num 1327) 2024-03-04T06:20:30.657 INFO:tasks.rados.rados.0.smithi064.stdout:2987: finishing write tid 3 to smithi06431743-42 2024-03-04T06:20:30.657 INFO:tasks.rados.rados.0.smithi064.stdout:2988: finishing write tid 1 to smithi06431743-8 2024-03-04T06:20:30.657 INFO:tasks.rados.rados.0.smithi064.stdout:2987: finishing write tid 4 to smithi06431743-42 2024-03-04T06:20:30.657 INFO:tasks.rados.rados.0.smithi064.stdout:2987: finishing write tid 5 to smithi06431743-42 2024-03-04T06:20:30.657 INFO:tasks.rados.rados.0.smithi064.stdout:2987: oid 42 updating version 0 to 712 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2987: oid 42 version 712 is already newer than 711 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2987: oid 42 version 712 is already newer than 710 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2987: oid 42 updating version 712 to 713 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 713 (ObjNum 1359 snap 378 seq_num 1359) dirty exists 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2987: left oid 42 (ObjNum 1359 snap 378 seq_num 1359) 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2988: finishing write tid 2 to smithi06431743-8 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2988: finishing write tid 3 to smithi06431743-8 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2988: finishing write tid 4 to smithi06431743-8 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2988: oid 8 updating version 0 to 779 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2988: oid 8 updating version 779 to 780 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2988: oid 8 updating version 780 to 781 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 781 (ObjNum 1360 snap 378 seq_num 1360) dirty exists 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2988: left oid 8 (ObjNum 1360 snap 378 seq_num 1360) 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2989: finishing write tid 1 to smithi06431743-14 2024-03-04T06:20:30.658 INFO:tasks.rados.rados.0.smithi064.stdout:2989: finishing write tid 2 to smithi06431743-14 2024-03-04T06:20:30.659 INFO:tasks.rados.rados.0.smithi064.stdout:2989: finishing write tid 3 to smithi06431743-14 2024-03-04T06:20:30.659 INFO:tasks.rados.rados.0.smithi064.stdout:2989: finishing write tid 4 to smithi06431743-14 2024-03-04T06:20:30.659 INFO:tasks.rados.rados.0.smithi064.stdout:2989: finishing write tid 5 to smithi06431743-14 2024-03-04T06:20:30.659 INFO:tasks.rados.rados.0.smithi064.stdout:2989: oid 14 updating version 0 to 1011 2024-03-04T06:20:30.659 INFO:tasks.rados.rados.0.smithi064.stdout:2989: oid 14 updating version 1011 to 1014 2024-03-04T06:20:30.659 INFO:tasks.rados.rados.0.smithi064.stdout:2989: oid 14 version 1014 is already newer than 1014 2024-03-04T06:20:30.659 INFO:tasks.rados.rados.0.smithi064.stdout:2989: oid 14 version 1014 is already newer than 1012 2024-03-04T06:20:30.659 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 1014 (ObjNum 1361 snap 378 seq_num 1361) dirty exists 2024-03-04T06:20:30.659 INFO:tasks.rados.rados.0.smithi064.stdout:2989: left oid 14 (ObjNum 1361 snap 378 seq_num 1361) 2024-03-04T06:20:31.313 INFO:tasks.rados.rados.0.smithi064.stdout:2984: done (7 left) 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2985: done (6 left) 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2986: done (5 left) 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2987: done (4 left) 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2988: done (3 left) 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2989: done (2 left) 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2990: done (1 left) 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2991: done (0 left) 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2992: read oid 10 snap -1 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2992: expect (ObjNum 1326 snap 368 seq_num 1326) 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2993: read oid 38 snap -1 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2993: expect (ObjNum 1337 snap 370 seq_num 1337) 2024-03-04T06:20:31.314 INFO:tasks.rados.rados.0.smithi064.stdout:2994: write oid 50 current snap is 379 2024-03-04T06:20:31.315 INFO:tasks.rados.rados.0.smithi064.stdout:2994: seq_num 1362 ranges {491134=516325,1734122=666468} 2024-03-04T06:20:31.318 INFO:tasks.rados.rados.0.smithi064.stdout:2994: writing smithi06431743-50 from 491134 to 1007459 tid 1 2024-03-04T06:20:31.326 INFO:tasks.rados.rados.0.smithi064.stdout:2994: writing smithi06431743-50 from 1734122 to 2400590 tid 2 2024-03-04T06:20:31.326 INFO:tasks.rados.rados.0.smithi064.stdout:2995: read oid 27 snap -1 2024-03-04T06:20:31.326 INFO:tasks.rados.rados.0.smithi064.stdout:2995: expect deleted 2024-03-04T06:20:31.326 INFO:tasks.rados.rados.0.smithi064.stdout:2996: read oid 43 snap -1 2024-03-04T06:20:31.327 INFO:tasks.rados.rados.0.smithi064.stdout:2996: expect deleted 2024-03-04T06:20:31.327 INFO:tasks.rados.rados.0.smithi064.stdout:2997: snap_remove snap 364 2024-03-04T06:20:31.902 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:32.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:32.013+0000 7f9af7b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:32.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:32.017+0000 7f9af7b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:32.314 INFO:tasks.rados.rados.0.smithi064.stdout:2997: done (5 left) 2024-03-04T06:20:32.314 INFO:tasks.rados.rados.0.smithi064.stdout:2998: write (excl) oid 11 current snap is 379 2024-03-04T06:20:32.314 INFO:tasks.rados.rados.0.smithi064.stdout:2998: seq_num 1363 ranges {507941=591574,1676251=526479,2746598=161343} 2024-03-04T06:20:32.320 INFO:tasks.rados.rados.0.smithi064.stdout:2998: writing smithi06431743-11 from 507941 to 1099515 tid 1 2024-03-04T06:20:32.326 INFO:tasks.rados.rados.0.smithi064.stdout:2998: writing smithi06431743-11 from 1676251 to 2202730 tid 2 2024-03-04T06:20:32.330 INFO:tasks.rados.rados.0.smithi064.stdout:2998: writing smithi06431743-11 from 2746598 to 2907941 tid 3 2024-03-04T06:20:32.330 INFO:tasks.rados.rados.0.smithi064.stdout:2999: write (excl) oid 32 current snap is 379 2024-03-04T06:20:32.331 INFO:tasks.rados.rados.0.smithi064.stdout:2999: seq_num 1364 ranges {524748=666823,1734733=670137} 2024-03-04T06:20:32.339 INFO:tasks.rados.rados.0.smithi064.stdout:2999: writing smithi06431743-32 from 524748 to 1191571 tid 1 2024-03-04T06:20:32.346 INFO:tasks.rados.rados.0.smithi064.stdout:2999: writing smithi06431743-32 from 1734733 to 2404870 tid 2 2024-03-04T06:20:32.346 INFO:tasks.rados.rados.0.smithi064.stdout:3000: read oid 44 snap -1 2024-03-04T06:20:32.347 INFO:tasks.rados.rados.0.smithi064.stdout:3000: expect deleted 2024-03-04T06:20:32.347 INFO:tasks.rados.rados.0.smithi064.stdout:3001: read oid 8 snap 372 2024-03-04T06:20:32.347 INFO:tasks.rados.rados.0.smithi064.stdout:3001: expect (ObjNum 1309 snap 360 seq_num 1309) 2024-03-04T06:20:32.347 INFO:tasks.rados.rados.0.smithi064.stdout:3002: write oid 18 current snap is 379 2024-03-04T06:20:32.347 INFO:tasks.rados.rados.0.smithi064.stdout:2992: expect (ObjNum 1326 snap 368 seq_num 1326) 2024-03-04T06:20:32.542 INFO:tasks.rados.rados.0.smithi064.stdout:2993: expect (ObjNum 1337 snap 370 seq_num 1337) 2024-03-04T06:20:32.634 INFO:tasks.rados.rados.0.smithi064.stdout:2994: finishing write tid 1 to smithi06431743-50 2024-03-04T06:20:32.634 INFO:tasks.rados.rados.0.smithi064.stdout:2994: finishing write tid 2 to smithi06431743-50 2024-03-04T06:20:32.634 INFO:tasks.rados.rados.0.smithi064.stdout:3002: seq_num 1365 ranges {541555=742072,1793215=530148,2801444=140111} 2024-03-04T06:20:32.640 INFO:tasks.rados.rados.0.smithi064.stdout:3002: writing smithi06431743-18 from 541555 to 1283627 tid 1 2024-03-04T06:20:32.646 INFO:tasks.rados.rados.0.smithi064.stdout:3002: writing smithi06431743-18 from 1793215 to 2323363 tid 2 2024-03-04T06:20:32.652 INFO:tasks.rados.rados.0.smithi064.stdout:3002: writing smithi06431743-18 from 2801444 to 2941555 tid 3 2024-03-04T06:20:32.652 INFO:tasks.rados.rados.0.smithi064.stdout:2992: done (9 left) 2024-03-04T06:20:32.652 INFO:tasks.rados.rados.0.smithi064.stdout:2993: done (8 left) 2024-03-04T06:20:32.652 INFO:tasks.rados.rados.0.smithi064.stdout:2995: done (7 left) 2024-03-04T06:20:32.652 INFO:tasks.rados.rados.0.smithi064.stdout:2996: done (6 left) 2024-03-04T06:20:32.653 INFO:tasks.rados.rados.0.smithi064.stdout:3003: snap_create 2024-03-04T06:20:32.653 INFO:tasks.rados.rados.0.smithi064.stdout:2994: finishing write tid 3 to smithi06431743-50 2024-03-04T06:20:32.653 INFO:tasks.rados.rados.0.smithi064.stdout:2994: finishing write tid 4 to smithi06431743-50 2024-03-04T06:20:32.653 INFO:tasks.rados.rados.0.smithi064.stdout:2994: oid 50 updating version 0 to 973 2024-03-04T06:20:32.653 INFO:tasks.rados.rados.0.smithi064.stdout:2994: oid 50 version 973 is already newer than 973 2024-03-04T06:20:32.653 INFO:tasks.rados.rados.0.smithi064.stdout:2994: oid 50 version 973 is already newer than 971 2024-03-04T06:20:32.653 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 973 (ObjNum 1362 snap 379 seq_num 1362) dirty exists 2024-03-04T06:20:32.653 INFO:tasks.rados.rados.0.smithi064.stdout:2994: left oid 50 (ObjNum 1362 snap 379 seq_num 1362) 2024-03-04T06:20:32.653 INFO:tasks.rados.rados.0.smithi064.stdout:3001: expect (ObjNum 1309 snap 360 seq_num 1309) 2024-03-04T06:20:32.784 INFO:tasks.rados.rados.0.smithi064.stdout:2999: finishing write tid 1 to smithi06431743-32 2024-03-04T06:20:32.784 INFO:tasks.rados.rados.0.smithi064.stdout:2998: finishing write tid 1 to smithi06431743-11 2024-03-04T06:20:32.784 INFO:tasks.rados.rados.0.smithi064.stdout:2999: finishing write tid 2 to smithi06431743-32 2024-03-04T06:20:32.785 INFO:tasks.rados.rados.0.smithi064.stdout:2999: finishing write tid 3 to smithi06431743-32 2024-03-04T06:20:32.785 INFO:tasks.rados.rados.0.smithi064.stdout:2999: finishing write tid 4 to smithi06431743-32 2024-03-04T06:20:32.785 INFO:tasks.rados.rados.0.smithi064.stdout:2999: oid 32 updating version 0 to 1337 2024-03-04T06:20:32.785 INFO:tasks.rados.rados.0.smithi064.stdout:2999: oid 32 version 1337 is already newer than 1335 2024-03-04T06:20:32.785 INFO:tasks.rados.rados.0.smithi064.stdout:2999: oid 32 version 1337 is already newer than 1336 2024-03-04T06:20:32.785 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1337 (ObjNum 1364 snap 379 seq_num 1364) dirty exists 2024-03-04T06:20:32.785 INFO:tasks.rados.rados.0.smithi064.stdout:2999: left oid 32 (ObjNum 1364 snap 379 seq_num 1364) 2024-03-04T06:20:32.785 INFO:tasks.rados.rados.0.smithi064.stdout:2998: finishing write tid 2 to smithi06431743-11 2024-03-04T06:20:32.785 INFO:tasks.rados.rados.0.smithi064.stdout:2998: finishing write tid 3 to smithi06431743-11 2024-03-04T06:20:32.786 INFO:tasks.rados.rados.0.smithi064.stdout:2998: finishing write tid 4 to smithi06431743-11 2024-03-04T06:20:32.786 INFO:tasks.rados.rados.0.smithi064.stdout:2998: finishing write tid 5 to smithi06431743-11 2024-03-04T06:20:32.786 INFO:tasks.rados.rados.0.smithi064.stdout:2998: oid 11 updating version 0 to 798 2024-03-04T06:20:32.786 INFO:tasks.rados.rados.0.smithi064.stdout:2998: oid 11 updating version 798 to 800 2024-03-04T06:20:32.786 INFO:tasks.rados.rados.0.smithi064.stdout:2998: oid 11 version 800 is already newer than 797 2024-03-04T06:20:32.786 INFO:tasks.rados.rados.0.smithi064.stdout:2998: oid 11 version 800 is already newer than 800 2024-03-04T06:20:32.786 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 800 (ObjNum 1363 snap 379 seq_num 1363) dirty exists 2024-03-04T06:20:32.786 INFO:tasks.rados.rados.0.smithi064.stdout:2998: left oid 11 (ObjNum 1363 snap 379 seq_num 1363) 2024-03-04T06:20:32.786 INFO:tasks.rados.rados.0.smithi064.stdout:3002: finishing write tid 1 to smithi06431743-18 2024-03-04T06:20:32.786 INFO:tasks.rados.rados.0.smithi064.stdout:3002: finishing write tid 2 to smithi06431743-18 2024-03-04T06:20:32.787 INFO:tasks.rados.rados.0.smithi064.stdout:3002: finishing write tid 3 to smithi06431743-18 2024-03-04T06:20:32.787 INFO:tasks.rados.rados.0.smithi064.stdout:3002: finishing write tid 4 to smithi06431743-18 2024-03-04T06:20:32.787 INFO:tasks.rados.rados.0.smithi064.stdout:3002: finishing write tid 5 to smithi06431743-18 2024-03-04T06:20:32.787 INFO:tasks.rados.rados.0.smithi064.stdout:3002: oid 18 updating version 0 to 599 2024-03-04T06:20:32.787 INFO:tasks.rados.rados.0.smithi064.stdout:3002: oid 18 version 599 is already newer than 596 2024-03-04T06:20:32.787 INFO:tasks.rados.rados.0.smithi064.stdout:3002: oid 18 version 599 is already newer than 597 2024-03-04T06:20:32.787 INFO:tasks.rados.rados.0.smithi064.stdout:3002: oid 18 version 599 is already newer than 598 2024-03-04T06:20:32.787 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 599 (ObjNum 1365 snap 379 seq_num 1365) dirty exists 2024-03-04T06:20:32.787 INFO:tasks.rados.rados.0.smithi064.stdout:3002: left oid 18 (ObjNum 1365 snap 379 seq_num 1365) 2024-03-04T06:20:33.319 INFO:tasks.rados.rados.0.smithi064.stdout:2994: done (6 left) 2024-03-04T06:20:33.319 INFO:tasks.rados.rados.0.smithi064.stdout:2998: done (5 left) 2024-03-04T06:20:33.319 INFO:tasks.rados.rados.0.smithi064.stdout:2999: done (4 left) 2024-03-04T06:20:33.319 INFO:tasks.rados.rados.0.smithi064.stdout:3000: done (3 left) 2024-03-04T06:20:33.320 INFO:tasks.rados.rados.0.smithi064.stdout:3001: done (2 left) 2024-03-04T06:20:33.320 INFO:tasks.rados.rados.0.smithi064.stdout:3002: done (1 left) 2024-03-04T06:20:33.320 INFO:tasks.rados.rados.0.smithi064.stdout:3003: done (0 left) 2024-03-04T06:20:33.320 INFO:tasks.rados.rados.0.smithi064.stdout:3004: write (excl) oid 48 current snap is 380 2024-03-04T06:20:33.320 INFO:tasks.rados.rados.0.smithi064.stdout:3004: seq_num 1366 ranges {558362=417321,1451697=673806,2712514=245848} 2024-03-04T06:20:33.324 INFO:tasks.rados.rados.0.smithi064.stdout:3004: writing smithi06431743-48 from 558362 to 975683 tid 1 2024-03-04T06:20:33.330 INFO:tasks.rados.rados.0.smithi064.stdout:3004: writing smithi06431743-48 from 1451697 to 2125503 tid 2 2024-03-04T06:20:33.335 INFO:tasks.rados.rados.0.smithi064.stdout:3004: writing smithi06431743-48 from 2712514 to 2958362 tid 3 2024-03-04T06:20:33.335 INFO:tasks.rados.rados.0.smithi064.stdout:3005: read oid 21 snap 376 2024-03-04T06:20:33.336 INFO:tasks.rados.rados.0.smithi064.stdout:3005: expect deleted 2024-03-04T06:20:33.336 INFO:tasks.rados.rados.0.smithi064.stdout:3006: snap_create 2024-03-04T06:20:33.375 INFO:tasks.rados.rados.0.smithi064.stdout:3004: finishing write tid 1 to smithi06431743-48 2024-03-04T06:20:33.388 INFO:tasks.rados.rados.0.smithi064.stdout:3004: finishing write tid 2 to smithi06431743-48 2024-03-04T06:20:33.400 INFO:tasks.rados.rados.0.smithi064.stdout:3004: finishing write tid 3 to smithi06431743-48 2024-03-04T06:20:33.406 INFO:tasks.rados.rados.0.smithi064.stdout:3004: finishing write tid 4 to smithi06431743-48 2024-03-04T06:20:33.408 INFO:tasks.rados.rados.0.smithi064.stdout:3004: finishing write tid 5 to smithi06431743-48 2024-03-04T06:20:33.408 INFO:tasks.rados.rados.0.smithi064.stdout:3004: oid 48 updating version 0 to 471 2024-03-04T06:20:33.408 INFO:tasks.rados.rados.0.smithi064.stdout:3004: oid 48 version 471 is already newer than 470 2024-03-04T06:20:33.408 INFO:tasks.rados.rados.0.smithi064.stdout:3004: oid 48 version 471 is already newer than 468 2024-03-04T06:20:33.408 INFO:tasks.rados.rados.0.smithi064.stdout:3004: oid 48 version 471 is already newer than 470 2024-03-04T06:20:33.408 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 471 (ObjNum 1366 snap 380 seq_num 1366) dirty exists 2024-03-04T06:20:33.408 INFO:tasks.rados.rados.0.smithi064.stdout:3004: left oid 48 (ObjNum 1366 snap 380 seq_num 1366) 2024-03-04T06:20:33.860 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:20:33.860 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:20:33.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:20:33.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:33.984+0000 7fde2a71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:33.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:33.991+0000 7fde2a71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:34.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:34.188 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:20:34.188 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:20:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:34.191+0000 7f945559d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:34.198 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:34.196+0000 7f945559d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:34.201 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1137,"stamp":"2024-03-04T06:20:33.317474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99493910,"num_objects":1010,"num_object_clones":812,"num_object_copies":3030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":161,"num_read":5484,"num_read_kb":4249959,"num_write":4675,"num_write_kb":1238429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11956,"ondisk_log_size":11956,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":739652,"kb_used_data":526884,"kb_used_omap":0,"kb_used_meta":212736,"kb_avail":376747708,"statfs":{"total":386547056640,"available":385789652992,"internally_reserved":0,"allocated":539529216,"data_stored":639385474,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":217841664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2655869,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":9,"num_read_kb":6,"num_write":27,"num_write_kb":7060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011743"},"pg_stats":[{"pgid":"3.f","version":"757'705","reported_seq":2607,"reported_epoch":759,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:30.342783+0000","last_change":"2024-03-04T06:20:28.357362+0000","last_active":"2024-03-04T06:20:30.342783+0000","last_peered":"2024-03-04T06:20:30.342783+0000","last_clean":"2024-03-04T06:20:30.342783+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:20:30.342783+0000","last_undegraded":"2024-03-04T06:20:30.342783+0000","last_fullsized":"2024-03-04T06:20:30.342783+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":311,"num_read_kb":251965,"num_write":274,"num_write_kb":72554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"756'592","reported_seq":2556,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-03-04T06:20:27.487354+0000","last_change":"2024-03-04T06:20:27.564971+0000","last_active":"2024-03-04T06:20:27.487354+0000","last_peered":"2024-03-04T06:20:27.487354+0000","last_clean":"2024-03-04T06:20:27.487354+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:20:27.487354+0000","last_undegraded":"2024-03-04T06:20:27.487354+0000","last_fullsized":"2024-03-04T06:20:27.487354+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5950305,"num_objects":52,"num_object_clones":37,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":186,"num_read_kb":130657,"num_write":213,"num_write_kb":55628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"759'791","reported_seq":2738,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.445588+0000","last_change":"2024-03-04T06:20:30.366767+0000","last_active":"2024-03-04T06:20:30.445588+0000","last_peered":"2024-03-04T06:20:30.445588+0000","last_clean":"2024-03-04T06:20:30.445588+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:20:30.445588+0000","last_undegraded":"2024-03-04T06:20:30.445588+0000","last_fullsized":"2024-03-04T06:20:30.445588+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10959451,"num_objects":76,"num_object_clones":61,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":446,"num_read_kb":359294,"num_write":335,"num_write_kb":90807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"759'688","reported_seq":2573,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.364383+0000","last_change":"2024-03-04T06:20:30.369039+0000","last_active":"2024-03-04T06:20:30.364383+0000","last_peered":"2024-03-04T06:20:30.364383+0000","last_clean":"2024-03-04T06:20:30.364383+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:20:30.364383+0000","last_undegraded":"2024-03-04T06:20:30.364383+0000","last_fullsized":"2024-03-04T06:20:30.364383+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8522106,"num_objects":53,"num_object_clones":43,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":267,"num_read_kb":202839,"num_write":254,"num_write_kb":71600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"759'279","reported_seq":2392,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.521387+0000","last_change":"2024-03-04T06:20:30.526106+0000","last_active":"2024-03-04T06:20:30.521387+0000","last_peered":"2024-03-04T06:20:30.521387+0000","last_clean":"2024-03-04T06:20:30.521387+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:20:30.521387+0000","last_undegraded":"2024-03-04T06:20:30.521387+0000","last_fullsized":"2024-03-04T06:20:30.521387+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672643,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":65484,"num_write":101,"num_write_kb":27678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"757'820","reported_seq":2658,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.357440+0000","last_change":"2024-03-04T06:20:30.363490+0000","last_active":"2024-03-04T06:20:30.357440+0000","last_peered":"2024-03-04T06:20:30.357440+0000","last_clean":"2024-03-04T06:20:30.357440+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:20:30.357440+0000","last_undegraded":"2024-03-04T06:20:30.357440+0000","last_fullsized":"2024-03-04T06:20:30.357440+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4757041,"num_objects":62,"num_object_clones":53,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":350,"num_read_kb":278962,"num_write":308,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2226,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.344070+0000","last_change":"2024-03-04T06:20:30.354686+0000","last_active":"2024-03-04T06:20:30.344070+0000","last_peered":"2024-03-04T06:20:30.344070+0000","last_clean":"2024-03-04T06:20:30.344070+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:20:30.344070+0000","last_undegraded":"2024-03-04T06:20:30.344070+0000","last_fullsized":"2024-03-04T06:20:30.344070+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"756'446","reported_seq":2480,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.319464+0000","last_change":"2024-03-04T06:20:30.325127+0000","last_active":"2024-03-04T06:20:30.319464+0000","last_peered":"2024-03-04T06:20:30.319464+0000","last_clean":"2024-03-04T06:20:30.319464+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:20:30.319464+0000","last_undegraded":"2024-03-04T06:20:30.319464+0000","last_fullsized":"2024-03-04T06:20:30.319464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403731,"num_objects":35,"num_object_clones":26,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":222,"num_read_kb":204266,"num_write":154,"num_write_kb":41095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2044,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.471194+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:20:30.471194+0000","last_peered":"2024-03-04T06:20:30.471194+0000","last_clean":"2024-03-04T06:20:30.471194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:20:30.471194+0000","last_undegraded":"2024-03-04T06:20:30.471194+0000","last_fullsized":"2024-03-04T06:20:30.471194+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1887,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.326408+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:20:30.326408+0000","last_peered":"2024-03-04T06:20:30.326408+0000","last_clean":"2024-03-04T06:20:30.326408+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:20:30.326408+0000","last_undegraded":"2024-03-04T06:20:30.326408+0000","last_fullsized":"2024-03-04T06:20:30.326408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"756'742","reported_seq":2660,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.325664+0000","last_change":"2024-03-04T06:20:30.329422+0000","last_active":"2024-03-04T06:20:30.325664+0000","last_peered":"2024-03-04T06:20:30.325664+0000","last_clean":"2024-03-04T06:20:30.325664+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:20:30.325664+0000","last_undegraded":"2024-03-04T06:20:30.325664+0000","last_fullsized":"2024-03-04T06:20:30.325664+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7463298,"num_objects":65,"num_object_clones":51,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":352,"num_read_kb":281385,"num_write":292,"num_write_kb":78583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1873,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.338755+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:20:30.338755+0000","last_peered":"2024-03-04T06:20:30.338755+0000","last_clean":"2024-03-04T06:20:30.338755+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:20:30.338755+0000","last_undegraded":"2024-03-04T06:20:30.338755+0000","last_fullsized":"2024-03-04T06:20:30.338755+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"759'464","reported_seq":2538,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.538392+0000","last_change":"2024-03-04T06:20:30.541972+0000","last_active":"2024-03-04T06:20:30.538392+0000","last_peered":"2024-03-04T06:20:30.538392+0000","last_clean":"2024-03-04T06:20:30.538392+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:20:30.538392+0000","last_undegraded":"2024-03-04T06:20:30.538392+0000","last_fullsized":"2024-03-04T06:20:30.538392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6454515,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":268,"num_read_kb":221302,"num_write":170,"num_write_kb":47985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1887,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.318211+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:20:30.318211+0000","last_peered":"2024-03-04T06:20:30.318211+0000","last_clean":"2024-03-04T06:20:30.318211+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:20:30.318211+0000","last_undegraded":"2024-03-04T06:20:30.318211+0000","last_fullsized":"2024-03-04T06:20:30.318211+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"757'1327","reported_seq":3067,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.485418+0000","last_change":"2024-03-04T06:20:30.489910+0000","last_active":"2024-03-04T06:20:30.485418+0000","last_peered":"2024-03-04T06:20:30.485418+0000","last_clean":"2024-03-04T06:20:30.485418+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:20:30.485418+0000","last_undegraded":"2024-03-04T06:20:30.485418+0000","last_fullsized":"2024-03-04T06:20:30.485418+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1327,"log_dups_size":0,"ondisk_log_size":1327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11278751,"num_objects":117,"num_object_clones":93,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":698,"num_read_kb":555711,"num_write":544,"num_write_kb":145500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1873,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.331628+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:20:30.331628+0000","last_peered":"2024-03-04T06:20:30.331628+0000","last_clean":"2024-03-04T06:20:30.331628+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:20:30.331628+0000","last_undegraded":"2024-03-04T06:20:30.331628+0000","last_fullsized":"2024-03-04T06:20:30.331628+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"756'1512","reported_seq":3031,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-03-04T06:20:27.491310+0000","last_change":"2024-03-04T06:20:27.571647+0000","last_active":"2024-03-04T06:20:27.491310+0000","last_peered":"2024-03-04T06:20:27.491310+0000","last_clean":"2024-03-04T06:20:27.491310+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:20:27.491310+0000","last_undegraded":"2024-03-04T06:20:27.491310+0000","last_fullsized":"2024-03-04T06:20:27.491310+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1512,"log_dups_size":0,"ondisk_log_size":1512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15985008,"num_objects":123,"num_object_clones":100,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":548,"num_read_kb":400606,"num_write":569,"num_write_kb":153035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"756'732","reported_seq":2666,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.330460+0000","last_change":"2024-03-04T06:20:30.335336+0000","last_active":"2024-03-04T06:20:30.330460+0000","last_peered":"2024-03-04T06:20:30.330460+0000","last_clean":"2024-03-04T06:20:30.330460+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:20:30.330460+0000","last_undegraded":"2024-03-04T06:20:30.330460+0000","last_fullsized":"2024-03-04T06:20:30.330460+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":732,"log_dups_size":0,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756678,"num_objects":59,"num_object_clones":47,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":393,"num_read_kb":329118,"num_write":273,"num_write_kb":73929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1932,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-03-04T06:20:27.507713+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:20:27.507713+0000","last_peered":"2024-03-04T06:20:27.507713+0000","last_clean":"2024-03-04T06:20:27.507713+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:20:27.507713+0000","last_undegraded":"2024-03-04T06:20:27.507713+0000","last_fullsized":"2024-03-04T06:20:27.507713+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"756'777","reported_seq":2694,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-03-04T06:20:27.595574+0000","last_change":"2024-03-04T06:20:27.610101+0000","last_active":"2024-03-04T06:20:27.595574+0000","last_peered":"2024-03-04T06:20:27.595574+0000","last_clean":"2024-03-04T06:20:27.595574+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:20:27.595574+0000","last_undegraded":"2024-03-04T06:20:27.595574+0000","last_fullsized":"2024-03-04T06:20:27.595574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7026316,"num_objects":71,"num_object_clones":55,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":327,"num_read_kb":236249,"num_write":301,"num_write_kb":81081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1911,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.482491+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:20:30.482491+0000","last_peered":"2024-03-04T06:20:30.482491+0000","last_clean":"2024-03-04T06:20:30.482491+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:20:30.482491+0000","last_undegraded":"2024-03-04T06:20:30.482491+0000","last_fullsized":"2024-03-04T06:20:30.482491+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"759'1014","reported_seq":2799,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.596827+0000","last_change":"2024-03-04T06:20:30.599377+0000","last_active":"2024-03-04T06:20:30.596827+0000","last_peered":"2024-03-04T06:20:30.596827+0000","last_clean":"2024-03-04T06:20:30.596827+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:20:30.596827+0000","last_undegraded":"2024-03-04T06:20:30.596827+0000","last_fullsized":"2024-03-04T06:20:30.596827+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1014,"log_dups_size":0,"ondisk_log_size":1014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12673699,"num_objects":84,"num_object_clones":70,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":422,"num_read_kb":309262,"num_write":394,"num_write_kb":110146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1932,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-03-04T06:20:27.499704+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:20:27.499704+0000","last_peered":"2024-03-04T06:20:27.499704+0000","last_clean":"2024-03-04T06:20:27.499704+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:20:27.499704+0000","last_undegraded":"2024-03-04T06:20:27.499704+0000","last_fullsized":"2024-03-04T06:20:27.499704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"756'970","reported_seq":2830,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-03-04T06:20:27.494926+0000","last_change":"2024-03-04T06:20:27.569072+0000","last_active":"2024-03-04T06:20:27.494926+0000","last_peered":"2024-03-04T06:20:27.494926+0000","last_clean":"2024-03-04T06:20:27.494926+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:20:27.494926+0000","last_undegraded":"2024-03-04T06:20:27.494926+0000","last_fullsized":"2024-03-04T06:20:27.494926+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":80,"num_object_clones":70,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":500,"num_read_kb":422768,"num_write":356,"num_write_kb":97261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1887,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-03-04T06:20:30.324605+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:20:30.324605+0000","last_peered":"2024-03-04T06:20:30.324605+0000","last_clean":"2024-03-04T06:20:30.324605+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:20:30.324605+0000","last_undegraded":"2024-03-04T06:20:30.324605+0000","last_fullsized":"2024-03-04T06:20:30.324605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98903542,"num_objects":1008,"num_object_clones":812,"num_object_copies":3024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":161,"num_read":5381,"num_read_kb":4249868,"num_write":4538,"num_write_kb":1236535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11859,"ondisk_log_size":11859,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706429,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":222212,"kb_used_data":164860,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94149628,"statfs":{"total":96636764160,"available":96409219072,"internally_reserved":0,"allocated":168816640,"data_stored":200039061,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706429,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":173176,"kb_used_data":119984,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94198664,"statfs":{"total":96636764160,"available":96459431936,"internally_reserved":0,"allocated":122863616,"data_stored":141351172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706428,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":168216,"kb_used_data":115024,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94203624,"statfs":{"total":96636764160,"available":96464510976,"internally_reserved":0,"allocated":117784576,"data_stored":139699645,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706428,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":176048,"kb_used_data":127016,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94195792,"statfs":{"total":96636764160,"available":96456491008,"internally_reserved":0,"allocated":130064384,"data_stored":158295596,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:20:34.321 INFO:tasks.rados.rados.0.smithi064.stdout:3004: done (2 left) 2024-03-04T06:20:34.321 INFO:tasks.rados.rados.0.smithi064.stdout:3005: done (1 left) 2024-03-04T06:20:34.321 INFO:tasks.rados.rados.0.smithi064.stdout:3006: done (0 left) 2024-03-04T06:20:34.321 INFO:tasks.rados.rados.0.smithi064.stdout:3007: delete oid 42 current snap is 381 2024-03-04T06:20:34.363 INFO:tasks.rados.rados.0.smithi064.stdout:3007: done (0 left) 2024-03-04T06:20:34.364 INFO:tasks.rados.rados.0.smithi064.stdout:3008: read oid 13 snap -1 2024-03-04T06:20:34.364 INFO:tasks.rados.rados.0.smithi064.stdout:3008: expect (ObjNum 1341 snap 370 seq_num 1341) 2024-03-04T06:20:34.364 INFO:tasks.rados.rados.0.smithi064.stdout:3009: delete oid 33 current snap is 381 2024-03-04T06:20:34.372 INFO:tasks.rados.rados.0.smithi064.stdout:3009: done (1 left) 2024-03-04T06:20:34.373 INFO:tasks.rados.rados.0.smithi064.stdout:3010: snap_create 2024-03-04T06:20:34.383 INFO:tasks.rados.rados.0.smithi064.stdout:3008: expect (ObjNum 1341 snap 370 seq_num 1341) 2024-03-04T06:20:35.325 INFO:tasks.rados.rados.0.smithi064.stdout:3008: done (1 left) 2024-03-04T06:20:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:3010: done (0 left) 2024-03-04T06:20:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:3011: write oid 13 current snap is 382 2024-03-04T06:20:35.326 INFO:tasks.rados.rados.0.smithi064.stdout:3011: seq_num 1369 ranges {608783=759421,2027143=421133} 2024-03-04T06:20:35.332 INFO:tasks.rados.rados.0.smithi064.stdout:3011: writing smithi06431743-13 from 608783 to 1368204 tid 1 2024-03-04T06:20:35.339 INFO:tasks.rados.rados.0.smithi064.stdout:3011: writing smithi06431743-13 from 2027143 to 2448276 tid 2 2024-03-04T06:20:35.339 INFO:tasks.rados.rados.0.smithi064.stdout:3012: snap_remove snap 350 2024-03-04T06:20:36.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:3012: done (1 left) 2024-03-04T06:20:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:3013: write (excl) oid 11 current snap is 382 2024-03-04T06:20:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:3011: finishing write tid 1 to smithi06431743-13 2024-03-04T06:20:36.326 INFO:tasks.rados.rados.0.smithi064.stdout:3013: seq_num 1370 ranges {625590=434670,1685625=681144,2822206=203384} 2024-03-04T06:20:36.332 INFO:tasks.rados.rados.0.smithi064.stdout:3013: writing smithi06431743-11 from 625590 to 1060260 tid 1 2024-03-04T06:20:36.339 INFO:tasks.rados.rados.0.smithi064.stdout:3013: writing smithi06431743-11 from 1685625 to 2366769 tid 2 2024-03-04T06:20:36.343 INFO:tasks.rados.rados.0.smithi064.stdout:3013: writing smithi06431743-11 from 2822206 to 3025590 tid 3 2024-03-04T06:20:36.343 INFO:tasks.rados.rados.0.smithi064.stdout:3014: read oid 9 snap -1 2024-03-04T06:20:36.343 INFO:tasks.rados.rados.0.smithi064.stdout:3014: expect deleted 2024-03-04T06:20:36.344 INFO:tasks.rados.rados.0.smithi064.stdout:3015: write oid 15 current snap is 382 2024-03-04T06:20:36.344 INFO:tasks.rados.rados.0.smithi064.stdout:3011: finishing write tid 2 to smithi06431743-13 2024-03-04T06:20:36.344 INFO:tasks.rados.rados.0.smithi064.stdout:3011: finishing write tid 3 to smithi06431743-13 2024-03-04T06:20:36.344 INFO:tasks.rados.rados.0.smithi064.stdout:3011: finishing write tid 4 to smithi06431743-13 2024-03-04T06:20:36.344 INFO:tasks.rados.rados.0.smithi064.stdout:3011: oid 13 updating version 0 to 1518 2024-03-04T06:20:36.344 INFO:tasks.rados.rados.0.smithi064.stdout:3011: oid 13 updating version 1518 to 1520 2024-03-04T06:20:36.344 INFO:tasks.rados.rados.0.smithi064.stdout:3011: oid 13 version 1520 is already newer than 1520 2024-03-04T06:20:36.344 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1520 (ObjNum 1369 snap 382 seq_num 1369) dirty exists 2024-03-04T06:20:36.344 INFO:tasks.rados.rados.0.smithi064.stdout:3011: left oid 13 (ObjNum 1369 snap 382 seq_num 1369) 2024-03-04T06:20:36.344 INFO:tasks.rados.rados.0.smithi064.stdout:3015: seq_num 1371 ranges {642397=509919,1627754=424802,2733276=309121} 2024-03-04T06:20:36.349 INFO:tasks.rados.rados.0.smithi064.stdout:3015: writing smithi06431743-15 from 642397 to 1152316 tid 1 2024-03-04T06:20:36.354 INFO:tasks.rados.rados.0.smithi064.stdout:3015: writing smithi06431743-15 from 1627754 to 2052556 tid 2 2024-03-04T06:20:36.360 INFO:tasks.rados.rados.0.smithi064.stdout:3015: writing smithi06431743-15 from 2733276 to 3042397 tid 3 2024-03-04T06:20:36.360 INFO:tasks.rados.rados.0.smithi064.stdout:3011: done (3 left) 2024-03-04T06:20:36.360 INFO:tasks.rados.rados.0.smithi064.stdout:3016: delete oid 1 current snap is 382 2024-03-04T06:20:36.402 INFO:tasks.rados.rados.0.smithi064.stdout:3016: done (3 left) 2024-03-04T06:20:36.402 INFO:tasks.rados.rados.0.smithi064.stdout:3017: snap_remove snap 361 2024-03-04T06:20:36.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:36.410+0000 7f54c2d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:36.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:36.416+0000 7f54c2d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:37.333 INFO:tasks.rados.rados.0.smithi064.stdout:3017: done (3 left) 2024-03-04T06:20:37.333 INFO:tasks.rados.rados.0.smithi064.stdout:3018: read oid 10 snap 365 2024-03-04T06:20:37.333 INFO:tasks.rados.rados.0.smithi064.stdout:3018: expect (ObjNum 1265 snap 341 seq_num 1265) 2024-03-04T06:20:37.333 INFO:tasks.rados.rados.0.smithi064.stdout:3019: snap_create 2024-03-04T06:20:37.333 INFO:tasks.rados.rados.0.smithi064.stdout:3015: finishing write tid 1 to smithi06431743-15 2024-03-04T06:20:37.333 INFO:tasks.rados.rados.0.smithi064.stdout:3013: finishing write tid 1 to smithi06431743-11 2024-03-04T06:20:37.333 INFO:tasks.rados.rados.0.smithi064.stdout:3013: finishing write tid 2 to smithi06431743-11 2024-03-04T06:20:37.333 INFO:tasks.rados.rados.0.smithi064.stdout:3015: finishing write tid 2 to smithi06431743-15 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:3013: finishing write tid 3 to smithi06431743-11 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:3015: finishing write tid 3 to smithi06431743-15 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:3013: finishing write tid 4 to smithi06431743-11 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:3013: finishing write tid 5 to smithi06431743-11 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:3013: oid 11 updating version 0 to 806 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:3013: oid 11 updating version 806 to 809 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:3013: oid 11 version 809 is already newer than 808 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:3013: oid 11 version 809 is already newer than 808 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 809 (ObjNum 1370 snap 382 seq_num 1370) dirty exists 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:3013: left oid 11 (ObjNum 1370 snap 382 seq_num 1370) 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:3015: finishing write tid 4 to smithi06431743-15 2024-03-04T06:20:37.334 INFO:tasks.rados.rados.0.smithi064.stdout:3015: finishing write tid 5 to smithi06431743-15 2024-03-04T06:20:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:3015: oid 15 updating version 0 to 743 2024-03-04T06:20:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:3015: oid 15 updating version 743 to 744 2024-03-04T06:20:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:3015: oid 15 updating version 744 to 745 2024-03-04T06:20:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:3015: oid 15 version 745 is already newer than 742 2024-03-04T06:20:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 745 (ObjNum 1371 snap 382 seq_num 1371) dirty exists 2024-03-04T06:20:37.335 INFO:tasks.rados.rados.0.smithi064.stdout:3015: left oid 15 (ObjNum 1371 snap 382 seq_num 1371) 2024-03-04T06:20:37.410 INFO:tasks.rados.rados.0.smithi064.stdout:3018: expect (ObjNum 1265 snap 341 seq_num 1265) 2024-03-04T06:20:38.305 INFO:tasks.rados.rados.0.smithi064.stdout:3013: done (4 left) 2024-03-04T06:20:38.305 INFO:tasks.rados.rados.0.smithi064.stdout:3014: done (3 left) 2024-03-04T06:20:38.305 INFO:tasks.rados.rados.0.smithi064.stdout:3015: done (2 left) 2024-03-04T06:20:38.305 INFO:tasks.rados.rados.0.smithi064.stdout:3018: done (1 left) 2024-03-04T06:20:38.305 INFO:tasks.rados.rados.0.smithi064.stdout:3019: done (0 left) 2024-03-04T06:20:38.306 INFO:tasks.rados.rados.0.smithi064.stdout:3020: read oid 3 snap -1 2024-03-04T06:20:38.306 INFO:tasks.rados.rados.0.smithi064.stdout:3020: expect (ObjNum 1331 snap 368 seq_num 1331) 2024-03-04T06:20:38.306 INFO:tasks.rados.rados.0.smithi064.stdout:3021: read oid 10 snap 345 2024-03-04T06:20:38.306 INFO:tasks.rados.rados.0.smithi064.stdout:3021: expect (ObjNum 1265 snap 341 seq_num 1265) 2024-03-04T06:20:38.306 INFO:tasks.rados.rados.0.smithi064.stdout:3022: read oid 32 snap -1 2024-03-04T06:20:38.306 INFO:tasks.rados.rados.0.smithi064.stdout:3022: expect (ObjNum 1364 snap 379 seq_num 1364) 2024-03-04T06:20:38.306 INFO:tasks.rados.rados.0.smithi064.stdout:3023: snap_create 2024-03-04T06:20:38.357 INFO:tasks.rados.rados.0.smithi064.stdout:3021: expect (ObjNum 1265 snap 341 seq_num 1265) 2024-03-04T06:20:38.551 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:38.585 INFO:tasks.rados.rados.0.smithi064.stdout:3020: expect (ObjNum 1331 snap 368 seq_num 1331) 2024-03-04T06:20:38.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:38.656+0000 7f7f7386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:38.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:38.660+0000 7f7f7386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:38.710 INFO:tasks.rados.rados.0.smithi064.stdout:3022: expect (ObjNum 1364 snap 379 seq_num 1364) 2024-03-04T06:20:39.203 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:20:39.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:20:39.305 INFO:tasks.rados.rados.0.smithi064.stdout:3020: done (3 left) 2024-03-04T06:20:39.305 INFO:tasks.rados.rados.0.smithi064.stdout:3021: done (2 left) 2024-03-04T06:20:39.305 INFO:tasks.rados.rados.0.smithi064.stdout:3022: done (1 left) 2024-03-04T06:20:39.305 INFO:tasks.rados.rados.0.smithi064.stdout:3023: done (0 left) 2024-03-04T06:20:39.305 INFO:tasks.rados.rados.0.smithi064.stdout:3024: read oid 28 snap -1 2024-03-04T06:20:39.305 INFO:tasks.rados.rados.0.smithi064.stdout:3024: expect (ObjNum 1338 snap 370 seq_num 1338) 2024-03-04T06:20:39.305 INFO:tasks.rados.rados.0.smithi064.stdout:3025: write oid 37 current snap is 384 2024-03-04T06:20:39.305 INFO:tasks.rados.rados.0.smithi064.stdout:3025: seq_num 1373 ranges {676011=660417,1744718=428471,2788122=287889} 2024-03-04T06:20:39.314 INFO:tasks.rados.rados.0.smithi064.stdout:3025: writing smithi06431743-37 from 676011 to 1336428 tid 1 2024-03-04T06:20:39.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:39.310+0000 7f8f45d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:39.319 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:39.319+0000 7f8f45d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:39.320 INFO:tasks.rados.rados.0.smithi064.stdout:3025: writing smithi06431743-37 from 1744718 to 2173189 tid 2 2024-03-04T06:20:39.326 INFO:tasks.rados.rados.0.smithi064.stdout:3025: writing smithi06431743-37 from 2788122 to 3076011 tid 3 2024-03-04T06:20:39.326 INFO:tasks.rados.rados.0.smithi064.stdout:3026: snap_remove snap 328 2024-03-04T06:20:40.303 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:20:40.306 INFO:tasks.rados.rados.0.smithi064.stdout:3026: done (2 left) 2024-03-04T06:20:40.306 INFO:tasks.rados.rados.0.smithi064.stdout:3027: delete oid 5 current snap is 384 2024-03-04T06:20:40.306 INFO:tasks.rados.rados.0.smithi064.stdout:3024: expect (ObjNum 1338 snap 370 seq_num 1338) 2024-03-04T06:20:40.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:20:40.452 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:40.447+0000 7fda1e914640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:40.455 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:40.455+0000 7fda1e914640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:40.475 INFO:tasks.rados.rados.0.smithi064.stdout:3025: finishing write tid 1 to smithi06431743-37 2024-03-04T06:20:40.475 INFO:tasks.rados.rados.0.smithi064.stdout:3025: finishing write tid 2 to smithi06431743-37 2024-03-04T06:20:40.475 INFO:tasks.rados.rados.0.smithi064.stdout:3024: done (2 left) 2024-03-04T06:20:40.475 INFO:tasks.rados.rados.0.smithi064.stdout:3027: done (1 left) 2024-03-04T06:20:40.475 INFO:tasks.rados.rados.0.smithi064.stdout:3028: write (excl) oid 13 current snap is 384 2024-03-04T06:20:40.475 INFO:tasks.rados.rados.0.smithi064.stdout:3025: finishing write tid 3 to smithi06431743-37 2024-03-04T06:20:40.475 INFO:tasks.rados.rados.0.smithi064.stdout:3028: seq_num 1375 ranges {709625=410915,1745329=432140,2726615=383010} 2024-03-04T06:20:40.482 INFO:tasks.rados.rados.0.smithi064.stdout:3028: writing smithi06431743-13 from 709625 to 1120540 tid 1 2024-03-04T06:20:40.491 INFO:tasks.rados.rados.0.smithi064.stdout:3028: writing smithi06431743-13 from 1745329 to 2177469 tid 2 2024-03-04T06:20:40.498 INFO:tasks.rados.rados.0.smithi064.stdout:3028: writing smithi06431743-13 from 2726615 to 3109625 tid 3 2024-03-04T06:20:40.498 INFO:tasks.rados.rados.0.smithi064.stdout:3029: snap_remove snap 371 2024-03-04T06:20:40.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:40.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:40.842+0000 7faa6319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:40.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:40.846+0000 7faa6319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:41.337 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:20:41.339 INFO:tasks.rados.rados.0.smithi064.stdout:3029: done (2 left) 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3030: snap_create 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3025: finishing write tid 4 to smithi06431743-37 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3025: finishing write tid 5 to smithi06431743-37 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3025: oid 37 updating version 0 to 606 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3025: oid 37 version 606 is already newer than 603 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3025: oid 37 version 606 is already newer than 604 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3025: oid 37 version 606 is already newer than 606 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 606 (ObjNum 1373 snap 384 seq_num 1373) dirty exists 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3025: left oid 37 (ObjNum 1373 snap 384 seq_num 1373) 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3028: finishing write tid 1 to smithi06431743-13 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3028: finishing write tid 2 to smithi06431743-13 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3028: finishing write tid 3 to smithi06431743-13 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3028: finishing write tid 4 to smithi06431743-13 2024-03-04T06:20:41.340 INFO:tasks.rados.rados.0.smithi064.stdout:3028: finishing write tid 5 to smithi06431743-13 2024-03-04T06:20:41.341 INFO:tasks.rados.rados.0.smithi064.stdout:3028: oid 13 updating version 0 to 1539 2024-03-04T06:20:41.341 INFO:tasks.rados.rados.0.smithi064.stdout:3028: oid 13 updating version 1539 to 1540 2024-03-04T06:20:41.341 INFO:tasks.rados.rados.0.smithi064.stdout:3028: oid 13 updating version 1540 to 1542 2024-03-04T06:20:41.341 INFO:tasks.rados.rados.0.smithi064.stdout:3028: oid 13 version 1542 is already newer than 1542 2024-03-04T06:20:41.341 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1542 (ObjNum 1375 snap 384 seq_num 1375) dirty exists 2024-03-04T06:20:41.341 INFO:tasks.rados.rados.0.smithi064.stdout:3028: left oid 13 (ObjNum 1375 snap 384 seq_num 1375) 2024-03-04T06:20:41.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:20:41.472 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:41.467+0000 7effe5213640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:41.475 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:41.475+0000 7effe5213640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:42.338 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:20:42.341 INFO:tasks.rados.rados.0.smithi064.stdout:3025: done (2 left) 2024-03-04T06:20:42.341 INFO:tasks.rados.rados.0.smithi064.stdout:3028: done (1 left) 2024-03-04T06:20:42.342 INFO:tasks.rados.rados.0.smithi064.stdout:3030: done (0 left) 2024-03-04T06:20:42.342 INFO:tasks.rados.rados.0.smithi064.stdout:3031: read oid 1 snap 375 2024-03-04T06:20:42.342 INFO:tasks.rados.rados.0.smithi064.stdout:3031: expect deleted 2024-03-04T06:20:42.342 INFO:tasks.rados.rados.0.smithi064.stdout:3032: read oid 37 snap -1 2024-03-04T06:20:42.342 INFO:tasks.rados.rados.0.smithi064.stdout:3032: expect (ObjNum 1373 snap 384 seq_num 1373) 2024-03-04T06:20:42.342 INFO:tasks.rados.rados.0.smithi064.stdout:3033: write oid 17 current snap is 385 2024-03-04T06:20:42.342 INFO:tasks.rados.rados.0.smithi064.stdout:3033: seq_num 1376 ranges {726432=486164,1803811=692151} 2024-03-04T06:20:42.347 INFO:tasks.rados.rados.0.smithi064.stdout:3033: writing smithi06431743-17 from 726432 to 1212596 tid 1 2024-03-04T06:20:42.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:20:42.355 INFO:tasks.rados.rados.0.smithi064.stdout:3033: writing smithi06431743-17 from 1803811 to 2495962 tid 2 2024-03-04T06:20:42.356 INFO:tasks.rados.rados.0.smithi064.stdout:3034: write (excl) oid 19 current snap is 385 2024-03-04T06:20:42.356 INFO:tasks.rados.rados.0.smithi064.stdout:3034: seq_num 1377 ranges {743239=677766,1978646=435809,2897814=245425} 2024-03-04T06:20:42.362 INFO:tasks.rados.rados.0.smithi064.stdout:3034: writing smithi06431743-19 from 743239 to 1421005 tid 1 2024-03-04T06:20:42.369 INFO:tasks.rados.rados.0.smithi064.stdout:3034: writing smithi06431743-19 from 1978646 to 2414455 tid 2 2024-03-04T06:20:42.373 INFO:tasks.rados.rados.0.smithi064.stdout:3034: writing smithi06431743-19 from 2897814 to 3143239 tid 3 2024-03-04T06:20:42.373 INFO:tasks.rados.rados.0.smithi064.stdout:3035: write (excl) oid 13 current snap is 385 2024-03-04T06:20:42.373 INFO:tasks.rados.rados.0.smithi064.stdout:3035: seq_num 1378 ranges {760046=753015,2037128=695820} 2024-03-04T06:20:42.383 INFO:tasks.rados.rados.0.smithi064.stdout:3035: writing smithi06431743-13 from 760046 to 1513061 tid 1 2024-03-04T06:20:42.391 INFO:tasks.rados.rados.0.smithi064.stdout:3035: writing smithi06431743-13 from 2037128 to 2732948 tid 2 2024-03-04T06:20:42.392 INFO:tasks.rados.rados.0.smithi064.stdout:3036: read oid 29 snap 375 2024-03-04T06:20:42.392 INFO:tasks.rados.rados.0.smithi064.stdout:3036: expect (ObjNum 1297 snap 355 seq_num 1297) 2024-03-04T06:20:42.392 INFO:tasks.rados.rados.0.smithi064.stdout:3037: write oid 27 current snap is 385 2024-03-04T06:20:42.392 INFO:tasks.rados.rados.0.smithi064.stdout:3037: seq_num 1379 ranges {776853=428264,1979257=439478,2836307=340546} 2024-03-04T06:20:42.398 INFO:tasks.rados.rados.0.smithi064.stdout:3037: writing smithi06431743-27 from 776853 to 1205117 tid 1 2024-03-04T06:20:42.405 INFO:tasks.rados.rados.0.smithi064.stdout:3037: writing smithi06431743-27 from 1979257 to 2418735 tid 2 2024-03-04T06:20:42.409 INFO:tasks.rados.rados.0.smithi064.stdout:3037: writing smithi06431743-27 from 2836307 to 3176853 tid 3 2024-03-04T06:20:42.409 INFO:tasks.rados.rados.0.smithi064.stdout:3038: delete oid 10 current snap is 385 2024-03-04T06:20:42.410 INFO:tasks.rados.rados.0.smithi064.stdout:3033: finishing write tid 1 to smithi06431743-17 2024-03-04T06:20:42.415 INFO:tasks.rados.rados.0.smithi064.stdout:3034: finishing write tid 1 to smithi06431743-19 2024-03-04T06:20:42.428 INFO:tasks.rados.rados.0.smithi064.stdout:3032: expect (ObjNum 1373 snap 384 seq_num 1373) 2024-03-04T06:20:42.484 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:42.478+0000 7f9c2f11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:42.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:42.487+0000 7f9c2f11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:42.602 INFO:tasks.rados.rados.0.smithi064.stdout:3033: finishing write tid 2 to smithi06431743-17 2024-03-04T06:20:42.602 INFO:tasks.rados.rados.0.smithi064.stdout:3034: finishing write tid 2 to smithi06431743-19 2024-03-04T06:20:42.602 INFO:tasks.rados.rados.0.smithi064.stdout:3033: finishing write tid 3 to smithi06431743-17 2024-03-04T06:20:42.602 INFO:tasks.rados.rados.0.smithi064.stdout:3031: done (7 left) 2024-03-04T06:20:42.602 INFO:tasks.rados.rados.0.smithi064.stdout:3032: done (6 left) 2024-03-04T06:20:42.602 INFO:tasks.rados.rados.0.smithi064.stdout:3038: done (5 left) 2024-03-04T06:20:42.602 INFO:tasks.rados.rados.0.smithi064.stdout:3039: read oid 14 snap 362 2024-03-04T06:20:42.602 INFO:tasks.rados.rados.0.smithi064.stdout:3039: expect (ObjNum 1226 snap 330 seq_num 1226) 2024-03-04T06:20:42.602 INFO:tasks.rados.rados.0.smithi064.stdout:3040: delete oid 39 current snap is 385 2024-03-04T06:20:42.602 INFO:tasks.rados.rados.0.smithi064.stdout:3033: finishing write tid 4 to smithi06431743-17 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:3033: oid 17 updating version 0 to 463 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:3033: oid 17 updating version 463 to 465 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:3033: oid 17 version 465 is already newer than 465 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 465 (ObjNum 1376 snap 385 seq_num 1376) dirty exists 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:3033: left oid 17 (ObjNum 1376 snap 385 seq_num 1376) 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:3037: finishing write tid 1 to smithi06431743-27 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:3034: finishing write tid 3 to smithi06431743-19 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:3037: finishing write tid 2 to smithi06431743-27 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:3034: finishing write tid 4 to smithi06431743-19 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:3034: finishing write tid 5 to smithi06431743-19 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:3034: oid 19 updating version 0 to 1548 2024-03-04T06:20:42.603 INFO:tasks.rados.rados.0.smithi064.stdout:3034: oid 19 updating version 1548 to 1549 2024-03-04T06:20:42.604 INFO:tasks.rados.rados.0.smithi064.stdout:3034: oid 19 version 1549 is already newer than 1548 2024-03-04T06:20:42.604 INFO:tasks.rados.rados.0.smithi064.stdout:3034: oid 19 version 1549 is already newer than 1546 2024-03-04T06:20:42.604 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 1549 (ObjNum 1377 snap 385 seq_num 1377) dirty exists 2024-03-04T06:20:42.604 INFO:tasks.rados.rados.0.smithi064.stdout:3034: left oid 19 (ObjNum 1377 snap 385 seq_num 1377) 2024-03-04T06:20:42.604 INFO:tasks.rados.rados.0.smithi064.stdout:3037: finishing write tid 3 to smithi06431743-27 2024-03-04T06:20:42.604 INFO:tasks.rados.rados.0.smithi064.stdout:3037: finishing write tid 4 to smithi06431743-27 2024-03-04T06:20:42.604 INFO:tasks.rados.rados.0.smithi064.stdout:3037: finishing write tid 5 to smithi06431743-27 2024-03-04T06:20:42.604 INFO:tasks.rados.rados.0.smithi064.stdout:3037: oid 27 updating version 0 to 617 2024-03-04T06:20:42.604 INFO:tasks.rados.rados.0.smithi064.stdout:3037: oid 27 version 617 is already newer than 615 2024-03-04T06:20:42.604 INFO:tasks.rados.rados.0.smithi064.stdout:3037: oid 27 updating version 617 to 618 2024-03-04T06:20:42.605 INFO:tasks.rados.rados.0.smithi064.stdout:3037: oid 27 version 618 is already newer than 617 2024-03-04T06:20:42.605 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 618 (ObjNum 1379 snap 385 seq_num 1379) dirty exists 2024-03-04T06:20:42.605 INFO:tasks.rados.rados.0.smithi064.stdout:3037: left oid 27 (ObjNum 1379 snap 385 seq_num 1379) 2024-03-04T06:20:42.605 INFO:tasks.rados.rados.0.smithi064.stdout:3035: finishing write tid 1 to smithi06431743-13 2024-03-04T06:20:42.605 INFO:tasks.rados.rados.0.smithi064.stdout:3035: finishing write tid 2 to smithi06431743-13 2024-03-04T06:20:42.605 INFO:tasks.rados.rados.0.smithi064.stdout:3035: finishing write tid 3 to smithi06431743-13 2024-03-04T06:20:42.605 INFO:tasks.rados.rados.0.smithi064.stdout:3035: finishing write tid 4 to smithi06431743-13 2024-03-04T06:20:42.605 INFO:tasks.rados.rados.0.smithi064.stdout:3035: oid 13 updating version 0 to 1552 2024-03-04T06:20:42.605 INFO:tasks.rados.rados.0.smithi064.stdout:3035: oid 13 version 1552 is already newer than 1551 2024-03-04T06:20:42.605 INFO:tasks.rados.rados.0.smithi064.stdout:3035: oid 13 updating version 1552 to 1553 2024-03-04T06:20:42.605 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1553 (ObjNum 1378 snap 385 seq_num 1378) dirty exists 2024-03-04T06:20:42.606 INFO:tasks.rados.rados.0.smithi064.stdout:3035: left oid 13 (ObjNum 1378 snap 385 seq_num 1378) 2024-03-04T06:20:42.606 INFO:tasks.rados.rados.0.smithi064.stdout:3036: expect (ObjNum 1297 snap 355 seq_num 1297) 2024-03-04T06:20:42.753 INFO:tasks.rados.rados.0.smithi064.stdout:3039: expect (ObjNum 1226 snap 330 seq_num 1226) 2024-03-04T06:20:42.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:42.933 INFO:tasks.rados.rados.0.smithi064.stdout:3033: done (6 left) 2024-03-04T06:20:42.933 INFO:tasks.rados.rados.0.smithi064.stdout:3034: done (5 left) 2024-03-04T06:20:42.933 INFO:tasks.rados.rados.0.smithi064.stdout:3035: done (4 left) 2024-03-04T06:20:42.933 INFO:tasks.rados.rados.0.smithi064.stdout:3036: done (3 left) 2024-03-04T06:20:42.933 INFO:tasks.rados.rados.0.smithi064.stdout:3037: done (2 left) 2024-03-04T06:20:42.933 INFO:tasks.rados.rados.0.smithi064.stdout:3039: done (1 left) 2024-03-04T06:20:42.933 INFO:tasks.rados.rados.0.smithi064.stdout:3040: done (0 left) 2024-03-04T06:20:42.933 INFO:tasks.rados.rados.0.smithi064.stdout:3041: read oid 49 snap 380 2024-03-04T06:20:42.933 INFO:tasks.rados.rados.0.smithi064.stdout:3041: expect (ObjNum 1333 snap 368 seq_num 1333) 2024-03-04T06:20:42.933 INFO:tasks.rados.rados.0.smithi064.stdout:3042: write oid 18 current snap is 385 2024-03-04T06:20:42.933 INFO:tasks.rados.rados.0.smithi064.stdout:3042: seq_num 1382 ranges {427274=654011,1754703=586805,2918576=308698} 2024-03-04T06:20:42.939 INFO:tasks.rados.rados.0.smithi064.stdout:3042: writing smithi06431743-18 from 427274 to 1081285 tid 1 2024-03-04T06:20:42.946 INFO:tasks.rados.rados.0.smithi064.stdout:3042: writing smithi06431743-18 from 1754703 to 2341508 tid 2 2024-03-04T06:20:42.951 INFO:tasks.rados.rados.0.smithi064.stdout:3042: writing smithi06431743-18 from 2918576 to 3227274 tid 3 2024-03-04T06:20:42.951 INFO:tasks.rados.rados.0.smithi064.stdout:3043: snap_create 2024-03-04T06:20:42.953 INFO:tasks.rados.rados.0.smithi064.stdout:3041: expect (ObjNum 1333 snap 368 seq_num 1333) 2024-03-04T06:20:43.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:43.020+0000 7f9881554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:43.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:43.024+0000 7f9881554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:43.165 INFO:tasks.rados.rados.0.smithi064.stdout:3042: finishing write tid 1 to smithi06431743-18 2024-03-04T06:20:43.165 INFO:tasks.rados.rados.0.smithi064.stdout:3042: finishing write tid 2 to smithi06431743-18 2024-03-04T06:20:43.166 INFO:tasks.rados.rados.0.smithi064.stdout:3042: finishing write tid 3 to smithi06431743-18 2024-03-04T06:20:43.166 INFO:tasks.rados.rados.0.smithi064.stdout:3042: finishing write tid 4 to smithi06431743-18 2024-03-04T06:20:43.166 INFO:tasks.rados.rados.0.smithi064.stdout:3042: finishing write tid 5 to smithi06431743-18 2024-03-04T06:20:43.166 INFO:tasks.rados.rados.0.smithi064.stdout:3042: oid 18 updating version 0 to 623 2024-03-04T06:20:43.166 INFO:tasks.rados.rados.0.smithi064.stdout:3042: oid 18 version 623 is already newer than 620 2024-03-04T06:20:43.166 INFO:tasks.rados.rados.0.smithi064.stdout:3042: oid 18 version 623 is already newer than 621 2024-03-04T06:20:43.166 INFO:tasks.rados.rados.0.smithi064.stdout:3042: oid 18 version 623 is already newer than 622 2024-03-04T06:20:43.166 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 623 (ObjNum 1382 snap 385 seq_num 1382) dirty exists 2024-03-04T06:20:43.166 INFO:tasks.rados.rados.0.smithi064.stdout:3042: left oid 18 (ObjNum 1382 snap 385 seq_num 1382) 2024-03-04T06:20:43.341 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:20:43.345 INFO:tasks.rados.rados.0.smithi064.stdout:3041: done (2 left) 2024-03-04T06:20:43.345 INFO:tasks.rados.rados.0.smithi064.stdout:3042: done (1 left) 2024-03-04T06:20:43.345 INFO:tasks.rados.rados.0.smithi064.stdout:3043: done (0 left) 2024-03-04T06:20:43.345 INFO:tasks.rados.rados.0.smithi064.stdout:3044: snap_create 2024-03-04T06:20:43.356 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:20:43.357 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:20:43.467 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:43.463+0000 7fbcd522d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:43.470 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:43.469+0000 7fbcd522d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:43.661 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:20:43.662 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:20:43.673 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1152,"stamp":"2024-03-04T06:20:43.342699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103259107,"num_objects":1020,"num_object_clones":822,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":160,"num_read":5562,"num_read_kb":4312176,"num_write":4727,"num_write_kb":1252387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12189,"ondisk_log_size":12189,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":752396,"kb_used_data":539628,"kb_used_omap":0,"kb_used_meta":212736,"kb_avail":376734964,"statfs":{"total":386547056640,"available":385776603136,"internally_reserved":0,"allocated":552579072,"data_stored":657801857,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":217841664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":168103,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":33848,"num_write":18,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039284"},"pg_stats":[{"pgid":"3.f","version":"766'727","reported_seq":2644,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.333926+0000","last_change":"2024-03-04T06:20:37.398309+0000","last_active":"2024-03-04T06:20:39.333926+0000","last_peered":"2024-03-04T06:20:39.333926+0000","last_clean":"2024-03-04T06:20:39.333926+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:20:39.333926+0000","last_undegraded":"2024-03-04T06:20:39.333926+0000","last_fullsized":"2024-03-04T06:20:39.333926+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":727,"log_dups_size":0,"ondisk_log_size":727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":313,"num_read_kb":251966,"num_write":280,"num_write_kb":73822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"768'606","reported_seq":2606,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.326101+0000","last_change":"2024-03-04T06:20:40.326101+0000","last_active":"2024-03-04T06:20:40.326101+0000","last_peered":"2024-03-04T06:20:40.326101+0000","last_clean":"2024-03-04T06:20:40.326101+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:20:40.326101+0000","last_undegraded":"2024-03-04T06:20:40.326101+0000","last_fullsized":"2024-03-04T06:20:40.326101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017566,"num_objects":54,"num_object_clones":39,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":188,"num_read_kb":130659,"num_write":223,"num_write_kb":58354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"767'817","reported_seq":2772,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.365702+0000","last_change":"2024-03-04T06:20:41.365702+0000","last_active":"2024-03-04T06:20:41.365702+0000","last_peered":"2024-03-04T06:20:41.365702+0000","last_clean":"2024-03-04T06:20:41.365702+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:20:41.365702+0000","last_undegraded":"2024-03-04T06:20:41.365702+0000","last_fullsized":"2024-03-04T06:20:41.365702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11900643,"num_objects":78,"num_object_clones":63,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":450,"num_read_kb":359296,"num_write":345,"num_write_kb":93348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"766'700","reported_seq":2602,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.323093+0000","last_change":"2024-03-04T06:20:37.393557+0000","last_active":"2024-03-04T06:20:39.323093+0000","last_peered":"2024-03-04T06:20:39.323093+0000","last_clean":"2024-03-04T06:20:39.323093+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:20:39.323093+0000","last_undegraded":"2024-03-04T06:20:39.323093+0000","last_fullsized":"2024-03-04T06:20:39.323093+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8522106,"num_objects":53,"num_object_clones":43,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":267,"num_read_kb":202839,"num_write":254,"num_write_kb":71600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"769'287","reported_seq":2419,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.333996+0000","last_change":"2024-03-04T06:20:40.411132+0000","last_active":"2024-03-04T06:20:40.333996+0000","last_peered":"2024-03-04T06:20:40.333996+0000","last_clean":"2024-03-04T06:20:40.333996+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:20:40.333996+0000","last_undegraded":"2024-03-04T06:20:40.333996+0000","last_fullsized":"2024-03-04T06:20:40.333996+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672643,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":65484,"num_write":101,"num_write_kb":27678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"766'834","reported_seq":2699,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.329890+0000","last_change":"2024-03-04T06:20:37.410599+0000","last_active":"2024-03-04T06:20:39.329890+0000","last_peered":"2024-03-04T06:20:39.329890+0000","last_clean":"2024-03-04T06:20:39.329890+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:20:39.329890+0000","last_undegraded":"2024-03-04T06:20:39.329890+0000","last_fullsized":"2024-03-04T06:20:39.329890+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4757041,"num_objects":62,"num_object_clones":53,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":374,"num_read_kb":306917,"num_write":308,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2254,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.328631+0000","last_change":"2024-03-04T06:20:37.367079+0000","last_active":"2024-03-04T06:20:39.328631+0000","last_peered":"2024-03-04T06:20:39.328631+0000","last_clean":"2024-03-04T06:20:39.328631+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:20:39.328631+0000","last_undegraded":"2024-03-04T06:20:39.328631+0000","last_fullsized":"2024-03-04T06:20:39.328631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"767'458","reported_seq":2504,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.363596+0000","last_change":"2024-03-04T06:20:41.363596+0000","last_active":"2024-03-04T06:20:41.363596+0000","last_peered":"2024-03-04T06:20:41.363596+0000","last_clean":"2024-03-04T06:20:41.363596+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:20:41.363596+0000","last_undegraded":"2024-03-04T06:20:41.363596+0000","last_fullsized":"2024-03-04T06:20:41.363596+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403731,"num_objects":35,"num_object_clones":26,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":222,"num_read_kb":204266,"num_write":154,"num_write_kb":41095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2066,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.316964+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:20:40.316964+0000","last_peered":"2024-03-04T06:20:40.316964+0000","last_clean":"2024-03-04T06:20:40.316964+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:20:40.316964+0000","last_undegraded":"2024-03-04T06:20:40.316964+0000","last_fullsized":"2024-03-04T06:20:40.316964+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1908,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-03-04T06:20:41.385424+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:20:41.385424+0000","last_peered":"2024-03-04T06:20:41.385424+0000","last_clean":"2024-03-04T06:20:41.385424+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:20:41.385424+0000","last_undegraded":"2024-03-04T06:20:41.385424+0000","last_fullsized":"2024-03-04T06:20:41.385424+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"767'752","reported_seq":2687,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.374909+0000","last_change":"2024-03-04T06:20:41.374909+0000","last_active":"2024-03-04T06:20:41.374909+0000","last_peered":"2024-03-04T06:20:41.374909+0000","last_clean":"2024-03-04T06:20:41.374909+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:20:41.374909+0000","last_undegraded":"2024-03-04T06:20:41.374909+0000","last_fullsized":"2024-03-04T06:20:41.374909+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7463298,"num_objects":65,"num_object_clones":51,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":358,"num_read_kb":286050,"num_write":292,"num_write_kb":78583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1898,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.332020+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:20:39.332020+0000","last_peered":"2024-03-04T06:20:39.332020+0000","last_clean":"2024-03-04T06:20:39.332020+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:20:39.332020+0000","last_undegraded":"2024-03-04T06:20:39.332020+0000","last_fullsized":"2024-03-04T06:20:39.332020+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"769'479","reported_seq":2570,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.321759+0000","last_change":"2024-03-04T06:20:40.380675+0000","last_active":"2024-03-04T06:20:40.321759+0000","last_peered":"2024-03-04T06:20:40.321759+0000","last_clean":"2024-03-04T06:20:40.321759+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:20:40.321759+0000","last_undegraded":"2024-03-04T06:20:40.321759+0000","last_fullsized":"2024-03-04T06:20:40.321759+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698233,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":270,"num_read_kb":221303,"num_write":175,"num_write_kb":49293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1908,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-03-04T06:20:41.362413+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:20:41.362413+0000","last_peered":"2024-03-04T06:20:41.362413+0000","last_clean":"2024-03-04T06:20:41.362413+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:20:41.362413+0000","last_undegraded":"2024-03-04T06:20:41.362413+0000","last_fullsized":"2024-03-04T06:20:41.362413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"769'1355","reported_seq":3105,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.330961+0000","last_change":"2024-03-04T06:20:40.446302+0000","last_active":"2024-03-04T06:20:40.330961+0000","last_peered":"2024-03-04T06:20:40.330961+0000","last_clean":"2024-03-04T06:20:40.330961+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:20:40.330961+0000","last_undegraded":"2024-03-04T06:20:40.330961+0000","last_fullsized":"2024-03-04T06:20:40.330961+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10287171,"num_objects":118,"num_object_clones":94,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":713,"num_read_kb":567605,"num_write":548,"num_write_kb":146807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1898,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.319938+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:20:39.319938+0000","last_peered":"2024-03-04T06:20:39.319938+0000","last_clean":"2024-03-04T06:20:39.319938+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:20:39.319938+0000","last_undegraded":"2024-03-04T06:20:39.319938+0000","last_fullsized":"2024-03-04T06:20:39.319938+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"766'1528","reported_seq":3081,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.322016+0000","last_change":"2024-03-04T06:20:40.322016+0000","last_active":"2024-03-04T06:20:40.322016+0000","last_peered":"2024-03-04T06:20:40.322016+0000","last_clean":"2024-03-04T06:20:40.322016+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:20:40.322016+0000","last_undegraded":"2024-03-04T06:20:40.322016+0000","last_fullsized":"2024-03-04T06:20:40.322016+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16455604,"num_objects":124,"num_object_clones":101,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":561,"num_read_kb":413234,"num_write":573,"num_write_kb":154189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"767'749","reported_seq":2695,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.387588+0000","last_change":"2024-03-04T06:20:41.387588+0000","last_active":"2024-03-04T06:20:41.387588+0000","last_peered":"2024-03-04T06:20:41.387588+0000","last_clean":"2024-03-04T06:20:41.387588+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:20:41.387588+0000","last_undegraded":"2024-03-04T06:20:41.387588+0000","last_fullsized":"2024-03-04T06:20:41.387588+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":749,"log_dups_size":0,"ondisk_log_size":749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042397,"num_objects":60,"num_object_clones":48,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":394,"num_read_kb":329119,"num_write":278,"num_write_kb":75144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1967,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.339391+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:20:40.339391+0000","last_peered":"2024-03-04T06:20:40.339391+0000","last_clean":"2024-03-04T06:20:40.339391+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:20:40.339391+0000","last_undegraded":"2024-03-04T06:20:40.339391+0000","last_fullsized":"2024-03-04T06:20:40.339391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"766'795","reported_seq":2741,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.340665+0000","last_change":"2024-03-04T06:20:40.340665+0000","last_active":"2024-03-04T06:20:40.340665+0000","last_peered":"2024-03-04T06:20:40.340665+0000","last_clean":"2024-03-04T06:20:40.340665+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:20:40.340665+0000","last_undegraded":"2024-03-04T06:20:40.340665+0000","last_fullsized":"2024-03-04T06:20:40.340665+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7883473,"num_objects":72,"num_object_clones":56,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":335,"num_read_kb":241316,"num_write":305,"num_write_kb":82364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1933,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.335440+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:20:40.335440+0000","last_peered":"2024-03-04T06:20:40.335440+0000","last_clean":"2024-03-04T06:20:40.335440+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:20:40.335440+0000","last_undegraded":"2024-03-04T06:20:40.335440+0000","last_fullsized":"2024-03-04T06:20:40.335440+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"769'1030","reported_seq":2826,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.319464+0000","last_change":"2024-03-04T06:20:40.444783+0000","last_active":"2024-03-04T06:20:40.319464+0000","last_peered":"2024-03-04T06:20:40.319464+0000","last_clean":"2024-03-04T06:20:40.319464+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:20:40.319464+0000","last_undegraded":"2024-03-04T06:20:40.319464+0000","last_fullsized":"2024-03-04T06:20:40.319464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1030,"log_dups_size":0,"ondisk_log_size":1030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12673699,"num_objects":84,"num_object_clones":70,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":422,"num_read_kb":309262,"num_write":394,"num_write_kb":110146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1967,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.320234+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:20:40.320234+0000","last_peered":"2024-03-04T06:20:40.320234+0000","last_clean":"2024-03-04T06:20:40.320234+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:20:40.320234+0000","last_undegraded":"2024-03-04T06:20:40.320234+0000","last_fullsized":"2024-03-04T06:20:40.320234+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"765'975","reported_seq":2875,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.324210+0000","last_change":"2024-03-04T06:20:40.324210+0000","last_active":"2024-03-04T06:20:40.324210+0000","last_peered":"2024-03-04T06:20:40.324210+0000","last_clean":"2024-03-04T06:20:40.324210+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:20:40.324210+0000","last_undegraded":"2024-03-04T06:20:40.324210+0000","last_fullsized":"2024-03-04T06:20:40.324210+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2891134,"num_objects":80,"num_object_clones":70,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":501,"num_read_kb":422769,"num_write":360,"num_write_kb":98417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1908,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-03-04T06:20:41.374122+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:20:41.374122+0000","last_peered":"2024-03-04T06:20:41.374122+0000","last_clean":"2024-03-04T06:20:41.374122+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:20:41.374122+0000","last_undegraded":"2024-03-04T06:20:41.374122+0000","last_fullsized":"2024-03-04T06:20:41.374122+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102668739,"num_objects":1018,"num_object_clones":822,"num_object_copies":3054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":160,"num_read":5459,"num_read_kb":4312085,"num_write":4590,"num_write_kb":1250493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12092,"ondisk_log_size":12092,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":218288,"kb_used_data":160936,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94153552,"statfs":{"total":96636764160,"available":96413237248,"internally_reserved":0,"allocated":164798464,"data_stored":195811052,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":172160,"kb_used_data":118968,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94199680,"statfs":{"total":96636764160,"available":96460472320,"internally_reserved":0,"allocated":121823232,"data_stored":139542283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":175848,"kb_used_data":122656,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94195992,"statfs":{"total":96636764160,"available":96456695808,"internally_reserved":0,"allocated":125599744,"data_stored":149793880,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":186100,"kb_used_data":137068,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94185740,"statfs":{"total":96636764160,"available":96446197760,"internally_reserved":0,"allocated":140357632,"data_stored":172654642,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:20:43.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:20:43.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:43.778+0000 7ff149f07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:43.785 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:43.784+0000 7ff149f07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:20:43.973 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:20:43.985 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1152,"stamp":"2024-03-04T06:20:43.342699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103259107,"num_objects":1020,"num_object_clones":822,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":160,"num_read":5562,"num_read_kb":4312176,"num_write":4727,"num_write_kb":1252387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12189,"ondisk_log_size":12189,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":752396,"kb_used_data":539628,"kb_used_omap":0,"kb_used_meta":212736,"kb_avail":376734964,"statfs":{"total":386547056640,"available":385776603136,"internally_reserved":0,"allocated":552579072,"data_stored":657801857,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":217841664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":168103,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":33848,"num_write":18,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039284"},"pg_stats":[{"pgid":"3.f","version":"766'727","reported_seq":2644,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.333926+0000","last_change":"2024-03-04T06:20:37.398309+0000","last_active":"2024-03-04T06:20:39.333926+0000","last_peered":"2024-03-04T06:20:39.333926+0000","last_clean":"2024-03-04T06:20:39.333926+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:20:39.333926+0000","last_undegraded":"2024-03-04T06:20:39.333926+0000","last_fullsized":"2024-03-04T06:20:39.333926+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":727,"log_dups_size":0,"ondisk_log_size":727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":313,"num_read_kb":251966,"num_write":280,"num_write_kb":73822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"768'606","reported_seq":2606,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.326101+0000","last_change":"2024-03-04T06:20:40.326101+0000","last_active":"2024-03-04T06:20:40.326101+0000","last_peered":"2024-03-04T06:20:40.326101+0000","last_clean":"2024-03-04T06:20:40.326101+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:20:40.326101+0000","last_undegraded":"2024-03-04T06:20:40.326101+0000","last_fullsized":"2024-03-04T06:20:40.326101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017566,"num_objects":54,"num_object_clones":39,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":188,"num_read_kb":130659,"num_write":223,"num_write_kb":58354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"767'817","reported_seq":2772,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.365702+0000","last_change":"2024-03-04T06:20:41.365702+0000","last_active":"2024-03-04T06:20:41.365702+0000","last_peered":"2024-03-04T06:20:41.365702+0000","last_clean":"2024-03-04T06:20:41.365702+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:20:41.365702+0000","last_undegraded":"2024-03-04T06:20:41.365702+0000","last_fullsized":"2024-03-04T06:20:41.365702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11900643,"num_objects":78,"num_object_clones":63,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":450,"num_read_kb":359296,"num_write":345,"num_write_kb":93348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"766'700","reported_seq":2602,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.323093+0000","last_change":"2024-03-04T06:20:37.393557+0000","last_active":"2024-03-04T06:20:39.323093+0000","last_peered":"2024-03-04T06:20:39.323093+0000","last_clean":"2024-03-04T06:20:39.323093+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:20:39.323093+0000","last_undegraded":"2024-03-04T06:20:39.323093+0000","last_fullsized":"2024-03-04T06:20:39.323093+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8522106,"num_objects":53,"num_object_clones":43,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":267,"num_read_kb":202839,"num_write":254,"num_write_kb":71600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"769'287","reported_seq":2419,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.333996+0000","last_change":"2024-03-04T06:20:40.411132+0000","last_active":"2024-03-04T06:20:40.333996+0000","last_peered":"2024-03-04T06:20:40.333996+0000","last_clean":"2024-03-04T06:20:40.333996+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:20:40.333996+0000","last_undegraded":"2024-03-04T06:20:40.333996+0000","last_fullsized":"2024-03-04T06:20:40.333996+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672643,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":65484,"num_write":101,"num_write_kb":27678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"766'834","reported_seq":2699,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.329890+0000","last_change":"2024-03-04T06:20:37.410599+0000","last_active":"2024-03-04T06:20:39.329890+0000","last_peered":"2024-03-04T06:20:39.329890+0000","last_clean":"2024-03-04T06:20:39.329890+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:20:39.329890+0000","last_undegraded":"2024-03-04T06:20:39.329890+0000","last_fullsized":"2024-03-04T06:20:39.329890+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4757041,"num_objects":62,"num_object_clones":53,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":374,"num_read_kb":306917,"num_write":308,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2254,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.328631+0000","last_change":"2024-03-04T06:20:37.367079+0000","last_active":"2024-03-04T06:20:39.328631+0000","last_peered":"2024-03-04T06:20:39.328631+0000","last_clean":"2024-03-04T06:20:39.328631+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:20:39.328631+0000","last_undegraded":"2024-03-04T06:20:39.328631+0000","last_fullsized":"2024-03-04T06:20:39.328631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"767'458","reported_seq":2504,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.363596+0000","last_change":"2024-03-04T06:20:41.363596+0000","last_active":"2024-03-04T06:20:41.363596+0000","last_peered":"2024-03-04T06:20:41.363596+0000","last_clean":"2024-03-04T06:20:41.363596+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:20:41.363596+0000","last_undegraded":"2024-03-04T06:20:41.363596+0000","last_fullsized":"2024-03-04T06:20:41.363596+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403731,"num_objects":35,"num_object_clones":26,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":222,"num_read_kb":204266,"num_write":154,"num_write_kb":41095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2066,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.316964+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:20:40.316964+0000","last_peered":"2024-03-04T06:20:40.316964+0000","last_clean":"2024-03-04T06:20:40.316964+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:20:40.316964+0000","last_undegraded":"2024-03-04T06:20:40.316964+0000","last_fullsized":"2024-03-04T06:20:40.316964+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1908,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-03-04T06:20:41.385424+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:20:41.385424+0000","last_peered":"2024-03-04T06:20:41.385424+0000","last_clean":"2024-03-04T06:20:41.385424+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:20:41.385424+0000","last_undegraded":"2024-03-04T06:20:41.385424+0000","last_fullsized":"2024-03-04T06:20:41.385424+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"767'752","reported_seq":2687,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.374909+0000","last_change":"2024-03-04T06:20:41.374909+0000","last_active":"2024-03-04T06:20:41.374909+0000","last_peered":"2024-03-04T06:20:41.374909+0000","last_clean":"2024-03-04T06:20:41.374909+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:20:41.374909+0000","last_undegraded":"2024-03-04T06:20:41.374909+0000","last_fullsized":"2024-03-04T06:20:41.374909+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7463298,"num_objects":65,"num_object_clones":51,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":358,"num_read_kb":286050,"num_write":292,"num_write_kb":78583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1898,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.332020+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:20:39.332020+0000","last_peered":"2024-03-04T06:20:39.332020+0000","last_clean":"2024-03-04T06:20:39.332020+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:20:39.332020+0000","last_undegraded":"2024-03-04T06:20:39.332020+0000","last_fullsized":"2024-03-04T06:20:39.332020+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"769'479","reported_seq":2570,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.321759+0000","last_change":"2024-03-04T06:20:40.380675+0000","last_active":"2024-03-04T06:20:40.321759+0000","last_peered":"2024-03-04T06:20:40.321759+0000","last_clean":"2024-03-04T06:20:40.321759+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:20:40.321759+0000","last_undegraded":"2024-03-04T06:20:40.321759+0000","last_fullsized":"2024-03-04T06:20:40.321759+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698233,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":270,"num_read_kb":221303,"num_write":175,"num_write_kb":49293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1908,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-03-04T06:20:41.362413+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:20:41.362413+0000","last_peered":"2024-03-04T06:20:41.362413+0000","last_clean":"2024-03-04T06:20:41.362413+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:20:41.362413+0000","last_undegraded":"2024-03-04T06:20:41.362413+0000","last_fullsized":"2024-03-04T06:20:41.362413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"769'1355","reported_seq":3105,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.330961+0000","last_change":"2024-03-04T06:20:40.446302+0000","last_active":"2024-03-04T06:20:40.330961+0000","last_peered":"2024-03-04T06:20:40.330961+0000","last_clean":"2024-03-04T06:20:40.330961+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:20:40.330961+0000","last_undegraded":"2024-03-04T06:20:40.330961+0000","last_fullsized":"2024-03-04T06:20:40.330961+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10287171,"num_objects":118,"num_object_clones":94,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":713,"num_read_kb":567605,"num_write":548,"num_write_kb":146807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1898,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.319938+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:20:39.319938+0000","last_peered":"2024-03-04T06:20:39.319938+0000","last_clean":"2024-03-04T06:20:39.319938+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:20:39.319938+0000","last_undegraded":"2024-03-04T06:20:39.319938+0000","last_fullsized":"2024-03-04T06:20:39.319938+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"766'1528","reported_seq":3081,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.322016+0000","last_change":"2024-03-04T06:20:40.322016+0000","last_active":"2024-03-04T06:20:40.322016+0000","last_peered":"2024-03-04T06:20:40.322016+0000","last_clean":"2024-03-04T06:20:40.322016+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:20:40.322016+0000","last_undegraded":"2024-03-04T06:20:40.322016+0000","last_fullsized":"2024-03-04T06:20:40.322016+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16455604,"num_objects":124,"num_object_clones":101,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":561,"num_read_kb":413234,"num_write":573,"num_write_kb":154189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"767'749","reported_seq":2695,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.387588+0000","last_change":"2024-03-04T06:20:41.387588+0000","last_active":"2024-03-04T06:20:41.387588+0000","last_peered":"2024-03-04T06:20:41.387588+0000","last_clean":"2024-03-04T06:20:41.387588+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:20:41.387588+0000","last_undegraded":"2024-03-04T06:20:41.387588+0000","last_fullsized":"2024-03-04T06:20:41.387588+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":749,"log_dups_size":0,"ondisk_log_size":749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042397,"num_objects":60,"num_object_clones":48,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":394,"num_read_kb":329119,"num_write":278,"num_write_kb":75144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1967,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.339391+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:20:40.339391+0000","last_peered":"2024-03-04T06:20:40.339391+0000","last_clean":"2024-03-04T06:20:40.339391+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:20:40.339391+0000","last_undegraded":"2024-03-04T06:20:40.339391+0000","last_fullsized":"2024-03-04T06:20:40.339391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"766'795","reported_seq":2741,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.340665+0000","last_change":"2024-03-04T06:20:40.340665+0000","last_active":"2024-03-04T06:20:40.340665+0000","last_peered":"2024-03-04T06:20:40.340665+0000","last_clean":"2024-03-04T06:20:40.340665+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:20:40.340665+0000","last_undegraded":"2024-03-04T06:20:40.340665+0000","last_fullsized":"2024-03-04T06:20:40.340665+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7883473,"num_objects":72,"num_object_clones":56,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":335,"num_read_kb":241316,"num_write":305,"num_write_kb":82364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1933,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.335440+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:20:40.335440+0000","last_peered":"2024-03-04T06:20:40.335440+0000","last_clean":"2024-03-04T06:20:40.335440+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:20:40.335440+0000","last_undegraded":"2024-03-04T06:20:40.335440+0000","last_fullsized":"2024-03-04T06:20:40.335440+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"769'1030","reported_seq":2826,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.319464+0000","last_change":"2024-03-04T06:20:40.444783+0000","last_active":"2024-03-04T06:20:40.319464+0000","last_peered":"2024-03-04T06:20:40.319464+0000","last_clean":"2024-03-04T06:20:40.319464+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:20:40.319464+0000","last_undegraded":"2024-03-04T06:20:40.319464+0000","last_fullsized":"2024-03-04T06:20:40.319464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1030,"log_dups_size":0,"ondisk_log_size":1030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12673699,"num_objects":84,"num_object_clones":70,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":422,"num_read_kb":309262,"num_write":394,"num_write_kb":110146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1967,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.320234+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:20:40.320234+0000","last_peered":"2024-03-04T06:20:40.320234+0000","last_clean":"2024-03-04T06:20:40.320234+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:20:40.320234+0000","last_undegraded":"2024-03-04T06:20:40.320234+0000","last_fullsized":"2024-03-04T06:20:40.320234+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"765'975","reported_seq":2875,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.324210+0000","last_change":"2024-03-04T06:20:40.324210+0000","last_active":"2024-03-04T06:20:40.324210+0000","last_peered":"2024-03-04T06:20:40.324210+0000","last_clean":"2024-03-04T06:20:40.324210+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:20:40.324210+0000","last_undegraded":"2024-03-04T06:20:40.324210+0000","last_fullsized":"2024-03-04T06:20:40.324210+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2891134,"num_objects":80,"num_object_clones":70,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":501,"num_read_kb":422769,"num_write":360,"num_write_kb":98417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1908,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-03-04T06:20:41.374122+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:20:41.374122+0000","last_peered":"2024-03-04T06:20:41.374122+0000","last_clean":"2024-03-04T06:20:41.374122+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:20:41.374122+0000","last_undegraded":"2024-03-04T06:20:41.374122+0000","last_fullsized":"2024-03-04T06:20:41.374122+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102668739,"num_objects":1018,"num_object_clones":822,"num_object_copies":3054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":160,"num_read":5459,"num_read_kb":4312085,"num_write":4590,"num_write_kb":1250493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12092,"ondisk_log_size":12092,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":218288,"kb_used_data":160936,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94153552,"statfs":{"total":96636764160,"available":96413237248,"internally_reserved":0,"allocated":164798464,"data_stored":195811052,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":172160,"kb_used_data":118968,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94199680,"statfs":{"total":96636764160,"available":96460472320,"internally_reserved":0,"allocated":121823232,"data_stored":139542283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":175848,"kb_used_data":122656,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94195992,"statfs":{"total":96636764160,"available":96456695808,"internally_reserved":0,"allocated":125599744,"data_stored":149793880,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":186100,"kb_used_data":137068,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94185740,"statfs":{"total":96636764160,"available":96446197760,"internally_reserved":0,"allocated":140357632,"data_stored":172654642,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:20:43.987 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:20:43.987 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:20:43.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:20:44.094 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:44.090+0000 7f841746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:44.097 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:44.096+0000 7f841746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:44.286 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:20:44.286 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:20:44.298 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1152,"stamp":"2024-03-04T06:20:43.342699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103259107,"num_objects":1020,"num_object_clones":822,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":160,"num_read":5562,"num_read_kb":4312176,"num_write":4727,"num_write_kb":1252387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12189,"ondisk_log_size":12189,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":752396,"kb_used_data":539628,"kb_used_omap":0,"kb_used_meta":212736,"kb_avail":376734964,"statfs":{"total":386547056640,"available":385776603136,"internally_reserved":0,"allocated":552579072,"data_stored":657801857,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":217841664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":168103,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":33848,"num_write":18,"num_write_kb":4263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039284"},"pg_stats":[{"pgid":"3.f","version":"766'727","reported_seq":2644,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.333926+0000","last_change":"2024-03-04T06:20:37.398309+0000","last_active":"2024-03-04T06:20:39.333926+0000","last_peered":"2024-03-04T06:20:39.333926+0000","last_clean":"2024-03-04T06:20:39.333926+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:20:39.333926+0000","last_undegraded":"2024-03-04T06:20:39.333926+0000","last_fullsized":"2024-03-04T06:20:39.333926+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":727,"log_dups_size":0,"ondisk_log_size":727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":313,"num_read_kb":251966,"num_write":280,"num_write_kb":73822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"768'606","reported_seq":2606,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.326101+0000","last_change":"2024-03-04T06:20:40.326101+0000","last_active":"2024-03-04T06:20:40.326101+0000","last_peered":"2024-03-04T06:20:40.326101+0000","last_clean":"2024-03-04T06:20:40.326101+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:20:40.326101+0000","last_undegraded":"2024-03-04T06:20:40.326101+0000","last_fullsized":"2024-03-04T06:20:40.326101+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017566,"num_objects":54,"num_object_clones":39,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":188,"num_read_kb":130659,"num_write":223,"num_write_kb":58354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"767'817","reported_seq":2772,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.365702+0000","last_change":"2024-03-04T06:20:41.365702+0000","last_active":"2024-03-04T06:20:41.365702+0000","last_peered":"2024-03-04T06:20:41.365702+0000","last_clean":"2024-03-04T06:20:41.365702+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:20:41.365702+0000","last_undegraded":"2024-03-04T06:20:41.365702+0000","last_fullsized":"2024-03-04T06:20:41.365702+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":817,"log_dups_size":0,"ondisk_log_size":817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11900643,"num_objects":78,"num_object_clones":63,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":450,"num_read_kb":359296,"num_write":345,"num_write_kb":93348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"766'700","reported_seq":2602,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.323093+0000","last_change":"2024-03-04T06:20:37.393557+0000","last_active":"2024-03-04T06:20:39.323093+0000","last_peered":"2024-03-04T06:20:39.323093+0000","last_clean":"2024-03-04T06:20:39.323093+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:20:39.323093+0000","last_undegraded":"2024-03-04T06:20:39.323093+0000","last_fullsized":"2024-03-04T06:20:39.323093+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8522106,"num_objects":53,"num_object_clones":43,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":267,"num_read_kb":202839,"num_write":254,"num_write_kb":71600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"769'287","reported_seq":2419,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.333996+0000","last_change":"2024-03-04T06:20:40.411132+0000","last_active":"2024-03-04T06:20:40.333996+0000","last_peered":"2024-03-04T06:20:40.333996+0000","last_clean":"2024-03-04T06:20:40.333996+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:20:40.333996+0000","last_undegraded":"2024-03-04T06:20:40.333996+0000","last_fullsized":"2024-03-04T06:20:40.333996+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672643,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":65484,"num_write":101,"num_write_kb":27678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"766'834","reported_seq":2699,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.329890+0000","last_change":"2024-03-04T06:20:37.410599+0000","last_active":"2024-03-04T06:20:39.329890+0000","last_peered":"2024-03-04T06:20:39.329890+0000","last_clean":"2024-03-04T06:20:39.329890+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:20:39.329890+0000","last_undegraded":"2024-03-04T06:20:39.329890+0000","last_fullsized":"2024-03-04T06:20:39.329890+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4757041,"num_objects":62,"num_object_clones":53,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":374,"num_read_kb":306917,"num_write":308,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2254,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.328631+0000","last_change":"2024-03-04T06:20:37.367079+0000","last_active":"2024-03-04T06:20:39.328631+0000","last_peered":"2024-03-04T06:20:39.328631+0000","last_clean":"2024-03-04T06:20:39.328631+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:20:39.328631+0000","last_undegraded":"2024-03-04T06:20:39.328631+0000","last_fullsized":"2024-03-04T06:20:39.328631+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"767'458","reported_seq":2504,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.363596+0000","last_change":"2024-03-04T06:20:41.363596+0000","last_active":"2024-03-04T06:20:41.363596+0000","last_peered":"2024-03-04T06:20:41.363596+0000","last_clean":"2024-03-04T06:20:41.363596+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:20:41.363596+0000","last_undegraded":"2024-03-04T06:20:41.363596+0000","last_fullsized":"2024-03-04T06:20:41.363596+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403731,"num_objects":35,"num_object_clones":26,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":222,"num_read_kb":204266,"num_write":154,"num_write_kb":41095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2066,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.316964+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:20:40.316964+0000","last_peered":"2024-03-04T06:20:40.316964+0000","last_clean":"2024-03-04T06:20:40.316964+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:20:40.316964+0000","last_undegraded":"2024-03-04T06:20:40.316964+0000","last_fullsized":"2024-03-04T06:20:40.316964+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1908,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-03-04T06:20:41.385424+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:20:41.385424+0000","last_peered":"2024-03-04T06:20:41.385424+0000","last_clean":"2024-03-04T06:20:41.385424+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:20:41.385424+0000","last_undegraded":"2024-03-04T06:20:41.385424+0000","last_fullsized":"2024-03-04T06:20:41.385424+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"767'752","reported_seq":2687,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.374909+0000","last_change":"2024-03-04T06:20:41.374909+0000","last_active":"2024-03-04T06:20:41.374909+0000","last_peered":"2024-03-04T06:20:41.374909+0000","last_clean":"2024-03-04T06:20:41.374909+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:20:41.374909+0000","last_undegraded":"2024-03-04T06:20:41.374909+0000","last_fullsized":"2024-03-04T06:20:41.374909+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7463298,"num_objects":65,"num_object_clones":51,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":358,"num_read_kb":286050,"num_write":292,"num_write_kb":78583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1898,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.332020+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:20:39.332020+0000","last_peered":"2024-03-04T06:20:39.332020+0000","last_clean":"2024-03-04T06:20:39.332020+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:20:39.332020+0000","last_undegraded":"2024-03-04T06:20:39.332020+0000","last_fullsized":"2024-03-04T06:20:39.332020+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"769'479","reported_seq":2570,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.321759+0000","last_change":"2024-03-04T06:20:40.380675+0000","last_active":"2024-03-04T06:20:40.321759+0000","last_peered":"2024-03-04T06:20:40.321759+0000","last_clean":"2024-03-04T06:20:40.321759+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:20:40.321759+0000","last_undegraded":"2024-03-04T06:20:40.321759+0000","last_fullsized":"2024-03-04T06:20:40.321759+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698233,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":270,"num_read_kb":221303,"num_write":175,"num_write_kb":49293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1908,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-03-04T06:20:41.362413+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:20:41.362413+0000","last_peered":"2024-03-04T06:20:41.362413+0000","last_clean":"2024-03-04T06:20:41.362413+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:20:41.362413+0000","last_undegraded":"2024-03-04T06:20:41.362413+0000","last_fullsized":"2024-03-04T06:20:41.362413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"769'1355","reported_seq":3105,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.330961+0000","last_change":"2024-03-04T06:20:40.446302+0000","last_active":"2024-03-04T06:20:40.330961+0000","last_peered":"2024-03-04T06:20:40.330961+0000","last_clean":"2024-03-04T06:20:40.330961+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:20:40.330961+0000","last_undegraded":"2024-03-04T06:20:40.330961+0000","last_fullsized":"2024-03-04T06:20:40.330961+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10287171,"num_objects":118,"num_object_clones":94,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":713,"num_read_kb":567605,"num_write":548,"num_write_kb":146807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1898,"reported_epoch":768,"state":"active+clean","last_fresh":"2024-03-04T06:20:39.319938+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:20:39.319938+0000","last_peered":"2024-03-04T06:20:39.319938+0000","last_clean":"2024-03-04T06:20:39.319938+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:20:39.319938+0000","last_undegraded":"2024-03-04T06:20:39.319938+0000","last_fullsized":"2024-03-04T06:20:39.319938+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"766'1528","reported_seq":3081,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.322016+0000","last_change":"2024-03-04T06:20:40.322016+0000","last_active":"2024-03-04T06:20:40.322016+0000","last_peered":"2024-03-04T06:20:40.322016+0000","last_clean":"2024-03-04T06:20:40.322016+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:20:40.322016+0000","last_undegraded":"2024-03-04T06:20:40.322016+0000","last_fullsized":"2024-03-04T06:20:40.322016+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16455604,"num_objects":124,"num_object_clones":101,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":561,"num_read_kb":413234,"num_write":573,"num_write_kb":154189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"767'749","reported_seq":2695,"reported_epoch":770,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:41.387588+0000","last_change":"2024-03-04T06:20:41.387588+0000","last_active":"2024-03-04T06:20:41.387588+0000","last_peered":"2024-03-04T06:20:41.387588+0000","last_clean":"2024-03-04T06:20:41.387588+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:20:41.387588+0000","last_undegraded":"2024-03-04T06:20:41.387588+0000","last_fullsized":"2024-03-04T06:20:41.387588+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":749,"log_dups_size":0,"ondisk_log_size":749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042397,"num_objects":60,"num_object_clones":48,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":394,"num_read_kb":329119,"num_write":278,"num_write_kb":75144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1967,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.339391+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:20:40.339391+0000","last_peered":"2024-03-04T06:20:40.339391+0000","last_clean":"2024-03-04T06:20:40.339391+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:20:40.339391+0000","last_undegraded":"2024-03-04T06:20:40.339391+0000","last_fullsized":"2024-03-04T06:20:40.339391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"766'795","reported_seq":2741,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.340665+0000","last_change":"2024-03-04T06:20:40.340665+0000","last_active":"2024-03-04T06:20:40.340665+0000","last_peered":"2024-03-04T06:20:40.340665+0000","last_clean":"2024-03-04T06:20:40.340665+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:20:40.340665+0000","last_undegraded":"2024-03-04T06:20:40.340665+0000","last_fullsized":"2024-03-04T06:20:40.340665+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":795,"log_dups_size":0,"ondisk_log_size":795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7883473,"num_objects":72,"num_object_clones":56,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":335,"num_read_kb":241316,"num_write":305,"num_write_kb":82364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1933,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.335440+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:20:40.335440+0000","last_peered":"2024-03-04T06:20:40.335440+0000","last_clean":"2024-03-04T06:20:40.335440+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:20:40.335440+0000","last_undegraded":"2024-03-04T06:20:40.335440+0000","last_fullsized":"2024-03-04T06:20:40.335440+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"769'1030","reported_seq":2826,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.319464+0000","last_change":"2024-03-04T06:20:40.444783+0000","last_active":"2024-03-04T06:20:40.319464+0000","last_peered":"2024-03-04T06:20:40.319464+0000","last_clean":"2024-03-04T06:20:40.319464+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:20:40.319464+0000","last_undegraded":"2024-03-04T06:20:40.319464+0000","last_fullsized":"2024-03-04T06:20:40.319464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1030,"log_dups_size":0,"ondisk_log_size":1030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12673699,"num_objects":84,"num_object_clones":70,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":422,"num_read_kb":309262,"num_write":394,"num_write_kb":110146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1967,"reported_epoch":769,"state":"active+clean","last_fresh":"2024-03-04T06:20:40.320234+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:20:40.320234+0000","last_peered":"2024-03-04T06:20:40.320234+0000","last_clean":"2024-03-04T06:20:40.320234+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:20:40.320234+0000","last_undegraded":"2024-03-04T06:20:40.320234+0000","last_fullsized":"2024-03-04T06:20:40.320234+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"765'975","reported_seq":2875,"reported_epoch":769,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:40.324210+0000","last_change":"2024-03-04T06:20:40.324210+0000","last_active":"2024-03-04T06:20:40.324210+0000","last_peered":"2024-03-04T06:20:40.324210+0000","last_clean":"2024-03-04T06:20:40.324210+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:20:40.324210+0000","last_undegraded":"2024-03-04T06:20:40.324210+0000","last_fullsized":"2024-03-04T06:20:40.324210+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2891134,"num_objects":80,"num_object_clones":70,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":501,"num_read_kb":422769,"num_write":360,"num_write_kb":98417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1908,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-03-04T06:20:41.374122+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:20:41.374122+0000","last_peered":"2024-03-04T06:20:41.374122+0000","last_clean":"2024-03-04T06:20:41.374122+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:20:41.374122+0000","last_undegraded":"2024-03-04T06:20:41.374122+0000","last_fullsized":"2024-03-04T06:20:41.374122+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102668739,"num_objects":1018,"num_object_clones":822,"num_object_copies":3054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":160,"num_read":5459,"num_read_kb":4312085,"num_write":4590,"num_write_kb":1250493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12092,"ondisk_log_size":12092,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":218288,"kb_used_data":160936,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94153552,"statfs":{"total":96636764160,"available":96413237248,"internally_reserved":0,"allocated":164798464,"data_stored":195811052,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":172160,"kb_used_data":118968,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94199680,"statfs":{"total":96636764160,"available":96460472320,"internally_reserved":0,"allocated":121823232,"data_stored":139542283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":175848,"kb_used_data":122656,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94195992,"statfs":{"total":96636764160,"available":96456695808,"internally_reserved":0,"allocated":125599744,"data_stored":149793880,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":186100,"kb_used_data":137068,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94185740,"statfs":{"total":96636764160,"available":96446197760,"internally_reserved":0,"allocated":140357632,"data_stored":172654642,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:20:44.347 INFO:tasks.rados.rados.0.smithi064.stdout:3044: done (0 left) 2024-03-04T06:20:44.347 INFO:tasks.rados.rados.0.smithi064.stdout:3045: read oid 49 snap -1 2024-03-04T06:20:44.347 INFO:tasks.rados.rados.0.smithi064.stdout:3045: expect (ObjNum 1333 snap 368 seq_num 1333) 2024-03-04T06:20:44.347 INFO:tasks.rados.rados.0.smithi064.stdout:3046: snap_create 2024-03-04T06:20:44.397 INFO:tasks.rados.rados.0.smithi064.stdout:3045: expect (ObjNum 1333 snap 368 seq_num 1333) 2024-03-04T06:20:45.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:45.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:45.210+0000 7f3064a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:45.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:45.215+0000 7f3064a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:45.347 INFO:tasks.rados.rados.0.smithi064.stdout:3045: done (1 left) 2024-03-04T06:20:45.347 INFO:tasks.rados.rados.0.smithi064.stdout:3046: done (0 left) 2024-03-04T06:20:45.347 INFO:tasks.rados.rados.0.smithi064.stdout:3047: write (excl) oid 30 current snap is 388 2024-03-04T06:20:45.347 INFO:tasks.rados.rados.0.smithi064.stdout:3047: seq_num 1383 ranges {444081=729260,1813185=446816,2945999=298082} 2024-03-04T06:20:45.353 INFO:tasks.rados.rados.0.smithi064.stdout:3047: writing smithi06431743-30 from 444081 to 1173341 tid 1 2024-03-04T06:20:45.359 INFO:tasks.rados.rados.0.smithi064.stdout:3047: writing smithi06431743-30 from 1813185 to 2260001 tid 2 2024-03-04T06:20:45.365 INFO:tasks.rados.rados.0.smithi064.stdout:3047: writing smithi06431743-30 from 2945999 to 3244081 tid 3 2024-03-04T06:20:45.365 INFO:tasks.rados.rados.0.smithi064.stdout:3048: read oid 4 snap -1 2024-03-04T06:20:45.365 INFO:tasks.rados.rados.0.smithi064.stdout:3048: expect (ObjNum 1277 snap 346 seq_num 1277) 2024-03-04T06:20:45.365 INFO:tasks.rados.rados.0.smithi064.stdout:3049: read oid 42 snap -1 2024-03-04T06:20:45.366 INFO:tasks.rados.rados.0.smithi064.stdout:3049: expect deleted 2024-03-04T06:20:45.366 INFO:tasks.rados.rados.0.smithi064.stdout:3050: write oid 50 current snap is 388 2024-03-04T06:20:45.366 INFO:tasks.rados.rados.0.smithi064.stdout:3050: seq_num 1384 ranges {460888=520862,1471667=590474,2573422=455407} 2024-03-04T06:20:45.370 INFO:tasks.rados.rados.0.smithi064.stdout:3050: writing smithi06431743-50 from 460888 to 981750 tid 1 2024-03-04T06:20:45.376 INFO:tasks.rados.rados.0.smithi064.stdout:3050: writing smithi06431743-50 from 1471667 to 2062141 tid 2 2024-03-04T06:20:45.383 INFO:tasks.rados.rados.0.smithi064.stdout:3050: writing smithi06431743-50 from 2573422 to 3028829 tid 3 2024-03-04T06:20:45.383 INFO:tasks.rados.rados.0.smithi064.stdout:3051: read oid 49 snap 369 2024-03-04T06:20:45.383 INFO:tasks.rados.rados.0.smithi064.stdout:3051: expect (ObjNum 1333 snap 368 seq_num 1333) 2024-03-04T06:20:45.384 INFO:tasks.rados.rados.0.smithi064.stdout:3052: snap_create 2024-03-04T06:20:45.424 INFO:tasks.rados.rados.0.smithi064.stdout:3051: expect (ObjNum 1333 snap 368 seq_num 1333) 2024-03-04T06:20:45.629 INFO:tasks.rados.rados.0.smithi064.stdout:3050: finishing write tid 1 to smithi06431743-50 2024-03-04T06:20:45.629 INFO:tasks.rados.rados.0.smithi064.stdout:3047: finishing write tid 1 to smithi06431743-30 2024-03-04T06:20:45.629 INFO:tasks.rados.rados.0.smithi064.stdout:3050: finishing write tid 2 to smithi06431743-50 2024-03-04T06:20:45.629 INFO:tasks.rados.rados.0.smithi064.stdout:3047: finishing write tid 2 to smithi06431743-30 2024-03-04T06:20:45.629 INFO:tasks.rados.rados.0.smithi064.stdout:3047: finishing write tid 3 to smithi06431743-30 2024-03-04T06:20:45.629 INFO:tasks.rados.rados.0.smithi064.stdout:3050: finishing write tid 3 to smithi06431743-50 2024-03-04T06:20:45.629 INFO:tasks.rados.rados.0.smithi064.stdout:3047: finishing write tid 4 to smithi06431743-30 2024-03-04T06:20:45.629 INFO:tasks.rados.rados.0.smithi064.stdout:3047: finishing write tid 5 to smithi06431743-30 2024-03-04T06:20:45.629 INFO:tasks.rados.rados.0.smithi064.stdout:3047: oid 30 updating version 0 to 1555 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3047: oid 30 updating version 1555 to 1558 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3047: oid 30 version 1558 is already newer than 1557 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3047: oid 30 version 1558 is already newer than 1557 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 1558 (ObjNum 1383 snap 388 seq_num 1383) dirty exists 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3047: left oid 30 (ObjNum 1383 snap 388 seq_num 1383) 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3050: finishing write tid 4 to smithi06431743-50 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3050: finishing write tid 5 to smithi06431743-50 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3050: oid 50 updating version 0 to 984 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3050: oid 50 updating version 984 to 985 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3050: oid 50 updating version 985 to 986 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3050: oid 50 version 986 is already newer than 984 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 986 (ObjNum 1384 snap 388 seq_num 1384) dirty exists 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3050: left oid 50 (ObjNum 1384 snap 388 seq_num 1384) 2024-03-04T06:20:45.630 INFO:tasks.rados.rados.0.smithi064.stdout:3048: expect (ObjNum 1277 snap 346 seq_num 1277) 2024-03-04T06:20:46.362 INFO:tasks.rados.rados.0.smithi064.stdout:3047: done (5 left) 2024-03-04T06:20:46.363 INFO:tasks.rados.rados.0.smithi064.stdout:3048: done (4 left) 2024-03-04T06:20:46.363 INFO:tasks.rados.rados.0.smithi064.stdout:3049: done (3 left) 2024-03-04T06:20:46.363 INFO:tasks.rados.rados.0.smithi064.stdout:3050: done (2 left) 2024-03-04T06:20:46.363 INFO:tasks.rados.rados.0.smithi064.stdout:3051: done (1 left) 2024-03-04T06:20:46.363 INFO:tasks.rados.rados.0.smithi064.stdout:3052: done (0 left) 2024-03-04T06:20:46.363 INFO:tasks.rados.rados.0.smithi064.stdout:3053: snap_create 2024-03-04T06:20:47.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:47.366 INFO:tasks.rados.rados.0.smithi064.stdout:3053: done (0 left) 2024-03-04T06:20:47.367 INFO:tasks.rados.rados.0.smithi064.stdout:3054: delete oid 37 current snap is 390 2024-03-04T06:20:47.405 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:47.403+0000 7f9e3e0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:47.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:47.408+0000 7f9e3e0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:47.419 INFO:tasks.rados.rados.0.smithi064.stdout:3054: done (0 left) 2024-03-04T06:20:47.419 INFO:tasks.rados.rados.0.smithi064.stdout:3055: delete oid 40 current snap is 390 2024-03-04T06:20:47.428 INFO:tasks.rados.rados.0.smithi064.stdout:3055: done (0 left) 2024-03-04T06:20:47.428 INFO:tasks.rados.rados.0.smithi064.stdout:3056: read oid 17 snap 376 2024-03-04T06:20:47.428 INFO:tasks.rados.rados.0.smithi064.stdout:3056: expect deleted 2024-03-04T06:20:47.428 INFO:tasks.rados.rados.0.smithi064.stdout:3057: snap_create 2024-03-04T06:20:48.307 INFO:tasks.rados.rados.0.smithi064.stdout:3056: done (1 left) 2024-03-04T06:20:48.307 INFO:tasks.rados.rados.0.smithi064.stdout:3057: done (0 left) 2024-03-04T06:20:48.307 INFO:tasks.rados.rados.0.smithi064.stdout:3058: delete oid 26 current snap is 391 2024-03-04T06:20:48.359 INFO:tasks.rados.rados.0.smithi064.stdout:3058: done (0 left) 2024-03-04T06:20:48.359 INFO:tasks.rados.rados.0.smithi064.stdout:3059: read oid 22 snap -1 2024-03-04T06:20:48.359 INFO:tasks.rados.rados.0.smithi064.stdout:3059: expect deleted 2024-03-04T06:20:48.359 INFO:tasks.rados.rados.0.smithi064.stdout:3060: snap_create 2024-03-04T06:20:49.300 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:20:49.300 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:20:49.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:20:49.311 INFO:tasks.rados.rados.0.smithi064.stdout:3059: done (1 left) 2024-03-04T06:20:49.311 INFO:tasks.rados.rados.0.smithi064.stdout:3060: done (0 left) 2024-03-04T06:20:49.311 INFO:tasks.rados.rados.0.smithi064.stdout:3061: write (excl) oid 13 current snap is 392 2024-03-04T06:20:49.311 INFO:tasks.rados.rados.0.smithi064.stdout:3061: seq_num 1388 ranges {528116=421858,1589242=597812,2966761=361355} 2024-03-04T06:20:49.317 INFO:tasks.rados.rados.0.smithi064.stdout:3061: writing smithi06431743-13 from 528116 to 949974 tid 1 2024-03-04T06:20:49.325 INFO:tasks.rados.rados.0.smithi064.stdout:3061: writing smithi06431743-13 from 1589242 to 2187054 tid 2 2024-03-04T06:20:49.333 INFO:tasks.rados.rados.0.smithi064.stdout:3061: writing smithi06431743-13 from 2966761 to 3328116 tid 3 2024-03-04T06:20:49.333 INFO:tasks.rados.rados.0.smithi064.stdout:3062: snap_remove snap 389 2024-03-04T06:20:49.426 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:49.422+0000 7fd1dfa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:49.429 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:49.428+0000 7fd1dfa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:49.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:49.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:49.598+0000 7f782139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:49.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:49.602+0000 7f782139d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:20:49.623 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:20:49.638 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1161,"stamp":"2024-03-04T06:20:49.308871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":105981841,"num_objects":1027,"num_object_clones":828,"num_object_copies":3081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":161,"num_read":5608,"num_read_kb":4350753,"num_write":4757,"num_write_kb":1260179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12300,"ondisk_log_size":12300,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":722504,"kb_used_data":509736,"kb_used_omap":0,"kb_used_meta":212736,"kb_avail":376764856,"statfs":{"total":386547056640,"available":385807212544,"internally_reserved":0,"allocated":521969664,"data_stored":618549988,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":217841664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2722734,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":46,"num_read_kb":38577,"num_write":30,"num_write_kb":7792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.962358"},"pg_stats":[{"pgid":"3.f","version":"771'735","reported_seq":2662,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.388594+0000","last_change":"2024-03-04T06:20:42.479792+0000","last_active":"2024-03-04T06:20:45.388594+0000","last_peered":"2024-03-04T06:20:45.388594+0000","last_clean":"2024-03-04T06:20:45.388594+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:20:45.388594+0000","last_undegraded":"2024-03-04T06:20:45.388594+0000","last_fullsized":"2024-03-04T06:20:45.388594+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":735,"log_dups_size":0,"ondisk_log_size":735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":313,"num_read_kb":251966,"num_write":280,"num_write_kb":73822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"771'623","reported_seq":2636,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.379001+0000","last_change":"2024-03-04T06:20:42.022722+0000","last_active":"2024-03-04T06:20:45.379001+0000","last_peered":"2024-03-04T06:20:45.379001+0000","last_clean":"2024-03-04T06:20:45.379001+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:20:45.379001+0000","last_undegraded":"2024-03-04T06:20:45.379001+0000","last_fullsized":"2024-03-04T06:20:45.379001+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9480138,"num_objects":56,"num_object_clones":40,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":196,"num_read_kb":138500,"num_write":233,"num_write_kb":61051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"771'829","reported_seq":2787,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.394097+0000","last_change":"2024-03-04T06:20:42.399173+0000","last_active":"2024-03-04T06:20:45.394097+0000","last_peered":"2024-03-04T06:20:45.394097+0000","last_clean":"2024-03-04T06:20:45.394097+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:20:45.394097+0000","last_undegraded":"2024-03-04T06:20:45.394097+0000","last_fullsized":"2024-03-04T06:20:45.394097+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":829,"log_dups_size":0,"ondisk_log_size":829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9328842,"num_objects":79,"num_object_clones":64,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":451,"num_read_kb":359296,"num_write":346,"num_write_kb":93348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"771'708","reported_seq":2620,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.378474+0000","last_change":"2024-03-04T06:20:42.475659+0000","last_active":"2024-03-04T06:20:45.378474+0000","last_peered":"2024-03-04T06:20:45.378474+0000","last_clean":"2024-03-04T06:20:45.378474+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:20:45.378474+0000","last_undegraded":"2024-03-04T06:20:45.378474+0000","last_fullsized":"2024-03-04T06:20:45.378474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8522106,"num_objects":53,"num_object_clones":43,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":267,"num_read_kb":202839,"num_write":254,"num_write_kb":71600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"770'289","reported_seq":2435,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.368927+0000","last_change":"2024-03-04T06:20:42.004321+0000","last_active":"2024-03-04T06:20:45.368927+0000","last_peered":"2024-03-04T06:20:45.368927+0000","last_clean":"2024-03-04T06:20:45.368927+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:20:45.368927+0000","last_undegraded":"2024-03-04T06:20:45.368927+0000","last_fullsized":"2024-03-04T06:20:45.368927+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672643,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":65484,"num_write":101,"num_write_kb":27678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"772'842","reported_seq":2717,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.386495+0000","last_change":"2024-03-04T06:20:42.437027+0000","last_active":"2024-03-04T06:20:45.386495+0000","last_peered":"2024-03-04T06:20:45.386495+0000","last_clean":"2024-03-04T06:20:45.386495+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:20:45.386495+0000","last_undegraded":"2024-03-04T06:20:45.386495+0000","last_fullsized":"2024-03-04T06:20:45.386495+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":842,"log_dups_size":0,"ondisk_log_size":842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470959,"num_objects":63,"num_object_clones":54,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":374,"num_read_kb":306917,"num_write":309,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2272,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.387173+0000","last_change":"2024-03-04T06:20:41.937237+0000","last_active":"2024-03-04T06:20:45.387173+0000","last_peered":"2024-03-04T06:20:45.387173+0000","last_clean":"2024-03-04T06:20:45.387173+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:20:45.387173+0000","last_undegraded":"2024-03-04T06:20:45.387173+0000","last_fullsized":"2024-03-04T06:20:45.387173+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"771'465","reported_seq":2531,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.417559+0000","last_change":"2024-03-04T06:20:42.369724+0000","last_active":"2024-03-04T06:20:45.417559+0000","last_peered":"2024-03-04T06:20:45.417559+0000","last_clean":"2024-03-04T06:20:45.417559+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:20:45.417559+0000","last_undegraded":"2024-03-04T06:20:45.417559+0000","last_fullsized":"2024-03-04T06:20:45.417559+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5530163,"num_objects":35,"num_object_clones":26,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":241,"num_read_kb":224065,"num_write":158,"num_write_kb":42246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2081,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.363461+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:20:45.363461+0000","last_peered":"2024-03-04T06:20:45.363461+0000","last_clean":"2024-03-04T06:20:45.363461+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:20:45.363461+0000","last_undegraded":"2024-03-04T06:20:45.363461+0000","last_fullsized":"2024-03-04T06:20:45.363461+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1920,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.405133+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:20:45.405133+0000","last_peered":"2024-03-04T06:20:45.405133+0000","last_clean":"2024-03-04T06:20:45.405133+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:20:45.405133+0000","last_undegraded":"2024-03-04T06:20:45.405133+0000","last_fullsized":"2024-03-04T06:20:45.405133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"770'754","reported_seq":2701,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.401620+0000","last_change":"2024-03-04T06:20:42.374253+0000","last_active":"2024-03-04T06:20:45.401620+0000","last_peered":"2024-03-04T06:20:45.401620+0000","last_clean":"2024-03-04T06:20:45.401620+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:20:45.401620+0000","last_undegraded":"2024-03-04T06:20:45.401620+0000","last_fullsized":"2024-03-04T06:20:45.401620+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7463298,"num_objects":65,"num_object_clones":51,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":358,"num_read_kb":286050,"num_write":292,"num_write_kb":78583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1914,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.384816+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:20:45.384816+0000","last_peered":"2024-03-04T06:20:45.384816+0000","last_clean":"2024-03-04T06:20:45.384816+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:20:45.384816+0000","last_undegraded":"2024-03-04T06:20:45.384816+0000","last_fullsized":"2024-03-04T06:20:45.384816+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"770'483","reported_seq":2586,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.354985+0000","last_change":"2024-03-04T06:20:42.014621+0000","last_active":"2024-03-04T06:20:45.354985+0000","last_peered":"2024-03-04T06:20:45.354985+0000","last_clean":"2024-03-04T06:20:45.354985+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:20:45.354985+0000","last_undegraded":"2024-03-04T06:20:45.354985+0000","last_fullsized":"2024-03-04T06:20:45.354985+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698233,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":270,"num_read_kb":221303,"num_write":175,"num_write_kb":49293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1920,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.396619+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:20:45.396619+0000","last_peered":"2024-03-04T06:20:45.396619+0000","last_clean":"2024-03-04T06:20:45.396619+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:20:45.396619+0000","last_undegraded":"2024-03-04T06:20:45.396619+0000","last_fullsized":"2024-03-04T06:20:45.396619+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"770'1357","reported_seq":3121,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.369679+0000","last_change":"2024-03-04T06:20:42.012608+0000","last_active":"2024-03-04T06:20:45.369679+0000","last_peered":"2024-03-04T06:20:45.369679+0000","last_clean":"2024-03-04T06:20:45.369679+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:20:45.369679+0000","last_undegraded":"2024-03-04T06:20:45.369679+0000","last_fullsized":"2024-03-04T06:20:45.369679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1357,"log_dups_size":0,"ondisk_log_size":1357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10287171,"num_objects":118,"num_object_clones":94,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":713,"num_read_kb":567605,"num_write":548,"num_write_kb":146807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1914,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.377087+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:20:45.377087+0000","last_peered":"2024-03-04T06:20:45.377087+0000","last_clean":"2024-03-04T06:20:45.377087+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:20:45.377087+0000","last_undegraded":"2024-03-04T06:20:45.377087+0000","last_fullsized":"2024-03-04T06:20:45.377087+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"771'1553","reported_seq":3115,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.372162+0000","last_change":"2024-03-04T06:20:42.015266+0000","last_active":"2024-03-04T06:20:45.372162+0000","last_peered":"2024-03-04T06:20:45.372162+0000","last_clean":"2024-03-04T06:20:45.372162+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:20:45.372162+0000","last_undegraded":"2024-03-04T06:20:45.372162+0000","last_fullsized":"2024-03-04T06:20:45.372162+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1553,"log_dups_size":0,"ondisk_log_size":1553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17447217,"num_objects":127,"num_object_clones":104,"num_object_copies":381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":573,"num_read_kb":418112,"num_write":587,"num_write_kb":158133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"770'753","reported_seq":2709,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.402831+0000","last_change":"2024-03-04T06:20:42.384368+0000","last_active":"2024-03-04T06:20:45.402831+0000","last_peered":"2024-03-04T06:20:45.402831+0000","last_clean":"2024-03-04T06:20:45.402831+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:20:45.402831+0000","last_undegraded":"2024-03-04T06:20:45.402831+0000","last_fullsized":"2024-03-04T06:20:45.402831+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042397,"num_objects":60,"num_object_clones":48,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":394,"num_read_kb":329119,"num_write":278,"num_write_kb":75144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1982,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.382962+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:20:45.382962+0000","last_peered":"2024-03-04T06:20:45.382962+0000","last_clean":"2024-03-04T06:20:45.382962+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:20:45.382962+0000","last_undegraded":"2024-03-04T06:20:45.382962+0000","last_fullsized":"2024-03-04T06:20:45.382962+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"770'799","reported_seq":2758,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.385791+0000","last_change":"2024-03-04T06:20:42.011264+0000","last_active":"2024-03-04T06:20:45.385791+0000","last_peered":"2024-03-04T06:20:45.385791+0000","last_clean":"2024-03-04T06:20:45.385791+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:20:45.385791+0000","last_undegraded":"2024-03-04T06:20:45.385791+0000","last_fullsized":"2024-03-04T06:20:45.385791+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7883473,"num_objects":72,"num_object_clones":56,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":335,"num_read_kb":241316,"num_write":305,"num_write_kb":82364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1948,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.370787+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:20:45.370787+0000","last_peered":"2024-03-04T06:20:45.370787+0000","last_clean":"2024-03-04T06:20:45.370787+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:20:45.370787+0000","last_undegraded":"2024-03-04T06:20:45.370787+0000","last_fullsized":"2024-03-04T06:20:45.370787+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"770'1032","reported_seq":2845,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.360985+0000","last_change":"2024-03-04T06:20:41.987524+0000","last_active":"2024-03-04T06:20:45.360985+0000","last_peered":"2024-03-04T06:20:45.360985+0000","last_clean":"2024-03-04T06:20:45.360985+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:20:45.360985+0000","last_undegraded":"2024-03-04T06:20:45.360985+0000","last_fullsized":"2024-03-04T06:20:45.360985+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1032,"log_dups_size":0,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12673699,"num_objects":84,"num_object_clones":70,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":428,"num_read_kb":315321,"num_write":394,"num_write_kb":110146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1982,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.380025+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:20:45.380025+0000","last_peered":"2024-03-04T06:20:45.380025+0000","last_clean":"2024-03-04T06:20:45.380025+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:20:45.380025+0000","last_undegraded":"2024-03-04T06:20:45.380025+0000","last_fullsized":"2024-03-04T06:20:45.380025+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"769'981","reported_seq":2892,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.381257+0000","last_change":"2024-03-04T06:20:41.944249+0000","last_active":"2024-03-04T06:20:45.381257+0000","last_peered":"2024-03-04T06:20:45.381257+0000","last_clean":"2024-03-04T06:20:45.381257+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:20:45.381257+0000","last_undegraded":"2024-03-04T06:20:45.381257+0000","last_fullsized":"2024-03-04T06:20:45.381257+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":981,"log_dups_size":0,"ondisk_log_size":981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2891134,"num_objects":80,"num_object_clones":70,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":501,"num_read_kb":422769,"num_write":360,"num_write_kb":98417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1920,"reported_epoch":774,"state":"active+clean","last_fresh":"2024-03-04T06:20:45.404192+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:20:45.404192+0000","last_peered":"2024-03-04T06:20:45.404192+0000","last_clean":"2024-03-04T06:20:45.404192+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:20:45.404192+0000","last_undegraded":"2024-03-04T06:20:45.404192+0000","last_fullsized":"2024-03-04T06:20:45.404192+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":105391473,"num_objects":1025,"num_object_clones":828,"num_object_copies":3075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":161,"num_read":5505,"num_read_kb":4350662,"num_write":4620,"num_write_kb":1258285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12203,"ondisk_log_size":12203,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":216964,"kb_used_data":159612,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94154876,"statfs":{"total":96636764160,"available":96414593024,"internally_reserved":0,"allocated":163442688,"data_stored":194579847,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":168640,"kb_used_data":115448,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94203200,"statfs":{"total":96636764160,"available":96464076800,"internally_reserved":0,"allocated":118218752,"data_stored":134107472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":161324,"kb_used_data":108132,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94210516,"statfs":{"total":96636764160,"available":96471568384,"internally_reserved":0,"allocated":110727168,"data_stored":131311636,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":175576,"kb_used_data":126544,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94196264,"statfs":{"total":96636764160,"available":96456974336,"internally_reserved":0,"allocated":129581056,"data_stored":158551033,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:20:50.311 INFO:tasks.rados.rados.0.smithi064.stdout:3062: done (1 left) 2024-03-04T06:20:50.311 INFO:tasks.rados.rados.0.smithi064.stdout:3063: write oid 5 current snap is 392 2024-03-04T06:20:50.311 INFO:tasks.rados.rados.0.smithi064.stdout:3061: finishing write tid 1 to smithi06431743-13 2024-03-04T06:20:50.311 INFO:tasks.rados.rados.0.smithi064.stdout:3063: seq_num 1389 ranges {544923=497107,1647724=457823,2710537=428120} 2024-03-04T06:20:50.316 INFO:tasks.rados.rados.0.smithi064.stdout:3063: writing smithi06431743-5 from 544923 to 1042030 tid 1 2024-03-04T06:20:50.322 INFO:tasks.rados.rados.0.smithi064.stdout:3063: writing smithi06431743-5 from 1647724 to 2105547 tid 2 2024-03-04T06:20:50.328 INFO:tasks.rados.rados.0.smithi064.stdout:3063: writing smithi06431743-5 from 2710537 to 3138657 tid 3 2024-03-04T06:20:50.328 INFO:tasks.rados.rados.0.smithi064.stdout:3064: read oid 10 snap -1 2024-03-04T06:20:50.328 INFO:tasks.rados.rados.0.smithi064.stdout:3064: expect deleted 2024-03-04T06:20:50.328 INFO:tasks.rados.rados.0.smithi064.stdout:3065: write (excl) oid 12 current snap is 392 2024-03-04T06:20:50.328 INFO:tasks.rados.rados.0.smithi064.stdout:3061: finishing write tid 2 to smithi06431743-13 2024-03-04T06:20:50.328 INFO:tasks.rados.rados.0.smithi064.stdout:3061: finishing write tid 3 to smithi06431743-13 2024-03-04T06:20:50.328 INFO:tasks.rados.rados.0.smithi064.stdout:3065: seq_num 1390 ranges {561730=572356,1706206=601481,3021607=340123} 2024-03-04T06:20:50.334 INFO:tasks.rados.rados.0.smithi064.stdout:3065: writing smithi06431743-12 from 561730 to 1134086 tid 1 2024-03-04T06:20:50.341 INFO:tasks.rados.rados.0.smithi064.stdout:3065: writing smithi06431743-12 from 1706206 to 2307687 tid 2 2024-03-04T06:20:50.348 INFO:tasks.rados.rados.0.smithi064.stdout:3065: writing smithi06431743-12 from 3021607 to 3361730 tid 3 2024-03-04T06:20:50.348 INFO:tasks.rados.rados.0.smithi064.stdout:3066: snap_create 2024-03-04T06:20:50.348 INFO:tasks.rados.rados.0.smithi064.stdout:3061: finishing write tid 4 to smithi06431743-13 2024-03-04T06:20:50.348 INFO:tasks.rados.rados.0.smithi064.stdout:3061: finishing write tid 5 to smithi06431743-13 2024-03-04T06:20:50.348 INFO:tasks.rados.rados.0.smithi064.stdout:3061: oid 13 updating version 0 to 1561 2024-03-04T06:20:50.348 INFO:tasks.rados.rados.0.smithi064.stdout:3061: oid 13 updating version 1561 to 1563 2024-03-04T06:20:50.349 INFO:tasks.rados.rados.0.smithi064.stdout:3061: oid 13 version 1563 is already newer than 1563 2024-03-04T06:20:50.349 INFO:tasks.rados.rados.0.smithi064.stdout:3061: oid 13 version 1563 is already newer than 1560 2024-03-04T06:20:50.349 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1563 (ObjNum 1388 snap 392 seq_num 1388) dirty exists 2024-03-04T06:20:50.349 INFO:tasks.rados.rados.0.smithi064.stdout:3061: left oid 13 (ObjNum 1388 snap 392 seq_num 1388) 2024-03-04T06:20:50.397 INFO:tasks.rados.rados.0.smithi064.stdout:3063: finishing write tid 1 to smithi06431743-5 2024-03-04T06:20:50.401 INFO:tasks.rados.rados.0.smithi064.stdout:3065: finishing write tid 1 to smithi06431743-12 2024-03-04T06:20:50.423 INFO:tasks.rados.rados.0.smithi064.stdout:3063: finishing write tid 2 to smithi06431743-5 2024-03-04T06:20:50.426 INFO:tasks.rados.rados.0.smithi064.stdout:3063: finishing write tid 3 to smithi06431743-5 2024-03-04T06:20:50.427 INFO:tasks.rados.rados.0.smithi064.stdout:3065: finishing write tid 2 to smithi06431743-12 2024-03-04T06:20:50.436 INFO:tasks.rados.rados.0.smithi064.stdout:3065: finishing write tid 3 to smithi06431743-12 2024-03-04T06:20:50.441 INFO:tasks.rados.rados.0.smithi064.stdout:3063: finishing write tid 4 to smithi06431743-5 2024-03-04T06:20:50.444 INFO:tasks.rados.rados.0.smithi064.stdout:3063: finishing write tid 5 to smithi06431743-5 2024-03-04T06:20:50.445 INFO:tasks.rados.rados.0.smithi064.stdout:3063: oid 5 updating version 0 to 1566 2024-03-04T06:20:50.445 INFO:tasks.rados.rados.0.smithi064.stdout:3063: oid 5 updating version 1566 to 1568 2024-03-04T06:20:50.445 INFO:tasks.rados.rados.0.smithi064.stdout:3063: oid 5 updating version 1568 to 1569 2024-03-04T06:20:50.445 INFO:tasks.rados.rados.0.smithi064.stdout:3063: oid 5 version 1569 is already newer than 1568 2024-03-04T06:20:50.445 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 1569 (ObjNum 1389 snap 392 seq_num 1389) dirty exists 2024-03-04T06:20:50.445 INFO:tasks.rados.rados.0.smithi064.stdout:3063: left oid 5 (ObjNum 1389 snap 392 seq_num 1389) 2024-03-04T06:20:50.446 INFO:tasks.rados.rados.0.smithi064.stdout:3065: finishing write tid 4 to smithi06431743-12 2024-03-04T06:20:50.447 INFO:tasks.rados.rados.0.smithi064.stdout:3065: finishing write tid 5 to smithi06431743-12 2024-03-04T06:20:50.447 INFO:tasks.rados.rados.0.smithi064.stdout:3065: oid 12 updating version 0 to 713 2024-03-04T06:20:50.447 INFO:tasks.rados.rados.0.smithi064.stdout:3065: oid 12 version 713 is already newer than 712 2024-03-04T06:20:50.447 INFO:tasks.rados.rados.0.smithi064.stdout:3065: oid 12 version 713 is already newer than 712 2024-03-04T06:20:50.448 INFO:tasks.rados.rados.0.smithi064.stdout:3065: oid 12 version 713 is already newer than 710 2024-03-04T06:20:50.448 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 713 (ObjNum 1390 snap 392 seq_num 1390) dirty exists 2024-03-04T06:20:50.448 INFO:tasks.rados.rados.0.smithi064.stdout:3065: left oid 12 (ObjNum 1390 snap 392 seq_num 1390) 2024-03-04T06:20:51.317 INFO:tasks.rados.rados.0.smithi064.stdout:3061: done (4 left) 2024-03-04T06:20:51.317 INFO:tasks.rados.rados.0.smithi064.stdout:3063: done (3 left) 2024-03-04T06:20:51.317 INFO:tasks.rados.rados.0.smithi064.stdout:3064: done (2 left) 2024-03-04T06:20:51.317 INFO:tasks.rados.rados.0.smithi064.stdout:3065: done (1 left) 2024-03-04T06:20:51.317 INFO:tasks.rados.rados.0.smithi064.stdout:3066: done (0 left) 2024-03-04T06:20:51.317 INFO:tasks.rados.rados.0.smithi064.stdout:3067: snap_create 2024-03-04T06:20:51.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:51.794 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:51.792+0000 7fb11999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:51.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:51.796+0000 7fb11999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:52.318 INFO:tasks.rados.rados.0.smithi064.stdout:3067: done (0 left) 2024-03-04T06:20:52.318 INFO:tasks.rados.rados.0.smithi064.stdout:3068: read oid 19 snap 378 2024-03-04T06:20:52.318 INFO:tasks.rados.rados.0.smithi064.stdout:3068: expect (ObjNum 1327 snap 368 seq_num 1327) 2024-03-04T06:20:52.318 INFO:tasks.rados.rados.0.smithi064.stdout:3069: write oid 16 current snap is 394 2024-03-04T06:20:52.318 INFO:tasks.rados.rados.0.smithi064.stdout:3069: seq_num 1391 ranges {578537=647605,1764688=745139,3049030=329507} 2024-03-04T06:20:52.324 INFO:tasks.rados.rados.0.smithi064.stdout:3069: writing smithi06431743-16 from 578537 to 1226142 tid 1 2024-03-04T06:20:52.331 INFO:tasks.rados.rados.0.smithi064.stdout:3069: writing smithi06431743-16 from 1764688 to 2509827 tid 2 2024-03-04T06:20:52.336 INFO:tasks.rados.rados.0.smithi064.stdout:3069: writing smithi06431743-16 from 3049030 to 3378537 tid 3 2024-03-04T06:20:52.336 INFO:tasks.rados.rados.0.smithi064.stdout:3070: write (excl) oid 22 current snap is 394 2024-03-04T06:20:52.336 INFO:tasks.rados.rados.0.smithi064.stdout:3070: seq_num 1392 ranges {595344=439207,1823170=605150,3076453=318891} 2024-03-04T06:20:52.342 INFO:tasks.rados.rados.0.smithi064.stdout:3070: writing smithi06431743-22 from 595344 to 1034551 tid 1 2024-03-04T06:20:52.349 INFO:tasks.rados.rados.0.smithi064.stdout:3070: writing smithi06431743-22 from 1823170 to 2428320 tid 2 2024-03-04T06:20:52.355 INFO:tasks.rados.rados.0.smithi064.stdout:3070: writing smithi06431743-22 from 3076453 to 3395344 tid 3 2024-03-04T06:20:52.355 INFO:tasks.rados.rados.0.smithi064.stdout:3071: delete oid 24 current snap is 394 2024-03-04T06:20:52.387 INFO:tasks.rados.rados.0.smithi064.stdout:3071: done (3 left) 2024-03-04T06:20:52.387 INFO:tasks.rados.rados.0.smithi064.stdout:3072: snap_create 2024-03-04T06:20:52.389 INFO:tasks.rados.rados.0.smithi064.stdout:3070: finishing write tid 1 to smithi06431743-22 2024-03-04T06:20:52.396 INFO:tasks.rados.rados.0.smithi064.stdout:3068: expect (ObjNum 1327 snap 368 seq_num 1327) 2024-03-04T06:20:52.567 INFO:tasks.rados.rados.0.smithi064.stdout:3070: finishing write tid 2 to smithi06431743-22 2024-03-04T06:20:52.567 INFO:tasks.rados.rados.0.smithi064.stdout:3069: finishing write tid 1 to smithi06431743-16 2024-03-04T06:20:52.567 INFO:tasks.rados.rados.0.smithi064.stdout:3070: finishing write tid 3 to smithi06431743-22 2024-03-04T06:20:52.567 INFO:tasks.rados.rados.0.smithi064.stdout:3069: finishing write tid 2 to smithi06431743-16 2024-03-04T06:20:52.567 INFO:tasks.rados.rados.0.smithi064.stdout:3070: finishing write tid 4 to smithi06431743-22 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3070: finishing write tid 5 to smithi06431743-22 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3070: oid 22 updating version 0 to 990 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3070: oid 22 version 990 is already newer than 988 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3070: oid 22 version 990 is already newer than 989 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3070: oid 22 version 990 is already newer than 987 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 990 (ObjNum 1392 snap 394 seq_num 1392) dirty exists 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3070: left oid 22 (ObjNum 1392 snap 394 seq_num 1392) 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3069: finishing write tid 3 to smithi06431743-16 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3069: finishing write tid 4 to smithi06431743-16 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3069: finishing write tid 5 to smithi06431743-16 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3069: oid 16 updating version 0 to 801 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3069: oid 16 updating version 801 to 803 2024-03-04T06:20:52.568 INFO:tasks.rados.rados.0.smithi064.stdout:3069: oid 16 updating version 803 to 804 2024-03-04T06:20:52.569 INFO:tasks.rados.rados.0.smithi064.stdout:3069: oid 16 version 804 is already newer than 802 2024-03-04T06:20:52.569 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 804 (ObjNum 1391 snap 394 seq_num 1391) dirty exists 2024-03-04T06:20:52.569 INFO:tasks.rados.rados.0.smithi064.stdout:3069: left oid 16 (ObjNum 1391 snap 394 seq_num 1391) 2024-03-04T06:20:53.328 INFO:tasks.rados.rados.0.smithi064.stdout:3068: done (3 left) 2024-03-04T06:20:53.328 INFO:tasks.rados.rados.0.smithi064.stdout:3069: done (2 left) 2024-03-04T06:20:53.329 INFO:tasks.rados.rados.0.smithi064.stdout:3070: done (1 left) 2024-03-04T06:20:53.329 INFO:tasks.rados.rados.0.smithi064.stdout:3072: done (0 left) 2024-03-04T06:20:53.329 INFO:tasks.rados.rados.0.smithi064.stdout:3073: write (excl) oid 14 current snap is 395 2024-03-04T06:20:53.329 INFO:tasks.rados.rados.0.smithi064.stdout:3073: seq_num 1394 ranges {628958=589705,1940134=608819,3131299=297659} 2024-03-04T06:20:53.334 INFO:tasks.rados.rados.0.smithi064.stdout:3073: writing smithi06431743-14 from 628958 to 1218663 tid 1 2024-03-04T06:20:53.341 INFO:tasks.rados.rados.0.smithi064.stdout:3073: writing smithi06431743-14 from 1940134 to 2548953 tid 2 2024-03-04T06:20:53.347 INFO:tasks.rados.rados.0.smithi064.stdout:3073: writing smithi06431743-14 from 3131299 to 3428958 tid 3 2024-03-04T06:20:53.347 INFO:tasks.rados.rados.0.smithi064.stdout:3074: snap_create 2024-03-04T06:20:53.387 INFO:tasks.rados.rados.0.smithi064.stdout:3073: finishing write tid 1 to smithi06431743-14 2024-03-04T06:20:53.408 INFO:tasks.rados.rados.0.smithi064.stdout:3073: finishing write tid 2 to smithi06431743-14 2024-03-04T06:20:53.417 INFO:tasks.rados.rados.0.smithi064.stdout:3073: finishing write tid 3 to smithi06431743-14 2024-03-04T06:20:53.433 INFO:tasks.rados.rados.0.smithi064.stdout:3073: finishing write tid 4 to smithi06431743-14 2024-03-04T06:20:53.434 INFO:tasks.rados.rados.0.smithi064.stdout:3073: finishing write tid 5 to smithi06431743-14 2024-03-04T06:20:53.434 INFO:tasks.rados.rados.0.smithi064.stdout:3073: oid 14 updating version 0 to 1036 2024-03-04T06:20:53.434 INFO:tasks.rados.rados.0.smithi064.stdout:3073: oid 14 updating version 1036 to 1039 2024-03-04T06:20:53.434 INFO:tasks.rados.rados.0.smithi064.stdout:3073: oid 14 version 1039 is already newer than 1037 2024-03-04T06:20:53.434 INFO:tasks.rados.rados.0.smithi064.stdout:3073: oid 14 version 1039 is already newer than 1038 2024-03-04T06:20:53.434 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 1039 (ObjNum 1394 snap 395 seq_num 1394) dirty exists 2024-03-04T06:20:53.435 INFO:tasks.rados.rados.0.smithi064.stdout:3073: left oid 14 (ObjNum 1394 snap 395 seq_num 1394) 2024-03-04T06:20:53.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:53.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:53.986+0000 7f3f6e8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:53.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:53.990+0000 7f3f6e8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:54.327 INFO:tasks.rados.rados.0.smithi064.stdout:3073: done (1 left) 2024-03-04T06:20:54.327 INFO:tasks.rados.rados.0.smithi064.stdout:3074: done (0 left) 2024-03-04T06:20:54.327 INFO:tasks.rados.rados.0.smithi064.stdout:3075: read oid 43 snap -1 2024-03-04T06:20:54.327 INFO:tasks.rados.rados.0.smithi064.stdout:3075: expect deleted 2024-03-04T06:20:54.327 INFO:tasks.rados.rados.0.smithi064.stdout:3076: delete oid 23 current snap is 396 2024-03-04T06:20:54.394 INFO:tasks.rados.rados.0.smithi064.stdout:3075: done (1 left) 2024-03-04T06:20:54.394 INFO:tasks.rados.rados.0.smithi064.stdout:3076: done (0 left) 2024-03-04T06:20:54.394 INFO:tasks.rados.rados.0.smithi064.stdout:3077: write (excl) oid 18 current snap is 396 2024-03-04T06:20:54.394 INFO:tasks.rados.rados.0.smithi064.stdout:3077: seq_num 1396 ranges {662572=740203,1940745=612488,3069792=392780} 2024-03-04T06:20:54.402 INFO:tasks.rados.rados.0.smithi064.stdout:3077: writing smithi06431743-18 from 662572 to 1402775 tid 1 2024-03-04T06:20:54.408 INFO:tasks.rados.rados.0.smithi064.stdout:3077: writing smithi06431743-18 from 1940745 to 2553233 tid 2 2024-03-04T06:20:54.414 INFO:tasks.rados.rados.0.smithi064.stdout:3077: writing smithi06431743-18 from 3069792 to 3462572 tid 3 2024-03-04T06:20:54.414 INFO:tasks.rados.rados.0.smithi064.stdout:3078: read oid 36 snap -1 2024-03-04T06:20:54.414 INFO:tasks.rados.rados.0.smithi064.stdout:3078: expect (ObjNum 1355 snap 377 seq_num 1355) 2024-03-04T06:20:54.414 INFO:tasks.rados.rados.0.smithi064.stdout:3079: write (excl) oid 11 current snap is 396 2024-03-04T06:20:54.414 INFO:tasks.rados.rados.0.smithi064.stdout:3079: seq_num 1397 ranges {679379=415452,1599227=756146,3097215=382164} 2024-03-04T06:20:54.419 INFO:tasks.rados.rados.0.smithi064.stdout:3079: writing smithi06431743-11 from 679379 to 1094831 tid 1 2024-03-04T06:20:54.426 INFO:tasks.rados.rados.0.smithi064.stdout:3079: writing smithi06431743-11 from 1599227 to 2355373 tid 2 2024-03-04T06:20:54.433 INFO:tasks.rados.rados.0.smithi064.stdout:3079: writing smithi06431743-11 from 3097215 to 3479379 tid 3 2024-03-04T06:20:54.433 INFO:tasks.rados.rados.0.smithi064.stdout:3077: finishing write tid 1 to smithi06431743-18 2024-03-04T06:20:54.433 INFO:tasks.rados.rados.0.smithi064.stdout:3080: read oid 45 snap 392 2024-03-04T06:20:54.433 INFO:tasks.rados.rados.0.smithi064.stdout:3080: expect (ObjNum 1358 snap 378 seq_num 1358) 2024-03-04T06:20:54.433 INFO:tasks.rados.rados.0.smithi064.stdout:3081: write (excl) oid 20 current snap is 396 2024-03-04T06:20:54.433 INFO:tasks.rados.rados.0.smithi064.stdout:3081: seq_num 1398 ranges {696186=490701,1657709=616157,2724638=562274} 2024-03-04T06:20:54.439 INFO:tasks.rados.rados.0.smithi064.stdout:3081: writing smithi06431743-20 from 696186 to 1186887 tid 1 2024-03-04T06:20:54.445 INFO:tasks.rados.rados.0.smithi064.stdout:3081: writing smithi06431743-20 from 1657709 to 2273866 tid 2 2024-03-04T06:20:54.452 INFO:tasks.rados.rados.0.smithi064.stdout:3081: writing smithi06431743-20 from 2724638 to 3286912 tid 3 2024-03-04T06:20:54.452 INFO:tasks.rados.rados.0.smithi064.stdout:3082: delete oid 9 current snap is 396 2024-03-04T06:20:54.452 INFO:tasks.rados.rados.0.smithi064.stdout:3078: expect (ObjNum 1355 snap 377 seq_num 1355) 2024-03-04T06:20:54.566 INFO:tasks.rados.rados.0.smithi064.stdout:3078: done (5 left) 2024-03-04T06:20:54.566 INFO:tasks.rados.rados.0.smithi064.stdout:3082: done (4 left) 2024-03-04T06:20:54.566 INFO:tasks.rados.rados.0.smithi064.stdout:3083: write oid 4 current snap is 396 2024-03-04T06:20:54.566 INFO:tasks.rados.rados.0.smithi064.stdout:3077: finishing write tid 2 to smithi06431743-18 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3079: finishing write tid 1 to smithi06431743-11 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3077: finishing write tid 3 to smithi06431743-18 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3079: finishing write tid 2 to smithi06431743-11 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3077: finishing write tid 4 to smithi06431743-18 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3077: finishing write tid 5 to smithi06431743-18 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3077: oid 18 updating version 0 to 631 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3077: oid 18 updating version 631 to 632 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3077: oid 18 version 632 is already newer than 629 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3077: oid 18 version 632 is already newer than 630 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 632 (ObjNum 1396 snap 396 seq_num 1396) dirty exists 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3077: left oid 18 (ObjNum 1396 snap 396 seq_num 1396) 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3081: finishing write tid 1 to smithi06431743-20 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3079: finishing write tid 3 to smithi06431743-11 2024-03-04T06:20:54.567 INFO:tasks.rados.rados.0.smithi064.stdout:3079: finishing write tid 4 to smithi06431743-11 2024-03-04T06:20:54.568 INFO:tasks.rados.rados.0.smithi064.stdout:3081: finishing write tid 2 to smithi06431743-20 2024-03-04T06:20:54.568 INFO:tasks.rados.rados.0.smithi064.stdout:3079: finishing write tid 5 to smithi06431743-11 2024-03-04T06:20:54.568 INFO:tasks.rados.rados.0.smithi064.stdout:3079: oid 11 updating version 0 to 832 2024-03-04T06:20:54.568 INFO:tasks.rados.rados.0.smithi064.stdout:3079: oid 11 version 832 is already newer than 831 2024-03-04T06:20:54.568 INFO:tasks.rados.rados.0.smithi064.stdout:3079: oid 11 updating version 832 to 834 2024-03-04T06:20:54.568 INFO:tasks.rados.rados.0.smithi064.stdout:3079: oid 11 version 834 is already newer than 833 2024-03-04T06:20:54.568 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 834 (ObjNum 1397 snap 396 seq_num 1397) dirty exists 2024-03-04T06:20:54.568 INFO:tasks.rados.rados.0.smithi064.stdout:3079: left oid 11 (ObjNum 1397 snap 396 seq_num 1397) 2024-03-04T06:20:54.568 INFO:tasks.rados.rados.0.smithi064.stdout:3083: seq_num 1400 ranges {729800=757552,2174673=619826} 2024-03-04T06:20:54.573 INFO:tasks.rados.rados.0.smithi064.stdout:3083: writing smithi06431743-4 from 729800 to 1487352 tid 1 2024-03-04T06:20:54.581 INFO:tasks.rados.rados.0.smithi064.stdout:3083: writing smithi06431743-4 from 2174673 to 2794499 tid 2 2024-03-04T06:20:54.581 INFO:tasks.rados.rados.0.smithi064.stdout:3077: done (4 left) 2024-03-04T06:20:54.581 INFO:tasks.rados.rados.0.smithi064.stdout:3079: done (3 left) 2024-03-04T06:20:54.581 INFO:tasks.rados.rados.0.smithi064.stdout:3084: delete oid 40 current snap is 396 2024-03-04T06:20:54.581 INFO:tasks.rados.rados.0.smithi064.stdout:3081: finishing write tid 3 to smithi06431743-20 2024-03-04T06:20:54.582 INFO:tasks.rados.rados.0.smithi064.stdout:3080: expect (ObjNum 1358 snap 378 seq_num 1358) 2024-03-04T06:20:54.640 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:20:54.640 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:20:54.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:20:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:54.743+0000 7fa1632d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:54.750 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:54.750+0000 7fa1632d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:3081: finishing write tid 4 to smithi06431743-20 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:3081: finishing write tid 5 to smithi06431743-20 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:3081: oid 20 updating version 0 to 1042 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:3081: oid 20 updating version 1042 to 1043 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:3081: oid 20 version 1043 is already newer than 1041 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:3081: oid 20 updating version 1043 to 1044 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 1044 (ObjNum 1398 snap 396 seq_num 1398) dirty exists 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:3081: left oid 20 (ObjNum 1398 snap 396 seq_num 1398) 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:3083: finishing write tid 1 to smithi06431743-4 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:3083: finishing write tid 2 to smithi06431743-4 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:3080: done (3 left) 2024-03-04T06:20:54.818 INFO:tasks.rados.rados.0.smithi064.stdout:3081: done (2 left) 2024-03-04T06:20:54.819 INFO:tasks.rados.rados.0.smithi064.stdout:3084: done (1 left) 2024-03-04T06:20:54.819 INFO:tasks.rados.rados.0.smithi064.stdout:3085: snap_remove snap 395 2024-03-04T06:20:54.940 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:20:54.940 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:20:54.952 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1169,"stamp":"2024-03-04T06:20:54.873152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101392932,"num_objects":1034,"num_object_clones":834,"num_object_copies":3102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":163,"num_read":5620,"num_read_kb":4358398,"num_write":4776,"num_write_kb":1265546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12326,"ondisk_log_size":12326,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":734040,"kb_used_data":521272,"kb_used_omap":0,"kb_used_meta":212736,"kb_avail":376753320,"statfs":{"total":386547056640,"available":385795399680,"internally_reserved":0,"allocated":533782528,"data_stored":636539128,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":217841664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2454119,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000951"},"pg_stats":[{"pgid":"3.f","version":"776'736","reported_seq":2676,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.348323+0000","last_change":"2024-03-04T06:20:50.356034+0000","last_active":"2024-03-04T06:20:50.348323+0000","last_peered":"2024-03-04T06:20:50.348323+0000","last_clean":"2024-03-04T06:20:50.348323+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:20:50.348323+0000","last_undegraded":"2024-03-04T06:20:50.348323+0000","last_fullsized":"2024-03-04T06:20:50.348323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":313,"num_read_kb":251966,"num_write":280,"num_write_kb":73822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"779'627","reported_seq":2653,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.319797+0000","last_change":"2024-03-04T06:20:50.367593+0000","last_active":"2024-03-04T06:20:50.319797+0000","last_peered":"2024-03-04T06:20:50.319797+0000","last_clean":"2024-03-04T06:20:50.319797+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:20:50.319797+0000","last_undegraded":"2024-03-04T06:20:50.319797+0000","last_fullsized":"2024-03-04T06:20:50.319797+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6404127,"num_objects":57,"num_object_clones":41,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":196,"num_read_kb":138500,"num_write":234,"num_write_kb":61051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"771'829","reported_seq":2801,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.360272+0000","last_change":"2024-03-04T06:20:50.366827+0000","last_active":"2024-03-04T06:20:50.360272+0000","last_peered":"2024-03-04T06:20:50.360272+0000","last_clean":"2024-03-04T06:20:50.360272+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:20:50.360272+0000","last_undegraded":"2024-03-04T06:20:50.360272+0000","last_fullsized":"2024-03-04T06:20:50.360272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":829,"log_dups_size":0,"ondisk_log_size":829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9328842,"num_objects":79,"num_object_clones":64,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":451,"num_read_kb":359296,"num_write":346,"num_write_kb":93348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"771'708","reported_seq":2635,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.347312+0000","last_change":"2024-03-04T06:20:50.347312+0000","last_active":"2024-03-04T06:20:50.347312+0000","last_peered":"2024-03-04T06:20:50.347312+0000","last_clean":"2024-03-04T06:20:50.347312+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:20:50.347312+0000","last_undegraded":"2024-03-04T06:20:50.347312+0000","last_fullsized":"2024-03-04T06:20:50.347312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8522106,"num_objects":54,"num_object_clones":44,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":268,"num_read_kb":202839,"num_write":255,"num_write_kb":72159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"770'289","reported_seq":2449,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.381734+0000","last_change":"2024-03-04T06:20:50.386532+0000","last_active":"2024-03-04T06:20:50.381734+0000","last_peered":"2024-03-04T06:20:50.381734+0000","last_clean":"2024-03-04T06:20:50.381734+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:20:50.381734+0000","last_undegraded":"2024-03-04T06:20:50.381734+0000","last_fullsized":"2024-03-04T06:20:50.381734+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672643,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":65484,"num_write":101,"num_write_kb":27678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"772'842","reported_seq":2732,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.351772+0000","last_change":"2024-03-04T06:20:50.356386+0000","last_active":"2024-03-04T06:20:50.351772+0000","last_peered":"2024-03-04T06:20:50.351772+0000","last_clean":"2024-03-04T06:20:50.351772+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:20:50.351772+0000","last_undegraded":"2024-03-04T06:20:50.351772+0000","last_fullsized":"2024-03-04T06:20:50.351772+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":842,"log_dups_size":0,"ondisk_log_size":842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470959,"num_objects":63,"num_object_clones":54,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":374,"num_read_kb":306917,"num_write":309,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2286,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.349513+0000","last_change":"2024-03-04T06:20:50.356566+0000","last_active":"2024-03-04T06:20:50.349513+0000","last_peered":"2024-03-04T06:20:50.349513+0000","last_clean":"2024-03-04T06:20:50.349513+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:20:50.349513+0000","last_undegraded":"2024-03-04T06:20:50.349513+0000","last_fullsized":"2024-03-04T06:20:50.349513+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"771'465","reported_seq":2545,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.356138+0000","last_change":"2024-03-04T06:20:50.366356+0000","last_active":"2024-03-04T06:20:50.356138+0000","last_peered":"2024-03-04T06:20:50.356138+0000","last_clean":"2024-03-04T06:20:50.356138+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:20:50.356138+0000","last_undegraded":"2024-03-04T06:20:50.356138+0000","last_fullsized":"2024-03-04T06:20:50.356138+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5530163,"num_objects":35,"num_object_clones":26,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":241,"num_read_kb":224065,"num_write":158,"num_write_kb":42246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2094,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.368448+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:20:50.368448+0000","last_peered":"2024-03-04T06:20:50.368448+0000","last_clean":"2024-03-04T06:20:50.368448+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:20:50.368448+0000","last_undegraded":"2024-03-04T06:20:50.368448+0000","last_fullsized":"2024-03-04T06:20:50.368448+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1933,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.370210+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:20:50.370210+0000","last_peered":"2024-03-04T06:20:50.370210+0000","last_clean":"2024-03-04T06:20:50.370210+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:20:50.370210+0000","last_undegraded":"2024-03-04T06:20:50.370210+0000","last_fullsized":"2024-03-04T06:20:50.370210+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"770'754","reported_seq":2715,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.369285+0000","last_change":"2024-03-04T06:20:50.371654+0000","last_active":"2024-03-04T06:20:50.369285+0000","last_peered":"2024-03-04T06:20:50.369285+0000","last_clean":"2024-03-04T06:20:50.369285+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:20:50.369285+0000","last_undegraded":"2024-03-04T06:20:50.369285+0000","last_fullsized":"2024-03-04T06:20:50.369285+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7463298,"num_objects":65,"num_object_clones":51,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":358,"num_read_kb":286050,"num_write":292,"num_write_kb":78583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1927,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.343899+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:20:50.343899+0000","last_peered":"2024-03-04T06:20:50.343899+0000","last_clean":"2024-03-04T06:20:50.343899+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:20:50.343899+0000","last_undegraded":"2024-03-04T06:20:50.343899+0000","last_fullsized":"2024-03-04T06:20:50.343899+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"770'483","reported_seq":2600,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.366229+0000","last_change":"2024-03-04T06:20:50.379722+0000","last_active":"2024-03-04T06:20:50.366229+0000","last_peered":"2024-03-04T06:20:50.366229+0000","last_clean":"2024-03-04T06:20:50.366229+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:20:50.366229+0000","last_undegraded":"2024-03-04T06:20:50.366229+0000","last_fullsized":"2024-03-04T06:20:50.366229+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698233,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":270,"num_read_kb":221303,"num_write":175,"num_write_kb":49293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1933,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.362730+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:20:50.362730+0000","last_peered":"2024-03-04T06:20:50.362730+0000","last_clean":"2024-03-04T06:20:50.362730+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:20:50.362730+0000","last_undegraded":"2024-03-04T06:20:50.362730+0000","last_fullsized":"2024-03-04T06:20:50.362730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"779'1361","reported_seq":3136,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.378510+0000","last_change":"2024-03-04T06:20:50.431554+0000","last_active":"2024-03-04T06:20:50.378510+0000","last_peered":"2024-03-04T06:20:50.378510+0000","last_clean":"2024-03-04T06:20:50.378510+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:20:50.378510+0000","last_undegraded":"2024-03-04T06:20:50.378510+0000","last_fullsized":"2024-03-04T06:20:50.378510+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1361,"log_dups_size":0,"ondisk_log_size":1361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7833052,"num_objects":119,"num_object_clones":95,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":713,"num_read_kb":567605,"num_write":549,"num_write_kb":146807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1927,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.335812+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:20:50.335812+0000","last_peered":"2024-03-04T06:20:50.335812+0000","last_clean":"2024-03-04T06:20:50.335812+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:20:50.335812+0000","last_undegraded":"2024-03-04T06:20:50.335812+0000","last_fullsized":"2024-03-04T06:20:50.335812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"779'1565","reported_seq":3144,"reported_epoch":779,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:20:50.321674+0000","last_change":"2024-03-04T06:20:50.321478+0000","last_active":"2024-03-04T06:20:50.321674+0000","last_peered":"2024-03-04T06:20:50.321674+0000","last_clean":"2024-03-04T06:20:50.321674+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:20:50.321674+0000","last_undegraded":"2024-03-04T06:20:50.321674+0000","last_fullsized":"2024-03-04T06:20:50.321674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1565,"log_dups_size":0,"ondisk_log_size":1565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18018684,"num_objects":130,"num_object_clones":106,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":583,"num_read_kb":425756,"num_write":598,"num_write_kb":161410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"770'753","reported_seq":2723,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.371600+0000","last_change":"2024-03-04T06:20:50.375971+0000","last_active":"2024-03-04T06:20:50.371600+0000","last_peered":"2024-03-04T06:20:50.371600+0000","last_clean":"2024-03-04T06:20:50.371600+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:20:50.371600+0000","last_undegraded":"2024-03-04T06:20:50.371600+0000","last_fullsized":"2024-03-04T06:20:50.371600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042397,"num_objects":60,"num_object_clones":48,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":394,"num_read_kb":329119,"num_write":278,"num_write_kb":75144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1997,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.330252+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:20:50.330252+0000","last_peered":"2024-03-04T06:20:50.330252+0000","last_clean":"2024-03-04T06:20:50.330252+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:20:50.330252+0000","last_undegraded":"2024-03-04T06:20:50.330252+0000","last_fullsized":"2024-03-04T06:20:50.330252+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"770'799","reported_seq":2774,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.335863+0000","last_change":"2024-03-04T06:20:50.339380+0000","last_active":"2024-03-04T06:20:50.335863+0000","last_peered":"2024-03-04T06:20:50.335863+0000","last_clean":"2024-03-04T06:20:50.335863+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:20:50.335863+0000","last_undegraded":"2024-03-04T06:20:50.335863+0000","last_fullsized":"2024-03-04T06:20:50.335863+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7883473,"num_objects":72,"num_object_clones":56,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":335,"num_read_kb":241316,"num_write":305,"num_write_kb":82364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1961,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.379749+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:20:50.379749+0000","last_peered":"2024-03-04T06:20:50.379749+0000","last_clean":"2024-03-04T06:20:50.379749+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:20:50.379749+0000","last_undegraded":"2024-03-04T06:20:50.379749+0000","last_fullsized":"2024-03-04T06:20:50.379749+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"770'1032","reported_seq":2859,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.359564+0000","last_change":"2024-03-04T06:20:50.379945+0000","last_active":"2024-03-04T06:20:50.359564+0000","last_peered":"2024-03-04T06:20:50.359564+0000","last_clean":"2024-03-04T06:20:50.359564+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:20:50.359564+0000","last_undegraded":"2024-03-04T06:20:50.359564+0000","last_fullsized":"2024-03-04T06:20:50.359564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1032,"log_dups_size":0,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12673699,"num_objects":84,"num_object_clones":70,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":428,"num_read_kb":315321,"num_write":394,"num_write_kb":110146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1997,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.327306+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:20:50.327306+0000","last_peered":"2024-03-04T06:20:50.327306+0000","last_clean":"2024-03-04T06:20:50.327306+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:20:50.327306+0000","last_undegraded":"2024-03-04T06:20:50.327306+0000","last_fullsized":"2024-03-04T06:20:50.327306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"774'986","reported_seq":2913,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.318873+0000","last_change":"2024-03-04T06:20:50.335362+0000","last_active":"2024-03-04T06:20:50.318873+0000","last_peered":"2024-03-04T06:20:50.318873+0000","last_clean":"2024-03-04T06:20:50.318873+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:20:50.318873+0000","last_undegraded":"2024-03-04T06:20:50.318873+0000","last_fullsized":"2024-03-04T06:20:50.318873+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":986,"log_dups_size":0,"ondisk_log_size":986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260888,"num_objects":81,"num_object_clones":71,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":502,"num_read_kb":422770,"num_write":365,"num_write_kb":99948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1933,"reported_epoch":779,"state":"active+clean","last_fresh":"2024-03-04T06:20:50.368437+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:20:50.368437+0000","last_peered":"2024-03-04T06:20:50.368437+0000","last_clean":"2024-03-04T06:20:50.368437+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:20:50.368437+0000","last_undegraded":"2024-03-04T06:20:50.368437+0000","last_fullsized":"2024-03-04T06:20:50.368437+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":100802564,"num_objects":1032,"num_object_clones":834,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":163,"num_read":5517,"num_read_kb":4358307,"num_write":4639,"num_write_kb":1263652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12229,"ondisk_log_size":12229,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706449,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":221216,"kb_used_data":163864,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94150624,"statfs":{"total":96636764160,"available":96410238976,"internally_reserved":0,"allocated":167796736,"data_stored":200826777,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706449,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":168680,"kb_used_data":115488,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94203160,"statfs":{"total":96636764160,"available":96464035840,"internally_reserved":0,"allocated":118259712,"data_stored":134126129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":165716,"kb_used_data":112524,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94206124,"statfs":{"total":96636764160,"available":96467070976,"internally_reserved":0,"allocated":115224576,"data_stored":138138863,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":178428,"kb_used_data":129396,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94193412,"statfs":{"total":96636764160,"available":96454053888,"internally_reserved":0,"allocated":132501504,"data_stored":163447359,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:20:55.345 INFO:tasks.rados.rados.0.smithi064.stdout:3085: done (1 left) 2024-03-04T06:20:55.345 INFO:tasks.rados.rados.0.smithi064.stdout:3086: delete oid 34 current snap is 396 2024-03-04T06:20:55.345 INFO:tasks.rados.rados.0.smithi064.stdout:3083: finishing write tid 3 to smithi06431743-4 2024-03-04T06:20:55.345 INFO:tasks.rados.rados.0.smithi064.stdout:3083: finishing write tid 4 to smithi06431743-4 2024-03-04T06:20:55.345 INFO:tasks.rados.rados.0.smithi064.stdout:3083: oid 4 updating version 0 to 1573 2024-03-04T06:20:55.345 INFO:tasks.rados.rados.0.smithi064.stdout:3083: oid 4 version 1573 is already newer than 1571 2024-03-04T06:20:55.345 INFO:tasks.rados.rados.0.smithi064.stdout:3083: oid 4 version 1573 is already newer than 1573 2024-03-04T06:20:55.346 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 1573 (ObjNum 1400 snap 396 seq_num 1400) dirty exists 2024-03-04T06:20:55.346 INFO:tasks.rados.rados.0.smithi064.stdout:3083: left oid 4 (ObjNum 1400 snap 396 seq_num 1400) 2024-03-04T06:20:55.382 INFO:tasks.rados.rados.0.smithi064.stdout:3083: done (1 left) 2024-03-04T06:20:55.382 INFO:tasks.rados.rados.0.smithi064.stdout:3086: done (0 left) 2024-03-04T06:20:55.382 INFO:tasks.rados.rados.0.smithi064.stdout:3087: write (excl) oid 2 current snap is 396 2024-03-04T06:20:55.383 INFO:tasks.rados.rados.0.smithi064.stdout:3087: seq_num 1403 ranges {780221=583299,1950119=767153,3261753=318468} 2024-03-04T06:20:55.389 INFO:tasks.rados.rados.0.smithi064.stdout:3087: writing smithi06431743-2 from 780221 to 1363520 tid 1 2024-03-04T06:20:55.397 INFO:tasks.rados.rados.0.smithi064.stdout:3087: writing smithi06431743-2 from 1950119 to 2717272 tid 2 2024-03-04T06:20:55.402 INFO:tasks.rados.rados.0.smithi064.stdout:3087: writing smithi06431743-2 from 3261753 to 3580221 tid 3 2024-03-04T06:20:55.403 INFO:tasks.rados.rados.0.smithi064.stdout:3088: write (excl) oid 17 current snap is 396 2024-03-04T06:20:55.403 INFO:tasks.rados.rados.0.smithi064.stdout:3088: seq_num 1404 ranges {797028=658548,1892248=510811,3172823=424205} 2024-03-04T06:20:55.412 INFO:tasks.rados.rados.0.smithi064.stdout:3088: writing smithi06431743-17 from 797028 to 1455576 tid 1 2024-03-04T06:20:55.419 INFO:tasks.rados.rados.0.smithi064.stdout:3088: writing smithi06431743-17 from 1892248 to 2403059 tid 2 2024-03-04T06:20:55.426 INFO:tasks.rados.rados.0.smithi064.stdout:3088: writing smithi06431743-17 from 3172823 to 3597028 tid 3 2024-03-04T06:20:55.426 INFO:tasks.rados.rados.0.smithi064.stdout:3089: delete oid 22 current snap is 396 2024-03-04T06:20:55.464 INFO:tasks.rados.rados.0.smithi064.stdout:3087: finishing write tid 1 to smithi06431743-2 2024-03-04T06:20:55.478 INFO:tasks.rados.rados.0.smithi064.stdout:3089: done (2 left) 2024-03-04T06:20:55.478 INFO:tasks.rados.rados.0.smithi064.stdout:3090: read oid 24 snap -1 2024-03-04T06:20:55.478 INFO:tasks.rados.rados.0.smithi064.stdout:3090: expect deleted 2024-03-04T06:20:55.479 INFO:tasks.rados.rados.0.smithi064.stdout:3091: write oid 50 current snap is 396 2024-03-04T06:20:55.479 INFO:tasks.rados.rados.0.smithi064.stdout:3091: seq_num 1406 ranges {430642=409046,1609212=514480,2827669=768803} 2024-03-04T06:20:55.483 INFO:tasks.rados.rados.0.smithi064.stdout:3091: writing smithi06431743-50 from 430642 to 839688 tid 1 2024-03-04T06:20:55.490 INFO:tasks.rados.rados.0.smithi064.stdout:3091: writing smithi06431743-50 from 1609212 to 2123692 tid 2 2024-03-04T06:20:55.499 INFO:tasks.rados.rados.0.smithi064.stdout:3091: writing smithi06431743-50 from 2827669 to 3596472 tid 3 2024-03-04T06:20:55.499 INFO:tasks.rados.rados.0.smithi064.stdout:3092: snap_create 2024-03-04T06:20:55.500 INFO:tasks.rados.rados.0.smithi064.stdout:3088: finishing write tid 1 to smithi06431743-17 2024-03-04T06:20:55.500 INFO:tasks.rados.rados.0.smithi064.stdout:3087: finishing write tid 2 to smithi06431743-2 2024-03-04T06:20:55.518 INFO:tasks.rados.rados.0.smithi064.stdout:3088: finishing write tid 2 to smithi06431743-17 2024-03-04T06:20:55.521 INFO:tasks.rados.rados.0.smithi064.stdout:3091: finishing write tid 1 to smithi06431743-50 2024-03-04T06:20:55.523 INFO:tasks.rados.rados.0.smithi064.stdout:3087: finishing write tid 3 to smithi06431743-2 2024-03-04T06:20:55.537 INFO:tasks.rados.rados.0.smithi064.stdout:3088: finishing write tid 3 to smithi06431743-17 2024-03-04T06:20:55.540 INFO:tasks.rados.rados.0.smithi064.stdout:3091: finishing write tid 2 to smithi06431743-50 2024-03-04T06:20:55.541 INFO:tasks.rados.rados.0.smithi064.stdout:3087: finishing write tid 4 to smithi06431743-2 2024-03-04T06:20:55.542 INFO:tasks.rados.rados.0.smithi064.stdout:3087: finishing write tid 5 to smithi06431743-2 2024-03-04T06:20:55.542 INFO:tasks.rados.rados.0.smithi064.stdout:3087: oid 2 updating version 0 to 716 2024-03-04T06:20:55.542 INFO:tasks.rados.rados.0.smithi064.stdout:3087: oid 2 updating version 716 to 718 2024-03-04T06:20:55.542 INFO:tasks.rados.rados.0.smithi064.stdout:3087: oid 2 version 718 is already newer than 717 2024-03-04T06:20:55.543 INFO:tasks.rados.rados.0.smithi064.stdout:3087: oid 2 version 718 is already newer than 715 2024-03-04T06:20:55.543 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 718 (ObjNum 1403 snap 396 seq_num 1403) dirty exists 2024-03-04T06:20:55.543 INFO:tasks.rados.rados.0.smithi064.stdout:3087: left oid 2 (ObjNum 1403 snap 396 seq_num 1403) 2024-03-04T06:20:55.547 INFO:tasks.rados.rados.0.smithi064.stdout:3088: finishing write tid 4 to smithi06431743-17 2024-03-04T06:20:55.548 INFO:tasks.rados.rados.0.smithi064.stdout:3091: finishing write tid 3 to smithi06431743-50 2024-03-04T06:20:55.548 INFO:tasks.rados.rados.0.smithi064.stdout:3088: finishing write tid 5 to smithi06431743-17 2024-03-04T06:20:55.548 INFO:tasks.rados.rados.0.smithi064.stdout:3088: oid 17 updating version 0 to 469 2024-03-04T06:20:55.549 INFO:tasks.rados.rados.0.smithi064.stdout:3088: oid 17 version 469 is already newer than 468 2024-03-04T06:20:55.549 INFO:tasks.rados.rados.0.smithi064.stdout:3088: oid 17 updating version 469 to 470 2024-03-04T06:20:55.549 INFO:tasks.rados.rados.0.smithi064.stdout:3088: oid 17 version 470 is already newer than 467 2024-03-04T06:20:55.549 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 470 (ObjNum 1404 snap 396 seq_num 1404) dirty exists 2024-03-04T06:20:55.549 INFO:tasks.rados.rados.0.smithi064.stdout:3088: left oid 17 (ObjNum 1404 snap 396 seq_num 1404) 2024-03-04T06:20:55.557 INFO:tasks.rados.rados.0.smithi064.stdout:3091: finishing write tid 4 to smithi06431743-50 2024-03-04T06:20:55.557 INFO:tasks.rados.rados.0.smithi064.stdout:3091: finishing write tid 5 to smithi06431743-50 2024-03-04T06:20:55.557 INFO:tasks.rados.rados.0.smithi064.stdout:3091: oid 50 updating version 0 to 995 2024-03-04T06:20:55.557 INFO:tasks.rados.rados.0.smithi064.stdout:3091: oid 50 updating version 995 to 997 2024-03-04T06:20:55.557 INFO:tasks.rados.rados.0.smithi064.stdout:3091: oid 50 version 997 is already newer than 997 2024-03-04T06:20:55.557 INFO:tasks.rados.rados.0.smithi064.stdout:3091: oid 50 updating version 997 to 998 2024-03-04T06:20:55.558 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 998 (ObjNum 1406 snap 396 seq_num 1406) dirty exists 2024-03-04T06:20:55.558 INFO:tasks.rados.rados.0.smithi064.stdout:3091: left oid 50 (ObjNum 1406 snap 396 seq_num 1406) 2024-03-04T06:20:56.062 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:20:56.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:56.170+0000 7f4eecb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:56.174 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:56.174+0000 7f4eecb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:56.341 INFO:tasks.rados.rados.0.smithi064.stdout:3087: done (4 left) 2024-03-04T06:20:56.341 INFO:tasks.rados.rados.0.smithi064.stdout:3088: done (3 left) 2024-03-04T06:20:56.341 INFO:tasks.rados.rados.0.smithi064.stdout:3090: done (2 left) 2024-03-04T06:20:56.341 INFO:tasks.rados.rados.0.smithi064.stdout:3091: done (1 left) 2024-03-04T06:20:56.341 INFO:tasks.rados.rados.0.smithi064.stdout:3092: done (0 left) 2024-03-04T06:20:56.341 INFO:tasks.rados.rados.0.smithi064.stdout:3093: snap_create 2024-03-04T06:20:57.342 INFO:tasks.rados.rados.0.smithi064.stdout:3093: done (0 left) 2024-03-04T06:20:57.343 INFO:tasks.rados.rados.0.smithi064.stdout:3094: write (excl) oid 45 current snap is 398 2024-03-04T06:20:57.343 INFO:tasks.rados.rados.0.smithi064.stdout:3094: seq_num 1407 ranges {447449=600648,1784047=774491,2971445=676004} 2024-03-04T06:20:57.348 INFO:tasks.rados.rados.0.smithi064.stdout:3094: writing smithi06431743-45 from 447449 to 1048097 tid 1 2024-03-04T06:20:57.358 INFO:tasks.rados.rados.0.smithi064.stdout:3094: writing smithi06431743-45 from 1784047 to 2558538 tid 2 2024-03-04T06:20:57.367 INFO:tasks.rados.rados.0.smithi064.stdout:3094: writing smithi06431743-45 from 2971445 to 3647449 tid 3 2024-03-04T06:20:57.367 INFO:tasks.rados.rados.0.smithi064.stdout:3095: read oid 15 snap -1 2024-03-04T06:20:57.367 INFO:tasks.rados.rados.0.smithi064.stdout:3095: expect (ObjNum 1371 snap 382 seq_num 1371) 2024-03-04T06:20:57.367 INFO:tasks.rados.rados.0.smithi064.stdout:3096: read oid 1 snap -1 2024-03-04T06:20:57.367 INFO:tasks.rados.rados.0.smithi064.stdout:3096: expect deleted 2024-03-04T06:20:57.367 INFO:tasks.rados.rados.0.smithi064.stdout:3097: snap_remove snap 382 2024-03-04T06:20:58.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:20:58.308 INFO:tasks.rados.rados.0.smithi064.stdout:3097: done (3 left) 2024-03-04T06:20:58.308 INFO:tasks.rados.rados.0.smithi064.stdout:3098: snap_remove snap 365 2024-03-04T06:20:58.368 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:58.366+0000 7efcafadd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:58.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:20:58.372+0000 7efcafadd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:20:59.314 INFO:tasks.rados.rados.0.smithi064.stdout:3098: done (3 left) 2024-03-04T06:20:59.314 INFO:tasks.rados.rados.0.smithi064.stdout:3099: snap_create 2024-03-04T06:20:59.314 INFO:tasks.rados.rados.0.smithi064.stdout:3095: expect (ObjNum 1371 snap 382 seq_num 1371) 2024-03-04T06:20:59.471 INFO:tasks.rados.rados.0.smithi064.stdout:3094: finishing write tid 1 to smithi06431743-45 2024-03-04T06:20:59.472 INFO:tasks.rados.rados.0.smithi064.stdout:3094: finishing write tid 2 to smithi06431743-45 2024-03-04T06:20:59.472 INFO:tasks.rados.rados.0.smithi064.stdout:3094: finishing write tid 3 to smithi06431743-45 2024-03-04T06:20:59.472 INFO:tasks.rados.rados.0.smithi064.stdout:3094: finishing write tid 4 to smithi06431743-45 2024-03-04T06:20:59.472 INFO:tasks.rados.rados.0.smithi064.stdout:3094: finishing write tid 5 to smithi06431743-45 2024-03-04T06:20:59.472 INFO:tasks.rados.rados.0.smithi064.stdout:3094: oid 45 updating version 0 to 841 2024-03-04T06:20:59.472 INFO:tasks.rados.rados.0.smithi064.stdout:3094: oid 45 version 841 is already newer than 840 2024-03-04T06:20:59.472 INFO:tasks.rados.rados.0.smithi064.stdout:3094: oid 45 version 841 is already newer than 838 2024-03-04T06:20:59.472 INFO:tasks.rados.rados.0.smithi064.stdout:3094: oid 45 version 841 is already newer than 840 2024-03-04T06:20:59.472 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 841 (ObjNum 1407 snap 398 seq_num 1407) dirty exists 2024-03-04T06:20:59.472 INFO:tasks.rados.rados.0.smithi064.stdout:3094: left oid 45 (ObjNum 1407 snap 398 seq_num 1407) 2024-03-04T06:20:59.954 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:20:59.955 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:20:59.955 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:20:59.955 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:21:00.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:00.068+0000 7ff94bd1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:00.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:00.076+0000 7ff94bd1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:00.089 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:21:00.089 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:21:00.089 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:21:00.089 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:21:00.089 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:21:00.089 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:21:00.089 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:21:00.089 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:21:00.089 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:21:00.089 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:21:00.090 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:21:00.090 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:21:00.092 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:21:00.092 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:21:00.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:21:00.200 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:00.195+0000 7fcf9979d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:00.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:00.202+0000 7fcf9979d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:21:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:21:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:21:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:21:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:21:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:21:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:21:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:21:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:21:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:21:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:21:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:21:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:21:00.218 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:21:00.218 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:21:00.312 INFO:tasks.rados.rados.0.smithi064.stdout:3094: done (3 left) 2024-03-04T06:21:00.312 INFO:tasks.rados.rados.0.smithi064.stdout:3095: done (2 left) 2024-03-04T06:21:00.313 INFO:tasks.rados.rados.0.smithi064.stdout:3096: done (1 left) 2024-03-04T06:21:00.313 INFO:tasks.rados.rados.0.smithi064.stdout:3099: done (0 left) 2024-03-04T06:21:00.313 INFO:tasks.rados.rados.0.smithi064.stdout:3100: snap_remove snap 381 2024-03-04T06:21:00.328 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:00.322+0000 7f8eed8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:00.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:00.330+0000 7f8eed8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:00.344 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:21:00.345 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:21:00.345 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:21:00.345 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:21:00.345 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:21:00.345 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:21:00.345 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:21:00.345 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:21:00.345 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:21:00.345 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:21:00.345 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:21:00.345 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:21:00.347 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:21:00.347 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:21:00.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:21:00.440 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:00.471 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:00.463+0000 7ff2d182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:00.476 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:00.475+0000 7ff2d182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:00.496 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:21:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:21:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:21:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:21:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:21:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:21:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:21:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:21:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:21:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:21:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:21:00.497 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:21:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:21:00.500 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:21:00.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:00.558+0000 7fc648d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:00.563 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:00.562+0000 7fc648d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:01.315 INFO:tasks.rados.rados.0.smithi064.stdout:3100: done (0 left) 2024-03-04T06:21:01.315 INFO:tasks.rados.rados.0.smithi064.stdout:3101: read oid 1 snap 376 2024-03-04T06:21:01.315 INFO:tasks.rados.rados.0.smithi064.stdout:3101: expect deleted 2024-03-04T06:21:01.315 INFO:tasks.rados.rados.0.smithi064.stdout:3102: snap_create 2024-03-04T06:21:02.314 INFO:tasks.rados.rados.0.smithi064.stdout:3101: done (1 left) 2024-03-04T06:21:02.314 INFO:tasks.rados.rados.0.smithi064.stdout:3102: done (0 left) 2024-03-04T06:21:02.314 INFO:tasks.rados.rados.0.smithi064.stdout:3103: read oid 17 snap -1 2024-03-04T06:21:02.315 INFO:tasks.rados.rados.0.smithi064.stdout:3103: expect (ObjNum 1404 snap 396 seq_num 1404) 2024-03-04T06:21:02.315 INFO:tasks.rados.rados.0.smithi064.stdout:3104: read oid 25 snap -1 2024-03-04T06:21:02.315 INFO:tasks.rados.rados.0.smithi064.stdout:3104: expect (ObjNum 1278 snap 346 seq_num 1278) 2024-03-04T06:21:02.315 INFO:tasks.rados.rados.0.smithi064.stdout:3105: delete oid 24 current snap is 400 2024-03-04T06:21:02.354 INFO:tasks.rados.rados.0.smithi064.stdout:3105: done (2 left) 2024-03-04T06:21:02.354 INFO:tasks.rados.rados.0.smithi064.stdout:3106: read oid 23 snap 375 2024-03-04T06:21:02.354 INFO:tasks.rados.rados.0.smithi064.stdout:3106: expect (ObjNum 1332 snap 368 seq_num 1332) 2024-03-04T06:21:02.355 INFO:tasks.rados.rados.0.smithi064.stdout:3107: delete oid 40 current snap is 400 2024-03-04T06:21:02.382 INFO:tasks.rados.rados.0.smithi064.stdout:3107: done (3 left) 2024-03-04T06:21:02.382 INFO:tasks.rados.rados.0.smithi064.stdout:3108: write oid 47 current snap is 400 2024-03-04T06:21:02.382 INFO:tasks.rados.rados.0.smithi064.stdout:3108: seq_num 1410 ranges {497870=426395,1443140=521818,2537361=530244,3583943=113927} 2024-03-04T06:21:02.387 INFO:tasks.rados.rados.0.smithi064.stdout:3108: writing smithi06431743-47 from 497870 to 924265 tid 1 2024-03-04T06:21:02.394 INFO:tasks.rados.rados.0.smithi064.stdout:3108: writing smithi06431743-47 from 1443140 to 1964958 tid 2 2024-03-04T06:21:02.401 INFO:tasks.rados.rados.0.smithi064.stdout:3108: writing smithi06431743-47 from 2537361 to 3067605 tid 3 2024-03-04T06:21:02.405 INFO:tasks.rados.rados.0.smithi064.stdout:3108: writing smithi06431743-47 from 3583943 to 3697870 tid 4 2024-03-04T06:21:02.405 INFO:tasks.rados.rados.0.smithi064.stdout:3109: snap_remove snap 385 2024-03-04T06:21:02.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:02.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:02.732+0000 7f8f18107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:02.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:02.736+0000 7f8f18107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:03.368 INFO:tasks.rados.rados.0.smithi064.stdout:3109: done (4 left) 2024-03-04T06:21:03.368 INFO:tasks.rados.rados.0.smithi064.stdout:3110: delete oid 18 current snap is 400 2024-03-04T06:21:03.368 INFO:tasks.rados.rados.0.smithi064.stdout:3108: finishing write tid 1 to smithi06431743-47 2024-03-04T06:21:03.368 INFO:tasks.rados.rados.0.smithi064.stdout:3106: expect (ObjNum 1332 snap 368 seq_num 1332) 2024-03-04T06:21:03.519 INFO:tasks.rados.rados.0.smithi064.stdout:3108: finishing write tid 2 to smithi06431743-47 2024-03-04T06:21:03.519 INFO:tasks.rados.rados.0.smithi064.stdout:3108: finishing write tid 3 to smithi06431743-47 2024-03-04T06:21:03.519 INFO:tasks.rados.rados.0.smithi064.stdout:3103: expect (ObjNum 1404 snap 396 seq_num 1404) 2024-03-04T06:21:03.701 INFO:tasks.rados.rados.0.smithi064.stdout:3104: expect (ObjNum 1278 snap 346 seq_num 1278) 2024-03-04T06:21:03.831 INFO:tasks.rados.rados.0.smithi064.stdout:3108: finishing write tid 4 to smithi06431743-47 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3108: finishing write tid 5 to smithi06431743-47 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3108: finishing write tid 6 to smithi06431743-47 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3108: oid 47 updating version 0 to 728 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3108: oid 47 updating version 728 to 731 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3108: oid 47 version 731 is already newer than 729 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3108: oid 47 version 731 is already newer than 730 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3108: oid 47 updating version 731 to 732 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 732 (ObjNum 1410 snap 400 seq_num 1410) dirty exists 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3108: left oid 47 (ObjNum 1410 snap 400 seq_num 1410) 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3103: done (4 left) 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3104: done (3 left) 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3106: done (2 left) 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3108: done (1 left) 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3110: done (0 left) 2024-03-04T06:21:03.832 INFO:tasks.rados.rados.0.smithi064.stdout:3111: delete oid 41 current snap is 400 2024-03-04T06:21:03.887 INFO:tasks.rados.rados.0.smithi064.stdout:3111: done (0 left) 2024-03-04T06:21:03.887 INFO:tasks.rados.rados.0.smithi064.stdout:3112: write (excl) oid 50 current snap is 400 2024-03-04T06:21:03.887 INFO:tasks.rados.rados.0.smithi064.stdout:3112: seq_num 1413 ranges {548291=652142,1902233=785498,3303277=445014} 2024-03-04T06:21:03.893 INFO:tasks.rados.rados.0.smithi064.stdout:3112: writing smithi06431743-50 from 548291 to 1200433 tid 1 2024-03-04T06:21:03.901 INFO:tasks.rados.rados.0.smithi064.stdout:3112: writing smithi06431743-50 from 1902233 to 2687731 tid 2 2024-03-04T06:21:03.907 INFO:tasks.rados.rados.0.smithi064.stdout:3112: writing smithi06431743-50 from 3303277 to 3748291 tid 3 2024-03-04T06:21:03.907 INFO:tasks.rados.rados.0.smithi064.stdout:3113: write (excl) oid 14 current snap is 400 2024-03-04T06:21:03.907 INFO:tasks.rados.rados.0.smithi064.stdout:3113: seq_num 1414 ranges {565098=727391,1960715=529156,2930700=691685} 2024-03-04T06:21:03.914 INFO:tasks.rados.rados.0.smithi064.stdout:3113: writing smithi06431743-14 from 565098 to 1292489 tid 1 2024-03-04T06:21:03.921 INFO:tasks.rados.rados.0.smithi064.stdout:3113: writing smithi06431743-14 from 1960715 to 2489871 tid 2 2024-03-04T06:21:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3113: writing smithi06431743-14 from 2930700 to 3622385 tid 3 2024-03-04T06:21:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3114: snap_create 2024-03-04T06:21:03.931 INFO:tasks.rados.rados.0.smithi064.stdout:3112: finishing write tid 1 to smithi06431743-50 2024-03-04T06:21:03.963 INFO:tasks.rados.rados.0.smithi064.stdout:3113: finishing write tid 1 to smithi06431743-14 2024-03-04T06:21:03.964 INFO:tasks.rados.rados.0.smithi064.stdout:3112: finishing write tid 2 to smithi06431743-50 2024-03-04T06:21:03.978 INFO:tasks.rados.rados.0.smithi064.stdout:3112: finishing write tid 3 to smithi06431743-50 2024-03-04T06:21:03.981 INFO:tasks.rados.rados.0.smithi064.stdout:3113: finishing write tid 2 to smithi06431743-14 2024-03-04T06:21:03.993 INFO:tasks.rados.rados.0.smithi064.stdout:3112: finishing write tid 4 to smithi06431743-50 2024-03-04T06:21:03.994 INFO:tasks.rados.rados.0.smithi064.stdout:3112: finishing write tid 5 to smithi06431743-50 2024-03-04T06:21:03.994 INFO:tasks.rados.rados.0.smithi064.stdout:3112: oid 50 updating version 0 to 1008 2024-03-04T06:21:03.994 INFO:tasks.rados.rados.0.smithi064.stdout:3112: oid 50 updating version 1008 to 1009 2024-03-04T06:21:03.994 INFO:tasks.rados.rados.0.smithi064.stdout:3112: oid 50 version 1009 is already newer than 1006 2024-03-04T06:21:03.994 INFO:tasks.rados.rados.0.smithi064.stdout:3112: oid 50 version 1009 is already newer than 1008 2024-03-04T06:21:03.994 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 1009 (ObjNum 1413 snap 400 seq_num 1413) dirty exists 2024-03-04T06:21:03.994 INFO:tasks.rados.rados.0.smithi064.stdout:3112: left oid 50 (ObjNum 1413 snap 400 seq_num 1413) 2024-03-04T06:21:03.996 INFO:tasks.rados.rados.0.smithi064.stdout:3113: finishing write tid 3 to smithi06431743-14 2024-03-04T06:21:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:3113: finishing write tid 4 to smithi06431743-14 2024-03-04T06:21:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:3113: finishing write tid 5 to smithi06431743-14 2024-03-04T06:21:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:3113: oid 14 updating version 0 to 1072 2024-03-04T06:21:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:3113: oid 14 updating version 1072 to 1075 2024-03-04T06:21:04.007 INFO:tasks.rados.rados.0.smithi064.stdout:3113: oid 14 version 1075 is already newer than 1074 2024-03-04T06:21:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:3113: oid 14 version 1075 is already newer than 1073 2024-03-04T06:21:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 1075 (ObjNum 1414 snap 400 seq_num 1414) dirty exists 2024-03-04T06:21:04.008 INFO:tasks.rados.rados.0.smithi064.stdout:3113: left oid 14 (ObjNum 1414 snap 400 seq_num 1414) 2024-03-04T06:21:04.370 INFO:tasks.rados.rados.0.smithi064.stdout:3112: done (2 left) 2024-03-04T06:21:04.370 INFO:tasks.rados.rados.0.smithi064.stdout:3113: done (1 left) 2024-03-04T06:21:04.370 INFO:tasks.rados.rados.0.smithi064.stdout:3114: done (0 left) 2024-03-04T06:21:04.370 INFO:tasks.rados.rados.0.smithi064.stdout:3115: write oid 37 current snap is 401 2024-03-04T06:21:04.370 INFO:tasks.rados.rados.0.smithi064.stdout:3115: seq_num 1415 ranges {581905=518993,1735550=672814,2958123=502957} 2024-03-04T06:21:04.376 INFO:tasks.rados.rados.0.smithi064.stdout:3115: writing smithi06431743-37 from 581905 to 1100898 tid 1 2024-03-04T06:21:04.384 INFO:tasks.rados.rados.0.smithi064.stdout:3115: writing smithi06431743-37 from 1735550 to 2408364 tid 2 2024-03-04T06:21:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:3115: writing smithi06431743-37 from 2958123 to 3461080 tid 3 2024-03-04T06:21:04.392 INFO:tasks.rados.rados.0.smithi064.stdout:3116: snap_create 2024-03-04T06:21:04.422 INFO:tasks.rados.rados.0.smithi064.stdout:3115: finishing write tid 1 to smithi06431743-37 2024-03-04T06:21:04.434 INFO:tasks.rados.rados.0.smithi064.stdout:3115: finishing write tid 2 to smithi06431743-37 2024-03-04T06:21:04.449 INFO:tasks.rados.rados.0.smithi064.stdout:3115: finishing write tid 3 to smithi06431743-37 2024-03-04T06:21:04.454 INFO:tasks.rados.rados.0.smithi064.stdout:3115: finishing write tid 4 to smithi06431743-37 2024-03-04T06:21:04.455 INFO:tasks.rados.rados.0.smithi064.stdout:3115: finishing write tid 5 to smithi06431743-37 2024-03-04T06:21:04.455 INFO:tasks.rados.rados.0.smithi064.stdout:3115: oid 37 updating version 0 to 651 2024-03-04T06:21:04.456 INFO:tasks.rados.rados.0.smithi064.stdout:3115: oid 37 version 651 is already newer than 649 2024-03-04T06:21:04.456 INFO:tasks.rados.rados.0.smithi064.stdout:3115: oid 37 version 651 is already newer than 650 2024-03-04T06:21:04.456 INFO:tasks.rados.rados.0.smithi064.stdout:3115: oid 37 updating version 651 to 652 2024-03-04T06:21:04.456 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 652 (ObjNum 1415 snap 401 seq_num 1415) dirty exists 2024-03-04T06:21:04.456 INFO:tasks.rados.rados.0.smithi064.stdout:3115: left oid 37 (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:21:04.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:04.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:04.909+0000 7f966086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:04.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:04.914+0000 7f966086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:05.373 INFO:tasks.rados.rados.0.smithi064.stdout:3115: done (1 left) 2024-03-04T06:21:05.374 INFO:tasks.rados.rados.0.smithi064.stdout:3116: done (0 left) 2024-03-04T06:21:05.374 INFO:tasks.rados.rados.0.smithi064.stdout:3117: snap_remove snap 400 2024-03-04T06:21:05.501 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:21:05.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:21:05.613 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:05.607+0000 7fc2d406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:05.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:05.615+0000 7fc2d406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:06.372 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:21:06.374 INFO:tasks.rados.rados.0.smithi064.stdout:3117: done (0 left) 2024-03-04T06:21:06.374 INFO:tasks.rados.rados.0.smithi064.stdout:3118: snap_remove snap 391 2024-03-04T06:21:06.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:21:06.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:06.489+0000 7f0b46e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:06.498 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:06.498+0000 7f0b46e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:06.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:07.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:07.093+0000 7f3b70b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:07.098 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:07.097+0000 7f3b70b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:07.377 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:21:07.380 INFO:tasks.rados.rados.0.smithi064.stdout:3118: done (0 left) 2024-03-04T06:21:07.380 INFO:tasks.rados.rados.0.smithi064.stdout:3119: read oid 3 snap -1 2024-03-04T06:21:07.381 INFO:tasks.rados.rados.0.smithi064.stdout:3119: expect (ObjNum 1331 snap 368 seq_num 1331) 2024-03-04T06:21:07.381 INFO:tasks.rados.rados.0.smithi064.stdout:3120: delete oid 18 current snap is 402 2024-03-04T06:21:07.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:21:07.457 INFO:tasks.rados.rados.0.smithi064.stdout:3119: expect (ObjNum 1331 snap 368 seq_num 1331) 2024-03-04T06:21:07.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:07.509+0000 7f9e1251d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:07.518 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:07.517+0000 7f9e1251d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:07.588 INFO:tasks.rados.rados.0.smithi064.stdout:3119: done (1 left) 2024-03-04T06:21:07.589 INFO:tasks.rados.rados.0.smithi064.stdout:3120: done (0 left) 2024-03-04T06:21:07.589 INFO:tasks.rados.rados.0.smithi064.stdout:3121: read oid 21 snap -1 2024-03-04T06:21:07.589 INFO:tasks.rados.rados.0.smithi064.stdout:3121: expect deleted 2024-03-04T06:21:07.589 INFO:tasks.rados.rados.0.smithi064.stdout:3122: write (excl) oid 27 current snap is 402 2024-03-04T06:21:07.589 INFO:tasks.rados.rados.0.smithi064.stdout:3122: seq_num 1417 ranges {615519=669491,1736161=676483,2896616=641854} 2024-03-04T06:21:07.595 INFO:tasks.rados.rados.0.smithi064.stdout:3122: writing smithi06431743-27 from 615519 to 1285010 tid 1 2024-03-04T06:21:07.601 INFO:tasks.rados.rados.0.smithi064.stdout:3122: writing smithi06431743-27 from 1736161 to 2412644 tid 2 2024-03-04T06:21:07.607 INFO:tasks.rados.rados.0.smithi064.stdout:3122: writing smithi06431743-27 from 2896616 to 3538470 tid 3 2024-03-04T06:21:07.608 INFO:tasks.rados.rados.0.smithi064.stdout:3123: write oid 35 current snap is 402 2024-03-04T06:21:07.608 INFO:tasks.rados.rados.0.smithi064.stdout:3123: seq_num 1418 ranges {632326=744740,1794643=536494,3040392=453126} 2024-03-04T06:21:07.615 INFO:tasks.rados.rados.0.smithi064.stdout:3123: writing smithi06431743-35 from 632326 to 1377066 tid 1 2024-03-04T06:21:07.620 INFO:tasks.rados.rados.0.smithi064.stdout:3123: writing smithi06431743-35 from 1794643 to 2331137 tid 2 2024-03-04T06:21:07.627 INFO:tasks.rados.rados.0.smithi064.stdout:3123: writing smithi06431743-35 from 3040392 to 3493518 tid 3 2024-03-04T06:21:07.627 INFO:tasks.rados.rados.0.smithi064.stdout:3124: delete oid 41 current snap is 402 2024-03-04T06:21:07.627 INFO:tasks.rados.rados.0.smithi064.stdout:3122: finishing write tid 1 to smithi06431743-27 2024-03-04T06:21:07.639 INFO:tasks.rados.rados.0.smithi064.stdout:3121: done (3 left) 2024-03-04T06:21:07.639 INFO:tasks.rados.rados.0.smithi064.stdout:3124: done (2 left) 2024-03-04T06:21:07.639 INFO:tasks.rados.rados.0.smithi064.stdout:3125: delete oid 45 current snap is 402 2024-03-04T06:21:07.663 INFO:tasks.rados.rados.0.smithi064.stdout:3122: finishing write tid 2 to smithi06431743-27 2024-03-04T06:21:07.667 INFO:tasks.rados.rados.0.smithi064.stdout:3123: finishing write tid 1 to smithi06431743-35 2024-03-04T06:21:07.672 INFO:tasks.rados.rados.0.smithi064.stdout:3125: done (2 left) 2024-03-04T06:21:07.673 INFO:tasks.rados.rados.0.smithi064.stdout:3126: delete oid 36 current snap is 402 2024-03-04T06:21:07.695 INFO:tasks.rados.rados.0.smithi064.stdout:3122: finishing write tid 3 to smithi06431743-27 2024-03-04T06:21:07.697 INFO:tasks.rados.rados.0.smithi064.stdout:3123: finishing write tid 2 to smithi06431743-35 2024-03-04T06:21:07.699 INFO:tasks.rados.rados.0.smithi064.stdout:3122: finishing write tid 4 to smithi06431743-27 2024-03-04T06:21:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:3122: finishing write tid 5 to smithi06431743-27 2024-03-04T06:21:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:3122: oid 27 updating version 0 to 656 2024-03-04T06:21:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:3122: oid 27 updating version 656 to 659 2024-03-04T06:21:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:3122: oid 27 version 659 is already newer than 657 2024-03-04T06:21:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:3122: oid 27 version 659 is already newer than 659 2024-03-04T06:21:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 659 (ObjNum 1417 snap 402 seq_num 1417) dirty exists 2024-03-04T06:21:07.701 INFO:tasks.rados.rados.0.smithi064.stdout:3122: left oid 27 (ObjNum 1417 snap 402 seq_num 1417) 2024-03-04T06:21:07.703 INFO:tasks.rados.rados.0.smithi064.stdout:3122: done (2 left) 2024-03-04T06:21:07.703 INFO:tasks.rados.rados.0.smithi064.stdout:3126: done (1 left) 2024-03-04T06:21:07.703 INFO:tasks.rados.rados.0.smithi064.stdout:3127: write oid 48 current snap is 402 2024-03-04T06:21:07.703 INFO:tasks.rados.rados.0.smithi064.stdout:3127: seq_num 1422 ranges {699554=762089,2028571=543832,3150084=614567} 2024-03-04T06:21:07.712 INFO:tasks.rados.rados.0.smithi064.stdout:3127: writing smithi06431743-48 from 699554 to 1461643 tid 1 2024-03-04T06:21:07.718 INFO:tasks.rados.rados.0.smithi064.stdout:3127: writing smithi06431743-48 from 2028571 to 2572403 tid 2 2024-03-04T06:21:07.726 INFO:tasks.rados.rados.0.smithi064.stdout:3127: writing smithi06431743-48 from 3150084 to 3764651 tid 3 2024-03-04T06:21:07.727 INFO:tasks.rados.rados.0.smithi064.stdout:3128: delete oid 41 current snap is 402 2024-03-04T06:21:07.727 INFO:tasks.rados.rados.0.smithi064.stdout:3123: finishing write tid 3 to smithi06431743-35 2024-03-04T06:21:07.727 INFO:tasks.rados.rados.0.smithi064.stdout:3123: finishing write tid 4 to smithi06431743-35 2024-03-04T06:21:07.727 INFO:tasks.rados.rados.0.smithi064.stdout:3123: finishing write tid 5 to smithi06431743-35 2024-03-04T06:21:07.727 INFO:tasks.rados.rados.0.smithi064.stdout:3123: oid 35 updating version 0 to 1083 2024-03-04T06:21:07.727 INFO:tasks.rados.rados.0.smithi064.stdout:3123: oid 35 updating version 1083 to 1084 2024-03-04T06:21:07.727 INFO:tasks.rados.rados.0.smithi064.stdout:3123: oid 35 version 1084 is already newer than 1081 2024-03-04T06:21:07.727 INFO:tasks.rados.rados.0.smithi064.stdout:3123: oid 35 version 1084 is already newer than 1083 2024-03-04T06:21:07.727 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 1084 (ObjNum 1418 snap 402 seq_num 1418) dirty exists 2024-03-04T06:21:07.727 INFO:tasks.rados.rados.0.smithi064.stdout:3123: left oid 35 (ObjNum 1418 snap 402 seq_num 1418) 2024-03-04T06:21:07.732 INFO:tasks.rados.rados.0.smithi064.stdout:3123: done (2 left) 2024-03-04T06:21:07.732 INFO:tasks.rados.rados.0.smithi064.stdout:3128: done (1 left) 2024-03-04T06:21:07.732 INFO:tasks.rados.rados.0.smithi064.stdout:3129: read oid 42 snap 393 2024-03-04T06:21:07.732 INFO:tasks.rados.rados.0.smithi064.stdout:3129: expect deleted 2024-03-04T06:21:07.732 INFO:tasks.rados.rados.0.smithi064.stdout:3130: read oid 29 snap 388 2024-03-04T06:21:07.732 INFO:tasks.rados.rados.0.smithi064.stdout:3130: expect (ObjNum 1297 snap 355 seq_num 1297) 2024-03-04T06:21:07.732 INFO:tasks.rados.rados.0.smithi064.stdout:3131: snap_create 2024-03-04T06:21:07.738 INFO:tasks.rados.rados.0.smithi064.stdout:3127: finishing write tid 1 to smithi06431743-48 2024-03-04T06:21:07.746 INFO:tasks.rados.rados.0.smithi064.stdout:3130: expect (ObjNum 1297 snap 355 seq_num 1297) 2024-03-04T06:21:07.899 INFO:tasks.rados.rados.0.smithi064.stdout:3127: finishing write tid 2 to smithi06431743-48 2024-03-04T06:21:07.899 INFO:tasks.rados.rados.0.smithi064.stdout:3127: finishing write tid 3 to smithi06431743-48 2024-03-04T06:21:07.899 INFO:tasks.rados.rados.0.smithi064.stdout:3127: finishing write tid 4 to smithi06431743-48 2024-03-04T06:21:07.899 INFO:tasks.rados.rados.0.smithi064.stdout:3127: finishing write tid 5 to smithi06431743-48 2024-03-04T06:21:07.899 INFO:tasks.rados.rados.0.smithi064.stdout:3127: oid 48 updating version 0 to 489 2024-03-04T06:21:07.899 INFO:tasks.rados.rados.0.smithi064.stdout:3127: oid 48 updating version 489 to 491 2024-03-04T06:21:07.899 INFO:tasks.rados.rados.0.smithi064.stdout:3127: oid 48 updating version 491 to 492 2024-03-04T06:21:07.899 INFO:tasks.rados.rados.0.smithi064.stdout:3127: oid 48 version 492 is already newer than 491 2024-03-04T06:21:07.899 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 492 (ObjNum 1422 snap 402 seq_num 1422) dirty exists 2024-03-04T06:21:07.899 INFO:tasks.rados.rados.0.smithi064.stdout:3127: left oid 48 (ObjNum 1422 snap 402 seq_num 1422) 2024-03-04T06:21:08.308 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:21:08.313 INFO:tasks.rados.rados.0.smithi064.stdout:3127: done (3 left) 2024-03-04T06:21:08.313 INFO:tasks.rados.rados.0.smithi064.stdout:3129: done (2 left) 2024-03-04T06:21:08.313 INFO:tasks.rados.rados.0.smithi064.stdout:3130: done (1 left) 2024-03-04T06:21:08.313 INFO:tasks.rados.rados.0.smithi064.stdout:3131: done (0 left) 2024-03-04T06:21:08.313 INFO:tasks.rados.rados.0.smithi064.stdout:3132: write (excl) oid 15 current snap is 403 2024-03-04T06:21:08.314 INFO:tasks.rados.rados.0.smithi064.stdout:3132: seq_num 1424 ranges {733168=512587,2029182=547501,3088577=637111} 2024-03-04T06:21:08.321 INFO:tasks.rados.rados.0.smithi064.stdout:3132: writing smithi06431743-15 from 733168 to 1245755 tid 1 2024-03-04T06:21:08.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:21:08.329 INFO:tasks.rados.rados.0.smithi064.stdout:3132: writing smithi06431743-15 from 2029182 to 2576683 tid 2 2024-03-04T06:21:08.337 INFO:tasks.rados.rados.0.smithi064.stdout:3132: writing smithi06431743-15 from 3088577 to 3725688 tid 3 2024-03-04T06:21:08.337 INFO:tasks.rados.rados.0.smithi064.stdout:3133: snap_remove snap 392 2024-03-04T06:21:08.456 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:08.450+0000 7f227b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:08.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:08.458+0000 7f227b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:09.170 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:09.280 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:09.278+0000 7f49ba02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:09.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:09.282+0000 7f49ba02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:09.316 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:21:09.319 INFO:tasks.rados.rados.0.smithi064.stdout:3133: done (1 left) 2024-03-04T06:21:09.319 INFO:tasks.rados.rados.0.smithi064.stdout:3134: read oid 22 snap 374 2024-03-04T06:21:09.319 INFO:tasks.rados.rados.0.smithi064.stdout:3134: expect deleted 2024-03-04T06:21:09.319 INFO:tasks.rados.rados.0.smithi064.stdout:3135: write oid 31 current snap is 403 2024-03-04T06:21:09.320 INFO:tasks.rados.rados.0.smithi064.stdout:3132: finishing write tid 1 to smithi06431743-15 2024-03-04T06:21:09.320 INFO:tasks.rados.rados.0.smithi064.stdout:3135: seq_num 1425 ranges {749975=587836,2087664=691159,3516000=433975} 2024-03-04T06:21:09.326 INFO:tasks.rados.rados.0.smithi064.stdout:3135: writing smithi06431743-31 from 749975 to 1337811 tid 1 2024-03-04T06:21:09.330 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:21:09.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:21:09.334 INFO:tasks.rados.rados.0.smithi064.stdout:3135: writing smithi06431743-31 from 2087664 to 2778823 tid 2 2024-03-04T06:21:09.341 INFO:tasks.rados.rados.0.smithi064.stdout:3135: writing smithi06431743-31 from 3516000 to 3949975 tid 3 2024-03-04T06:21:09.341 INFO:tasks.rados.rados.0.smithi064.stdout:3136: write oid 32 current snap is 403 2024-03-04T06:21:09.341 INFO:tasks.rados.rados.0.smithi064.stdout:3132: finishing write tid 2 to smithi06431743-15 2024-03-04T06:21:09.341 INFO:tasks.rados.rados.0.smithi064.stdout:3132: finishing write tid 3 to smithi06431743-15 2024-03-04T06:21:09.341 INFO:tasks.rados.rados.0.smithi064.stdout:3132: finishing write tid 4 to smithi06431743-15 2024-03-04T06:21:09.342 INFO:tasks.rados.rados.0.smithi064.stdout:3132: finishing write tid 5 to smithi06431743-15 2024-03-04T06:21:09.342 INFO:tasks.rados.rados.0.smithi064.stdout:3132: oid 15 updating version 0 to 764 2024-03-04T06:21:09.342 INFO:tasks.rados.rados.0.smithi064.stdout:3132: oid 15 version 764 is already newer than 764 2024-03-04T06:21:09.342 INFO:tasks.rados.rados.0.smithi064.stdout:3132: oid 15 version 764 is already newer than 762 2024-03-04T06:21:09.342 INFO:tasks.rados.rados.0.smithi064.stdout:3132: oid 15 version 764 is already newer than 761 2024-03-04T06:21:09.342 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 764 (ObjNum 1424 snap 403 seq_num 1424) dirty exists 2024-03-04T06:21:09.342 INFO:tasks.rados.rados.0.smithi064.stdout:3132: left oid 15 (ObjNum 1424 snap 403 seq_num 1424) 2024-03-04T06:21:09.342 INFO:tasks.rados.rados.0.smithi064.stdout:3136: seq_num 1426 ranges {766782=663085,2146146=551170,3143423=776008} 2024-03-04T06:21:09.348 INFO:tasks.rados.rados.0.smithi064.stdout:3136: writing smithi06431743-32 from 766782 to 1429867 tid 1 2024-03-04T06:21:09.355 INFO:tasks.rados.rados.0.smithi064.stdout:3136: writing smithi06431743-32 from 2146146 to 2697316 tid 2 2024-03-04T06:21:09.362 INFO:tasks.rados.rados.0.smithi064.stdout:3136: writing smithi06431743-32 from 3143423 to 3919431 tid 3 2024-03-04T06:21:09.362 INFO:tasks.rados.rados.0.smithi064.stdout:3132: done (3 left) 2024-03-04T06:21:09.363 INFO:tasks.rados.rados.0.smithi064.stdout:3137: delete oid 48 current snap is 403 2024-03-04T06:21:09.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:09.444+0000 7f0a97d07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:09.451 INFO:tasks.rados.rados.0.smithi064.stdout:3135: finishing write tid 1 to smithi06431743-31 2024-03-04T06:21:09.451 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:09.451+0000 7f0a97d07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:09.472 INFO:tasks.rados.rados.0.smithi064.stdout:3136: finishing write tid 1 to smithi06431743-32 2024-03-04T06:21:09.480 INFO:tasks.rados.rados.0.smithi064.stdout:3135: finishing write tid 2 to smithi06431743-31 2024-03-04T06:21:09.488 INFO:tasks.rados.rados.0.smithi064.stdout:3134: done (3 left) 2024-03-04T06:21:09.488 INFO:tasks.rados.rados.0.smithi064.stdout:3137: done (2 left) 2024-03-04T06:21:09.488 INFO:tasks.rados.rados.0.smithi064.stdout:3138: write (excl) oid 47 current snap is 403 2024-03-04T06:21:09.488 INFO:tasks.rados.rados.0.smithi064.stdout:3138: seq_num 1428 ranges {400396=413583,1463110=438486} 2024-03-04T06:21:09.494 INFO:tasks.rados.rados.0.smithi064.stdout:3138: writing smithi06431743-47 from 400396 to 813979 tid 1 2024-03-04T06:21:09.502 INFO:tasks.rados.rados.0.smithi064.stdout:3138: writing smithi06431743-47 from 1463110 to 1901596 tid 2 2024-03-04T06:21:09.503 INFO:tasks.rados.rados.0.smithi064.stdout:3139: read oid 17 snap 362 2024-03-04T06:21:09.503 INFO:tasks.rados.rados.0.smithi064.stdout:3139: expect (ObjNum 1251 snap 337 seq_num 1251) 2024-03-04T06:21:09.503 INFO:tasks.rados.rados.0.smithi064.stdout:3140: delete oid 42 current snap is 403 2024-03-04T06:21:09.503 INFO:tasks.rados.rados.0.smithi064.stdout:3136: finishing write tid 2 to smithi06431743-32 2024-03-04T06:21:09.503 INFO:tasks.rados.rados.0.smithi064.stdout:3135: finishing write tid 3 to smithi06431743-31 2024-03-04T06:21:09.508 INFO:tasks.rados.rados.0.smithi064.stdout:3136: finishing write tid 3 to smithi06431743-32 2024-03-04T06:21:09.515 INFO:tasks.rados.rados.0.smithi064.stdout:3135: finishing write tid 4 to smithi06431743-31 2024-03-04T06:21:09.518 INFO:tasks.rados.rados.0.smithi064.stdout:3135: finishing write tid 5 to smithi06431743-31 2024-03-04T06:21:09.518 INFO:tasks.rados.rados.0.smithi064.stdout:3135: oid 31 updating version 0 to 296 2024-03-04T06:21:09.518 INFO:tasks.rados.rados.0.smithi064.stdout:3135: oid 31 version 296 is already newer than 294 2024-03-04T06:21:09.518 INFO:tasks.rados.rados.0.smithi064.stdout:3135: oid 31 version 296 is already newer than 295 2024-03-04T06:21:09.518 INFO:tasks.rados.rados.0.smithi064.stdout:3135: oid 31 version 296 is already newer than 293 2024-03-04T06:21:09.518 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 296 (ObjNum 1425 snap 403 seq_num 1425) dirty exists 2024-03-04T06:21:09.518 INFO:tasks.rados.rados.0.smithi064.stdout:3135: left oid 31 (ObjNum 1425 snap 403 seq_num 1425) 2024-03-04T06:21:09.528 INFO:tasks.rados.rados.0.smithi064.stdout:3136: finishing write tid 4 to smithi06431743-32 2024-03-04T06:21:09.529 INFO:tasks.rados.rados.0.smithi064.stdout:3136: finishing write tid 5 to smithi06431743-32 2024-03-04T06:21:09.529 INFO:tasks.rados.rados.0.smithi064.stdout:3136: oid 32 updating version 0 to 1379 2024-03-04T06:21:09.529 INFO:tasks.rados.rados.0.smithi064.stdout:3136: oid 32 version 1379 is already newer than 1377 2024-03-04T06:21:09.529 INFO:tasks.rados.rados.0.smithi064.stdout:3136: oid 32 version 1379 is already newer than 1378 2024-03-04T06:21:09.529 INFO:tasks.rados.rados.0.smithi064.stdout:3136: oid 32 updating version 1379 to 1380 2024-03-04T06:21:09.529 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1380 (ObjNum 1426 snap 403 seq_num 1426) dirty exists 2024-03-04T06:21:09.529 INFO:tasks.rados.rados.0.smithi064.stdout:3136: left oid 32 (ObjNum 1426 snap 403 seq_num 1426) 2024-03-04T06:21:09.556 INFO:tasks.rados.rados.0.smithi064.stdout:3135: done (4 left) 2024-03-04T06:21:09.556 INFO:tasks.rados.rados.0.smithi064.stdout:3136: done (3 left) 2024-03-04T06:21:09.556 INFO:tasks.rados.rados.0.smithi064.stdout:3140: done (2 left) 2024-03-04T06:21:09.556 INFO:tasks.rados.rados.0.smithi064.stdout:3141: snap_remove snap 378 2024-03-04T06:21:09.704 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:21:09.705 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:21:09.716 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1191,"stamp":"2024-03-04T06:21:09.317523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103257985,"num_objects":1053,"num_object_clones":852,"num_object_copies":3159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":167,"num_read":5690,"num_read_kb":4402088,"num_write":4864,"num_write_kb":1291821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12581,"ondisk_log_size":12581,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":817156,"kb_used_data":600228,"kb_used_omap":0,"kb_used_meta":216896,"kb_avail":376670204,"statfs":{"total":386547056640,"available":385710288896,"internally_reserved":0,"allocated":614633472,"data_stored":744664106,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":222101504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1277629,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":24,"num_read_kb":19169,"num_write":23,"num_write_kb":6958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.945369"},"pg_stats":[{"pgid":"3.f","version":"791'742","reported_seq":2716,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.392262+0000","last_change":"2024-03-04T06:21:04.402567+0000","last_active":"2024-03-04T06:21:05.392262+0000","last_peered":"2024-03-04T06:21:05.392262+0000","last_clean":"2024-03-04T06:21:05.392262+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:21:05.392262+0000","last_undegraded":"2024-03-04T06:21:05.392262+0000","last_fullsized":"2024-03-04T06:21:05.392262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":313,"num_read_kb":251966,"num_write":280,"num_write_kb":73822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"793'652","reported_seq":2705,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.447916+0000","last_change":"2024-03-04T06:21:03.473984+0000","last_active":"2024-03-04T06:21:04.447916+0000","last_peered":"2024-03-04T06:21:04.447916+0000","last_clean":"2024-03-04T06:21:04.447916+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:04.447916+0000","last_undegraded":"2024-03-04T06:21:04.447916+0000","last_fullsized":"2024-03-04T06:21:04.447916+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6958758,"num_objects":59,"num_object_clones":43,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":199,"num_read_kb":138502,"num_write":245,"num_write_kb":64414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"792'853","reported_seq":2847,"reported_epoch":794,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:05.932464+0000","last_change":"2024-03-04T06:21:02.359969+0000","last_active":"2024-03-04T06:21:05.932464+0000","last_peered":"2024-03-04T06:21:05.932464+0000","last_clean":"2024-03-04T06:21:05.932464+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:21:05.932464+0000","last_undegraded":"2024-03-04T06:21:05.932464+0000","last_fullsized":"2024-03-04T06:21:05.932464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":853,"log_dups_size":0,"ondisk_log_size":853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10606174,"num_objects":81,"num_object_clones":66,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":467,"num_read_kb":374978,"num_write":356,"num_write_kb":96872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"793'738","reported_seq":2690,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.381214+0000","last_change":"2024-03-04T06:21:04.436565+0000","last_active":"2024-03-04T06:21:05.381214+0000","last_peered":"2024-03-04T06:21:05.381214+0000","last_clean":"2024-03-04T06:21:05.381214+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:21:05.381214+0000","last_undegraded":"2024-03-04T06:21:05.381214+0000","last_fullsized":"2024-03-04T06:21:05.381214+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":738,"log_dups_size":0,"ondisk_log_size":738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639821,"num_objects":56,"num_object_clones":46,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":272,"num_read_kb":202842,"num_write":270,"num_write_kb":76269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"789'291","reported_seq":2493,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.929148+0000","last_change":"2024-03-04T06:21:03.404428+0000","last_active":"2024-03-04T06:21:05.929148+0000","last_peered":"2024-03-04T06:21:05.929148+0000","last_clean":"2024-03-04T06:21:05.929148+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:21:05.929148+0000","last_undegraded":"2024-03-04T06:21:05.929148+0000","last_fullsized":"2024-03-04T06:21:05.929148+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672643,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":65484,"num_write":101,"num_write_kb":27678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"790'846","reported_seq":2772,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.389757+0000","last_change":"2024-03-04T06:21:04.403099+0000","last_active":"2024-03-04T06:21:05.389757+0000","last_peered":"2024-03-04T06:21:05.389757+0000","last_clean":"2024-03-04T06:21:05.389757+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:21:05.389757+0000","last_undegraded":"2024-03-04T06:21:05.389757+0000","last_fullsized":"2024-03-04T06:21:05.389757+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":846,"log_dups_size":0,"ondisk_log_size":846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470959,"num_objects":63,"num_object_clones":54,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":374,"num_read_kb":306917,"num_write":309,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2326,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.390414+0000","last_change":"2024-03-04T06:21:04.403012+0000","last_active":"2024-03-04T06:21:05.390414+0000","last_peered":"2024-03-04T06:21:05.390414+0000","last_clean":"2024-03-04T06:21:05.390414+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:21:05.390414+0000","last_undegraded":"2024-03-04T06:21:05.390414+0000","last_fullsized":"2024-03-04T06:21:05.390414+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"792'476","reported_seq":2587,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.925754+0000","last_change":"2024-03-04T06:21:03.888193+0000","last_active":"2024-03-04T06:21:05.925754+0000","last_peered":"2024-03-04T06:21:05.925754+0000","last_clean":"2024-03-04T06:21:05.925754+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:21:05.925754+0000","last_undegraded":"2024-03-04T06:21:05.925754+0000","last_fullsized":"2024-03-04T06:21:05.925754+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6000759,"num_objects":36,"num_object_clones":27,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":249,"num_read_kb":229475,"num_write":163,"num_write_kb":43804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2133,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.912389+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:21:05.912389+0000","last_peered":"2024-03-04T06:21:05.912389+0000","last_clean":"2024-03-04T06:21:05.912389+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:21:05.912389+0000","last_undegraded":"2024-03-04T06:21:05.912389+0000","last_fullsized":"2024-03-04T06:21:05.912389+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.948764+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:21:05.948764+0000","last_peered":"2024-03-04T06:21:05.948764+0000","last_clean":"2024-03-04T06:21:05.948764+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:21:05.948764+0000","last_undegraded":"2024-03-04T06:21:05.948764+0000","last_fullsized":"2024-03-04T06:21:05.948764+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"792'766","reported_seq":2754,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.944229+0000","last_change":"2024-03-04T06:21:04.476739+0000","last_active":"2024-03-04T06:21:05.944229+0000","last_peered":"2024-03-04T06:21:05.944229+0000","last_clean":"2024-03-04T06:21:05.944229+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:21:05.944229+0000","last_undegraded":"2024-03-04T06:21:05.944229+0000","last_fullsized":"2024-03-04T06:21:05.944229+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370117,"num_objects":67,"num_object_clones":53,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":364,"num_read_kb":291649,"num_write":294,"num_write_kb":78583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.389244+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:21:05.389244+0000","last_peered":"2024-03-04T06:21:05.389244+0000","last_clean":"2024-03-04T06:21:05.389244+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:21:05.389244+0000","last_undegraded":"2024-03-04T06:21:05.389244+0000","last_fullsized":"2024-03-04T06:21:05.389244+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"789'487","reported_seq":2644,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.923158+0000","last_change":"2024-03-04T06:21:03.390956+0000","last_active":"2024-03-04T06:21:05.923158+0000","last_peered":"2024-03-04T06:21:05.923158+0000","last_clean":"2024-03-04T06:21:05.923158+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:21:05.923158+0000","last_undegraded":"2024-03-04T06:21:05.923158+0000","last_fullsized":"2024-03-04T06:21:05.923158+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698233,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":270,"num_read_kb":221303,"num_write":175,"num_write_kb":49293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.930065+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:21:05.930065+0000","last_peered":"2024-03-04T06:21:05.930065+0000","last_clean":"2024-03-04T06:21:05.930065+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:21:05.930065+0000","last_undegraded":"2024-03-04T06:21:05.930065+0000","last_fullsized":"2024-03-04T06:21:05.930065+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"792'1375","reported_seq":3181,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.925369+0000","last_change":"2024-03-04T06:21:03.448109+0000","last_active":"2024-03-04T06:21:05.925369+0000","last_peered":"2024-03-04T06:21:05.925369+0000","last_clean":"2024-03-04T06:21:05.925369+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:21:05.925369+0000","last_undegraded":"2024-03-04T06:21:05.925369+0000","last_fullsized":"2024-03-04T06:21:05.925369+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1375,"log_dups_size":0,"ondisk_log_size":1375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7833052,"num_objects":119,"num_object_clones":95,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":713,"num_read_kb":567605,"num_write":549,"num_write_kb":146807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.383520+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:21:05.383520+0000","last_peered":"2024-03-04T06:21:05.383520+0000","last_clean":"2024-03-04T06:21:05.383520+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:21:05.383520+0000","last_undegraded":"2024-03-04T06:21:05.383520+0000","last_fullsized":"2024-03-04T06:21:05.383520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"792'1601","reported_seq":3197,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.392623+0000","last_change":"2024-03-04T06:21:03.478422+0000","last_active":"2024-03-04T06:21:04.392623+0000","last_peered":"2024-03-04T06:21:04.392623+0000","last_clean":"2024-03-04T06:21:04.392623+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:04.392623+0000","last_undegraded":"2024-03-04T06:21:04.392623+0000","last_fullsized":"2024-03-04T06:21:04.392623+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1601,"log_dups_size":0,"ondisk_log_size":1601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20388838,"num_objects":131,"num_object_clones":107,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":591,"num_read_kb":430948,"num_write":606,"num_write_kb":163623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"792'759","reported_seq":2760,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.945620+0000","last_change":"2024-03-04T06:21:04.481076+0000","last_active":"2024-03-04T06:21:05.945620+0000","last_peered":"2024-03-04T06:21:05.945620+0000","last_clean":"2024-03-04T06:21:05.945620+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:21:05.945620+0000","last_undegraded":"2024-03-04T06:21:05.945620+0000","last_fullsized":"2024-03-04T06:21:05.945620+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":759,"log_dups_size":0,"ondisk_log_size":759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042397,"num_objects":60,"num_object_clones":48,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":400,"num_read_kb":335889,"num_write":278,"num_write_kb":75144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2033,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.399133+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:04.399133+0000","last_peered":"2024-03-04T06:21:04.399133+0000","last_clean":"2024-03-04T06:21:04.399133+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:04.399133+0000","last_undegraded":"2024-03-04T06:21:04.399133+0000","last_fullsized":"2024-03-04T06:21:04.399133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"792'814","reported_seq":2823,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.404105+0000","last_change":"2024-03-04T06:21:03.445748+0000","last_active":"2024-03-04T06:21:04.404105+0000","last_peered":"2024-03-04T06:21:04.404105+0000","last_clean":"2024-03-04T06:21:04.404105+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:04.404105+0000","last_undegraded":"2024-03-04T06:21:04.404105+0000","last_fullsized":"2024-03-04T06:21:04.404105+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":814,"log_dups_size":0,"ondisk_log_size":814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9009542,"num_objects":73,"num_object_clones":57,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":342,"num_read_kb":246342,"num_write":310,"num_write_kb":84047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2000,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.923932+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:21:05.923932+0000","last_peered":"2024-03-04T06:21:05.923932+0000","last_clean":"2024-03-04T06:21:05.923932+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:21:05.923932+0000","last_undegraded":"2024-03-04T06:21:05.923932+0000","last_fullsized":"2024-03-04T06:21:05.923932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"792'1075","reported_seq":2920,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.922107+0000","last_change":"2024-03-04T06:21:03.433911+0000","last_active":"2024-03-04T06:21:05.922107+0000","last_peered":"2024-03-04T06:21:05.922107+0000","last_clean":"2024-03-04T06:21:05.922107+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:21:05.922107+0000","last_undegraded":"2024-03-04T06:21:05.922107+0000","last_fullsized":"2024-03-04T06:21:05.922107+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1075,"log_dups_size":0,"ondisk_log_size":1075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11228033,"num_objects":88,"num_object_clones":74,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":435,"num_read_kb":315324,"num_write":410,"num_write_kb":115144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2033,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.394677+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:04.394677+0000","last_peered":"2024-03-04T06:21:04.394677+0000","last_clean":"2024-03-04T06:21:04.394677+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:04.394677+0000","last_undegraded":"2024-03-04T06:21:04.394677+0000","last_fullsized":"2024-03-04T06:21:04.394677+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"792'1009","reported_seq":2970,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.389633+0000","last_change":"2024-03-04T06:21:03.444502+0000","last_active":"2024-03-04T06:21:04.389633+0000","last_peered":"2024-03-04T06:21:04.389633+0000","last_clean":"2024-03-04T06:21:04.389633+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:04.389633+0000","last_undegraded":"2024-03-04T06:21:04.389633+0000","last_fullsized":"2024-03-04T06:21:04.389633+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1009,"log_dups_size":0,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3748291,"num_objects":85,"num_object_clones":74,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":507,"num_read_kb":422773,"num_write":381,"num_write_kb":104774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.943514+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:21:05.943514+0000","last_peered":"2024-03-04T06:21:05.943514+0000","last_clean":"2024-03-04T06:21:05.943514+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:21:05.943514+0000","last_undegraded":"2024-03-04T06:21:05.943514+0000","last_fullsized":"2024-03-04T06:21:05.943514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102667617,"num_objects":1051,"num_object_clones":852,"num_object_copies":3153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":167,"num_read":5587,"num_read_kb":4401997,"num_write":4727,"num_write_kb":1289927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12484,"ondisk_log_size":12484,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706464,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":245588,"kb_used_data":188236,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94126252,"statfs":{"total":96636764160,"available":96385282048,"internally_reserved":0,"allocated":192753664,"data_stored":234570560,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706464,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":186796,"kb_used_data":133604,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94185044,"statfs":{"total":96636764160,"available":96445485056,"internally_reserved":0,"allocated":136810496,"data_stored":158346788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":184756,"kb_used_data":131564,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94187084,"statfs":{"total":96636764160,"available":96447574016,"internally_reserved":0,"allocated":134721536,"data_stored":164117065,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":200016,"kb_used_data":146824,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94171824,"statfs":{"total":96636764160,"available":96431947776,"internally_reserved":0,"allocated":150347776,"data_stored":187629693,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:21:09.717 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:21:09.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:09.821+0000 7f3b6946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:09.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:09.827+0000 7f3b6946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:10.017 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:21:10.018 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:21:10.029 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1191,"stamp":"2024-03-04T06:21:09.317523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103257985,"num_objects":1053,"num_object_clones":852,"num_object_copies":3159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":167,"num_read":5690,"num_read_kb":4402088,"num_write":4864,"num_write_kb":1291821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12581,"ondisk_log_size":12581,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":817156,"kb_used_data":600228,"kb_used_omap":0,"kb_used_meta":216896,"kb_avail":376670204,"statfs":{"total":386547056640,"available":385710288896,"internally_reserved":0,"allocated":614633472,"data_stored":744664106,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":222101504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1277629,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":24,"num_read_kb":19169,"num_write":23,"num_write_kb":6958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.945369"},"pg_stats":[{"pgid":"3.f","version":"791'742","reported_seq":2716,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.392262+0000","last_change":"2024-03-04T06:21:04.402567+0000","last_active":"2024-03-04T06:21:05.392262+0000","last_peered":"2024-03-04T06:21:05.392262+0000","last_clean":"2024-03-04T06:21:05.392262+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:21:05.392262+0000","last_undegraded":"2024-03-04T06:21:05.392262+0000","last_fullsized":"2024-03-04T06:21:05.392262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":313,"num_read_kb":251966,"num_write":280,"num_write_kb":73822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"793'652","reported_seq":2705,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.447916+0000","last_change":"2024-03-04T06:21:03.473984+0000","last_active":"2024-03-04T06:21:04.447916+0000","last_peered":"2024-03-04T06:21:04.447916+0000","last_clean":"2024-03-04T06:21:04.447916+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:04.447916+0000","last_undegraded":"2024-03-04T06:21:04.447916+0000","last_fullsized":"2024-03-04T06:21:04.447916+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6958758,"num_objects":59,"num_object_clones":43,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":199,"num_read_kb":138502,"num_write":245,"num_write_kb":64414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"792'853","reported_seq":2847,"reported_epoch":794,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:05.932464+0000","last_change":"2024-03-04T06:21:02.359969+0000","last_active":"2024-03-04T06:21:05.932464+0000","last_peered":"2024-03-04T06:21:05.932464+0000","last_clean":"2024-03-04T06:21:05.932464+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:21:05.932464+0000","last_undegraded":"2024-03-04T06:21:05.932464+0000","last_fullsized":"2024-03-04T06:21:05.932464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":853,"log_dups_size":0,"ondisk_log_size":853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10606174,"num_objects":81,"num_object_clones":66,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":467,"num_read_kb":374978,"num_write":356,"num_write_kb":96872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"793'738","reported_seq":2690,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.381214+0000","last_change":"2024-03-04T06:21:04.436565+0000","last_active":"2024-03-04T06:21:05.381214+0000","last_peered":"2024-03-04T06:21:05.381214+0000","last_clean":"2024-03-04T06:21:05.381214+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:21:05.381214+0000","last_undegraded":"2024-03-04T06:21:05.381214+0000","last_fullsized":"2024-03-04T06:21:05.381214+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":738,"log_dups_size":0,"ondisk_log_size":738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639821,"num_objects":56,"num_object_clones":46,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":272,"num_read_kb":202842,"num_write":270,"num_write_kb":76269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"789'291","reported_seq":2493,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.929148+0000","last_change":"2024-03-04T06:21:03.404428+0000","last_active":"2024-03-04T06:21:05.929148+0000","last_peered":"2024-03-04T06:21:05.929148+0000","last_clean":"2024-03-04T06:21:05.929148+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:21:05.929148+0000","last_undegraded":"2024-03-04T06:21:05.929148+0000","last_fullsized":"2024-03-04T06:21:05.929148+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672643,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":65484,"num_write":101,"num_write_kb":27678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"790'846","reported_seq":2772,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.389757+0000","last_change":"2024-03-04T06:21:04.403099+0000","last_active":"2024-03-04T06:21:05.389757+0000","last_peered":"2024-03-04T06:21:05.389757+0000","last_clean":"2024-03-04T06:21:05.389757+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:21:05.389757+0000","last_undegraded":"2024-03-04T06:21:05.389757+0000","last_fullsized":"2024-03-04T06:21:05.389757+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":846,"log_dups_size":0,"ondisk_log_size":846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470959,"num_objects":63,"num_object_clones":54,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":374,"num_read_kb":306917,"num_write":309,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2326,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.390414+0000","last_change":"2024-03-04T06:21:04.403012+0000","last_active":"2024-03-04T06:21:05.390414+0000","last_peered":"2024-03-04T06:21:05.390414+0000","last_clean":"2024-03-04T06:21:05.390414+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:21:05.390414+0000","last_undegraded":"2024-03-04T06:21:05.390414+0000","last_fullsized":"2024-03-04T06:21:05.390414+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"792'476","reported_seq":2587,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.925754+0000","last_change":"2024-03-04T06:21:03.888193+0000","last_active":"2024-03-04T06:21:05.925754+0000","last_peered":"2024-03-04T06:21:05.925754+0000","last_clean":"2024-03-04T06:21:05.925754+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:21:05.925754+0000","last_undegraded":"2024-03-04T06:21:05.925754+0000","last_fullsized":"2024-03-04T06:21:05.925754+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6000759,"num_objects":36,"num_object_clones":27,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":249,"num_read_kb":229475,"num_write":163,"num_write_kb":43804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2133,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.912389+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:21:05.912389+0000","last_peered":"2024-03-04T06:21:05.912389+0000","last_clean":"2024-03-04T06:21:05.912389+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:21:05.912389+0000","last_undegraded":"2024-03-04T06:21:05.912389+0000","last_fullsized":"2024-03-04T06:21:05.912389+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.948764+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:21:05.948764+0000","last_peered":"2024-03-04T06:21:05.948764+0000","last_clean":"2024-03-04T06:21:05.948764+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:21:05.948764+0000","last_undegraded":"2024-03-04T06:21:05.948764+0000","last_fullsized":"2024-03-04T06:21:05.948764+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"792'766","reported_seq":2754,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.944229+0000","last_change":"2024-03-04T06:21:04.476739+0000","last_active":"2024-03-04T06:21:05.944229+0000","last_peered":"2024-03-04T06:21:05.944229+0000","last_clean":"2024-03-04T06:21:05.944229+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:21:05.944229+0000","last_undegraded":"2024-03-04T06:21:05.944229+0000","last_fullsized":"2024-03-04T06:21:05.944229+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370117,"num_objects":67,"num_object_clones":53,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":364,"num_read_kb":291649,"num_write":294,"num_write_kb":78583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.389244+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:21:05.389244+0000","last_peered":"2024-03-04T06:21:05.389244+0000","last_clean":"2024-03-04T06:21:05.389244+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:21:05.389244+0000","last_undegraded":"2024-03-04T06:21:05.389244+0000","last_fullsized":"2024-03-04T06:21:05.389244+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"789'487","reported_seq":2644,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.923158+0000","last_change":"2024-03-04T06:21:03.390956+0000","last_active":"2024-03-04T06:21:05.923158+0000","last_peered":"2024-03-04T06:21:05.923158+0000","last_clean":"2024-03-04T06:21:05.923158+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:21:05.923158+0000","last_undegraded":"2024-03-04T06:21:05.923158+0000","last_fullsized":"2024-03-04T06:21:05.923158+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698233,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":270,"num_read_kb":221303,"num_write":175,"num_write_kb":49293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.930065+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:21:05.930065+0000","last_peered":"2024-03-04T06:21:05.930065+0000","last_clean":"2024-03-04T06:21:05.930065+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:21:05.930065+0000","last_undegraded":"2024-03-04T06:21:05.930065+0000","last_fullsized":"2024-03-04T06:21:05.930065+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"792'1375","reported_seq":3181,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.925369+0000","last_change":"2024-03-04T06:21:03.448109+0000","last_active":"2024-03-04T06:21:05.925369+0000","last_peered":"2024-03-04T06:21:05.925369+0000","last_clean":"2024-03-04T06:21:05.925369+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:21:05.925369+0000","last_undegraded":"2024-03-04T06:21:05.925369+0000","last_fullsized":"2024-03-04T06:21:05.925369+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1375,"log_dups_size":0,"ondisk_log_size":1375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7833052,"num_objects":119,"num_object_clones":95,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":713,"num_read_kb":567605,"num_write":549,"num_write_kb":146807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.383520+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:21:05.383520+0000","last_peered":"2024-03-04T06:21:05.383520+0000","last_clean":"2024-03-04T06:21:05.383520+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:21:05.383520+0000","last_undegraded":"2024-03-04T06:21:05.383520+0000","last_fullsized":"2024-03-04T06:21:05.383520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"792'1601","reported_seq":3197,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.392623+0000","last_change":"2024-03-04T06:21:03.478422+0000","last_active":"2024-03-04T06:21:04.392623+0000","last_peered":"2024-03-04T06:21:04.392623+0000","last_clean":"2024-03-04T06:21:04.392623+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:04.392623+0000","last_undegraded":"2024-03-04T06:21:04.392623+0000","last_fullsized":"2024-03-04T06:21:04.392623+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1601,"log_dups_size":0,"ondisk_log_size":1601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20388838,"num_objects":131,"num_object_clones":107,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":591,"num_read_kb":430948,"num_write":606,"num_write_kb":163623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"792'759","reported_seq":2760,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.945620+0000","last_change":"2024-03-04T06:21:04.481076+0000","last_active":"2024-03-04T06:21:05.945620+0000","last_peered":"2024-03-04T06:21:05.945620+0000","last_clean":"2024-03-04T06:21:05.945620+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:21:05.945620+0000","last_undegraded":"2024-03-04T06:21:05.945620+0000","last_fullsized":"2024-03-04T06:21:05.945620+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":759,"log_dups_size":0,"ondisk_log_size":759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042397,"num_objects":60,"num_object_clones":48,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":400,"num_read_kb":335889,"num_write":278,"num_write_kb":75144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2033,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.399133+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:04.399133+0000","last_peered":"2024-03-04T06:21:04.399133+0000","last_clean":"2024-03-04T06:21:04.399133+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:04.399133+0000","last_undegraded":"2024-03-04T06:21:04.399133+0000","last_fullsized":"2024-03-04T06:21:04.399133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"792'814","reported_seq":2823,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.404105+0000","last_change":"2024-03-04T06:21:03.445748+0000","last_active":"2024-03-04T06:21:04.404105+0000","last_peered":"2024-03-04T06:21:04.404105+0000","last_clean":"2024-03-04T06:21:04.404105+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:04.404105+0000","last_undegraded":"2024-03-04T06:21:04.404105+0000","last_fullsized":"2024-03-04T06:21:04.404105+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":814,"log_dups_size":0,"ondisk_log_size":814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9009542,"num_objects":73,"num_object_clones":57,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":342,"num_read_kb":246342,"num_write":310,"num_write_kb":84047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2000,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.923932+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:21:05.923932+0000","last_peered":"2024-03-04T06:21:05.923932+0000","last_clean":"2024-03-04T06:21:05.923932+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:21:05.923932+0000","last_undegraded":"2024-03-04T06:21:05.923932+0000","last_fullsized":"2024-03-04T06:21:05.923932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"792'1075","reported_seq":2920,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.922107+0000","last_change":"2024-03-04T06:21:03.433911+0000","last_active":"2024-03-04T06:21:05.922107+0000","last_peered":"2024-03-04T06:21:05.922107+0000","last_clean":"2024-03-04T06:21:05.922107+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:21:05.922107+0000","last_undegraded":"2024-03-04T06:21:05.922107+0000","last_fullsized":"2024-03-04T06:21:05.922107+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1075,"log_dups_size":0,"ondisk_log_size":1075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11228033,"num_objects":88,"num_object_clones":74,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":435,"num_read_kb":315324,"num_write":410,"num_write_kb":115144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2033,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.394677+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:04.394677+0000","last_peered":"2024-03-04T06:21:04.394677+0000","last_clean":"2024-03-04T06:21:04.394677+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:04.394677+0000","last_undegraded":"2024-03-04T06:21:04.394677+0000","last_fullsized":"2024-03-04T06:21:04.394677+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"792'1009","reported_seq":2970,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.389633+0000","last_change":"2024-03-04T06:21:03.444502+0000","last_active":"2024-03-04T06:21:04.389633+0000","last_peered":"2024-03-04T06:21:04.389633+0000","last_clean":"2024-03-04T06:21:04.389633+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:04.389633+0000","last_undegraded":"2024-03-04T06:21:04.389633+0000","last_fullsized":"2024-03-04T06:21:04.389633+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1009,"log_dups_size":0,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3748291,"num_objects":85,"num_object_clones":74,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":507,"num_read_kb":422773,"num_write":381,"num_write_kb":104774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.943514+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:21:05.943514+0000","last_peered":"2024-03-04T06:21:05.943514+0000","last_clean":"2024-03-04T06:21:05.943514+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:21:05.943514+0000","last_undegraded":"2024-03-04T06:21:05.943514+0000","last_fullsized":"2024-03-04T06:21:05.943514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102667617,"num_objects":1051,"num_object_clones":852,"num_object_copies":3153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":167,"num_read":5587,"num_read_kb":4401997,"num_write":4727,"num_write_kb":1289927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12484,"ondisk_log_size":12484,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706464,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":245588,"kb_used_data":188236,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94126252,"statfs":{"total":96636764160,"available":96385282048,"internally_reserved":0,"allocated":192753664,"data_stored":234570560,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706464,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":186796,"kb_used_data":133604,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94185044,"statfs":{"total":96636764160,"available":96445485056,"internally_reserved":0,"allocated":136810496,"data_stored":158346788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":184756,"kb_used_data":131564,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94187084,"statfs":{"total":96636764160,"available":96447574016,"internally_reserved":0,"allocated":134721536,"data_stored":164117065,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":200016,"kb_used_data":146824,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94171824,"statfs":{"total":96636764160,"available":96431947776,"internally_reserved":0,"allocated":150347776,"data_stored":187629693,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:21:10.031 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:21:10.031 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:21:10.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:21:10.139 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:10.135+0000 7f3f6f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:10.142 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:10.141+0000 7f3f6f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:10.328 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:21:10.329 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:21:10.341 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1191,"stamp":"2024-03-04T06:21:09.317523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103257985,"num_objects":1053,"num_object_clones":852,"num_object_copies":3159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":167,"num_read":5690,"num_read_kb":4402088,"num_write":4864,"num_write_kb":1291821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12581,"ondisk_log_size":12581,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":817156,"kb_used_data":600228,"kb_used_omap":0,"kb_used_meta":216896,"kb_avail":376670204,"statfs":{"total":386547056640,"available":385710288896,"internally_reserved":0,"allocated":614633472,"data_stored":744664106,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":222101504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1277629,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":24,"num_read_kb":19169,"num_write":23,"num_write_kb":6958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.945369"},"pg_stats":[{"pgid":"3.f","version":"791'742","reported_seq":2716,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.392262+0000","last_change":"2024-03-04T06:21:04.402567+0000","last_active":"2024-03-04T06:21:05.392262+0000","last_peered":"2024-03-04T06:21:05.392262+0000","last_clean":"2024-03-04T06:21:05.392262+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:21:05.392262+0000","last_undegraded":"2024-03-04T06:21:05.392262+0000","last_fullsized":"2024-03-04T06:21:05.392262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":313,"num_read_kb":251966,"num_write":280,"num_write_kb":73822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"793'652","reported_seq":2705,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.447916+0000","last_change":"2024-03-04T06:21:03.473984+0000","last_active":"2024-03-04T06:21:04.447916+0000","last_peered":"2024-03-04T06:21:04.447916+0000","last_clean":"2024-03-04T06:21:04.447916+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:04.447916+0000","last_undegraded":"2024-03-04T06:21:04.447916+0000","last_fullsized":"2024-03-04T06:21:04.447916+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6958758,"num_objects":59,"num_object_clones":43,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":199,"num_read_kb":138502,"num_write":245,"num_write_kb":64414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"792'853","reported_seq":2847,"reported_epoch":794,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:05.932464+0000","last_change":"2024-03-04T06:21:02.359969+0000","last_active":"2024-03-04T06:21:05.932464+0000","last_peered":"2024-03-04T06:21:05.932464+0000","last_clean":"2024-03-04T06:21:05.932464+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:21:05.932464+0000","last_undegraded":"2024-03-04T06:21:05.932464+0000","last_fullsized":"2024-03-04T06:21:05.932464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":853,"log_dups_size":0,"ondisk_log_size":853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10606174,"num_objects":81,"num_object_clones":66,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":467,"num_read_kb":374978,"num_write":356,"num_write_kb":96872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"793'738","reported_seq":2690,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.381214+0000","last_change":"2024-03-04T06:21:04.436565+0000","last_active":"2024-03-04T06:21:05.381214+0000","last_peered":"2024-03-04T06:21:05.381214+0000","last_clean":"2024-03-04T06:21:05.381214+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:21:05.381214+0000","last_undegraded":"2024-03-04T06:21:05.381214+0000","last_fullsized":"2024-03-04T06:21:05.381214+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":738,"log_dups_size":0,"ondisk_log_size":738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10639821,"num_objects":56,"num_object_clones":46,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":272,"num_read_kb":202842,"num_write":270,"num_write_kb":76269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"789'291","reported_seq":2493,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.929148+0000","last_change":"2024-03-04T06:21:03.404428+0000","last_active":"2024-03-04T06:21:05.929148+0000","last_peered":"2024-03-04T06:21:05.929148+0000","last_clean":"2024-03-04T06:21:05.929148+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:21:05.929148+0000","last_undegraded":"2024-03-04T06:21:05.929148+0000","last_fullsized":"2024-03-04T06:21:05.929148+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672643,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":91,"num_read_kb":65484,"num_write":101,"num_write_kb":27678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"790'846","reported_seq":2772,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.389757+0000","last_change":"2024-03-04T06:21:04.403099+0000","last_active":"2024-03-04T06:21:05.389757+0000","last_peered":"2024-03-04T06:21:05.389757+0000","last_clean":"2024-03-04T06:21:05.389757+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:21:05.389757+0000","last_undegraded":"2024-03-04T06:21:05.389757+0000","last_fullsized":"2024-03-04T06:21:05.389757+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":846,"log_dups_size":0,"ondisk_log_size":846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470959,"num_objects":63,"num_object_clones":54,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":374,"num_read_kb":306917,"num_write":309,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2326,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.390414+0000","last_change":"2024-03-04T06:21:04.403012+0000","last_active":"2024-03-04T06:21:05.390414+0000","last_peered":"2024-03-04T06:21:05.390414+0000","last_clean":"2024-03-04T06:21:05.390414+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:21:05.390414+0000","last_undegraded":"2024-03-04T06:21:05.390414+0000","last_fullsized":"2024-03-04T06:21:05.390414+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"792'476","reported_seq":2587,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.925754+0000","last_change":"2024-03-04T06:21:03.888193+0000","last_active":"2024-03-04T06:21:05.925754+0000","last_peered":"2024-03-04T06:21:05.925754+0000","last_clean":"2024-03-04T06:21:05.925754+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:21:05.925754+0000","last_undegraded":"2024-03-04T06:21:05.925754+0000","last_fullsized":"2024-03-04T06:21:05.925754+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6000759,"num_objects":36,"num_object_clones":27,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":249,"num_read_kb":229475,"num_write":163,"num_write_kb":43804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2133,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.912389+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:21:05.912389+0000","last_peered":"2024-03-04T06:21:05.912389+0000","last_clean":"2024-03-04T06:21:05.912389+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:21:05.912389+0000","last_undegraded":"2024-03-04T06:21:05.912389+0000","last_fullsized":"2024-03-04T06:21:05.912389+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.948764+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:21:05.948764+0000","last_peered":"2024-03-04T06:21:05.948764+0000","last_clean":"2024-03-04T06:21:05.948764+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:21:05.948764+0000","last_undegraded":"2024-03-04T06:21:05.948764+0000","last_fullsized":"2024-03-04T06:21:05.948764+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"792'766","reported_seq":2754,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.944229+0000","last_change":"2024-03-04T06:21:04.476739+0000","last_active":"2024-03-04T06:21:05.944229+0000","last_peered":"2024-03-04T06:21:05.944229+0000","last_clean":"2024-03-04T06:21:05.944229+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:21:05.944229+0000","last_undegraded":"2024-03-04T06:21:05.944229+0000","last_fullsized":"2024-03-04T06:21:05.944229+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370117,"num_objects":67,"num_object_clones":53,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":364,"num_read_kb":291649,"num_write":294,"num_write_kb":78583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.389244+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:21:05.389244+0000","last_peered":"2024-03-04T06:21:05.389244+0000","last_clean":"2024-03-04T06:21:05.389244+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:21:05.389244+0000","last_undegraded":"2024-03-04T06:21:05.389244+0000","last_fullsized":"2024-03-04T06:21:05.389244+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"789'487","reported_seq":2644,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.923158+0000","last_change":"2024-03-04T06:21:03.390956+0000","last_active":"2024-03-04T06:21:05.923158+0000","last_peered":"2024-03-04T06:21:05.923158+0000","last_clean":"2024-03-04T06:21:05.923158+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:21:05.923158+0000","last_undegraded":"2024-03-04T06:21:05.923158+0000","last_fullsized":"2024-03-04T06:21:05.923158+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698233,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":270,"num_read_kb":221303,"num_write":175,"num_write_kb":49293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.930065+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:21:05.930065+0000","last_peered":"2024-03-04T06:21:05.930065+0000","last_clean":"2024-03-04T06:21:05.930065+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:21:05.930065+0000","last_undegraded":"2024-03-04T06:21:05.930065+0000","last_fullsized":"2024-03-04T06:21:05.930065+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"792'1375","reported_seq":3181,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.925369+0000","last_change":"2024-03-04T06:21:03.448109+0000","last_active":"2024-03-04T06:21:05.925369+0000","last_peered":"2024-03-04T06:21:05.925369+0000","last_clean":"2024-03-04T06:21:05.925369+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:21:05.925369+0000","last_undegraded":"2024-03-04T06:21:05.925369+0000","last_fullsized":"2024-03-04T06:21:05.925369+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1375,"log_dups_size":0,"ondisk_log_size":1375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7833052,"num_objects":119,"num_object_clones":95,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":713,"num_read_kb":567605,"num_write":549,"num_write_kb":146807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.383520+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:21:05.383520+0000","last_peered":"2024-03-04T06:21:05.383520+0000","last_clean":"2024-03-04T06:21:05.383520+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:21:05.383520+0000","last_undegraded":"2024-03-04T06:21:05.383520+0000","last_fullsized":"2024-03-04T06:21:05.383520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"792'1601","reported_seq":3197,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.392623+0000","last_change":"2024-03-04T06:21:03.478422+0000","last_active":"2024-03-04T06:21:04.392623+0000","last_peered":"2024-03-04T06:21:04.392623+0000","last_clean":"2024-03-04T06:21:04.392623+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:04.392623+0000","last_undegraded":"2024-03-04T06:21:04.392623+0000","last_fullsized":"2024-03-04T06:21:04.392623+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1601,"log_dups_size":0,"ondisk_log_size":1601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20388838,"num_objects":131,"num_object_clones":107,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":591,"num_read_kb":430948,"num_write":606,"num_write_kb":163623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"792'759","reported_seq":2760,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.945620+0000","last_change":"2024-03-04T06:21:04.481076+0000","last_active":"2024-03-04T06:21:05.945620+0000","last_peered":"2024-03-04T06:21:05.945620+0000","last_clean":"2024-03-04T06:21:05.945620+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:21:05.945620+0000","last_undegraded":"2024-03-04T06:21:05.945620+0000","last_fullsized":"2024-03-04T06:21:05.945620+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":759,"log_dups_size":0,"ondisk_log_size":759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3042397,"num_objects":60,"num_object_clones":48,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":400,"num_read_kb":335889,"num_write":278,"num_write_kb":75144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2033,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.399133+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:04.399133+0000","last_peered":"2024-03-04T06:21:04.399133+0000","last_clean":"2024-03-04T06:21:04.399133+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:04.399133+0000","last_undegraded":"2024-03-04T06:21:04.399133+0000","last_fullsized":"2024-03-04T06:21:04.399133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"792'814","reported_seq":2823,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.404105+0000","last_change":"2024-03-04T06:21:03.445748+0000","last_active":"2024-03-04T06:21:04.404105+0000","last_peered":"2024-03-04T06:21:04.404105+0000","last_clean":"2024-03-04T06:21:04.404105+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:04.404105+0000","last_undegraded":"2024-03-04T06:21:04.404105+0000","last_fullsized":"2024-03-04T06:21:04.404105+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":814,"log_dups_size":0,"ondisk_log_size":814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9009542,"num_objects":73,"num_object_clones":57,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":342,"num_read_kb":246342,"num_write":310,"num_write_kb":84047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2000,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.923932+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:21:05.923932+0000","last_peered":"2024-03-04T06:21:05.923932+0000","last_clean":"2024-03-04T06:21:05.923932+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:21:05.923932+0000","last_undegraded":"2024-03-04T06:21:05.923932+0000","last_fullsized":"2024-03-04T06:21:05.923932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"792'1075","reported_seq":2920,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.922107+0000","last_change":"2024-03-04T06:21:03.433911+0000","last_active":"2024-03-04T06:21:05.922107+0000","last_peered":"2024-03-04T06:21:05.922107+0000","last_clean":"2024-03-04T06:21:05.922107+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:21:05.922107+0000","last_undegraded":"2024-03-04T06:21:05.922107+0000","last_fullsized":"2024-03-04T06:21:05.922107+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1075,"log_dups_size":0,"ondisk_log_size":1075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11228033,"num_objects":88,"num_object_clones":74,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":435,"num_read_kb":315324,"num_write":410,"num_write_kb":115144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2033,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.394677+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:04.394677+0000","last_peered":"2024-03-04T06:21:04.394677+0000","last_clean":"2024-03-04T06:21:04.394677+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:04.394677+0000","last_undegraded":"2024-03-04T06:21:04.394677+0000","last_fullsized":"2024-03-04T06:21:04.394677+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"792'1009","reported_seq":2970,"reported_epoch":793,"state":"active+clean","last_fresh":"2024-03-04T06:21:04.389633+0000","last_change":"2024-03-04T06:21:03.444502+0000","last_active":"2024-03-04T06:21:04.389633+0000","last_peered":"2024-03-04T06:21:04.389633+0000","last_clean":"2024-03-04T06:21:04.389633+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:04.389633+0000","last_undegraded":"2024-03-04T06:21:04.389633+0000","last_fullsized":"2024-03-04T06:21:04.389633+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1009,"log_dups_size":0,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3748291,"num_objects":85,"num_object_clones":74,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":507,"num_read_kb":422773,"num_write":381,"num_write_kb":104774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1962,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-04T06:21:05.943514+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:21:05.943514+0000","last_peered":"2024-03-04T06:21:05.943514+0000","last_clean":"2024-03-04T06:21:05.943514+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:21:05.943514+0000","last_undegraded":"2024-03-04T06:21:05.943514+0000","last_fullsized":"2024-03-04T06:21:05.943514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102667617,"num_objects":1051,"num_object_clones":852,"num_object_copies":3153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":167,"num_read":5587,"num_read_kb":4401997,"num_write":4727,"num_write_kb":1289927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12484,"ondisk_log_size":12484,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706464,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":245588,"kb_used_data":188236,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94126252,"statfs":{"total":96636764160,"available":96385282048,"internally_reserved":0,"allocated":192753664,"data_stored":234570560,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706464,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":186796,"kb_used_data":133604,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94185044,"statfs":{"total":96636764160,"available":96445485056,"internally_reserved":0,"allocated":136810496,"data_stored":158346788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":184756,"kb_used_data":131564,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94187084,"statfs":{"total":96636764160,"available":96447574016,"internally_reserved":0,"allocated":134721536,"data_stored":164117065,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":200016,"kb_used_data":146824,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94171824,"statfs":{"total":96636764160,"available":96431947776,"internally_reserved":0,"allocated":150347776,"data_stored":187629693,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:21:10.387 INFO:tasks.rados.rados.0.smithi064.stdout:3141: done (2 left) 2024-03-04T06:21:10.387 INFO:tasks.rados.rados.0.smithi064.stdout:3142: read oid 14 snap -1 2024-03-04T06:21:10.387 INFO:tasks.rados.rados.0.smithi064.stdout:3142: expect (ObjNum 1414 snap 400 seq_num 1414) 2024-03-04T06:21:10.387 INFO:tasks.rados.rados.0.smithi064.stdout:3143: delete oid 15 current snap is 403 2024-03-04T06:21:10.387 INFO:tasks.rados.rados.0.smithi064.stdout:3138: finishing write tid 1 to smithi06431743-47 2024-03-04T06:21:10.387 INFO:tasks.rados.rados.0.smithi064.stdout:3138: finishing write tid 2 to smithi06431743-47 2024-03-04T06:21:10.387 INFO:tasks.rados.rados.0.smithi064.stdout:3139: expect (ObjNum 1251 snap 337 seq_num 1251) 2024-03-04T06:21:10.579 INFO:tasks.rados.rados.0.smithi064.stdout:3138: finishing write tid 3 to smithi06431743-47 2024-03-04T06:21:10.580 INFO:tasks.rados.rados.0.smithi064.stdout:3138: finishing write tid 4 to smithi06431743-47 2024-03-04T06:21:10.580 INFO:tasks.rados.rados.0.smithi064.stdout:3138: oid 47 updating version 0 to 750 2024-03-04T06:21:10.580 INFO:tasks.rados.rados.0.smithi064.stdout:3138: oid 47 version 750 is already newer than 750 2024-03-04T06:21:10.580 INFO:tasks.rados.rados.0.smithi064.stdout:3138: oid 47 version 750 is already newer than 748 2024-03-04T06:21:10.580 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 750 (ObjNum 1428 snap 403 seq_num 1428) dirty exists 2024-03-04T06:21:10.580 INFO:tasks.rados.rados.0.smithi064.stdout:3138: left oid 47 (ObjNum 1428 snap 403 seq_num 1428) 2024-03-04T06:21:10.580 INFO:tasks.rados.rados.0.smithi064.stdout:3138: done (3 left) 2024-03-04T06:21:10.580 INFO:tasks.rados.rados.0.smithi064.stdout:3139: done (2 left) 2024-03-04T06:21:10.580 INFO:tasks.rados.rados.0.smithi064.stdout:3143: done (1 left) 2024-03-04T06:21:10.580 INFO:tasks.rados.rados.0.smithi064.stdout:3144: snap_remove snap 390 2024-03-04T06:21:11.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:11.393 INFO:tasks.rados.rados.0.smithi064.stdout:3144: done (1 left) 2024-03-04T06:21:11.394 INFO:tasks.rados.rados.0.smithi064.stdout:3145: snap_create 2024-03-04T06:21:11.394 INFO:tasks.rados.rados.0.smithi064.stdout:3142: expect (ObjNum 1414 snap 400 seq_num 1414) 2024-03-04T06:21:11.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:11.507+0000 7f20c5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:11.512 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:11.511+0000 7f20c5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:12.395 INFO:tasks.rados.rados.0.smithi064.stdout:3142: done (1 left) 2024-03-04T06:21:12.395 INFO:tasks.rados.rados.0.smithi064.stdout:3145: done (0 left) 2024-03-04T06:21:12.395 INFO:tasks.rados.rados.0.smithi064.stdout:3146: delete oid 49 current snap is 404 2024-03-04T06:21:12.481 INFO:tasks.rados.rados.0.smithi064.stdout:3146: done (0 left) 2024-03-04T06:21:12.481 INFO:tasks.rados.rados.0.smithi064.stdout:3147: delete oid 33 current snap is 404 2024-03-04T06:21:12.515 INFO:tasks.rados.rados.0.smithi064.stdout:3147: done (0 left) 2024-03-04T06:21:12.515 INFO:tasks.rados.rados.0.smithi064.stdout:3148: snap_remove snap 386 2024-03-04T06:21:13.399 INFO:tasks.rados.rados.0.smithi064.stdout:3148: done (0 left) 2024-03-04T06:21:13.399 INFO:tasks.rados.rados.0.smithi064.stdout:3149: snap_remove snap 362 2024-03-04T06:21:13.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:13.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:13.704+0000 7f8a6339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:13.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:13.708+0000 7f8a6339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:14.400 INFO:tasks.rados.rados.0.smithi064.stdout:3149: done (0 left) 2024-03-04T06:21:14.400 INFO:tasks.rados.rados.0.smithi064.stdout:3150: read oid 4 snap -1 2024-03-04T06:21:14.400 INFO:tasks.rados.rados.0.smithi064.stdout:3150: expect (ObjNum 1400 snap 396 seq_num 1400) 2024-03-04T06:21:14.400 INFO:tasks.rados.rados.0.smithi064.stdout:3151: snap_remove snap 374 2024-03-04T06:21:15.343 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:21:15.343 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:21:15.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:21:15.405 INFO:tasks.rados.rados.0.smithi064.stdout:3151: done (1 left) 2024-03-04T06:21:15.405 INFO:tasks.rados.rados.0.smithi064.stdout:3152: read oid 21 snap -1 2024-03-04T06:21:15.405 INFO:tasks.rados.rados.0.smithi064.stdout:3152: expect deleted 2024-03-04T06:21:15.405 INFO:tasks.rados.rados.0.smithi064.stdout:3153: write (excl) oid 36 current snap is 404 2024-03-04T06:21:15.405 INFO:tasks.rados.rados.0.smithi064.stdout:3153: seq_num 1433 ranges {484431=506181,1639167=445264} 2024-03-04T06:21:15.410 INFO:tasks.rados.rados.0.smithi064.stdout:3153: writing smithi06431743-36 from 484431 to 990612 tid 1 2024-03-04T06:21:15.417 INFO:tasks.rados.rados.0.smithi064.stdout:3153: writing smithi06431743-36 from 1639167 to 2084431 tid 2 2024-03-04T06:21:15.417 INFO:tasks.rados.rados.0.smithi064.stdout:3154: write (excl) oid 33 current snap is 404 2024-03-04T06:21:15.417 INFO:tasks.rados.rados.0.smithi064.stdout:3154: seq_num 1434 ranges {501238=581430,1697649=403589} 2024-03-04T06:21:15.424 INFO:tasks.rados.rados.0.smithi064.stdout:3154: writing smithi06431743-33 from 501238 to 1082668 tid 1 2024-03-04T06:21:15.431 INFO:tasks.rados.rados.0.smithi064.stdout:3154: writing smithi06431743-33 from 1697649 to 2101238 tid 2 2024-03-04T06:21:15.431 INFO:tasks.rados.rados.0.smithi064.stdout:3155: read oid 38 snap -1 2024-03-04T06:21:15.431 INFO:tasks.rados.rados.0.smithi064.stdout:3155: expect (ObjNum 1337 snap 370 seq_num 1337) 2024-03-04T06:21:15.431 INFO:tasks.rados.rados.0.smithi064.stdout:3156: write oid 3 current snap is 404 2024-03-04T06:21:15.431 INFO:tasks.rados.rados.0.smithi064.stdout:3156: seq_num 1435 ranges {518045=656679,1756131=361914} 2024-03-04T06:21:15.438 INFO:tasks.rados.rados.0.smithi064.stdout:3156: writing smithi06431743-3 from 518045 to 1174724 tid 1 2024-03-04T06:21:15.444 INFO:tasks.rados.rados.0.smithi064.stdout:3156: writing smithi06431743-3 from 1756131 to 2118045 tid 2 2024-03-04T06:21:15.444 INFO:tasks.rados.rados.0.smithi064.stdout:3157: write oid 41 current snap is 404 2024-03-04T06:21:15.444 INFO:tasks.rados.rados.0.smithi064.stdout:3150: expect (ObjNum 1400 snap 396 seq_num 1400) 2024-03-04T06:21:15.485 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:15.479+0000 7efd96f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:15.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:15.489+0000 7efd96f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:21:15.735 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:21:15.747 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1200,"stamp":"2024-03-04T06:21:15.403828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91962427,"num_objects":1063,"num_object_clones":862,"num_object_copies":3189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":171,"num_read":5737,"num_read_kb":4438112,"num_write":4902,"num_write_kb":1303447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12791,"ondisk_log_size":12791,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":851652,"kb_used_data":630564,"kb_used_omap":0,"kb_used_meta":221056,"kb_avail":376635708,"statfs":{"total":386547056640,"available":385674964992,"internally_reserved":0,"allocated":645697536,"data_stored":789944022,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":226361344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7463265,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":34,"num_read_kb":31147,"num_write":28,"num_write_kb":8851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012037"},"pg_stats":[{"pgid":"3.f","version":"799'744","reported_seq":2741,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-03-04T06:21:13.933523+0000","last_change":"2024-03-04T06:21:13.936532+0000","last_active":"2024-03-04T06:21:13.933523+0000","last_peered":"2024-03-04T06:21:13.933523+0000","last_clean":"2024-03-04T06:21:13.933523+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:21:13.933523+0000","last_undegraded":"2024-03-04T06:21:13.933523+0000","last_fullsized":"2024-03-04T06:21:13.933523+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":314,"num_read_kb":251966,"num_write":280,"num_write_kb":73822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"803'679","reported_seq":2748,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-03-04T06:21:14.414321+0000","last_change":"2024-03-04T06:21:14.479212+0000","last_active":"2024-03-04T06:21:14.414321+0000","last_peered":"2024-03-04T06:21:14.414321+0000","last_clean":"2024-03-04T06:21:14.414321+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:14.414321+0000","last_undegraded":"2024-03-04T06:21:14.414321+0000","last_fullsized":"2024-03-04T06:21:14.414321+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7597424,"num_objects":60,"num_object_clones":44,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":202,"num_read_kb":138503,"num_write":250,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"799'871","reported_seq":2869,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.413706+0000","last_change":"2024-03-04T06:21:10.527934+0000","last_active":"2024-03-04T06:21:10.413706+0000","last_peered":"2024-03-04T06:21:10.413706+0000","last_clean":"2024-03-04T06:21:10.413706+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:21:10.413706+0000","last_undegraded":"2024-03-04T06:21:10.413706+0000","last_fullsized":"2024-03-04T06:21:10.413706+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6958725,"num_objects":82,"num_object_clones":67,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":468,"num_read_kb":374978,"num_write":357,"num_write_kb":96872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"799'756","reported_seq":2716,"reported_epoch":802,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:13.911941+0000","last_change":"2024-03-04T06:21:13.911941+0000","last_active":"2024-03-04T06:21:13.911941+0000","last_peered":"2024-03-04T06:21:13.911941+0000","last_clean":"2024-03-04T06:21:13.911941+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:21:13.911941+0000","last_undegraded":"2024-03-04T06:21:13.911941+0000","last_fullsized":"2024-03-04T06:21:13.911941+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8942347,"num_objects":57,"num_object_clones":47,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":274,"num_read_kb":202843,"num_write":274,"num_write_kb":77102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"799'298","reported_seq":2515,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.417316+0000","last_change":"2024-03-04T06:21:10.461322+0000","last_active":"2024-03-04T06:21:10.417316+0000","last_peered":"2024-03-04T06:21:10.417316+0000","last_clean":"2024-03-04T06:21:10.417316+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:21:10.417316+0000","last_undegraded":"2024-03-04T06:21:10.417316+0000","last_fullsized":"2024-03-04T06:21:10.417316+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3949975,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":92,"num_read_kb":65485,"num_write":106,"num_write_kb":29352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"799'848","reported_seq":2796,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-03-04T06:21:13.931997+0000","last_change":"2024-03-04T06:21:13.936273+0000","last_active":"2024-03-04T06:21:13.931997+0000","last_peered":"2024-03-04T06:21:13.931997+0000","last_clean":"2024-03-04T06:21:13.931997+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:21:13.931997+0000","last_undegraded":"2024-03-04T06:21:13.931997+0000","last_fullsized":"2024-03-04T06:21:13.931997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":848,"log_dups_size":0,"ondisk_log_size":848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470959,"num_objects":63,"num_object_clones":54,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":374,"num_read_kb":306917,"num_write":309,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2350,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-03-04T06:21:13.934470+0000","last_change":"2024-03-04T06:21:13.936867+0000","last_active":"2024-03-04T06:21:13.934470+0000","last_peered":"2024-03-04T06:21:13.934470+0000","last_clean":"2024-03-04T06:21:13.934470+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:21:13.934470+0000","last_undegraded":"2024-03-04T06:21:13.934470+0000","last_fullsized":"2024-03-04T06:21:13.934470+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"798'480","reported_seq":2607,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.422517+0000","last_change":"2024-03-04T06:21:10.437728+0000","last_active":"2024-03-04T06:21:10.422517+0000","last_peered":"2024-03-04T06:21:10.422517+0000","last_clean":"2024-03-04T06:21:10.422517+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:21:10.422517+0000","last_undegraded":"2024-03-04T06:21:10.422517+0000","last_fullsized":"2024-03-04T06:21:10.422517+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6000759,"num_objects":36,"num_object_clones":27,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":255,"num_read_kb":237043,"num_write":163,"num_write_kb":43804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2146,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.398560+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:21:10.398560+0000","last_peered":"2024-03-04T06:21:10.398560+0000","last_clean":"2024-03-04T06:21:10.398560+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:21:10.398560+0000","last_undegraded":"2024-03-04T06:21:10.398560+0000","last_fullsized":"2024-03-04T06:21:10.398560+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1975,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.430534+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:21:10.430534+0000","last_peered":"2024-03-04T06:21:10.430534+0000","last_clean":"2024-03-04T06:21:10.430534+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:21:10.430534+0000","last_undegraded":"2024-03-04T06:21:10.430534+0000","last_fullsized":"2024-03-04T06:21:10.430534+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"799'778","reported_seq":2776,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.424461+0000","last_change":"2024-03-04T06:21:10.485927+0000","last_active":"2024-03-04T06:21:10.424461+0000","last_peered":"2024-03-04T06:21:10.424461+0000","last_clean":"2024-03-04T06:21:10.424461+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:21:10.424461+0000","last_undegraded":"2024-03-04T06:21:10.424461+0000","last_fullsized":"2024-03-04T06:21:10.424461+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370117,"num_objects":67,"num_object_clones":53,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":372,"num_read_kb":297454,"num_write":294,"num_write_kb":78583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1980,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-03-04T06:21:13.928710+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:21:13.928710+0000","last_peered":"2024-03-04T06:21:13.928710+0000","last_clean":"2024-03-04T06:21:13.928710+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:21:13.928710+0000","last_undegraded":"2024-03-04T06:21:13.928710+0000","last_fullsized":"2024-03-04T06:21:13.928710+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"799'498","reported_seq":2667,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.399911+0000","last_change":"2024-03-04T06:21:10.466745+0000","last_active":"2024-03-04T06:21:10.399911+0000","last_peered":"2024-03-04T06:21:10.399911+0000","last_clean":"2024-03-04T06:21:10.399911+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:21:10.399911+0000","last_undegraded":"2024-03-04T06:21:10.399911+0000","last_fullsized":"2024-03-04T06:21:10.399911+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2739871,"num_objects":45,"num_object_clones":36,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":271,"num_read_kb":221304,"num_write":181,"num_write_kb":51171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1975,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.418019+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:21:10.418019+0000","last_peered":"2024-03-04T06:21:10.418019+0000","last_clean":"2024-03-04T06:21:10.418019+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:21:10.418019+0000","last_undegraded":"2024-03-04T06:21:10.418019+0000","last_fullsized":"2024-03-04T06:21:10.418019+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"799'1384","reported_seq":3203,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.416097+0000","last_change":"2024-03-04T06:21:10.485891+0000","last_active":"2024-03-04T06:21:10.416097+0000","last_peered":"2024-03-04T06:21:10.416097+0000","last_clean":"2024-03-04T06:21:10.416097+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:21:10.416097+0000","last_undegraded":"2024-03-04T06:21:10.416097+0000","last_fullsized":"2024-03-04T06:21:10.416097+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1384,"log_dups_size":0,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8875086,"num_objects":120,"num_object_clones":96,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":714,"num_read_kb":567606,"num_write":554,"num_write_kb":148752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1980,"reported_epoch":802,"state":"active+clean","last_fresh":"2024-03-04T06:21:13.905623+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:21:13.905623+0000","last_peered":"2024-03-04T06:21:13.905623+0000","last_clean":"2024-03-04T06:21:13.905623+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:21:13.905623+0000","last_undegraded":"2024-03-04T06:21:13.905623+0000","last_fullsized":"2024-03-04T06:21:13.905623+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"803'1633","reported_seq":3240,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-03-04T06:21:14.482626+0000","last_change":"2024-03-04T06:21:14.509281+0000","last_active":"2024-03-04T06:21:14.482626+0000","last_peered":"2024-03-04T06:21:14.482626+0000","last_clean":"2024-03-04T06:21:14.482626+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:14.482626+0000","last_undegraded":"2024-03-04T06:21:14.482626+0000","last_fullsized":"2024-03-04T06:21:14.482626+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1633,"log_dups_size":0,"ondisk_log_size":1633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20388838,"num_objects":131,"num_object_clones":107,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":603,"num_read_kb":444849,"num_write":606,"num_write_kb":163623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"799'769","reported_seq":2783,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.484128+0000","last_change":"2024-03-04T06:21:10.494857+0000","last_active":"2024-03-04T06:21:10.484128+0000","last_peered":"2024-03-04T06:21:10.484128+0000","last_clean":"2024-03-04T06:21:10.484128+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:21:10.484128+0000","last_undegraded":"2024-03-04T06:21:10.484128+0000","last_fullsized":"2024-03-04T06:21:10.484128+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":769,"log_dups_size":0,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":61,"num_object_clones":49,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":403,"num_read_kb":335890,"num_write":284,"num_write_kb":76803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2063,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-03-04T06:21:14.420913+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:14.420913+0000","last_peered":"2024-03-04T06:21:14.420913+0000","last_clean":"2024-03-04T06:21:14.420913+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:14.420913+0000","last_undegraded":"2024-03-04T06:21:14.420913+0000","last_fullsized":"2024-03-04T06:21:14.420913+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"803'838","reported_seq":2861,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-03-04T06:21:14.417800+0000","last_change":"2024-03-04T06:21:14.474968+0000","last_active":"2024-03-04T06:21:14.417800+0000","last_peered":"2024-03-04T06:21:14.417800+0000","last_clean":"2024-03-04T06:21:14.417800+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:14.417800+0000","last_undegraded":"2024-03-04T06:21:14.417800+0000","last_fullsized":"2024-03-04T06:21:14.417800+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6236057,"num_objects":74,"num_object_clones":58,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":343,"num_read_kb":246342,"num_write":311,"num_write_kb":84047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2013,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.413123+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:21:10.413123+0000","last_peered":"2024-03-04T06:21:10.413123+0000","last_clean":"2024-03-04T06:21:10.413123+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:21:10.413123+0000","last_undegraded":"2024-03-04T06:21:10.413123+0000","last_fullsized":"2024-03-04T06:21:10.413123+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"799'1100","reported_seq":2945,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.469397+0000","last_change":"2024-03-04T06:21:10.522661+0000","last_active":"2024-03-04T06:21:10.469397+0000","last_peered":"2024-03-04T06:21:10.469397+0000","last_clean":"2024-03-04T06:21:10.469397+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:21:10.469397+0000","last_undegraded":"2024-03-04T06:21:10.469397+0000","last_fullsized":"2024-03-04T06:21:10.469397+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1100,"log_dups_size":0,"ondisk_log_size":1100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11093610,"num_objects":89,"num_object_clones":75,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":442,"num_read_kb":324068,"num_write":415,"num_write_kb":116839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2063,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-03-04T06:21:14.415291+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:14.415291+0000","last_peered":"2024-03-04T06:21:14.415291+0000","last_clean":"2024-03-04T06:21:14.415291+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:14.415291+0000","last_undegraded":"2024-03-04T06:21:14.415291+0000","last_fullsized":"2024-03-04T06:21:14.415291+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"802'1018","reported_seq":3007,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-03-04T06:21:14.407633+0000","last_change":"2024-03-04T06:21:14.422003+0000","last_active":"2024-03-04T06:21:14.407633+0000","last_peered":"2024-03-04T06:21:14.407633+0000","last_clean":"2024-03-04T06:21:14.407633+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:14.407633+0000","last_undegraded":"2024-03-04T06:21:14.407633+0000","last_fullsized":"2024-03-04T06:21:14.407633+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1018,"log_dups_size":0,"ondisk_log_size":1018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3748291,"num_objects":85,"num_object_clones":74,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":507,"num_read_kb":422773,"num_write":381,"num_write_kb":104774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1975,"reported_epoch":799,"state":"active+clean","last_fresh":"2024-03-04T06:21:10.426186+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:21:10.426186+0000","last_peered":"2024-03-04T06:21:10.426186+0000","last_clean":"2024-03-04T06:21:10.426186+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:21:10.426186+0000","last_undegraded":"2024-03-04T06:21:10.426186+0000","last_fullsized":"2024-03-04T06:21:10.426186+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91372059,"num_objects":1061,"num_object_clones":862,"num_object_copies":3183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":171,"num_read":5634,"num_read_kb":4438021,"num_write":4765,"num_write_kb":1301553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12694,"ondisk_log_size":12694,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706469,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":259828,"kb_used_data":198316,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94112012,"statfs":{"total":96636764160,"available":96370700288,"internally_reserved":0,"allocated":203075584,"data_stored":249648504,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706469,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":194352,"kb_used_data":141160,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94177488,"statfs":{"total":96636764160,"available":96437747712,"internally_reserved":0,"allocated":144547840,"data_stored":169594942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706473,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":191268,"kb_used_data":138076,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94180572,"statfs":{"total":96636764160,"available":96440905728,"internally_reserved":0,"allocated":141389824,"data_stored":172983722,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706473,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":206204,"kb_used_data":153012,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94165636,"statfs":{"total":96636764160,"available":96425611264,"internally_reserved":0,"allocated":156684288,"data_stored":197716854,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:21:15.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:15.799 INFO:tasks.rados.rados.0.smithi064.stdout:3157: seq_num 1436 ranges {534852=731928,1814613=320239} 2024-03-04T06:21:15.807 INFO:tasks.rados.rados.0.smithi064.stdout:3157: writing smithi06431743-41 from 534852 to 1266780 tid 1 2024-03-04T06:21:15.812 INFO:tasks.rados.rados.0.smithi064.stdout:3157: writing smithi06431743-41 from 1814613 to 2134852 tid 2 2024-03-04T06:21:15.812 INFO:tasks.rados.rados.0.smithi064.stdout:3150: done (6 left) 2024-03-04T06:21:15.812 INFO:tasks.rados.rados.0.smithi064.stdout:3158: snap_remove snap 403 2024-03-04T06:21:15.887 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:15.885+0000 7fe60ea6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:15.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:15.889+0000 7fe60ea6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:16.406 INFO:tasks.rados.rados.0.smithi064.stdout:3158: done (6 left) 2024-03-04T06:21:16.406 INFO:tasks.rados.rados.0.smithi064.stdout:3159: snap_remove snap 360 2024-03-04T06:21:17.413 INFO:tasks.rados.rados.0.smithi064.stdout:3159: done (6 left) 2024-03-04T06:21:17.414 INFO:tasks.rados.rados.0.smithi064.stdout:3160: read oid 11 snap 376 2024-03-04T06:21:17.414 INFO:tasks.rados.rados.0.smithi064.stdout:3160: expect (ObjNum 1314 snap 361 seq_num 1314) 2024-03-04T06:21:17.414 INFO:tasks.rados.rados.0.smithi064.stdout:3161: snap_remove snap 394 2024-03-04T06:21:17.957 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:18.069 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:18.067+0000 7f328062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:18.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:18.072+0000 7f328062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:18.315 INFO:tasks.rados.rados.0.smithi064.stdout:3161: done (7 left) 2024-03-04T06:21:18.315 INFO:tasks.rados.rados.0.smithi064.stdout:3162: read oid 9 snap 399 2024-03-04T06:21:18.315 INFO:tasks.rados.rados.0.smithi064.stdout:3162: expect deleted 2024-03-04T06:21:18.315 INFO:tasks.rados.rados.0.smithi064.stdout:3163: write oid 20 current snap is 404 2024-03-04T06:21:18.315 INFO:tasks.rados.rados.0.smithi064.stdout:3163: seq_num 1437 ranges {551659=523530,1873095=278564} 2024-03-04T06:21:18.320 INFO:tasks.rados.rados.0.smithi064.stdout:3163: writing smithi06431743-20 from 551659 to 1075189 tid 1 2024-03-04T06:21:18.327 INFO:tasks.rados.rados.0.smithi064.stdout:3163: writing smithi06431743-20 from 1873095 to 2151659 tid 2 2024-03-04T06:21:18.327 INFO:tasks.rados.rados.0.smithi064.stdout:3164: write (excl) oid 13 current snap is 404 2024-03-04T06:21:18.327 INFO:tasks.rados.rados.0.smithi064.stdout:3154: finishing write tid 1 to smithi06431743-33 2024-03-04T06:21:18.327 INFO:tasks.rados.rados.0.smithi064.stdout:3153: finishing write tid 1 to smithi06431743-36 2024-03-04T06:21:18.328 INFO:tasks.rados.rados.0.smithi064.stdout:3154: finishing write tid 2 to smithi06431743-33 2024-03-04T06:21:18.328 INFO:tasks.rados.rados.0.smithi064.stdout:3154: finishing write tid 3 to smithi06431743-33 2024-03-04T06:21:18.328 INFO:tasks.rados.rados.0.smithi064.stdout:3154: finishing write tid 4 to smithi06431743-33 2024-03-04T06:21:18.328 INFO:tasks.rados.rados.0.smithi064.stdout:3154: oid 33 updating version 0 to 1020 2024-03-04T06:21:18.328 INFO:tasks.rados.rados.0.smithi064.stdout:3154: oid 33 updating version 1020 to 1021 2024-03-04T06:21:18.328 INFO:tasks.rados.rados.0.smithi064.stdout:3154: oid 33 version 1021 is already newer than 1019 2024-03-04T06:21:18.328 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 1021 (ObjNum 1434 snap 404 seq_num 1434) dirty exists 2024-03-04T06:21:18.328 INFO:tasks.rados.rados.0.smithi064.stdout:3154: left oid 33 (ObjNum 1434 snap 404 seq_num 1434) 2024-03-04T06:21:18.328 INFO:tasks.rados.rados.0.smithi064.stdout:3164: seq_num 1438 ranges {568466=598779,1931577=236889} 2024-03-04T06:21:18.333 INFO:tasks.rados.rados.0.smithi064.stdout:3164: writing smithi06431743-13 from 568466 to 1167245 tid 1 2024-03-04T06:21:18.338 INFO:tasks.rados.rados.0.smithi064.stdout:3164: writing smithi06431743-13 from 1931577 to 2168466 tid 2 2024-03-04T06:21:18.338 INFO:tasks.rados.rados.0.smithi064.stdout:3152: done (9 left) 2024-03-04T06:21:18.339 INFO:tasks.rados.rados.0.smithi064.stdout:3154: done (8 left) 2024-03-04T06:21:18.339 INFO:tasks.rados.rados.0.smithi064.stdout:3165: write oid 5 current snap is 404 2024-03-04T06:21:18.339 INFO:tasks.rados.rados.0.smithi064.stdout:3165: seq_num 1439 ranges {585273=674028,1990059=195214} 2024-03-04T06:21:18.345 INFO:tasks.rados.rados.0.smithi064.stdout:3165: writing smithi06431743-5 from 585273 to 1259301 tid 1 2024-03-04T06:21:18.350 INFO:tasks.rados.rados.0.smithi064.stdout:3165: writing smithi06431743-5 from 1990059 to 2185273 tid 2 2024-03-04T06:21:18.350 INFO:tasks.rados.rados.0.smithi064.stdout:3166: read oid 27 snap -1 2024-03-04T06:21:18.350 INFO:tasks.rados.rados.0.smithi064.stdout:3166: expect (ObjNum 1417 snap 402 seq_num 1417) 2024-03-04T06:21:18.351 INFO:tasks.rados.rados.0.smithi064.stdout:3167: write oid 2 current snap is 404 2024-03-04T06:21:18.351 INFO:tasks.rados.rados.0.smithi064.stdout:3167: seq_num 1440 ranges {602080=749277,2048541=153539} 2024-03-04T06:21:18.357 INFO:tasks.rados.rados.0.smithi064.stdout:3167: writing smithi06431743-2 from 602080 to 1351357 tid 1 2024-03-04T06:21:18.363 INFO:tasks.rados.rados.0.smithi064.stdout:3167: writing smithi06431743-2 from 2048541 to 2202080 tid 2 2024-03-04T06:21:18.363 INFO:tasks.rados.rados.0.smithi064.stdout:3168: delete oid 44 current snap is 404 2024-03-04T06:21:18.363 INFO:tasks.rados.rados.0.smithi064.stdout:3153: finishing write tid 2 to smithi06431743-36 2024-03-04T06:21:18.363 INFO:tasks.rados.rados.0.smithi064.stdout:3155: expect (ObjNum 1337 snap 370 seq_num 1337) 2024-03-04T06:21:18.481 INFO:tasks.rados.rados.0.smithi064.stdout:3156: finishing write tid 1 to smithi06431743-3 2024-03-04T06:21:18.481 INFO:tasks.rados.rados.0.smithi064.stdout:3153: finishing write tid 3 to smithi06431743-36 2024-03-04T06:21:18.481 INFO:tasks.rados.rados.0.smithi064.stdout:3153: finishing write tid 4 to smithi06431743-36 2024-03-04T06:21:18.481 INFO:tasks.rados.rados.0.smithi064.stdout:3153: oid 36 updating version 0 to 840 2024-03-04T06:21:18.481 INFO:tasks.rados.rados.0.smithi064.stdout:3153: oid 36 updating version 840 to 841 2024-03-04T06:21:18.481 INFO:tasks.rados.rados.0.smithi064.stdout:3153: oid 36 version 841 is already newer than 839 2024-03-04T06:21:18.481 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 841 (ObjNum 1433 snap 404 seq_num 1433) dirty exists 2024-03-04T06:21:18.481 INFO:tasks.rados.rados.0.smithi064.stdout:3153: left oid 36 (ObjNum 1433 snap 404 seq_num 1433) 2024-03-04T06:21:18.481 INFO:tasks.rados.rados.0.smithi064.stdout:3156: finishing write tid 2 to smithi06431743-3 2024-03-04T06:21:18.481 INFO:tasks.rados.rados.0.smithi064.stdout:3156: finishing write tid 3 to smithi06431743-3 2024-03-04T06:21:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:3153: done (11 left) 2024-03-04T06:21:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:3155: done (10 left) 2024-03-04T06:21:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:3168: done (9 left) 2024-03-04T06:21:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:3169: write oid 36 current snap is 404 2024-03-04T06:21:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:3156: finishing write tid 4 to smithi06431743-3 2024-03-04T06:21:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:3156: oid 3 updating version 0 to 794 2024-03-04T06:21:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:3156: oid 3 updating version 794 to 796 2024-03-04T06:21:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:3156: oid 3 version 796 is already newer than 795 2024-03-04T06:21:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 796 (ObjNum 1435 snap 404 seq_num 1435) dirty exists 2024-03-04T06:21:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:3156: left oid 3 (ObjNum 1435 snap 404 seq_num 1435) 2024-03-04T06:21:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:3169: seq_num 1442 ranges {635694=499775,1649152=464169} 2024-03-04T06:21:18.489 INFO:tasks.rados.rados.0.smithi064.stdout:3169: writing smithi06431743-36 from 635694 to 1135469 tid 1 2024-03-04T06:21:18.497 INFO:tasks.rados.rados.0.smithi064.stdout:3169: writing smithi06431743-36 from 1649152 to 2113321 tid 2 2024-03-04T06:21:18.497 INFO:tasks.rados.rados.0.smithi064.stdout:3156: done (9 left) 2024-03-04T06:21:18.497 INFO:tasks.rados.rados.0.smithi064.stdout:3170: delete oid 33 current snap is 404 2024-03-04T06:21:18.497 INFO:tasks.rados.rados.0.smithi064.stdout:3157: finishing write tid 1 to smithi06431743-41 2024-03-04T06:21:18.497 INFO:tasks.rados.rados.0.smithi064.stdout:3157: finishing write tid 2 to smithi06431743-41 2024-03-04T06:21:18.497 INFO:tasks.rados.rados.0.smithi064.stdout:3157: finishing write tid 3 to smithi06431743-41 2024-03-04T06:21:18.497 INFO:tasks.rados.rados.0.smithi064.stdout:3157: finishing write tid 4 to smithi06431743-41 2024-03-04T06:21:18.497 INFO:tasks.rados.rados.0.smithi064.stdout:3157: oid 41 updating version 0 to 799 2024-03-04T06:21:18.497 INFO:tasks.rados.rados.0.smithi064.stdout:3157: oid 41 version 799 is already newer than 797 2024-03-04T06:21:18.497 INFO:tasks.rados.rados.0.smithi064.stdout:3157: oid 41 version 799 is already newer than 799 2024-03-04T06:21:18.498 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 799 (ObjNum 1436 snap 404 seq_num 1436) dirty exists 2024-03-04T06:21:18.498 INFO:tasks.rados.rados.0.smithi064.stdout:3157: left oid 41 (ObjNum 1436 snap 404 seq_num 1436) 2024-03-04T06:21:18.498 INFO:tasks.rados.rados.0.smithi064.stdout:3160: expect (ObjNum 1314 snap 361 seq_num 1314) 2024-03-04T06:21:18.592 INFO:tasks.rados.rados.0.smithi064.stdout:3163: finishing write tid 1 to smithi06431743-20 2024-03-04T06:21:18.593 INFO:tasks.rados.rados.0.smithi064.stdout:3163: finishing write tid 2 to smithi06431743-20 2024-03-04T06:21:18.593 INFO:tasks.rados.rados.0.smithi064.stdout:3166: expect (ObjNum 1417 snap 402 seq_num 1417) 2024-03-04T06:21:18.791 INFO:tasks.rados.rados.0.smithi064.stdout:3163: finishing write tid 3 to smithi06431743-20 2024-03-04T06:21:18.791 INFO:tasks.rados.rados.0.smithi064.stdout:3167: finishing write tid 1 to smithi06431743-2 2024-03-04T06:21:18.791 INFO:tasks.rados.rados.0.smithi064.stdout:3164: finishing write tid 1 to smithi06431743-13 2024-03-04T06:21:18.791 INFO:tasks.rados.rados.0.smithi064.stdout:3163: finishing write tid 4 to smithi06431743-20 2024-03-04T06:21:18.791 INFO:tasks.rados.rados.0.smithi064.stdout:3163: oid 20 updating version 0 to 1130 2024-03-04T06:21:18.791 INFO:tasks.rados.rados.0.smithi064.stdout:3163: oid 20 updating version 1130 to 1132 2024-03-04T06:21:18.791 INFO:tasks.rados.rados.0.smithi064.stdout:3163: oid 20 version 1132 is already newer than 1132 2024-03-04T06:21:18.791 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 1132 (ObjNum 1437 snap 404 seq_num 1437) dirty exists 2024-03-04T06:21:18.791 INFO:tasks.rados.rados.0.smithi064.stdout:3163: left oid 20 (ObjNum 1437 snap 404 seq_num 1437) 2024-03-04T06:21:18.791 INFO:tasks.rados.rados.0.smithi064.stdout:3167: finishing write tid 2 to smithi06431743-2 2024-03-04T06:21:18.791 INFO:tasks.rados.rados.0.smithi064.stdout:3169: finishing write tid 1 to smithi06431743-36 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3157: done (9 left) 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3160: done (8 left) 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3162: done (7 left) 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3163: done (6 left) 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3166: done (5 left) 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3170: done (4 left) 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3171: read oid 6 snap 379 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3171: expect deleted 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3172: read oid 1 snap -1 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3172: expect deleted 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3173: delete oid 29 current snap is 404 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3164: finishing write tid 2 to smithi06431743-13 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3167: finishing write tid 3 to smithi06431743-2 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3164: finishing write tid 3 to smithi06431743-13 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3167: finishing write tid 4 to smithi06431743-2 2024-03-04T06:21:18.792 INFO:tasks.rados.rados.0.smithi064.stdout:3167: oid 2 updating version 0 to 780 2024-03-04T06:21:18.793 INFO:tasks.rados.rados.0.smithi064.stdout:3167: oid 2 version 780 is already newer than 778 2024-03-04T06:21:18.793 INFO:tasks.rados.rados.0.smithi064.stdout:3167: oid 2 version 780 is already newer than 780 2024-03-04T06:21:18.793 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 780 (ObjNum 1440 snap 404 seq_num 1440) dirty exists 2024-03-04T06:21:18.793 INFO:tasks.rados.rados.0.smithi064.stdout:3167: left oid 2 (ObjNum 1440 snap 404 seq_num 1440) 2024-03-04T06:21:18.794 INFO:tasks.rados.rados.0.smithi064.stdout:3164: finishing write tid 4 to smithi06431743-13 2024-03-04T06:21:18.794 INFO:tasks.rados.rados.0.smithi064.stdout:3164: oid 13 updating version 0 to 1647 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3164: oid 13 updating version 1647 to 1649 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3164: oid 13 version 1649 is already newer than 1649 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1649 (ObjNum 1438 snap 404 seq_num 1438) dirty exists 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3164: left oid 13 (ObjNum 1438 snap 404 seq_num 1438) 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3169: finishing write tid 2 to smithi06431743-36 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3165: finishing write tid 1 to smithi06431743-5 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3169: finishing write tid 3 to smithi06431743-36 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3169: finishing write tid 4 to smithi06431743-36 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3169: oid 36 updating version 0 to 851 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3169: oid 36 updating version 851 to 852 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3169: oid 36 version 852 is already newer than 850 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 852 (ObjNum 1442 snap 404 seq_num 1442) dirty exists 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3169: left oid 36 (ObjNum 1442 snap 404 seq_num 1442) 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3165: finishing write tid 2 to smithi06431743-5 2024-03-04T06:21:18.795 INFO:tasks.rados.rados.0.smithi064.stdout:3165: finishing write tid 3 to smithi06431743-5 2024-03-04T06:21:18.796 INFO:tasks.rados.rados.0.smithi064.stdout:3165: finishing write tid 4 to smithi06431743-5 2024-03-04T06:21:18.796 INFO:tasks.rados.rados.0.smithi064.stdout:3165: oid 5 updating version 0 to 1651 2024-03-04T06:21:18.796 INFO:tasks.rados.rados.0.smithi064.stdout:3165: oid 5 updating version 1651 to 1653 2024-03-04T06:21:18.796 INFO:tasks.rados.rados.0.smithi064.stdout:3165: oid 5 version 1653 is already newer than 1653 2024-03-04T06:21:18.796 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 1653 (ObjNum 1439 snap 404 seq_num 1439) dirty exists 2024-03-04T06:21:18.796 INFO:tasks.rados.rados.0.smithi064.stdout:3165: left oid 5 (ObjNum 1439 snap 404 seq_num 1439) 2024-03-04T06:21:18.812 INFO:tasks.rados.rados.0.smithi064.stdout:3164: done (6 left) 2024-03-04T06:21:18.812 INFO:tasks.rados.rados.0.smithi064.stdout:3165: done (5 left) 2024-03-04T06:21:18.812 INFO:tasks.rados.rados.0.smithi064.stdout:3167: done (4 left) 2024-03-04T06:21:18.812 INFO:tasks.rados.rados.0.smithi064.stdout:3169: done (3 left) 2024-03-04T06:21:18.812 INFO:tasks.rados.rados.0.smithi064.stdout:3171: done (2 left) 2024-03-04T06:21:18.812 INFO:tasks.rados.rados.0.smithi064.stdout:3172: done (1 left) 2024-03-04T06:21:18.812 INFO:tasks.rados.rados.0.smithi064.stdout:3173: done (0 left) 2024-03-04T06:21:18.812 INFO:tasks.rados.rados.0.smithi064.stdout:3174: snap_remove snap 383 2024-03-04T06:21:19.417 INFO:tasks.rados.rados.0.smithi064.stdout:3174: done (0 left) 2024-03-04T06:21:19.417 INFO:tasks.rados.rados.0.smithi064.stdout:3175: snap_remove snap 379 2024-03-04T06:21:20.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:20.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:20.252+0000 7ff1d7d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:20.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:20.257+0000 7ff1d7d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:20.418 INFO:tasks.rados.rados.0.smithi064.stdout:3175: done (0 left) 2024-03-04T06:21:20.418 INFO:tasks.rados.rados.0.smithi064.stdout:3176: write oid 21 current snap is 404 2024-03-04T06:21:20.418 INFO:tasks.rados.rados.0.smithi064.stdout:3176: seq_num 1445 ranges {686115=441875,1824598=461517} 2024-03-04T06:21:20.424 INFO:tasks.rados.rados.0.smithi064.stdout:3176: writing smithi06431743-21 from 686115 to 1127990 tid 1 2024-03-04T06:21:20.431 INFO:tasks.rados.rados.0.smithi064.stdout:3176: writing smithi06431743-21 from 1824598 to 2286115 tid 2 2024-03-04T06:21:20.431 INFO:tasks.rados.rados.0.smithi064.stdout:3177: snap_remove snap 376 2024-03-04T06:21:20.749 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:21:20.749 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:21:20.749 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:21:20.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:20.854+0000 7f65e375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:20.861 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:20.861+0000 7f65e375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:21:21.051 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:21:21.063 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1208,"stamp":"2024-03-04T06:21:20.878467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":86181050,"num_objects":1069,"num_object_clones":868,"num_object_copies":3207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":171,"num_read":5761,"num_read_kb":4452271,"num_write":4937,"num_write_kb":1310856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12988,"ondisk_log_size":12988,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":862896,"kb_used_data":641040,"kb_used_omap":0,"kb_used_meta":221056,"kb_avail":376624464,"statfs":{"total":386547056640,"available":385663451136,"internally_reserved":0,"allocated":656424960,"data_stored":803014851,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":226361344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5260426,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":14157,"num_write":26,"num_write_kb":5385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000659"},"pg_stats":[{"pgid":"3.f","version":"806'754","reported_seq":2761,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-04T06:21:19.510619+0000","last_change":"2024-03-04T06:21:19.443044+0000","last_active":"2024-03-04T06:21:19.510619+0000","last_peered":"2024-03-04T06:21:19.510619+0000","last_clean":"2024-03-04T06:21:19.510619+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:21:19.510619+0000","last_undegraded":"2024-03-04T06:21:19.510619+0000","last_fullsized":"2024-03-04T06:21:19.510619+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":314,"num_read_kb":251966,"num_write":280,"num_write_kb":73822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"807'687","reported_seq":2763,"reported_epoch":808,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:20.355353+0000","last_change":"2024-03-04T06:21:18.352717+0000","last_active":"2024-03-04T06:21:20.355353+0000","last_peered":"2024-03-04T06:21:20.355353+0000","last_clean":"2024-03-04T06:21:20.355353+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:20.355353+0000","last_undegraded":"2024-03-04T06:21:20.355353+0000","last_fullsized":"2024-03-04T06:21:20.355353+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7597424,"num_objects":60,"num_object_clones":44,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":208,"num_read_kb":147964,"num_write":250,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"804'891","reported_seq":2889,"reported_epoch":805,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:16.411933+0000","last_change":"2024-03-04T06:21:16.411933+0000","last_active":"2024-03-04T06:21:16.411933+0000","last_peered":"2024-03-04T06:21:16.411933+0000","last_clean":"2024-03-04T06:21:16.411933+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:21:16.411933+0000","last_undegraded":"2024-03-04T06:21:16.411933+0000","last_fullsized":"2024-03-04T06:21:16.411933+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6958725,"num_objects":82,"num_object_clones":67,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":468,"num_read_kb":374978,"num_write":357,"num_write_kb":96872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"808'786","reported_seq":2742,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-04T06:21:19.470541+0000","last_change":"2024-03-04T06:21:19.464568+0000","last_active":"2024-03-04T06:21:19.470541+0000","last_peered":"2024-03-04T06:21:19.470541+0000","last_clean":"2024-03-04T06:21:19.470541+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:21:19.470541+0000","last_undegraded":"2024-03-04T06:21:19.470541+0000","last_fullsized":"2024-03-04T06:21:19.470541+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":786,"log_dups_size":0,"ondisk_log_size":786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7564206,"num_objects":58,"num_object_clones":48,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":275,"num_read_kb":202844,"num_write":278,"num_write_kb":77984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"802'302","reported_seq":2528,"reported_epoch":804,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:15.445710+0000","last_change":"2024-03-04T06:21:15.445710+0000","last_active":"2024-03-04T06:21:15.445710+0000","last_peered":"2024-03-04T06:21:15.445710+0000","last_clean":"2024-03-04T06:21:15.445710+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:21:15.445710+0000","last_undegraded":"2024-03-04T06:21:15.445710+0000","last_fullsized":"2024-03-04T06:21:15.445710+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3949975,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":92,"num_read_kb":65485,"num_write":106,"num_write_kb":29352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"808'858","reported_seq":2820,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-04T06:21:19.509840+0000","last_change":"2024-03-04T06:21:19.454774+0000","last_active":"2024-03-04T06:21:19.509840+0000","last_peered":"2024-03-04T06:21:19.509840+0000","last_clean":"2024-03-04T06:21:19.509840+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:21:19.509840+0000","last_undegraded":"2024-03-04T06:21:19.509840+0000","last_fullsized":"2024-03-04T06:21:19.509840+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470959,"num_objects":63,"num_object_clones":54,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":380,"num_read_kb":311607,"num_write":309,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2370,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-04T06:21:19.436869+0000","last_change":"2024-03-04T06:21:19.442410+0000","last_active":"2024-03-04T06:21:19.436869+0000","last_peered":"2024-03-04T06:21:19.436869+0000","last_clean":"2024-03-04T06:21:19.436869+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:21:19.436869+0000","last_undegraded":"2024-03-04T06:21:19.436869+0000","last_fullsized":"2024-03-04T06:21:19.436869+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"804'494","reported_seq":2628,"reported_epoch":805,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:16.419543+0000","last_change":"2024-03-04T06:21:16.419543+0000","last_active":"2024-03-04T06:21:16.419543+0000","last_peered":"2024-03-04T06:21:16.419543+0000","last_clean":"2024-03-04T06:21:16.419543+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:21:16.419543+0000","last_undegraded":"2024-03-04T06:21:16.419543+0000","last_fullsized":"2024-03-04T06:21:16.419543+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3597028,"num_objects":37,"num_object_clones":28,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":237043,"num_write":164,"num_write_kb":43804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2157,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-03-04T06:21:15.433043+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:21:15.433043+0000","last_peered":"2024-03-04T06:21:15.433043+0000","last_clean":"2024-03-04T06:21:15.433043+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:21:15.433043+0000","last_undegraded":"2024-03-04T06:21:15.433043+0000","last_fullsized":"2024-03-04T06:21:15.433043+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1988,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-03-04T06:21:15.429699+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:21:15.429699+0000","last_peered":"2024-03-04T06:21:15.429699+0000","last_clean":"2024-03-04T06:21:15.429699+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:21:15.429699+0000","last_undegraded":"2024-03-04T06:21:15.429699+0000","last_fullsized":"2024-03-04T06:21:15.429699+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"804'799","reported_seq":2801,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-03-04T06:21:15.828091+0000","last_change":"2024-03-04T06:21:15.616900+0000","last_active":"2024-03-04T06:21:15.828091+0000","last_peered":"2024-03-04T06:21:15.828091+0000","last_clean":"2024-03-04T06:21:15.828091+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:21:15.828091+0000","last_undegraded":"2024-03-04T06:21:15.828091+0000","last_fullsized":"2024-03-04T06:21:15.828091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4252897,"num_objects":68,"num_object_clones":54,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":374,"num_read_kb":297456,"num_write":302,"num_write_kb":80607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1994,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-04T06:21:19.438743+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:21:19.438743+0000","last_peered":"2024-03-04T06:21:19.438743+0000","last_clean":"2024-03-04T06:21:19.438743+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:21:19.438743+0000","last_undegraded":"2024-03-04T06:21:19.438743+0000","last_fullsized":"2024-03-04T06:21:19.438743+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"802'502","reported_seq":2680,"reported_epoch":804,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:15.420177+0000","last_change":"2024-03-04T06:21:15.420177+0000","last_active":"2024-03-04T06:21:15.420177+0000","last_peered":"2024-03-04T06:21:15.420177+0000","last_clean":"2024-03-04T06:21:15.420177+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:21:15.420177+0000","last_undegraded":"2024-03-04T06:21:15.420177+0000","last_fullsized":"2024-03-04T06:21:15.420177+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2739871,"num_objects":45,"num_object_clones":36,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":271,"num_read_kb":221304,"num_write":181,"num_write_kb":51171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1989,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-03-04T06:21:15.415302+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:21:15.415302+0000","last_peered":"2024-03-04T06:21:15.415302+0000","last_clean":"2024-03-04T06:21:15.415302+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:21:15.415302+0000","last_undegraded":"2024-03-04T06:21:15.415302+0000","last_fullsized":"2024-03-04T06:21:15.415302+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"802'1392","reported_seq":3216,"reported_epoch":804,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:15.447454+0000","last_change":"2024-03-04T06:21:15.447454+0000","last_active":"2024-03-04T06:21:15.447454+0000","last_peered":"2024-03-04T06:21:15.447454+0000","last_clean":"2024-03-04T06:21:15.447454+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:21:15.447454+0000","last_undegraded":"2024-03-04T06:21:15.447454+0000","last_fullsized":"2024-03-04T06:21:15.447454+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1392,"log_dups_size":0,"ondisk_log_size":1392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8875086,"num_objects":120,"num_object_clones":96,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":714,"num_read_kb":567606,"num_write":554,"num_write_kb":148752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1994,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-04T06:21:19.425432+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:21:19.425432+0000","last_peered":"2024-03-04T06:21:19.425432+0000","last_clean":"2024-03-04T06:21:19.425432+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:21:19.425432+0000","last_undegraded":"2024-03-04T06:21:19.425432+0000","last_fullsized":"2024-03-04T06:21:19.425432+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"807'1655","reported_seq":3261,"reported_epoch":808,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:20.359152+0000","last_change":"2024-03-04T06:21:18.358156+0000","last_active":"2024-03-04T06:21:20.359152+0000","last_peered":"2024-03-04T06:21:20.359152+0000","last_clean":"2024-03-04T06:21:20.359152+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:20.359152+0000","last_undegraded":"2024-03-04T06:21:20.359152+0000","last_fullsized":"2024-03-04T06:21:20.359152+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1655,"log_dups_size":0,"ondisk_log_size":1655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14270859,"num_objects":134,"num_object_clones":110,"num_object_copies":402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":606,"num_read_kb":444851,"num_write":615,"num_write_kb":165290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"803'777","reported_seq":2800,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-03-04T06:21:15.431548+0000","last_change":"2024-03-04T06:21:15.462988+0000","last_active":"2024-03-04T06:21:15.431548+0000","last_peered":"2024-03-04T06:21:15.431548+0000","last_clean":"2024-03-04T06:21:15.431548+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:21:15.431548+0000","last_undegraded":"2024-03-04T06:21:15.431548+0000","last_fullsized":"2024-03-04T06:21:15.431548+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":61,"num_object_clones":49,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":403,"num_read_kb":335890,"num_write":284,"num_write_kb":76803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2074,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-04T06:21:20.367470+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:20.367470+0000","last_peered":"2024-03-04T06:21:20.367470+0000","last_clean":"2024-03-04T06:21:20.367470+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:20.367470+0000","last_undegraded":"2024-03-04T06:21:20.367470+0000","last_fullsized":"2024-03-04T06:21:20.367470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"807'852","reported_seq":2881,"reported_epoch":808,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:20.371820+0000","last_change":"2024-03-04T06:21:18.378740+0000","last_active":"2024-03-04T06:21:20.371820+0000","last_peered":"2024-03-04T06:21:20.371820+0000","last_clean":"2024-03-04T06:21:20.371820+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:20.371820+0000","last_undegraded":"2024-03-04T06:21:20.371820+0000","last_fullsized":"2024-03-04T06:21:20.371820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":852,"log_dups_size":0,"ondisk_log_size":852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8471751,"num_objects":74,"num_object_clones":58,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":346,"num_read_kb":246344,"num_write":319,"num_write_kb":85920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2024,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-03-04T06:21:15.436660+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:21:15.436660+0000","last_peered":"2024-03-04T06:21:15.436660+0000","last_clean":"2024-03-04T06:21:15.436660+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:21:15.436660+0000","last_undegraded":"2024-03-04T06:21:15.436660+0000","last_fullsized":"2024-03-04T06:21:15.436660+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"802'1116","reported_seq":2958,"reported_epoch":804,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:15.423011+0000","last_change":"2024-03-04T06:21:15.423011+0000","last_active":"2024-03-04T06:21:15.423011+0000","last_peered":"2024-03-04T06:21:15.423011+0000","last_clean":"2024-03-04T06:21:15.423011+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:21:15.423011+0000","last_undegraded":"2024-03-04T06:21:15.423011+0000","last_fullsized":"2024-03-04T06:21:15.423011+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1116,"log_dups_size":0,"ondisk_log_size":1116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11093610,"num_objects":89,"num_object_clones":75,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":442,"num_read_kb":324068,"num_write":415,"num_write_kb":116839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2074,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-04T06:21:20.365330+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:20.365330+0000","last_peered":"2024-03-04T06:21:20.365330+0000","last_clean":"2024-03-04T06:21:20.365330+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:20.365330+0000","last_undegraded":"2024-03-04T06:21:20.365330+0000","last_fullsized":"2024-03-04T06:21:20.365330+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"807'1026","reported_seq":3025,"reported_epoch":808,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:20.357782+0000","last_change":"2024-03-04T06:21:18.347530+0000","last_active":"2024-03-04T06:21:20.357782+0000","last_peered":"2024-03-04T06:21:20.357782+0000","last_clean":"2024-03-04T06:21:20.357782+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:20.357782+0000","last_undegraded":"2024-03-04T06:21:20.357782+0000","last_fullsized":"2024-03-04T06:21:20.357782+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3748291,"num_objects":85,"num_object_clones":74,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":509,"num_read_kb":422774,"num_write":386,"num_write_kb":105737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1988,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-03-04T06:21:15.426157+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:21:15.426157+0000","last_peered":"2024-03-04T06:21:15.426157+0000","last_clean":"2024-03-04T06:21:15.426157+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:21:15.426157+0000","last_undegraded":"2024-03-04T06:21:15.426157+0000","last_fullsized":"2024-03-04T06:21:15.426157+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":85590682,"num_objects":1067,"num_object_clones":868,"num_object_copies":3201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":171,"num_read":5658,"num_read_kb":4452180,"num_write":4800,"num_write_kb":1308962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12891,"ondisk_log_size":12891,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706474,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":262772,"kb_used_data":201260,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94109068,"statfs":{"total":96636764160,"available":96367685632,"internally_reserved":0,"allocated":206090240,"data_stored":252736325,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706474,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":196364,"kb_used_data":143172,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94175476,"statfs":{"total":96636764160,"available":96435687424,"internally_reserved":0,"allocated":146608128,"data_stored":171731318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706478,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":192704,"kb_used_data":138744,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94179136,"statfs":{"total":96636764160,"available":96439435264,"internally_reserved":0,"allocated":142073856,"data_stored":174460650,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706478,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":211056,"kb_used_data":157864,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94160784,"statfs":{"total":96636764160,"available":96420642816,"internally_reserved":0,"allocated":161652736,"data_stored":204086558,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:21:21.432 INFO:tasks.rados.rados.0.smithi064.stdout:3177: done (1 left) 2024-03-04T06:21:21.433 INFO:tasks.rados.rados.0.smithi064.stdout:3178: read oid 47 snap -1 2024-03-04T06:21:21.433 INFO:tasks.rados.rados.0.smithi064.stdout:3178: expect (ObjNum 1428 snap 403 seq_num 1428) 2024-03-04T06:21:21.433 INFO:tasks.rados.rados.0.smithi064.stdout:3179: read oid 33 snap 401 2024-03-04T06:21:21.433 INFO:tasks.rados.rados.0.smithi064.stdout:3179: expect deleted 2024-03-04T06:21:21.433 INFO:tasks.rados.rados.0.smithi064.stdout:3180: write (excl) oid 20 current snap is 404 2024-03-04T06:21:21.433 INFO:tasks.rados.rados.0.smithi064.stdout:3176: finishing write tid 1 to smithi06431743-21 2024-03-04T06:21:21.433 INFO:tasks.rados.rados.0.smithi064.stdout:3180: seq_num 1446 ranges {702922=517124,1883080=419842} 2024-03-04T06:21:21.439 INFO:tasks.rados.rados.0.smithi064.stdout:3180: writing smithi06431743-20 from 702922 to 1220046 tid 1 2024-03-04T06:21:21.445 INFO:tasks.rados.rados.0.smithi064.stdout:3180: writing smithi06431743-20 from 1883080 to 2302922 tid 2 2024-03-04T06:21:21.445 INFO:tasks.rados.rados.0.smithi064.stdout:3181: snap_create 2024-03-04T06:21:21.445 INFO:tasks.rados.rados.0.smithi064.stdout:3176: finishing write tid 2 to smithi06431743-21 2024-03-04T06:21:21.445 INFO:tasks.rados.rados.0.smithi064.stdout:3176: finishing write tid 3 to smithi06431743-21 2024-03-04T06:21:21.445 INFO:tasks.rados.rados.0.smithi064.stdout:3176: finishing write tid 4 to smithi06431743-21 2024-03-04T06:21:21.445 INFO:tasks.rados.rados.0.smithi064.stdout:3176: oid 21 updating version 0 to 790 2024-03-04T06:21:21.445 INFO:tasks.rados.rados.0.smithi064.stdout:3176: oid 21 version 790 is already newer than 788 2024-03-04T06:21:21.446 INFO:tasks.rados.rados.0.smithi064.stdout:3176: oid 21 version 790 is already newer than 790 2024-03-04T06:21:21.446 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 790 (ObjNum 1445 snap 404 seq_num 1445) dirty exists 2024-03-04T06:21:21.446 INFO:tasks.rados.rados.0.smithi064.stdout:3176: left oid 21 (ObjNum 1445 snap 404 seq_num 1445) 2024-03-04T06:21:21.477 INFO:tasks.rados.rados.0.smithi064.stdout:3178: expect (ObjNum 1428 snap 403 seq_num 1428) 2024-03-04T06:21:21.624 INFO:tasks.rados.rados.0.smithi064.stdout:3180: finishing write tid 1 to smithi06431743-20 2024-03-04T06:21:21.624 INFO:tasks.rados.rados.0.smithi064.stdout:3180: finishing write tid 2 to smithi06431743-20 2024-03-04T06:21:21.624 INFO:tasks.rados.rados.0.smithi064.stdout:3180: finishing write tid 3 to smithi06431743-20 2024-03-04T06:21:21.624 INFO:tasks.rados.rados.0.smithi064.stdout:3180: finishing write tid 4 to smithi06431743-20 2024-03-04T06:21:21.624 INFO:tasks.rados.rados.0.smithi064.stdout:3180: oid 20 updating version 0 to 1149 2024-03-04T06:21:21.624 INFO:tasks.rados.rados.0.smithi064.stdout:3180: oid 20 version 1149 is already newer than 1147 2024-03-04T06:21:21.624 INFO:tasks.rados.rados.0.smithi064.stdout:3180: oid 20 version 1149 is already newer than 1148 2024-03-04T06:21:21.624 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 1149 (ObjNum 1446 snap 404 seq_num 1446) dirty exists 2024-03-04T06:21:21.625 INFO:tasks.rados.rados.0.smithi064.stdout:3180: left oid 20 (ObjNum 1446 snap 404 seq_num 1446) 2024-03-04T06:21:22.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:22.431 INFO:tasks.rados.rados.0.smithi064.stdout:3176: done (4 left) 2024-03-04T06:21:22.431 INFO:tasks.rados.rados.0.smithi064.stdout:3178: done (3 left) 2024-03-04T06:21:22.431 INFO:tasks.rados.rados.0.smithi064.stdout:3179: done (2 left) 2024-03-04T06:21:22.431 INFO:tasks.rados.rados.0.smithi064.stdout:3180: done (1 left) 2024-03-04T06:21:22.431 INFO:tasks.rados.rados.0.smithi064.stdout:3181: done (0 left) 2024-03-04T06:21:22.431 INFO:tasks.rados.rados.0.smithi064.stdout:3182: read oid 37 snap 399 2024-03-04T06:21:22.431 INFO:tasks.rados.rados.0.smithi064.stdout:3182: expect deleted 2024-03-04T06:21:22.431 INFO:tasks.rados.rados.0.smithi064.stdout:3183: delete oid 11 current snap is 405 2024-03-04T06:21:22.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:22.433+0000 7fe9bf95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:22.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:22.438+0000 7fe9bf95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:22.500 INFO:tasks.rados.rados.0.smithi064.stdout:3182: done (1 left) 2024-03-04T06:21:22.500 INFO:tasks.rados.rados.0.smithi064.stdout:3183: done (0 left) 2024-03-04T06:21:22.500 INFO:tasks.rados.rados.0.smithi064.stdout:3184: read oid 38 snap -1 2024-03-04T06:21:22.500 INFO:tasks.rados.rados.0.smithi064.stdout:3184: expect (ObjNum 1337 snap 370 seq_num 1337) 2024-03-04T06:21:22.500 INFO:tasks.rados.rados.0.smithi064.stdout:3185: write (excl) oid 39 current snap is 405 2024-03-04T06:21:22.500 INFO:tasks.rados.rados.0.smithi064.stdout:3185: seq_num 1448 ranges {736536=667622,2000044=336492} 2024-03-04T06:21:22.507 INFO:tasks.rados.rados.0.smithi064.stdout:3185: writing smithi06431743-39 from 736536 to 1404158 tid 1 2024-03-04T06:21:22.514 INFO:tasks.rados.rados.0.smithi064.stdout:3185: writing smithi06431743-39 from 2000044 to 2336536 tid 2 2024-03-04T06:21:22.514 INFO:tasks.rados.rados.0.smithi064.stdout:3186: read oid 35 snap -1 2024-03-04T06:21:22.514 INFO:tasks.rados.rados.0.smithi064.stdout:3186: expect (ObjNum 1418 snap 402 seq_num 1418) 2024-03-04T06:21:22.514 INFO:tasks.rados.rados.0.smithi064.stdout:3187: read oid 5 snap 380 2024-03-04T06:21:22.514 INFO:tasks.rados.rados.0.smithi064.stdout:3187: expect deleted 2024-03-04T06:21:22.514 INFO:tasks.rados.rados.0.smithi064.stdout:3188: write oid 48 current snap is 405 2024-03-04T06:21:22.514 INFO:tasks.rados.rados.0.smithi064.stdout:3188: seq_num 1449 ranges {753343=742871,1942173=411170} 2024-03-04T06:21:22.525 INFO:tasks.rados.rados.0.smithi064.stdout:3188: writing smithi06431743-48 from 753343 to 1496214 tid 1 2024-03-04T06:21:22.530 INFO:tasks.rados.rados.0.smithi064.stdout:3188: writing smithi06431743-48 from 1942173 to 2353343 tid 2 2024-03-04T06:21:22.531 INFO:tasks.rados.rados.0.smithi064.stdout:3189: write oid 42 current snap is 405 2024-03-04T06:21:22.531 INFO:tasks.rados.rados.0.smithi064.stdout:3189: seq_num 1450 ranges {770150=418120,1600655=478845} 2024-03-04T06:21:22.537 INFO:tasks.rados.rados.0.smithi064.stdout:3189: writing smithi06431743-42 from 770150 to 1188270 tid 1 2024-03-04T06:21:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:3189: writing smithi06431743-42 from 1600655 to 2079500 tid 2 2024-03-04T06:21:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:3190: read oid 13 snap 375 2024-03-04T06:21:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:3190: expect (ObjNum 1341 snap 370 seq_num 1341) 2024-03-04T06:21:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:3191: read oid 37 snap -1 2024-03-04T06:21:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:3191: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:21:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:3192: write oid 28 current snap is 405 2024-03-04T06:21:22.543 INFO:tasks.rados.rados.0.smithi064.stdout:3192: seq_num 1451 ranges {786957=493369,2059137=327820} 2024-03-04T06:21:22.550 INFO:tasks.rados.rados.0.smithi064.stdout:3192: writing smithi06431743-28 from 786957 to 1280326 tid 1 2024-03-04T06:21:22.558 INFO:tasks.rados.rados.0.smithi064.stdout:3192: writing smithi06431743-28 from 2059137 to 2386957 tid 2 2024-03-04T06:21:22.558 INFO:tasks.rados.rados.0.smithi064.stdout:3193: snap_create 2024-03-04T06:21:22.558 INFO:tasks.rados.rados.0.smithi064.stdout:3185: finishing write tid 1 to smithi06431743-39 2024-03-04T06:21:22.559 INFO:tasks.rados.rados.0.smithi064.stdout:3184: expect (ObjNum 1337 snap 370 seq_num 1337) 2024-03-04T06:21:22.656 INFO:tasks.rados.rados.0.smithi064.stdout:3190: expect (ObjNum 1341 snap 370 seq_num 1341) 2024-03-04T06:21:22.868 INFO:tasks.rados.rados.0.smithi064.stdout:3189: finishing write tid 1 to smithi06431743-42 2024-03-04T06:21:22.868 INFO:tasks.rados.rados.0.smithi064.stdout:3191: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:21:23.009 INFO:tasks.rados.rados.0.smithi064.stdout:3185: finishing write tid 2 to smithi06431743-39 2024-03-04T06:21:23.009 INFO:tasks.rados.rados.0.smithi064.stdout:3186: expect (ObjNum 1418 snap 402 seq_num 1418) 2024-03-04T06:21:23.344 INFO:tasks.rados.rados.0.smithi064.stdout:3188: finishing write tid 1 to smithi06431743-48 2024-03-04T06:21:23.344 INFO:tasks.rados.rados.0.smithi064.stdout:3192: finishing write tid 1 to smithi06431743-28 2024-03-04T06:21:23.344 INFO:tasks.rados.rados.0.smithi064.stdout:3189: finishing write tid 2 to smithi06431743-42 2024-03-04T06:21:23.344 INFO:tasks.rados.rados.0.smithi064.stdout:3189: finishing write tid 3 to smithi06431743-42 2024-03-04T06:21:23.344 INFO:tasks.rados.rados.0.smithi064.stdout:3189: finishing write tid 4 to smithi06431743-42 2024-03-04T06:21:23.344 INFO:tasks.rados.rados.0.smithi064.stdout:3189: oid 42 updating version 0 to 761 2024-03-04T06:21:23.344 INFO:tasks.rados.rados.0.smithi064.stdout:3189: oid 42 version 761 is already newer than 761 2024-03-04T06:21:23.344 INFO:tasks.rados.rados.0.smithi064.stdout:3189: oid 42 version 761 is already newer than 759 2024-03-04T06:21:23.344 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 761 (ObjNum 1450 snap 405 seq_num 1450) dirty exists 2024-03-04T06:21:23.344 INFO:tasks.rados.rados.0.smithi064.stdout:3189: left oid 42 (ObjNum 1450 snap 405 seq_num 1450) 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3185: finishing write tid 3 to smithi06431743-39 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3188: finishing write tid 2 to smithi06431743-48 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3185: finishing write tid 4 to smithi06431743-39 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3185: oid 39 updating version 0 to 920 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3185: oid 39 updating version 920 to 922 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3185: oid 39 version 922 is already newer than 921 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 922 (ObjNum 1448 snap 405 seq_num 1448) dirty exists 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3185: left oid 39 (ObjNum 1448 snap 405 seq_num 1448) 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3192: finishing write tid 2 to smithi06431743-28 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3188: finishing write tid 3 to smithi06431743-48 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3188: finishing write tid 4 to smithi06431743-48 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3188: oid 48 updating version 0 to 517 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3188: oid 48 updating version 517 to 518 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3188: oid 48 updating version 518 to 519 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 519 (ObjNum 1449 snap 405 seq_num 1449) dirty exists 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3188: left oid 48 (ObjNum 1449 snap 405 seq_num 1449) 2024-03-04T06:21:23.345 INFO:tasks.rados.rados.0.smithi064.stdout:3192: finishing write tid 3 to smithi06431743-28 2024-03-04T06:21:23.346 INFO:tasks.rados.rados.0.smithi064.stdout:3192: finishing write tid 4 to smithi06431743-28 2024-03-04T06:21:23.346 INFO:tasks.rados.rados.0.smithi064.stdout:3192: oid 28 updating version 0 to 1422 2024-03-04T06:21:23.346 INFO:tasks.rados.rados.0.smithi064.stdout:3192: oid 28 version 1422 is already newer than 1422 2024-03-04T06:21:23.346 INFO:tasks.rados.rados.0.smithi064.stdout:3192: oid 28 version 1422 is already newer than 1420 2024-03-04T06:21:23.346 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 1422 (ObjNum 1451 snap 405 seq_num 1451) dirty exists 2024-03-04T06:21:23.346 INFO:tasks.rados.rados.0.smithi064.stdout:3192: left oid 28 (ObjNum 1451 snap 405 seq_num 1451) 2024-03-04T06:21:23.433 INFO:tasks.rados.rados.0.smithi064.stdout:3184: done (9 left) 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3185: done (8 left) 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3186: done (7 left) 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3187: done (6 left) 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3188: done (5 left) 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3189: done (4 left) 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3190: done (3 left) 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3191: done (2 left) 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3192: done (1 left) 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3193: done (0 left) 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3194: read oid 24 snap -1 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3194: expect deleted 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3195: write (excl) oid 3 current snap is 406 2024-03-04T06:21:23.434 INFO:tasks.rados.rados.0.smithi064.stdout:3195: seq_num 1452 ranges {403764=568618,1717619=686145} 2024-03-04T06:21:23.438 INFO:tasks.rados.rados.0.smithi064.stdout:3195: writing smithi06431743-3 from 403764 to 972382 tid 1 2024-03-04T06:21:23.447 INFO:tasks.rados.rados.0.smithi064.stdout:3195: writing smithi06431743-3 from 1717619 to 2403764 tid 2 2024-03-04T06:21:23.447 INFO:tasks.rados.rados.0.smithi064.stdout:3196: snap_remove snap 387 2024-03-04T06:21:24.437 INFO:tasks.rados.rados.0.smithi064.stdout:3196: done (2 left) 2024-03-04T06:21:24.437 INFO:tasks.rados.rados.0.smithi064.stdout:3197: snap_remove snap 401 2024-03-04T06:21:24.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:24.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:24.649+0000 7f975bedd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:24.654 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:24.654+0000 7f975bedd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3197: done (2 left) 2024-03-04T06:21:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3198: read oid 15 snap 397 2024-03-04T06:21:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3198: expect (ObjNum 1371 snap 382 seq_num 1371) 2024-03-04T06:21:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3199: delete oid 11 current snap is 406 2024-03-04T06:21:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3195: finishing write tid 1 to smithi06431743-3 2024-03-04T06:21:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3195: finishing write tid 2 to smithi06431743-3 2024-03-04T06:21:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3195: finishing write tid 3 to smithi06431743-3 2024-03-04T06:21:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3195: finishing write tid 4 to smithi06431743-3 2024-03-04T06:21:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3195: oid 3 updating version 0 to 831 2024-03-04T06:21:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3195: oid 3 updating version 831 to 833 2024-03-04T06:21:25.442 INFO:tasks.rados.rados.0.smithi064.stdout:3195: oid 3 version 833 is already newer than 833 2024-03-04T06:21:25.442 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 833 (ObjNum 1452 snap 406 seq_num 1452) dirty exists 2024-03-04T06:21:25.442 INFO:tasks.rados.rados.0.smithi064.stdout:3195: left oid 3 (ObjNum 1452 snap 406 seq_num 1452) 2024-03-04T06:21:25.494 INFO:tasks.rados.rados.0.smithi064.stdout:3194: done (3 left) 2024-03-04T06:21:25.494 INFO:tasks.rados.rados.0.smithi064.stdout:3195: done (2 left) 2024-03-04T06:21:25.494 INFO:tasks.rados.rados.0.smithi064.stdout:3199: done (1 left) 2024-03-04T06:21:25.494 INFO:tasks.rados.rados.0.smithi064.stdout:3200: read oid 44 snap -1 2024-03-04T06:21:25.494 INFO:tasks.rados.rados.0.smithi064.stdout:3200: expect deleted 2024-03-04T06:21:25.494 INFO:tasks.rados.rados.0.smithi064.stdout:3201: snap_remove snap 345 2024-03-04T06:21:26.065 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:21:26.066 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:21:26.066 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:21:26.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:21:26.185 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:26.180+0000 7fddfed9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:26.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:26.188+0000 7fddfed9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:21:26.201 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:21:26.203 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:21:26.204 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:21:26.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:21:26.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:26.307+0000 7fe65899d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:26.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:26.314+0000 7fe65899d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:21:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:21:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:21:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:21:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:21:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:21:26.327 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:21:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:21:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:21:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:21:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:21:26.328 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:21:26.330 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:21:26.330 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:21:26.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:21:26.441 INFO:tasks.rados.rados.0.smithi064.stdout:3201: done (2 left) 2024-03-04T06:21:26.441 INFO:tasks.rados.rados.0.smithi064.stdout:3202: read oid 50 snap -1 2024-03-04T06:21:26.441 INFO:tasks.rados.rados.0.smithi064.stdout:3202: expect (ObjNum 1413 snap 400 seq_num 1413) 2024-03-04T06:21:26.441 INFO:tasks.rados.rados.0.smithi064.stdout:3198: expect (ObjNum 1371 snap 382 seq_num 1371) 2024-03-04T06:21:26.445 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:26.439+0000 7fb1e5d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:26.449 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:26.448+0000 7fb1e5d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:21:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:21:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:21:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:21:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:21:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:21:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:21:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:21:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:21:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:21:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:21:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:21:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:21:26.473 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:21:26.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:21:26.623 INFO:tasks.rados.rados.0.smithi064.stdout:3198: done (2 left) 2024-03-04T06:21:26.623 INFO:tasks.rados.rados.0.smithi064.stdout:3200: done (1 left) 2024-03-04T06:21:26.623 INFO:tasks.rados.rados.0.smithi064.stdout:3203: read oid 30 snap 372 2024-03-04T06:21:26.623 INFO:tasks.rados.rados.0.smithi064.stdout:3203: expect (ObjNum 1302 snap 358 seq_num 1302) 2024-03-04T06:21:26.623 INFO:tasks.rados.rados.0.smithi064.stdout:3204: write (excl) oid 45 current snap is 406 2024-03-04T06:21:26.623 INFO:tasks.rados.rados.0.smithi064.stdout:3204: seq_num 1454 ranges {437378=435469,1434583=769830} 2024-03-04T06:21:26.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:26.618+0000 7f96a882d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:26.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:26.626+0000 7f96a882d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:26.628 INFO:tasks.rados.rados.0.smithi064.stdout:3204: writing smithi06431743-45 from 437378 to 872847 tid 1 2024-03-04T06:21:26.636 INFO:tasks.rados.rados.0.smithi064.stdout:3204: writing smithi06431743-45 from 1434583 to 2204413 tid 2 2024-03-04T06:21:26.636 INFO:tasks.rados.rados.0.smithi064.stdout:3205: snap_remove snap 369 2024-03-04T06:21:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:21:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:21:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:21:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:21:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:21:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:21:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:21:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:21:26.652 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:21:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:21:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:21:26.653 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:21:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:21:26.656 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:21:26.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:26.825 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:26.824+0000 7f5548424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:26.828 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:26.828+0000 7f5548424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:27.448 INFO:tasks.rados.rados.0.smithi064.stdout:3205: done (3 left) 2024-03-04T06:21:27.449 INFO:tasks.rados.rados.0.smithi064.stdout:3206: write (excl) oid 5 current snap is 406 2024-03-04T06:21:27.449 INFO:tasks.rados.rados.0.smithi064.stdout:3206: seq_num 1455 ranges {454185=510718,1493065=629841} 2024-03-04T06:21:27.453 INFO:tasks.rados.rados.0.smithi064.stdout:3206: writing smithi06431743-5 from 454185 to 964903 tid 1 2024-03-04T06:21:27.460 INFO:tasks.rados.rados.0.smithi064.stdout:3206: writing smithi06431743-5 from 1493065 to 2122906 tid 2 2024-03-04T06:21:27.460 INFO:tasks.rados.rados.0.smithi064.stdout:3207: read oid 10 snap -1 2024-03-04T06:21:27.460 INFO:tasks.rados.rados.0.smithi064.stdout:3207: expect deleted 2024-03-04T06:21:27.460 INFO:tasks.rados.rados.0.smithi064.stdout:3208: snap_create 2024-03-04T06:21:27.460 INFO:tasks.rados.rados.0.smithi064.stdout:3204: finishing write tid 1 to smithi06431743-45 2024-03-04T06:21:27.461 INFO:tasks.rados.rados.0.smithi064.stdout:3202: expect (ObjNum 1413 snap 400 seq_num 1413) 2024-03-04T06:21:27.708 INFO:tasks.rados.rados.0.smithi064.stdout:3204: finishing write tid 2 to smithi06431743-45 2024-03-04T06:21:27.708 INFO:tasks.rados.rados.0.smithi064.stdout:3203: expect (ObjNum 1302 snap 358 seq_num 1302) 2024-03-04T06:21:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:3204: finishing write tid 3 to smithi06431743-45 2024-03-04T06:21:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:3204: finishing write tid 4 to smithi06431743-45 2024-03-04T06:21:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:3204: oid 45 updating version 0 to 940 2024-03-04T06:21:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:3204: oid 45 version 940 is already newer than 939 2024-03-04T06:21:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:3204: oid 45 updating version 940 to 941 2024-03-04T06:21:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 941 (ObjNum 1454 snap 406 seq_num 1454) dirty exists 2024-03-04T06:21:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:3204: left oid 45 (ObjNum 1454 snap 406 seq_num 1454) 2024-03-04T06:21:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:3206: finishing write tid 1 to smithi06431743-5 2024-03-04T06:21:27.907 INFO:tasks.rados.rados.0.smithi064.stdout:3206: finishing write tid 2 to smithi06431743-5 2024-03-04T06:21:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:3206: finishing write tid 3 to smithi06431743-5 2024-03-04T06:21:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:3206: finishing write tid 4 to smithi06431743-5 2024-03-04T06:21:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:3206: oid 5 updating version 0 to 1699 2024-03-04T06:21:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:3206: oid 5 updating version 1699 to 1700 2024-03-04T06:21:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:3206: oid 5 updating version 1700 to 1701 2024-03-04T06:21:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 1701 (ObjNum 1455 snap 406 seq_num 1455) dirty exists 2024-03-04T06:21:27.908 INFO:tasks.rados.rados.0.smithi064.stdout:3206: left oid 5 (ObjNum 1455 snap 406 seq_num 1455) 2024-03-04T06:21:28.314 INFO:tasks.rados.rados.0.smithi064.stdout:3202: done (5 left) 2024-03-04T06:21:28.314 INFO:tasks.rados.rados.0.smithi064.stdout:3203: done (4 left) 2024-03-04T06:21:28.314 INFO:tasks.rados.rados.0.smithi064.stdout:3204: done (3 left) 2024-03-04T06:21:28.314 INFO:tasks.rados.rados.0.smithi064.stdout:3206: done (2 left) 2024-03-04T06:21:28.314 INFO:tasks.rados.rados.0.smithi064.stdout:3207: done (1 left) 2024-03-04T06:21:28.314 INFO:tasks.rados.rados.0.smithi064.stdout:3208: done (0 left) 2024-03-04T06:21:28.314 INFO:tasks.rados.rados.0.smithi064.stdout:3209: snap_remove snap 402 2024-03-04T06:21:28.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:29.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:29.017+0000 7f93f0e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:29.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:29.021+0000 7f93f0e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:29.314 INFO:tasks.rados.rados.0.smithi064.stdout:3209: done (0 left) 2024-03-04T06:21:29.314 INFO:tasks.rados.rados.0.smithi064.stdout:3210: read oid 3 snap -1 2024-03-04T06:21:29.314 INFO:tasks.rados.rados.0.smithi064.stdout:3210: expect (ObjNum 1452 snap 406 seq_num 1452) 2024-03-04T06:21:29.314 INFO:tasks.rados.rados.0.smithi064.stdout:3211: read oid 32 snap -1 2024-03-04T06:21:29.314 INFO:tasks.rados.rados.0.smithi064.stdout:3211: expect (ObjNum 1426 snap 403 seq_num 1426) 2024-03-04T06:21:29.315 INFO:tasks.rados.rados.0.smithi064.stdout:3212: write oid 6 current snap is 407 2024-03-04T06:21:29.315 INFO:tasks.rados.rados.0.smithi064.stdout:3212: seq_num 1456 ranges {470992=585967,1551547=773499} 2024-03-04T06:21:29.322 INFO:tasks.rados.rados.0.smithi064.stdout:3212: writing smithi06431743-6 from 470992 to 1056959 tid 1 2024-03-04T06:21:29.331 INFO:tasks.rados.rados.0.smithi064.stdout:3212: writing smithi06431743-6 from 1551547 to 2325046 tid 2 2024-03-04T06:21:29.331 INFO:tasks.rados.rados.0.smithi064.stdout:3213: snap_remove snap 355 2024-03-04T06:21:30.315 INFO:tasks.rados.rados.0.smithi064.stdout:3213: done (3 left) 2024-03-04T06:21:30.316 INFO:tasks.rados.rados.0.smithi064.stdout:3214: write (excl) oid 17 current snap is 407 2024-03-04T06:21:30.316 INFO:tasks.rados.rados.0.smithi064.stdout:3214: seq_num 1457 ranges {487799=661216,1893676=594123} 2024-03-04T06:21:30.323 INFO:tasks.rados.rados.0.smithi064.stdout:3214: writing smithi06431743-17 from 487799 to 1149015 tid 1 2024-03-04T06:21:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:3214: writing smithi06431743-17 from 1893676 to 2487799 tid 2 2024-03-04T06:21:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:3215: write oid 23 current snap is 407 2024-03-04T06:21:30.333 INFO:tasks.rados.rados.0.smithi064.stdout:3215: seq_num 1458 ranges {504606=736465,1952158=552448} 2024-03-04T06:21:30.341 INFO:tasks.rados.rados.0.smithi064.stdout:3215: writing smithi06431743-23 from 504606 to 1241071 tid 1 2024-03-04T06:21:30.351 INFO:tasks.rados.rados.0.smithi064.stdout:3215: writing smithi06431743-23 from 1952158 to 2504606 tid 2 2024-03-04T06:21:30.351 INFO:tasks.rados.rados.0.smithi064.stdout:3216: read oid 7 snap -1 2024-03-04T06:21:30.351 INFO:tasks.rados.rados.0.smithi064.stdout:3216: expect (ObjNum 1353 snap 377 seq_num 1353) 2024-03-04T06:21:30.351 INFO:tasks.rados.rados.0.smithi064.stdout:3217: read oid 12 snap -1 2024-03-04T06:21:30.351 INFO:tasks.rados.rados.0.smithi064.stdout:3217: expect (ObjNum 1390 snap 392 seq_num 1390) 2024-03-04T06:21:30.351 INFO:tasks.rados.rados.0.smithi064.stdout:3218: delete oid 46 current snap is 407 2024-03-04T06:21:30.351 INFO:tasks.rados.rados.0.smithi064.stdout:3211: expect (ObjNum 1426 snap 403 seq_num 1426) 2024-03-04T06:21:30.655 INFO:tasks.rados.rados.0.smithi064.stdout:3210: expect (ObjNum 1452 snap 406 seq_num 1452) 2024-03-04T06:21:30.780 INFO:tasks.rados.rados.0.smithi064.stdout:3212: finishing write tid 1 to smithi06431743-6 2024-03-04T06:21:30.780 INFO:tasks.rados.rados.0.smithi064.stdout:3212: finishing write tid 2 to smithi06431743-6 2024-03-04T06:21:30.780 INFO:tasks.rados.rados.0.smithi064.stdout:3212: finishing write tid 3 to smithi06431743-6 2024-03-04T06:21:30.780 INFO:tasks.rados.rados.0.smithi064.stdout:3212: finishing write tid 4 to smithi06431743-6 2024-03-04T06:21:30.780 INFO:tasks.rados.rados.0.smithi064.stdout:3212: oid 6 updating version 0 to 802 2024-03-04T06:21:30.781 INFO:tasks.rados.rados.0.smithi064.stdout:3212: oid 6 version 802 is already newer than 801 2024-03-04T06:21:30.781 INFO:tasks.rados.rados.0.smithi064.stdout:3212: oid 6 updating version 802 to 803 2024-03-04T06:21:30.781 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 803 (ObjNum 1456 snap 407 seq_num 1456) dirty exists 2024-03-04T06:21:30.781 INFO:tasks.rados.rados.0.smithi064.stdout:3212: left oid 6 (ObjNum 1456 snap 407 seq_num 1456) 2024-03-04T06:21:30.781 INFO:tasks.rados.rados.0.smithi064.stdout:3210: done (7 left) 2024-03-04T06:21:30.781 INFO:tasks.rados.rados.0.smithi064.stdout:3211: done (6 left) 2024-03-04T06:21:30.781 INFO:tasks.rados.rados.0.smithi064.stdout:3212: done (5 left) 2024-03-04T06:21:30.781 INFO:tasks.rados.rados.0.smithi064.stdout:3218: done (4 left) 2024-03-04T06:21:30.781 INFO:tasks.rados.rados.0.smithi064.stdout:3219: write oid 25 current snap is 407 2024-03-04T06:21:30.781 INFO:tasks.rados.rados.0.smithi064.stdout:3219: seq_num 1460 ranges {538220=603316,1785475=752745} 2024-03-04T06:21:30.786 INFO:tasks.rados.rados.0.smithi064.stdout:3219: writing smithi06431743-25 from 538220 to 1141536 tid 1 2024-03-04T06:21:30.796 INFO:tasks.rados.rados.0.smithi064.stdout:3219: writing smithi06431743-25 from 1785475 to 2538220 tid 2 2024-03-04T06:21:30.796 INFO:tasks.rados.rados.0.smithi064.stdout:3220: read oid 32 snap -1 2024-03-04T06:21:30.796 INFO:tasks.rados.rados.0.smithi064.stdout:3220: expect (ObjNum 1426 snap 403 seq_num 1426) 2024-03-04T06:21:30.796 INFO:tasks.rados.rados.0.smithi064.stdout:3221: write oid 31 current snap is 407 2024-03-04T06:21:30.796 INFO:tasks.rados.rados.0.smithi064.stdout:3221: seq_num 1461 ranges {555027=678565,1727604=640848} 2024-03-04T06:21:30.805 INFO:tasks.rados.rados.0.smithi064.stdout:3221: writing smithi06431743-31 from 555027 to 1233592 tid 1 2024-03-04T06:21:30.814 INFO:tasks.rados.rados.0.smithi064.stdout:3221: writing smithi06431743-31 from 1727604 to 2368452 tid 2 2024-03-04T06:21:30.814 INFO:tasks.rados.rados.0.smithi064.stdout:3222: write oid 24 current snap is 407 2024-03-04T06:21:30.814 INFO:tasks.rados.rados.0.smithi064.stdout:3217: expect (ObjNum 1390 snap 392 seq_num 1390) 2024-03-04T06:21:30.978 INFO:tasks.rados.rados.0.smithi064.stdout:3215: finishing write tid 1 to smithi06431743-23 2024-03-04T06:21:30.978 INFO:tasks.rados.rados.0.smithi064.stdout:3215: finishing write tid 2 to smithi06431743-23 2024-03-04T06:21:30.978 INFO:tasks.rados.rados.0.smithi064.stdout:3216: expect (ObjNum 1353 snap 377 seq_num 1353) 2024-03-04T06:21:31.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:31.177 INFO:tasks.rados.rados.0.smithi064.stdout:3214: finishing write tid 1 to smithi06431743-17 2024-03-04T06:21:31.177 INFO:tasks.rados.rados.0.smithi064.stdout:3215: finishing write tid 3 to smithi06431743-23 2024-03-04T06:21:31.177 INFO:tasks.rados.rados.0.smithi064.stdout:3214: finishing write tid 2 to smithi06431743-17 2024-03-04T06:21:31.177 INFO:tasks.rados.rados.0.smithi064.stdout:3222: seq_num 1462 ranges {571834=753814,1786086=784506} 2024-03-04T06:21:31.187 INFO:tasks.rados.rados.0.smithi064.stdout:3222: writing smithi06431743-24 from 571834 to 1325648 tid 1 2024-03-04T06:21:31.198 INFO:tasks.rados.rados.0.smithi064.stdout:3222: writing smithi06431743-24 from 1786086 to 2570592 tid 2 2024-03-04T06:21:31.198 INFO:tasks.rados.rados.0.smithi064.stdout:3216: done (7 left) 2024-03-04T06:21:31.198 INFO:tasks.rados.rados.0.smithi064.stdout:3217: done (6 left) 2024-03-04T06:21:31.198 INFO:tasks.rados.rados.0.smithi064.stdout:3223: snap_create 2024-03-04T06:21:31.198 INFO:tasks.rados.rados.0.smithi064.stdout:3215: finishing write tid 4 to smithi06431743-23 2024-03-04T06:21:31.198 INFO:tasks.rados.rados.0.smithi064.stdout:3215: oid 23 updating version 0 to 855 2024-03-04T06:21:31.198 INFO:tasks.rados.rados.0.smithi064.stdout:3215: oid 23 updating version 855 to 856 2024-03-04T06:21:31.198 INFO:tasks.rados.rados.0.smithi064.stdout:3215: oid 23 version 856 is already newer than 854 2024-03-04T06:21:31.198 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 856 (ObjNum 1458 snap 407 seq_num 1458) dirty exists 2024-03-04T06:21:31.199 INFO:tasks.rados.rados.0.smithi064.stdout:3215: left oid 23 (ObjNum 1458 snap 407 seq_num 1458) 2024-03-04T06:21:31.199 INFO:tasks.rados.rados.0.smithi064.stdout:3214: finishing write tid 3 to smithi06431743-17 2024-03-04T06:21:31.199 INFO:tasks.rados.rados.0.smithi064.stdout:3214: finishing write tid 4 to smithi06431743-17 2024-03-04T06:21:31.199 INFO:tasks.rados.rados.0.smithi064.stdout:3214: oid 17 updating version 0 to 528 2024-03-04T06:21:31.199 INFO:tasks.rados.rados.0.smithi064.stdout:3214: oid 17 updating version 528 to 530 2024-03-04T06:21:31.199 INFO:tasks.rados.rados.0.smithi064.stdout:3214: oid 17 version 530 is already newer than 529 2024-03-04T06:21:31.199 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 530 (ObjNum 1457 snap 407 seq_num 1457) dirty exists 2024-03-04T06:21:31.199 INFO:tasks.rados.rados.0.smithi064.stdout:3214: left oid 17 (ObjNum 1457 snap 407 seq_num 1457) 2024-03-04T06:21:31.199 INFO:tasks.rados.rados.0.smithi064.stdout:3220: expect (ObjNum 1426 snap 403 seq_num 1426) 2024-03-04T06:21:31.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:31.204+0000 7fe52339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:31.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:31.210+0000 7fe52339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:31.471 INFO:tasks.rados.rados.0.smithi064.stdout:3219: finishing write tid 1 to smithi06431743-25 2024-03-04T06:21:31.471 INFO:tasks.rados.rados.0.smithi064.stdout:3221: finishing write tid 1 to smithi06431743-31 2024-03-04T06:21:31.471 INFO:tasks.rados.rados.0.smithi064.stdout:3219: finishing write tid 2 to smithi06431743-25 2024-03-04T06:21:31.471 INFO:tasks.rados.rados.0.smithi064.stdout:3221: finishing write tid 2 to smithi06431743-31 2024-03-04T06:21:31.471 INFO:tasks.rados.rados.0.smithi064.stdout:3219: finishing write tid 3 to smithi06431743-25 2024-03-04T06:21:31.471 INFO:tasks.rados.rados.0.smithi064.stdout:3219: finishing write tid 4 to smithi06431743-25 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3219: oid 25 updating version 0 to 953 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3219: oid 25 version 953 is already newer than 952 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3219: oid 25 version 953 is already newer than 951 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 953 (ObjNum 1460 snap 407 seq_num 1460) dirty exists 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3219: left oid 25 (ObjNum 1460 snap 407 seq_num 1460) 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3221: finishing write tid 3 to smithi06431743-31 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3214: done (6 left) 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3215: done (5 left) 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3219: done (4 left) 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3220: done (3 left) 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3223: done (2 left) 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3224: read oid 37 snap 375 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3224: expect (ObjNum 1296 snap 355 seq_num 1296) 2024-03-04T06:21:31.472 INFO:tasks.rados.rados.0.smithi064.stdout:3225: snap_remove snap 372 2024-03-04T06:21:31.657 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:21:31.658 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:21:31.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:21:31.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:31.761+0000 7f66340dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:31.768 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:31.767+0000 7f66340dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:31.959 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:21:31.960 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:21:31.972 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1224,"stamp":"2024-03-04T06:21:31.316513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91308373,"num_objects":1074,"num_object_clones":873,"num_object_copies":3222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":168,"num_read":5828,"num_read_kb":4515814,"num_write":4974,"num_write_kb":1319572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13409,"ondisk_log_size":13409,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":849944,"kb_used_data":627448,"kb_used_omap":0,"kb_used_meta":222464,"kb_avail":376637416,"statfs":{"total":386547056640,"available":385676713984,"internally_reserved":0,"allocated":642506752,"data_stored":769741810,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":227803136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":268912,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":18098,"num_write":4,"num_write_kb":1115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.871601"},"pg_stats":[{"pgid":"3.f","version":"816'769","reported_seq":2801,"reported_epoch":819,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:30.381265+0000","last_change":"2024-03-04T06:21:30.381265+0000","last_active":"2024-03-04T06:21:30.381265+0000","last_peered":"2024-03-04T06:21:30.381265+0000","last_clean":"2024-03-04T06:21:30.381265+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:21:30.381265+0000","last_undegraded":"2024-03-04T06:21:30.381265+0000","last_fullsized":"2024-03-04T06:21:30.381265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":769,"log_dups_size":0,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370150,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":315,"num_read_kb":251967,"num_write":284,"num_write_kb":74699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"816'717","reported_seq":2801,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-03-04T06:21:29.911159+0000","last_change":"2024-03-04T06:21:29.921766+0000","last_active":"2024-03-04T06:21:29.911159+0000","last_peered":"2024-03-04T06:21:29.911159+0000","last_clean":"2024-03-04T06:21:29.911159+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:29.911159+0000","last_undegraded":"2024-03-04T06:21:29.911159+0000","last_fullsized":"2024-03-04T06:21:29.911159+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7597424,"num_objects":60,"num_object_clones":44,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":214,"num_read_kb":154970,"num_write":250,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"814'930","reported_seq":2933,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-03-04T06:21:25.491668+0000","last_change":"2024-03-04T06:21:25.496847+0000","last_active":"2024-03-04T06:21:25.491668+0000","last_peered":"2024-03-04T06:21:25.491668+0000","last_clean":"2024-03-04T06:21:25.491668+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:21:25.491668+0000","last_undegraded":"2024-03-04T06:21:25.491668+0000","last_fullsized":"2024-03-04T06:21:25.491668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5815882,"num_objects":83,"num_object_clones":68,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":476,"num_read_kb":379931,"num_write":362,"num_write_kb":97853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"818'824","reported_seq":2781,"reported_epoch":819,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:30.378192+0000","last_change":"2024-03-04T06:21:30.378192+0000","last_active":"2024-03-04T06:21:30.378192+0000","last_peered":"2024-03-04T06:21:30.378192+0000","last_clean":"2024-03-04T06:21:30.378192+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:21:30.378192+0000","last_undegraded":"2024-03-04T06:21:30.378192+0000","last_fullsized":"2024-03-04T06:21:30.378192+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7564206,"num_objects":58,"num_object_clones":48,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":281,"num_read_kb":208015,"num_write":278,"num_write_kb":77984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"813'314","reported_seq":2561,"reported_epoch":814,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:26.374801+0000","last_change":"2024-03-04T06:21:26.374801+0000","last_active":"2024-03-04T06:21:26.374801+0000","last_peered":"2024-03-04T06:21:26.374801+0000","last_clean":"2024-03-04T06:21:26.374801+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:21:26.374801+0000","last_undegraded":"2024-03-04T06:21:26.374801+0000","last_fullsized":"2024-03-04T06:21:26.374801+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3949975,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":92,"num_read_kb":65485,"num_write":106,"num_write_kb":29352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"816'870","reported_seq":2859,"reported_epoch":819,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:30.383739+0000","last_change":"2024-03-04T06:21:30.383739+0000","last_active":"2024-03-04T06:21:30.383739+0000","last_peered":"2024-03-04T06:21:30.383739+0000","last_clean":"2024-03-04T06:21:30.383739+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:21:30.383739+0000","last_undegraded":"2024-03-04T06:21:30.383739+0000","last_fullsized":"2024-03-04T06:21:30.383739+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":870,"log_dups_size":0,"ondisk_log_size":870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470959,"num_objects":63,"num_object_clones":54,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":386,"num_read_kb":317573,"num_write":309,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2406,"reported_epoch":819,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:30.386578+0000","last_change":"2024-03-04T06:21:30.386578+0000","last_active":"2024-03-04T06:21:30.386578+0000","last_peered":"2024-03-04T06:21:30.386578+0000","last_clean":"2024-03-04T06:21:30.386578+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:21:30.386578+0000","last_undegraded":"2024-03-04T06:21:30.386578+0000","last_fullsized":"2024-03-04T06:21:30.386578+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"814'516","reported_seq":2663,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-03-04T06:21:25.451341+0000","last_change":"2024-03-04T06:21:25.489853+0000","last_active":"2024-03-04T06:21:25.451341+0000","last_peered":"2024-03-04T06:21:25.451341+0000","last_clean":"2024-03-04T06:21:25.451341+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:21:25.451341+0000","last_undegraded":"2024-03-04T06:21:25.451341+0000","last_fullsized":"2024-03-04T06:21:25.451341+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3597028,"num_objects":37,"num_object_clones":28,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":256,"num_read_kb":237043,"num_write":164,"num_write_kb":43804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2181,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-03-04T06:21:26.346534+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:21:26.346534+0000","last_peered":"2024-03-04T06:21:26.346534+0000","last_clean":"2024-03-04T06:21:26.346534+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:21:26.346534+0000","last_undegraded":"2024-03-04T06:21:26.346534+0000","last_fullsized":"2024-03-04T06:21:26.346534+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2018,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-03-04T06:21:25.462842+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:21:25.462842+0000","last_peered":"2024-03-04T06:21:25.462842+0000","last_clean":"2024-03-04T06:21:25.462842+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:21:25.462842+0000","last_undegraded":"2024-03-04T06:21:25.462842+0000","last_fullsized":"2024-03-04T06:21:25.462842+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"814'841","reported_seq":2843,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-03-04T06:21:25.460461+0000","last_change":"2024-03-04T06:21:25.489428+0000","last_active":"2024-03-04T06:21:25.460461+0000","last_peered":"2024-03-04T06:21:25.460461+0000","last_clean":"2024-03-04T06:21:25.460461+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:21:25.460461+0000","last_undegraded":"2024-03-04T06:21:25.460461+0000","last_fullsized":"2024-03-04T06:21:25.460461+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4538616,"num_objects":69,"num_object_clones":55,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":376,"num_read_kb":297457,"num_write":306,"num_write_kb":81834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2023,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-03-04T06:21:30.381261+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:21:30.381261+0000","last_peered":"2024-03-04T06:21:30.381261+0000","last_clean":"2024-03-04T06:21:30.381261+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:21:30.381261+0000","last_undegraded":"2024-03-04T06:21:30.381261+0000","last_fullsized":"2024-03-04T06:21:30.381261+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"813'521","reported_seq":2717,"reported_epoch":814,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:26.353441+0000","last_change":"2024-03-04T06:21:26.353441+0000","last_active":"2024-03-04T06:21:26.353441+0000","last_peered":"2024-03-04T06:21:26.353441+0000","last_clean":"2024-03-04T06:21:26.353441+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:21:26.353441+0000","last_undegraded":"2024-03-04T06:21:26.353441+0000","last_fullsized":"2024-03-04T06:21:26.353441+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093214,"num_objects":45,"num_object_clones":36,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":272,"num_read_kb":221305,"num_write":185,"num_write_kb":52299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2018,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-03-04T06:21:25.454925+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:21:25.454925+0000","last_peered":"2024-03-04T06:21:25.454925+0000","last_clean":"2024-03-04T06:21:25.454925+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:21:25.454925+0000","last_undegraded":"2024-03-04T06:21:25.454925+0000","last_fullsized":"2024-03-04T06:21:25.454925+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"813'1428","reported_seq":3253,"reported_epoch":814,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:26.370425+0000","last_change":"2024-03-04T06:21:26.370425+0000","last_active":"2024-03-04T06:21:26.370425+0000","last_peered":"2024-03-04T06:21:26.370425+0000","last_clean":"2024-03-04T06:21:26.370425+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:21:26.370425+0000","last_undegraded":"2024-03-04T06:21:26.370425+0000","last_fullsized":"2024-03-04T06:21:26.370425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1428,"log_dups_size":0,"ondisk_log_size":1428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8774277,"num_objects":121,"num_object_clones":97,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":715,"num_read_kb":567607,"num_write":558,"num_write_kb":149555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2023,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-03-04T06:21:30.373135+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:21:30.373135+0000","last_peered":"2024-03-04T06:21:30.373135+0000","last_clean":"2024-03-04T06:21:30.373135+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:21:30.373135+0000","last_undegraded":"2024-03-04T06:21:30.373135+0000","last_fullsized":"2024-03-04T06:21:30.373135+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"818'1713","reported_seq":3306,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-03-04T06:21:29.910198+0000","last_change":"2024-03-04T06:21:29.961208+0000","last_active":"2024-03-04T06:21:29.910198+0000","last_peered":"2024-03-04T06:21:29.910198+0000","last_clean":"2024-03-04T06:21:29.910198+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:29.910198+0000","last_undegraded":"2024-03-04T06:21:29.910198+0000","last_fullsized":"2024-03-04T06:21:29.910198+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1713,"log_dups_size":0,"ondisk_log_size":1713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14539771,"num_objects":135,"num_object_clones":111,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":620,"num_read_kb":459267,"num_write":619,"num_write_kb":166405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"814'794","reported_seq":2845,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-03-04T06:21:25.503705+0000","last_change":"2024-03-04T06:21:25.506070+0000","last_active":"2024-03-04T06:21:25.503705+0000","last_peered":"2024-03-04T06:21:25.503705+0000","last_clean":"2024-03-04T06:21:25.503705+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:21:25.503705+0000","last_undegraded":"2024-03-04T06:21:25.503705+0000","last_fullsized":"2024-03-04T06:21:25.503705+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2286115,"num_objects":61,"num_object_clones":49,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":410,"num_read_kb":341588,"num_write":288,"num_write_kb":77686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2098,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-03-04T06:21:29.922150+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:29.922150+0000","last_peered":"2024-03-04T06:21:29.922150+0000","last_clean":"2024-03-04T06:21:29.922150+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:29.922150+0000","last_undegraded":"2024-03-04T06:21:29.922150+0000","last_fullsized":"2024-03-04T06:21:29.922150+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"816'878","reported_seq":2916,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-03-04T06:21:29.926622+0000","last_change":"2024-03-04T06:21:29.929523+0000","last_active":"2024-03-04T06:21:29.926622+0000","last_peered":"2024-03-04T06:21:29.926622+0000","last_clean":"2024-03-04T06:21:29.926622+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:29.926622+0000","last_undegraded":"2024-03-04T06:21:29.926622+0000","last_fullsized":"2024-03-04T06:21:29.926622+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8471751,"num_objects":74,"num_object_clones":58,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":346,"num_read_kb":246344,"num_write":319,"num_write_kb":85920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2048,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-03-04T06:21:26.367267+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:21:26.367267+0000","last_peered":"2024-03-04T06:21:26.367267+0000","last_clean":"2024-03-04T06:21:26.367267+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:21:26.367267+0000","last_undegraded":"2024-03-04T06:21:26.367267+0000","last_fullsized":"2024-03-04T06:21:26.367267+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"813'1163","reported_seq":3002,"reported_epoch":814,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:26.355942+0000","last_change":"2024-03-04T06:21:26.355942+0000","last_active":"2024-03-04T06:21:26.355942+0000","last_peered":"2024-03-04T06:21:26.355942+0000","last_clean":"2024-03-04T06:21:26.355942+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:21:26.355942+0000","last_undegraded":"2024-03-04T06:21:26.355942+0000","last_fullsized":"2024-03-04T06:21:26.355942+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9900346,"num_objects":90,"num_object_clones":76,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":451,"num_read_kb":334784,"num_write":423,"num_write_kb":118541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2098,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-03-04T06:21:29.905748+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:29.905748+0000","last_peered":"2024-03-04T06:21:29.905748+0000","last_clean":"2024-03-04T06:21:29.905748+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:29.905748+0000","last_undegraded":"2024-03-04T06:21:29.905748+0000","last_fullsized":"2024-03-04T06:21:29.905748+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"815'1034","reported_seq":3062,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-03-04T06:21:29.909332+0000","last_change":"2024-03-04T06:21:29.920192+0000","last_active":"2024-03-04T06:21:29.909332+0000","last_peered":"2024-03-04T06:21:29.909332+0000","last_clean":"2024-03-04T06:21:29.909332+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:29.909332+0000","last_undegraded":"2024-03-04T06:21:29.909332+0000","last_fullsized":"2024-03-04T06:21:29.909332+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1034,"log_dups_size":0,"ondisk_log_size":1034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3748291,"num_objects":85,"num_object_clones":74,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":515,"num_read_kb":432387,"num_write":386,"num_write_kb":105737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2018,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-03-04T06:21:25.458856+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:21:25.458856+0000","last_peered":"2024-03-04T06:21:25.458856+0000","last_clean":"2024-03-04T06:21:25.458856+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:21:25.458856+0000","last_undegraded":"2024-03-04T06:21:25.458856+0000","last_fullsized":"2024-03-04T06:21:25.458856+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":90718005,"num_objects":1072,"num_object_clones":873,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":168,"num_read":5725,"num_read_kb":4515723,"num_write":4837,"num_write_kb":1317678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13312,"ondisk_log_size":13312,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706484,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":270500,"kb_used_data":207580,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94101340,"statfs":{"total":96636764160,"available":96359772160,"internally_reserved":0,"allocated":212561920,"data_stored":259971629,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706484,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":202196,"kb_used_data":149004,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94169644,"statfs":{"total":96636764160,"available":96429715456,"internally_reserved":0,"allocated":152580096,"data_stored":179120684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706488,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181476,"kb_used_data":128284,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94190364,"statfs":{"total":96636764160,"available":96450932736,"internally_reserved":0,"allocated":131362816,"data_stored":156027164,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706488,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":195772,"kb_used_data":142580,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94176068,"statfs":{"total":96636764160,"available":96436293632,"internally_reserved":0,"allocated":146001920,"data_stored":174622333,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:21:32.321 INFO:tasks.rados.rados.0.smithi064.stdout:3225: done (3 left) 2024-03-04T06:21:32.321 INFO:tasks.rados.rados.0.smithi064.stdout:3226: snap_create 2024-03-04T06:21:32.321 INFO:tasks.rados.rados.0.smithi064.stdout:3221: finishing write tid 4 to smithi06431743-31 2024-03-04T06:21:32.321 INFO:tasks.rados.rados.0.smithi064.stdout:3221: oid 31 updating version 0 to 328 2024-03-04T06:21:32.321 INFO:tasks.rados.rados.0.smithi064.stdout:3221: oid 31 version 328 is already newer than 327 2024-03-04T06:21:32.321 INFO:tasks.rados.rados.0.smithi064.stdout:3221: oid 31 version 328 is already newer than 326 2024-03-04T06:21:32.321 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 328 (ObjNum 1461 snap 407 seq_num 1461) dirty exists 2024-03-04T06:21:32.321 INFO:tasks.rados.rados.0.smithi064.stdout:3221: left oid 31 (ObjNum 1461 snap 407 seq_num 1461) 2024-03-04T06:21:32.321 INFO:tasks.rados.rados.0.smithi064.stdout:3222: finishing write tid 1 to smithi06431743-24 2024-03-04T06:21:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:3222: finishing write tid 2 to smithi06431743-24 2024-03-04T06:21:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:3222: finishing write tid 3 to smithi06431743-24 2024-03-04T06:21:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:3222: finishing write tid 4 to smithi06431743-24 2024-03-04T06:21:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:3222: oid 24 updating version 0 to 1195 2024-03-04T06:21:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:3222: oid 24 updating version 1195 to 1196 2024-03-04T06:21:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:3222: oid 24 version 1196 is already newer than 1194 2024-03-04T06:21:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 1196 (ObjNum 1462 snap 407 seq_num 1462) dirty exists 2024-03-04T06:21:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:3222: left oid 24 (ObjNum 1462 snap 407 seq_num 1462) 2024-03-04T06:21:32.322 INFO:tasks.rados.rados.0.smithi064.stdout:3224: expect (ObjNum 1296 snap 355 seq_num 1296) 2024-03-04T06:21:33.290 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:33.331 INFO:tasks.rados.rados.0.smithi064.stdout:3221: done (3 left) 2024-03-04T06:21:33.331 INFO:tasks.rados.rados.0.smithi064.stdout:3222: done (2 left) 2024-03-04T06:21:33.331 INFO:tasks.rados.rados.0.smithi064.stdout:3224: done (1 left) 2024-03-04T06:21:33.331 INFO:tasks.rados.rados.0.smithi064.stdout:3226: done (0 left) 2024-03-04T06:21:33.331 INFO:tasks.rados.rados.0.smithi064.stdout:3227: snap_create 2024-03-04T06:21:33.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:33.409+0000 7f5fd6224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:33.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:33.413+0000 7f5fd6224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:34.332 INFO:tasks.rados.rados.0.smithi064.stdout:3227: done (0 left) 2024-03-04T06:21:34.333 INFO:tasks.rados.rados.0.smithi064.stdout:3228: snap_create 2024-03-04T06:21:35.331 INFO:tasks.rados.rados.0.smithi064.stdout:3228: done (0 left) 2024-03-04T06:21:35.331 INFO:tasks.rados.rados.0.smithi064.stdout:3229: delete oid 31 current snap is 411 2024-03-04T06:21:35.383 INFO:tasks.rados.rados.0.smithi064.stdout:3229: done (0 left) 2024-03-04T06:21:35.383 INFO:tasks.rados.rados.0.smithi064.stdout:3230: delete oid 27 current snap is 411 2024-03-04T06:21:35.436 INFO:tasks.rados.rados.0.smithi064.stdout:3230: done (0 left) 2024-03-04T06:21:35.436 INFO:tasks.rados.rados.0.smithi064.stdout:3231: snap_remove snap 396 2024-03-04T06:21:35.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:35.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:35.598+0000 7f5a69914640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:35.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:35.603+0000 7f5a69914640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:36.342 INFO:tasks.rados.rados.0.smithi064.stdout:3231: done (0 left) 2024-03-04T06:21:36.342 INFO:tasks.rados.rados.0.smithi064.stdout:3232: snap_create 2024-03-04T06:21:36.973 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:21:36.974 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:21:36.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:21:37.095 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:37.091+0000 7f8ee8d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:37.099 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:37.098+0000 7f8ee8d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:37.287 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:21:37.288 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:21:37.299 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1232,"stamp":"2024-03-04T06:21:36.881360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92872315,"num_objects":1079,"num_object_clones":878,"num_object_copies":3237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":166,"num_read":5874,"num_read_kb":4566687,"num_write":5004,"num_write_kb":1328685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13679,"ondisk_log_size":13679,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":666660,"kb_used_data":444164,"kb_used_omap":0,"kb_used_meta":222464,"kb_avail":376820700,"statfs":{"total":386547056640,"available":385864396800,"internally_reserved":0,"allocated":454823936,"data_stored":537582151,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":227803136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2555027,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":12,"num_read_kb":17632,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000777"},"pg_stats":[{"pgid":"3.f","version":"821'775","reported_seq":2813,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.480713+0000","last_change":"2024-03-04T06:21:32.488187+0000","last_active":"2024-03-04T06:21:34.480713+0000","last_peered":"2024-03-04T06:21:34.480713+0000","last_clean":"2024-03-04T06:21:34.480713+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:21:34.480713+0000","last_undegraded":"2024-03-04T06:21:34.480713+0000","last_fullsized":"2024-03-04T06:21:34.480713+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2370150,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":315,"num_read_kb":251967,"num_write":284,"num_write_kb":74699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"821'725","reported_seq":2817,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.500148+0000","last_change":"2024-03-04T06:21:32.478151+0000","last_active":"2024-03-04T06:21:34.500148+0000","last_peered":"2024-03-04T06:21:34.500148+0000","last_clean":"2024-03-04T06:21:34.500148+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:34.500148+0000","last_undegraded":"2024-03-04T06:21:34.500148+0000","last_fullsized":"2024-03-04T06:21:34.500148+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7597424,"num_objects":60,"num_object_clones":44,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":220,"num_read_kb":164051,"num_write":250,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"823'965","reported_seq":2967,"reported_epoch":824,"state":"active+clean","last_fresh":"2024-03-04T06:21:35.343038+0000","last_change":"2024-03-04T06:21:32.519914+0000","last_active":"2024-03-04T06:21:35.343038+0000","last_peered":"2024-03-04T06:21:35.343038+0000","last_clean":"2024-03-04T06:21:35.343038+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:21:35.343038+0000","last_undegraded":"2024-03-04T06:21:35.343038+0000","last_fullsized":"2024-03-04T06:21:35.343038+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":965,"log_dups_size":0,"ondisk_log_size":965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7312134,"num_objects":84,"num_object_clones":69,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":479,"num_read_kb":379933,"num_write":370,"num_write_kb":100357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"821'836","reported_seq":2796,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.471042+0000","last_change":"2024-03-04T06:21:32.504124+0000","last_active":"2024-03-04T06:21:34.471042+0000","last_peered":"2024-03-04T06:21:34.471042+0000","last_clean":"2024-03-04T06:21:34.471042+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:21:34.471042+0000","last_undegraded":"2024-03-04T06:21:34.471042+0000","last_fullsized":"2024-03-04T06:21:34.471042+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":836,"log_dups_size":0,"ondisk_log_size":836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7564206,"num_objects":58,"num_object_clones":48,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":287,"num_read_kb":216566,"num_write":278,"num_write_kb":77984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"825'334","reported_seq":2606,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-03-04T06:21:36.365688+0000","last_change":"2024-03-04T06:21:36.393632+0000","last_active":"2024-03-04T06:21:36.365688+0000","last_peered":"2024-03-04T06:21:36.365688+0000","last_clean":"2024-03-04T06:21:36.365688+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:21:36.365688+0000","last_undegraded":"2024-03-04T06:21:36.365688+0000","last_fullsized":"2024-03-04T06:21:36.365688+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":93,"num_read_kb":65486,"num_write":111,"num_write_kb":30641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"821'878","reported_seq":2871,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.481975+0000","last_change":"2024-03-04T06:21:32.458441+0000","last_active":"2024-03-04T06:21:34.481975+0000","last_peered":"2024-03-04T06:21:34.481975+0000","last_clean":"2024-03-04T06:21:34.481975+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:21:34.481975+0000","last_undegraded":"2024-03-04T06:21:34.481975+0000","last_fullsized":"2024-03-04T06:21:34.481975+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470959,"num_objects":63,"num_object_clones":54,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":386,"num_read_kb":317573,"num_write":309,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2418,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.477918+0000","last_change":"2024-03-04T06:21:32.432355+0000","last_active":"2024-03-04T06:21:34.477918+0000","last_peered":"2024-03-04T06:21:34.477918+0000","last_clean":"2024-03-04T06:21:34.477918+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:21:34.477918+0000","last_undegraded":"2024-03-04T06:21:34.477918+0000","last_fullsized":"2024-03-04T06:21:34.477918+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"821'532","reported_seq":2694,"reported_epoch":824,"state":"active+clean","last_fresh":"2024-03-04T06:21:35.344912+0000","last_change":"2024-03-04T06:21:32.378008+0000","last_active":"2024-03-04T06:21:35.344912+0000","last_peered":"2024-03-04T06:21:35.344912+0000","last_clean":"2024-03-04T06:21:35.344912+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:21:35.344912+0000","last_undegraded":"2024-03-04T06:21:35.344912+0000","last_fullsized":"2024-03-04T06:21:35.344912+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487799,"num_objects":38,"num_object_clones":29,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":258,"num_read_kb":237044,"num_write":168,"num_write_kb":45031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2214,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-03-04T06:21:36.358091+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:21:36.358091+0000","last_peered":"2024-03-04T06:21:36.358091+0000","last_clean":"2024-03-04T06:21:36.358091+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:21:36.358091+0000","last_undegraded":"2024-03-04T06:21:36.358091+0000","last_fullsized":"2024-03-04T06:21:36.358091+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2040,"reported_epoch":824,"state":"active+clean","last_fresh":"2024-03-04T06:21:35.350789+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:21:35.350789+0000","last_peered":"2024-03-04T06:21:35.350789+0000","last_clean":"2024-03-04T06:21:35.350789+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:21:35.350789+0000","last_undegraded":"2024-03-04T06:21:35.350789+0000","last_fullsized":"2024-03-04T06:21:35.350789+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"821'860","reported_seq":2877,"reported_epoch":824,"state":"active+clean","last_fresh":"2024-03-04T06:21:35.358991+0000","last_change":"2024-03-04T06:21:32.411055+0000","last_active":"2024-03-04T06:21:35.358991+0000","last_peered":"2024-03-04T06:21:35.358991+0000","last_clean":"2024-03-04T06:21:35.358991+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:21:35.358991+0000","last_undegraded":"2024-03-04T06:21:35.358991+0000","last_fullsized":"2024-03-04T06:21:35.358991+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":860,"log_dups_size":0,"ondisk_log_size":860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7043222,"num_objects":69,"num_object_clones":55,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":383,"num_read_kb":303688,"num_write":310,"num_write_kb":83094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2033,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.477618+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:21:34.477618+0000","last_peered":"2024-03-04T06:21:34.477618+0000","last_clean":"2024-03-04T06:21:34.477618+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:21:34.477618+0000","last_undegraded":"2024-03-04T06:21:34.477618+0000","last_fullsized":"2024-03-04T06:21:34.477618+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"825'541","reported_seq":2760,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-03-04T06:21:36.349303+0000","last_change":"2024-03-04T06:21:36.395906+0000","last_active":"2024-03-04T06:21:36.349303+0000","last_peered":"2024-03-04T06:21:36.349303+0000","last_clean":"2024-03-04T06:21:36.349303+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:21:36.349303+0000","last_undegraded":"2024-03-04T06:21:36.349303+0000","last_fullsized":"2024-03-04T06:21:36.349303+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093214,"num_objects":45,"num_object_clones":36,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":278,"num_read_kb":227061,"num_write":185,"num_write_kb":52299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2040,"reported_epoch":824,"state":"active+clean","last_fresh":"2024-03-04T06:21:35.340032+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:21:35.340032+0000","last_peered":"2024-03-04T06:21:35.340032+0000","last_clean":"2024-03-04T06:21:35.340032+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:21:35.340032+0000","last_undegraded":"2024-03-04T06:21:35.340032+0000","last_fullsized":"2024-03-04T06:21:35.340032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"821'1468","reported_seq":3299,"reported_epoch":825,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:36.360833+0000","last_change":"2024-03-04T06:21:36.360833+0000","last_active":"2024-03-04T06:21:36.360833+0000","last_peered":"2024-03-04T06:21:36.360833+0000","last_clean":"2024-03-04T06:21:36.360833+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:21:36.360833+0000","last_undegraded":"2024-03-04T06:21:36.360833+0000","last_fullsized":"2024-03-04T06:21:36.360833+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1468,"log_dups_size":0,"ondisk_log_size":1468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353739,"num_objects":122,"num_object_clones":98,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":728,"num_read_kb":588855,"num_write":559,"num_write_kb":149555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2033,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.468304+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:21:34.468304+0000","last_peered":"2024-03-04T06:21:34.468304+0000","last_clean":"2024-03-04T06:21:34.468304+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:21:34.468304+0000","last_undegraded":"2024-03-04T06:21:34.468304+0000","last_fullsized":"2024-03-04T06:21:34.468304+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"821'1733","reported_seq":3319,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.502848+0000","last_change":"2024-03-04T06:21:32.530354+0000","last_active":"2024-03-04T06:21:34.502848+0000","last_peered":"2024-03-04T06:21:34.502848+0000","last_clean":"2024-03-04T06:21:34.502848+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:34.502848+0000","last_undegraded":"2024-03-04T06:21:34.502848+0000","last_fullsized":"2024-03-04T06:21:34.502848+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1733,"log_dups_size":0,"ondisk_log_size":1733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14539771,"num_objects":135,"num_object_clones":111,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":620,"num_read_kb":459267,"num_write":619,"num_write_kb":166405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"819'807","reported_seq":2876,"reported_epoch":824,"state":"active+clean","last_fresh":"2024-03-04T06:21:35.354341+0000","last_change":"2024-03-04T06:21:32.363673+0000","last_active":"2024-03-04T06:21:35.354341+0000","last_peered":"2024-03-04T06:21:35.354341+0000","last_clean":"2024-03-04T06:21:35.354341+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:21:35.354341+0000","last_undegraded":"2024-03-04T06:21:35.354341+0000","last_fullsized":"2024-03-04T06:21:35.354341+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":807,"log_dups_size":0,"ondisk_log_size":807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4757107,"num_objects":61,"num_object_clones":49,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":411,"num_read_kb":341589,"num_write":292,"num_write_kb":79015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2109,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.513435+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:34.513435+0000","last_peered":"2024-03-04T06:21:34.513435+0000","last_clean":"2024-03-04T06:21:34.513435+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:34.513435+0000","last_undegraded":"2024-03-04T06:21:34.513435+0000","last_fullsized":"2024-03-04T06:21:34.513435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"821'888","reported_seq":2929,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.512004+0000","last_change":"2024-03-04T06:21:32.474365+0000","last_active":"2024-03-04T06:21:34.512004+0000","last_peered":"2024-03-04T06:21:34.512004+0000","last_clean":"2024-03-04T06:21:34.512004+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:34.512004+0000","last_undegraded":"2024-03-04T06:21:34.512004+0000","last_fullsized":"2024-03-04T06:21:34.512004+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":888,"log_dups_size":0,"ondisk_log_size":888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8471751,"num_objects":74,"num_object_clones":58,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":346,"num_read_kb":246344,"num_write":319,"num_write_kb":85920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2081,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-03-04T06:21:36.363510+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:21:36.363510+0000","last_peered":"2024-03-04T06:21:36.363510+0000","last_clean":"2024-03-04T06:21:36.363510+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:21:36.363510+0000","last_undegraded":"2024-03-04T06:21:36.363510+0000","last_fullsized":"2024-03-04T06:21:36.363510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"821'1204","reported_seq":3045,"reported_epoch":825,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:36.355139+0000","last_change":"2024-03-04T06:21:36.355139+0000","last_active":"2024-03-04T06:21:36.355139+0000","last_peered":"2024-03-04T06:21:36.355139+0000","last_clean":"2024-03-04T06:21:36.355139+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:21:36.355139+0000","last_undegraded":"2024-03-04T06:21:36.355139+0000","last_fullsized":"2024-03-04T06:21:36.355139+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12472180,"num_objects":90,"num_object_clones":76,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":452,"num_read_kb":334785,"num_write":427,"num_write_kb":120045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2109,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.496495+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:34.496495+0000","last_peered":"2024-03-04T06:21:34.496495+0000","last_clean":"2024-03-04T06:21:34.496495+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:34.496495+0000","last_undegraded":"2024-03-04T06:21:34.496495+0000","last_fullsized":"2024-03-04T06:21:34.496495+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"820'1036","reported_seq":3075,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-03-04T06:21:34.489714+0000","last_change":"2024-03-04T06:21:32.402371+0000","last_active":"2024-03-04T06:21:34.489714+0000","last_peered":"2024-03-04T06:21:34.489714+0000","last_clean":"2024-03-04T06:21:34.489714+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:34.489714+0000","last_undegraded":"2024-03-04T06:21:34.489714+0000","last_fullsized":"2024-03-04T06:21:34.489714+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1036,"log_dups_size":0,"ondisk_log_size":1036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3748291,"num_objects":85,"num_object_clones":74,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":515,"num_read_kb":432387,"num_write":386,"num_write_kb":105737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2040,"reported_epoch":824,"state":"active+clean","last_fresh":"2024-03-04T06:21:35.350215+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:21:35.350215+0000","last_peered":"2024-03-04T06:21:35.350215+0000","last_clean":"2024-03-04T06:21:35.350215+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:21:35.350215+0000","last_undegraded":"2024-03-04T06:21:35.350215+0000","last_fullsized":"2024-03-04T06:21:35.350215+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":92281947,"num_objects":1077,"num_object_clones":878,"num_object_copies":3231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":166,"num_read":5771,"num_read_kb":4566596,"num_write":4867,"num_write_kb":1326791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13582,"ondisk_log_size":13582,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706494,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":207664,"kb_used_data":144744,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94164176,"statfs":{"total":96636764160,"available":96424116224,"internally_reserved":0,"allocated":148217856,"data_stored":176188583,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706494,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153640,"kb_used_data":100448,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94218200,"statfs":{"total":96636764160,"available":96479436800,"internally_reserved":0,"allocated":102858752,"data_stored":117573968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706493,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":145616,"kb_used_data":92424,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94226224,"statfs":{"total":96636764160,"available":96487653376,"internally_reserved":0,"allocated":94642176,"data_stored":114024799,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706493,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":159740,"kb_used_data":106548,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94212100,"statfs":{"total":96636764160,"available":96473190400,"internally_reserved":0,"allocated":109105152,"data_stored":129794801,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:21:37.342 INFO:tasks.rados.rados.0.smithi064.stdout:3232: done (0 left) 2024-03-04T06:21:37.343 INFO:tasks.rados.rados.0.smithi064.stdout:3233: delete oid 33 current snap is 412 2024-03-04T06:21:37.413 INFO:tasks.rados.rados.0.smithi064.stdout:3233: done (0 left) 2024-03-04T06:21:37.413 INFO:tasks.rados.rados.0.smithi064.stdout:3234: read oid 14 snap 406 2024-03-04T06:21:37.414 INFO:tasks.rados.rados.0.smithi064.stdout:3234: expect (ObjNum 1414 snap 400 seq_num 1414) 2024-03-04T06:21:37.414 INFO:tasks.rados.rados.0.smithi064.stdout:3235: read oid 17 snap -1 2024-03-04T06:21:37.414 INFO:tasks.rados.rados.0.smithi064.stdout:3235: expect (ObjNum 1457 snap 407 seq_num 1457) 2024-03-04T06:21:37.414 INFO:tasks.rados.rados.0.smithi064.stdout:3236: write oid 29 current snap is 412 2024-03-04T06:21:37.414 INFO:tasks.rados.rados.0.smithi064.stdout:3236: seq_num 1466 ranges {639062=654810,1903661=735401} 2024-03-04T06:21:37.421 INFO:tasks.rados.rados.0.smithi064.stdout:3236: writing smithi06431743-29 from 639062 to 1293872 tid 1 2024-03-04T06:21:37.429 INFO:tasks.rados.rados.0.smithi064.stdout:3236: writing smithi06431743-29 from 1903661 to 2639062 tid 2 2024-03-04T06:21:37.429 INFO:tasks.rados.rados.0.smithi064.stdout:3237: read oid 2 snap -1 2024-03-04T06:21:37.429 INFO:tasks.rados.rados.0.smithi064.stdout:3237: expect (ObjNum 1440 snap 404 seq_num 1440) 2024-03-04T06:21:37.430 INFO:tasks.rados.rados.0.smithi064.stdout:3238: read oid 38 snap -1 2024-03-04T06:21:37.430 INFO:tasks.rados.rados.0.smithi064.stdout:3238: expect (ObjNum 1337 snap 370 seq_num 1337) 2024-03-04T06:21:37.430 INFO:tasks.rados.rados.0.smithi064.stdout:3239: delete oid 41 current snap is 412 2024-03-04T06:21:37.437 INFO:tasks.rados.rados.0.smithi064.stdout:3234: expect (ObjNum 1414 snap 400 seq_num 1414) 2024-03-04T06:21:37.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:37.777 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:37.775+0000 7f5d5d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:37.780 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:37.779+0000 7f5d5d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:37.873 INFO:tasks.rados.rados.0.smithi064.stdout:3235: expect (ObjNum 1457 snap 407 seq_num 1457) 2024-03-04T06:21:38.033 INFO:tasks.rados.rados.0.smithi064.stdout:3236: finishing write tid 1 to smithi06431743-29 2024-03-04T06:21:38.033 INFO:tasks.rados.rados.0.smithi064.stdout:3238: expect (ObjNum 1337 snap 370 seq_num 1337) 2024-03-04T06:21:38.125 INFO:tasks.rados.rados.0.smithi064.stdout:3236: finishing write tid 2 to smithi06431743-29 2024-03-04T06:21:38.125 INFO:tasks.rados.rados.0.smithi064.stdout:3236: finishing write tid 3 to smithi06431743-29 2024-03-04T06:21:38.125 INFO:tasks.rados.rados.0.smithi064.stdout:3236: finishing write tid 4 to smithi06431743-29 2024-03-04T06:21:38.125 INFO:tasks.rados.rados.0.smithi064.stdout:3236: oid 29 updating version 0 to 1738 2024-03-04T06:21:38.125 INFO:tasks.rados.rados.0.smithi064.stdout:3236: oid 29 updating version 1738 to 1740 2024-03-04T06:21:38.125 INFO:tasks.rados.rados.0.smithi064.stdout:3236: oid 29 version 1740 is already newer than 1740 2024-03-04T06:21:38.125 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 1740 (ObjNum 1466 snap 412 seq_num 1466) dirty exists 2024-03-04T06:21:38.125 INFO:tasks.rados.rados.0.smithi064.stdout:3236: left oid 29 (ObjNum 1466 snap 412 seq_num 1466) 2024-03-04T06:21:38.125 INFO:tasks.rados.rados.0.smithi064.stdout:3237: expect (ObjNum 1440 snap 404 seq_num 1440) 2024-03-04T06:21:38.319 INFO:tasks.rados.rados.0.smithi064.stdout:3234: done (5 left) 2024-03-04T06:21:38.320 INFO:tasks.rados.rados.0.smithi064.stdout:3235: done (4 left) 2024-03-04T06:21:38.320 INFO:tasks.rados.rados.0.smithi064.stdout:3236: done (3 left) 2024-03-04T06:21:38.320 INFO:tasks.rados.rados.0.smithi064.stdout:3237: done (2 left) 2024-03-04T06:21:38.320 INFO:tasks.rados.rados.0.smithi064.stdout:3238: done (1 left) 2024-03-04T06:21:38.320 INFO:tasks.rados.rados.0.smithi064.stdout:3239: done (0 left) 2024-03-04T06:21:38.320 INFO:tasks.rados.rados.0.smithi064.stdout:3240: delete oid 38 current snap is 412 2024-03-04T06:21:38.375 INFO:tasks.rados.rados.0.smithi064.stdout:3240: done (0 left) 2024-03-04T06:21:38.375 INFO:tasks.rados.rados.0.smithi064.stdout:3241: read oid 48 snap 388 2024-03-04T06:21:38.375 INFO:tasks.rados.rados.0.smithi064.stdout:3241: expect (ObjNum 1366 snap 380 seq_num 1366) 2024-03-04T06:21:38.375 INFO:tasks.rados.rados.0.smithi064.stdout:3242: write oid 40 current snap is 412 2024-03-04T06:21:38.376 INFO:tasks.rados.rados.0.smithi064.stdout:3242: seq_num 1469 ranges {689483=596910,2079107=539171} 2024-03-04T06:21:38.382 INFO:tasks.rados.rados.0.smithi064.stdout:3242: writing smithi06431743-40 from 689483 to 1286393 tid 1 2024-03-04T06:21:38.391 INFO:tasks.rados.rados.0.smithi064.stdout:3242: writing smithi06431743-40 from 2079107 to 2618278 tid 2 2024-03-04T06:21:38.391 INFO:tasks.rados.rados.0.smithi064.stdout:3243: write oid 31 current snap is 412 2024-03-04T06:21:38.391 INFO:tasks.rados.rados.0.smithi064.stdout:3243: seq_num 1470 ranges {706290=672159,2137589=568701} 2024-03-04T06:21:38.399 INFO:tasks.rados.rados.0.smithi064.stdout:3243: writing smithi06431743-31 from 706290 to 1378449 tid 1 2024-03-04T06:21:38.408 INFO:tasks.rados.rados.0.smithi064.stdout:3243: writing smithi06431743-31 from 2137589 to 2706290 tid 2 2024-03-04T06:21:38.408 INFO:tasks.rados.rados.0.smithi064.stdout:3244: snap_remove snap 411 2024-03-04T06:21:39.351 INFO:tasks.rados.rados.0.smithi064.stdout:3244: done (3 left) 2024-03-04T06:21:39.351 INFO:tasks.rados.rados.0.smithi064.stdout:3245: snap_remove snap 380 2024-03-04T06:21:39.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:39.971 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:39.969+0000 7f6f51c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:39.974 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:39.973+0000 7f6f51c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:40.354 INFO:tasks.rados.rados.0.smithi064.stdout:3245: done (3 left) 2024-03-04T06:21:40.354 INFO:tasks.rados.rados.0.smithi064.stdout:3246: delete oid 20 current snap is 412 2024-03-04T06:21:40.354 INFO:tasks.rados.rados.0.smithi064.stdout:3241: expect (ObjNum 1366 snap 380 seq_num 1366) 2024-03-04T06:21:40.513 INFO:tasks.rados.rados.0.smithi064.stdout:3242: finishing write tid 1 to smithi06431743-40 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3243: finishing write tid 1 to smithi06431743-31 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3243: finishing write tid 2 to smithi06431743-31 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3242: finishing write tid 2 to smithi06431743-40 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3243: finishing write tid 3 to smithi06431743-31 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3242: finishing write tid 3 to smithi06431743-40 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3241: done (3 left) 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3246: done (2 left) 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3247: read oid 14 snap -1 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3247: expect (ObjNum 1414 snap 400 seq_num 1414) 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3248: delete oid 6 current snap is 412 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3243: finishing write tid 4 to smithi06431743-31 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3243: oid 31 updating version 0 to 336 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3243: oid 31 version 336 is already newer than 335 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:3243: oid 31 updating version 336 to 337 2024-03-04T06:21:40.514 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 337 (ObjNum 1470 snap 412 seq_num 1470) dirty exists 2024-03-04T06:21:40.515 INFO:tasks.rados.rados.0.smithi064.stdout:3243: left oid 31 (ObjNum 1470 snap 412 seq_num 1470) 2024-03-04T06:21:40.515 INFO:tasks.rados.rados.0.smithi064.stdout:3242: finishing write tid 4 to smithi06431743-40 2024-03-04T06:21:40.515 INFO:tasks.rados.rados.0.smithi064.stdout:3242: oid 40 updating version 0 to 776 2024-03-04T06:21:40.515 INFO:tasks.rados.rados.0.smithi064.stdout:3242: oid 40 updating version 776 to 778 2024-03-04T06:21:40.515 INFO:tasks.rados.rados.0.smithi064.stdout:3242: oid 40 version 778 is already newer than 777 2024-03-04T06:21:40.515 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 778 (ObjNum 1469 snap 412 seq_num 1469) dirty exists 2024-03-04T06:21:40.515 INFO:tasks.rados.rados.0.smithi064.stdout:3242: left oid 40 (ObjNum 1469 snap 412 seq_num 1469) 2024-03-04T06:21:40.545 INFO:tasks.rados.rados.0.smithi064.stdout:3242: done (3 left) 2024-03-04T06:21:40.545 INFO:tasks.rados.rados.0.smithi064.stdout:3243: done (2 left) 2024-03-04T06:21:40.545 INFO:tasks.rados.rados.0.smithi064.stdout:3248: done (1 left) 2024-03-04T06:21:40.545 INFO:tasks.rados.rados.0.smithi064.stdout:3249: delete oid 47 current snap is 412 2024-03-04T06:21:40.546 INFO:tasks.rados.rados.0.smithi064.stdout:3247: expect (ObjNum 1414 snap 400 seq_num 1414) 2024-03-04T06:21:40.944 INFO:tasks.rados.rados.0.smithi064.stdout:3247: done (1 left) 2024-03-04T06:21:40.944 INFO:tasks.rados.rados.0.smithi064.stdout:3249: done (0 left) 2024-03-04T06:21:40.944 INFO:tasks.rados.rados.0.smithi064.stdout:3250: read oid 47 snap 404 2024-03-04T06:21:40.944 INFO:tasks.rados.rados.0.smithi064.stdout:3250: expect (ObjNum 1428 snap 403 seq_num 1428) 2024-03-04T06:21:40.945 INFO:tasks.rados.rados.0.smithi064.stdout:3251: snap_create 2024-03-04T06:21:40.958 INFO:tasks.rados.rados.0.smithi064.stdout:3250: expect (ObjNum 1428 snap 403 seq_num 1428) 2024-03-04T06:21:41.361 INFO:tasks.rados.rados.0.smithi064.stdout:3250: done (1 left) 2024-03-04T06:21:41.361 INFO:tasks.rados.rados.0.smithi064.stdout:3251: done (0 left) 2024-03-04T06:21:41.361 INFO:tasks.rados.rados.0.smithi064.stdout:3252: snap_remove snap 393 2024-03-04T06:21:42.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:42.164 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:42.162+0000 7f8e0191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:42.167 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:42.167+0000 7f8e0191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:42.301 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:21:42.301 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:21:42.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:21:42.362 INFO:tasks.rados.rados.0.smithi064.stdout:3252: done (0 left) 2024-03-04T06:21:42.362 INFO:tasks.rados.rados.0.smithi064.stdout:3253: read oid 31 snap -1 2024-03-04T06:21:42.362 INFO:tasks.rados.rados.0.smithi064.stdout:3253: expect (ObjNum 1470 snap 412 seq_num 1470) 2024-03-04T06:21:42.362 INFO:tasks.rados.rados.0.smithi064.stdout:3254: write (excl) oid 50 current snap is 413 2024-03-04T06:21:42.362 INFO:tasks.rados.rados.0.smithi064.stdout:3254: seq_num 1474 ranges {773518=573155,1855164=406520} 2024-03-04T06:21:42.369 INFO:tasks.rados.rados.0.smithi064.stdout:3254: writing smithi06431743-50 from 773518 to 1346673 tid 1 2024-03-04T06:21:42.374 INFO:tasks.rados.rados.0.smithi064.stdout:3254: writing smithi06431743-50 from 1855164 to 2261684 tid 2 2024-03-04T06:21:42.374 INFO:tasks.rados.rados.0.smithi064.stdout:3255: write oid 21 current snap is 413 2024-03-04T06:21:42.374 INFO:tasks.rados.rados.0.smithi064.stdout:3255: seq_num 1475 ranges {790325=764757,2029999=550178} 2024-03-04T06:21:42.382 INFO:tasks.rados.rados.0.smithi064.stdout:3255: writing smithi06431743-21 from 790325 to 1555082 tid 1 2024-03-04T06:21:42.390 INFO:tasks.rados.rados.0.smithi064.stdout:3255: writing smithi06431743-21 from 2029999 to 2580177 tid 2 2024-03-04T06:21:42.390 INFO:tasks.rados.rados.0.smithi064.stdout:3256: snap_create 2024-03-04T06:21:42.427 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:42.418+0000 7fb6eb11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:42.433 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:42.432+0000 7fb6eb11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:42.450 INFO:tasks.rados.rados.0.smithi064.stdout:3253: expect (ObjNum 1470 snap 412 seq_num 1470) 2024-03-04T06:21:42.598 INFO:tasks.rados.rados.0.smithi064.stdout:3255: finishing write tid 1 to smithi06431743-21 2024-03-04T06:21:42.598 INFO:tasks.rados.rados.0.smithi064.stdout:3254: finishing write tid 1 to smithi06431743-50 2024-03-04T06:21:42.598 INFO:tasks.rados.rados.0.smithi064.stdout:3255: finishing write tid 2 to smithi06431743-21 2024-03-04T06:21:42.598 INFO:tasks.rados.rados.0.smithi064.stdout:3255: finishing write tid 3 to smithi06431743-21 2024-03-04T06:21:42.599 INFO:tasks.rados.rados.0.smithi064.stdout:3255: finishing write tid 4 to smithi06431743-21 2024-03-04T06:21:42.599 INFO:tasks.rados.rados.0.smithi064.stdout:3255: oid 21 updating version 0 to 819 2024-03-04T06:21:42.599 INFO:tasks.rados.rados.0.smithi064.stdout:3255: oid 21 version 819 is already newer than 818 2024-03-04T06:21:42.599 INFO:tasks.rados.rados.0.smithi064.stdout:3255: oid 21 version 819 is already newer than 817 2024-03-04T06:21:42.599 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 21 v 819 (ObjNum 1475 snap 413 seq_num 1475) dirty exists 2024-03-04T06:21:42.599 INFO:tasks.rados.rados.0.smithi064.stdout:3255: left oid 21 (ObjNum 1475 snap 413 seq_num 1475) 2024-03-04T06:21:42.599 INFO:tasks.rados.rados.0.smithi064.stdout:3254: finishing write tid 2 to smithi06431743-50 2024-03-04T06:21:42.604 INFO:tasks.rados.rados.0.smithi064.stdout:3254: finishing write tid 3 to smithi06431743-50 2024-03-04T06:21:42.606 INFO:tasks.rados.rados.0.smithi064.stdout:3254: finishing write tid 4 to smithi06431743-50 2024-03-04T06:21:42.606 INFO:tasks.rados.rados.0.smithi064.stdout:3254: oid 50 updating version 0 to 1046 2024-03-04T06:21:42.606 INFO:tasks.rados.rados.0.smithi064.stdout:3254: oid 50 version 1046 is already newer than 1045 2024-03-04T06:21:42.607 INFO:tasks.rados.rados.0.smithi064.stdout:3254: oid 50 updating version 1046 to 1047 2024-03-04T06:21:42.607 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 1047 (ObjNum 1474 snap 413 seq_num 1474) dirty exists 2024-03-04T06:21:42.607 INFO:tasks.rados.rados.0.smithi064.stdout:3254: left oid 50 (ObjNum 1474 snap 413 seq_num 1474) 2024-03-04T06:21:42.677 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:21:42.678 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:21:42.689 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1240,"stamp":"2024-03-04T06:21:42.359682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87711906,"num_objects":1085,"num_object_clones":883,"num_object_copies":3255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":5915,"num_read_kb":4611250,"num_write":5021,"num_write_kb":1332367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13747,"ondisk_log_size":13747,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":673364,"kb_used_data":450868,"kb_used_omap":0,"kb_used_meta":222464,"kb_avail":376813996,"statfs":{"total":386547056640,"available":385857531904,"internally_reserved":0,"allocated":461688832,"data_stored":543185492,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":227803136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5160409,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":41,"num_read_kb":44563,"num_write":17,"num_write_kb":3682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.045979"},"pg_stats":[{"pgid":"3.f","version":"827'778","reported_seq":2826,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-04T06:21:38.430072+0000","last_change":"2024-03-04T06:21:37.516414+0000","last_active":"2024-03-04T06:21:38.430072+0000","last_peered":"2024-03-04T06:21:38.430072+0000","last_clean":"2024-03-04T06:21:38.430072+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:21:38.430072+0000","last_undegraded":"2024-03-04T06:21:38.430072+0000","last_fullsized":"2024-03-04T06:21:38.430072+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5059633,"num_objects":69,"num_object_clones":55,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":316,"num_read_kb":251968,"num_write":288,"num_write_kb":75809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"826'731","reported_seq":2833,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-04T06:21:39.361681+0000","last_change":"2024-03-04T06:21:39.372741+0000","last_active":"2024-03-04T06:21:39.361681+0000","last_peered":"2024-03-04T06:21:39.361681+0000","last_clean":"2024-03-04T06:21:39.361681+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:39.361681+0000","last_undegraded":"2024-03-04T06:21:39.361681+0000","last_fullsized":"2024-03-04T06:21:39.361681+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781905,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":220,"num_read_kb":164051,"num_write":251,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"826'971","reported_seq":2980,"reported_epoch":829,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:40.371180+0000","last_change":"2024-03-04T06:21:40.371180+0000","last_active":"2024-03-04T06:21:40.371180+0000","last_peered":"2024-03-04T06:21:40.371180+0000","last_clean":"2024-03-04T06:21:40.371180+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:21:40.371180+0000","last_undegraded":"2024-03-04T06:21:40.371180+0000","last_fullsized":"2024-03-04T06:21:40.371180+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":971,"log_dups_size":0,"ondisk_log_size":971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7312134,"num_objects":84,"num_object_clones":69,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":479,"num_read_kb":379933,"num_write":370,"num_write_kb":100357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"826'840","reported_seq":2808,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-04T06:21:38.324764+0000","last_change":"2024-03-04T06:21:37.605290+0000","last_active":"2024-03-04T06:21:38.324764+0000","last_peered":"2024-03-04T06:21:38.324764+0000","last_clean":"2024-03-04T06:21:38.324764+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:21:38.324764+0000","last_undegraded":"2024-03-04T06:21:38.324764+0000","last_fullsized":"2024-03-04T06:21:38.324764+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":840,"log_dups_size":0,"ondisk_log_size":840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7564206,"num_objects":58,"num_object_clones":48,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":293,"num_read_kb":223019,"num_write":278,"num_write_kb":77984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"827'337","reported_seq":2620,"reported_epoch":829,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:40.387973+0000","last_change":"2024-03-04T06:21:40.387973+0000","last_active":"2024-03-04T06:21:40.387973+0000","last_peered":"2024-03-04T06:21:40.387973+0000","last_clean":"2024-03-04T06:21:40.387973+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:21:40.387973+0000","last_undegraded":"2024-03-04T06:21:40.387973+0000","last_fullsized":"2024-03-04T06:21:40.387973+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":94,"num_read_kb":65487,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"821'878","reported_seq":2884,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-04T06:21:38.330104+0000","last_change":"2024-03-04T06:21:37.514356+0000","last_active":"2024-03-04T06:21:38.330104+0000","last_peered":"2024-03-04T06:21:38.330104+0000","last_clean":"2024-03-04T06:21:38.330104+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:21:38.330104+0000","last_undegraded":"2024-03-04T06:21:38.330104+0000","last_fullsized":"2024-03-04T06:21:38.330104+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":64,"num_object_clones":55,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":393,"num_read_kb":322263,"num_write":310,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2427,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-04T06:21:38.328807+0000","last_change":"2024-03-04T06:21:37.516723+0000","last_active":"2024-03-04T06:21:38.328807+0000","last_peered":"2024-03-04T06:21:38.328807+0000","last_clean":"2024-03-04T06:21:38.328807+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:21:38.328807+0000","last_undegraded":"2024-03-04T06:21:38.328807+0000","last_fullsized":"2024-03-04T06:21:38.328807+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"826'536","reported_seq":2709,"reported_epoch":829,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:40.370354+0000","last_change":"2024-03-04T06:21:40.370354+0000","last_active":"2024-03-04T06:21:40.370354+0000","last_peered":"2024-03-04T06:21:40.370354+0000","last_clean":"2024-03-04T06:21:40.370354+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:21:40.370354+0000","last_undegraded":"2024-03-04T06:21:40.370354+0000","last_fullsized":"2024-03-04T06:21:40.370354+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487799,"num_objects":38,"num_object_clones":29,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":264,"num_read_kb":243518,"num_write":168,"num_write_kb":45031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2224,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-03-04T06:21:40.376355+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:21:40.376355+0000","last_peered":"2024-03-04T06:21:40.376355+0000","last_clean":"2024-03-04T06:21:40.376355+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:21:40.376355+0000","last_undegraded":"2024-03-04T06:21:40.376355+0000","last_fullsized":"2024-03-04T06:21:40.376355+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2051,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-03-04T06:21:40.381372+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:21:40.381372+0000","last_peered":"2024-03-04T06:21:40.381372+0000","last_clean":"2024-03-04T06:21:40.381372+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:21:40.381372+0000","last_undegraded":"2024-03-04T06:21:40.381372+0000","last_fullsized":"2024-03-04T06:21:40.381372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"826'866","reported_seq":2890,"reported_epoch":829,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:40.379866+0000","last_change":"2024-03-04T06:21:40.379866+0000","last_active":"2024-03-04T06:21:40.379866+0000","last_peered":"2024-03-04T06:21:40.379866+0000","last_clean":"2024-03-04T06:21:40.379866+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:21:40.379866+0000","last_undegraded":"2024-03-04T06:21:40.379866+0000","last_fullsized":"2024-03-04T06:21:40.379866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":866,"log_dups_size":0,"ondisk_log_size":866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4908370,"num_objects":70,"num_object_clones":56,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":383,"num_read_kb":303688,"num_write":311,"num_write_kb":83094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2041,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-04T06:21:38.330603+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:21:38.330603+0000","last_peered":"2024-03-04T06:21:38.330603+0000","last_clean":"2024-03-04T06:21:38.330603+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:21:38.330603+0000","last_undegraded":"2024-03-04T06:21:38.330603+0000","last_fullsized":"2024-03-04T06:21:38.330603+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"825'541","reported_seq":2773,"reported_epoch":829,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:40.380558+0000","last_change":"2024-03-04T06:21:40.380558+0000","last_active":"2024-03-04T06:21:40.380558+0000","last_peered":"2024-03-04T06:21:40.380558+0000","last_clean":"2024-03-04T06:21:40.380558+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:21:40.380558+0000","last_undegraded":"2024-03-04T06:21:40.380558+0000","last_fullsized":"2024-03-04T06:21:40.380558+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093214,"num_objects":45,"num_object_clones":36,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":284,"num_read_kb":234230,"num_write":185,"num_write_kb":52299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2051,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-03-04T06:21:40.368510+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:21:40.368510+0000","last_peered":"2024-03-04T06:21:40.368510+0000","last_clean":"2024-03-04T06:21:40.368510+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:21:40.368510+0000","last_undegraded":"2024-03-04T06:21:40.368510+0000","last_fullsized":"2024-03-04T06:21:40.368510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"825'1474","reported_seq":3310,"reported_epoch":829,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:40.387054+0000","last_change":"2024-03-04T06:21:40.387054+0000","last_active":"2024-03-04T06:21:40.387054+0000","last_peered":"2024-03-04T06:21:40.387054+0000","last_clean":"2024-03-04T06:21:40.387054+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:21:40.387054+0000","last_undegraded":"2024-03-04T06:21:40.387054+0000","last_fullsized":"2024-03-04T06:21:40.387054+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1474,"log_dups_size":0,"ondisk_log_size":1474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6353739,"num_objects":122,"num_object_clones":98,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":728,"num_read_kb":588855,"num_write":559,"num_write_kb":149555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2041,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-04T06:21:38.322897+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:21:38.322897+0000","last_peered":"2024-03-04T06:21:38.322897+0000","last_clean":"2024-03-04T06:21:38.322897+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:21:38.322897+0000","last_undegraded":"2024-03-04T06:21:38.322897+0000","last_fullsized":"2024-03-04T06:21:38.322897+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"826'1742","reported_seq":3338,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-04T06:21:39.357959+0000","last_change":"2024-03-04T06:21:39.373306+0000","last_active":"2024-03-04T06:21:39.357959+0000","last_peered":"2024-03-04T06:21:39.357959+0000","last_clean":"2024-03-04T06:21:39.357959+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:39.357959+0000","last_undegraded":"2024-03-04T06:21:39.357959+0000","last_fullsized":"2024-03-04T06:21:39.357959+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1742,"log_dups_size":0,"ondisk_log_size":1742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17178833,"num_objects":135,"num_object_clones":111,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":621,"num_read_kb":459268,"num_write":623,"num_write_kb":167764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"829'811","reported_seq":2889,"reported_epoch":829,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:40.380086+0000","last_change":"2024-03-04T06:21:40.379935+0000","last_active":"2024-03-04T06:21:40.380086+0000","last_peered":"2024-03-04T06:21:40.380086+0000","last_clean":"2024-03-04T06:21:40.380086+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:21:40.380086+0000","last_undegraded":"2024-03-04T06:21:40.380086+0000","last_fullsized":"2024-03-04T06:21:40.380086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":811,"log_dups_size":0,"ondisk_log_size":811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2286115,"num_objects":62,"num_object_clones":50,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":412,"num_read_kb":341589,"num_write":293,"num_write_kb":79015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2122,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-04T06:21:39.374253+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:39.374253+0000","last_peered":"2024-03-04T06:21:39.374253+0000","last_clean":"2024-03-04T06:21:39.374253+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:39.374253+0000","last_undegraded":"2024-03-04T06:21:39.374253+0000","last_fullsized":"2024-03-04T06:21:39.374253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"826'890","reported_seq":2944,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-04T06:21:39.383012+0000","last_change":"2024-03-04T06:21:39.387203+0000","last_active":"2024-03-04T06:21:39.383012+0000","last_peered":"2024-03-04T06:21:39.383012+0000","last_clean":"2024-03-04T06:21:39.383012+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:39.383012+0000","last_undegraded":"2024-03-04T06:21:39.383012+0000","last_fullsized":"2024-03-04T06:21:39.383012+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8471751,"num_objects":74,"num_object_clones":58,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":346,"num_read_kb":246344,"num_write":319,"num_write_kb":85920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2091,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-03-04T06:21:40.387450+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:21:40.387450+0000","last_peered":"2024-03-04T06:21:40.387450+0000","last_clean":"2024-03-04T06:21:40.387450+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:21:40.387450+0000","last_undegraded":"2024-03-04T06:21:40.387450+0000","last_fullsized":"2024-03-04T06:21:40.387450+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"829'1216","reported_seq":3063,"reported_epoch":829,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:40.529969+0000","last_change":"2024-03-04T06:21:40.378209+0000","last_active":"2024-03-04T06:21:40.529969+0000","last_peered":"2024-03-04T06:21:40.529969+0000","last_clean":"2024-03-04T06:21:40.529969+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:21:40.529969+0000","last_undegraded":"2024-03-04T06:21:40.529969+0000","last_fullsized":"2024-03-04T06:21:40.529969+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1216,"log_dups_size":0,"ondisk_log_size":1216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10169258,"num_objects":91,"num_object_clones":77,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":464,"num_read_kb":354559,"num_write":428,"num_write_kb":120045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2122,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-04T06:21:39.362445+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:39.362445+0000","last_peered":"2024-03-04T06:21:39.362445+0000","last_clean":"2024-03-04T06:21:39.362445+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:39.362445+0000","last_undegraded":"2024-03-04T06:21:39.362445+0000","last_fullsized":"2024-03-04T06:21:39.362445+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"826'1039","reported_seq":3090,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-04T06:21:39.367861+0000","last_change":"2024-03-04T06:21:39.372565+0000","last_active":"2024-03-04T06:21:39.367861+0000","last_peered":"2024-03-04T06:21:39.367861+0000","last_clean":"2024-03-04T06:21:39.367861+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:39.367861+0000","last_undegraded":"2024-03-04T06:21:39.367861+0000","last_fullsized":"2024-03-04T06:21:39.367861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1039,"log_dups_size":0,"ondisk_log_size":1039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3748291,"num_objects":85,"num_object_clones":74,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":515,"num_read_kb":432387,"num_write":386,"num_write_kb":105737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2051,"reported_epoch":829,"state":"active+clean","last_fresh":"2024-03-04T06:21:40.381353+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:21:40.381353+0000","last_peered":"2024-03-04T06:21:40.381353+0000","last_clean":"2024-03-04T06:21:40.381353+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:21:40.381353+0000","last_undegraded":"2024-03-04T06:21:40.381353+0000","last_fullsized":"2024-03-04T06:21:40.381353+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":87121538,"num_objects":1083,"num_object_clones":883,"num_object_copies":3249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":5812,"num_read_kb":4611159,"num_write":4884,"num_write_kb":1330473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13650,"ondisk_log_size":13650,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706499,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":209128,"kb_used_data":146208,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162712,"statfs":{"total":96636764160,"available":96422617088,"internally_reserved":0,"allocated":149716992,"data_stored":177090199,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706499,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":152520,"kb_used_data":99328,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94219320,"statfs":{"total":96636764160,"available":96480583680,"internally_reserved":0,"allocated":101711872,"data_stored":115844513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706498,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":148240,"kb_used_data":95048,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94223600,"statfs":{"total":96636764160,"available":96484966400,"internally_reserved":0,"allocated":97329152,"data_stored":116674243,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706498,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":163476,"kb_used_data":110284,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94208364,"statfs":{"total":96636764160,"available":96469364736,"internally_reserved":0,"allocated":112930816,"data_stored":133576537,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:21:43.382 INFO:tasks.rados.rados.0.smithi064.stdout:3253: done (3 left) 2024-03-04T06:21:43.382 INFO:tasks.rados.rados.0.smithi064.stdout:3254: done (2 left) 2024-03-04T06:21:43.382 INFO:tasks.rados.rados.0.smithi064.stdout:3255: done (1 left) 2024-03-04T06:21:43.382 INFO:tasks.rados.rados.0.smithi064.stdout:3256: done (0 left) 2024-03-04T06:21:43.382 INFO:tasks.rados.rados.0.smithi064.stdout:3257: write oid 32 current snap is 414 2024-03-04T06:21:43.382 INFO:tasks.rados.rados.0.smithi064.stdout:3257: seq_num 1476 ranges {407132=440006,1288481=693836,2616455=190677} 2024-03-04T06:21:43.387 INFO:tasks.rados.rados.0.smithi064.stdout:3257: writing smithi06431743-32 from 407132 to 847138 tid 1 2024-03-04T06:21:43.394 INFO:tasks.rados.rados.0.smithi064.stdout:3257: writing smithi06431743-32 from 1288481 to 1982317 tid 2 2024-03-04T06:21:43.399 INFO:tasks.rados.rados.0.smithi064.stdout:3257: writing smithi06431743-32 from 2616455 to 2807132 tid 3 2024-03-04T06:21:43.400 INFO:tasks.rados.rados.0.smithi064.stdout:3258: write oid 46 current snap is 414 2024-03-04T06:21:43.400 INFO:tasks.rados.rados.0.smithi064.stdout:3258: seq_num 1477 ranges {423939=515255,1346963=553847,2643878=180061} 2024-03-04T06:21:43.405 INFO:tasks.rados.rados.0.smithi064.stdout:3258: writing smithi06431743-46 from 423939 to 939194 tid 1 2024-03-04T06:21:43.410 INFO:tasks.rados.rados.0.smithi064.stdout:3258: writing smithi06431743-46 from 1346963 to 1900810 tid 2 2024-03-04T06:21:43.416 INFO:tasks.rados.rados.0.smithi064.stdout:3258: writing smithi06431743-46 from 2643878 to 2823939 tid 3 2024-03-04T06:21:43.416 INFO:tasks.rados.rados.0.smithi064.stdout:3259: snap_remove snap 407 2024-03-04T06:21:44.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:44.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:44.365+0000 7fedb879d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:44.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:44.370+0000 7fedb879d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:44.385 INFO:tasks.rados.rados.0.smithi064.stdout:3259: done (2 left) 2024-03-04T06:21:44.385 INFO:tasks.rados.rados.0.smithi064.stdout:3260: read oid 5 snap -1 2024-03-04T06:21:44.385 INFO:tasks.rados.rados.0.smithi064.stdout:3260: expect (ObjNum 1455 snap 406 seq_num 1455) 2024-03-04T06:21:44.385 INFO:tasks.rados.rados.0.smithi064.stdout:3261: read oid 7 snap -1 2024-03-04T06:21:44.385 INFO:tasks.rados.rados.0.smithi064.stdout:3261: expect (ObjNum 1353 snap 377 seq_num 1353) 2024-03-04T06:21:44.385 INFO:tasks.rados.rados.0.smithi064.stdout:3262: snap_remove snap 397 2024-03-04T06:21:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:3262: done (4 left) 2024-03-04T06:21:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:3263: read oid 25 snap 398 2024-03-04T06:21:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:3263: expect (ObjNum 1278 snap 346 seq_num 1278) 2024-03-04T06:21:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:3264: delete oid 26 current snap is 414 2024-03-04T06:21:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:3257: finishing write tid 1 to smithi06431743-32 2024-03-04T06:21:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:3257: finishing write tid 2 to smithi06431743-32 2024-03-04T06:21:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:3257: finishing write tid 3 to smithi06431743-32 2024-03-04T06:21:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:3257: finishing write tid 4 to smithi06431743-32 2024-03-04T06:21:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:3257: finishing write tid 5 to smithi06431743-32 2024-03-04T06:21:45.389 INFO:tasks.rados.rados.0.smithi064.stdout:3257: oid 32 updating version 0 to 1488 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3257: oid 32 updating version 1488 to 1491 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3257: oid 32 version 1491 is already newer than 1490 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3257: oid 32 version 1491 is already newer than 1489 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1491 (ObjNum 1476 snap 414 seq_num 1476) dirty exists 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3257: left oid 32 (ObjNum 1476 snap 414 seq_num 1476) 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3258: finishing write tid 1 to smithi06431743-46 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3258: finishing write tid 2 to smithi06431743-46 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3258: finishing write tid 3 to smithi06431743-46 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3258: finishing write tid 4 to smithi06431743-46 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3258: finishing write tid 5 to smithi06431743-46 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3258: oid 46 updating version 0 to 1495 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3258: oid 46 version 1495 is already newer than 1495 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3258: oid 46 version 1495 is already newer than 1495 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3258: oid 46 version 1495 is already newer than 1495 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 1495 (ObjNum 1477 snap 414 seq_num 1477) dirty exists 2024-03-04T06:21:45.390 INFO:tasks.rados.rados.0.smithi064.stdout:3258: left oid 46 (ObjNum 1477 snap 414 seq_num 1477) 2024-03-04T06:21:45.391 INFO:tasks.rados.rados.0.smithi064.stdout:3261: expect (ObjNum 1353 snap 377 seq_num 1353) 2024-03-04T06:21:45.616 INFO:tasks.rados.rados.0.smithi064.stdout:3260: expect (ObjNum 1455 snap 406 seq_num 1455) 2024-03-04T06:21:45.740 INFO:tasks.rados.rados.0.smithi064.stdout:3263: expect (ObjNum 1278 snap 346 seq_num 1278) 2024-03-04T06:21:45.871 INFO:tasks.rados.rados.0.smithi064.stdout:3257: done (5 left) 2024-03-04T06:21:45.872 INFO:tasks.rados.rados.0.smithi064.stdout:3258: done (4 left) 2024-03-04T06:21:45.872 INFO:tasks.rados.rados.0.smithi064.stdout:3260: done (3 left) 2024-03-04T06:21:45.872 INFO:tasks.rados.rados.0.smithi064.stdout:3261: done (2 left) 2024-03-04T06:21:45.872 INFO:tasks.rados.rados.0.smithi064.stdout:3263: done (1 left) 2024-03-04T06:21:45.872 INFO:tasks.rados.rados.0.smithi064.stdout:3264: done (0 left) 2024-03-04T06:21:45.872 INFO:tasks.rados.rados.0.smithi064.stdout:3265: snap_remove snap 413 2024-03-04T06:21:46.396 INFO:tasks.rados.rados.0.smithi064.stdout:3265: done (0 left) 2024-03-04T06:21:46.396 INFO:tasks.rados.rados.0.smithi064.stdout:3266: snap_remove snap 409 2024-03-04T06:21:46.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:46.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:46.597+0000 7f2e71813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:46.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:46.601+0000 7f2e71813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:47.399 INFO:tasks.rados.rados.0.smithi064.stdout:3266: done (0 left) 2024-03-04T06:21:47.399 INFO:tasks.rados.rados.0.smithi064.stdout:3267: write oid 17 current snap is 414 2024-03-04T06:21:47.399 INFO:tasks.rados.rados.0.smithi064.stdout:3267: seq_num 1479 ranges {457553=665753,1747574=557516} 2024-03-04T06:21:47.405 INFO:tasks.rados.rados.0.smithi064.stdout:3267: writing smithi06431743-17 from 457553 to 1123306 tid 1 2024-03-04T06:21:47.412 INFO:tasks.rados.rados.0.smithi064.stdout:3267: writing smithi06431743-17 from 1747574 to 2305090 tid 2 2024-03-04T06:21:47.412 INFO:tasks.rados.rados.0.smithi064.stdout:3268: read oid 27 snap 410 2024-03-04T06:21:47.413 INFO:tasks.rados.rados.0.smithi064.stdout:3268: expect (ObjNum 1417 snap 402 seq_num 1417) 2024-03-04T06:21:47.413 INFO:tasks.rados.rados.0.smithi064.stdout:3269: write (excl) oid 34 current snap is 414 2024-03-04T06:21:47.413 INFO:tasks.rados.rados.0.smithi064.stdout:3269: seq_num 1480 ranges {474360=741002,1806056=701174} 2024-03-04T06:21:47.421 INFO:tasks.rados.rados.0.smithi064.stdout:3269: writing smithi06431743-34 from 474360 to 1215362 tid 1 2024-03-04T06:21:47.430 INFO:tasks.rados.rados.0.smithi064.stdout:3269: writing smithi06431743-34 from 1806056 to 2507230 tid 2 2024-03-04T06:21:47.430 INFO:tasks.rados.rados.0.smithi064.stdout:3270: write oid 32 current snap is 414 2024-03-04T06:21:47.431 INFO:tasks.rados.rados.0.smithi064.stdout:3270: seq_num 1481 ranges {491167=416251,1464538=561185,2753570=137597} 2024-03-04T06:21:47.437 INFO:tasks.rados.rados.0.smithi064.stdout:3270: writing smithi06431743-32 from 491167 to 907418 tid 1 2024-03-04T06:21:47.444 INFO:tasks.rados.rados.0.smithi064.stdout:3270: writing smithi06431743-32 from 1464538 to 2025723 tid 2 2024-03-04T06:21:47.449 INFO:tasks.rados.rados.0.smithi064.stdout:3270: writing smithi06431743-32 from 2753570 to 2891167 tid 3 2024-03-04T06:21:47.450 INFO:tasks.rados.rados.0.smithi064.stdout:3271: read oid 3 snap 388 2024-03-04T06:21:47.450 INFO:tasks.rados.rados.0.smithi064.stdout:3271: expect (ObjNum 1331 snap 368 seq_num 1331) 2024-03-04T06:21:47.450 INFO:tasks.rados.rados.0.smithi064.stdout:3272: write oid 48 current snap is 414 2024-03-04T06:21:47.450 INFO:tasks.rados.rados.0.smithi064.stdout:3272: seq_num 1482 ranges {507974=491500,1406667=704843,2548287=359687} 2024-03-04T06:21:47.456 INFO:tasks.rados.rados.0.smithi064.stdout:3272: writing smithi06431743-48 from 507974 to 999474 tid 1 2024-03-04T06:21:47.464 INFO:tasks.rados.rados.0.smithi064.stdout:3272: writing smithi06431743-48 from 1406667 to 2111510 tid 2 2024-03-04T06:21:47.470 INFO:tasks.rados.rados.0.smithi064.stdout:3272: writing smithi06431743-48 from 2548287 to 2907974 tid 3 2024-03-04T06:21:47.470 INFO:tasks.rados.rados.0.smithi064.stdout:3273: write oid 22 current snap is 414 2024-03-04T06:21:47.470 INFO:tasks.rados.rados.0.smithi064.stdout:3273: seq_num 1483 ranges {524781=683102,1981502=564854} 2024-03-04T06:21:47.478 INFO:tasks.rados.rados.0.smithi064.stdout:3273: writing smithi06431743-22 from 524781 to 1207883 tid 1 2024-03-04T06:21:47.488 INFO:tasks.rados.rados.0.smithi064.stdout:3273: writing smithi06431743-22 from 1981502 to 2546356 tid 2 2024-03-04T06:21:47.488 INFO:tasks.rados.rados.0.smithi064.stdout:3274: read oid 49 snap -1 2024-03-04T06:21:47.488 INFO:tasks.rados.rados.0.smithi064.stdout:3274: expect deleted 2024-03-04T06:21:47.488 INFO:tasks.rados.rados.0.smithi064.stdout:3275: read oid 8 snap -1 2024-03-04T06:21:47.488 INFO:tasks.rados.rados.0.smithi064.stdout:3275: expect (ObjNum 1360 snap 378 seq_num 1360) 2024-03-04T06:21:47.488 INFO:tasks.rados.rados.0.smithi064.stdout:3276: read oid 50 snap -1 2024-03-04T06:21:47.488 INFO:tasks.rados.rados.0.smithi064.stdout:3276: expect (ObjNum 1474 snap 413 seq_num 1474) 2024-03-04T06:21:47.488 INFO:tasks.rados.rados.0.smithi064.stdout:3277: delete oid 7 current snap is 414 2024-03-04T06:21:47.509 INFO:tasks.rados.rados.0.smithi064.stdout:3271: expect (ObjNum 1331 snap 368 seq_num 1331) 2024-03-04T06:21:47.635 INFO:tasks.rados.rados.0.smithi064.stdout:3268: expect (ObjNum 1417 snap 402 seq_num 1417) 2024-03-04T06:21:47.691 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:21:47.692 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:21:47.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:21:47.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:47.796+0000 7efee8b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:47.803 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:47.802+0000 7efee8b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:47.843 INFO:tasks.rados.rados.0.smithi064.stdout:3275: expect (ObjNum 1360 snap 378 seq_num 1360) 2024-03-04T06:21:48.010 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:21:48.010 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:21:48.022 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1248,"stamp":"2024-03-04T06:21:47.396784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":86905236,"num_objects":1089,"num_object_clones":887,"num_object_copies":3267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":5950,"num_read_kb":4646957,"num_write":5040,"num_write_kb":1337125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13932,"ondisk_log_size":13932,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":685644,"kb_used_data":463148,"kb_used_omap":0,"kb_used_meta":222464,"kb_avail":376801716,"statfs":{"total":386547056640,"available":385844957184,"internally_reserved":0,"allocated":474263552,"data_stored":562851200,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":227803136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-806670,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":35707,"num_write":19,"num_write_kb":4758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018690"},"pg_stats":[{"pgid":"3.f","version":"829'780","reported_seq":2842,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-04T06:21:43.398314+0000","last_change":"2024-03-04T06:21:42.431567+0000","last_active":"2024-03-04T06:21:43.398314+0000","last_peered":"2024-03-04T06:21:43.398314+0000","last_clean":"2024-03-04T06:21:43.398314+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:21:43.398314+0000","last_undegraded":"2024-03-04T06:21:43.398314+0000","last_fullsized":"2024-03-04T06:21:43.398314+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":780,"log_dups_size":0,"ondisk_log_size":780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5059633,"num_objects":69,"num_object_clones":55,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":316,"num_read_kb":251968,"num_write":288,"num_write_kb":75809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"833'741","reported_seq":2849,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-03-04T06:21:44.398977+0000","last_change":"2024-03-04T06:21:44.461513+0000","last_active":"2024-03-04T06:21:44.398977+0000","last_peered":"2024-03-04T06:21:44.398977+0000","last_clean":"2024-03-04T06:21:44.398977+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:44.398977+0000","last_undegraded":"2024-03-04T06:21:44.398977+0000","last_fullsized":"2024-03-04T06:21:44.398977+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":741,"log_dups_size":0,"ondisk_log_size":741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781905,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":220,"num_read_kb":164051,"num_write":251,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"834'989","reported_seq":3001,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.508114+0000","last_change":"2024-03-04T06:21:45.534016+0000","last_active":"2024-03-04T06:21:45.508114+0000","last_peered":"2024-03-04T06:21:45.508114+0000","last_clean":"2024-03-04T06:21:45.508114+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:21:45.508114+0000","last_undegraded":"2024-03-04T06:21:45.508114+0000","last_fullsized":"2024-03-04T06:21:45.508114+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":989,"log_dups_size":0,"ondisk_log_size":989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7312134,"num_objects":84,"num_object_clones":69,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":485,"num_read_kb":390127,"num_write":370,"num_write_kb":100357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"831'850","reported_seq":2828,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-04T06:21:43.387501+0000","last_change":"2024-03-04T06:21:42.486846+0000","last_active":"2024-03-04T06:21:43.387501+0000","last_peered":"2024-03-04T06:21:43.387501+0000","last_clean":"2024-03-04T06:21:43.387501+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:21:43.387501+0000","last_undegraded":"2024-03-04T06:21:43.387501+0000","last_fullsized":"2024-03-04T06:21:43.387501+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":850,"log_dups_size":0,"ondisk_log_size":850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563810,"num_objects":59,"num_object_clones":49,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":299,"num_read_kb":228881,"num_write":279,"num_write_kb":77984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"834'345","reported_seq":2642,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.471899+0000","last_change":"2024-03-04T06:21:45.471899+0000","last_active":"2024-03-04T06:21:45.471899+0000","last_peered":"2024-03-04T06:21:45.471899+0000","last_clean":"2024-03-04T06:21:45.471899+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:21:45.471899+0000","last_undegraded":"2024-03-04T06:21:45.471899+0000","last_fullsized":"2024-03-04T06:21:45.471899+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":100,"num_read_kb":71985,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"831'886","reported_seq":2900,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-04T06:21:43.395825+0000","last_change":"2024-03-04T06:21:42.487793+0000","last_active":"2024-03-04T06:21:43.395825+0000","last_peered":"2024-03-04T06:21:43.395825+0000","last_clean":"2024-03-04T06:21:43.395825+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:21:43.395825+0000","last_undegraded":"2024-03-04T06:21:43.395825+0000","last_fullsized":"2024-03-04T06:21:43.395825+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":886,"log_dups_size":0,"ondisk_log_size":886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":64,"num_object_clones":55,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":393,"num_read_kb":322263,"num_write":310,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2443,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-04T06:21:43.397392+0000","last_change":"2024-03-04T06:21:42.428194+0000","last_active":"2024-03-04T06:21:43.397392+0000","last_peered":"2024-03-04T06:21:43.397392+0000","last_clean":"2024-03-04T06:21:43.397392+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:21:43.397392+0000","last_undegraded":"2024-03-04T06:21:43.397392+0000","last_fullsized":"2024-03-04T06:21:43.397392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"834'546","reported_seq":2727,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.404974+0000","last_change":"2024-03-04T06:21:45.492543+0000","last_active":"2024-03-04T06:21:45.404974+0000","last_peered":"2024-03-04T06:21:45.404974+0000","last_clean":"2024-03-04T06:21:45.404974+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:21:45.404974+0000","last_undegraded":"2024-03-04T06:21:45.404974+0000","last_fullsized":"2024-03-04T06:21:45.404974+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487799,"num_objects":38,"num_object_clones":29,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":264,"num_read_kb":243518,"num_write":168,"num_write_kb":45031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2237,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.404618+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:21:45.404618+0000","last_peered":"2024-03-04T06:21:45.404618+0000","last_clean":"2024-03-04T06:21:45.404618+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:21:45.404618+0000","last_undegraded":"2024-03-04T06:21:45.404618+0000","last_fullsized":"2024-03-04T06:21:45.404618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2064,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.418174+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:21:45.418174+0000","last_peered":"2024-03-04T06:21:45.418174+0000","last_clean":"2024-03-04T06:21:45.418174+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:21:45.418174+0000","last_undegraded":"2024-03-04T06:21:45.418174+0000","last_fullsized":"2024-03-04T06:21:45.418174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"834'884","reported_seq":2908,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.412452+0000","last_change":"2024-03-04T06:21:45.492512+0000","last_active":"2024-03-04T06:21:45.412452+0000","last_peered":"2024-03-04T06:21:45.412452+0000","last_clean":"2024-03-04T06:21:45.412452+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:21:45.412452+0000","last_undegraded":"2024-03-04T06:21:45.412452+0000","last_fullsized":"2024-03-04T06:21:45.412452+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":884,"log_dups_size":0,"ondisk_log_size":884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4908370,"num_objects":70,"num_object_clones":56,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":383,"num_read_kb":303688,"num_write":311,"num_write_kb":83094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2054,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-04T06:21:43.402161+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:21:43.402161+0000","last_peered":"2024-03-04T06:21:43.402161+0000","last_clean":"2024-03-04T06:21:43.402161+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:21:43.402161+0000","last_undegraded":"2024-03-04T06:21:43.402161+0000","last_fullsized":"2024-03-04T06:21:43.402161+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"834'547","reported_seq":2795,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.473425+0000","last_change":"2024-03-04T06:21:45.473425+0000","last_active":"2024-03-04T06:21:45.473425+0000","last_peered":"2024-03-04T06:21:45.473425+0000","last_clean":"2024-03-04T06:21:45.473425+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:21:45.473425+0000","last_undegraded":"2024-03-04T06:21:45.473425+0000","last_fullsized":"2024-03-04T06:21:45.473425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5093214,"num_objects":45,"num_object_clones":36,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":290,"num_read_kb":241122,"num_write":185,"num_write_kb":52299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2064,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.398836+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:21:45.398836+0000","last_peered":"2024-03-04T06:21:45.398836+0000","last_clean":"2024-03-04T06:21:45.398836+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:21:45.398836+0000","last_undegraded":"2024-03-04T06:21:45.398836+0000","last_fullsized":"2024-03-04T06:21:45.398836+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"834'1503","reported_seq":3339,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.449714+0000","last_change":"2024-03-04T06:21:45.522051+0000","last_active":"2024-03-04T06:21:45.449714+0000","last_peered":"2024-03-04T06:21:45.449714+0000","last_clean":"2024-03-04T06:21:45.449714+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:21:45.449714+0000","last_undegraded":"2024-03-04T06:21:45.449714+0000","last_fullsized":"2024-03-04T06:21:45.449714+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1503,"log_dups_size":0,"ondisk_log_size":1503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8018028,"num_objects":123,"num_object_clones":99,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":730,"num_read_kb":588857,"num_write":569,"num_write_kb":152071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2054,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-04T06:21:43.390157+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:21:43.390157+0000","last_peered":"2024-03-04T06:21:43.390157+0000","last_clean":"2024-03-04T06:21:43.390157+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:21:43.390157+0000","last_undegraded":"2024-03-04T06:21:43.390157+0000","last_fullsized":"2024-03-04T06:21:43.390157+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"831'1760","reported_seq":3357,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-03-04T06:21:44.452216+0000","last_change":"2024-03-04T06:21:44.413347+0000","last_active":"2024-03-04T06:21:44.452216+0000","last_peered":"2024-03-04T06:21:44.452216+0000","last_clean":"2024-03-04T06:21:44.452216+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:44.452216+0000","last_undegraded":"2024-03-04T06:21:44.452216+0000","last_fullsized":"2024-03-04T06:21:44.452216+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1760,"log_dups_size":0,"ondisk_log_size":1760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17178833,"num_objects":135,"num_object_clones":111,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":627,"num_read_kb":465525,"num_write":623,"num_write_kb":167764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"834'825","reported_seq":2911,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.421115+0000","last_change":"2024-03-04T06:21:45.496503+0000","last_active":"2024-03-04T06:21:45.421115+0000","last_peered":"2024-03-04T06:21:45.421115+0000","last_clean":"2024-03-04T06:21:45.421115+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:21:45.421115+0000","last_undegraded":"2024-03-04T06:21:45.421115+0000","last_fullsized":"2024-03-04T06:21:45.421115+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2790325,"num_objects":63,"num_object_clones":51,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":413,"num_read_kb":341590,"num_write":297,"num_write_kb":80300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2135,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-03-04T06:21:44.423340+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:44.423340+0000","last_peered":"2024-03-04T06:21:44.423340+0000","last_clean":"2024-03-04T06:21:44.423340+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:44.423340+0000","last_undegraded":"2024-03-04T06:21:44.423340+0000","last_fullsized":"2024-03-04T06:21:44.423340+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"831'898","reported_seq":2960,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-03-04T06:21:44.419038+0000","last_change":"2024-03-04T06:21:44.428456+0000","last_active":"2024-03-04T06:21:44.419038+0000","last_peered":"2024-03-04T06:21:44.419038+0000","last_clean":"2024-03-04T06:21:44.419038+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:44.419038+0000","last_undegraded":"2024-03-04T06:21:44.419038+0000","last_fullsized":"2024-03-04T06:21:44.419038+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":898,"log_dups_size":0,"ondisk_log_size":898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8471751,"num_objects":74,"num_object_clones":58,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":346,"num_read_kb":246344,"num_write":319,"num_write_kb":85920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2104,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.416379+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:21:45.416379+0000","last_peered":"2024-03-04T06:21:45.416379+0000","last_clean":"2024-03-04T06:21:45.416379+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:21:45.416379+0000","last_undegraded":"2024-03-04T06:21:45.416379+0000","last_fullsized":"2024-03-04T06:21:45.416379+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"834'1232","reported_seq":3081,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.399767+0000","last_change":"2024-03-04T06:21:45.519481+0000","last_active":"2024-03-04T06:21:45.399767+0000","last_peered":"2024-03-04T06:21:45.399767+0000","last_clean":"2024-03-04T06:21:45.399767+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:21:45.399767+0000","last_undegraded":"2024-03-04T06:21:45.399767+0000","last_fullsized":"2024-03-04T06:21:45.399767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1232,"log_dups_size":0,"ondisk_log_size":1232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10169258,"num_objects":91,"num_object_clones":77,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":464,"num_read_kb":354559,"num_write":428,"num_write_kb":120045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2135,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-03-04T06:21:44.399784+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:44.399784+0000","last_peered":"2024-03-04T06:21:44.399784+0000","last_clean":"2024-03-04T06:21:44.399784+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:44.399784+0000","last_undegraded":"2024-03-04T06:21:44.399784+0000","last_fullsized":"2024-03-04T06:21:44.399784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"833'1049","reported_seq":3110,"reported_epoch":833,"state":"active+clean","last_fresh":"2024-03-04T06:21:44.397953+0000","last_change":"2024-03-04T06:21:44.454702+0000","last_active":"2024-03-04T06:21:44.397953+0000","last_peered":"2024-03-04T06:21:44.397953+0000","last_clean":"2024-03-04T06:21:44.397953+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:44.397953+0000","last_undegraded":"2024-03-04T06:21:44.397953+0000","last_fullsized":"2024-03-04T06:21:44.397953+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2773518,"num_objects":86,"num_object_clones":75,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":517,"num_read_kb":432388,"num_write":390,"num_write_kb":106694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2064,"reported_epoch":834,"state":"active+clean","last_fresh":"2024-03-04T06:21:45.410134+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:21:45.410134+0000","last_peered":"2024-03-04T06:21:45.410134+0000","last_clean":"2024-03-04T06:21:45.410134+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:21:45.410134+0000","last_undegraded":"2024-03-04T06:21:45.410134+0000","last_fullsized":"2024-03-04T06:21:45.410134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":86314868,"num_objects":1087,"num_object_clones":887,"num_object_copies":3261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":5847,"num_read_kb":4646866,"num_write":4903,"num_write_kb":1335231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13835,"ondisk_log_size":13835,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706504,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":213968,"kb_used_data":151048,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94157872,"statfs":{"total":96636764160,"available":96417660928,"internally_reserved":0,"allocated":154673152,"data_stored":184600511,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706504,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":157360,"kb_used_data":104168,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94214480,"statfs":{"total":96636764160,"available":96475627520,"internally_reserved":0,"allocated":106668032,"data_stored":123354825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706503,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153080,"kb_used_data":99888,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94218760,"statfs":{"total":96636764160,"available":96480010240,"internally_reserved":0,"allocated":102285312,"data_stored":124184555,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706503,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":161236,"kb_used_data":108044,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94210604,"statfs":{"total":96636764160,"available":96471658496,"internally_reserved":0,"allocated":110637056,"data_stored":130711309,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:21:48.051 INFO:tasks.rados.rados.0.smithi064.stdout:3267: finishing write tid 1 to smithi06431743-17 2024-03-04T06:21:48.051 INFO:tasks.rados.rados.0.smithi064.stdout:3272: finishing write tid 1 to smithi06431743-48 2024-03-04T06:21:48.051 INFO:tasks.rados.rados.0.smithi064.stdout:3269: finishing write tid 1 to smithi06431743-34 2024-03-04T06:21:48.051 INFO:tasks.rados.rados.0.smithi064.stdout:3273: finishing write tid 1 to smithi06431743-22 2024-03-04T06:21:48.051 INFO:tasks.rados.rados.0.smithi064.stdout:3267: finishing write tid 2 to smithi06431743-17 2024-03-04T06:21:48.051 INFO:tasks.rados.rados.0.smithi064.stdout:3269: finishing write tid 2 to smithi06431743-34 2024-03-04T06:21:48.051 INFO:tasks.rados.rados.0.smithi064.stdout:3272: finishing write tid 2 to smithi06431743-48 2024-03-04T06:21:48.051 INFO:tasks.rados.rados.0.smithi064.stdout:3268: done (10 left) 2024-03-04T06:21:48.052 INFO:tasks.rados.rados.0.smithi064.stdout:3271: done (9 left) 2024-03-04T06:21:48.052 INFO:tasks.rados.rados.0.smithi064.stdout:3275: done (8 left) 2024-03-04T06:21:48.052 INFO:tasks.rados.rados.0.smithi064.stdout:3277: done (7 left) 2024-03-04T06:21:48.052 INFO:tasks.rados.rados.0.smithi064.stdout:3278: snap_create 2024-03-04T06:21:48.052 INFO:tasks.rados.rados.0.smithi064.stdout:3267: finishing write tid 3 to smithi06431743-17 2024-03-04T06:21:48.052 INFO:tasks.rados.rados.0.smithi064.stdout:3267: finishing write tid 4 to smithi06431743-17 2024-03-04T06:21:48.052 INFO:tasks.rados.rados.0.smithi064.stdout:3267: oid 17 updating version 0 to 550 2024-03-04T06:21:48.052 INFO:tasks.rados.rados.0.smithi064.stdout:3267: oid 17 version 550 is already newer than 548 2024-03-04T06:21:48.053 INFO:tasks.rados.rados.0.smithi064.stdout:3267: oid 17 version 550 is already newer than 550 2024-03-04T06:21:48.053 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 17 v 550 (ObjNum 1479 snap 414 seq_num 1479) dirty exists 2024-03-04T06:21:48.053 INFO:tasks.rados.rados.0.smithi064.stdout:3267: left oid 17 (ObjNum 1479 snap 414 seq_num 1479) 2024-03-04T06:21:48.053 INFO:tasks.rados.rados.0.smithi064.stdout:3272: finishing write tid 3 to smithi06431743-48 2024-03-04T06:21:48.053 INFO:tasks.rados.rados.0.smithi064.stdout:3273: finishing write tid 2 to smithi06431743-22 2024-03-04T06:21:48.053 INFO:tasks.rados.rados.0.smithi064.stdout:3273: finishing write tid 3 to smithi06431743-22 2024-03-04T06:21:48.053 INFO:tasks.rados.rados.0.smithi064.stdout:3272: finishing write tid 4 to smithi06431743-48 2024-03-04T06:21:48.053 INFO:tasks.rados.rados.0.smithi064.stdout:3273: finishing write tid 4 to smithi06431743-22 2024-03-04T06:21:48.053 INFO:tasks.rados.rados.0.smithi064.stdout:3273: oid 22 updating version 0 to 1054 2024-03-04T06:21:48.053 INFO:tasks.rados.rados.0.smithi064.stdout:3273: oid 22 updating version 1054 to 1055 2024-03-04T06:21:48.054 INFO:tasks.rados.rados.0.smithi064.stdout:3273: oid 22 updating version 1055 to 1056 2024-03-04T06:21:48.054 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 1056 (ObjNum 1483 snap 414 seq_num 1483) dirty exists 2024-03-04T06:21:48.054 INFO:tasks.rados.rados.0.smithi064.stdout:3273: left oid 22 (ObjNum 1483 snap 414 seq_num 1483) 2024-03-04T06:21:48.054 INFO:tasks.rados.rados.0.smithi064.stdout:3269: finishing write tid 3 to smithi06431743-34 2024-03-04T06:21:48.054 INFO:tasks.rados.rados.0.smithi064.stdout:3272: finishing write tid 5 to smithi06431743-48 2024-03-04T06:21:48.054 INFO:tasks.rados.rados.0.smithi064.stdout:3272: oid 48 updating version 0 to 551 2024-03-04T06:21:48.054 INFO:tasks.rados.rados.0.smithi064.stdout:3272: oid 48 version 551 is already newer than 549 2024-03-04T06:21:48.054 INFO:tasks.rados.rados.0.smithi064.stdout:3272: oid 48 updating version 551 to 552 2024-03-04T06:21:48.054 INFO:tasks.rados.rados.0.smithi064.stdout:3272: oid 48 version 552 is already newer than 551 2024-03-04T06:21:48.055 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 552 (ObjNum 1482 snap 414 seq_num 1482) dirty exists 2024-03-04T06:21:48.055 INFO:tasks.rados.rados.0.smithi064.stdout:3272: left oid 48 (ObjNum 1482 snap 414 seq_num 1482) 2024-03-04T06:21:48.055 INFO:tasks.rados.rados.0.smithi064.stdout:3269: finishing write tid 4 to smithi06431743-34 2024-03-04T06:21:48.055 INFO:tasks.rados.rados.0.smithi064.stdout:3269: oid 34 updating version 0 to 1510 2024-03-04T06:21:48.055 INFO:tasks.rados.rados.0.smithi064.stdout:3269: oid 34 version 1510 is already newer than 1509 2024-03-04T06:21:48.055 INFO:tasks.rados.rados.0.smithi064.stdout:3269: oid 34 version 1510 is already newer than 1508 2024-03-04T06:21:48.055 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 1510 (ObjNum 1480 snap 414 seq_num 1480) dirty exists 2024-03-04T06:21:48.055 INFO:tasks.rados.rados.0.smithi064.stdout:3269: left oid 34 (ObjNum 1480 snap 414 seq_num 1480) 2024-03-04T06:21:48.055 INFO:tasks.rados.rados.0.smithi064.stdout:3276: expect (ObjNum 1474 snap 413 seq_num 1474) 2024-03-04T06:21:48.263 INFO:tasks.rados.rados.0.smithi064.stdout:3270: finishing write tid 1 to smithi06431743-32 2024-03-04T06:21:48.263 INFO:tasks.rados.rados.0.smithi064.stdout:3270: finishing write tid 2 to smithi06431743-32 2024-03-04T06:21:48.263 INFO:tasks.rados.rados.0.smithi064.stdout:3270: finishing write tid 3 to smithi06431743-32 2024-03-04T06:21:48.263 INFO:tasks.rados.rados.0.smithi064.stdout:3270: finishing write tid 4 to smithi06431743-32 2024-03-04T06:21:48.263 INFO:tasks.rados.rados.0.smithi064.stdout:3270: finishing write tid 5 to smithi06431743-32 2024-03-04T06:21:48.263 INFO:tasks.rados.rados.0.smithi064.stdout:3270: oid 32 updating version 0 to 1514 2024-03-04T06:21:48.263 INFO:tasks.rados.rados.0.smithi064.stdout:3270: oid 32 version 1514 is already newer than 1514 2024-03-04T06:21:48.263 INFO:tasks.rados.rados.0.smithi064.stdout:3270: oid 32 version 1514 is already newer than 1514 2024-03-04T06:21:48.263 INFO:tasks.rados.rados.0.smithi064.stdout:3270: oid 32 version 1514 is already newer than 1514 2024-03-04T06:21:48.264 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1514 (ObjNum 1481 snap 414 seq_num 1481) dirty exists 2024-03-04T06:21:48.264 INFO:tasks.rados.rados.0.smithi064.stdout:3270: left oid 32 (ObjNum 1481 snap 414 seq_num 1481) 2024-03-04T06:21:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:3267: done (7 left) 2024-03-04T06:21:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:3269: done (6 left) 2024-03-04T06:21:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:3270: done (5 left) 2024-03-04T06:21:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:3272: done (4 left) 2024-03-04T06:21:48.317 INFO:tasks.rados.rados.0.smithi064.stdout:3273: done (3 left) 2024-03-04T06:21:48.318 INFO:tasks.rados.rados.0.smithi064.stdout:3274: done (2 left) 2024-03-04T06:21:48.318 INFO:tasks.rados.rados.0.smithi064.stdout:3276: done (1 left) 2024-03-04T06:21:48.318 INFO:tasks.rados.rados.0.smithi064.stdout:3278: done (0 left) 2024-03-04T06:21:48.318 INFO:tasks.rados.rados.0.smithi064.stdout:3279: write oid 24 current snap is 415 2024-03-04T06:21:48.318 INFO:tasks.rados.rados.0.smithi064.stdout:3279: seq_num 1485 ranges {558395=433600,1698466=568523,2863262=95133} 2024-03-04T06:21:48.322 INFO:tasks.rados.rados.0.smithi064.stdout:3279: writing smithi06431743-24 from 558395 to 991995 tid 1 2024-03-04T06:21:48.329 INFO:tasks.rados.rados.0.smithi064.stdout:3279: writing smithi06431743-24 from 1698466 to 2266989 tid 2 2024-03-04T06:21:48.333 INFO:tasks.rados.rados.0.smithi064.stdout:3279: writing smithi06431743-24 from 2863262 to 2958395 tid 3 2024-03-04T06:21:48.333 INFO:tasks.rados.rados.0.smithi064.stdout:3280: write oid 50 current snap is 415 2024-03-04T06:21:48.333 INFO:tasks.rados.rados.0.smithi064.stdout:3280: seq_num 1486 ranges {575202=508849,1640595=712181} 2024-03-04T06:21:48.339 INFO:tasks.rados.rados.0.smithi064.stdout:3280: writing smithi06431743-50 from 575202 to 1084051 tid 1 2024-03-04T06:21:48.346 INFO:tasks.rados.rados.0.smithi064.stdout:3280: writing smithi06431743-50 from 1640595 to 2352776 tid 2 2024-03-04T06:21:48.346 INFO:tasks.rados.rados.0.smithi064.stdout:3281: read oid 18 snap 406 2024-03-04T06:21:48.346 INFO:tasks.rados.rados.0.smithi064.stdout:3281: expect deleted 2024-03-04T06:21:48.346 INFO:tasks.rados.rados.0.smithi064.stdout:3282: write (excl) oid 16 current snap is 415 2024-03-04T06:21:48.346 INFO:tasks.rados.rados.0.smithi064.stdout:3282: seq_num 1487 ranges {592009=584098,1699077=572192,2801755=190254} 2024-03-04T06:21:48.352 INFO:tasks.rados.rados.0.smithi064.stdout:3282: writing smithi06431743-16 from 592009 to 1176107 tid 1 2024-03-04T06:21:48.358 INFO:tasks.rados.rados.0.smithi064.stdout:3282: writing smithi06431743-16 from 1699077 to 2271269 tid 2 2024-03-04T06:21:48.362 INFO:tasks.rados.rados.0.smithi064.stdout:3282: writing smithi06431743-16 from 2801755 to 2992009 tid 3 2024-03-04T06:21:48.362 INFO:tasks.rados.rados.0.smithi064.stdout:3283: snap_remove snap 410 2024-03-04T06:21:48.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:48.776 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:48.774+0000 7f4688c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:48.779 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:48.778+0000 7f4688c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:49.320 INFO:tasks.rados.rados.0.smithi064.stdout:3283: done (4 left) 2024-03-04T06:21:49.320 INFO:tasks.rados.rados.0.smithi064.stdout:3284: delete oid 8 current snap is 415 2024-03-04T06:21:49.320 INFO:tasks.rados.rados.0.smithi064.stdout:3279: finishing write tid 1 to smithi06431743-24 2024-03-04T06:21:49.320 INFO:tasks.rados.rados.0.smithi064.stdout:3280: finishing write tid 1 to smithi06431743-50 2024-03-04T06:21:49.320 INFO:tasks.rados.rados.0.smithi064.stdout:3279: finishing write tid 2 to smithi06431743-24 2024-03-04T06:21:49.320 INFO:tasks.rados.rados.0.smithi064.stdout:3282: finishing write tid 1 to smithi06431743-16 2024-03-04T06:21:49.320 INFO:tasks.rados.rados.0.smithi064.stdout:3279: finishing write tid 3 to smithi06431743-24 2024-03-04T06:21:49.320 INFO:tasks.rados.rados.0.smithi064.stdout:3280: finishing write tid 2 to smithi06431743-50 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3279: finishing write tid 4 to smithi06431743-24 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3279: finishing write tid 5 to smithi06431743-24 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3279: oid 24 updating version 0 to 1238 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3279: oid 24 version 1238 is already newer than 1238 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3279: oid 24 version 1238 is already newer than 1236 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3279: oid 24 updating version 1238 to 1239 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 1239 (ObjNum 1485 snap 415 seq_num 1485) dirty exists 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3279: left oid 24 (ObjNum 1485 snap 415 seq_num 1485) 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3280: finishing write tid 3 to smithi06431743-50 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3282: finishing write tid 2 to smithi06431743-16 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3280: finishing write tid 4 to smithi06431743-50 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3280: oid 50 updating version 0 to 1059 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3280: oid 50 version 1059 is already newer than 1059 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3280: oid 50 updating version 1059 to 1060 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 1060 (ObjNum 1486 snap 415 seq_num 1486) dirty exists 2024-03-04T06:21:49.321 INFO:tasks.rados.rados.0.smithi064.stdout:3280: left oid 50 (ObjNum 1486 snap 415 seq_num 1486) 2024-03-04T06:21:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:3282: finishing write tid 3 to smithi06431743-16 2024-03-04T06:21:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:3282: finishing write tid 4 to smithi06431743-16 2024-03-04T06:21:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:3282: finishing write tid 5 to smithi06431743-16 2024-03-04T06:21:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:3282: oid 16 updating version 0 to 904 2024-03-04T06:21:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:3282: oid 16 version 904 is already newer than 904 2024-03-04T06:21:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:3282: oid 16 updating version 904 to 905 2024-03-04T06:21:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:3282: oid 16 version 905 is already newer than 902 2024-03-04T06:21:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 905 (ObjNum 1487 snap 415 seq_num 1487) dirty exists 2024-03-04T06:21:49.322 INFO:tasks.rados.rados.0.smithi064.stdout:3282: left oid 16 (ObjNum 1487 snap 415 seq_num 1487) 2024-03-04T06:21:49.407 INFO:tasks.rados.rados.0.smithi064.stdout:3279: done (4 left) 2024-03-04T06:21:49.407 INFO:tasks.rados.rados.0.smithi064.stdout:3280: done (3 left) 2024-03-04T06:21:49.408 INFO:tasks.rados.rados.0.smithi064.stdout:3281: done (2 left) 2024-03-04T06:21:49.408 INFO:tasks.rados.rados.0.smithi064.stdout:3282: done (1 left) 2024-03-04T06:21:49.408 INFO:tasks.rados.rados.0.smithi064.stdout:3284: done (0 left) 2024-03-04T06:21:49.408 INFO:tasks.rados.rados.0.smithi064.stdout:3285: snap_create 2024-03-04T06:21:50.402 INFO:tasks.rados.rados.0.smithi064.stdout:3285: done (0 left) 2024-03-04T06:21:50.402 INFO:tasks.rados.rados.0.smithi064.stdout:3286: snap_remove snap 405 2024-03-04T06:21:50.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:50.963 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:50.961+0000 7f110415d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:50.966 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:50.966+0000 7f110415d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:51.403 INFO:tasks.rados.rados.0.smithi064.stdout:3286: done (0 left) 2024-03-04T06:21:51.403 INFO:tasks.rados.rados.0.smithi064.stdout:3287: write (excl) oid 11 current snap is 416 2024-03-04T06:21:51.403 INFO:tasks.rados.rados.0.smithi064.stdout:3287: seq_num 1489 ranges {625623=734596,1816041=459508,2740248=285375} 2024-03-04T06:21:51.410 INFO:tasks.rados.rados.0.smithi064.stdout:3287: writing smithi06431743-11 from 625623 to 1360219 tid 1 2024-03-04T06:21:51.416 INFO:tasks.rados.rados.0.smithi064.stdout:3287: writing smithi06431743-11 from 1816041 to 2275549 tid 2 2024-03-04T06:21:51.421 INFO:tasks.rados.rados.0.smithi064.stdout:3287: writing smithi06431743-11 from 2740248 to 3025623 tid 3 2024-03-04T06:21:51.421 INFO:tasks.rados.rados.0.smithi064.stdout:3288: write oid 16 current snap is 416 2024-03-04T06:21:51.421 INFO:tasks.rados.rados.0.smithi064.stdout:3288: seq_num 1490 ranges {642430=409845,1758170=719519} 2024-03-04T06:21:51.428 INFO:tasks.rados.rados.0.smithi064.stdout:3288: writing smithi06431743-16 from 642430 to 1052275 tid 1 2024-03-04T06:21:51.439 INFO:tasks.rados.rados.0.smithi064.stdout:3288: writing smithi06431743-16 from 1758170 to 2477689 tid 2 2024-03-04T06:21:51.439 INFO:tasks.rados.rados.0.smithi064.stdout:3289: write (excl) oid 49 current snap is 416 2024-03-04T06:21:51.439 INFO:tasks.rados.rados.0.smithi064.stdout:3289: seq_num 1491 ranges {659237=601447,1933005=463177} 2024-03-04T06:21:51.447 INFO:tasks.rados.rados.0.smithi064.stdout:3289: writing smithi06431743-49 from 659237 to 1260684 tid 1 2024-03-04T06:21:51.455 INFO:tasks.rados.rados.0.smithi064.stdout:3289: writing smithi06431743-49 from 1933005 to 2396182 tid 2 2024-03-04T06:21:51.455 INFO:tasks.rados.rados.0.smithi064.stdout:3290: snap_remove snap 412 2024-03-04T06:21:52.408 INFO:tasks.rados.rados.0.smithi064.stdout:3290: done (3 left) 2024-03-04T06:21:52.408 INFO:tasks.rados.rados.0.smithi064.stdout:3291: snap_remove snap 408 2024-03-04T06:21:53.024 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:21:53.024 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:21:53.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:21:53.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:53.145 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:53.140+0000 7f8df406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:53.149 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:53.148+0000 7f8df406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:53.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:53.179+0000 7f727d39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:53.184 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:53.183+0000 7f727d39d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:53.350 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:21:53.350 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:21:53.361 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1256,"stamp":"2024-03-04T06:21:52.884628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":88317123,"num_objects":1098,"num_object_clones":894,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":171,"num_read":5985,"num_read_kb":4677011,"num_write":5078,"num_write_kb":1347144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14039,"ondisk_log_size":14039,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":714224,"kb_used_data":490824,"kb_used_omap":0,"kb_used_meta":222464,"kb_avail":376773136,"statfs":{"total":386547056640,"available":385815691264,"internally_reserved":0,"allocated":502603776,"data_stored":601549156,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":227803136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1411887,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":35,"num_read_kb":30054,"num_write":38,"num_write_kb":10019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000861"},"pg_stats":[{"pgid":"3.f","version":"829'780","reported_seq":2861,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.360861+0000","last_change":"2024-03-04T06:21:50.373827+0000","last_active":"2024-03-04T06:21:50.360861+0000","last_peered":"2024-03-04T06:21:50.360861+0000","last_clean":"2024-03-04T06:21:50.360861+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:21:50.360861+0000","last_undegraded":"2024-03-04T06:21:50.360861+0000","last_fullsized":"2024-03-04T06:21:50.360861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":780,"log_dups_size":0,"ondisk_log_size":780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5059633,"num_objects":69,"num_object_clones":55,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":316,"num_read_kb":251968,"num_write":288,"num_write_kb":75809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"838'749","reported_seq":2869,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-03-04T06:21:49.391954+0000","last_change":"2024-03-04T06:21:49.391954+0000","last_active":"2024-03-04T06:21:49.391954+0000","last_peered":"2024-03-04T06:21:49.391954+0000","last_clean":"2024-03-04T06:21:49.391954+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:49.391954+0000","last_undegraded":"2024-03-04T06:21:49.391954+0000","last_fullsized":"2024-03-04T06:21:49.391954+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":749,"log_dups_size":0,"ondisk_log_size":749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781905,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":226,"num_read_kb":171792,"num_write":251,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"834'989","reported_seq":3018,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.417792+0000","last_change":"2024-03-04T06:21:50.393428+0000","last_active":"2024-03-04T06:21:50.417792+0000","last_peered":"2024-03-04T06:21:50.417792+0000","last_clean":"2024-03-04T06:21:50.417792+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:21:50.417792+0000","last_undegraded":"2024-03-04T06:21:50.417792+0000","last_fullsized":"2024-03-04T06:21:50.417792+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":989,"log_dups_size":0,"ondisk_log_size":989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7312134,"num_objects":84,"num_object_clones":69,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":485,"num_read_kb":390127,"num_write":370,"num_write_kb":100357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"838'858","reported_seq":2847,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.355665+0000","last_change":"2024-03-04T06:21:50.376492+0000","last_active":"2024-03-04T06:21:50.355665+0000","last_peered":"2024-03-04T06:21:50.355665+0000","last_clean":"2024-03-04T06:21:50.355665+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:21:50.355665+0000","last_undegraded":"2024-03-04T06:21:50.355665+0000","last_fullsized":"2024-03-04T06:21:50.355665+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5563810,"num_objects":59,"num_object_clones":49,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":299,"num_read_kb":228881,"num_write":279,"num_write_kb":77984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"838'349","reported_seq":2659,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.499405+0000","last_change":"2024-03-04T06:21:49.398523+0000","last_active":"2024-03-04T06:21:50.499405+0000","last_peered":"2024-03-04T06:21:50.499405+0000","last_clean":"2024-03-04T06:21:50.499405+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:21:50.499405+0000","last_undegraded":"2024-03-04T06:21:50.499405+0000","last_fullsized":"2024-03-04T06:21:50.499405+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":100,"num_read_kb":71985,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"836'890","reported_seq":2918,"reported_epoch":838,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:50.359652+0000","last_change":"2024-03-04T06:21:50.359652+0000","last_active":"2024-03-04T06:21:50.359652+0000","last_peered":"2024-03-04T06:21:50.359652+0000","last_clean":"2024-03-04T06:21:50.359652+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:21:50.359652+0000","last_undegraded":"2024-03-04T06:21:50.359652+0000","last_fullsized":"2024-03-04T06:21:50.359652+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":890,"log_dups_size":0,"ondisk_log_size":890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":64,"num_object_clones":55,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":393,"num_read_kb":322263,"num_write":310,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2462,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.366949+0000","last_change":"2024-03-04T06:21:50.374507+0000","last_active":"2024-03-04T06:21:50.366949+0000","last_peered":"2024-03-04T06:21:50.366949+0000","last_clean":"2024-03-04T06:21:50.366949+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:21:50.366949+0000","last_undegraded":"2024-03-04T06:21:50.366949+0000","last_fullsized":"2024-03-04T06:21:50.366949+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"838'552","reported_seq":2750,"reported_epoch":840,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:21:51.429701+0000","last_change":"2024-03-04T06:21:51.429701+0000","last_active":"2024-03-04T06:21:51.429701+0000","last_peered":"2024-03-04T06:21:51.429701+0000","last_clean":"2024-03-04T06:21:51.429701+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:21:51.429701+0000","last_undegraded":"2024-03-04T06:21:51.429701+0000","last_fullsized":"2024-03-04T06:21:51.429701+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857553,"num_objects":39,"num_object_clones":30,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":265,"num_read_kb":243519,"num_write":172,"num_write_kb":46227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2253,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.493626+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:21:50.493626+0000","last_peered":"2024-03-04T06:21:50.493626+0000","last_clean":"2024-03-04T06:21:50.493626+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:21:50.493626+0000","last_undegraded":"2024-03-04T06:21:50.493626+0000","last_fullsized":"2024-03-04T06:21:50.493626+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2077,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.431624+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:21:50.431624+0000","last_peered":"2024-03-04T06:21:50.431624+0000","last_clean":"2024-03-04T06:21:50.431624+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:21:50.431624+0000","last_undegraded":"2024-03-04T06:21:50.431624+0000","last_fullsized":"2024-03-04T06:21:50.431624+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"838'888","reported_seq":2927,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.429346+0000","last_change":"2024-03-04T06:21:50.429346+0000","last_active":"2024-03-04T06:21:50.429346+0000","last_peered":"2024-03-04T06:21:50.429346+0000","last_clean":"2024-03-04T06:21:50.429346+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:21:50.429346+0000","last_undegraded":"2024-03-04T06:21:50.429346+0000","last_fullsized":"2024-03-04T06:21:50.429346+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":888,"log_dups_size":0,"ondisk_log_size":888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4908370,"num_objects":70,"num_object_clones":56,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":389,"num_read_kb":309493,"num_write":311,"num_write_kb":83094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2068,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.365846+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:21:50.365846+0000","last_peered":"2024-03-04T06:21:50.365846+0000","last_clean":"2024-03-04T06:21:50.365846+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:21:50.365846+0000","last_undegraded":"2024-03-04T06:21:50.365846+0000","last_fullsized":"2024-03-04T06:21:50.365846+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"838'558","reported_seq":2818,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.489443+0000","last_change":"2024-03-04T06:21:49.398907+0000","last_active":"2024-03-04T06:21:50.489443+0000","last_peered":"2024-03-04T06:21:50.489443+0000","last_clean":"2024-03-04T06:21:50.489443+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:21:50.489443+0000","last_undegraded":"2024-03-04T06:21:50.489443+0000","last_fullsized":"2024-03-04T06:21:50.489443+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2907974,"num_objects":47,"num_object_clones":38,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":292,"num_read_kb":241123,"num_write":191,"num_write_kb":53820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2078,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.424080+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:21:50.424080+0000","last_peered":"2024-03-04T06:21:50.424080+0000","last_clean":"2024-03-04T06:21:50.424080+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:21:50.424080+0000","last_undegraded":"2024-03-04T06:21:50.424080+0000","last_fullsized":"2024-03-04T06:21:50.424080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"838'1516","reported_seq":3366,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.500271+0000","last_change":"2024-03-04T06:21:49.398950+0000","last_active":"2024-03-04T06:21:50.500271+0000","last_peered":"2024-03-04T06:21:50.500271+0000","last_clean":"2024-03-04T06:21:50.500271+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:21:50.500271+0000","last_undegraded":"2024-03-04T06:21:50.500271+0000","last_fullsized":"2024-03-04T06:21:50.500271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1516,"log_dups_size":0,"ondisk_log_size":1516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10976423,"num_objects":124,"num_object_clones":99,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":733,"num_read_kb":588859,"num_write":578,"num_write_kb":154571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2068,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.353675+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:21:50.353675+0000","last_peered":"2024-03-04T06:21:50.353675+0000","last_clean":"2024-03-04T06:21:50.353675+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:21:50.353675+0000","last_undegraded":"2024-03-04T06:21:50.353675+0000","last_fullsized":"2024-03-04T06:21:50.353675+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"834'1766","reported_seq":3374,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-03-04T06:21:49.333399+0000","last_change":"2024-03-04T06:21:49.354715+0000","last_active":"2024-03-04T06:21:49.333399+0000","last_peered":"2024-03-04T06:21:49.333399+0000","last_clean":"2024-03-04T06:21:49.333399+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:49.333399+0000","last_undegraded":"2024-03-04T06:21:49.333399+0000","last_fullsized":"2024-03-04T06:21:49.333399+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1766,"log_dups_size":0,"ondisk_log_size":1766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17178833,"num_objects":135,"num_object_clones":111,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":627,"num_read_kb":465525,"num_write":623,"num_write_kb":167764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"838'833","reported_seq":2927,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.429636+0000","last_change":"2024-03-04T06:21:50.429636+0000","last_active":"2024-03-04T06:21:50.429636+0000","last_peered":"2024-03-04T06:21:50.429636+0000","last_clean":"2024-03-04T06:21:50.429636+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:21:50.429636+0000","last_undegraded":"2024-03-04T06:21:50.429636+0000","last_fullsized":"2024-03-04T06:21:50.429636+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":833,"log_dups_size":0,"ondisk_log_size":833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2790325,"num_objects":63,"num_object_clones":51,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":413,"num_read_kb":341590,"num_write":297,"num_write_kb":80300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2148,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-03-04T06:21:49.345122+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:49.345122+0000","last_peered":"2024-03-04T06:21:49.345122+0000","last_clean":"2024-03-04T06:21:49.345122+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:49.345122+0000","last_undegraded":"2024-03-04T06:21:49.345122+0000","last_fullsized":"2024-03-04T06:21:49.345122+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"838'909","reported_seq":2986,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-03-04T06:21:49.371098+0000","last_change":"2024-03-04T06:21:49.396349+0000","last_active":"2024-03-04T06:21:49.371098+0000","last_peered":"2024-03-04T06:21:49.371098+0000","last_clean":"2024-03-04T06:21:49.371098+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:49.371098+0000","last_undegraded":"2024-03-04T06:21:49.371098+0000","last_fullsized":"2024-03-04T06:21:49.371098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227703,"num_objects":76,"num_object_clones":60,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":354,"num_read_kb":254718,"num_write":325,"num_write_kb":87236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2119,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.502944+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:21:50.502944+0000","last_peered":"2024-03-04T06:21:50.502944+0000","last_clean":"2024-03-04T06:21:50.502944+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:21:50.502944+0000","last_undegraded":"2024-03-04T06:21:50.502944+0000","last_fullsized":"2024-03-04T06:21:50.502944+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"838'1243","reported_seq":3104,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.492577+0000","last_change":"2024-03-04T06:21:49.394369+0000","last_active":"2024-03-04T06:21:50.492577+0000","last_peered":"2024-03-04T06:21:50.492577+0000","last_clean":"2024-03-04T06:21:50.492577+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:21:50.492577+0000","last_undegraded":"2024-03-04T06:21:50.492577+0000","last_fullsized":"2024-03-04T06:21:50.492577+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1243,"log_dups_size":0,"ondisk_log_size":1243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10555819,"num_objects":92,"num_object_clones":78,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":465,"num_read_kb":354560,"num_write":433,"num_write_kb":121118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2148,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-03-04T06:21:49.330134+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:49.330134+0000","last_peered":"2024-03-04T06:21:49.330134+0000","last_clean":"2024-03-04T06:21:49.330134+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:49.330134+0000","last_undegraded":"2024-03-04T06:21:49.330134+0000","last_fullsized":"2024-03-04T06:21:49.330134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"838'1062","reported_seq":3138,"reported_epoch":838,"state":"active+clean","last_fresh":"2024-03-04T06:21:49.390913+0000","last_change":"2024-03-04T06:21:49.390913+0000","last_active":"2024-03-04T06:21:49.390913+0000","last_peered":"2024-03-04T06:21:49.390913+0000","last_clean":"2024-03-04T06:21:49.390913+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:49.390913+0000","last_undegraded":"2024-03-04T06:21:49.390913+0000","last_fullsized":"2024-03-04T06:21:49.390913+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1062,"log_dups_size":0,"ondisk_log_size":1062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5899983,"num_objects":88,"num_object_clones":76,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":525,"num_read_kb":440517,"num_write":398,"num_write_kb":109107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2077,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-03-04T06:21:50.428106+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:21:50.428106+0000","last_peered":"2024-03-04T06:21:50.428106+0000","last_clean":"2024-03-04T06:21:50.428106+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:21:50.428106+0000","last_undegraded":"2024-03-04T06:21:50.428106+0000","last_fullsized":"2024-03-04T06:21:50.428106+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":87726755,"num_objects":1096,"num_object_clones":894,"num_object_copies":3288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":171,"num_read":5882,"num_read_kb":4676920,"num_write":4941,"num_write_kb":1345250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13942,"ondisk_log_size":13942,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706509,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":223180,"kb_used_data":160260,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94148660,"statfs":{"total":96636764160,"available":96408227840,"internally_reserved":0,"allocated":164106240,"data_stored":197493520,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706509,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":165244,"kb_used_data":112052,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94206596,"statfs":{"total":96636764160,"available":96467554304,"internally_reserved":0,"allocated":114741248,"data_stored":134043555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706508,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":160584,"kb_used_data":106488,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94211256,"statfs":{"total":96636764160,"available":96472326144,"internally_reserved":0,"allocated":109043712,"data_stored":133138036,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706508,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":165216,"kb_used_data":112024,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94206624,"statfs":{"total":96636764160,"available":96467582976,"internally_reserved":0,"allocated":114712576,"data_stored":136874045,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:21:53.416 INFO:tasks.rados.rados.0.smithi064.stdout:3291: done (3 left) 2024-03-04T06:21:53.416 INFO:tasks.rados.rados.0.smithi064.stdout:3292: write (excl) oid 9 current snap is 416 2024-03-04T06:21:53.416 INFO:tasks.rados.rados.0.smithi064.stdout:3289: finishing write tid 1 to smithi06431743-49 2024-03-04T06:21:53.416 INFO:tasks.rados.rados.0.smithi064.stdout:3292: seq_num 1492 ranges {676044=676696,1991487=723188} 2024-03-04T06:21:53.423 INFO:tasks.rados.rados.0.smithi064.stdout:3292: writing smithi06431743-9 from 676044 to 1352740 tid 1 2024-03-04T06:21:53.431 INFO:tasks.rados.rados.0.smithi064.stdout:3292: writing smithi06431743-9 from 1991487 to 2714675 tid 2 2024-03-04T06:21:53.431 INFO:tasks.rados.rados.0.smithi064.stdout:3293: delete oid 4 current snap is 416 2024-03-04T06:21:53.431 INFO:tasks.rados.rados.0.smithi064.stdout:3287: finishing write tid 1 to smithi06431743-11 2024-03-04T06:21:53.431 INFO:tasks.rados.rados.0.smithi064.stdout:3289: finishing write tid 2 to smithi06431743-49 2024-03-04T06:21:53.431 INFO:tasks.rados.rados.0.smithi064.stdout:3288: finishing write tid 1 to smithi06431743-16 2024-03-04T06:21:53.431 INFO:tasks.rados.rados.0.smithi064.stdout:3287: finishing write tid 2 to smithi06431743-11 2024-03-04T06:21:53.431 INFO:tasks.rados.rados.0.smithi064.stdout:3289: finishing write tid 3 to smithi06431743-49 2024-03-04T06:21:53.431 INFO:tasks.rados.rados.0.smithi064.stdout:3289: finishing write tid 4 to smithi06431743-49 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3289: oid 49 updating version 0 to 556 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3289: oid 49 updating version 556 to 557 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3289: oid 49 version 557 is already newer than 555 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 557 (ObjNum 1491 snap 416 seq_num 1491) dirty exists 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3289: left oid 49 (ObjNum 1491 snap 416 seq_num 1491) 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3287: finishing write tid 3 to smithi06431743-11 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3288: finishing write tid 2 to smithi06431743-16 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3288: finishing write tid 3 to smithi06431743-16 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3287: finishing write tid 4 to smithi06431743-11 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3288: finishing write tid 4 to smithi06431743-16 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3288: oid 16 updating version 0 to 917 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3288: oid 16 version 917 is already newer than 915 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:3288: oid 16 version 917 is already newer than 916 2024-03-04T06:21:53.432 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 917 (ObjNum 1490 snap 416 seq_num 1490) dirty exists 2024-03-04T06:21:53.433 INFO:tasks.rados.rados.0.smithi064.stdout:3288: left oid 16 (ObjNum 1490 snap 416 seq_num 1490) 2024-03-04T06:21:53.433 INFO:tasks.rados.rados.0.smithi064.stdout:3287: finishing write tid 5 to smithi06431743-11 2024-03-04T06:21:53.433 INFO:tasks.rados.rados.0.smithi064.stdout:3287: oid 11 updating version 0 to 992 2024-03-04T06:21:53.433 INFO:tasks.rados.rados.0.smithi064.stdout:3287: oid 11 updating version 992 to 994 2024-03-04T06:21:53.433 INFO:tasks.rados.rados.0.smithi064.stdout:3287: oid 11 version 994 is already newer than 993 2024-03-04T06:21:53.433 INFO:tasks.rados.rados.0.smithi064.stdout:3287: oid 11 updating version 994 to 995 2024-03-04T06:21:53.433 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 995 (ObjNum 1489 snap 416 seq_num 1489) dirty exists 2024-03-04T06:21:53.433 INFO:tasks.rados.rados.0.smithi064.stdout:3287: left oid 11 (ObjNum 1489 snap 416 seq_num 1489) 2024-03-04T06:21:53.524 INFO:tasks.rados.rados.0.smithi064.stdout:3292: finishing write tid 1 to smithi06431743-9 2024-03-04T06:21:53.533 INFO:tasks.rados.rados.0.smithi064.stdout:3287: done (4 left) 2024-03-04T06:21:53.534 INFO:tasks.rados.rados.0.smithi064.stdout:3288: done (3 left) 2024-03-04T06:21:53.534 INFO:tasks.rados.rados.0.smithi064.stdout:3289: done (2 left) 2024-03-04T06:21:53.534 INFO:tasks.rados.rados.0.smithi064.stdout:3293: done (1 left) 2024-03-04T06:21:53.534 INFO:tasks.rados.rados.0.smithi064.stdout:3294: snap_create 2024-03-04T06:21:53.537 INFO:tasks.rados.rados.0.smithi064.stdout:3292: finishing write tid 2 to smithi06431743-9 2024-03-04T06:21:53.545 INFO:tasks.rados.rados.0.smithi064.stdout:3292: finishing write tid 3 to smithi06431743-9 2024-03-04T06:21:53.546 INFO:tasks.rados.rados.0.smithi064.stdout:3292: finishing write tid 4 to smithi06431743-9 2024-03-04T06:21:53.546 INFO:tasks.rados.rados.0.smithi064.stdout:3292: oid 9 updating version 0 to 1068 2024-03-04T06:21:53.546 INFO:tasks.rados.rados.0.smithi064.stdout:3292: oid 9 version 1068 is already newer than 1067 2024-03-04T06:21:53.546 INFO:tasks.rados.rados.0.smithi064.stdout:3292: oid 9 updating version 1068 to 1069 2024-03-04T06:21:53.546 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 1069 (ObjNum 1492 snap 416 seq_num 1492) dirty exists 2024-03-04T06:21:53.546 INFO:tasks.rados.rados.0.smithi064.stdout:3292: left oid 9 (ObjNum 1492 snap 416 seq_num 1492) 2024-03-04T06:21:54.418 INFO:tasks.rados.rados.0.smithi064.stdout:3292: done (1 left) 2024-03-04T06:21:54.418 INFO:tasks.rados.rados.0.smithi064.stdout:3294: done (0 left) 2024-03-04T06:21:54.418 INFO:tasks.rados.rados.0.smithi064.stdout:3295: write (excl) oid 4 current snap is 417 2024-03-04T06:21:54.418 INFO:tasks.rados.rados.0.smithi064.stdout:3295: seq_num 1494 ranges {709658=427194,1592098=726857,2761010=348648} 2024-03-04T06:21:54.424 INFO:tasks.rados.rados.0.smithi064.stdout:3295: writing smithi06431743-4 from 709658 to 1136852 tid 1 2024-03-04T06:21:54.431 INFO:tasks.rados.rados.0.smithi064.stdout:3295: writing smithi06431743-4 from 1592098 to 2318955 tid 2 2024-03-04T06:21:54.435 INFO:tasks.rados.rados.0.smithi064.stdout:3295: writing smithi06431743-4 from 2761010 to 3109658 tid 3 2024-03-04T06:21:54.435 INFO:tasks.rados.rados.0.smithi064.stdout:3296: read oid 7 snap -1 2024-03-04T06:21:54.435 INFO:tasks.rados.rados.0.smithi064.stdout:3296: expect deleted 2024-03-04T06:21:54.435 INFO:tasks.rados.rados.0.smithi064.stdout:3297: delete oid 2 current snap is 417 2024-03-04T06:21:54.492 INFO:tasks.rados.rados.0.smithi064.stdout:3295: finishing write tid 1 to smithi06431743-4 2024-03-04T06:21:54.498 INFO:tasks.rados.rados.0.smithi064.stdout:3296: done (2 left) 2024-03-04T06:21:54.498 INFO:tasks.rados.rados.0.smithi064.stdout:3297: done (1 left) 2024-03-04T06:21:54.498 INFO:tasks.rados.rados.0.smithi064.stdout:3298: snap_create 2024-03-04T06:21:54.508 INFO:tasks.rados.rados.0.smithi064.stdout:3295: finishing write tid 2 to smithi06431743-4 2024-03-04T06:21:54.515 INFO:tasks.rados.rados.0.smithi064.stdout:3295: finishing write tid 3 to smithi06431743-4 2024-03-04T06:21:54.526 INFO:tasks.rados.rados.0.smithi064.stdout:3295: finishing write tid 4 to smithi06431743-4 2024-03-04T06:21:54.527 INFO:tasks.rados.rados.0.smithi064.stdout:3295: finishing write tid 5 to smithi06431743-4 2024-03-04T06:21:54.527 INFO:tasks.rados.rados.0.smithi064.stdout:3295: oid 4 updating version 0 to 1771 2024-03-04T06:21:54.527 INFO:tasks.rados.rados.0.smithi064.stdout:3295: oid 4 updating version 1771 to 1773 2024-03-04T06:21:54.527 INFO:tasks.rados.rados.0.smithi064.stdout:3295: oid 4 updating version 1773 to 1774 2024-03-04T06:21:54.527 INFO:tasks.rados.rados.0.smithi064.stdout:3295: oid 4 version 1774 is already newer than 1773 2024-03-04T06:21:54.528 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 1774 (ObjNum 1494 snap 417 seq_num 1494) dirty exists 2024-03-04T06:21:54.528 INFO:tasks.rados.rados.0.smithi064.stdout:3295: left oid 4 (ObjNum 1494 snap 417 seq_num 1494) 2024-03-04T06:21:55.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:55.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:55.365+0000 7fd33195d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:55.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:55.370+0000 7fd33195d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:55.422 INFO:tasks.rados.rados.0.smithi064.stdout:3295: done (1 left) 2024-03-04T06:21:55.422 INFO:tasks.rados.rados.0.smithi064.stdout:3298: done (0 left) 2024-03-04T06:21:55.423 INFO:tasks.rados.rados.0.smithi064.stdout:3299: read oid 6 snap -1 2024-03-04T06:21:55.423 INFO:tasks.rados.rados.0.smithi064.stdout:3299: expect deleted 2024-03-04T06:21:55.423 INFO:tasks.rados.rados.0.smithi064.stdout:3300: write (excl) oid 48 current snap is 418 2024-03-04T06:21:55.423 INFO:tasks.rados.rados.0.smithi064.stdout:3300: seq_num 1496 ranges {743272=577692,2109062=730526} 2024-03-04T06:21:55.430 INFO:tasks.rados.rados.0.smithi064.stdout:3300: writing smithi06431743-48 from 743272 to 1320964 tid 1 2024-03-04T06:21:55.439 INFO:tasks.rados.rados.0.smithi064.stdout:3300: writing smithi06431743-48 from 2109062 to 2839588 tid 2 2024-03-04T06:21:55.439 INFO:tasks.rados.rados.0.smithi064.stdout:3301: read oid 43 snap -1 2024-03-04T06:21:55.439 INFO:tasks.rados.rados.0.smithi064.stdout:3301: expect deleted 2024-03-04T06:21:55.439 INFO:tasks.rados.rados.0.smithi064.stdout:3302: write (excl) oid 34 current snap is 418 2024-03-04T06:21:55.439 INFO:tasks.rados.rados.0.smithi064.stdout:3302: seq_num 1497 ranges {760079=652941,2167544=474184} 2024-03-04T06:21:55.448 INFO:tasks.rados.rados.0.smithi064.stdout:3302: writing smithi06431743-34 from 760079 to 1413020 tid 1 2024-03-04T06:21:55.455 INFO:tasks.rados.rados.0.smithi064.stdout:3302: writing smithi06431743-34 from 2167544 to 2641728 tid 2 2024-03-04T06:21:55.455 INFO:tasks.rados.rados.0.smithi064.stdout:3303: read oid 38 snap -1 2024-03-04T06:21:55.455 INFO:tasks.rados.rados.0.smithi064.stdout:3303: expect deleted 2024-03-04T06:21:55.455 INFO:tasks.rados.rados.0.smithi064.stdout:3304: write oid 26 current snap is 418 2024-03-04T06:21:55.455 INFO:tasks.rados.rados.0.smithi064.stdout:3304: seq_num 1498 ranges {776886=444543,1826026=734195,2987055=189831} 2024-03-04T06:21:55.462 INFO:tasks.rados.rados.0.smithi064.stdout:3304: writing smithi06431743-26 from 776886 to 1221429 tid 1 2024-03-04T06:21:55.471 INFO:tasks.rados.rados.0.smithi064.stdout:3304: writing smithi06431743-26 from 1826026 to 2560221 tid 2 2024-03-04T06:21:55.476 INFO:tasks.rados.rados.0.smithi064.stdout:3304: writing smithi06431743-26 from 2987055 to 3176886 tid 3 2024-03-04T06:21:55.476 INFO:tasks.rados.rados.0.smithi064.stdout:3305: read oid 19 snap -1 2024-03-04T06:21:55.476 INFO:tasks.rados.rados.0.smithi064.stdout:3305: expect (ObjNum 1377 snap 385 seq_num 1377) 2024-03-04T06:21:55.476 INFO:tasks.rados.rados.0.smithi064.stdout:3306: snap_remove snap 377 2024-03-04T06:21:56.426 INFO:tasks.rados.rados.0.smithi064.stdout:3306: done (7 left) 2024-03-04T06:21:56.426 INFO:tasks.rados.rados.0.smithi064.stdout:3307: delete oid 13 current snap is 418 2024-03-04T06:21:56.426 INFO:tasks.rados.rados.0.smithi064.stdout:3300: finishing write tid 1 to smithi06431743-48 2024-03-04T06:21:56.426 INFO:tasks.rados.rados.0.smithi064.stdout:3305: expect (ObjNum 1377 snap 385 seq_num 1377) 2024-03-04T06:21:56.642 INFO:tasks.rados.rados.0.smithi064.stdout:3302: finishing write tid 1 to smithi06431743-34 2024-03-04T06:21:56.642 INFO:tasks.rados.rados.0.smithi064.stdout:3300: finishing write tid 2 to smithi06431743-48 2024-03-04T06:21:56.642 INFO:tasks.rados.rados.0.smithi064.stdout:3300: finishing write tid 3 to smithi06431743-48 2024-03-04T06:21:56.642 INFO:tasks.rados.rados.0.smithi064.stdout:3300: finishing write tid 4 to smithi06431743-48 2024-03-04T06:21:56.642 INFO:tasks.rados.rados.0.smithi064.stdout:3300: oid 48 updating version 0 to 570 2024-03-04T06:21:56.642 INFO:tasks.rados.rados.0.smithi064.stdout:3300: oid 48 version 570 is already newer than 570 2024-03-04T06:21:56.642 INFO:tasks.rados.rados.0.smithi064.stdout:3300: oid 48 version 570 is already newer than 568 2024-03-04T06:21:56.642 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 570 (ObjNum 1496 snap 418 seq_num 1496) dirty exists 2024-03-04T06:21:56.642 INFO:tasks.rados.rados.0.smithi064.stdout:3300: left oid 48 (ObjNum 1496 snap 418 seq_num 1496) 2024-03-04T06:21:56.642 INFO:tasks.rados.rados.0.smithi064.stdout:3302: finishing write tid 2 to smithi06431743-34 2024-03-04T06:21:56.642 INFO:tasks.rados.rados.0.smithi064.stdout:3302: finishing write tid 3 to smithi06431743-34 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3302: finishing write tid 4 to smithi06431743-34 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3302: oid 34 updating version 0 to 1524 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3302: oid 34 updating version 1524 to 1525 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3302: oid 34 updating version 1525 to 1526 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 1526 (ObjNum 1497 snap 418 seq_num 1497) dirty exists 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3302: left oid 34 (ObjNum 1497 snap 418 seq_num 1497) 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3304: finishing write tid 1 to smithi06431743-26 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3304: finishing write tid 2 to smithi06431743-26 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3299: done (7 left) 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3300: done (6 left) 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3301: done (5 left) 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3302: done (4 left) 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3303: done (3 left) 2024-03-04T06:21:56.643 INFO:tasks.rados.rados.0.smithi064.stdout:3305: done (2 left) 2024-03-04T06:21:56.644 INFO:tasks.rados.rados.0.smithi064.stdout:3307: done (1 left) 2024-03-04T06:21:56.644 INFO:tasks.rados.rados.0.smithi064.stdout:3308: write oid 48 current snap is 418 2024-03-04T06:21:56.644 INFO:tasks.rados.rados.0.smithi064.stdout:3304: finishing write tid 3 to smithi06431743-26 2024-03-04T06:21:56.644 INFO:tasks.rados.rados.0.smithi064.stdout:3304: finishing write tid 4 to smithi06431743-26 2024-03-04T06:21:56.644 INFO:tasks.rados.rados.0.smithi064.stdout:3308: seq_num 1500 ranges {410500=595041,1542990=621511,2925548=284952} 2024-03-04T06:21:56.647 INFO:tasks.rados.rados.0.smithi064.stdout:3308: writing smithi06431743-48 from 410500 to 1005541 tid 1 2024-03-04T06:21:56.654 INFO:tasks.rados.rados.0.smithi064.stdout:3308: writing smithi06431743-48 from 1542990 to 2164501 tid 2 2024-03-04T06:21:56.660 INFO:tasks.rados.rados.0.smithi064.stdout:3308: writing smithi06431743-48 from 2925548 to 3210500 tid 3 2024-03-04T06:21:56.660 INFO:tasks.rados.rados.0.smithi064.stdout:3309: read oid 38 snap -1 2024-03-04T06:21:56.660 INFO:tasks.rados.rados.0.smithi064.stdout:3309: expect deleted 2024-03-04T06:21:56.661 INFO:tasks.rados.rados.0.smithi064.stdout:3310: write (excl) oid 12 current snap is 418 2024-03-04T06:21:56.661 INFO:tasks.rados.rados.0.smithi064.stdout:3310: seq_num 1501 ranges {427307=670290,1601472=481522,2552971=483056} 2024-03-04T06:21:56.666 INFO:tasks.rados.rados.0.smithi064.stdout:3310: writing smithi06431743-12 from 427307 to 1097597 tid 1 2024-03-04T06:21:56.672 INFO:tasks.rados.rados.0.smithi064.stdout:3310: writing smithi06431743-12 from 1601472 to 2082994 tid 2 2024-03-04T06:21:56.677 INFO:tasks.rados.rados.0.smithi064.stdout:3310: writing smithi06431743-12 from 2552971 to 3036027 tid 3 2024-03-04T06:21:56.677 INFO:tasks.rados.rados.0.smithi064.stdout:3311: snap_create 2024-03-04T06:21:56.677 INFO:tasks.rados.rados.0.smithi064.stdout:3304: finishing write tid 5 to smithi06431743-26 2024-03-04T06:21:56.677 INFO:tasks.rados.rados.0.smithi064.stdout:3304: oid 26 updating version 0 to 1530 2024-03-04T06:21:56.677 INFO:tasks.rados.rados.0.smithi064.stdout:3304: oid 26 version 1530 is already newer than 1530 2024-03-04T06:21:56.677 INFO:tasks.rados.rados.0.smithi064.stdout:3304: oid 26 version 1530 is already newer than 1530 2024-03-04T06:21:56.677 INFO:tasks.rados.rados.0.smithi064.stdout:3304: oid 26 version 1530 is already newer than 1530 2024-03-04T06:21:56.678 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 1530 (ObjNum 1498 snap 418 seq_num 1498) dirty exists 2024-03-04T06:21:56.678 INFO:tasks.rados.rados.0.smithi064.stdout:3304: left oid 26 (ObjNum 1498 snap 418 seq_num 1498) 2024-03-04T06:21:56.683 INFO:tasks.rados.rados.0.smithi064.stdout:3308: finishing write tid 1 to smithi06431743-48 2024-03-04T06:21:56.711 INFO:tasks.rados.rados.0.smithi064.stdout:3308: finishing write tid 2 to smithi06431743-48 2024-03-04T06:21:56.718 INFO:tasks.rados.rados.0.smithi064.stdout:3310: finishing write tid 1 to smithi06431743-12 2024-03-04T06:21:56.724 INFO:tasks.rados.rados.0.smithi064.stdout:3308: finishing write tid 3 to smithi06431743-48 2024-03-04T06:21:56.732 INFO:tasks.rados.rados.0.smithi064.stdout:3308: finishing write tid 4 to smithi06431743-48 2024-03-04T06:21:56.733 INFO:tasks.rados.rados.0.smithi064.stdout:3308: finishing write tid 5 to smithi06431743-48 2024-03-04T06:21:56.733 INFO:tasks.rados.rados.0.smithi064.stdout:3308: oid 48 updating version 0 to 576 2024-03-04T06:21:56.733 INFO:tasks.rados.rados.0.smithi064.stdout:3308: oid 48 version 576 is already newer than 575 2024-03-04T06:21:56.734 INFO:tasks.rados.rados.0.smithi064.stdout:3308: oid 48 updating version 576 to 577 2024-03-04T06:21:56.734 INFO:tasks.rados.rados.0.smithi064.stdout:3308: oid 48 updating version 577 to 578 2024-03-04T06:21:56.734 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 578 (ObjNum 1500 snap 418 seq_num 1500) dirty exists 2024-03-04T06:21:56.734 INFO:tasks.rados.rados.0.smithi064.stdout:3308: left oid 48 (ObjNum 1500 snap 418 seq_num 1500) 2024-03-04T06:21:56.735 INFO:tasks.rados.rados.0.smithi064.stdout:3310: finishing write tid 2 to smithi06431743-12 2024-03-04T06:21:56.744 INFO:tasks.rados.rados.0.smithi064.stdout:3310: finishing write tid 3 to smithi06431743-12 2024-03-04T06:21:56.753 INFO:tasks.rados.rados.0.smithi064.stdout:3310: finishing write tid 4 to smithi06431743-12 2024-03-04T06:21:56.754 INFO:tasks.rados.rados.0.smithi064.stdout:3310: finishing write tid 5 to smithi06431743-12 2024-03-04T06:21:56.754 INFO:tasks.rados.rados.0.smithi064.stdout:3310: oid 12 updating version 0 to 871 2024-03-04T06:21:56.754 INFO:tasks.rados.rados.0.smithi064.stdout:3310: oid 12 updating version 871 to 873 2024-03-04T06:21:56.754 INFO:tasks.rados.rados.0.smithi064.stdout:3310: oid 12 version 873 is already newer than 870 2024-03-04T06:21:56.754 INFO:tasks.rados.rados.0.smithi064.stdout:3310: oid 12 version 873 is already newer than 872 2024-03-04T06:21:56.754 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 873 (ObjNum 1501 snap 418 seq_num 1501) dirty exists 2024-03-04T06:21:56.754 INFO:tasks.rados.rados.0.smithi064.stdout:3310: left oid 12 (ObjNum 1501 snap 418 seq_num 1501) 2024-03-04T06:21:57.437 INFO:tasks.rados.rados.0.smithi064.stdout:3304: done (4 left) 2024-03-04T06:21:57.437 INFO:tasks.rados.rados.0.smithi064.stdout:3308: done (3 left) 2024-03-04T06:21:57.437 INFO:tasks.rados.rados.0.smithi064.stdout:3309: done (2 left) 2024-03-04T06:21:57.437 INFO:tasks.rados.rados.0.smithi064.stdout:3310: done (1 left) 2024-03-04T06:21:57.437 INFO:tasks.rados.rados.0.smithi064.stdout:3311: done (0 left) 2024-03-04T06:21:57.438 INFO:tasks.rados.rados.0.smithi064.stdout:3312: snap_create 2024-03-04T06:21:57.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:21:57.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:57.547+0000 7f6bd442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:57.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:57.552+0000 7f6bd442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:58.319 INFO:tasks.rados.rados.0.smithi064.stdout:3312: done (0 left) 2024-03-04T06:21:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:3313: write (excl) oid 45 current snap is 420 2024-03-04T06:21:58.320 INFO:tasks.rados.rados.0.smithi064.stdout:3313: seq_num 1502 ranges {444114=745539,1943601=625180} 2024-03-04T06:21:58.325 INFO:tasks.rados.rados.0.smithi064.stdout:3313: writing smithi06431743-45 from 444114 to 1189653 tid 1 2024-03-04T06:21:58.334 INFO:tasks.rados.rados.0.smithi064.stdout:3313: writing smithi06431743-45 from 1943601 to 2568781 tid 2 2024-03-04T06:21:58.334 INFO:tasks.rados.rados.0.smithi064.stdout:3314: snap_remove snap 375 2024-03-04T06:21:58.363 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:21:58.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:21:58.492 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:58.486+0000 7fbe03107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:58.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:58.495+0000 7fbe03107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:59.326 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:21:59.328 INFO:tasks.rados.rados.0.smithi064.stdout:3314: done (1 left) 2024-03-04T06:21:59.329 INFO:tasks.rados.rados.0.smithi064.stdout:3315: snap_create 2024-03-04T06:21:59.329 INFO:tasks.rados.rados.0.smithi064.stdout:3313: finishing write tid 1 to smithi06431743-45 2024-03-04T06:21:59.329 INFO:tasks.rados.rados.0.smithi064.stdout:3313: finishing write tid 2 to smithi06431743-45 2024-03-04T06:21:59.329 INFO:tasks.rados.rados.0.smithi064.stdout:3313: finishing write tid 3 to smithi06431743-45 2024-03-04T06:21:59.329 INFO:tasks.rados.rados.0.smithi064.stdout:3313: finishing write tid 4 to smithi06431743-45 2024-03-04T06:21:59.329 INFO:tasks.rados.rados.0.smithi064.stdout:3313: oid 45 updating version 0 to 1003 2024-03-04T06:21:59.329 INFO:tasks.rados.rados.0.smithi064.stdout:3313: oid 45 version 1003 is already newer than 1003 2024-03-04T06:21:59.329 INFO:tasks.rados.rados.0.smithi064.stdout:3313: oid 45 version 1003 is already newer than 1001 2024-03-04T06:21:59.329 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 1003 (ObjNum 1502 snap 420 seq_num 1502) dirty exists 2024-03-04T06:21:59.329 INFO:tasks.rados.rados.0.smithi064.stdout:3313: left oid 45 (ObjNum 1502 snap 420 seq_num 1502) 2024-03-04T06:21:59.339 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:21:59.460 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:59.454+0000 7f4ebef5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:59.463 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:59.462+0000 7f4ebef5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:59.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:21:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:59.726+0000 7f4ba991d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:21:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:21:59.730+0000 7f4ba991d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:00.327 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:22:00.330 INFO:tasks.rados.rados.0.smithi064.stdout:3313: done (1 left) 2024-03-04T06:22:00.330 INFO:tasks.rados.rados.0.smithi064.stdout:3315: done (0 left) 2024-03-04T06:22:00.330 INFO:tasks.rados.rados.0.smithi064.stdout:3316: snap_remove snap 420 2024-03-04T06:22:00.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:22:00.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:00.456+0000 7fa996907640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:00.465 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:00.464+0000 7fa996907640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:01.327 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:22:01.331 INFO:tasks.rados.rados.0.smithi064.stdout:3316: done (0 left) 2024-03-04T06:22:01.331 INFO:tasks.rados.rados.0.smithi064.stdout:3317: read oid 49 snap -1 2024-03-04T06:22:01.331 INFO:tasks.rados.rados.0.smithi064.stdout:3317: expect (ObjNum 1491 snap 416 seq_num 1491) 2024-03-04T06:22:01.331 INFO:tasks.rados.rados.0.smithi064.stdout:3318: read oid 43 snap 416 2024-03-04T06:22:01.331 INFO:tasks.rados.rados.0.smithi064.stdout:3318: expect deleted 2024-03-04T06:22:01.331 INFO:tasks.rados.rados.0.smithi064.stdout:3319: write (excl) oid 28 current snap is 421 2024-03-04T06:22:01.331 INFO:tasks.rados.rados.0.smithi064.stdout:3319: seq_num 1503 ranges {460921=420788,1602083=485191,2491464=621953} 2024-03-04T06:22:01.335 INFO:tasks.rados.rados.0.smithi064.stdout:3319: writing smithi06431743-28 from 460921 to 881709 tid 1 2024-03-04T06:22:01.342 INFO:tasks.rados.rados.0.smithi064.stdout:3319: writing smithi06431743-28 from 1602083 to 2087274 tid 2 2024-03-04T06:22:01.349 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:22:01.350 INFO:tasks.rados.rados.0.smithi064.stdout:3319: writing smithi06431743-28 from 2491464 to 3113417 tid 3 2024-03-04T06:22:01.350 INFO:tasks.rados.rados.0.smithi064.stdout:3320: write (excl) oid 33 current snap is 421 2024-03-04T06:22:01.350 INFO:tasks.rados.rados.0.smithi064.stdout:3320: seq_num 1504 ranges {477728=496037,1660565=628849,2918887=358841} 2024-03-04T06:22:01.354 INFO:tasks.rados.rados.0.smithi064.stdout:3320: writing smithi06431743-33 from 477728 to 973765 tid 1 2024-03-04T06:22:01.364 INFO:tasks.rados.rados.0.smithi064.stdout:3320: writing smithi06431743-33 from 1660565 to 2289414 tid 2 2024-03-04T06:22:01.370 INFO:tasks.rados.rados.0.smithi064.stdout:3320: writing smithi06431743-33 from 2918887 to 3277728 tid 3 2024-03-04T06:22:01.370 INFO:tasks.rados.rados.0.smithi064.stdout:3321: write (excl) oid 41 current snap is 421 2024-03-04T06:22:01.370 INFO:tasks.rados.rados.0.smithi064.stdout:3321: seq_num 1505 ranges {494535=571286,1719047=488860,2946310=348225} 2024-03-04T06:22:01.378 INFO:tasks.rados.rados.0.smithi064.stdout:3321: writing smithi06431743-41 from 494535 to 1065821 tid 1 2024-03-04T06:22:01.386 INFO:tasks.rados.rados.0.smithi064.stdout:3321: writing smithi06431743-41 from 1719047 to 2207907 tid 2 2024-03-04T06:22:01.393 INFO:tasks.rados.rados.0.smithi064.stdout:3321: writing smithi06431743-41 from 2946310 to 3294535 tid 3 2024-03-04T06:22:01.393 INFO:tasks.rados.rados.0.smithi064.stdout:3322: write (excl) oid 4 current snap is 421 2024-03-04T06:22:01.393 INFO:tasks.rados.rados.0.smithi064.stdout:3322: seq_num 1506 ranges {511342=762888,1777529=632518,2973733=337609} 2024-03-04T06:22:01.402 INFO:tasks.rados.rados.0.smithi064.stdout:3322: writing smithi06431743-4 from 511342 to 1274230 tid 1 2024-03-04T06:22:01.410 INFO:tasks.rados.rados.0.smithi064.stdout:3322: writing smithi06431743-4 from 1777529 to 2410047 tid 2 2024-03-04T06:22:01.416 INFO:tasks.rados.rados.0.smithi064.stdout:3322: writing smithi06431743-4 from 2973733 to 3311342 tid 3 2024-03-04T06:22:01.416 INFO:tasks.rados.rados.0.smithi064.stdout:3323: delete oid 2 current snap is 421 2024-03-04T06:22:01.441 INFO:tasks.rados.rados.0.smithi064.stdout:3318: done (6 left) 2024-03-04T06:22:01.441 INFO:tasks.rados.rados.0.smithi064.stdout:3323: done (5 left) 2024-03-04T06:22:01.441 INFO:tasks.rados.rados.0.smithi064.stdout:3324: write (excl) oid 23 current snap is 421 2024-03-04T06:22:01.441 INFO:tasks.rados.rados.0.smithi064.stdout:3324: seq_num 1508 ranges {544956=513386,1494493=636187,2628579=594666} 2024-03-04T06:22:01.442 INFO:tasks.rados.rados.0.smithi064.stdout:3324: writing smithi06431743-23 from 544956 to 1058342 tid 1 2024-03-04T06:22:01.450 INFO:tasks.rados.rados.0.smithi064.stdout:3324: writing smithi06431743-23 from 1494493 to 2130680 tid 2 2024-03-04T06:22:01.459 INFO:tasks.rados.rados.0.smithi064.stdout:3324: writing smithi06431743-23 from 2628579 to 3223245 tid 3 2024-03-04T06:22:01.459 INFO:tasks.rados.rados.0.smithi064.stdout:3325: delete oid 48 current snap is 421 2024-03-04T06:22:01.459 INFO:tasks.rados.rados.0.smithi064.stdout:3317: expect (ObjNum 1491 snap 416 seq_num 1491) 2024-03-04T06:22:01.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:01.490+0000 7f7b14f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:01.501 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:01.501+0000 7f7b14f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:01.622 INFO:tasks.rados.rados.0.smithi064.stdout:3319: finishing write tid 1 to smithi06431743-28 2024-03-04T06:22:01.622 INFO:tasks.rados.rados.0.smithi064.stdout:3321: finishing write tid 1 to smithi06431743-41 2024-03-04T06:22:01.622 INFO:tasks.rados.rados.0.smithi064.stdout:3322: finishing write tid 1 to smithi06431743-4 2024-03-04T06:22:01.622 INFO:tasks.rados.rados.0.smithi064.stdout:3320: finishing write tid 1 to smithi06431743-33 2024-03-04T06:22:01.622 INFO:tasks.rados.rados.0.smithi064.stdout:3317: done (6 left) 2024-03-04T06:22:01.622 INFO:tasks.rados.rados.0.smithi064.stdout:3325: done (5 left) 2024-03-04T06:22:01.622 INFO:tasks.rados.rados.0.smithi064.stdout:3326: write (excl) oid 15 current snap is 421 2024-03-04T06:22:01.622 INFO:tasks.rados.rados.0.smithi064.stdout:3321: finishing write tid 2 to smithi06431743-41 2024-03-04T06:22:01.622 INFO:tasks.rados.rados.0.smithi064.stdout:3322: finishing write tid 2 to smithi06431743-4 2024-03-04T06:22:01.622 INFO:tasks.rados.rados.0.smithi064.stdout:3319: finishing write tid 2 to smithi06431743-28 2024-03-04T06:22:01.623 INFO:tasks.rados.rados.0.smithi064.stdout:3322: finishing write tid 3 to smithi06431743-4 2024-03-04T06:22:01.623 INFO:tasks.rados.rados.0.smithi064.stdout:3326: seq_num 1510 ranges {578570=663884,1895104=639856,2967072=411498} 2024-03-04T06:22:01.628 INFO:tasks.rados.rados.0.smithi064.stdout:3326: writing smithi06431743-15 from 578570 to 1242454 tid 1 2024-03-04T06:22:01.636 INFO:tasks.rados.rados.0.smithi064.stdout:3326: writing smithi06431743-15 from 1895104 to 2534960 tid 2 2024-03-04T06:22:01.643 INFO:tasks.rados.rados.0.smithi064.stdout:3326: writing smithi06431743-15 from 2967072 to 3378570 tid 3 2024-03-04T06:22:01.643 INFO:tasks.rados.rados.0.smithi064.stdout:3327: snap_create 2024-03-04T06:22:01.643 INFO:tasks.rados.rados.0.smithi064.stdout:3321: finishing write tid 3 to smithi06431743-41 2024-03-04T06:22:01.643 INFO:tasks.rados.rados.0.smithi064.stdout:3320: finishing write tid 2 to smithi06431743-33 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3321: finishing write tid 4 to smithi06431743-41 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3321: finishing write tid 5 to smithi06431743-41 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3321: oid 41 updating version 0 to 907 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3321: oid 41 updating version 907 to 908 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3321: oid 41 version 908 is already newer than 906 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3321: oid 41 version 908 is already newer than 905 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 908 (ObjNum 1505 snap 421 seq_num 1505) dirty exists 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3321: left oid 41 (ObjNum 1505 snap 421 seq_num 1505) 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3322: finishing write tid 4 to smithi06431743-4 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3322: finishing write tid 5 to smithi06431743-4 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3322: oid 4 updating version 0 to 1794 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3322: oid 4 version 1794 is already newer than 1792 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3322: oid 4 updating version 1794 to 1795 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3322: oid 4 version 1795 is already newer than 1794 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 1795 (ObjNum 1506 snap 421 seq_num 1506) dirty exists 2024-03-04T06:22:01.644 INFO:tasks.rados.rados.0.smithi064.stdout:3322: left oid 4 (ObjNum 1506 snap 421 seq_num 1506) 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3320: finishing write tid 3 to smithi06431743-33 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3319: finishing write tid 3 to smithi06431743-28 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3320: finishing write tid 4 to smithi06431743-33 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3320: finishing write tid 5 to smithi06431743-33 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3320: oid 33 updating version 0 to 1071 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3320: oid 33 updating version 1071 to 1073 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3320: oid 33 version 1073 is already newer than 1073 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3320: oid 33 version 1073 is already newer than 1070 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 1073 (ObjNum 1504 snap 421 seq_num 1504) dirty exists 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3320: left oid 33 (ObjNum 1504 snap 421 seq_num 1504) 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3319: finishing write tid 4 to smithi06431743-28 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3319: finishing write tid 5 to smithi06431743-28 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3319: oid 28 updating version 0 to 1551 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3319: oid 28 version 1551 is already newer than 1550 2024-03-04T06:22:01.645 INFO:tasks.rados.rados.0.smithi064.stdout:3319: oid 28 version 1551 is already newer than 1548 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3319: oid 28 version 1551 is already newer than 1549 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 1551 (ObjNum 1503 snap 421 seq_num 1503) dirty exists 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3319: left oid 28 (ObjNum 1503 snap 421 seq_num 1503) 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3324: finishing write tid 1 to smithi06431743-23 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3324: finishing write tid 2 to smithi06431743-23 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3324: finishing write tid 3 to smithi06431743-23 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3324: finishing write tid 4 to smithi06431743-23 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3324: finishing write tid 5 to smithi06431743-23 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3324: oid 23 updating version 0 to 910 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3324: oid 23 updating version 910 to 913 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3324: oid 23 version 913 is already newer than 913 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3324: oid 23 version 913 is already newer than 913 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 913 (ObjNum 1508 snap 421 seq_num 1508) dirty exists 2024-03-04T06:22:01.646 INFO:tasks.rados.rados.0.smithi064.stdout:3324: left oid 23 (ObjNum 1508 snap 421 seq_num 1508) 2024-03-04T06:22:01.653 INFO:tasks.rados.rados.0.smithi064.stdout:3326: finishing write tid 1 to smithi06431743-15 2024-03-04T06:22:01.670 INFO:tasks.rados.rados.0.smithi064.stdout:3326: finishing write tid 2 to smithi06431743-15 2024-03-04T06:22:01.678 INFO:tasks.rados.rados.0.smithi064.stdout:3326: finishing write tid 3 to smithi06431743-15 2024-03-04T06:22:01.681 INFO:tasks.rados.rados.0.smithi064.stdout:3326: finishing write tid 4 to smithi06431743-15 2024-03-04T06:22:01.682 INFO:tasks.rados.rados.0.smithi064.stdout:3326: finishing write tid 5 to smithi06431743-15 2024-03-04T06:22:01.683 INFO:tasks.rados.rados.0.smithi064.stdout:3326: oid 15 updating version 0 to 847 2024-03-04T06:22:01.683 INFO:tasks.rados.rados.0.smithi064.stdout:3326: oid 15 version 847 is already newer than 844 2024-03-04T06:22:01.683 INFO:tasks.rados.rados.0.smithi064.stdout:3326: oid 15 version 847 is already newer than 847 2024-03-04T06:22:01.683 INFO:tasks.rados.rados.0.smithi064.stdout:3326: oid 15 version 847 is already newer than 845 2024-03-04T06:22:01.683 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 847 (ObjNum 1510 snap 421 seq_num 1510) dirty exists 2024-03-04T06:22:01.683 INFO:tasks.rados.rados.0.smithi064.stdout:3326: left oid 15 (ObjNum 1510 snap 421 seq_num 1510) 2024-03-04T06:22:01.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:01.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:01.906+0000 7f0ba726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:01.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:01.910+0000 7f0ba726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:02.447 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:22:02.450 INFO:tasks.rados.rados.0.smithi064.stdout:3319: done (6 left) 2024-03-04T06:22:02.450 INFO:tasks.rados.rados.0.smithi064.stdout:3320: done (5 left) 2024-03-04T06:22:02.450 INFO:tasks.rados.rados.0.smithi064.stdout:3321: done (4 left) 2024-03-04T06:22:02.450 INFO:tasks.rados.rados.0.smithi064.stdout:3322: done (3 left) 2024-03-04T06:22:02.450 INFO:tasks.rados.rados.0.smithi064.stdout:3324: done (2 left) 2024-03-04T06:22:02.450 INFO:tasks.rados.rados.0.smithi064.stdout:3326: done (1 left) 2024-03-04T06:22:02.450 INFO:tasks.rados.rados.0.smithi064.stdout:3327: done (0 left) 2024-03-04T06:22:02.450 INFO:tasks.rados.rados.0.smithi064.stdout:3328: write (excl) oid 32 current snap is 422 2024-03-04T06:22:02.450 INFO:tasks.rados.rados.0.smithi064.stdout:3328: seq_num 1511 ranges {595377=739133,1953586=499867,3110848=284529} 2024-03-04T06:22:02.457 INFO:tasks.rados.rados.0.smithi064.stdout:3328: writing smithi06431743-32 from 595377 to 1334510 tid 1 2024-03-04T06:22:02.462 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:22:02.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:22:02.464 INFO:tasks.rados.rados.0.smithi064.stdout:3328: writing smithi06431743-32 from 1953586 to 2453453 tid 2 2024-03-04T06:22:02.471 INFO:tasks.rados.rados.0.smithi064.stdout:3328: writing smithi06431743-32 from 3110848 to 3395377 tid 3 2024-03-04T06:22:02.471 INFO:tasks.rados.rados.0.smithi064.stdout:3329: read oid 25 snap 384 2024-03-04T06:22:02.471 INFO:tasks.rados.rados.0.smithi064.stdout:3329: expect (ObjNum 1278 snap 346 seq_num 1278) 2024-03-04T06:22:02.471 INFO:tasks.rados.rados.0.smithi064.stdout:3330: snap_create 2024-03-04T06:22:02.496 INFO:tasks.rados.rados.0.smithi064.stdout:3329: expect (ObjNum 1278 snap 346 seq_num 1278) 2024-03-04T06:22:02.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:02.569+0000 7f68ab02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:02.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:02.576+0000 7f68ab02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:02.645 INFO:tasks.rados.rados.0.smithi064.stdout:3328: finishing write tid 1 to smithi06431743-32 2024-03-04T06:22:02.646 INFO:tasks.rados.rados.0.smithi064.stdout:3328: finishing write tid 2 to smithi06431743-32 2024-03-04T06:22:02.646 INFO:tasks.rados.rados.0.smithi064.stdout:3328: finishing write tid 3 to smithi06431743-32 2024-03-04T06:22:02.646 INFO:tasks.rados.rados.0.smithi064.stdout:3328: finishing write tid 4 to smithi06431743-32 2024-03-04T06:22:02.646 INFO:tasks.rados.rados.0.smithi064.stdout:3328: finishing write tid 5 to smithi06431743-32 2024-03-04T06:22:02.646 INFO:tasks.rados.rados.0.smithi064.stdout:3328: oid 32 updating version 0 to 1554 2024-03-04T06:22:02.646 INFO:tasks.rados.rados.0.smithi064.stdout:3328: oid 32 version 1554 is already newer than 1553 2024-03-04T06:22:02.646 INFO:tasks.rados.rados.0.smithi064.stdout:3328: oid 32 updating version 1554 to 1556 2024-03-04T06:22:02.646 INFO:tasks.rados.rados.0.smithi064.stdout:3328: oid 32 version 1556 is already newer than 1555 2024-03-04T06:22:02.646 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1556 (ObjNum 1511 snap 422 seq_num 1511) dirty exists 2024-03-04T06:22:02.646 INFO:tasks.rados.rados.0.smithi064.stdout:3328: left oid 32 (ObjNum 1511 snap 422 seq_num 1511) 2024-03-04T06:22:02.790 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:22:02.790 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:22:02.802 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1270,"stamp":"2024-03-04T06:22:02.448675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98241025,"num_objects":1108,"num_object_clones":903,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":6017,"num_read_kb":4685290,"num_write":5132,"num_write_kb":1362666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14270,"ondisk_log_size":14270,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":722240,"kb_used_data":499744,"kb_used_omap":0,"kb_used_meta":222464,"kb_avail":376765120,"statfs":{"total":386547056640,"available":385807482880,"internally_reserved":0,"allocated":511737856,"data_stored":608162785,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":227803136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-363104,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":8271,"num_write":17,"num_write_kb":5375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.131711"},"pg_stats":[{"pgid":"3.f","version":"849'784","reported_seq":2893,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.358471+0000","last_change":"2024-03-04T06:22:00.394425+0000","last_active":"2024-03-04T06:22:00.358471+0000","last_peered":"2024-03-04T06:22:00.358471+0000","last_clean":"2024-03-04T06:22:00.358471+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:22:00.358471+0000","last_undegraded":"2024-03-04T06:22:00.358471+0000","last_fullsized":"2024-03-04T06:22:00.358471+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":784,"log_dups_size":0,"ondisk_log_size":784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5059633,"num_objects":69,"num_object_clones":55,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":316,"num_read_kb":251968,"num_write":288,"num_write_kb":75809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"845'757","reported_seq":2896,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.471593+0000","last_change":"2024-03-04T06:21:56.548337+0000","last_active":"2024-03-04T06:21:58.471593+0000","last_peered":"2024-03-04T06:21:58.471593+0000","last_clean":"2024-03-04T06:21:58.471593+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:58.471593+0000","last_undegraded":"2024-03-04T06:21:58.471593+0000","last_fullsized":"2024-03-04T06:21:58.471593+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781905,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":226,"num_read_kb":171792,"num_write":251,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"847'1007","reported_seq":3053,"reported_epoch":850,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:01.365595+0000","last_change":"2024-03-04T06:22:01.365595+0000","last_active":"2024-03-04T06:22:01.365595+0000","last_peered":"2024-03-04T06:22:01.365595+0000","last_clean":"2024-03-04T06:22:01.365595+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:22:01.365595+0000","last_undegraded":"2024-03-04T06:22:01.365595+0000","last_fullsized":"2024-03-04T06:22:01.365595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11144493,"num_objects":85,"num_object_clones":70,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":489,"num_read_kb":390129,"num_write":379,"num_write_kb":103143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"849'879","reported_seq":2885,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.348417+0000","last_change":"2024-03-04T06:22:00.401557+0000","last_active":"2024-03-04T06:22:00.348417+0000","last_peered":"2024-03-04T06:22:00.348417+0000","last_clean":"2024-03-04T06:22:00.348417+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:22:00.348417+0000","last_undegraded":"2024-03-04T06:22:00.348417+0000","last_fullsized":"2024-03-04T06:22:00.348417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":879,"log_dups_size":0,"ondisk_log_size":879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227307,"num_objects":61,"num_object_clones":51,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":301,"num_read_kb":228882,"num_write":285,"num_write_kb":79582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"848'357","reported_seq":2692,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.377437+0000","last_change":"2024-03-04T06:22:01.381565+0000","last_active":"2024-03-04T06:22:01.377437+0000","last_peered":"2024-03-04T06:22:01.377437+0000","last_clean":"2024-03-04T06:22:01.377437+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:22:01.377437+0000","last_undegraded":"2024-03-04T06:22:01.377437+0000","last_fullsized":"2024-03-04T06:22:01.377437+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":100,"num_read_kb":71985,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"849'902","reported_seq":2951,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.359971+0000","last_change":"2024-03-04T06:22:00.402301+0000","last_active":"2024-03-04T06:22:00.359971+0000","last_peered":"2024-03-04T06:22:00.359971+0000","last_clean":"2024-03-04T06:22:00.359971+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:22:00.359971+0000","last_undegraded":"2024-03-04T06:22:00.359971+0000","last_fullsized":"2024-03-04T06:22:00.359971+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":64,"num_object_clones":55,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":393,"num_read_kb":322263,"num_write":310,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2494,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.357523+0000","last_change":"2024-03-04T06:22:00.373338+0000","last_active":"2024-03-04T06:22:00.357523+0000","last_peered":"2024-03-04T06:22:00.357523+0000","last_clean":"2024-03-04T06:22:00.357523+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:22:00.357523+0000","last_undegraded":"2024-03-04T06:22:00.357523+0000","last_fullsized":"2024-03-04T06:22:00.357523+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"850'565","reported_seq":2779,"reported_epoch":850,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:01.368427+0000","last_change":"2024-03-04T06:22:01.368260+0000","last_active":"2024-03-04T06:22:01.368427+0000","last_peered":"2024-03-04T06:22:01.368427+0000","last_clean":"2024-03-04T06:22:01.368427+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:22:01.368427+0000","last_undegraded":"2024-03-04T06:22:01.368427+0000","last_fullsized":"2024-03-04T06:22:01.368427+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916790,"num_objects":39,"num_object_clones":30,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":271,"num_read_kb":244560,"num_write":176,"num_write_kb":47268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2279,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.356227+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:22:01.356227+0000","last_peered":"2024-03-04T06:22:01.356227+0000","last_clean":"2024-03-04T06:22:01.356227+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:22:01.356227+0000","last_undegraded":"2024-03-04T06:22:01.356227+0000","last_fullsized":"2024-03-04T06:22:01.356227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2100,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.392372+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:22:01.392372+0000","last_peered":"2024-03-04T06:22:01.392372+0000","last_clean":"2024-03-04T06:22:01.392372+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:22:01.392372+0000","last_undegraded":"2024-03-04T06:22:01.392372+0000","last_fullsized":"2024-03-04T06:22:01.392372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"850'904","reported_seq":2955,"reported_epoch":850,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:01.384712+0000","last_change":"2024-03-04T06:22:01.384545+0000","last_active":"2024-03-04T06:22:01.384712+0000","last_peered":"2024-03-04T06:22:01.384712+0000","last_clean":"2024-03-04T06:22:01.384712+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:22:01.384712+0000","last_undegraded":"2024-03-04T06:22:01.384712+0000","last_fullsized":"2024-03-04T06:22:01.384712+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":904,"log_dups_size":0,"ondisk_log_size":904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5974191,"num_objects":70,"num_object_clones":56,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":390,"num_read_kb":309493,"num_write":312,"num_write_kb":83652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2095,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.354749+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:22:00.354749+0000","last_peered":"2024-03-04T06:22:00.354749+0000","last_clean":"2024-03-04T06:22:00.354749+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:22:00.354749+0000","last_undegraded":"2024-03-04T06:22:00.354749+0000","last_fullsized":"2024-03-04T06:22:00.354749+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"848'582","reported_seq":2860,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.357907+0000","last_change":"2024-03-04T06:22:01.372275+0000","last_active":"2024-03-04T06:22:01.357907+0000","last_peered":"2024-03-04T06:22:01.357907+0000","last_clean":"2024-03-04T06:22:01.357907+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:22:01.357907+0000","last_undegraded":"2024-03-04T06:22:01.357907+0000","last_fullsized":"2024-03-04T06:22:01.357907+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3210500,"num_objects":48,"num_object_clones":39,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":295,"num_read_kb":241125,"num_write":200,"num_write_kb":56567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2100,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.369563+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:22:01.369563+0000","last_peered":"2024-03-04T06:22:01.369563+0000","last_clean":"2024-03-04T06:22:01.369563+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:22:01.369563+0000","last_undegraded":"2024-03-04T06:22:01.369563+0000","last_fullsized":"2024-03-04T06:22:01.369563+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"848'1546","reported_seq":3409,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.382098+0000","last_change":"2024-03-04T06:22:01.382098+0000","last_active":"2024-03-04T06:22:01.382098+0000","last_peered":"2024-03-04T06:22:01.382098+0000","last_clean":"2024-03-04T06:22:01.382098+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:22:01.382098+0000","last_undegraded":"2024-03-04T06:22:01.382098+0000","last_fullsized":"2024-03-04T06:22:01.382098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14439028,"num_objects":126,"num_object_clones":101,"num_object_copies":378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":737,"num_read_kb":588861,"num_write":588,"num_write_kb":157422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2095,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.347084+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:22:00.347084+0000","last_peered":"2024-03-04T06:22:00.347084+0000","last_clean":"2024-03-04T06:22:00.347084+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:22:00.347084+0000","last_undegraded":"2024-03-04T06:22:00.347084+0000","last_fullsized":"2024-03-04T06:22:00.347084+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"845'1786","reported_seq":3411,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.473254+0000","last_change":"2024-03-04T06:21:56.594592+0000","last_active":"2024-03-04T06:21:58.473254+0000","last_peered":"2024-03-04T06:21:58.473254+0000","last_clean":"2024-03-04T06:21:58.473254+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:58.473254+0000","last_undegraded":"2024-03-04T06:21:58.473254+0000","last_fullsized":"2024-03-04T06:21:58.473254+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1786,"log_dups_size":0,"ondisk_log_size":1786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14590225,"num_objects":137,"num_object_clones":113,"num_object_copies":411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":636,"num_read_kb":472754,"num_write":630,"num_write_kb":169233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"847'843","reported_seq":2956,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.417364+0000","last_change":"2024-03-04T06:22:01.433064+0000","last_active":"2024-03-04T06:22:01.417364+0000","last_peered":"2024-03-04T06:22:01.417364+0000","last_clean":"2024-03-04T06:22:01.417364+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:22:01.417364+0000","last_undegraded":"2024-03-04T06:22:01.417364+0000","last_fullsized":"2024-03-04T06:22:01.417364+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2790325,"num_objects":63,"num_object_clones":51,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":413,"num_read_kb":341590,"num_write":297,"num_write_kb":80300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2171,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.489023+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:58.489023+0000","last_peered":"2024-03-04T06:21:58.489023+0000","last_clean":"2024-03-04T06:21:58.489023+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:58.489023+0000","last_undegraded":"2024-03-04T06:21:58.489023+0000","last_fullsized":"2024-03-04T06:21:58.489023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"845'927","reported_seq":3017,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.482942+0000","last_change":"2024-03-04T06:21:56.575152+0000","last_active":"2024-03-04T06:21:58.482942+0000","last_peered":"2024-03-04T06:21:58.482942+0000","last_clean":"2024-03-04T06:21:58.482942+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:58.482942+0000","last_undegraded":"2024-03-04T06:21:58.482942+0000","last_fullsized":"2024-03-04T06:21:58.482942+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":927,"log_dups_size":0,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5278124,"num_objects":77,"num_object_clones":61,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":355,"num_read_kb":254719,"num_write":329,"num_write_kb":88340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2146,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.377364+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:22:01.377364+0000","last_peered":"2024-03-04T06:22:01.377364+0000","last_clean":"2024-03-04T06:22:01.377364+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:22:01.377364+0000","last_undegraded":"2024-03-04T06:22:01.377364+0000","last_fullsized":"2024-03-04T06:22:01.377364+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"848'1265","reported_seq":3137,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.362769+0000","last_change":"2024-03-04T06:22:01.371963+0000","last_active":"2024-03-04T06:22:01.362769+0000","last_peered":"2024-03-04T06:22:01.362769+0000","last_clean":"2024-03-04T06:22:01.362769+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:22:01.362769+0000","last_undegraded":"2024-03-04T06:22:01.362769+0000","last_fullsized":"2024-03-04T06:22:01.362769+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1265,"log_dups_size":0,"ondisk_log_size":1265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10555819,"num_objects":92,"num_object_clones":78,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":465,"num_read_kb":354560,"num_write":433,"num_write_kb":121118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2171,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.474715+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:58.474715+0000","last_peered":"2024-03-04T06:21:58.474715+0000","last_clean":"2024-03-04T06:21:58.474715+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:58.474715+0000","last_undegraded":"2024-03-04T06:21:58.474715+0000","last_fullsized":"2024-03-04T06:21:58.474715+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"842'1069","reported_seq":3169,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.470036+0000","last_change":"2024-03-04T06:21:56.458442+0000","last_active":"2024-03-04T06:21:58.470036+0000","last_peered":"2024-03-04T06:21:58.470036+0000","last_clean":"2024-03-04T06:21:58.470036+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:58.470036+0000","last_undegraded":"2024-03-04T06:21:58.470036+0000","last_fullsized":"2024-03-04T06:21:58.470036+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1069,"log_dups_size":0,"ondisk_log_size":1069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8976027,"num_objects":89,"num_object_clones":76,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":527,"num_read_kb":440518,"num_write":402,"num_write_kb":110475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2100,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.383702+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:22:01.383702+0000","last_peered":"2024-03-04T06:22:01.383702+0000","last_clean":"2024-03-04T06:22:01.383702+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:22:01.383702+0000","last_undegraded":"2024-03-04T06:22:01.383702+0000","last_fullsized":"2024-03-04T06:22:01.383702+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":97650657,"num_objects":1106,"num_object_clones":903,"num_object_copies":3318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":5914,"num_read_kb":4685199,"num_write":4995,"num_write_kb":1360772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14173,"ondisk_log_size":14173,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":225048,"kb_used_data":162128,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94146792,"statfs":{"total":96636764160,"available":96406315008,"internally_reserved":0,"allocated":166019072,"data_stored":197960727,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":164932,"kb_used_data":111740,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94206908,"statfs":{"total":96636764160,"available":96467873792,"internally_reserved":0,"allocated":114421760,"data_stored":131892123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162436,"kb_used_data":109244,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94209404,"statfs":{"total":96636764160,"available":96470429696,"internally_reserved":0,"allocated":111865856,"data_stored":136273814,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":169824,"kb_used_data":116632,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94202016,"statfs":{"total":96636764160,"available":96462864384,"internally_reserved":0,"allocated":119431168,"data_stored":142036121,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:22:02.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:22:02.949 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:02.946+0000 7f8a6cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:02.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:02.952+0000 7f8a6cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:03.140 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:22:03.141 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:22:03.153 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1271,"stamp":"2024-03-04T06:22:02.886807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98241025,"num_objects":1108,"num_object_clones":903,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":6017,"num_read_kb":4685290,"num_write":5132,"num_write_kb":1362666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14270,"ondisk_log_size":14270,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":722240,"kb_used_data":499744,"kb_used_omap":0,"kb_used_meta":222464,"kb_avail":376765120,"statfs":{"total":386547056640,"available":385807482880,"internally_reserved":0,"allocated":511737856,"data_stored":608162785,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":227803136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-363104,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":8271,"num_write":17,"num_write_kb":5375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000705"},"pg_stats":[{"pgid":"3.f","version":"849'784","reported_seq":2893,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.358471+0000","last_change":"2024-03-04T06:22:00.394425+0000","last_active":"2024-03-04T06:22:00.358471+0000","last_peered":"2024-03-04T06:22:00.358471+0000","last_clean":"2024-03-04T06:22:00.358471+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:22:00.358471+0000","last_undegraded":"2024-03-04T06:22:00.358471+0000","last_fullsized":"2024-03-04T06:22:00.358471+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":784,"log_dups_size":0,"ondisk_log_size":784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5059633,"num_objects":69,"num_object_clones":55,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":316,"num_read_kb":251968,"num_write":288,"num_write_kb":75809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"845'757","reported_seq":2896,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.471593+0000","last_change":"2024-03-04T06:21:56.548337+0000","last_active":"2024-03-04T06:21:58.471593+0000","last_peered":"2024-03-04T06:21:58.471593+0000","last_clean":"2024-03-04T06:21:58.471593+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:58.471593+0000","last_undegraded":"2024-03-04T06:21:58.471593+0000","last_fullsized":"2024-03-04T06:21:58.471593+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781905,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":226,"num_read_kb":171792,"num_write":251,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"847'1007","reported_seq":3053,"reported_epoch":850,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:01.365595+0000","last_change":"2024-03-04T06:22:01.365595+0000","last_active":"2024-03-04T06:22:01.365595+0000","last_peered":"2024-03-04T06:22:01.365595+0000","last_clean":"2024-03-04T06:22:01.365595+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:22:01.365595+0000","last_undegraded":"2024-03-04T06:22:01.365595+0000","last_fullsized":"2024-03-04T06:22:01.365595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11144493,"num_objects":85,"num_object_clones":70,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":489,"num_read_kb":390129,"num_write":379,"num_write_kb":103143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"849'879","reported_seq":2885,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.348417+0000","last_change":"2024-03-04T06:22:00.401557+0000","last_active":"2024-03-04T06:22:00.348417+0000","last_peered":"2024-03-04T06:22:00.348417+0000","last_clean":"2024-03-04T06:22:00.348417+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:22:00.348417+0000","last_undegraded":"2024-03-04T06:22:00.348417+0000","last_fullsized":"2024-03-04T06:22:00.348417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":879,"log_dups_size":0,"ondisk_log_size":879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227307,"num_objects":61,"num_object_clones":51,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":301,"num_read_kb":228882,"num_write":285,"num_write_kb":79582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"848'357","reported_seq":2692,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.377437+0000","last_change":"2024-03-04T06:22:01.381565+0000","last_active":"2024-03-04T06:22:01.377437+0000","last_peered":"2024-03-04T06:22:01.377437+0000","last_clean":"2024-03-04T06:22:01.377437+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:22:01.377437+0000","last_undegraded":"2024-03-04T06:22:01.377437+0000","last_fullsized":"2024-03-04T06:22:01.377437+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":100,"num_read_kb":71985,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"849'902","reported_seq":2951,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.359971+0000","last_change":"2024-03-04T06:22:00.402301+0000","last_active":"2024-03-04T06:22:00.359971+0000","last_peered":"2024-03-04T06:22:00.359971+0000","last_clean":"2024-03-04T06:22:00.359971+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:22:00.359971+0000","last_undegraded":"2024-03-04T06:22:00.359971+0000","last_fullsized":"2024-03-04T06:22:00.359971+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":64,"num_object_clones":55,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":393,"num_read_kb":322263,"num_write":310,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2494,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.357523+0000","last_change":"2024-03-04T06:22:00.373338+0000","last_active":"2024-03-04T06:22:00.357523+0000","last_peered":"2024-03-04T06:22:00.357523+0000","last_clean":"2024-03-04T06:22:00.357523+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:22:00.357523+0000","last_undegraded":"2024-03-04T06:22:00.357523+0000","last_fullsized":"2024-03-04T06:22:00.357523+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"850'565","reported_seq":2779,"reported_epoch":850,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:01.368427+0000","last_change":"2024-03-04T06:22:01.368260+0000","last_active":"2024-03-04T06:22:01.368427+0000","last_peered":"2024-03-04T06:22:01.368427+0000","last_clean":"2024-03-04T06:22:01.368427+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:22:01.368427+0000","last_undegraded":"2024-03-04T06:22:01.368427+0000","last_fullsized":"2024-03-04T06:22:01.368427+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916790,"num_objects":39,"num_object_clones":30,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":271,"num_read_kb":244560,"num_write":176,"num_write_kb":47268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2279,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.356227+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:22:01.356227+0000","last_peered":"2024-03-04T06:22:01.356227+0000","last_clean":"2024-03-04T06:22:01.356227+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:22:01.356227+0000","last_undegraded":"2024-03-04T06:22:01.356227+0000","last_fullsized":"2024-03-04T06:22:01.356227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2100,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.392372+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:22:01.392372+0000","last_peered":"2024-03-04T06:22:01.392372+0000","last_clean":"2024-03-04T06:22:01.392372+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:22:01.392372+0000","last_undegraded":"2024-03-04T06:22:01.392372+0000","last_fullsized":"2024-03-04T06:22:01.392372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"850'904","reported_seq":2955,"reported_epoch":850,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:01.384712+0000","last_change":"2024-03-04T06:22:01.384545+0000","last_active":"2024-03-04T06:22:01.384712+0000","last_peered":"2024-03-04T06:22:01.384712+0000","last_clean":"2024-03-04T06:22:01.384712+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:22:01.384712+0000","last_undegraded":"2024-03-04T06:22:01.384712+0000","last_fullsized":"2024-03-04T06:22:01.384712+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":904,"log_dups_size":0,"ondisk_log_size":904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5974191,"num_objects":70,"num_object_clones":56,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":390,"num_read_kb":309493,"num_write":312,"num_write_kb":83652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2095,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.354749+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:22:00.354749+0000","last_peered":"2024-03-04T06:22:00.354749+0000","last_clean":"2024-03-04T06:22:00.354749+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:22:00.354749+0000","last_undegraded":"2024-03-04T06:22:00.354749+0000","last_fullsized":"2024-03-04T06:22:00.354749+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"848'582","reported_seq":2860,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.357907+0000","last_change":"2024-03-04T06:22:01.372275+0000","last_active":"2024-03-04T06:22:01.357907+0000","last_peered":"2024-03-04T06:22:01.357907+0000","last_clean":"2024-03-04T06:22:01.357907+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:22:01.357907+0000","last_undegraded":"2024-03-04T06:22:01.357907+0000","last_fullsized":"2024-03-04T06:22:01.357907+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3210500,"num_objects":48,"num_object_clones":39,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":295,"num_read_kb":241125,"num_write":200,"num_write_kb":56567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2100,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.369563+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:22:01.369563+0000","last_peered":"2024-03-04T06:22:01.369563+0000","last_clean":"2024-03-04T06:22:01.369563+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:22:01.369563+0000","last_undegraded":"2024-03-04T06:22:01.369563+0000","last_fullsized":"2024-03-04T06:22:01.369563+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"848'1546","reported_seq":3409,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.382098+0000","last_change":"2024-03-04T06:22:01.382098+0000","last_active":"2024-03-04T06:22:01.382098+0000","last_peered":"2024-03-04T06:22:01.382098+0000","last_clean":"2024-03-04T06:22:01.382098+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:22:01.382098+0000","last_undegraded":"2024-03-04T06:22:01.382098+0000","last_fullsized":"2024-03-04T06:22:01.382098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14439028,"num_objects":126,"num_object_clones":101,"num_object_copies":378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":737,"num_read_kb":588861,"num_write":588,"num_write_kb":157422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2095,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.347084+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:22:00.347084+0000","last_peered":"2024-03-04T06:22:00.347084+0000","last_clean":"2024-03-04T06:22:00.347084+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:22:00.347084+0000","last_undegraded":"2024-03-04T06:22:00.347084+0000","last_fullsized":"2024-03-04T06:22:00.347084+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"845'1786","reported_seq":3411,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.473254+0000","last_change":"2024-03-04T06:21:56.594592+0000","last_active":"2024-03-04T06:21:58.473254+0000","last_peered":"2024-03-04T06:21:58.473254+0000","last_clean":"2024-03-04T06:21:58.473254+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:58.473254+0000","last_undegraded":"2024-03-04T06:21:58.473254+0000","last_fullsized":"2024-03-04T06:21:58.473254+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1786,"log_dups_size":0,"ondisk_log_size":1786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14590225,"num_objects":137,"num_object_clones":113,"num_object_copies":411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":636,"num_read_kb":472754,"num_write":630,"num_write_kb":169233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"847'843","reported_seq":2956,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.417364+0000","last_change":"2024-03-04T06:22:01.433064+0000","last_active":"2024-03-04T06:22:01.417364+0000","last_peered":"2024-03-04T06:22:01.417364+0000","last_clean":"2024-03-04T06:22:01.417364+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:22:01.417364+0000","last_undegraded":"2024-03-04T06:22:01.417364+0000","last_fullsized":"2024-03-04T06:22:01.417364+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2790325,"num_objects":63,"num_object_clones":51,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":413,"num_read_kb":341590,"num_write":297,"num_write_kb":80300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2171,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.489023+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:58.489023+0000","last_peered":"2024-03-04T06:21:58.489023+0000","last_clean":"2024-03-04T06:21:58.489023+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:58.489023+0000","last_undegraded":"2024-03-04T06:21:58.489023+0000","last_fullsized":"2024-03-04T06:21:58.489023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"845'927","reported_seq":3017,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.482942+0000","last_change":"2024-03-04T06:21:56.575152+0000","last_active":"2024-03-04T06:21:58.482942+0000","last_peered":"2024-03-04T06:21:58.482942+0000","last_clean":"2024-03-04T06:21:58.482942+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:58.482942+0000","last_undegraded":"2024-03-04T06:21:58.482942+0000","last_fullsized":"2024-03-04T06:21:58.482942+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":927,"log_dups_size":0,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5278124,"num_objects":77,"num_object_clones":61,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":355,"num_read_kb":254719,"num_write":329,"num_write_kb":88340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2146,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.377364+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:22:01.377364+0000","last_peered":"2024-03-04T06:22:01.377364+0000","last_clean":"2024-03-04T06:22:01.377364+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:22:01.377364+0000","last_undegraded":"2024-03-04T06:22:01.377364+0000","last_fullsized":"2024-03-04T06:22:01.377364+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"848'1265","reported_seq":3137,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.362769+0000","last_change":"2024-03-04T06:22:01.371963+0000","last_active":"2024-03-04T06:22:01.362769+0000","last_peered":"2024-03-04T06:22:01.362769+0000","last_clean":"2024-03-04T06:22:01.362769+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:22:01.362769+0000","last_undegraded":"2024-03-04T06:22:01.362769+0000","last_fullsized":"2024-03-04T06:22:01.362769+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1265,"log_dups_size":0,"ondisk_log_size":1265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10555819,"num_objects":92,"num_object_clones":78,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":465,"num_read_kb":354560,"num_write":433,"num_write_kb":121118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2171,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.474715+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:58.474715+0000","last_peered":"2024-03-04T06:21:58.474715+0000","last_clean":"2024-03-04T06:21:58.474715+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:58.474715+0000","last_undegraded":"2024-03-04T06:21:58.474715+0000","last_fullsized":"2024-03-04T06:21:58.474715+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"842'1069","reported_seq":3169,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.470036+0000","last_change":"2024-03-04T06:21:56.458442+0000","last_active":"2024-03-04T06:21:58.470036+0000","last_peered":"2024-03-04T06:21:58.470036+0000","last_clean":"2024-03-04T06:21:58.470036+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:58.470036+0000","last_undegraded":"2024-03-04T06:21:58.470036+0000","last_fullsized":"2024-03-04T06:21:58.470036+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1069,"log_dups_size":0,"ondisk_log_size":1069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8976027,"num_objects":89,"num_object_clones":76,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":527,"num_read_kb":440518,"num_write":402,"num_write_kb":110475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2100,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.383702+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:22:01.383702+0000","last_peered":"2024-03-04T06:22:01.383702+0000","last_clean":"2024-03-04T06:22:01.383702+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:22:01.383702+0000","last_undegraded":"2024-03-04T06:22:01.383702+0000","last_fullsized":"2024-03-04T06:22:01.383702+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":97650657,"num_objects":1106,"num_object_clones":903,"num_object_copies":3318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":5914,"num_read_kb":4685199,"num_write":4995,"num_write_kb":1360772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14173,"ondisk_log_size":14173,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":225048,"kb_used_data":162128,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94146792,"statfs":{"total":96636764160,"available":96406315008,"internally_reserved":0,"allocated":166019072,"data_stored":197960727,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":164932,"kb_used_data":111740,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94206908,"statfs":{"total":96636764160,"available":96467873792,"internally_reserved":0,"allocated":114421760,"data_stored":131892123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162436,"kb_used_data":109244,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94209404,"statfs":{"total":96636764160,"available":96470429696,"internally_reserved":0,"allocated":111865856,"data_stored":136273814,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":169824,"kb_used_data":116632,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94202016,"statfs":{"total":96636764160,"available":96462864384,"internally_reserved":0,"allocated":119431168,"data_stored":142036121,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:22:03.154 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:22:03.154 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:22:03.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:22:03.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:03.258+0000 7fa12479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:03.264+0000 7fa12479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:03.454 INFO:tasks.rados.rados.0.smithi064.stdout:3328: done (2 left) 2024-03-04T06:22:03.454 INFO:tasks.rados.rados.0.smithi064.stdout:3329: done (1 left) 2024-03-04T06:22:03.455 INFO:tasks.rados.rados.0.smithi064.stdout:3330: done (0 left) 2024-03-04T06:22:03.455 INFO:tasks.rados.rados.0.smithi064.stdout:3331: write oid 8 current snap is 423 2024-03-04T06:22:03.455 INFO:tasks.rados.rados.0.smithi064.stdout:3331: seq_num 1512 ranges {612184=414382,1612068=643525,3021918=390266} 2024-03-04T06:22:03.459 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:22:03.460 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:22:03.461 INFO:tasks.rados.rados.0.smithi064.stdout:3331: writing smithi06431743-8 from 612184 to 1026566 tid 1 2024-03-04T06:22:03.467 INFO:tasks.rados.rados.0.smithi064.stdout:3331: writing smithi06431743-8 from 1612068 to 2255593 tid 2 2024-03-04T06:22:03.474 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1272,"stamp":"2024-03-04T06:22:03.451963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":98241025,"num_objects":1108,"num_object_clones":903,"num_object_copies":3324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":6017,"num_read_kb":4685290,"num_write":5132,"num_write_kb":1362666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14270,"ondisk_log_size":14270,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":722240,"kb_used_data":499744,"kb_used_omap":0,"kb_used_meta":222464,"kb_avail":376765120,"statfs":{"total":386547056640,"available":385807482880,"internally_reserved":0,"allocated":511737856,"data_stored":608162785,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":227803136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-363104,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":8271,"num_write":17,"num_write_kb":5375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.124464"},"pg_stats":[{"pgid":"3.f","version":"849'784","reported_seq":2893,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.358471+0000","last_change":"2024-03-04T06:22:00.394425+0000","last_active":"2024-03-04T06:22:00.358471+0000","last_peered":"2024-03-04T06:22:00.358471+0000","last_clean":"2024-03-04T06:22:00.358471+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:22:00.358471+0000","last_undegraded":"2024-03-04T06:22:00.358471+0000","last_fullsized":"2024-03-04T06:22:00.358471+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":784,"log_dups_size":0,"ondisk_log_size":784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5059633,"num_objects":69,"num_object_clones":55,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":316,"num_read_kb":251968,"num_write":288,"num_write_kb":75809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"845'757","reported_seq":2896,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.471593+0000","last_change":"2024-03-04T06:21:56.548337+0000","last_active":"2024-03-04T06:21:58.471593+0000","last_peered":"2024-03-04T06:21:58.471593+0000","last_clean":"2024-03-04T06:21:58.471593+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:21:58.471593+0000","last_undegraded":"2024-03-04T06:21:58.471593+0000","last_fullsized":"2024-03-04T06:21:58.471593+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781905,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":226,"num_read_kb":171792,"num_write":251,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"847'1007","reported_seq":3053,"reported_epoch":850,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:01.365595+0000","last_change":"2024-03-04T06:22:01.365595+0000","last_active":"2024-03-04T06:22:01.365595+0000","last_peered":"2024-03-04T06:22:01.365595+0000","last_clean":"2024-03-04T06:22:01.365595+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:22:01.365595+0000","last_undegraded":"2024-03-04T06:22:01.365595+0000","last_fullsized":"2024-03-04T06:22:01.365595+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11144493,"num_objects":85,"num_object_clones":70,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":489,"num_read_kb":390129,"num_write":379,"num_write_kb":103143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"849'879","reported_seq":2885,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.348417+0000","last_change":"2024-03-04T06:22:00.401557+0000","last_active":"2024-03-04T06:22:00.348417+0000","last_peered":"2024-03-04T06:22:00.348417+0000","last_clean":"2024-03-04T06:22:00.348417+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:22:00.348417+0000","last_undegraded":"2024-03-04T06:22:00.348417+0000","last_fullsized":"2024-03-04T06:22:00.348417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":879,"log_dups_size":0,"ondisk_log_size":879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227307,"num_objects":61,"num_object_clones":51,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":301,"num_read_kb":228882,"num_write":285,"num_write_kb":79582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"848'357","reported_seq":2692,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.377437+0000","last_change":"2024-03-04T06:22:01.381565+0000","last_active":"2024-03-04T06:22:01.377437+0000","last_peered":"2024-03-04T06:22:01.377437+0000","last_clean":"2024-03-04T06:22:01.377437+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:22:01.377437+0000","last_undegraded":"2024-03-04T06:22:01.377437+0000","last_fullsized":"2024-03-04T06:22:01.377437+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":100,"num_read_kb":71985,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"849'902","reported_seq":2951,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.359971+0000","last_change":"2024-03-04T06:22:00.402301+0000","last_active":"2024-03-04T06:22:00.359971+0000","last_peered":"2024-03-04T06:22:00.359971+0000","last_clean":"2024-03-04T06:22:00.359971+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:22:00.359971+0000","last_undegraded":"2024-03-04T06:22:00.359971+0000","last_fullsized":"2024-03-04T06:22:00.359971+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":64,"num_object_clones":55,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":393,"num_read_kb":322263,"num_write":310,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2494,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.357523+0000","last_change":"2024-03-04T06:22:00.373338+0000","last_active":"2024-03-04T06:22:00.357523+0000","last_peered":"2024-03-04T06:22:00.357523+0000","last_clean":"2024-03-04T06:22:00.357523+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:22:00.357523+0000","last_undegraded":"2024-03-04T06:22:00.357523+0000","last_fullsized":"2024-03-04T06:22:00.357523+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"850'565","reported_seq":2779,"reported_epoch":850,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:01.368427+0000","last_change":"2024-03-04T06:22:01.368260+0000","last_active":"2024-03-04T06:22:01.368427+0000","last_peered":"2024-03-04T06:22:01.368427+0000","last_clean":"2024-03-04T06:22:01.368427+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:22:01.368427+0000","last_undegraded":"2024-03-04T06:22:01.368427+0000","last_fullsized":"2024-03-04T06:22:01.368427+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916790,"num_objects":39,"num_object_clones":30,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":271,"num_read_kb":244560,"num_write":176,"num_write_kb":47268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2279,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.356227+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:22:01.356227+0000","last_peered":"2024-03-04T06:22:01.356227+0000","last_clean":"2024-03-04T06:22:01.356227+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:22:01.356227+0000","last_undegraded":"2024-03-04T06:22:01.356227+0000","last_fullsized":"2024-03-04T06:22:01.356227+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2100,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.392372+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:22:01.392372+0000","last_peered":"2024-03-04T06:22:01.392372+0000","last_clean":"2024-03-04T06:22:01.392372+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:22:01.392372+0000","last_undegraded":"2024-03-04T06:22:01.392372+0000","last_fullsized":"2024-03-04T06:22:01.392372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"850'904","reported_seq":2955,"reported_epoch":850,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:01.384712+0000","last_change":"2024-03-04T06:22:01.384545+0000","last_active":"2024-03-04T06:22:01.384712+0000","last_peered":"2024-03-04T06:22:01.384712+0000","last_clean":"2024-03-04T06:22:01.384712+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:22:01.384712+0000","last_undegraded":"2024-03-04T06:22:01.384712+0000","last_fullsized":"2024-03-04T06:22:01.384712+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":904,"log_dups_size":0,"ondisk_log_size":904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5974191,"num_objects":70,"num_object_clones":56,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":390,"num_read_kb":309493,"num_write":312,"num_write_kb":83652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2095,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.354749+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:22:00.354749+0000","last_peered":"2024-03-04T06:22:00.354749+0000","last_clean":"2024-03-04T06:22:00.354749+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:22:00.354749+0000","last_undegraded":"2024-03-04T06:22:00.354749+0000","last_fullsized":"2024-03-04T06:22:00.354749+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"848'582","reported_seq":2860,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.357907+0000","last_change":"2024-03-04T06:22:01.372275+0000","last_active":"2024-03-04T06:22:01.357907+0000","last_peered":"2024-03-04T06:22:01.357907+0000","last_clean":"2024-03-04T06:22:01.357907+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:22:01.357907+0000","last_undegraded":"2024-03-04T06:22:01.357907+0000","last_fullsized":"2024-03-04T06:22:01.357907+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3210500,"num_objects":48,"num_object_clones":39,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":295,"num_read_kb":241125,"num_write":200,"num_write_kb":56567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2100,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.369563+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:22:01.369563+0000","last_peered":"2024-03-04T06:22:01.369563+0000","last_clean":"2024-03-04T06:22:01.369563+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:22:01.369563+0000","last_undegraded":"2024-03-04T06:22:01.369563+0000","last_fullsized":"2024-03-04T06:22:01.369563+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"848'1546","reported_seq":3409,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.382098+0000","last_change":"2024-03-04T06:22:01.382098+0000","last_active":"2024-03-04T06:22:01.382098+0000","last_peered":"2024-03-04T06:22:01.382098+0000","last_clean":"2024-03-04T06:22:01.382098+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:22:01.382098+0000","last_undegraded":"2024-03-04T06:22:01.382098+0000","last_fullsized":"2024-03-04T06:22:01.382098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14439028,"num_objects":126,"num_object_clones":101,"num_object_copies":378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":737,"num_read_kb":588861,"num_write":588,"num_write_kb":157422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2095,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-04T06:22:00.347084+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:22:00.347084+0000","last_peered":"2024-03-04T06:22:00.347084+0000","last_clean":"2024-03-04T06:22:00.347084+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:22:00.347084+0000","last_undegraded":"2024-03-04T06:22:00.347084+0000","last_fullsized":"2024-03-04T06:22:00.347084+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"845'1786","reported_seq":3411,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.473254+0000","last_change":"2024-03-04T06:21:56.594592+0000","last_active":"2024-03-04T06:21:58.473254+0000","last_peered":"2024-03-04T06:21:58.473254+0000","last_clean":"2024-03-04T06:21:58.473254+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:21:58.473254+0000","last_undegraded":"2024-03-04T06:21:58.473254+0000","last_fullsized":"2024-03-04T06:21:58.473254+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1786,"log_dups_size":0,"ondisk_log_size":1786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14590225,"num_objects":137,"num_object_clones":113,"num_object_copies":411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":636,"num_read_kb":472754,"num_write":630,"num_write_kb":169233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"847'843","reported_seq":2956,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.417364+0000","last_change":"2024-03-04T06:22:01.433064+0000","last_active":"2024-03-04T06:22:01.417364+0000","last_peered":"2024-03-04T06:22:01.417364+0000","last_clean":"2024-03-04T06:22:01.417364+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:22:01.417364+0000","last_undegraded":"2024-03-04T06:22:01.417364+0000","last_fullsized":"2024-03-04T06:22:01.417364+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2790325,"num_objects":63,"num_object_clones":51,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":413,"num_read_kb":341590,"num_write":297,"num_write_kb":80300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2171,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.489023+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:21:58.489023+0000","last_peered":"2024-03-04T06:21:58.489023+0000","last_clean":"2024-03-04T06:21:58.489023+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:21:58.489023+0000","last_undegraded":"2024-03-04T06:21:58.489023+0000","last_fullsized":"2024-03-04T06:21:58.489023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"845'927","reported_seq":3017,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.482942+0000","last_change":"2024-03-04T06:21:56.575152+0000","last_active":"2024-03-04T06:21:58.482942+0000","last_peered":"2024-03-04T06:21:58.482942+0000","last_clean":"2024-03-04T06:21:58.482942+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:21:58.482942+0000","last_undegraded":"2024-03-04T06:21:58.482942+0000","last_fullsized":"2024-03-04T06:21:58.482942+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":927,"log_dups_size":0,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5278124,"num_objects":77,"num_object_clones":61,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":355,"num_read_kb":254719,"num_write":329,"num_write_kb":88340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2146,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.377364+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:22:01.377364+0000","last_peered":"2024-03-04T06:22:01.377364+0000","last_clean":"2024-03-04T06:22:01.377364+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:22:01.377364+0000","last_undegraded":"2024-03-04T06:22:01.377364+0000","last_fullsized":"2024-03-04T06:22:01.377364+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"848'1265","reported_seq":3137,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.362769+0000","last_change":"2024-03-04T06:22:01.371963+0000","last_active":"2024-03-04T06:22:01.362769+0000","last_peered":"2024-03-04T06:22:01.362769+0000","last_clean":"2024-03-04T06:22:01.362769+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:22:01.362769+0000","last_undegraded":"2024-03-04T06:22:01.362769+0000","last_fullsized":"2024-03-04T06:22:01.362769+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1265,"log_dups_size":0,"ondisk_log_size":1265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10555819,"num_objects":92,"num_object_clones":78,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":465,"num_read_kb":354560,"num_write":433,"num_write_kb":121118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2171,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.474715+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:21:58.474715+0000","last_peered":"2024-03-04T06:21:58.474715+0000","last_clean":"2024-03-04T06:21:58.474715+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:21:58.474715+0000","last_undegraded":"2024-03-04T06:21:58.474715+0000","last_fullsized":"2024-03-04T06:21:58.474715+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"842'1069","reported_seq":3169,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-03-04T06:21:58.470036+0000","last_change":"2024-03-04T06:21:56.458442+0000","last_active":"2024-03-04T06:21:58.470036+0000","last_peered":"2024-03-04T06:21:58.470036+0000","last_clean":"2024-03-04T06:21:58.470036+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:21:58.470036+0000","last_undegraded":"2024-03-04T06:21:58.470036+0000","last_fullsized":"2024-03-04T06:21:58.470036+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1069,"log_dups_size":0,"ondisk_log_size":1069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8976027,"num_objects":89,"num_object_clones":76,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":527,"num_read_kb":440518,"num_write":402,"num_write_kb":110475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2100,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-04T06:22:01.383702+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:22:01.383702+0000","last_peered":"2024-03-04T06:22:01.383702+0000","last_clean":"2024-03-04T06:22:01.383702+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:22:01.383702+0000","last_undegraded":"2024-03-04T06:22:01.383702+0000","last_fullsized":"2024-03-04T06:22:01.383702+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":97650657,"num_objects":1106,"num_object_clones":903,"num_object_copies":3318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":5914,"num_read_kb":4685199,"num_write":4995,"num_write_kb":1360772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14173,"ondisk_log_size":14173,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":225048,"kb_used_data":162128,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94146792,"statfs":{"total":96636764160,"available":96406315008,"internally_reserved":0,"allocated":166019072,"data_stored":197960727,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":164932,"kb_used_data":111740,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94206908,"statfs":{"total":96636764160,"available":96467873792,"internally_reserved":0,"allocated":114421760,"data_stored":131892123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162436,"kb_used_data":109244,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94209404,"statfs":{"total":96636764160,"available":96470429696,"internally_reserved":0,"allocated":111865856,"data_stored":136273814,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":169824,"kb_used_data":116632,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94202016,"statfs":{"total":96636764160,"available":96462864384,"internally_reserved":0,"allocated":119431168,"data_stored":142036121,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:22:03.476 INFO:tasks.rados.rados.0.smithi064.stdout:3331: writing smithi06431743-8 from 3021918 to 3412184 tid 3 2024-03-04T06:22:03.476 INFO:tasks.rados.rados.0.smithi064.stdout:3332: snap_create 2024-03-04T06:22:03.541 INFO:tasks.rados.rados.0.smithi064.stdout:3331: finishing write tid 1 to smithi06431743-8 2024-03-04T06:22:03.559 INFO:tasks.rados.rados.0.smithi064.stdout:3331: finishing write tid 2 to smithi06431743-8 2024-03-04T06:22:03.571 INFO:tasks.rados.rados.0.smithi064.stdout:3331: finishing write tid 3 to smithi06431743-8 2024-03-04T06:22:03.575 INFO:tasks.rados.rados.0.smithi064.stdout:3331: finishing write tid 4 to smithi06431743-8 2024-03-04T06:22:03.576 INFO:tasks.rados.rados.0.smithi064.stdout:3331: finishing write tid 5 to smithi06431743-8 2024-03-04T06:22:03.576 INFO:tasks.rados.rados.0.smithi064.stdout:3331: oid 8 updating version 0 to 934 2024-03-04T06:22:03.576 INFO:tasks.rados.rados.0.smithi064.stdout:3331: oid 8 updating version 934 to 935 2024-03-04T06:22:03.577 INFO:tasks.rados.rados.0.smithi064.stdout:3331: oid 8 version 935 is already newer than 933 2024-03-04T06:22:03.577 INFO:tasks.rados.rados.0.smithi064.stdout:3331: oid 8 version 935 is already newer than 932 2024-03-04T06:22:03.577 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 935 (ObjNum 1512 snap 423 seq_num 1512) dirty exists 2024-03-04T06:22:03.577 INFO:tasks.rados.rados.0.smithi064.stdout:3331: left oid 8 (ObjNum 1512 snap 423 seq_num 1512) 2024-03-04T06:22:03.982 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:04.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:04.088+0000 7f9fe3a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:04.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:04.092+0000 7f9fe3a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:04.454 INFO:tasks.rados.rados.0.smithi064.stdout:3331: done (1 left) 2024-03-04T06:22:04.454 INFO:tasks.rados.rados.0.smithi064.stdout:3332: done (0 left) 2024-03-04T06:22:04.454 INFO:tasks.rados.rados.0.smithi064.stdout:3333: delete oid 32 current snap is 424 2024-03-04T06:22:04.521 INFO:tasks.rados.rados.0.smithi064.stdout:3333: done (0 left) 2024-03-04T06:22:04.522 INFO:tasks.rados.rados.0.smithi064.stdout:3334: read oid 35 snap -1 2024-03-04T06:22:04.522 INFO:tasks.rados.rados.0.smithi064.stdout:3334: expect (ObjNum 1418 snap 402 seq_num 1418) 2024-03-04T06:22:04.522 INFO:tasks.rados.rados.0.smithi064.stdout:3335: read oid 33 snap -1 2024-03-04T06:22:04.522 INFO:tasks.rados.rados.0.smithi064.stdout:3335: expect (ObjNum 1504 snap 421 seq_num 1504) 2024-03-04T06:22:04.522 INFO:tasks.rados.rados.0.smithi064.stdout:3336: write oid 11 current snap is 424 2024-03-04T06:22:04.522 INFO:tasks.rados.rados.0.smithi064.stdout:3336: seq_num 1514 ranges {645798=681233,1729032=647194,3076764=369034} 2024-03-04T06:22:04.527 INFO:tasks.rados.rados.0.smithi064.stdout:3336: writing smithi06431743-11 from 645798 to 1327031 tid 1 2024-03-04T06:22:04.532 INFO:tasks.rados.rados.0.smithi064.stdout:3336: writing smithi06431743-11 from 1729032 to 2376226 tid 2 2024-03-04T06:22:04.539 INFO:tasks.rados.rados.0.smithi064.stdout:3336: writing smithi06431743-11 from 3076764 to 3445798 tid 3 2024-03-04T06:22:04.539 INFO:tasks.rados.rados.0.smithi064.stdout:3337: read oid 2 snap 422 2024-03-04T06:22:04.539 INFO:tasks.rados.rados.0.smithi064.stdout:3337: expect deleted 2024-03-04T06:22:04.539 INFO:tasks.rados.rados.0.smithi064.stdout:3338: read oid 4 snap -1 2024-03-04T06:22:04.539 INFO:tasks.rados.rados.0.smithi064.stdout:3338: expect (ObjNum 1506 snap 421 seq_num 1506) 2024-03-04T06:22:04.539 INFO:tasks.rados.rados.0.smithi064.stdout:3339: snap_create 2024-03-04T06:22:04.556 INFO:tasks.rados.rados.0.smithi064.stdout:3334: expect (ObjNum 1418 snap 402 seq_num 1418) 2024-03-04T06:22:04.898 INFO:tasks.rados.rados.0.smithi064.stdout:3336: finishing write tid 1 to smithi06431743-11 2024-03-04T06:22:04.898 INFO:tasks.rados.rados.0.smithi064.stdout:3335: expect (ObjNum 1504 snap 421 seq_num 1504) 2024-03-04T06:22:05.019 INFO:tasks.rados.rados.0.smithi064.stdout:3338: expect (ObjNum 1506 snap 421 seq_num 1506) 2024-03-04T06:22:05.185 INFO:tasks.rados.rados.0.smithi064.stdout:3336: finishing write tid 2 to smithi06431743-11 2024-03-04T06:22:05.186 INFO:tasks.rados.rados.0.smithi064.stdout:3336: finishing write tid 3 to smithi06431743-11 2024-03-04T06:22:05.186 INFO:tasks.rados.rados.0.smithi064.stdout:3336: finishing write tid 4 to smithi06431743-11 2024-03-04T06:22:05.186 INFO:tasks.rados.rados.0.smithi064.stdout:3336: finishing write tid 5 to smithi06431743-11 2024-03-04T06:22:05.186 INFO:tasks.rados.rados.0.smithi064.stdout:3336: oid 11 updating version 0 to 1019 2024-03-04T06:22:05.186 INFO:tasks.rados.rados.0.smithi064.stdout:3336: oid 11 version 1019 is already newer than 1017 2024-03-04T06:22:05.186 INFO:tasks.rados.rados.0.smithi064.stdout:3336: oid 11 updating version 1019 to 1020 2024-03-04T06:22:05.186 INFO:tasks.rados.rados.0.smithi064.stdout:3336: oid 11 version 1020 is already newer than 1018 2024-03-04T06:22:05.186 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 1020 (ObjNum 1514 snap 424 seq_num 1514) dirty exists 2024-03-04T06:22:05.186 INFO:tasks.rados.rados.0.smithi064.stdout:3336: left oid 11 (ObjNum 1514 snap 424 seq_num 1514) 2024-03-04T06:22:05.459 INFO:tasks.rados.rados.0.smithi064.stdout:3334: done (5 left) 2024-03-04T06:22:05.459 INFO:tasks.rados.rados.0.smithi064.stdout:3335: done (4 left) 2024-03-04T06:22:05.459 INFO:tasks.rados.rados.0.smithi064.stdout:3336: done (3 left) 2024-03-04T06:22:05.459 INFO:tasks.rados.rados.0.smithi064.stdout:3337: done (2 left) 2024-03-04T06:22:05.459 INFO:tasks.rados.rados.0.smithi064.stdout:3338: done (1 left) 2024-03-04T06:22:05.459 INFO:tasks.rados.rados.0.smithi064.stdout:3339: done (0 left) 2024-03-04T06:22:05.459 INFO:tasks.rados.rados.0.smithi064.stdout:3340: write (excl) oid 29 current snap is 425 2024-03-04T06:22:05.459 INFO:tasks.rados.rados.0.smithi064.stdout:3340: seq_num 1515 ranges {662605=756482,2187514=790852} 2024-03-04T06:22:05.466 INFO:tasks.rados.rados.0.smithi064.stdout:3340: writing smithi06431743-29 from 662605 to 1419087 tid 1 2024-03-04T06:22:05.475 INFO:tasks.rados.rados.0.smithi064.stdout:3340: writing smithi06431743-29 from 2187514 to 2978366 tid 2 2024-03-04T06:22:05.475 INFO:tasks.rados.rados.0.smithi064.stdout:3341: read oid 28 snap -1 2024-03-04T06:22:05.475 INFO:tasks.rados.rados.0.smithi064.stdout:3341: expect (ObjNum 1503 snap 421 seq_num 1503) 2024-03-04T06:22:05.475 INFO:tasks.rados.rados.0.smithi064.stdout:3342: delete oid 14 current snap is 425 2024-03-04T06:22:05.534 INFO:tasks.rados.rados.0.smithi064.stdout:3340: finishing write tid 1 to smithi06431743-29 2024-03-04T06:22:05.557 INFO:tasks.rados.rados.0.smithi064.stdout:3342: done (2 left) 2024-03-04T06:22:05.558 INFO:tasks.rados.rados.0.smithi064.stdout:3343: read oid 31 snap 398 2024-03-04T06:22:05.558 INFO:tasks.rados.rados.0.smithi064.stdout:3343: expect (ObjNum 1349 snap 372 seq_num 1349) 2024-03-04T06:22:05.558 INFO:tasks.rados.rados.0.smithi064.stdout:3344: read oid 42 snap -1 2024-03-04T06:22:05.558 INFO:tasks.rados.rados.0.smithi064.stdout:3344: expect (ObjNum 1450 snap 405 seq_num 1450) 2024-03-04T06:22:05.558 INFO:tasks.rados.rados.0.smithi064.stdout:3345: snap_create 2024-03-04T06:22:05.559 INFO:tasks.rados.rados.0.smithi064.stdout:3341: expect (ObjNum 1503 snap 421 seq_num 1503) 2024-03-04T06:22:05.781 INFO:tasks.rados.rados.0.smithi064.stdout:3340: finishing write tid 2 to smithi06431743-29 2024-03-04T06:22:05.781 INFO:tasks.rados.rados.0.smithi064.stdout:3340: finishing write tid 3 to smithi06431743-29 2024-03-04T06:22:05.781 INFO:tasks.rados.rados.0.smithi064.stdout:3340: finishing write tid 4 to smithi06431743-29 2024-03-04T06:22:05.781 INFO:tasks.rados.rados.0.smithi064.stdout:3340: oid 29 updating version 0 to 1803 2024-03-04T06:22:05.781 INFO:tasks.rados.rados.0.smithi064.stdout:3340: oid 29 version 1803 is already newer than 1801 2024-03-04T06:22:05.781 INFO:tasks.rados.rados.0.smithi064.stdout:3340: oid 29 version 1803 is already newer than 1802 2024-03-04T06:22:05.781 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 1803 (ObjNum 1515 snap 425 seq_num 1515) dirty exists 2024-03-04T06:22:05.781 INFO:tasks.rados.rados.0.smithi064.stdout:3340: left oid 29 (ObjNum 1515 snap 425 seq_num 1515) 2024-03-04T06:22:05.781 INFO:tasks.rados.rados.0.smithi064.stdout:3343: expect (ObjNum 1349 snap 372 seq_num 1349) 2024-03-04T06:22:06.001 INFO:tasks.rados.rados.0.smithi064.stdout:3344: expect (ObjNum 1450 snap 405 seq_num 1450) 2024-03-04T06:22:06.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:06.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:06.266+0000 7f870459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:06.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:06.270+0000 7f870459d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:06.462 INFO:tasks.rados.rados.0.smithi064.stdout:3340: done (4 left) 2024-03-04T06:22:06.462 INFO:tasks.rados.rados.0.smithi064.stdout:3341: done (3 left) 2024-03-04T06:22:06.462 INFO:tasks.rados.rados.0.smithi064.stdout:3343: done (2 left) 2024-03-04T06:22:06.462 INFO:tasks.rados.rados.0.smithi064.stdout:3344: done (1 left) 2024-03-04T06:22:06.462 INFO:tasks.rados.rados.0.smithi064.stdout:3345: done (0 left) 2024-03-04T06:22:06.462 INFO:tasks.rados.rados.0.smithi064.stdout:3346: read oid 11 snap -1 2024-03-04T06:22:06.462 INFO:tasks.rados.rados.0.smithi064.stdout:3346: expect (ObjNum 1514 snap 424 seq_num 1514) 2024-03-04T06:22:06.462 INFO:tasks.rados.rados.0.smithi064.stdout:3347: snap_remove snap 406 2024-03-04T06:22:07.466 INFO:tasks.rados.rados.0.smithi064.stdout:3347: done (1 left) 2024-03-04T06:22:07.466 INFO:tasks.rados.rados.0.smithi064.stdout:3348: write oid 20 current snap is 426 2024-03-04T06:22:07.466 INFO:tasks.rados.rados.0.smithi064.stdout:3348: seq_num 1517 ranges {696219=506980,1904478=794521,3159033=337186} 2024-03-04T06:22:07.472 INFO:tasks.rados.rados.0.smithi064.stdout:3348: writing smithi06431743-20 from 696219 to 1203199 tid 1 2024-03-04T06:22:07.481 INFO:tasks.rados.rados.0.smithi064.stdout:3348: writing smithi06431743-20 from 1904478 to 2698999 tid 2 2024-03-04T06:22:07.485 INFO:tasks.rados.rados.0.smithi064.stdout:3348: writing smithi06431743-20 from 3159033 to 3496219 tid 3 2024-03-04T06:22:07.486 INFO:tasks.rados.rados.0.smithi064.stdout:3349: delete oid 48 current snap is 426 2024-03-04T06:22:07.486 INFO:tasks.rados.rados.0.smithi064.stdout:3346: expect (ObjNum 1514 snap 424 seq_num 1514) 2024-03-04T06:22:07.692 INFO:tasks.rados.rados.0.smithi064.stdout:3348: finishing write tid 1 to smithi06431743-20 2024-03-04T06:22:07.693 INFO:tasks.rados.rados.0.smithi064.stdout:3348: finishing write tid 2 to smithi06431743-20 2024-03-04T06:22:07.693 INFO:tasks.rados.rados.0.smithi064.stdout:3348: finishing write tid 3 to smithi06431743-20 2024-03-04T06:22:07.693 INFO:tasks.rados.rados.0.smithi064.stdout:3346: done (2 left) 2024-03-04T06:22:07.693 INFO:tasks.rados.rados.0.smithi064.stdout:3349: done (1 left) 2024-03-04T06:22:07.693 INFO:tasks.rados.rados.0.smithi064.stdout:3350: read oid 46 snap -1 2024-03-04T06:22:07.693 INFO:tasks.rados.rados.0.smithi064.stdout:3350: expect (ObjNum 1477 snap 414 seq_num 1477) 2024-03-04T06:22:07.693 INFO:tasks.rados.rados.0.smithi064.stdout:3351: read oid 40 snap 423 2024-03-04T06:22:07.693 INFO:tasks.rados.rados.0.smithi064.stdout:3351: expect (ObjNum 1469 snap 412 seq_num 1469) 2024-03-04T06:22:07.693 INFO:tasks.rados.rados.0.smithi064.stdout:3348: finishing write tid 4 to smithi06431743-20 2024-03-04T06:22:07.693 INFO:tasks.rados.rados.0.smithi064.stdout:3348: finishing write tid 5 to smithi06431743-20 2024-03-04T06:22:07.694 INFO:tasks.rados.rados.0.smithi064.stdout:3348: oid 20 updating version 0 to 1273 2024-03-04T06:22:07.694 INFO:tasks.rados.rados.0.smithi064.stdout:3348: oid 20 updating version 1273 to 1274 2024-03-04T06:22:07.694 INFO:tasks.rados.rados.0.smithi064.stdout:3348: oid 20 version 1274 is already newer than 1272 2024-03-04T06:22:07.694 INFO:tasks.rados.rados.0.smithi064.stdout:3348: oid 20 updating version 1274 to 1275 2024-03-04T06:22:07.694 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 1275 (ObjNum 1517 snap 426 seq_num 1517) dirty exists 2024-03-04T06:22:07.694 INFO:tasks.rados.rados.0.smithi064.stdout:3348: left oid 20 (ObjNum 1517 snap 426 seq_num 1517) 2024-03-04T06:22:07.694 INFO:tasks.rados.rados.0.smithi064.stdout:3352: read oid 43 snap 419 2024-03-04T06:22:07.694 INFO:tasks.rados.rados.0.smithi064.stdout:3352: expect deleted 2024-03-04T06:22:07.694 INFO:tasks.rados.rados.0.smithi064.stdout:3348: done (3 left) 2024-03-04T06:22:07.694 INFO:tasks.rados.rados.0.smithi064.stdout:3353: snap_remove snap 414 2024-03-04T06:22:08.320 INFO:tasks.rados.rados.0.smithi064.stdout:3353: done (3 left) 2024-03-04T06:22:08.320 INFO:tasks.rados.rados.0.smithi064.stdout:3354: read oid 38 snap -1 2024-03-04T06:22:08.320 INFO:tasks.rados.rados.0.smithi064.stdout:3354: expect deleted 2024-03-04T06:22:08.320 INFO:tasks.rados.rados.0.smithi064.stdout:3355: write (excl) oid 42 current snap is 426 2024-03-04T06:22:08.321 INFO:tasks.rados.rados.0.smithi064.stdout:3355: seq_num 1519 ranges {729833=657478,1905089=798190,3497526=32307} 2024-03-04T06:22:08.327 INFO:tasks.rados.rados.0.smithi064.stdout:3355: writing smithi06431743-42 from 729833 to 1387311 tid 1 2024-03-04T06:22:08.335 INFO:tasks.rados.rados.0.smithi064.stdout:3355: writing smithi06431743-42 from 1905089 to 2703279 tid 2 2024-03-04T06:22:08.340 INFO:tasks.rados.rados.0.smithi064.stdout:3355: writing smithi06431743-42 from 3497526 to 3529833 tid 3 2024-03-04T06:22:08.340 INFO:tasks.rados.rados.0.smithi064.stdout:3356: snap_create 2024-03-04T06:22:08.340 INFO:tasks.rados.rados.0.smithi064.stdout:3350: expect (ObjNum 1477 snap 414 seq_num 1477) 2024-03-04T06:22:08.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:08.476 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:22:08.476 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:22:08.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:22:08.477 INFO:tasks.rados.rados.0.smithi064.stdout:3351: expect (ObjNum 1469 snap 412 seq_num 1469) 2024-03-04T06:22:08.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:08.481+0000 7f360a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:08.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:08.485+0000 7f360a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:08.580 INFO:tasks.rados.rados.0.smithi064.stdout:3355: finishing write tid 1 to smithi06431743-42 2024-03-04T06:22:08.580 INFO:tasks.rados.rados.0.smithi064.stdout:3355: finishing write tid 2 to smithi06431743-42 2024-03-04T06:22:08.580 INFO:tasks.rados.rados.0.smithi064.stdout:3355: finishing write tid 3 to smithi06431743-42 2024-03-04T06:22:08.580 INFO:tasks.rados.rados.0.smithi064.stdout:3355: finishing write tid 4 to smithi06431743-42 2024-03-04T06:22:08.580 INFO:tasks.rados.rados.0.smithi064.stdout:3355: finishing write tid 5 to smithi06431743-42 2024-03-04T06:22:08.580 INFO:tasks.rados.rados.0.smithi064.stdout:3355: oid 42 updating version 0 to 789 2024-03-04T06:22:08.580 INFO:tasks.rados.rados.0.smithi064.stdout:3355: oid 42 version 789 is already newer than 786 2024-03-04T06:22:08.580 INFO:tasks.rados.rados.0.smithi064.stdout:3355: oid 42 version 789 is already newer than 787 2024-03-04T06:22:08.580 INFO:tasks.rados.rados.0.smithi064.stdout:3355: oid 42 version 789 is already newer than 789 2024-03-04T06:22:08.581 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 789 (ObjNum 1519 snap 426 seq_num 1519) dirty exists 2024-03-04T06:22:08.581 INFO:tasks.rados.rados.0.smithi064.stdout:3355: left oid 42 (ObjNum 1519 snap 426 seq_num 1519) 2024-03-04T06:22:08.591 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:08.587+0000 7ff6b3c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:08.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:08.593+0000 7ff6b3c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:08.786 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:22:08.787 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:22:08.798 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1279,"stamp":"2024-03-04T06:22:08.318437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103007629,"num_objects":1116,"num_object_clones":910,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":170,"num_read":6070,"num_read_kb":4740263,"num_write":5178,"num_write_kb":1375658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14339,"ondisk_log_size":14339,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":749012,"kb_used_data":522356,"kb_used_omap":0,"kb_used_meta":226624,"kb_avail":376738348,"statfs":{"total":386547056640,"available":385780068352,"internally_reserved":0,"allocated":534892544,"data_stored":630328266,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":232062976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4766604,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":53,"num_read_kb":54973,"num_write":46,"num_write_kb":12992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.865860"},"pg_stats":[{"pgid":"3.f","version":"849'784","reported_seq":2904,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.562143+0000","last_change":"2024-03-04T06:22:01.361679+0000","last_active":"2024-03-04T06:22:04.562143+0000","last_peered":"2024-03-04T06:22:04.562143+0000","last_clean":"2024-03-04T06:22:04.562143+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:22:04.562143+0000","last_undegraded":"2024-03-04T06:22:04.562143+0000","last_fullsized":"2024-03-04T06:22:04.562143+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":784,"log_dups_size":0,"ondisk_log_size":784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5059633,"num_objects":69,"num_object_clones":55,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":316,"num_read_kb":251968,"num_write":288,"num_write_kb":75809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"850'761","reported_seq":2910,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.534083+0000","last_change":"2024-03-04T06:22:03.506090+0000","last_active":"2024-03-04T06:22:04.534083+0000","last_peered":"2024-03-04T06:22:04.534083+0000","last_clean":"2024-03-04T06:22:04.534083+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:22:04.534083+0000","last_undegraded":"2024-03-04T06:22:04.534083+0000","last_fullsized":"2024-03-04T06:22:04.534083+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781905,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":226,"num_read_kb":171792,"num_write":251,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"853'1020","reported_seq":3072,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.610899+0000","last_change":"2024-03-04T06:22:02.466759+0000","last_active":"2024-03-04T06:22:04.610899+0000","last_peered":"2024-03-04T06:22:04.610899+0000","last_clean":"2024-03-04T06:22:04.610899+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:22:04.610899+0000","last_undegraded":"2024-03-04T06:22:04.610899+0000","last_fullsized":"2024-03-04T06:22:04.610899+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1020,"log_dups_size":0,"ondisk_log_size":1020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11564668,"num_objects":86,"num_object_clones":71,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":496,"num_read_kb":398287,"num_write":384,"num_write_kb":104803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"849'879","reported_seq":2897,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.559655+0000","last_change":"2024-03-04T06:22:01.352106+0000","last_active":"2024-03-04T06:22:04.559655+0000","last_peered":"2024-03-04T06:22:04.559655+0000","last_clean":"2024-03-04T06:22:04.559655+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:22:04.559655+0000","last_undegraded":"2024-03-04T06:22:04.559655+0000","last_fullsized":"2024-03-04T06:22:04.559655+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":879,"log_dups_size":0,"ondisk_log_size":879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227307,"num_objects":61,"num_object_clones":51,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":301,"num_read_kb":228882,"num_write":285,"num_write_kb":79582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"848'357","reported_seq":2707,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-03-04T06:22:05.576179+0000","last_change":"2024-03-04T06:22:01.381565+0000","last_active":"2024-03-04T06:22:05.576179+0000","last_peered":"2024-03-04T06:22:05.576179+0000","last_clean":"2024-03-04T06:22:05.576179+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:22:05.576179+0000","last_undegraded":"2024-03-04T06:22:05.576179+0000","last_fullsized":"2024-03-04T06:22:05.576179+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":106,"num_read_kb":79818,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"849'902","reported_seq":2962,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.564179+0000","last_change":"2024-03-04T06:22:01.362720+0000","last_active":"2024-03-04T06:22:04.564179+0000","last_peered":"2024-03-04T06:22:04.564179+0000","last_clean":"2024-03-04T06:22:04.564179+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:22:04.564179+0000","last_undegraded":"2024-03-04T06:22:04.564179+0000","last_fullsized":"2024-03-04T06:22:04.564179+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":64,"num_object_clones":55,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":393,"num_read_kb":322263,"num_write":310,"num_write_kb":89653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2505,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.562762+0000","last_change":"2024-03-04T06:22:01.362226+0000","last_active":"2024-03-04T06:22:04.562762+0000","last_peered":"2024-03-04T06:22:04.562762+0000","last_clean":"2024-03-04T06:22:04.562762+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:22:04.562762+0000","last_undegraded":"2024-03-04T06:22:04.562762+0000","last_fullsized":"2024-03-04T06:22:04.562762+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"850'565","reported_seq":2792,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.470166+0000","last_change":"2024-03-04T06:22:02.467760+0000","last_active":"2024-03-04T06:22:04.470166+0000","last_peered":"2024-03-04T06:22:04.470166+0000","last_clean":"2024-03-04T06:22:04.470166+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:22:04.470166+0000","last_undegraded":"2024-03-04T06:22:04.470166+0000","last_fullsized":"2024-03-04T06:22:04.470166+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":565,"log_dups_size":0,"ondisk_log_size":565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916790,"num_objects":39,"num_object_clones":30,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":273,"num_read_kb":250536,"num_write":176,"num_write_kb":47268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2291,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-03-04T06:22:05.493547+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:22:05.493547+0000","last_peered":"2024-03-04T06:22:05.493547+0000","last_clean":"2024-03-04T06:22:05.493547+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:22:05.493547+0000","last_undegraded":"2024-03-04T06:22:05.493547+0000","last_fullsized":"2024-03-04T06:22:05.493547+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2109,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.477786+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:22:04.477786+0000","last_peered":"2024-03-04T06:22:04.477786+0000","last_clean":"2024-03-04T06:22:04.477786+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:22:04.477786+0000","last_undegraded":"2024-03-04T06:22:04.477786+0000","last_fullsized":"2024-03-04T06:22:04.477786+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"850'913","reported_seq":2975,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.477668+0000","last_change":"2024-03-04T06:22:02.475997+0000","last_active":"2024-03-04T06:22:04.477668+0000","last_peered":"2024-03-04T06:22:04.477668+0000","last_clean":"2024-03-04T06:22:04.477668+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:22:04.477668+0000","last_undegraded":"2024-03-04T06:22:04.477668+0000","last_fullsized":"2024-03-04T06:22:04.477668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":913,"log_dups_size":0,"ondisk_log_size":913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":393,"num_read_kb":309495,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2105,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.561869+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:22:04.561869+0000","last_peered":"2024-03-04T06:22:04.561869+0000","last_clean":"2024-03-04T06:22:04.561869+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:22:04.561869+0000","last_undegraded":"2024-03-04T06:22:04.561869+0000","last_fullsized":"2024-03-04T06:22:04.561869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"850'584","reported_seq":2873,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-03-04T06:22:05.491332+0000","last_change":"2024-03-04T06:22:01.372275+0000","last_active":"2024-03-04T06:22:05.491332+0000","last_peered":"2024-03-04T06:22:05.491332+0000","last_clean":"2024-03-04T06:22:05.491332+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:22:05.491332+0000","last_undegraded":"2024-03-04T06:22:05.491332+0000","last_fullsized":"2024-03-04T06:22:05.491332+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":49,"num_object_clones":40,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":296,"num_read_kb":241125,"num_write":201,"num_write_kb":56567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2109,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.463608+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:22:04.463608+0000","last_peered":"2024-03-04T06:22:04.463608+0000","last_clean":"2024-03-04T06:22:04.463608+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:22:04.463608+0000","last_undegraded":"2024-03-04T06:22:04.463608+0000","last_fullsized":"2024-03-04T06:22:04.463608+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"853'1558","reported_seq":3434,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-03-04T06:22:05.546117+0000","last_change":"2024-03-04T06:22:01.382098+0000","last_active":"2024-03-04T06:22:05.546117+0000","last_peered":"2024-03-04T06:22:05.546117+0000","last_clean":"2024-03-04T06:22:05.546117+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:22:05.546117+0000","last_undegraded":"2024-03-04T06:22:05.546117+0000","last_fullsized":"2024-03-04T06:22:05.546117+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1558,"log_dups_size":0,"ondisk_log_size":1558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12421825,"num_objects":128,"num_object_clones":103,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":746,"num_read_kb":596498,"num_write":598,"num_write_kb":159993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2105,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.553958+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:22:04.553958+0000","last_peered":"2024-03-04T06:22:04.553958+0000","last_clean":"2024-03-04T06:22:04.553958+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:22:04.553958+0000","last_undegraded":"2024-03-04T06:22:04.553958+0000","last_fullsized":"2024-03-04T06:22:04.553958+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"850'1799","reported_seq":3432,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.569013+0000","last_change":"2024-03-04T06:22:03.529074+0000","last_active":"2024-03-04T06:22:04.569013+0000","last_peered":"2024-03-04T06:22:04.569013+0000","last_clean":"2024-03-04T06:22:04.569013+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:22:04.569013+0000","last_undegraded":"2024-03-04T06:22:04.569013+0000","last_fullsized":"2024-03-04T06:22:04.569013+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1799,"log_dups_size":0,"ondisk_log_size":1799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14791909,"num_objects":138,"num_object_clones":114,"num_object_copies":414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":644,"num_read_kb":481305,"num_write":635,"num_write_kb":170927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"850'847","reported_seq":2970,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.476052+0000","last_change":"2024-03-04T06:22:01.433064+0000","last_active":"2024-03-04T06:22:04.476052+0000","last_peered":"2024-03-04T06:22:04.476052+0000","last_clean":"2024-03-04T06:22:04.476052+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:22:04.476052+0000","last_undegraded":"2024-03-04T06:22:04.476052+0000","last_fullsized":"2024-03-04T06:22:04.476052+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":847,"log_dups_size":0,"ondisk_log_size":847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168895,"num_objects":63,"num_object_clones":51,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":415,"num_read_kb":341591,"num_write":302,"num_write_kb":81976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2183,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.539252+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:22:04.539252+0000","last_peered":"2024-03-04T06:22:04.539252+0000","last_clean":"2024-03-04T06:22:04.539252+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:22:04.539252+0000","last_undegraded":"2024-03-04T06:22:04.539252+0000","last_fullsized":"2024-03-04T06:22:04.539252+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"852'935","reported_seq":3036,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.544701+0000","last_change":"2024-03-04T06:22:03.479045+0000","last_active":"2024-03-04T06:22:04.544701+0000","last_peered":"2024-03-04T06:22:04.544701+0000","last_clean":"2024-03-04T06:22:04.544701+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:22:04.544701+0000","last_undegraded":"2024-03-04T06:22:04.544701+0000","last_fullsized":"2024-03-04T06:22:04.544701+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8690308,"num_objects":77,"num_object_clones":61,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":356,"num_read_kb":254720,"num_write":334,"num_write_kb":89756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2158,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-03-04T06:22:05.504694+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:22:05.504694+0000","last_peered":"2024-03-04T06:22:05.504694+0000","last_clean":"2024-03-04T06:22:05.504694+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:22:05.504694+0000","last_undegraded":"2024-03-04T06:22:05.504694+0000","last_fullsized":"2024-03-04T06:22:05.504694+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"848'1265","reported_seq":3153,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-03-04T06:22:05.494727+0000","last_change":"2024-03-04T06:22:01.371963+0000","last_active":"2024-03-04T06:22:05.494727+0000","last_peered":"2024-03-04T06:22:05.494727+0000","last_clean":"2024-03-04T06:22:05.494727+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:22:05.494727+0000","last_undegraded":"2024-03-04T06:22:05.494727+0000","last_fullsized":"2024-03-04T06:22:05.494727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1265,"log_dups_size":0,"ondisk_log_size":1265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790721,"num_objects":93,"num_object_clones":79,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":471,"num_read_kb":365274,"num_write":434,"num_write_kb":121118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2183,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.535696+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:22:04.535696+0000","last_peered":"2024-03-04T06:22:04.535696+0000","last_clean":"2024-03-04T06:22:04.535696+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:22:04.535696+0000","last_undegraded":"2024-03-04T06:22:04.535696+0000","last_fullsized":"2024-03-04T06:22:04.535696+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"850'1073","reported_seq":3191,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.567255+0000","last_change":"2024-03-04T06:22:01.419397+0000","last_active":"2024-03-04T06:22:04.567255+0000","last_peered":"2024-03-04T06:22:04.567255+0000","last_clean":"2024-03-04T06:22:04.567255+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:22:04.567255+0000","last_undegraded":"2024-03-04T06:22:04.567255+0000","last_fullsized":"2024-03-04T06:22:04.567255+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1073,"log_dups_size":0,"ondisk_log_size":1073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12253755,"num_objects":90,"num_object_clones":76,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":535,"num_read_kb":446618,"num_write":407,"num_write_kb":111926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2109,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-04T06:22:04.475156+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:22:04.475156+0000","last_peered":"2024-03-04T06:22:04.475156+0000","last_clean":"2024-03-04T06:22:04.475156+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:22:04.475156+0000","last_undegraded":"2024-03-04T06:22:04.475156+0000","last_fullsized":"2024-03-04T06:22:04.475156+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102417261,"num_objects":1114,"num_object_clones":910,"num_object_copies":3342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":170,"num_read":5967,"num_read_kb":4740172,"num_write":5041,"num_write_kb":1373764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14242,"ondisk_log_size":14242,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706524,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":234664,"kb_used_data":171744,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94137176,"statfs":{"total":96636764160,"available":96396468224,"internally_reserved":0,"allocated":175865856,"data_stored":208304333,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706524,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":176208,"kb_used_data":118856,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94195632,"statfs":{"total":96636764160,"available":96456327168,"internally_reserved":0,"allocated":121708544,"data_stored":140417734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706523,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166736,"kb_used_data":113544,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94205104,"statfs":{"total":96636764160,"available":96466026496,"internally_reserved":0,"allocated":116269056,"data_stored":140023668,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706523,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":171404,"kb_used_data":118212,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94200436,"statfs":{"total":96636764160,"available":96461246464,"internally_reserved":0,"allocated":121049088,"data_stored":141582531,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:22:09.322 INFO:tasks.rados.rados.0.smithi064.stdout:3350: done (5 left) 2024-03-04T06:22:09.323 INFO:tasks.rados.rados.0.smithi064.stdout:3351: done (4 left) 2024-03-04T06:22:09.323 INFO:tasks.rados.rados.0.smithi064.stdout:3352: done (3 left) 2024-03-04T06:22:09.323 INFO:tasks.rados.rados.0.smithi064.stdout:3354: done (2 left) 2024-03-04T06:22:09.323 INFO:tasks.rados.rados.0.smithi064.stdout:3355: done (1 left) 2024-03-04T06:22:09.323 INFO:tasks.rados.rados.0.smithi064.stdout:3356: done (0 left) 2024-03-04T06:22:09.323 INFO:tasks.rados.rados.0.smithi064.stdout:3357: delete oid 24 current snap is 427 2024-03-04T06:22:09.369 INFO:tasks.rados.rados.0.smithi064.stdout:3357: done (0 left) 2024-03-04T06:22:09.369 INFO:tasks.rados.rados.0.smithi064.stdout:3358: write (excl) oid 45 current snap is 427 2024-03-04T06:22:09.369 INFO:tasks.rados.rados.0.smithi064.stdout:3358: seq_num 1521 ranges {763447=524329,1738406=401859,2868725=490261} 2024-03-04T06:22:09.376 INFO:tasks.rados.rados.0.smithi064.stdout:3358: writing smithi06431743-45 from 763447 to 1287776 tid 1 2024-03-04T06:22:09.381 INFO:tasks.rados.rados.0.smithi064.stdout:3358: writing smithi06431743-45 from 1738406 to 2140265 tid 2 2024-03-04T06:22:09.388 INFO:tasks.rados.rados.0.smithi064.stdout:3358: writing smithi06431743-45 from 2868725 to 3358986 tid 3 2024-03-04T06:22:09.388 INFO:tasks.rados.rados.0.smithi064.stdout:3359: snap_remove snap 388 2024-03-04T06:22:10.323 INFO:tasks.rados.rados.0.smithi064.stdout:3359: done (1 left) 2024-03-04T06:22:10.323 INFO:tasks.rados.rados.0.smithi064.stdout:3360: snap_remove snap 424 2024-03-04T06:22:10.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:10.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:10.661+0000 7f93dd314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:10.667 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:10.666+0000 7f93dd314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:11.324 INFO:tasks.rados.rados.0.smithi064.stdout:3360: done (1 left) 2024-03-04T06:22:11.325 INFO:tasks.rados.rados.0.smithi064.stdout:3361: write (excl) oid 7 current snap is 427 2024-03-04T06:22:11.325 INFO:tasks.rados.rados.0.smithi064.stdout:3358: finishing write tid 1 to smithi06431743-45 2024-03-04T06:22:11.325 INFO:tasks.rados.rados.0.smithi064.stdout:3361: seq_num 1522 ranges {780254=599578,1796888=661870,2896148=684106} 2024-03-04T06:22:11.332 INFO:tasks.rados.rados.0.smithi064.stdout:3361: writing smithi06431743-7 from 780254 to 1379832 tid 1 2024-03-04T06:22:11.337 INFO:tasks.rados.rados.0.smithi064.stdout:3361: writing smithi06431743-7 from 1796888 to 2458758 tid 2 2024-03-04T06:22:11.344 INFO:tasks.rados.rados.0.smithi064.stdout:3361: writing smithi06431743-7 from 2896148 to 3580254 tid 3 2024-03-04T06:22:11.344 INFO:tasks.rados.rados.0.smithi064.stdout:3362: delete oid 5 current snap is 427 2024-03-04T06:22:11.344 INFO:tasks.rados.rados.0.smithi064.stdout:3358: finishing write tid 2 to smithi06431743-45 2024-03-04T06:22:11.344 INFO:tasks.rados.rados.0.smithi064.stdout:3358: finishing write tid 3 to smithi06431743-45 2024-03-04T06:22:11.345 INFO:tasks.rados.rados.0.smithi064.stdout:3358: finishing write tid 4 to smithi06431743-45 2024-03-04T06:22:11.345 INFO:tasks.rados.rados.0.smithi064.stdout:3358: finishing write tid 5 to smithi06431743-45 2024-03-04T06:22:11.345 INFO:tasks.rados.rados.0.smithi064.stdout:3358: oid 45 updating version 0 to 1031 2024-03-04T06:22:11.345 INFO:tasks.rados.rados.0.smithi064.stdout:3358: oid 45 version 1031 is already newer than 1029 2024-03-04T06:22:11.345 INFO:tasks.rados.rados.0.smithi064.stdout:3358: oid 45 version 1031 is already newer than 1030 2024-03-04T06:22:11.345 INFO:tasks.rados.rados.0.smithi064.stdout:3358: oid 45 version 1031 is already newer than 1028 2024-03-04T06:22:11.345 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 1031 (ObjNum 1521 snap 427 seq_num 1521) dirty exists 2024-03-04T06:22:11.345 INFO:tasks.rados.rados.0.smithi064.stdout:3358: left oid 45 (ObjNum 1521 snap 427 seq_num 1521) 2024-03-04T06:22:11.392 INFO:tasks.rados.rados.0.smithi064.stdout:3361: finishing write tid 1 to smithi06431743-7 2024-03-04T06:22:11.415 INFO:tasks.rados.rados.0.smithi064.stdout:3361: finishing write tid 2 to smithi06431743-7 2024-03-04T06:22:11.445 INFO:tasks.rados.rados.0.smithi064.stdout:3361: finishing write tid 3 to smithi06431743-7 2024-03-04T06:22:11.453 INFO:tasks.rados.rados.0.smithi064.stdout:3361: finishing write tid 4 to smithi06431743-7 2024-03-04T06:22:11.454 INFO:tasks.rados.rados.0.smithi064.stdout:3358: done (2 left) 2024-03-04T06:22:11.455 INFO:tasks.rados.rados.0.smithi064.stdout:3362: done (1 left) 2024-03-04T06:22:11.455 INFO:tasks.rados.rados.0.smithi064.stdout:3363: read oid 23 snap -1 2024-03-04T06:22:11.455 INFO:tasks.rados.rados.0.smithi064.stdout:3363: expect (ObjNum 1508 snap 421 seq_num 1508) 2024-03-04T06:22:11.455 INFO:tasks.rados.rados.0.smithi064.stdout:3364: write oid 44 current snap is 427 2024-03-04T06:22:11.455 INFO:tasks.rados.rados.0.smithi064.stdout:3364: seq_num 1524 ranges {413868=750076,1797499=549186,3118288=495580} 2024-03-04T06:22:11.461 INFO:tasks.rados.rados.0.smithi064.stdout:3364: writing smithi06431743-44 from 413868 to 1163944 tid 1 2024-03-04T06:22:11.467 INFO:tasks.rados.rados.0.smithi064.stdout:3364: writing smithi06431743-44 from 1797499 to 2346685 tid 2 2024-03-04T06:22:11.476 INFO:tasks.rados.rados.0.smithi064.stdout:3364: writing smithi06431743-44 from 3118288 to 3613868 tid 3 2024-03-04T06:22:11.476 INFO:tasks.rados.rados.0.smithi064.stdout:3365: delete oid 1 current snap is 427 2024-03-04T06:22:11.477 INFO:tasks.rados.rados.0.smithi064.stdout:3361: finishing write tid 5 to smithi06431743-7 2024-03-04T06:22:11.477 INFO:tasks.rados.rados.0.smithi064.stdout:3361: oid 7 updating version 0 to 597 2024-03-04T06:22:11.477 INFO:tasks.rados.rados.0.smithi064.stdout:3361: oid 7 version 597 is already newer than 595 2024-03-04T06:22:11.477 INFO:tasks.rados.rados.0.smithi064.stdout:3361: oid 7 version 597 is already newer than 596 2024-03-04T06:22:11.477 INFO:tasks.rados.rados.0.smithi064.stdout:3361: oid 7 updating version 597 to 598 2024-03-04T06:22:11.477 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 598 (ObjNum 1522 snap 427 seq_num 1522) dirty exists 2024-03-04T06:22:11.477 INFO:tasks.rados.rados.0.smithi064.stdout:3361: left oid 7 (ObjNum 1522 snap 427 seq_num 1522) 2024-03-04T06:22:11.483 INFO:tasks.rados.rados.0.smithi064.stdout:3361: done (3 left) 2024-03-04T06:22:11.483 INFO:tasks.rados.rados.0.smithi064.stdout:3365: done (2 left) 2024-03-04T06:22:11.483 INFO:tasks.rados.rados.0.smithi064.stdout:3366: write (excl) oid 33 current snap is 427 2024-03-04T06:22:11.483 INFO:tasks.rados.rados.0.smithi064.stdout:3366: seq_num 1526 ranges {447482=500574,1514463=552855,2489487=462974,3645668=1814} 2024-03-04T06:22:11.489 INFO:tasks.rados.rados.0.smithi064.stdout:3366: writing smithi06431743-33 from 447482 to 948056 tid 1 2024-03-04T06:22:11.498 INFO:tasks.rados.rados.0.smithi064.stdout:3366: writing smithi06431743-33 from 1514463 to 2067318 tid 2 2024-03-04T06:22:11.504 INFO:tasks.rados.rados.0.smithi064.stdout:3366: writing smithi06431743-33 from 2489487 to 2952461 tid 3 2024-03-04T06:22:11.508 INFO:tasks.rados.rados.0.smithi064.stdout:3366: writing smithi06431743-33 from 3645668 to 3647482 tid 4 2024-03-04T06:22:11.508 INFO:tasks.rados.rados.0.smithi064.stdout:3367: write oid 47 current snap is 427 2024-03-04T06:22:11.509 INFO:tasks.rados.rados.0.smithi064.stdout:3367: seq_num 1527 ranges {464289=575823,1456592=412866,2400557=674246,3595554=68735} 2024-03-04T06:22:11.516 INFO:tasks.rados.rados.0.smithi064.stdout:3367: writing smithi06431743-47 from 464289 to 1040112 tid 1 2024-03-04T06:22:11.521 INFO:tasks.rados.rados.0.smithi064.stdout:3367: writing smithi06431743-47 from 1456592 to 1869458 tid 2 2024-03-04T06:22:11.531 INFO:tasks.rados.rados.0.smithi064.stdout:3367: writing smithi06431743-47 from 2400557 to 3074803 tid 3 2024-03-04T06:22:11.534 INFO:tasks.rados.rados.0.smithi064.stdout:3367: writing smithi06431743-47 from 3595554 to 3664289 tid 4 2024-03-04T06:22:11.534 INFO:tasks.rados.rados.0.smithi064.stdout:3368: write (excl) oid 45 current snap is 427 2024-03-04T06:22:11.534 INFO:tasks.rados.rados.0.smithi064.stdout:3368: seq_num 1528 ranges {481096=651072,1915074=556524,3227980=453116} 2024-03-04T06:22:11.540 INFO:tasks.rados.rados.0.smithi064.stdout:3368: writing smithi06431743-45 from 481096 to 1132168 tid 1 2024-03-04T06:22:11.550 INFO:tasks.rados.rados.0.smithi064.stdout:3368: writing smithi06431743-45 from 1915074 to 2471598 tid 2 2024-03-04T06:22:11.558 INFO:tasks.rados.rados.0.smithi064.stdout:3368: writing smithi06431743-45 from 3227980 to 3681096 tid 3 2024-03-04T06:22:11.558 INFO:tasks.rados.rados.0.smithi064.stdout:3369: read oid 37 snap 421 2024-03-04T06:22:11.558 INFO:tasks.rados.rados.0.smithi064.stdout:3369: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:22:11.559 INFO:tasks.rados.rados.0.smithi064.stdout:3370: snap_remove snap 384 2024-03-04T06:22:12.329 INFO:tasks.rados.rados.0.smithi064.stdout:3370: done (6 left) 2024-03-04T06:22:12.330 INFO:tasks.rados.rados.0.smithi064.stdout:3371: snap_create 2024-03-04T06:22:12.330 INFO:tasks.rados.rados.0.smithi064.stdout:3363: expect (ObjNum 1508 snap 421 seq_num 1508) 2024-03-04T06:22:12.514 INFO:tasks.rados.rados.0.smithi064.stdout:3366: finishing write tid 1 to smithi06431743-33 2024-03-04T06:22:12.514 INFO:tasks.rados.rados.0.smithi064.stdout:3366: finishing write tid 2 to smithi06431743-33 2024-03-04T06:22:12.514 INFO:tasks.rados.rados.0.smithi064.stdout:3368: finishing write tid 1 to smithi06431743-45 2024-03-04T06:22:12.514 INFO:tasks.rados.rados.0.smithi064.stdout:3369: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:22:12.655 INFO:tasks.rados.rados.0.smithi064.stdout:3367: finishing write tid 1 to smithi06431743-47 2024-03-04T06:22:12.655 INFO:tasks.rados.rados.0.smithi064.stdout:3366: finishing write tid 3 to smithi06431743-33 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3364: finishing write tid 1 to smithi06431743-44 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3368: finishing write tid 2 to smithi06431743-45 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3367: finishing write tid 2 to smithi06431743-47 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3366: finishing write tid 4 to smithi06431743-33 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3364: finishing write tid 2 to smithi06431743-44 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3367: finishing write tid 3 to smithi06431743-47 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3368: finishing write tid 3 to smithi06431743-45 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3367: finishing write tid 4 to smithi06431743-47 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3366: finishing write tid 5 to smithi06431743-33 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3368: finishing write tid 4 to smithi06431743-45 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3364: finishing write tid 3 to smithi06431743-44 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3364: finishing write tid 4 to smithi06431743-44 2024-03-04T06:22:12.656 INFO:tasks.rados.rados.0.smithi064.stdout:3366: finishing write tid 6 to smithi06431743-33 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3366: oid 33 updating version 0 to 1081 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3366: oid 33 updating version 1081 to 1082 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3366: oid 33 updating version 1082 to 1085 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3366: oid 33 version 1085 is already newer than 1084 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3366: oid 33 version 1085 is already newer than 1083 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 1085 (ObjNum 1526 snap 427 seq_num 1526) dirty exists 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3366: left oid 33 (ObjNum 1526 snap 427 seq_num 1526) 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3368: finishing write tid 5 to smithi06431743-45 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3368: oid 45 updating version 0 to 1038 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3368: oid 45 version 1038 is already newer than 1037 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3368: oid 45 version 1038 is already newer than 1036 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3368: oid 45 updating version 1038 to 1039 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 1039 (ObjNum 1528 snap 427 seq_num 1528) dirty exists 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3368: left oid 45 (ObjNum 1528 snap 427 seq_num 1528) 2024-03-04T06:22:12.657 INFO:tasks.rados.rados.0.smithi064.stdout:3364: finishing write tid 5 to smithi06431743-44 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3364: oid 44 updating version 0 to 908 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3364: oid 44 updating version 908 to 910 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3364: oid 44 version 910 is already newer than 910 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3364: oid 44 version 910 is already newer than 907 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 910 (ObjNum 1524 snap 427 seq_num 1524) dirty exists 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3364: left oid 44 (ObjNum 1524 snap 427 seq_num 1524) 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3367: finishing write tid 5 to smithi06431743-47 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3367: finishing write tid 6 to smithi06431743-47 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3367: oid 47 updating version 0 to 895 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3367: oid 47 updating version 895 to 897 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3367: oid 47 updating version 897 to 898 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3367: oid 47 version 898 is already newer than 897 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3367: oid 47 version 898 is already newer than 894 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 898 (ObjNum 1527 snap 427 seq_num 1527) dirty exists 2024-03-04T06:22:12.658 INFO:tasks.rados.rados.0.smithi064.stdout:3367: left oid 47 (ObjNum 1527 snap 427 seq_num 1527) 2024-03-04T06:22:12.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:12.841 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:12.839+0000 7fddff554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:12.844 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:12.843+0000 7fddff554640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:13.335 INFO:tasks.rados.rados.0.smithi064.stdout:3363: done (6 left) 2024-03-04T06:22:13.335 INFO:tasks.rados.rados.0.smithi064.stdout:3364: done (5 left) 2024-03-04T06:22:13.335 INFO:tasks.rados.rados.0.smithi064.stdout:3366: done (4 left) 2024-03-04T06:22:13.335 INFO:tasks.rados.rados.0.smithi064.stdout:3367: done (3 left) 2024-03-04T06:22:13.335 INFO:tasks.rados.rados.0.smithi064.stdout:3368: done (2 left) 2024-03-04T06:22:13.335 INFO:tasks.rados.rados.0.smithi064.stdout:3369: done (1 left) 2024-03-04T06:22:13.335 INFO:tasks.rados.rados.0.smithi064.stdout:3371: done (0 left) 2024-03-04T06:22:13.335 INFO:tasks.rados.rados.0.smithi064.stdout:3372: snap_create 2024-03-04T06:22:13.801 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:22:13.801 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:22:13.802 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:22:13.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:22:13.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:13.910+0000 7fd50682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:13.917 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:13.917+0000 7fd50682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:13.930 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:13.931 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:13.933 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:13.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:22:14.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:14.037+0000 7f037b91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:14.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:14.044+0000 7f037b91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:14.057 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:14.057 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:14.057 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:14.057 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:14.057 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:14.057 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:14.057 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:14.057 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:14.057 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:14.057 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:14.058 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:14.058 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:14.060 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:14.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:22:14.168 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:14.163+0000 7f5ee219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:14.171 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:14.170+0000 7f5ee219d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:14.184 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:14.184 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:14.184 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:14.184 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:14.184 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:14.185 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:14.185 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:14.185 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:14.185 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:14.185 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:14.185 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:14.185 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:14.187 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:14.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:22:14.296 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:14.291+0000 7fd19a62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:14.299 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:14.298+0000 7fd19a62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:14.312 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:14.312 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:14.312 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:14.312 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:14.313 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:14.313 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:14.313 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:14.313 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:14.313 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:14.313 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:14.313 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:14.313 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:14.315 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:14.315 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:14.338 INFO:tasks.rados.rados.0.smithi064.stdout:3372: done (0 left) 2024-03-04T06:22:14.339 INFO:tasks.rados.rados.0.smithi064.stdout:3373: read oid 10 snap -1 2024-03-04T06:22:14.339 INFO:tasks.rados.rados.0.smithi064.stdout:3373: expect deleted 2024-03-04T06:22:14.339 INFO:tasks.rados.rados.0.smithi064.stdout:3374: read oid 15 snap 426 2024-03-04T06:22:14.339 INFO:tasks.rados.rados.0.smithi064.stdout:3374: expect (ObjNum 1510 snap 421 seq_num 1510) 2024-03-04T06:22:14.339 INFO:tasks.rados.rados.0.smithi064.stdout:3375: write (excl) oid 25 current snap is 429 2024-03-04T06:22:14.339 INFO:tasks.rados.rados.0.smithi064.stdout:3375: seq_num 1529 ranges {497903=442674,1689909=416535,2571756=696790} 2024-03-04T06:22:14.343 INFO:tasks.rados.rados.0.smithi064.stdout:3375: writing smithi06431743-25 from 497903 to 940577 tid 1 2024-03-04T06:22:14.349 INFO:tasks.rados.rados.0.smithi064.stdout:3375: writing smithi06431743-25 from 1689909 to 2106444 tid 2 2024-03-04T06:22:14.356 INFO:tasks.rados.rados.0.smithi064.stdout:3375: writing smithi06431743-25 from 2571756 to 3268546 tid 3 2024-03-04T06:22:14.356 INFO:tasks.rados.rados.0.smithi064.stdout:3376: read oid 45 snap 422 2024-03-04T06:22:14.356 INFO:tasks.rados.rados.0.smithi064.stdout:3376: expect (ObjNum 1502 snap 420 seq_num 1502) 2024-03-04T06:22:14.356 INFO:tasks.rados.rados.0.smithi064.stdout:3377: snap_remove snap 419 2024-03-04T06:22:14.910 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:15.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:15.015+0000 7f1c66d54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:15.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:15.019+0000 7f1c66d54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:15.349 INFO:tasks.rados.rados.0.smithi064.stdout:3377: done (4 left) 2024-03-04T06:22:15.349 INFO:tasks.rados.rados.0.smithi064.stdout:3378: read oid 41 snap -1 2024-03-04T06:22:15.349 INFO:tasks.rados.rados.0.smithi064.stdout:3378: expect (ObjNum 1505 snap 421 seq_num 1505) 2024-03-04T06:22:15.349 INFO:tasks.rados.rados.0.smithi064.stdout:3379: read oid 42 snap 404 2024-03-04T06:22:15.350 INFO:tasks.rados.rados.0.smithi064.stdout:3379: expect deleted 2024-03-04T06:22:15.350 INFO:tasks.rados.rados.0.smithi064.stdout:3380: snap_create 2024-03-04T06:22:15.350 INFO:tasks.rados.rados.0.smithi064.stdout:3374: expect (ObjNum 1510 snap 421 seq_num 1510) 2024-03-04T06:22:15.485 INFO:tasks.rados.rados.0.smithi064.stdout:3375: finishing write tid 1 to smithi06431743-25 2024-03-04T06:22:15.485 INFO:tasks.rados.rados.0.smithi064.stdout:3375: finishing write tid 2 to smithi06431743-25 2024-03-04T06:22:15.485 INFO:tasks.rados.rados.0.smithi064.stdout:3375: finishing write tid 3 to smithi06431743-25 2024-03-04T06:22:15.485 INFO:tasks.rados.rados.0.smithi064.stdout:3375: finishing write tid 4 to smithi06431743-25 2024-03-04T06:22:15.485 INFO:tasks.rados.rados.0.smithi064.stdout:3375: finishing write tid 5 to smithi06431743-25 2024-03-04T06:22:15.485 INFO:tasks.rados.rados.0.smithi064.stdout:3375: oid 25 updating version 0 to 1050 2024-03-04T06:22:15.485 INFO:tasks.rados.rados.0.smithi064.stdout:3375: oid 25 version 1050 is already newer than 1048 2024-03-04T06:22:15.485 INFO:tasks.rados.rados.0.smithi064.stdout:3375: oid 25 version 1050 is already newer than 1047 2024-03-04T06:22:15.485 INFO:tasks.rados.rados.0.smithi064.stdout:3375: oid 25 version 1050 is already newer than 1049 2024-03-04T06:22:15.486 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 1050 (ObjNum 1529 snap 429 seq_num 1529) dirty exists 2024-03-04T06:22:15.486 INFO:tasks.rados.rados.0.smithi064.stdout:3375: left oid 25 (ObjNum 1529 snap 429 seq_num 1529) 2024-03-04T06:22:15.486 INFO:tasks.rados.rados.0.smithi064.stdout:3376: expect (ObjNum 1502 snap 420 seq_num 1502) 2024-03-04T06:22:15.645 INFO:tasks.rados.rados.0.smithi064.stdout:3378: expect (ObjNum 1505 snap 421 seq_num 1505) 2024-03-04T06:22:16.346 INFO:tasks.rados.rados.0.smithi064.stdout:3373: done (6 left) 2024-03-04T06:22:16.346 INFO:tasks.rados.rados.0.smithi064.stdout:3374: done (5 left) 2024-03-04T06:22:16.346 INFO:tasks.rados.rados.0.smithi064.stdout:3375: done (4 left) 2024-03-04T06:22:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:3376: done (3 left) 2024-03-04T06:22:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:3378: done (2 left) 2024-03-04T06:22:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:3379: done (1 left) 2024-03-04T06:22:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:3380: done (0 left) 2024-03-04T06:22:16.347 INFO:tasks.rados.rados.0.smithi064.stdout:3381: snap_remove snap 417 2024-03-04T06:22:17.088 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:17.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:17.201+0000 7fe0b2424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:17.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:17.206+0000 7fe0b2424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:17.346 INFO:tasks.rados.rados.0.smithi064.stdout:3381: done (0 left) 2024-03-04T06:22:17.346 INFO:tasks.rados.rados.0.smithi064.stdout:3382: snap_create 2024-03-04T06:22:18.321 INFO:tasks.rados.rados.0.smithi064.stdout:3382: done (0 left) 2024-03-04T06:22:18.322 INFO:tasks.rados.rados.0.smithi064.stdout:3383: delete oid 40 current snap is 431 2024-03-04T06:22:18.358 INFO:tasks.rados.rados.0.smithi064.stdout:3383: done (0 left) 2024-03-04T06:22:18.358 INFO:tasks.rados.rados.0.smithi064.stdout:3384: delete oid 21 current snap is 431 2024-03-04T06:22:18.396 INFO:tasks.rados.rados.0.smithi064.stdout:3384: done (0 left) 2024-03-04T06:22:18.397 INFO:tasks.rados.rados.0.smithi064.stdout:3385: snap_remove snap 425 2024-03-04T06:22:19.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:19.316 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:22:19.316 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:22:19.316 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:22:19.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:22:19.364 INFO:tasks.rados.rados.0.smithi064.stdout:3385: done (0 left) 2024-03-04T06:22:19.364 INFO:tasks.rados.rados.0.smithi064.stdout:3386: snap_remove snap 421 2024-03-04T06:22:19.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:19.386+0000 7f5c9306d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:19.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:19.391+0000 7f5c9306d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:19.434 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:19.429+0000 7f1fd986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:19.437 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:19.437+0000 7f1fd986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:19.450 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:19.451 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:19.453 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:19.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:22:19.562 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:19.558+0000 7f4286806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:19.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:19.565+0000 7f4286806640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:19.577 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:19.577 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:19.581 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:19.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:22:19.688 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:19.684+0000 7f205239d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:19.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:19.691+0000 7f205239d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:19.704 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:19.705 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:19.705 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:19.705 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:19.705 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:19.705 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:19.705 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:19.705 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:19.705 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:19.705 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:19.705 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:19.705 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:19.708 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:19.708 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:22:19.816 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:19.812+0000 7fc9b0cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:19.819 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:19.819+0000 7fc9b0cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:19.832 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:19.833 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:19.833 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:19.833 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:19.833 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:19.833 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:19.833 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:19.833 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:19.833 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:19.833 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:19.833 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:19.833 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:19.835 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:19.835 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:20.367 INFO:tasks.rados.rados.0.smithi064.stdout:3386: done (0 left) 2024-03-04T06:22:20.367 INFO:tasks.rados.rados.0.smithi064.stdout:3387: write (excl) oid 1 current snap is 431 2024-03-04T06:22:20.367 INFO:tasks.rados.rados.0.smithi064.stdout:3387: seq_num 1532 ranges {548324=668421,1749002=563862,2937672=646959} 2024-03-04T06:22:20.374 INFO:tasks.rados.rados.0.smithi064.stdout:3387: writing smithi06431743-1 from 548324 to 1216745 tid 1 2024-03-04T06:22:20.381 INFO:tasks.rados.rados.0.smithi064.stdout:3387: writing smithi06431743-1 from 1749002 to 2312864 tid 2 2024-03-04T06:22:20.389 INFO:tasks.rados.rados.0.smithi064.stdout:3387: writing smithi06431743-1 from 2937672 to 3584631 tid 3 2024-03-04T06:22:20.389 INFO:tasks.rados.rados.0.smithi064.stdout:3388: write (excl) oid 19 current snap is 431 2024-03-04T06:22:20.389 INFO:tasks.rados.rados.0.smithi064.stdout:3388: seq_num 1533 ranges {565131=743670,1807484=423873,2965095=458231} 2024-03-04T06:22:20.398 INFO:tasks.rados.rados.0.smithi064.stdout:3388: writing smithi06431743-19 from 565131 to 1308801 tid 1 2024-03-04T06:22:20.405 INFO:tasks.rados.rados.0.smithi064.stdout:3388: writing smithi06431743-19 from 1807484 to 2231357 tid 2 2024-03-04T06:22:20.414 INFO:tasks.rados.rados.0.smithi064.stdout:3388: writing smithi06431743-19 from 2965095 to 3423326 tid 3 2024-03-04T06:22:20.414 INFO:tasks.rados.rados.0.smithi064.stdout:3389: read oid 4 snap -1 2024-03-04T06:22:20.414 INFO:tasks.rados.rados.0.smithi064.stdout:3389: expect (ObjNum 1506 snap 421 seq_num 1506) 2024-03-04T06:22:20.414 INFO:tasks.rados.rados.0.smithi064.stdout:3390: write oid 13 current snap is 431 2024-03-04T06:22:20.414 INFO:tasks.rados.rados.0.smithi064.stdout:3390: seq_num 1534 ranges {581938=418919,1465966=567531,2592518=669503} 2024-03-04T06:22:20.419 INFO:tasks.rados.rados.0.smithi064.stdout:3390: writing smithi06431743-13 from 581938 to 1000857 tid 1 2024-03-04T06:22:20.427 INFO:tasks.rados.rados.0.smithi064.stdout:3390: writing smithi06431743-13 from 1465966 to 2033497 tid 2 2024-03-04T06:22:20.434 INFO:tasks.rados.rados.0.smithi064.stdout:3390: writing smithi06431743-13 from 2592518 to 3262021 tid 3 2024-03-04T06:22:20.434 INFO:tasks.rados.rados.0.smithi064.stdout:3391: write oid 6 current snap is 431 2024-03-04T06:22:20.434 INFO:tasks.rados.rados.0.smithi064.stdout:3391: seq_num 1535 ranges {598745=494168,1808095=427542,2903588=597128} 2024-03-04T06:22:20.440 INFO:tasks.rados.rados.0.smithi064.stdout:3391: writing smithi06431743-6 from 598745 to 1092913 tid 1 2024-03-04T06:22:20.448 INFO:tasks.rados.rados.0.smithi064.stdout:3391: writing smithi06431743-6 from 1808095 to 2235637 tid 2 2024-03-04T06:22:20.457 INFO:tasks.rados.rados.0.smithi064.stdout:3391: writing smithi06431743-6 from 2903588 to 3500716 tid 3 2024-03-04T06:22:20.457 INFO:tasks.rados.rados.0.smithi064.stdout:3392: write (excl) oid 2 current snap is 431 2024-03-04T06:22:20.457 INFO:tasks.rados.rados.0.smithi064.stdout:3392: seq_num 1536 ranges {615552=685770,1982930=571200,3047364=408400} 2024-03-04T06:22:20.467 INFO:tasks.rados.rados.0.smithi064.stdout:3392: writing smithi06431743-2 from 615552 to 1301322 tid 1 2024-03-04T06:22:20.474 INFO:tasks.rados.rados.0.smithi064.stdout:3392: writing smithi06431743-2 from 1982930 to 2554130 tid 2 2024-03-04T06:22:20.480 INFO:tasks.rados.rados.0.smithi064.stdout:3392: writing smithi06431743-2 from 3047364 to 3455764 tid 3 2024-03-04T06:22:20.480 INFO:tasks.rados.rados.0.smithi064.stdout:3393: read oid 12 snap 404 2024-03-04T06:22:20.480 INFO:tasks.rados.rados.0.smithi064.stdout:3393: expect (ObjNum 1390 snap 392 seq_num 1390) 2024-03-04T06:22:20.480 INFO:tasks.rados.rados.0.smithi064.stdout:3394: snap_create 2024-03-04T06:22:20.482 INFO:tasks.rados.rados.0.smithi064.stdout:3388: finishing write tid 1 to smithi06431743-19 2024-03-04T06:22:20.502 INFO:tasks.rados.rados.0.smithi064.stdout:3392: finishing write tid 1 to smithi06431743-2 2024-03-04T06:22:20.507 INFO:tasks.rados.rados.0.smithi064.stdout:3387: finishing write tid 1 to smithi06431743-1 2024-03-04T06:22:20.512 INFO:tasks.rados.rados.0.smithi064.stdout:3388: finishing write tid 2 to smithi06431743-19 2024-03-04T06:22:20.515 INFO:tasks.rados.rados.0.smithi064.stdout:3391: finishing write tid 1 to smithi06431743-6 2024-03-04T06:22:20.534 INFO:tasks.rados.rados.0.smithi064.stdout:3392: finishing write tid 2 to smithi06431743-2 2024-03-04T06:22:20.535 INFO:tasks.rados.rados.0.smithi064.stdout:3387: finishing write tid 2 to smithi06431743-1 2024-03-04T06:22:20.544 INFO:tasks.rados.rados.0.smithi064.stdout:3387: finishing write tid 3 to smithi06431743-1 2024-03-04T06:22:20.549 INFO:tasks.rados.rados.0.smithi064.stdout:3392: finishing write tid 3 to smithi06431743-2 2024-03-04T06:22:20.552 INFO:tasks.rados.rados.0.smithi064.stdout:3388: finishing write tid 3 to smithi06431743-19 2024-03-04T06:22:20.553 INFO:tasks.rados.rados.0.smithi064.stdout:3391: finishing write tid 2 to smithi06431743-6 2024-03-04T06:22:20.554 INFO:tasks.rados.rados.0.smithi064.stdout:3392: finishing write tid 4 to smithi06431743-2 2024-03-04T06:22:20.557 INFO:tasks.rados.rados.0.smithi064.stdout:3392: finishing write tid 5 to smithi06431743-2 2024-03-04T06:22:20.557 INFO:tasks.rados.rados.0.smithi064.stdout:3392: oid 2 updating version 0 to 910 2024-03-04T06:22:20.557 INFO:tasks.rados.rados.0.smithi064.stdout:3392: oid 2 version 910 is already newer than 908 2024-03-04T06:22:20.557 INFO:tasks.rados.rados.0.smithi064.stdout:3392: oid 2 version 910 is already newer than 907 2024-03-04T06:22:20.557 INFO:tasks.rados.rados.0.smithi064.stdout:3392: oid 2 version 910 is already newer than 910 2024-03-04T06:22:20.557 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 910 (ObjNum 1536 snap 431 seq_num 1536) dirty exists 2024-03-04T06:22:20.557 INFO:tasks.rados.rados.0.smithi064.stdout:3392: left oid 2 (ObjNum 1536 snap 431 seq_num 1536) 2024-03-04T06:22:20.559 INFO:tasks.rados.rados.0.smithi064.stdout:3387: finishing write tid 4 to smithi06431743-1 2024-03-04T06:22:20.561 INFO:tasks.rados.rados.0.smithi064.stdout:3387: finishing write tid 5 to smithi06431743-1 2024-03-04T06:22:20.561 INFO:tasks.rados.rados.0.smithi064.stdout:3387: oid 1 updating version 0 to 1599 2024-03-04T06:22:20.561 INFO:tasks.rados.rados.0.smithi064.stdout:3387: oid 1 version 1599 is already newer than 1598 2024-03-04T06:22:20.561 INFO:tasks.rados.rados.0.smithi064.stdout:3387: oid 1 version 1599 is already newer than 1596 2024-03-04T06:22:20.562 INFO:tasks.rados.rados.0.smithi064.stdout:3387: oid 1 version 1599 is already newer than 1598 2024-03-04T06:22:20.562 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 1599 (ObjNum 1532 snap 431 seq_num 1532) dirty exists 2024-03-04T06:22:20.562 INFO:tasks.rados.rados.0.smithi064.stdout:3387: left oid 1 (ObjNum 1532 snap 431 seq_num 1532) 2024-03-04T06:22:20.566 INFO:tasks.rados.rados.0.smithi064.stdout:3388: finishing write tid 4 to smithi06431743-19 2024-03-04T06:22:20.570 INFO:tasks.rados.rados.0.smithi064.stdout:3388: finishing write tid 5 to smithi06431743-19 2024-03-04T06:22:20.570 INFO:tasks.rados.rados.0.smithi064.stdout:3388: oid 19 updating version 0 to 1850 2024-03-04T06:22:20.570 INFO:tasks.rados.rados.0.smithi064.stdout:3388: oid 19 updating version 1850 to 1851 2024-03-04T06:22:20.570 INFO:tasks.rados.rados.0.smithi064.stdout:3388: oid 19 updating version 1851 to 1852 2024-03-04T06:22:20.570 INFO:tasks.rados.rados.0.smithi064.stdout:3388: oid 19 version 1852 is already newer than 1849 2024-03-04T06:22:20.570 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 1852 (ObjNum 1533 snap 431 seq_num 1533) dirty exists 2024-03-04T06:22:20.570 INFO:tasks.rados.rados.0.smithi064.stdout:3388: left oid 19 (ObjNum 1533 snap 431 seq_num 1533) 2024-03-04T06:22:20.572 INFO:tasks.rados.rados.0.smithi064.stdout:3391: finishing write tid 3 to smithi06431743-6 2024-03-04T06:22:20.580 INFO:tasks.rados.rados.0.smithi064.stdout:3393: expect (ObjNum 1390 snap 392 seq_num 1390) 2024-03-04T06:22:20.744 INFO:tasks.rados.rados.0.smithi064.stdout:3389: expect (ObjNum 1506 snap 421 seq_num 1506) 2024-03-04T06:22:20.910 INFO:tasks.rados.rados.0.smithi064.stdout:3391: finishing write tid 4 to smithi06431743-6 2024-03-04T06:22:20.910 INFO:tasks.rados.rados.0.smithi064.stdout:3391: finishing write tid 5 to smithi06431743-6 2024-03-04T06:22:20.910 INFO:tasks.rados.rados.0.smithi064.stdout:3391: oid 6 updating version 0 to 861 2024-03-04T06:22:20.910 INFO:tasks.rados.rados.0.smithi064.stdout:3391: oid 6 version 861 is already newer than 860 2024-03-04T06:22:20.911 INFO:tasks.rados.rados.0.smithi064.stdout:3391: oid 6 version 861 is already newer than 859 2024-03-04T06:22:20.911 INFO:tasks.rados.rados.0.smithi064.stdout:3391: oid 6 version 861 is already newer than 858 2024-03-04T06:22:20.911 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 861 (ObjNum 1535 snap 431 seq_num 1535) dirty exists 2024-03-04T06:22:20.911 INFO:tasks.rados.rados.0.smithi064.stdout:3391: left oid 6 (ObjNum 1535 snap 431 seq_num 1535) 2024-03-04T06:22:20.911 INFO:tasks.rados.rados.0.smithi064.stdout:3390: finishing write tid 1 to smithi06431743-13 2024-03-04T06:22:20.911 INFO:tasks.rados.rados.0.smithi064.stdout:3390: finishing write tid 2 to smithi06431743-13 2024-03-04T06:22:20.911 INFO:tasks.rados.rados.0.smithi064.stdout:3390: finishing write tid 3 to smithi06431743-13 2024-03-04T06:22:20.911 INFO:tasks.rados.rados.0.smithi064.stdout:3390: finishing write tid 4 to smithi06431743-13 2024-03-04T06:22:20.911 INFO:tasks.rados.rados.0.smithi064.stdout:3390: finishing write tid 5 to smithi06431743-13 2024-03-04T06:22:20.911 INFO:tasks.rados.rados.0.smithi064.stdout:3390: oid 13 updating version 0 to 1856 2024-03-04T06:22:20.912 INFO:tasks.rados.rados.0.smithi064.stdout:3390: oid 13 version 1856 is already newer than 1854 2024-03-04T06:22:20.912 INFO:tasks.rados.rados.0.smithi064.stdout:3390: oid 13 version 1856 is already newer than 1853 2024-03-04T06:22:20.912 INFO:tasks.rados.rados.0.smithi064.stdout:3390: oid 13 version 1856 is already newer than 1856 2024-03-04T06:22:20.912 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1856 (ObjNum 1534 snap 431 seq_num 1534) dirty exists 2024-03-04T06:22:20.912 INFO:tasks.rados.rados.0.smithi064.stdout:3390: left oid 13 (ObjNum 1534 snap 431 seq_num 1534) 2024-03-04T06:22:21.387 INFO:tasks.rados.rados.0.smithi064.stdout:3387: done (7 left) 2024-03-04T06:22:21.387 INFO:tasks.rados.rados.0.smithi064.stdout:3388: done (6 left) 2024-03-04T06:22:21.387 INFO:tasks.rados.rados.0.smithi064.stdout:3389: done (5 left) 2024-03-04T06:22:21.387 INFO:tasks.rados.rados.0.smithi064.stdout:3390: done (4 left) 2024-03-04T06:22:21.387 INFO:tasks.rados.rados.0.smithi064.stdout:3391: done (3 left) 2024-03-04T06:22:21.387 INFO:tasks.rados.rados.0.smithi064.stdout:3392: done (2 left) 2024-03-04T06:22:21.387 INFO:tasks.rados.rados.0.smithi064.stdout:3393: done (1 left) 2024-03-04T06:22:21.387 INFO:tasks.rados.rados.0.smithi064.stdout:3394: done (0 left) 2024-03-04T06:22:21.388 INFO:tasks.rados.rados.0.smithi064.stdout:3395: write (excl) oid 28 current snap is 432 2024-03-04T06:22:21.388 INFO:tasks.rados.rados.0.smithi064.stdout:3395: seq_num 1537 ranges {632359=761019,2041412=714858,3358434=473925} 2024-03-04T06:22:21.394 INFO:tasks.rados.rados.0.smithi064.stdout:3395: writing smithi06431743-28 from 632359 to 1393378 tid 1 2024-03-04T06:22:21.401 INFO:tasks.rados.rados.0.smithi064.stdout:3395: writing smithi06431743-28 from 2041412 to 2756270 tid 2 2024-03-04T06:22:21.407 INFO:tasks.rados.rados.0.smithi064.stdout:3395: writing smithi06431743-28 from 3358434 to 3832359 tid 3 2024-03-04T06:22:21.407 INFO:tasks.rados.rados.0.smithi064.stdout:3396: snap_create 2024-03-04T06:22:21.456 INFO:tasks.rados.rados.0.smithi064.stdout:3395: finishing write tid 1 to smithi06431743-28 2024-03-04T06:22:21.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:21.475 INFO:tasks.rados.rados.0.smithi064.stdout:3395: finishing write tid 2 to smithi06431743-28 2024-03-04T06:22:21.484 INFO:tasks.rados.rados.0.smithi064.stdout:3395: finishing write tid 3 to smithi06431743-28 2024-03-04T06:22:21.492 INFO:tasks.rados.rados.0.smithi064.stdout:3395: finishing write tid 4 to smithi06431743-28 2024-03-04T06:22:21.493 INFO:tasks.rados.rados.0.smithi064.stdout:3395: finishing write tid 5 to smithi06431743-28 2024-03-04T06:22:21.493 INFO:tasks.rados.rados.0.smithi064.stdout:3395: oid 28 updating version 0 to 1603 2024-03-04T06:22:21.493 INFO:tasks.rados.rados.0.smithi064.stdout:3395: oid 28 updating version 1603 to 1604 2024-03-04T06:22:21.493 INFO:tasks.rados.rados.0.smithi064.stdout:3395: oid 28 version 1604 is already newer than 1603 2024-03-04T06:22:21.493 INFO:tasks.rados.rados.0.smithi064.stdout:3395: oid 28 version 1604 is already newer than 1601 2024-03-04T06:22:21.494 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 1604 (ObjNum 1537 snap 432 seq_num 1537) dirty exists 2024-03-04T06:22:21.494 INFO:tasks.rados.rados.0.smithi064.stdout:3395: left oid 28 (ObjNum 1537 snap 432 seq_num 1537) 2024-03-04T06:22:21.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:21.575+0000 7f68270dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:21.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:21.579+0000 7f68270dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:22.386 INFO:tasks.rados.rados.0.smithi064.stdout:3395: done (1 left) 2024-03-04T06:22:22.386 INFO:tasks.rados.rados.0.smithi064.stdout:3396: done (0 left) 2024-03-04T06:22:22.386 INFO:tasks.rados.rados.0.smithi064.stdout:3397: snap_create 2024-03-04T06:22:23.389 INFO:tasks.rados.rados.0.smithi064.stdout:3397: done (0 left) 2024-03-04T06:22:23.389 INFO:tasks.rados.rados.0.smithi064.stdout:3398: write oid 43 current snap is 434 2024-03-04T06:22:23.390 INFO:tasks.rados.rados.0.smithi064.stdout:3398: seq_num 1538 ranges {649166=436268,1699894=574869,2702210=430944} 2024-03-04T06:22:23.395 INFO:tasks.rados.rados.0.smithi064.stdout:3398: writing smithi06431743-43 from 649166 to 1085434 tid 1 2024-03-04T06:22:23.402 INFO:tasks.rados.rados.0.smithi064.stdout:3398: writing smithi06431743-43 from 1699894 to 2274763 tid 2 2024-03-04T06:22:23.407 INFO:tasks.rados.rados.0.smithi064.stdout:3398: writing smithi06431743-43 from 2702210 to 3133154 tid 3 2024-03-04T06:22:23.407 INFO:tasks.rados.rados.0.smithi064.stdout:3399: write oid 6 current snap is 434 2024-03-04T06:22:23.407 INFO:tasks.rados.rados.0.smithi064.stdout:3399: seq_num 1539 ranges {665973=511517,1642023=718527,2896927=642216} 2024-03-04T06:22:23.414 INFO:tasks.rados.rados.0.smithi064.stdout:3399: writing smithi06431743-6 from 665973 to 1177490 tid 1 2024-03-04T06:22:23.422 INFO:tasks.rados.rados.0.smithi064.stdout:3399: writing smithi06431743-6 from 1642023 to 2360550 tid 2 2024-03-04T06:22:23.431 INFO:tasks.rados.rados.0.smithi064.stdout:3399: writing smithi06431743-6 from 2896927 to 3539143 tid 3 2024-03-04T06:22:23.431 INFO:tasks.rados.rados.0.smithi064.stdout:3400: snap_create 2024-03-04T06:22:23.449 INFO:tasks.rados.rados.0.smithi064.stdout:3398: finishing write tid 1 to smithi06431743-43 2024-03-04T06:22:23.475 INFO:tasks.rados.rados.0.smithi064.stdout:3399: finishing write tid 1 to smithi06431743-6 2024-03-04T06:22:23.478 INFO:tasks.rados.rados.0.smithi064.stdout:3398: finishing write tid 2 to smithi06431743-43 2024-03-04T06:22:23.493 INFO:tasks.rados.rados.0.smithi064.stdout:3399: finishing write tid 2 to smithi06431743-6 2024-03-04T06:22:23.493 INFO:tasks.rados.rados.0.smithi064.stdout:3398: finishing write tid 3 to smithi06431743-43 2024-03-04T06:22:23.502 INFO:tasks.rados.rados.0.smithi064.stdout:3398: finishing write tid 4 to smithi06431743-43 2024-03-04T06:22:23.503 INFO:tasks.rados.rados.0.smithi064.stdout:3398: finishing write tid 5 to smithi06431743-43 2024-03-04T06:22:23.503 INFO:tasks.rados.rados.0.smithi064.stdout:3398: oid 43 updating version 0 to 809 2024-03-04T06:22:23.504 INFO:tasks.rados.rados.0.smithi064.stdout:3398: oid 43 version 809 is already newer than 806 2024-03-04T06:22:23.504 INFO:tasks.rados.rados.0.smithi064.stdout:3398: oid 43 version 809 is already newer than 807 2024-03-04T06:22:23.504 INFO:tasks.rados.rados.0.smithi064.stdout:3398: oid 43 version 809 is already newer than 808 2024-03-04T06:22:23.504 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 809 (ObjNum 1538 snap 434 seq_num 1538) dirty exists 2024-03-04T06:22:23.504 INFO:tasks.rados.rados.0.smithi064.stdout:3398: left oid 43 (ObjNum 1538 snap 434 seq_num 1538) 2024-03-04T06:22:23.505 INFO:tasks.rados.rados.0.smithi064.stdout:3399: finishing write tid 3 to smithi06431743-6 2024-03-04T06:22:23.511 INFO:tasks.rados.rados.0.smithi064.stdout:3399: finishing write tid 4 to smithi06431743-6 2024-03-04T06:22:23.511 INFO:tasks.rados.rados.0.smithi064.stdout:3399: finishing write tid 5 to smithi06431743-6 2024-03-04T06:22:23.511 INFO:tasks.rados.rados.0.smithi064.stdout:3399: oid 6 updating version 0 to 863 2024-03-04T06:22:23.511 INFO:tasks.rados.rados.0.smithi064.stdout:3399: oid 6 updating version 863 to 865 2024-03-04T06:22:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:3399: oid 6 version 865 is already newer than 865 2024-03-04T06:22:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:3399: oid 6 updating version 865 to 866 2024-03-04T06:22:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 866 (ObjNum 1539 snap 434 seq_num 1539) dirty exists 2024-03-04T06:22:23.512 INFO:tasks.rados.rados.0.smithi064.stdout:3399: left oid 6 (ObjNum 1539 snap 434 seq_num 1539) 2024-03-04T06:22:23.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:23.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:23.750+0000 7fec438ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:23.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:23.754+0000 7fec438ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:24.406 INFO:tasks.rados.rados.0.smithi064.stdout:3398: done (2 left) 2024-03-04T06:22:24.406 INFO:tasks.rados.rados.0.smithi064.stdout:3399: done (1 left) 2024-03-04T06:22:24.407 INFO:tasks.rados.rados.0.smithi064.stdout:3400: done (0 left) 2024-03-04T06:22:24.407 INFO:tasks.rados.rados.0.smithi064.stdout:3401: write oid 46 current snap is 435 2024-03-04T06:22:24.407 INFO:tasks.rados.rados.0.smithi064.stdout:3401: seq_num 1540 ranges {682780=586766,1700505=578538,3040703=569841} 2024-03-04T06:22:24.413 INFO:tasks.rados.rados.0.smithi064.stdout:3401: writing smithi06431743-46 from 682780 to 1269546 tid 1 2024-03-04T06:22:24.418 INFO:tasks.rados.rados.0.smithi064.stdout:3401: writing smithi06431743-46 from 1700505 to 2279043 tid 2 2024-03-04T06:22:24.426 INFO:tasks.rados.rados.0.smithi064.stdout:3401: writing smithi06431743-46 from 3040703 to 3610544 tid 3 2024-03-04T06:22:24.426 INFO:tasks.rados.rados.0.smithi064.stdout:3402: read oid 3 snap -1 2024-03-04T06:22:24.426 INFO:tasks.rados.rados.0.smithi064.stdout:3402: expect (ObjNum 1452 snap 406 seq_num 1452) 2024-03-04T06:22:24.426 INFO:tasks.rados.rados.0.smithi064.stdout:3403: snap_remove snap 399 2024-03-04T06:22:24.836 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:22:24.836 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:22:24.952 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:24.947+0000 7f154319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:24.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:24.955+0000 7f154319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:25.408 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:22:25.415 INFO:tasks.rados.rados.0.smithi064.stdout:3403: done (2 left) 2024-03-04T06:22:25.415 INFO:tasks.rados.rados.0.smithi064.stdout:3404: snap_create 2024-03-04T06:22:25.415 INFO:tasks.rados.rados.0.smithi064.stdout:3402: expect (ObjNum 1452 snap 406 seq_num 1452) 2024-03-04T06:22:25.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:22:25.545 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:25.539+0000 7fc5d791d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:25.548 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:25.547+0000 7fc5d791d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:25.557 INFO:tasks.rados.rados.0.smithi064.stdout:3401: finishing write tid 1 to smithi06431743-46 2024-03-04T06:22:25.558 INFO:tasks.rados.rados.0.smithi064.stdout:3401: finishing write tid 2 to smithi06431743-46 2024-03-04T06:22:25.558 INFO:tasks.rados.rados.0.smithi064.stdout:3401: finishing write tid 3 to smithi06431743-46 2024-03-04T06:22:25.558 INFO:tasks.rados.rados.0.smithi064.stdout:3401: finishing write tid 4 to smithi06431743-46 2024-03-04T06:22:25.558 INFO:tasks.rados.rados.0.smithi064.stdout:3401: finishing write tid 5 to smithi06431743-46 2024-03-04T06:22:25.558 INFO:tasks.rados.rados.0.smithi064.stdout:3401: oid 46 updating version 0 to 1607 2024-03-04T06:22:25.558 INFO:tasks.rados.rados.0.smithi064.stdout:3401: oid 46 updating version 1607 to 1609 2024-03-04T06:22:25.558 INFO:tasks.rados.rados.0.smithi064.stdout:3401: oid 46 version 1609 is already newer than 1607 2024-03-04T06:22:25.558 INFO:tasks.rados.rados.0.smithi064.stdout:3401: oid 46 version 1609 is already newer than 1608 2024-03-04T06:22:25.558 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 1609 (ObjNum 1540 snap 435 seq_num 1540) dirty exists 2024-03-04T06:22:25.558 INFO:tasks.rados.rados.0.smithi064.stdout:3401: left oid 46 (ObjNum 1540 snap 435 seq_num 1540) 2024-03-04T06:22:25.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:25.945 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:25.943+0000 7fadddd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:25.948 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:25.948+0000 7fadddd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:26.409 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:22:26.414 INFO:tasks.rados.rados.0.smithi064.stdout:3401: done (2 left) 2024-03-04T06:22:26.414 INFO:tasks.rados.rados.0.smithi064.stdout:3402: done (1 left) 2024-03-04T06:22:26.415 INFO:tasks.rados.rados.0.smithi064.stdout:3404: done (0 left) 2024-03-04T06:22:26.415 INFO:tasks.rados.rados.0.smithi064.stdout:3405: read oid 18 snap -1 2024-03-04T06:22:26.415 INFO:tasks.rados.rados.0.smithi064.stdout:3405: expect deleted 2024-03-04T06:22:26.415 INFO:tasks.rados.rados.0.smithi064.stdout:3406: read oid 26 snap -1 2024-03-04T06:22:26.415 INFO:tasks.rados.rados.0.smithi064.stdout:3406: expect (ObjNum 1498 snap 418 seq_num 1498) 2024-03-04T06:22:26.415 INFO:tasks.rados.rados.0.smithi064.stdout:3407: read oid 36 snap -1 2024-03-04T06:22:26.415 INFO:tasks.rados.rados.0.smithi064.stdout:3407: expect (ObjNum 1442 snap 404 seq_num 1442) 2024-03-04T06:22:26.415 INFO:tasks.rados.rados.0.smithi064.stdout:3408: snap_create 2024-03-04T06:22:26.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:22:26.464 INFO:tasks.rados.rados.0.smithi064.stdout:3407: expect (ObjNum 1442 snap 404 seq_num 1442) 2024-03-04T06:22:26.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:26.541+0000 7f6c5291d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:26.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:26.549+0000 7f6c5291d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:26.591 INFO:tasks.rados.rados.0.smithi064.stdout:3406: expect (ObjNum 1498 snap 418 seq_num 1498) 2024-03-04T06:22:27.410 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:22:27.413 INFO:tasks.rados.rados.0.smithi064.stdout:3405: done (3 left) 2024-03-04T06:22:27.413 INFO:tasks.rados.rados.0.smithi064.stdout:3406: done (2 left) 2024-03-04T06:22:27.413 INFO:tasks.rados.rados.0.smithi064.stdout:3407: done (1 left) 2024-03-04T06:22:27.413 INFO:tasks.rados.rados.0.smithi064.stdout:3408: done (0 left) 2024-03-04T06:22:27.413 INFO:tasks.rados.rados.0.smithi064.stdout:3409: write (excl) oid 36 current snap is 437 2024-03-04T06:22:27.413 INFO:tasks.rados.rados.0.smithi064.stdout:3409: seq_num 1541 ranges {699587=662015,2158987=722196,3351773=547814} 2024-03-04T06:22:27.420 INFO:tasks.rados.rados.0.smithi064.stdout:3409: writing smithi06431743-36 from 699587 to 1361602 tid 1 2024-03-04T06:22:27.429 INFO:tasks.rados.rados.0.smithi064.stdout:3409: writing smithi06431743-36 from 2158987 to 2881183 tid 2 2024-03-04T06:22:27.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:22:27.437 INFO:tasks.rados.rados.0.smithi064.stdout:3409: writing smithi06431743-36 from 3351773 to 3899587 tid 3 2024-03-04T06:22:27.437 INFO:tasks.rados.rados.0.smithi064.stdout:3410: read oid 7 snap 431 2024-03-04T06:22:27.437 INFO:tasks.rados.rados.0.smithi064.stdout:3410: expect (ObjNum 1522 snap 427 seq_num 1522) 2024-03-04T06:22:27.437 INFO:tasks.rados.rados.0.smithi064.stdout:3411: write (excl) oid 24 current snap is 437 2024-03-04T06:22:27.437 INFO:tasks.rados.rados.0.smithi064.stdout:3411: seq_num 1542 ranges {716394=737264,2217469=582207,3495549=420845} 2024-03-04T06:22:27.446 INFO:tasks.rados.rados.0.smithi064.stdout:3411: writing smithi06431743-24 from 716394 to 1453658 tid 1 2024-03-04T06:22:27.455 INFO:tasks.rados.rados.0.smithi064.stdout:3411: writing smithi06431743-24 from 2217469 to 2799676 tid 2 2024-03-04T06:22:27.461 INFO:tasks.rados.rados.0.smithi064.stdout:3411: writing smithi06431743-24 from 3495549 to 3916394 tid 3 2024-03-04T06:22:27.462 INFO:tasks.rados.rados.0.smithi064.stdout:3412: write oid 32 current snap is 437 2024-03-04T06:22:27.462 INFO:tasks.rados.rados.0.smithi064.stdout:3412: seq_num 1543 ranges {733201=412513,1759598=725865,3006619=403657} 2024-03-04T06:22:27.468 INFO:tasks.rados.rados.0.smithi064.stdout:3412: writing smithi06431743-32 from 733201 to 1145714 tid 1 2024-03-04T06:22:27.475 INFO:tasks.rados.rados.0.smithi064.stdout:3412: writing smithi06431743-32 from 1759598 to 2485463 tid 2 2024-03-04T06:22:27.480 INFO:tasks.rados.rados.0.smithi064.stdout:3412: writing smithi06431743-32 from 3006619 to 3410276 tid 3 2024-03-04T06:22:27.480 INFO:tasks.rados.rados.0.smithi064.stdout:3413: write (excl) oid 49 current snap is 437 2024-03-04T06:22:27.480 INFO:tasks.rados.rados.0.smithi064.stdout:3413: seq_num 1544 ranges {750008=604115,1934433=585876,3150395=731282} 2024-03-04T06:22:27.487 INFO:tasks.rados.rados.0.smithi064.stdout:3413: writing smithi06431743-49 from 750008 to 1354123 tid 1 2024-03-04T06:22:27.497 INFO:tasks.rados.rados.0.smithi064.stdout:3413: writing smithi06431743-49 from 1934433 to 2520309 tid 2 2024-03-04T06:22:27.506 INFO:tasks.rados.rados.0.smithi064.stdout:3413: writing smithi06431743-49 from 3150395 to 3881677 tid 3 2024-03-04T06:22:27.507 INFO:tasks.rados.rados.0.smithi064.stdout:3414: read oid 23 snap -1 2024-03-04T06:22:27.507 INFO:tasks.rados.rados.0.smithi064.stdout:3414: expect (ObjNum 1508 snap 421 seq_num 1508) 2024-03-04T06:22:27.507 INFO:tasks.rados.rados.0.smithi064.stdout:3415: snap_create 2024-03-04T06:22:27.507 INFO:tasks.rados.rados.0.smithi064.stdout:3409: finishing write tid 1 to smithi06431743-36 2024-03-04T06:22:27.538 INFO:tasks.rados.rados.0.smithi064.stdout:3409: finishing write tid 2 to smithi06431743-36 2024-03-04T06:22:27.544 INFO:tasks.rados.rados.0.smithi064.stdout:3410: expect (ObjNum 1522 snap 427 seq_num 1522) 2024-03-04T06:22:27.585 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:27.579+0000 7fe96a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:27.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:27.588+0000 7fe96a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:27.690 INFO:tasks.rados.rados.0.smithi064.stdout:3412: finishing write tid 1 to smithi06431743-32 2024-03-04T06:22:27.690 INFO:tasks.rados.rados.0.smithi064.stdout:3411: finishing write tid 1 to smithi06431743-24 2024-03-04T06:22:27.690 INFO:tasks.rados.rados.0.smithi064.stdout:3413: finishing write tid 1 to smithi06431743-49 2024-03-04T06:22:27.690 INFO:tasks.rados.rados.0.smithi064.stdout:3414: expect (ObjNum 1508 snap 421 seq_num 1508) 2024-03-04T06:22:27.860 INFO:tasks.rados.rados.0.smithi064.stdout:3409: finishing write tid 3 to smithi06431743-36 2024-03-04T06:22:27.861 INFO:tasks.rados.rados.0.smithi064.stdout:3409: finishing write tid 4 to smithi06431743-36 2024-03-04T06:22:27.861 INFO:tasks.rados.rados.0.smithi064.stdout:3409: finishing write tid 5 to smithi06431743-36 2024-03-04T06:22:27.861 INFO:tasks.rados.rados.0.smithi064.stdout:3409: oid 36 updating version 0 to 959 2024-03-04T06:22:27.861 INFO:tasks.rados.rados.0.smithi064.stdout:3409: oid 36 version 959 is already newer than 959 2024-03-04T06:22:27.861 INFO:tasks.rados.rados.0.smithi064.stdout:3409: oid 36 updating version 959 to 960 2024-03-04T06:22:27.861 INFO:tasks.rados.rados.0.smithi064.stdout:3409: oid 36 version 960 is already newer than 957 2024-03-04T06:22:27.861 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 960 (ObjNum 1541 snap 437 seq_num 1541) dirty exists 2024-03-04T06:22:27.861 INFO:tasks.rados.rados.0.smithi064.stdout:3409: left oid 36 (ObjNum 1541 snap 437 seq_num 1541) 2024-03-04T06:22:27.861 INFO:tasks.rados.rados.0.smithi064.stdout:3413: finishing write tid 2 to smithi06431743-49 2024-03-04T06:22:27.861 INFO:tasks.rados.rados.0.smithi064.stdout:3411: finishing write tid 2 to smithi06431743-24 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:3412: finishing write tid 2 to smithi06431743-32 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:3412: finishing write tid 3 to smithi06431743-32 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:3413: finishing write tid 3 to smithi06431743-49 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:3411: finishing write tid 3 to smithi06431743-24 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:3413: finishing write tid 4 to smithi06431743-49 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:3412: finishing write tid 4 to smithi06431743-32 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:3412: finishing write tid 5 to smithi06431743-32 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:3412: oid 32 updating version 0 to 1618 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:3412: oid 32 updating version 1618 to 1619 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:3412: oid 32 version 1619 is already newer than 1618 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:3412: oid 32 version 1619 is already newer than 1616 2024-03-04T06:22:27.862 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1619 (ObjNum 1543 snap 437 seq_num 1543) dirty exists 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:3412: left oid 32 (ObjNum 1543 snap 437 seq_num 1543) 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:3413: finishing write tid 5 to smithi06431743-49 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:3413: oid 49 updating version 0 to 581 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:3413: oid 49 updating version 581 to 582 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:3413: oid 49 version 582 is already newer than 579 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:3413: oid 49 version 582 is already newer than 580 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 582 (ObjNum 1544 snap 437 seq_num 1544) dirty exists 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:3413: left oid 49 (ObjNum 1544 snap 437 seq_num 1544) 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:3411: finishing write tid 4 to smithi06431743-24 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:3411: finishing write tid 5 to smithi06431743-24 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:3411: oid 24 updating version 0 to 1320 2024-03-04T06:22:27.863 INFO:tasks.rados.rados.0.smithi064.stdout:3411: oid 24 version 1320 is already newer than 1319 2024-03-04T06:22:27.864 INFO:tasks.rados.rados.0.smithi064.stdout:3411: oid 24 updating version 1320 to 1321 2024-03-04T06:22:27.864 INFO:tasks.rados.rados.0.smithi064.stdout:3411: oid 24 version 1321 is already newer than 1318 2024-03-04T06:22:27.864 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 1321 (ObjNum 1542 snap 437 seq_num 1542) dirty exists 2024-03-04T06:22:27.864 INFO:tasks.rados.rados.0.smithi064.stdout:3411: left oid 24 (ObjNum 1542 snap 437 seq_num 1542) 2024-03-04T06:22:28.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:28.183 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:28.181+0000 7f9b94cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:28.186 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:28.186+0000 7f9b94cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:28.321 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:22:28.327 INFO:tasks.rados.rados.0.smithi064.stdout:3409: done (6 left) 2024-03-04T06:22:28.327 INFO:tasks.rados.rados.0.smithi064.stdout:3410: done (5 left) 2024-03-04T06:22:28.327 INFO:tasks.rados.rados.0.smithi064.stdout:3411: done (4 left) 2024-03-04T06:22:28.328 INFO:tasks.rados.rados.0.smithi064.stdout:3412: done (3 left) 2024-03-04T06:22:28.328 INFO:tasks.rados.rados.0.smithi064.stdout:3413: done (2 left) 2024-03-04T06:22:28.328 INFO:tasks.rados.rados.0.smithi064.stdout:3414: done (1 left) 2024-03-04T06:22:28.328 INFO:tasks.rados.rados.0.smithi064.stdout:3415: done (0 left) 2024-03-04T06:22:28.328 INFO:tasks.rados.rados.0.smithi064.stdout:3416: delete oid 16 current snap is 438 2024-03-04T06:22:28.336 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:22:28.337 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:22:28.375 INFO:tasks.rados.rados.0.smithi064.stdout:3416: done (0 left) 2024-03-04T06:22:28.376 INFO:tasks.rados.rados.0.smithi064.stdout:3417: delete oid 44 current snap is 438 2024-03-04T06:22:28.440 INFO:tasks.rados.rados.0.smithi064.stdout:3417: done (0 left) 2024-03-04T06:22:28.440 INFO:tasks.rados.rados.0.smithi064.stdout:3418: snap_create 2024-03-04T06:22:28.459 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:28.455+0000 7f55b4c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:28.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:28.461+0000 7f55b4c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:28.648 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:22:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:22:28.661 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1309,"stamp":"2024-03-04T06:22:28.323815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131732640,"num_objects":1133,"num_object_clones":923,"num_object_copies":3399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":6174,"num_read_kb":4821709,"num_write":5278,"num_write_kb":1406588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14766,"ondisk_log_size":14766,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":753008,"kb_used_data":518032,"kb_used_omap":0,"kb_used_meta":234944,"kb_avail":376734352,"statfs":{"total":386547056640,"available":385775976448,"internally_reserved":0,"allocated":530464768,"data_stored":595099071,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":240582656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13766055,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":28,"num_read_kb":21372,"num_write":35,"num_write_kb":11677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.918628"},"pg_stats":[{"pgid":"3.f","version":"872'809","reported_seq":2979,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.494771+0000","last_change":"2024-03-04T06:22:20.501166+0000","last_active":"2024-03-04T06:22:23.494771+0000","last_peered":"2024-03-04T06:22:23.494771+0000","last_clean":"2024-03-04T06:22:23.494771+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:22:23.494771+0000","last_undegraded":"2024-03-04T06:22:23.494771+0000","last_fullsized":"2024-03-04T06:22:23.494771+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7378999,"num_objects":72,"num_object_clones":57,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":331,"num_read_kb":262323,"num_write":299,"num_write_kb":78674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"861'773","reported_seq":2968,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.399018+0000","last_change":"2024-03-04T06:22:20.431723+0000","last_active":"2024-03-04T06:22:22.399018+0000","last_peered":"2024-03-04T06:22:22.399018+0000","last_clean":"2024-03-04T06:22:22.399018+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:22:22.399018+0000","last_undegraded":"2024-03-04T06:22:22.399018+0000","last_fullsized":"2024-03-04T06:22:22.399018+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781905,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":232,"num_read_kb":180836,"num_write":251,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"874'1080","reported_seq":3156,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.437549+0000","last_change":"2024-03-04T06:22:25.499334+0000","last_active":"2024-03-04T06:22:25.437549+0000","last_peered":"2024-03-04T06:22:25.437549+0000","last_clean":"2024-03-04T06:22:25.437549+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:22:25.437549+0000","last_undegraded":"2024-03-04T06:22:25.437549+0000","last_fullsized":"2024-03-04T06:22:25.437549+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1080,"log_dups_size":0,"ondisk_log_size":1080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13161333,"num_objects":88,"num_object_clones":73,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":514,"num_read_kb":411112,"num_write":399,"num_write_kb":109332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"869'910","reported_seq":2969,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.407561+0000","last_change":"2024-03-04T06:22:20.414474+0000","last_active":"2024-03-04T06:22:23.407561+0000","last_peered":"2024-03-04T06:22:23.407561+0000","last_clean":"2024-03-04T06:22:23.407561+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:22:23.407561+0000","last_undegraded":"2024-03-04T06:22:23.407561+0000","last_fullsized":"2024-03-04T06:22:23.407561+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10707148,"num_objects":61,"num_object_clones":51,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":310,"num_read_kb":237435,"num_write":296,"num_write_kb":82903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"874'365","reported_seq":2771,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.935266+0000","last_change":"2024-03-04T06:22:26.010041+0000","last_active":"2024-03-04T06:22:25.935266+0000","last_peered":"2024-03-04T06:22:25.935266+0000","last_clean":"2024-03-04T06:22:25.935266+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:22:25.935266+0000","last_undegraded":"2024-03-04T06:22:25.935266+0000","last_fullsized":"2024-03-04T06:22:25.935266+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":106,"num_read_kb":79818,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"861'914","reported_seq":3025,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.424039+0000","last_change":"2024-03-04T06:22:20.422361+0000","last_active":"2024-03-04T06:22:23.424039+0000","last_peered":"2024-03-04T06:22:23.424039+0000","last_clean":"2024-03-04T06:22:23.424039+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:22:23.424039+0000","last_undegraded":"2024-03-04T06:22:23.424039+0000","last_fullsized":"2024-03-04T06:22:23.424039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613868,"num_objects":65,"num_object_clones":55,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":394,"num_read_kb":322264,"num_write":315,"num_write_kb":91407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2563,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.419571+0000","last_change":"2024-03-04T06:22:20.422757+0000","last_active":"2024-03-04T06:22:23.419571+0000","last_peered":"2024-03-04T06:22:23.419571+0000","last_clean":"2024-03-04T06:22:23.419571+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:22:23.419571+0000","last_undegraded":"2024-03-04T06:22:23.419571+0000","last_fullsized":"2024-03-04T06:22:23.419571+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"874'577","reported_seq":2855,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.432217+0000","last_change":"2024-03-04T06:22:25.481920+0000","last_active":"2024-03-04T06:22:25.432217+0000","last_peered":"2024-03-04T06:22:25.432217+0000","last_clean":"2024-03-04T06:22:25.432217+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:22:25.432217+0000","last_undegraded":"2024-03-04T06:22:25.432217+0000","last_fullsized":"2024-03-04T06:22:25.432217+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916790,"num_objects":39,"num_object_clones":30,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":273,"num_read_kb":250536,"num_write":176,"num_write_kb":47268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2345,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.927133+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:22:25.927133+0000","last_peered":"2024-03-04T06:22:25.927133+0000","last_clean":"2024-03-04T06:22:25.927133+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:22:25.927133+0000","last_undegraded":"2024-03-04T06:22:25.927133+0000","last_fullsized":"2024-03-04T06:22:25.927133+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2162,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.442141+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:22:25.442141+0000","last_peered":"2024-03-04T06:22:25.442141+0000","last_clean":"2024-03-04T06:22:25.442141+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:22:25.442141+0000","last_undegraded":"2024-03-04T06:22:25.442141+0000","last_fullsized":"2024-03-04T06:22:25.442141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"874'937","reported_seq":3047,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.447892+0000","last_change":"2024-03-04T06:22:25.485438+0000","last_active":"2024-03-04T06:22:25.447892+0000","last_peered":"2024-03-04T06:22:25.447892+0000","last_clean":"2024-03-04T06:22:25.447892+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:22:25.447892+0000","last_undegraded":"2024-03-04T06:22:25.447892+0000","last_fullsized":"2024-03-04T06:22:25.447892+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":937,"log_dups_size":0,"ondisk_log_size":937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":411,"num_read_kb":327742,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2154,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.414059+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:22:23.414059+0000","last_peered":"2024-03-04T06:22:23.414059+0000","last_clean":"2024-03-04T06:22:23.414059+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:22:23.414059+0000","last_undegraded":"2024-03-04T06:22:23.414059+0000","last_fullsized":"2024-03-04T06:22:23.414059+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"874'610","reported_seq":2943,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.922898+0000","last_change":"2024-03-04T06:22:25.997590+0000","last_active":"2024-03-04T06:22:25.922898+0000","last_peered":"2024-03-04T06:22:25.922898+0000","last_clean":"2024-03-04T06:22:25.922898+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:22:25.922898+0000","last_undegraded":"2024-03-04T06:22:25.922898+0000","last_fullsized":"2024-03-04T06:22:25.922898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3580254,"num_objects":49,"num_object_clones":40,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":299,"num_read_kb":241126,"num_write":206,"num_write_kb":58469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2162,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.431393+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:22:25.431393+0000","last_peered":"2024-03-04T06:22:25.431393+0000","last_clean":"2024-03-04T06:22:25.431393+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:22:25.431393+0000","last_undegraded":"2024-03-04T06:22:25.431393+0000","last_fullsized":"2024-03-04T06:22:25.431393+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"874'1615","reported_seq":3516,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.943109+0000","last_change":"2024-03-04T06:22:26.052846+0000","last_active":"2024-03-04T06:22:25.943109+0000","last_peered":"2024-03-04T06:22:25.943109+0000","last_clean":"2024-03-04T06:22:25.943109+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:22:25.943109+0000","last_undegraded":"2024-03-04T06:22:25.943109+0000","last_fullsized":"2024-03-04T06:22:25.943109+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17800428,"num_objects":131,"num_object_clones":105,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":757,"num_read_kb":603237,"num_write":613,"num_write_kb":165431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2154,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.411696+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:22:23.411696+0000","last_peered":"2024-03-04T06:22:23.411696+0000","last_clean":"2024-03-04T06:22:23.411696+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:22:23.411696+0000","last_undegraded":"2024-03-04T06:22:23.411696+0000","last_fullsized":"2024-03-04T06:22:23.411696+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"869'1856","reported_seq":3506,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.400653+0000","last_change":"2024-03-04T06:22:21.404263+0000","last_active":"2024-03-04T06:22:22.400653+0000","last_peered":"2024-03-04T06:22:22.400653+0000","last_clean":"2024-03-04T06:22:22.400653+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:22:22.400653+0000","last_undegraded":"2024-03-04T06:22:22.400653+0000","last_fullsized":"2024-03-04T06:22:22.400653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1856,"log_dups_size":0,"ondisk_log_size":1856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17565097,"num_objects":141,"num_object_clones":117,"num_object_copies":423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":655,"num_read_kb":488706,"num_write":650,"num_write_kb":175647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"874'868","reported_seq":3047,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.441409+0000","last_change":"2024-03-04T06:22:25.480589+0000","last_active":"2024-03-04T06:22:25.441409+0000","last_peered":"2024-03-04T06:22:25.441409+0000","last_clean":"2024-03-04T06:22:25.441409+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:22:25.441409+0000","last_undegraded":"2024-03-04T06:22:25.441409+0000","last_fullsized":"2024-03-04T06:22:25.441409+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":868,"log_dups_size":0,"ondisk_log_size":868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7244543,"num_objects":66,"num_object_clones":53,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":424,"num_read_kb":349869,"num_write":313,"num_write_kb":85291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2229,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.405954+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:22:22.405954+0000","last_peered":"2024-03-04T06:22:22.405954+0000","last_clean":"2024-03-04T06:22:22.405954+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:22:22.405954+0000","last_undegraded":"2024-03-04T06:22:22.405954+0000","last_fullsized":"2024-03-04T06:22:22.405954+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"861'949","reported_seq":3091,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.409511+0000","last_change":"2024-03-04T06:22:20.413265+0000","last_active":"2024-03-04T06:22:22.409511+0000","last_peered":"2024-03-04T06:22:22.409511+0000","last_clean":"2024-03-04T06:22:22.409511+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:22:22.409511+0000","last_undegraded":"2024-03-04T06:22:22.409511+0000","last_fullsized":"2024-03-04T06:22:22.409511+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":949,"log_dups_size":0,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8690308,"num_objects":77,"num_object_clones":61,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":356,"num_read_kb":254720,"num_write":334,"num_write_kb":89756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2212,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.951039+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:22:25.951039+0000","last_peered":"2024-03-04T06:22:25.951039+0000","last_clean":"2024-03-04T06:22:25.951039+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:22:25.951039+0000","last_undegraded":"2024-03-04T06:22:25.951039+0000","last_fullsized":"2024-03-04T06:22:25.951039+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"874'1317","reported_seq":3223,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.931654+0000","last_change":"2024-03-04T06:22:26.037403+0000","last_active":"2024-03-04T06:22:25.931654+0000","last_peered":"2024-03-04T06:22:25.931654+0000","last_clean":"2024-03-04T06:22:25.931654+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:22:25.931654+0000","last_undegraded":"2024-03-04T06:22:25.931654+0000","last_fullsized":"2024-03-04T06:22:25.931654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1317,"log_dups_size":0,"ondisk_log_size":1317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7328545,"num_objects":94,"num_object_clones":80,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":472,"num_read_kb":365275,"num_write":440,"num_write_kb":122720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2229,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.397467+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:22:22.397467+0000","last_peered":"2024-03-04T06:22:22.397467+0000","last_clean":"2024-03-04T06:22:22.397467+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:22:22.397467+0000","last_undegraded":"2024-03-04T06:22:22.397467+0000","last_fullsized":"2024-03-04T06:22:22.397467+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"869'1089","reported_seq":3252,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.396753+0000","last_change":"2024-03-04T06:22:21.404726+0000","last_active":"2024-03-04T06:22:22.396753+0000","last_peered":"2024-03-04T06:22:22.396753+0000","last_clean":"2024-03-04T06:22:22.396753+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:22:22.396753+0000","last_undegraded":"2024-03-04T06:22:22.396753+0000","last_fullsized":"2024-03-04T06:22:22.396753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1089,"log_dups_size":0,"ondisk_log_size":1089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12623509,"num_objects":91,"num_object_clones":77,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":537,"num_read_kb":446619,"num_write":413,"num_write_kb":113410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2162,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.442717+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:22:25.442717+0000","last_peered":"2024-03-04T06:22:25.442717+0000","last_clean":"2024-03-04T06:22:25.442717+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:22:25.442717+0000","last_undegraded":"2024-03-04T06:22:25.442717+0000","last_fullsized":"2024-03-04T06:22:25.442717+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":131142272,"num_objects":1131,"num_object_clones":923,"num_object_copies":3393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":6071,"num_read_kb":4821618,"num_write":5141,"num_write_kb":1404694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14669,"ondisk_log_size":14669,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":236092,"kb_used_data":169012,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94135748,"statfs":{"total":96636764160,"available":96395005952,"internally_reserved":0,"allocated":173068288,"data_stored":195053601,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":177076,"kb_used_data":119724,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94194764,"statfs":{"total":96636764160,"available":96455438336,"internally_reserved":0,"allocated":122597376,"data_stored":136887870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":170892,"kb_used_data":113540,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94200948,"statfs":{"total":96636764160,"available":96461770752,"internally_reserved":0,"allocated":116264960,"data_stored":132530392,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":168948,"kb_used_data":115756,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94202892,"statfs":{"total":96636764160,"available":96463761408,"internally_reserved":0,"allocated":118534144,"data_stored":130627208,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:22:28.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:22:28.809 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:28.805+0000 7f2a6962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:28.812 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:28.811+0000 7f2a6962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:29.001 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:22:29.001 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:22:29.013 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1310,"stamp":"2024-03-04T06:22:28.892606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131732640,"num_objects":1133,"num_object_clones":923,"num_object_copies":3399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":6174,"num_read_kb":4821709,"num_write":5278,"num_write_kb":1406588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14766,"ondisk_log_size":14766,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":753008,"kb_used_data":518032,"kb_used_omap":0,"kb_used_meta":234944,"kb_avail":376734352,"statfs":{"total":386547056640,"available":385775976448,"internally_reserved":0,"allocated":530464768,"data_stored":595099071,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":240582656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13766055,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":28,"num_read_kb":21372,"num_write":35,"num_write_kb":11677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001065"},"pg_stats":[{"pgid":"3.f","version":"872'809","reported_seq":2979,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.494771+0000","last_change":"2024-03-04T06:22:20.501166+0000","last_active":"2024-03-04T06:22:23.494771+0000","last_peered":"2024-03-04T06:22:23.494771+0000","last_clean":"2024-03-04T06:22:23.494771+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:22:23.494771+0000","last_undegraded":"2024-03-04T06:22:23.494771+0000","last_fullsized":"2024-03-04T06:22:23.494771+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7378999,"num_objects":72,"num_object_clones":57,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":331,"num_read_kb":262323,"num_write":299,"num_write_kb":78674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"861'773","reported_seq":2968,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.399018+0000","last_change":"2024-03-04T06:22:20.431723+0000","last_active":"2024-03-04T06:22:22.399018+0000","last_peered":"2024-03-04T06:22:22.399018+0000","last_clean":"2024-03-04T06:22:22.399018+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:22:22.399018+0000","last_undegraded":"2024-03-04T06:22:22.399018+0000","last_fullsized":"2024-03-04T06:22:22.399018+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781905,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":232,"num_read_kb":180836,"num_write":251,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"874'1080","reported_seq":3156,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.437549+0000","last_change":"2024-03-04T06:22:25.499334+0000","last_active":"2024-03-04T06:22:25.437549+0000","last_peered":"2024-03-04T06:22:25.437549+0000","last_clean":"2024-03-04T06:22:25.437549+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:22:25.437549+0000","last_undegraded":"2024-03-04T06:22:25.437549+0000","last_fullsized":"2024-03-04T06:22:25.437549+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1080,"log_dups_size":0,"ondisk_log_size":1080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13161333,"num_objects":88,"num_object_clones":73,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":514,"num_read_kb":411112,"num_write":399,"num_write_kb":109332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"869'910","reported_seq":2969,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.407561+0000","last_change":"2024-03-04T06:22:20.414474+0000","last_active":"2024-03-04T06:22:23.407561+0000","last_peered":"2024-03-04T06:22:23.407561+0000","last_clean":"2024-03-04T06:22:23.407561+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:22:23.407561+0000","last_undegraded":"2024-03-04T06:22:23.407561+0000","last_fullsized":"2024-03-04T06:22:23.407561+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10707148,"num_objects":61,"num_object_clones":51,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":310,"num_read_kb":237435,"num_write":296,"num_write_kb":82903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"874'365","reported_seq":2771,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.935266+0000","last_change":"2024-03-04T06:22:26.010041+0000","last_active":"2024-03-04T06:22:25.935266+0000","last_peered":"2024-03-04T06:22:25.935266+0000","last_clean":"2024-03-04T06:22:25.935266+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:22:25.935266+0000","last_undegraded":"2024-03-04T06:22:25.935266+0000","last_fullsized":"2024-03-04T06:22:25.935266+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":106,"num_read_kb":79818,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"861'914","reported_seq":3025,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.424039+0000","last_change":"2024-03-04T06:22:20.422361+0000","last_active":"2024-03-04T06:22:23.424039+0000","last_peered":"2024-03-04T06:22:23.424039+0000","last_clean":"2024-03-04T06:22:23.424039+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:22:23.424039+0000","last_undegraded":"2024-03-04T06:22:23.424039+0000","last_fullsized":"2024-03-04T06:22:23.424039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613868,"num_objects":65,"num_object_clones":55,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":394,"num_read_kb":322264,"num_write":315,"num_write_kb":91407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2563,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.419571+0000","last_change":"2024-03-04T06:22:20.422757+0000","last_active":"2024-03-04T06:22:23.419571+0000","last_peered":"2024-03-04T06:22:23.419571+0000","last_clean":"2024-03-04T06:22:23.419571+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:22:23.419571+0000","last_undegraded":"2024-03-04T06:22:23.419571+0000","last_fullsized":"2024-03-04T06:22:23.419571+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"874'577","reported_seq":2855,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.432217+0000","last_change":"2024-03-04T06:22:25.481920+0000","last_active":"2024-03-04T06:22:25.432217+0000","last_peered":"2024-03-04T06:22:25.432217+0000","last_clean":"2024-03-04T06:22:25.432217+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:22:25.432217+0000","last_undegraded":"2024-03-04T06:22:25.432217+0000","last_fullsized":"2024-03-04T06:22:25.432217+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916790,"num_objects":39,"num_object_clones":30,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":273,"num_read_kb":250536,"num_write":176,"num_write_kb":47268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2345,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.927133+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:22:25.927133+0000","last_peered":"2024-03-04T06:22:25.927133+0000","last_clean":"2024-03-04T06:22:25.927133+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:22:25.927133+0000","last_undegraded":"2024-03-04T06:22:25.927133+0000","last_fullsized":"2024-03-04T06:22:25.927133+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2162,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.442141+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:22:25.442141+0000","last_peered":"2024-03-04T06:22:25.442141+0000","last_clean":"2024-03-04T06:22:25.442141+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:22:25.442141+0000","last_undegraded":"2024-03-04T06:22:25.442141+0000","last_fullsized":"2024-03-04T06:22:25.442141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"874'937","reported_seq":3047,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.447892+0000","last_change":"2024-03-04T06:22:25.485438+0000","last_active":"2024-03-04T06:22:25.447892+0000","last_peered":"2024-03-04T06:22:25.447892+0000","last_clean":"2024-03-04T06:22:25.447892+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:22:25.447892+0000","last_undegraded":"2024-03-04T06:22:25.447892+0000","last_fullsized":"2024-03-04T06:22:25.447892+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":937,"log_dups_size":0,"ondisk_log_size":937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":411,"num_read_kb":327742,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2154,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.414059+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:22:23.414059+0000","last_peered":"2024-03-04T06:22:23.414059+0000","last_clean":"2024-03-04T06:22:23.414059+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:22:23.414059+0000","last_undegraded":"2024-03-04T06:22:23.414059+0000","last_fullsized":"2024-03-04T06:22:23.414059+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"874'610","reported_seq":2943,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.922898+0000","last_change":"2024-03-04T06:22:25.997590+0000","last_active":"2024-03-04T06:22:25.922898+0000","last_peered":"2024-03-04T06:22:25.922898+0000","last_clean":"2024-03-04T06:22:25.922898+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:22:25.922898+0000","last_undegraded":"2024-03-04T06:22:25.922898+0000","last_fullsized":"2024-03-04T06:22:25.922898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3580254,"num_objects":49,"num_object_clones":40,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":299,"num_read_kb":241126,"num_write":206,"num_write_kb":58469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2162,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.431393+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:22:25.431393+0000","last_peered":"2024-03-04T06:22:25.431393+0000","last_clean":"2024-03-04T06:22:25.431393+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:22:25.431393+0000","last_undegraded":"2024-03-04T06:22:25.431393+0000","last_fullsized":"2024-03-04T06:22:25.431393+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"874'1615","reported_seq":3516,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.943109+0000","last_change":"2024-03-04T06:22:26.052846+0000","last_active":"2024-03-04T06:22:25.943109+0000","last_peered":"2024-03-04T06:22:25.943109+0000","last_clean":"2024-03-04T06:22:25.943109+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:22:25.943109+0000","last_undegraded":"2024-03-04T06:22:25.943109+0000","last_fullsized":"2024-03-04T06:22:25.943109+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17800428,"num_objects":131,"num_object_clones":105,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":757,"num_read_kb":603237,"num_write":613,"num_write_kb":165431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2154,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.411696+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:22:23.411696+0000","last_peered":"2024-03-04T06:22:23.411696+0000","last_clean":"2024-03-04T06:22:23.411696+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:22:23.411696+0000","last_undegraded":"2024-03-04T06:22:23.411696+0000","last_fullsized":"2024-03-04T06:22:23.411696+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"869'1856","reported_seq":3506,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.400653+0000","last_change":"2024-03-04T06:22:21.404263+0000","last_active":"2024-03-04T06:22:22.400653+0000","last_peered":"2024-03-04T06:22:22.400653+0000","last_clean":"2024-03-04T06:22:22.400653+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:22:22.400653+0000","last_undegraded":"2024-03-04T06:22:22.400653+0000","last_fullsized":"2024-03-04T06:22:22.400653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1856,"log_dups_size":0,"ondisk_log_size":1856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17565097,"num_objects":141,"num_object_clones":117,"num_object_copies":423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":655,"num_read_kb":488706,"num_write":650,"num_write_kb":175647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"874'868","reported_seq":3047,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.441409+0000","last_change":"2024-03-04T06:22:25.480589+0000","last_active":"2024-03-04T06:22:25.441409+0000","last_peered":"2024-03-04T06:22:25.441409+0000","last_clean":"2024-03-04T06:22:25.441409+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:22:25.441409+0000","last_undegraded":"2024-03-04T06:22:25.441409+0000","last_fullsized":"2024-03-04T06:22:25.441409+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":868,"log_dups_size":0,"ondisk_log_size":868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7244543,"num_objects":66,"num_object_clones":53,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":424,"num_read_kb":349869,"num_write":313,"num_write_kb":85291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2229,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.405954+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:22:22.405954+0000","last_peered":"2024-03-04T06:22:22.405954+0000","last_clean":"2024-03-04T06:22:22.405954+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:22:22.405954+0000","last_undegraded":"2024-03-04T06:22:22.405954+0000","last_fullsized":"2024-03-04T06:22:22.405954+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"861'949","reported_seq":3091,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.409511+0000","last_change":"2024-03-04T06:22:20.413265+0000","last_active":"2024-03-04T06:22:22.409511+0000","last_peered":"2024-03-04T06:22:22.409511+0000","last_clean":"2024-03-04T06:22:22.409511+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:22:22.409511+0000","last_undegraded":"2024-03-04T06:22:22.409511+0000","last_fullsized":"2024-03-04T06:22:22.409511+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":949,"log_dups_size":0,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8690308,"num_objects":77,"num_object_clones":61,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":356,"num_read_kb":254720,"num_write":334,"num_write_kb":89756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2212,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.951039+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:22:25.951039+0000","last_peered":"2024-03-04T06:22:25.951039+0000","last_clean":"2024-03-04T06:22:25.951039+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:22:25.951039+0000","last_undegraded":"2024-03-04T06:22:25.951039+0000","last_fullsized":"2024-03-04T06:22:25.951039+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"874'1317","reported_seq":3223,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.931654+0000","last_change":"2024-03-04T06:22:26.037403+0000","last_active":"2024-03-04T06:22:25.931654+0000","last_peered":"2024-03-04T06:22:25.931654+0000","last_clean":"2024-03-04T06:22:25.931654+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:22:25.931654+0000","last_undegraded":"2024-03-04T06:22:25.931654+0000","last_fullsized":"2024-03-04T06:22:25.931654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1317,"log_dups_size":0,"ondisk_log_size":1317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7328545,"num_objects":94,"num_object_clones":80,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":472,"num_read_kb":365275,"num_write":440,"num_write_kb":122720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2229,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.397467+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:22:22.397467+0000","last_peered":"2024-03-04T06:22:22.397467+0000","last_clean":"2024-03-04T06:22:22.397467+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:22:22.397467+0000","last_undegraded":"2024-03-04T06:22:22.397467+0000","last_fullsized":"2024-03-04T06:22:22.397467+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"869'1089","reported_seq":3252,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.396753+0000","last_change":"2024-03-04T06:22:21.404726+0000","last_active":"2024-03-04T06:22:22.396753+0000","last_peered":"2024-03-04T06:22:22.396753+0000","last_clean":"2024-03-04T06:22:22.396753+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:22:22.396753+0000","last_undegraded":"2024-03-04T06:22:22.396753+0000","last_fullsized":"2024-03-04T06:22:22.396753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1089,"log_dups_size":0,"ondisk_log_size":1089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12623509,"num_objects":91,"num_object_clones":77,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":537,"num_read_kb":446619,"num_write":413,"num_write_kb":113410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2162,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.442717+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:22:25.442717+0000","last_peered":"2024-03-04T06:22:25.442717+0000","last_clean":"2024-03-04T06:22:25.442717+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:22:25.442717+0000","last_undegraded":"2024-03-04T06:22:25.442717+0000","last_fullsized":"2024-03-04T06:22:25.442717+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":131142272,"num_objects":1131,"num_object_clones":923,"num_object_copies":3393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":6071,"num_read_kb":4821618,"num_write":5141,"num_write_kb":1404694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14669,"ondisk_log_size":14669,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":236092,"kb_used_data":169012,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94135748,"statfs":{"total":96636764160,"available":96395005952,"internally_reserved":0,"allocated":173068288,"data_stored":195053601,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":177076,"kb_used_data":119724,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94194764,"statfs":{"total":96636764160,"available":96455438336,"internally_reserved":0,"allocated":122597376,"data_stored":136887870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":170892,"kb_used_data":113540,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94200948,"statfs":{"total":96636764160,"available":96461770752,"internally_reserved":0,"allocated":116264960,"data_stored":132530392,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":168948,"kb_used_data":115756,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94202892,"statfs":{"total":96636764160,"available":96463761408,"internally_reserved":0,"allocated":118534144,"data_stored":130627208,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:22:29.014 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:22:29.014 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:22:29.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:22:29.121 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:29.118+0000 7f30a206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:29.124 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:29.124+0000 7f30a206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:29.315 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:22:29.315 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:22:29.327 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1310,"stamp":"2024-03-04T06:22:28.892606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131732640,"num_objects":1133,"num_object_clones":923,"num_object_copies":3399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":6174,"num_read_kb":4821709,"num_write":5278,"num_write_kb":1406588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14766,"ondisk_log_size":14766,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":753008,"kb_used_data":518032,"kb_used_omap":0,"kb_used_meta":234944,"kb_avail":376734352,"statfs":{"total":386547056640,"available":385775976448,"internally_reserved":0,"allocated":530464768,"data_stored":595099071,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":240582656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13766055,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-2,"num_read":28,"num_read_kb":21372,"num_write":35,"num_write_kb":11677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001065"},"pg_stats":[{"pgid":"3.f","version":"872'809","reported_seq":2979,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.494771+0000","last_change":"2024-03-04T06:22:20.501166+0000","last_active":"2024-03-04T06:22:23.494771+0000","last_peered":"2024-03-04T06:22:23.494771+0000","last_clean":"2024-03-04T06:22:23.494771+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:22:23.494771+0000","last_undegraded":"2024-03-04T06:22:23.494771+0000","last_fullsized":"2024-03-04T06:22:23.494771+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7378999,"num_objects":72,"num_object_clones":57,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":331,"num_read_kb":262323,"num_write":299,"num_write_kb":78674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"861'773","reported_seq":2968,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.399018+0000","last_change":"2024-03-04T06:22:20.431723+0000","last_active":"2024-03-04T06:22:22.399018+0000","last_peered":"2024-03-04T06:22:22.399018+0000","last_clean":"2024-03-04T06:22:22.399018+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:22:22.399018+0000","last_undegraded":"2024-03-04T06:22:22.399018+0000","last_fullsized":"2024-03-04T06:22:22.399018+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781905,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":232,"num_read_kb":180836,"num_write":251,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"874'1080","reported_seq":3156,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.437549+0000","last_change":"2024-03-04T06:22:25.499334+0000","last_active":"2024-03-04T06:22:25.437549+0000","last_peered":"2024-03-04T06:22:25.437549+0000","last_clean":"2024-03-04T06:22:25.437549+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:22:25.437549+0000","last_undegraded":"2024-03-04T06:22:25.437549+0000","last_fullsized":"2024-03-04T06:22:25.437549+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1080,"log_dups_size":0,"ondisk_log_size":1080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13161333,"num_objects":88,"num_object_clones":73,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":514,"num_read_kb":411112,"num_write":399,"num_write_kb":109332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"869'910","reported_seq":2969,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.407561+0000","last_change":"2024-03-04T06:22:20.414474+0000","last_active":"2024-03-04T06:22:23.407561+0000","last_peered":"2024-03-04T06:22:23.407561+0000","last_clean":"2024-03-04T06:22:23.407561+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:22:23.407561+0000","last_undegraded":"2024-03-04T06:22:23.407561+0000","last_fullsized":"2024-03-04T06:22:23.407561+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10707148,"num_objects":61,"num_object_clones":51,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":310,"num_read_kb":237435,"num_write":296,"num_write_kb":82903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"874'365","reported_seq":2771,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.935266+0000","last_change":"2024-03-04T06:22:26.010041+0000","last_active":"2024-03-04T06:22:25.935266+0000","last_peered":"2024-03-04T06:22:25.935266+0000","last_clean":"2024-03-04T06:22:25.935266+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:22:25.935266+0000","last_undegraded":"2024-03-04T06:22:25.935266+0000","last_fullsized":"2024-03-04T06:22:25.935266+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":106,"num_read_kb":79818,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"861'914","reported_seq":3025,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.424039+0000","last_change":"2024-03-04T06:22:20.422361+0000","last_active":"2024-03-04T06:22:23.424039+0000","last_peered":"2024-03-04T06:22:23.424039+0000","last_clean":"2024-03-04T06:22:23.424039+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:22:23.424039+0000","last_undegraded":"2024-03-04T06:22:23.424039+0000","last_fullsized":"2024-03-04T06:22:23.424039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613868,"num_objects":65,"num_object_clones":55,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":394,"num_read_kb":322264,"num_write":315,"num_write_kb":91407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2563,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.419571+0000","last_change":"2024-03-04T06:22:20.422757+0000","last_active":"2024-03-04T06:22:23.419571+0000","last_peered":"2024-03-04T06:22:23.419571+0000","last_clean":"2024-03-04T06:22:23.419571+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:22:23.419571+0000","last_undegraded":"2024-03-04T06:22:23.419571+0000","last_fullsized":"2024-03-04T06:22:23.419571+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"874'577","reported_seq":2855,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.432217+0000","last_change":"2024-03-04T06:22:25.481920+0000","last_active":"2024-03-04T06:22:25.432217+0000","last_peered":"2024-03-04T06:22:25.432217+0000","last_clean":"2024-03-04T06:22:25.432217+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:22:25.432217+0000","last_undegraded":"2024-03-04T06:22:25.432217+0000","last_fullsized":"2024-03-04T06:22:25.432217+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916790,"num_objects":39,"num_object_clones":30,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":273,"num_read_kb":250536,"num_write":176,"num_write_kb":47268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2345,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.927133+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:22:25.927133+0000","last_peered":"2024-03-04T06:22:25.927133+0000","last_clean":"2024-03-04T06:22:25.927133+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:22:25.927133+0000","last_undegraded":"2024-03-04T06:22:25.927133+0000","last_fullsized":"2024-03-04T06:22:25.927133+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2162,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.442141+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:22:25.442141+0000","last_peered":"2024-03-04T06:22:25.442141+0000","last_clean":"2024-03-04T06:22:25.442141+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:22:25.442141+0000","last_undegraded":"2024-03-04T06:22:25.442141+0000","last_fullsized":"2024-03-04T06:22:25.442141+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"874'937","reported_seq":3047,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.447892+0000","last_change":"2024-03-04T06:22:25.485438+0000","last_active":"2024-03-04T06:22:25.447892+0000","last_peered":"2024-03-04T06:22:25.447892+0000","last_clean":"2024-03-04T06:22:25.447892+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:22:25.447892+0000","last_undegraded":"2024-03-04T06:22:25.447892+0000","last_fullsized":"2024-03-04T06:22:25.447892+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":937,"log_dups_size":0,"ondisk_log_size":937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":411,"num_read_kb":327742,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2154,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.414059+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:22:23.414059+0000","last_peered":"2024-03-04T06:22:23.414059+0000","last_clean":"2024-03-04T06:22:23.414059+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:22:23.414059+0000","last_undegraded":"2024-03-04T06:22:23.414059+0000","last_fullsized":"2024-03-04T06:22:23.414059+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"874'610","reported_seq":2943,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.922898+0000","last_change":"2024-03-04T06:22:25.997590+0000","last_active":"2024-03-04T06:22:25.922898+0000","last_peered":"2024-03-04T06:22:25.922898+0000","last_clean":"2024-03-04T06:22:25.922898+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:22:25.922898+0000","last_undegraded":"2024-03-04T06:22:25.922898+0000","last_fullsized":"2024-03-04T06:22:25.922898+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3580254,"num_objects":49,"num_object_clones":40,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":299,"num_read_kb":241126,"num_write":206,"num_write_kb":58469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2162,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.431393+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:22:25.431393+0000","last_peered":"2024-03-04T06:22:25.431393+0000","last_clean":"2024-03-04T06:22:25.431393+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:22:25.431393+0000","last_undegraded":"2024-03-04T06:22:25.431393+0000","last_fullsized":"2024-03-04T06:22:25.431393+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"874'1615","reported_seq":3516,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.943109+0000","last_change":"2024-03-04T06:22:26.052846+0000","last_active":"2024-03-04T06:22:25.943109+0000","last_peered":"2024-03-04T06:22:25.943109+0000","last_clean":"2024-03-04T06:22:25.943109+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:22:25.943109+0000","last_undegraded":"2024-03-04T06:22:25.943109+0000","last_fullsized":"2024-03-04T06:22:25.943109+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1615,"log_dups_size":0,"ondisk_log_size":1615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17800428,"num_objects":131,"num_object_clones":105,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":757,"num_read_kb":603237,"num_write":613,"num_write_kb":165431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2154,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-04T06:22:23.411696+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:22:23.411696+0000","last_peered":"2024-03-04T06:22:23.411696+0000","last_clean":"2024-03-04T06:22:23.411696+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:22:23.411696+0000","last_undegraded":"2024-03-04T06:22:23.411696+0000","last_fullsized":"2024-03-04T06:22:23.411696+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"869'1856","reported_seq":3506,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.400653+0000","last_change":"2024-03-04T06:22:21.404263+0000","last_active":"2024-03-04T06:22:22.400653+0000","last_peered":"2024-03-04T06:22:22.400653+0000","last_clean":"2024-03-04T06:22:22.400653+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:22:22.400653+0000","last_undegraded":"2024-03-04T06:22:22.400653+0000","last_fullsized":"2024-03-04T06:22:22.400653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1856,"log_dups_size":0,"ondisk_log_size":1856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17565097,"num_objects":141,"num_object_clones":117,"num_object_copies":423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":655,"num_read_kb":488706,"num_write":650,"num_write_kb":175647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"874'868","reported_seq":3047,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.441409+0000","last_change":"2024-03-04T06:22:25.480589+0000","last_active":"2024-03-04T06:22:25.441409+0000","last_peered":"2024-03-04T06:22:25.441409+0000","last_clean":"2024-03-04T06:22:25.441409+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:22:25.441409+0000","last_undegraded":"2024-03-04T06:22:25.441409+0000","last_fullsized":"2024-03-04T06:22:25.441409+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":868,"log_dups_size":0,"ondisk_log_size":868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7244543,"num_objects":66,"num_object_clones":53,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":424,"num_read_kb":349869,"num_write":313,"num_write_kb":85291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2229,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.405954+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:22:22.405954+0000","last_peered":"2024-03-04T06:22:22.405954+0000","last_clean":"2024-03-04T06:22:22.405954+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:22:22.405954+0000","last_undegraded":"2024-03-04T06:22:22.405954+0000","last_fullsized":"2024-03-04T06:22:22.405954+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"861'949","reported_seq":3091,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.409511+0000","last_change":"2024-03-04T06:22:20.413265+0000","last_active":"2024-03-04T06:22:22.409511+0000","last_peered":"2024-03-04T06:22:22.409511+0000","last_clean":"2024-03-04T06:22:22.409511+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:22:22.409511+0000","last_undegraded":"2024-03-04T06:22:22.409511+0000","last_fullsized":"2024-03-04T06:22:22.409511+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":949,"log_dups_size":0,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8690308,"num_objects":77,"num_object_clones":61,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":356,"num_read_kb":254720,"num_write":334,"num_write_kb":89756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2212,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.951039+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:22:25.951039+0000","last_peered":"2024-03-04T06:22:25.951039+0000","last_clean":"2024-03-04T06:22:25.951039+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:22:25.951039+0000","last_undegraded":"2024-03-04T06:22:25.951039+0000","last_fullsized":"2024-03-04T06:22:25.951039+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"874'1317","reported_seq":3223,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.931654+0000","last_change":"2024-03-04T06:22:26.037403+0000","last_active":"2024-03-04T06:22:25.931654+0000","last_peered":"2024-03-04T06:22:25.931654+0000","last_clean":"2024-03-04T06:22:25.931654+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:22:25.931654+0000","last_undegraded":"2024-03-04T06:22:25.931654+0000","last_fullsized":"2024-03-04T06:22:25.931654+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1317,"log_dups_size":0,"ondisk_log_size":1317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7328545,"num_objects":94,"num_object_clones":80,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":472,"num_read_kb":365275,"num_write":440,"num_write_kb":122720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2229,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.397467+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:22:22.397467+0000","last_peered":"2024-03-04T06:22:22.397467+0000","last_clean":"2024-03-04T06:22:22.397467+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:22:22.397467+0000","last_undegraded":"2024-03-04T06:22:22.397467+0000","last_fullsized":"2024-03-04T06:22:22.397467+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"869'1089","reported_seq":3252,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-03-04T06:22:22.396753+0000","last_change":"2024-03-04T06:22:21.404726+0000","last_active":"2024-03-04T06:22:22.396753+0000","last_peered":"2024-03-04T06:22:22.396753+0000","last_clean":"2024-03-04T06:22:22.396753+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:22:22.396753+0000","last_undegraded":"2024-03-04T06:22:22.396753+0000","last_fullsized":"2024-03-04T06:22:22.396753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1089,"log_dups_size":0,"ondisk_log_size":1089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12623509,"num_objects":91,"num_object_clones":77,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":537,"num_read_kb":446619,"num_write":413,"num_write_kb":113410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2162,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-04T06:22:25.442717+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:22:25.442717+0000","last_peered":"2024-03-04T06:22:25.442717+0000","last_clean":"2024-03-04T06:22:25.442717+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:22:25.442717+0000","last_undegraded":"2024-03-04T06:22:25.442717+0000","last_fullsized":"2024-03-04T06:22:25.442717+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":131142272,"num_objects":1131,"num_object_clones":923,"num_object_copies":3393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":169,"num_read":6071,"num_read_kb":4821618,"num_write":5141,"num_write_kb":1404694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14669,"ondisk_log_size":14669,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":236092,"kb_used_data":169012,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94135748,"statfs":{"total":96636764160,"available":96395005952,"internally_reserved":0,"allocated":173068288,"data_stored":195053601,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706544,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":177076,"kb_used_data":119724,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94194764,"statfs":{"total":96636764160,"available":96455438336,"internally_reserved":0,"allocated":122597376,"data_stored":136887870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":170892,"kb_used_data":113540,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94200948,"statfs":{"total":96636764160,"available":96461770752,"internally_reserved":0,"allocated":116264960,"data_stored":132530392,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706543,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":168948,"kb_used_data":115756,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94202892,"statfs":{"total":96636764160,"available":96463761408,"internally_reserved":0,"allocated":118534144,"data_stored":130627208,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:22:29.431 INFO:tasks.rados.rados.0.smithi064.stdout:3418: done (0 left) 2024-03-04T06:22:29.431 INFO:tasks.rados.rados.0.smithi064.stdout:3419: read oid 49 snap 432 2024-03-04T06:22:29.431 INFO:tasks.rados.rados.0.smithi064.stdout:3419: expect (ObjNum 1491 snap 416 seq_num 1491) 2024-03-04T06:22:29.431 INFO:tasks.rados.rados.0.smithi064.stdout:3420: read oid 9 snap -1 2024-03-04T06:22:29.431 INFO:tasks.rados.rados.0.smithi064.stdout:3420: expect (ObjNum 1492 snap 416 seq_num 1492) 2024-03-04T06:22:29.432 INFO:tasks.rados.rados.0.smithi064.stdout:3421: delete oid 38 current snap is 439 2024-03-04T06:22:29.463 INFO:tasks.rados.rados.0.smithi064.stdout:3421: done (2 left) 2024-03-04T06:22:29.463 INFO:tasks.rados.rados.0.smithi064.stdout:3422: read oid 24 snap -1 2024-03-04T06:22:29.463 INFO:tasks.rados.rados.0.smithi064.stdout:3422: expect (ObjNum 1542 snap 437 seq_num 1542) 2024-03-04T06:22:29.463 INFO:tasks.rados.rados.0.smithi064.stdout:3423: delete oid 48 current snap is 439 2024-03-04T06:22:29.470 INFO:tasks.rados.rados.0.smithi064.stdout:3419: expect (ObjNum 1491 snap 416 seq_num 1491) 2024-03-04T06:22:29.586 INFO:tasks.rados.rados.0.smithi064.stdout:3420: expect (ObjNum 1492 snap 416 seq_num 1492) 2024-03-04T06:22:29.701 INFO:tasks.rados.rados.0.smithi064.stdout:3422: expect (ObjNum 1542 snap 437 seq_num 1542) 2024-03-04T06:22:29.847 INFO:tasks.rados.rados.0.smithi064.stdout:3419: done (3 left) 2024-03-04T06:22:29.847 INFO:tasks.rados.rados.0.smithi064.stdout:3420: done (2 left) 2024-03-04T06:22:29.848 INFO:tasks.rados.rados.0.smithi064.stdout:3422: done (1 left) 2024-03-04T06:22:29.848 INFO:tasks.rados.rados.0.smithi064.stdout:3423: done (0 left) 2024-03-04T06:22:29.848 INFO:tasks.rados.rados.0.smithi064.stdout:3424: snap_remove snap 431 2024-03-04T06:22:30.254 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:30.363 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:30.361+0000 7f633895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:30.365 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:30.365+0000 7f633895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:30.431 INFO:tasks.rados.rados.0.smithi064.stdout:3424: done (0 left) 2024-03-04T06:22:30.431 INFO:tasks.rados.rados.0.smithi064.stdout:3425: snap_remove snap 398 2024-03-04T06:22:31.436 INFO:tasks.rados.rados.0.smithi064.stdout:3425: done (0 left) 2024-03-04T06:22:31.436 INFO:tasks.rados.rados.0.smithi064.stdout:3426: write oid 28 current snap is 439 2024-03-04T06:22:31.436 INFO:tasks.rados.rados.0.smithi064.stdout:3426: seq_num 1549 ranges {434043=580360,1710490=323553} 2024-03-04T06:22:31.442 INFO:tasks.rados.rados.0.smithi064.stdout:3426: writing smithi06431743-28 from 434043 to 1014403 tid 1 2024-03-04T06:22:31.447 INFO:tasks.rados.rados.0.smithi064.stdout:3426: writing smithi06431743-28 from 1710490 to 2034043 tid 2 2024-03-04T06:22:31.447 INFO:tasks.rados.rados.0.smithi064.stdout:3427: write oid 13 current snap is 439 2024-03-04T06:22:31.447 INFO:tasks.rados.rados.0.smithi064.stdout:3427: seq_num 1550 ranges {450850=655609,1768972=281878} 2024-03-04T06:22:31.455 INFO:tasks.rados.rados.0.smithi064.stdout:3427: writing smithi06431743-13 from 450850 to 1106459 tid 1 2024-03-04T06:22:31.463 INFO:tasks.rados.rados.0.smithi064.stdout:3427: writing smithi06431743-13 from 1768972 to 2050850 tid 2 2024-03-04T06:22:31.463 INFO:tasks.rados.rados.0.smithi064.stdout:3428: delete oid 25 current snap is 439 2024-03-04T06:22:31.597 INFO:tasks.rados.rados.0.smithi064.stdout:3426: finishing write tid 1 to smithi06431743-28 2024-03-04T06:22:31.612 INFO:tasks.rados.rados.0.smithi064.stdout:3426: finishing write tid 2 to smithi06431743-28 2024-03-04T06:22:31.619 INFO:tasks.rados.rados.0.smithi064.stdout:3427: finishing write tid 1 to smithi06431743-13 2024-03-04T06:22:31.641 INFO:tasks.rados.rados.0.smithi064.stdout:3428: done (2 left) 2024-03-04T06:22:31.641 INFO:tasks.rados.rados.0.smithi064.stdout:3429: snap_create 2024-03-04T06:22:31.652 INFO:tasks.rados.rados.0.smithi064.stdout:3427: finishing write tid 2 to smithi06431743-13 2024-03-04T06:22:31.653 INFO:tasks.rados.rados.0.smithi064.stdout:3426: finishing write tid 3 to smithi06431743-28 2024-03-04T06:22:31.666 INFO:tasks.rados.rados.0.smithi064.stdout:3427: finishing write tid 3 to smithi06431743-13 2024-03-04T06:22:31.668 INFO:tasks.rados.rados.0.smithi064.stdout:3427: finishing write tid 4 to smithi06431743-13 2024-03-04T06:22:31.668 INFO:tasks.rados.rados.0.smithi064.stdout:3427: oid 13 updating version 0 to 1874 2024-03-04T06:22:31.668 INFO:tasks.rados.rados.0.smithi064.stdout:3427: oid 13 version 1874 is already newer than 1874 2024-03-04T06:22:31.668 INFO:tasks.rados.rados.0.smithi064.stdout:3427: oid 13 version 1874 is already newer than 1872 2024-03-04T06:22:31.668 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1874 (ObjNum 1550 snap 439 seq_num 1550) dirty exists 2024-03-04T06:22:31.668 INFO:tasks.rados.rados.0.smithi064.stdout:3427: left oid 13 (ObjNum 1550 snap 439 seq_num 1550) 2024-03-04T06:22:31.670 INFO:tasks.rados.rados.0.smithi064.stdout:3426: finishing write tid 4 to smithi06431743-28 2024-03-04T06:22:31.670 INFO:tasks.rados.rados.0.smithi064.stdout:3426: oid 28 updating version 0 to 1626 2024-03-04T06:22:31.670 INFO:tasks.rados.rados.0.smithi064.stdout:3426: oid 28 updating version 1626 to 1627 2024-03-04T06:22:31.670 INFO:tasks.rados.rados.0.smithi064.stdout:3426: oid 28 version 1627 is already newer than 1625 2024-03-04T06:22:31.671 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 1627 (ObjNum 1549 snap 439 seq_num 1549) dirty exists 2024-03-04T06:22:31.671 INFO:tasks.rados.rados.0.smithi064.stdout:3426: left oid 28 (ObjNum 1549 snap 439 seq_num 1549) 2024-03-04T06:22:32.435 INFO:tasks.rados.rados.0.smithi064.stdout:3426: done (2 left) 2024-03-04T06:22:32.436 INFO:tasks.rados.rados.0.smithi064.stdout:3427: done (1 left) 2024-03-04T06:22:32.436 INFO:tasks.rados.rados.0.smithi064.stdout:3429: done (0 left) 2024-03-04T06:22:32.436 INFO:tasks.rados.rados.0.smithi064.stdout:3430: delete oid 20 current snap is 440 2024-03-04T06:22:32.437 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:32.484 INFO:tasks.rados.rados.0.smithi064.stdout:3430: done (0 left) 2024-03-04T06:22:32.484 INFO:tasks.rados.rados.0.smithi064.stdout:3431: write (excl) oid 16 current snap is 440 2024-03-04T06:22:32.485 INFO:tasks.rados.rados.0.smithi064.stdout:3431: seq_num 1553 ranges {501271=597709,1544418=556853} 2024-03-04T06:22:32.490 INFO:tasks.rados.rados.0.smithi064.stdout:3431: writing smithi06431743-16 from 501271 to 1098980 tid 1 2024-03-04T06:22:32.496 INFO:tasks.rados.rados.0.smithi064.stdout:3431: writing smithi06431743-16 from 1544418 to 2101271 tid 2 2024-03-04T06:22:32.496 INFO:tasks.rados.rados.0.smithi064.stdout:3432: snap_remove snap 404 2024-03-04T06:22:32.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:32.557+0000 7f476b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:32.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:32.561+0000 7f476b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:33.441 INFO:tasks.rados.rados.0.smithi064.stdout:3432: done (1 left) 2024-03-04T06:22:33.441 INFO:tasks.rados.rados.0.smithi064.stdout:3433: write oid 9 current snap is 440 2024-03-04T06:22:33.441 INFO:tasks.rados.rados.0.smithi064.stdout:3431: finishing write tid 1 to smithi06431743-16 2024-03-04T06:22:33.441 INFO:tasks.rados.rados.0.smithi064.stdout:3433: seq_num 1554 ranges {518078=672958,1602900=487868} 2024-03-04T06:22:33.447 INFO:tasks.rados.rados.0.smithi064.stdout:3433: writing smithi06431743-9 from 518078 to 1191036 tid 1 2024-03-04T06:22:33.453 INFO:tasks.rados.rados.0.smithi064.stdout:3433: writing smithi06431743-9 from 1602900 to 2090768 tid 2 2024-03-04T06:22:33.453 INFO:tasks.rados.rados.0.smithi064.stdout:3431: finishing write tid 2 to smithi06431743-16 2024-03-04T06:22:33.453 INFO:tasks.rados.rados.0.smithi064.stdout:3434: read oid 43 snap -1 2024-03-04T06:22:33.453 INFO:tasks.rados.rados.0.smithi064.stdout:3434: expect (ObjNum 1538 snap 434 seq_num 1538) 2024-03-04T06:22:33.454 INFO:tasks.rados.rados.0.smithi064.stdout:3435: read oid 34 snap -1 2024-03-04T06:22:33.454 INFO:tasks.rados.rados.0.smithi064.stdout:3435: expect (ObjNum 1497 snap 418 seq_num 1497) 2024-03-04T06:22:33.454 INFO:tasks.rados.rados.0.smithi064.stdout:3436: snap_create 2024-03-04T06:22:33.454 INFO:tasks.rados.rados.0.smithi064.stdout:3431: finishing write tid 3 to smithi06431743-16 2024-03-04T06:22:33.454 INFO:tasks.rados.rados.0.smithi064.stdout:3431: finishing write tid 4 to smithi06431743-16 2024-03-04T06:22:33.454 INFO:tasks.rados.rados.0.smithi064.stdout:3431: oid 16 updating version 0 to 975 2024-03-04T06:22:33.454 INFO:tasks.rados.rados.0.smithi064.stdout:3431: oid 16 version 975 is already newer than 973 2024-03-04T06:22:33.454 INFO:tasks.rados.rados.0.smithi064.stdout:3431: oid 16 version 975 is already newer than 974 2024-03-04T06:22:33.454 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 975 (ObjNum 1553 snap 440 seq_num 1553) dirty exists 2024-03-04T06:22:33.454 INFO:tasks.rados.rados.0.smithi064.stdout:3431: left oid 16 (ObjNum 1553 snap 440 seq_num 1553) 2024-03-04T06:22:33.548 INFO:tasks.rados.rados.0.smithi064.stdout:3434: expect (ObjNum 1538 snap 434 seq_num 1538) 2024-03-04T06:22:33.698 INFO:tasks.rados.rados.0.smithi064.stdout:3433: finishing write tid 1 to smithi06431743-9 2024-03-04T06:22:33.698 INFO:tasks.rados.rados.0.smithi064.stdout:3435: expect (ObjNum 1497 snap 418 seq_num 1497) 2024-03-04T06:22:33.859 INFO:tasks.rados.rados.0.smithi064.stdout:3433: finishing write tid 2 to smithi06431743-9 2024-03-04T06:22:33.859 INFO:tasks.rados.rados.0.smithi064.stdout:3433: finishing write tid 3 to smithi06431743-9 2024-03-04T06:22:33.859 INFO:tasks.rados.rados.0.smithi064.stdout:3433: finishing write tid 4 to smithi06431743-9 2024-03-04T06:22:33.859 INFO:tasks.rados.rados.0.smithi064.stdout:3433: oid 9 updating version 0 to 1097 2024-03-04T06:22:33.859 INFO:tasks.rados.rados.0.smithi064.stdout:3433: oid 9 updating version 1097 to 1099 2024-03-04T06:22:33.860 INFO:tasks.rados.rados.0.smithi064.stdout:3433: oid 9 version 1099 is already newer than 1099 2024-03-04T06:22:33.860 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 1099 (ObjNum 1554 snap 440 seq_num 1554) dirty exists 2024-03-04T06:22:33.860 INFO:tasks.rados.rados.0.smithi064.stdout:3433: left oid 9 (ObjNum 1554 snap 440 seq_num 1554) 2024-03-04T06:22:34.329 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:22:34.330 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:22:34.330 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:22:34.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:22:34.441 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:34.436+0000 7f557ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:34.442 INFO:tasks.rados.rados.0.smithi064.stdout:3431: done (4 left) 2024-03-04T06:22:34.442 INFO:tasks.rados.rados.0.smithi064.stdout:3433: done (3 left) 2024-03-04T06:22:34.442 INFO:tasks.rados.rados.0.smithi064.stdout:3434: done (2 left) 2024-03-04T06:22:34.442 INFO:tasks.rados.rados.0.smithi064.stdout:3435: done (1 left) 2024-03-04T06:22:34.442 INFO:tasks.rados.rados.0.smithi064.stdout:3436: done (0 left) 2024-03-04T06:22:34.442 INFO:tasks.rados.rados.0.smithi064.stdout:3437: read oid 29 snap -1 2024-03-04T06:22:34.442 INFO:tasks.rados.rados.0.smithi064.stdout:3437: expect (ObjNum 1515 snap 425 seq_num 1515) 2024-03-04T06:22:34.442 INFO:tasks.rados.rados.0.smithi064.stdout:3438: delete oid 28 current snap is 441 2024-03-04T06:22:34.444 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:34.444+0000 7f557ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:34.456 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:34.456 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:34.457 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:34.457 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:34.457 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:34.457 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:34.457 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:34.457 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:34.457 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:34.457 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:34.457 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:34.457 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:34.459 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:34.459 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:34.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:22:34.493 INFO:tasks.rados.rados.0.smithi064.stdout:3438: done (1 left) 2024-03-04T06:22:34.493 INFO:tasks.rados.rados.0.smithi064.stdout:3439: snap_create 2024-03-04T06:22:34.500 INFO:tasks.rados.rados.0.smithi064.stdout:3437: expect (ObjNum 1515 snap 425 seq_num 1515) 2024-03-04T06:22:34.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:34.564+0000 7f5d6d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:34.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:34.571+0000 7f5d6d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:34.587 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:34.588 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:34.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:22:34.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:34.736 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:34.734+0000 7fedb5024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:34.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:34.734+0000 7f2a7359d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:34.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:34.738+0000 7fedb5024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:34.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:34.742+0000 7f2a7359d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:34.758 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:34.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:22:34.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:34.862+0000 7f7cb022d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:34.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:34.870+0000 7f7cb022d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:34.882 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:34.883 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:34.885 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:35.447 INFO:tasks.rados.rados.0.smithi064.stdout:3437: done (1 left) 2024-03-04T06:22:35.447 INFO:tasks.rados.rados.0.smithi064.stdout:3439: done (0 left) 2024-03-04T06:22:35.448 INFO:tasks.rados.rados.0.smithi064.stdout:3440: delete oid 16 current snap is 442 2024-03-04T06:22:35.500 INFO:tasks.rados.rados.0.smithi064.stdout:3440: done (0 left) 2024-03-04T06:22:35.500 INFO:tasks.rados.rados.0.smithi064.stdout:3441: write oid 45 current snap is 442 2024-03-04T06:22:35.500 INFO:tasks.rados.rados.0.smithi064.stdout:3441: seq_num 1557 ranges {568499=498705,1661993=506506} 2024-03-04T06:22:35.506 INFO:tasks.rados.rados.0.smithi064.stdout:3441: writing smithi06431743-45 from 568499 to 1067204 tid 1 2024-03-04T06:22:35.513 INFO:tasks.rados.rados.0.smithi064.stdout:3441: writing smithi06431743-45 from 1661993 to 2168499 tid 2 2024-03-04T06:22:35.513 INFO:tasks.rados.rados.0.smithi064.stdout:3442: write oid 5 current snap is 442 2024-03-04T06:22:35.513 INFO:tasks.rados.rados.0.smithi064.stdout:3442: seq_num 1558 ranges {585306=573954,1720475=464831} 2024-03-04T06:22:35.519 INFO:tasks.rados.rados.0.smithi064.stdout:3442: writing smithi06431743-5 from 585306 to 1159260 tid 1 2024-03-04T06:22:35.525 INFO:tasks.rados.rados.0.smithi064.stdout:3442: writing smithi06431743-5 from 1720475 to 2185306 tid 2 2024-03-04T06:22:35.525 INFO:tasks.rados.rados.0.smithi064.stdout:3443: delete oid 10 current snap is 442 2024-03-04T06:22:35.544 INFO:tasks.rados.rados.0.smithi064.stdout:3442: finishing write tid 1 to smithi06431743-5 2024-03-04T06:22:35.573 INFO:tasks.rados.rados.0.smithi064.stdout:3442: finishing write tid 2 to smithi06431743-5 2024-03-04T06:22:35.575 INFO:tasks.rados.rados.0.smithi064.stdout:3442: finishing write tid 3 to smithi06431743-5 2024-03-04T06:22:35.576 INFO:tasks.rados.rados.0.smithi064.stdout:3442: finishing write tid 4 to smithi06431743-5 2024-03-04T06:22:35.576 INFO:tasks.rados.rados.0.smithi064.stdout:3442: oid 5 updating version 0 to 1889 2024-03-04T06:22:35.576 INFO:tasks.rados.rados.0.smithi064.stdout:3442: oid 5 updating version 1889 to 1891 2024-03-04T06:22:35.576 INFO:tasks.rados.rados.0.smithi064.stdout:3442: oid 5 version 1891 is already newer than 1891 2024-03-04T06:22:35.576 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 1891 (ObjNum 1558 snap 442 seq_num 1558) dirty exists 2024-03-04T06:22:35.576 INFO:tasks.rados.rados.0.smithi064.stdout:3442: left oid 5 (ObjNum 1558 snap 442 seq_num 1558) 2024-03-04T06:22:35.584 INFO:tasks.rados.rados.0.smithi064.stdout:3442: done (2 left) 2024-03-04T06:22:35.584 INFO:tasks.rados.rados.0.smithi064.stdout:3443: done (1 left) 2024-03-04T06:22:35.585 INFO:tasks.rados.rados.0.smithi064.stdout:3444: snap_remove snap 430 2024-03-04T06:22:36.451 INFO:tasks.rados.rados.0.smithi064.stdout:3444: done (1 left) 2024-03-04T06:22:36.451 INFO:tasks.rados.rados.0.smithi064.stdout:3445: delete oid 29 current snap is 442 2024-03-04T06:22:36.451 INFO:tasks.rados.rados.0.smithi064.stdout:3441: finishing write tid 1 to smithi06431743-45 2024-03-04T06:22:36.451 INFO:tasks.rados.rados.0.smithi064.stdout:3441: finishing write tid 2 to smithi06431743-45 2024-03-04T06:22:36.451 INFO:tasks.rados.rados.0.smithi064.stdout:3441: finishing write tid 3 to smithi06431743-45 2024-03-04T06:22:36.451 INFO:tasks.rados.rados.0.smithi064.stdout:3441: finishing write tid 4 to smithi06431743-45 2024-03-04T06:22:36.451 INFO:tasks.rados.rados.0.smithi064.stdout:3441: oid 45 updating version 0 to 1094 2024-03-04T06:22:36.451 INFO:tasks.rados.rados.0.smithi064.stdout:3441: oid 45 updating version 1094 to 1096 2024-03-04T06:22:36.451 INFO:tasks.rados.rados.0.smithi064.stdout:3441: oid 45 version 1096 is already newer than 1096 2024-03-04T06:22:36.451 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 1096 (ObjNum 1557 snap 442 seq_num 1557) dirty exists 2024-03-04T06:22:36.452 INFO:tasks.rados.rados.0.smithi064.stdout:3441: left oid 45 (ObjNum 1557 snap 442 seq_num 1557) 2024-03-04T06:22:36.525 INFO:tasks.rados.rados.0.smithi064.stdout:3441: done (1 left) 2024-03-04T06:22:36.525 INFO:tasks.rados.rados.0.smithi064.stdout:3445: done (0 left) 2024-03-04T06:22:36.525 INFO:tasks.rados.rados.0.smithi064.stdout:3446: read oid 43 snap -1 2024-03-04T06:22:36.525 INFO:tasks.rados.rados.0.smithi064.stdout:3446: expect (ObjNum 1538 snap 434 seq_num 1538) 2024-03-04T06:22:36.525 INFO:tasks.rados.rados.0.smithi064.stdout:3447: snap_create 2024-03-04T06:22:36.538 INFO:tasks.rados.rados.0.smithi064.stdout:3446: expect (ObjNum 1538 snap 434 seq_num 1538) 2024-03-04T06:22:36.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:36.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:36.910+0000 7f72ce714640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:36.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:36.914+0000 7f72ce714640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:37.461 INFO:tasks.rados.rados.0.smithi064.stdout:3446: done (1 left) 2024-03-04T06:22:37.461 INFO:tasks.rados.rados.0.smithi064.stdout:3447: done (0 left) 2024-03-04T06:22:37.461 INFO:tasks.rados.rados.0.smithi064.stdout:3448: write (excl) oid 45 current snap is 443 2024-03-04T06:22:37.461 INFO:tasks.rados.rados.0.smithi064.stdout:3448: seq_num 1561 ranges {635727=516054,1895921=339806} 2024-03-04T06:22:37.467 INFO:tasks.rados.rados.0.smithi064.stdout:3448: writing smithi06431743-45 from 635727 to 1151781 tid 1 2024-03-04T06:22:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:3448: writing smithi06431743-45 from 1895921 to 2235727 tid 2 2024-03-04T06:22:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:3449: read oid 11 snap -1 2024-03-04T06:22:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:3449: expect (ObjNum 1514 snap 424 seq_num 1514) 2024-03-04T06:22:37.473 INFO:tasks.rados.rados.0.smithi064.stdout:3450: delete oid 27 current snap is 443 2024-03-04T06:22:37.515 INFO:tasks.rados.rados.0.smithi064.stdout:3450: done (2 left) 2024-03-04T06:22:37.515 INFO:tasks.rados.rados.0.smithi064.stdout:3451: snap_remove snap 435 2024-03-04T06:22:38.326 INFO:tasks.rados.rados.0.smithi064.stdout:3451: done (2 left) 2024-03-04T06:22:38.326 INFO:tasks.rados.rados.0.smithi064.stdout:3452: write (excl) oid 4 current snap is 443 2024-03-04T06:22:38.326 INFO:tasks.rados.rados.0.smithi064.stdout:3448: finishing write tid 1 to smithi06431743-45 2024-03-04T06:22:38.326 INFO:tasks.rados.rados.0.smithi064.stdout:3448: finishing write tid 2 to smithi06431743-45 2024-03-04T06:22:38.327 INFO:tasks.rados.rados.0.smithi064.stdout:3448: finishing write tid 3 to smithi06431743-45 2024-03-04T06:22:38.327 INFO:tasks.rados.rados.0.smithi064.stdout:3448: finishing write tid 4 to smithi06431743-45 2024-03-04T06:22:38.327 INFO:tasks.rados.rados.0.smithi064.stdout:3448: oid 45 updating version 0 to 1104 2024-03-04T06:22:38.327 INFO:tasks.rados.rados.0.smithi064.stdout:3448: oid 45 version 1104 is already newer than 1103 2024-03-04T06:22:38.327 INFO:tasks.rados.rados.0.smithi064.stdout:3448: oid 45 version 1104 is already newer than 1102 2024-03-04T06:22:38.327 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 1104 (ObjNum 1561 snap 443 seq_num 1561) dirty exists 2024-03-04T06:22:38.327 INFO:tasks.rados.rados.0.smithi064.stdout:3448: left oid 45 (ObjNum 1561 snap 443 seq_num 1561) 2024-03-04T06:22:38.327 INFO:tasks.rados.rados.0.smithi064.stdout:3449: expect (ObjNum 1514 snap 424 seq_num 1514) 2024-03-04T06:22:38.533 INFO:tasks.rados.rados.0.smithi064.stdout:3452: seq_num 1563 ranges {669341=666552,2012885=256456} 2024-03-04T06:22:38.540 INFO:tasks.rados.rados.0.smithi064.stdout:3452: writing smithi06431743-4 from 669341 to 1335893 tid 1 2024-03-04T06:22:38.547 INFO:tasks.rados.rados.0.smithi064.stdout:3452: writing smithi06431743-4 from 2012885 to 2269341 tid 2 2024-03-04T06:22:38.547 INFO:tasks.rados.rados.0.smithi064.stdout:3448: done (2 left) 2024-03-04T06:22:38.547 INFO:tasks.rados.rados.0.smithi064.stdout:3449: done (1 left) 2024-03-04T06:22:38.547 INFO:tasks.rados.rados.0.smithi064.stdout:3453: write oid 34 current snap is 443 2024-03-04T06:22:38.547 INFO:tasks.rados.rados.0.smithi064.stdout:3453: seq_num 1564 ranges {686148=741801,1955014=331134} 2024-03-04T06:22:38.554 INFO:tasks.rados.rados.0.smithi064.stdout:3453: writing smithi06431743-34 from 686148 to 1427949 tid 1 2024-03-04T06:22:38.560 INFO:tasks.rados.rados.0.smithi064.stdout:3453: writing smithi06431743-34 from 1955014 to 2286148 tid 2 2024-03-04T06:22:38.560 INFO:tasks.rados.rados.0.smithi064.stdout:3454: read oid 22 snap -1 2024-03-04T06:22:38.560 INFO:tasks.rados.rados.0.smithi064.stdout:3454: expect (ObjNum 1483 snap 414 seq_num 1483) 2024-03-04T06:22:38.560 INFO:tasks.rados.rados.0.smithi064.stdout:3455: read oid 31 snap -1 2024-03-04T06:22:38.560 INFO:tasks.rados.rados.0.smithi064.stdout:3455: expect (ObjNum 1470 snap 412 seq_num 1470) 2024-03-04T06:22:38.560 INFO:tasks.rados.rados.0.smithi064.stdout:3456: read oid 6 snap -1 2024-03-04T06:22:38.560 INFO:tasks.rados.rados.0.smithi064.stdout:3456: expect (ObjNum 1539 snap 434 seq_num 1539) 2024-03-04T06:22:38.560 INFO:tasks.rados.rados.0.smithi064.stdout:3457: read oid 5 snap -1 2024-03-04T06:22:38.560 INFO:tasks.rados.rados.0.smithi064.stdout:3457: expect (ObjNum 1558 snap 442 seq_num 1558) 2024-03-04T06:22:38.560 INFO:tasks.rados.rados.0.smithi064.stdout:3458: delete oid 10 current snap is 443 2024-03-04T06:22:38.569 INFO:tasks.rados.rados.0.smithi064.stdout:3458: done (6 left) 2024-03-04T06:22:38.569 INFO:tasks.rados.rados.0.smithi064.stdout:3459: delete oid 35 current snap is 443 2024-03-04T06:22:38.583 INFO:tasks.rados.rados.0.smithi064.stdout:3456: expect (ObjNum 1539 snap 434 seq_num 1539) 2024-03-04T06:22:38.803 INFO:tasks.rados.rados.0.smithi064.stdout:3454: expect (ObjNum 1483 snap 414 seq_num 1483) 2024-03-04T06:22:38.913 INFO:tasks.rados.rados.0.smithi064.stdout:3452: finishing write tid 1 to smithi06431743-4 2024-03-04T06:22:38.942 INFO:tasks.rados.rados.0.smithi064.stdout:3455: expect (ObjNum 1470 snap 412 seq_num 1470) 2024-03-04T06:22:38.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:39.015 INFO:tasks.rados.rados.0.smithi064.stdout:3453: finishing write tid 1 to smithi06431743-34 2024-03-04T06:22:39.015 INFO:tasks.rados.rados.0.smithi064.stdout:3452: finishing write tid 2 to smithi06431743-4 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3453: finishing write tid 2 to smithi06431743-34 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3453: finishing write tid 3 to smithi06431743-34 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3453: finishing write tid 4 to smithi06431743-34 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3453: oid 34 updating version 0 to 1649 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3453: oid 34 updating version 1649 to 1650 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3453: oid 34 updating version 1650 to 1651 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 34 v 1651 (ObjNum 1564 snap 443 seq_num 1564) dirty exists 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3453: left oid 34 (ObjNum 1564 snap 443 seq_num 1564) 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3452: finishing write tid 3 to smithi06431743-4 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3452: finishing write tid 4 to smithi06431743-4 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3452: oid 4 updating version 0 to 1903 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3452: oid 4 version 1903 is already newer than 1902 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3452: oid 4 version 1903 is already newer than 1901 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 1903 (ObjNum 1563 snap 443 seq_num 1563) dirty exists 2024-03-04T06:22:39.016 INFO:tasks.rados.rados.0.smithi064.stdout:3452: left oid 4 (ObjNum 1563 snap 443 seq_num 1563) 2024-03-04T06:22:39.017 INFO:tasks.rados.rados.0.smithi064.stdout:3452: done (6 left) 2024-03-04T06:22:39.017 INFO:tasks.rados.rados.0.smithi064.stdout:3453: done (5 left) 2024-03-04T06:22:39.017 INFO:tasks.rados.rados.0.smithi064.stdout:3454: done (4 left) 2024-03-04T06:22:39.017 INFO:tasks.rados.rados.0.smithi064.stdout:3455: done (3 left) 2024-03-04T06:22:39.017 INFO:tasks.rados.rados.0.smithi064.stdout:3456: done (2 left) 2024-03-04T06:22:39.017 INFO:tasks.rados.rados.0.smithi064.stdout:3459: done (1 left) 2024-03-04T06:22:39.017 INFO:tasks.rados.rados.0.smithi064.stdout:3460: read oid 30 snap -1 2024-03-04T06:22:39.017 INFO:tasks.rados.rados.0.smithi064.stdout:3460: expect (ObjNum 1383 snap 388 seq_num 1383) 2024-03-04T06:22:39.017 INFO:tasks.rados.rados.0.smithi064.stdout:3461: snap_remove snap 432 2024-03-04T06:22:39.090 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:39.088+0000 7f5f9c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:39.093 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:39.092+0000 7f5f9c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:39.465 INFO:tasks.rados.rados.0.smithi064.stdout:3461: done (2 left) 2024-03-04T06:22:39.465 INFO:tasks.rados.rados.0.smithi064.stdout:3462: read oid 22 snap -1 2024-03-04T06:22:39.465 INFO:tasks.rados.rados.0.smithi064.stdout:3462: expect (ObjNum 1483 snap 414 seq_num 1483) 2024-03-04T06:22:39.465 INFO:tasks.rados.rados.0.smithi064.stdout:3463: delete oid 39 current snap is 443 2024-03-04T06:22:39.465 INFO:tasks.rados.rados.0.smithi064.stdout:3457: expect (ObjNum 1558 snap 442 seq_num 1558) 2024-03-04T06:22:39.568 INFO:tasks.rados.rados.0.smithi064.stdout:3460: expect (ObjNum 1383 snap 388 seq_num 1383) 2024-03-04T06:22:39.755 INFO:tasks.rados.rados.0.smithi064.stdout:3462: expect (ObjNum 1483 snap 414 seq_num 1483) 2024-03-04T06:22:39.865 INFO:tasks.rados.rados.0.smithi064.stdout:3457: done (3 left) 2024-03-04T06:22:39.865 INFO:tasks.rados.rados.0.smithi064.stdout:3460: done (2 left) 2024-03-04T06:22:39.865 INFO:tasks.rados.rados.0.smithi064.stdout:3462: done (1 left) 2024-03-04T06:22:39.865 INFO:tasks.rados.rados.0.smithi064.stdout:3463: done (0 left) 2024-03-04T06:22:39.865 INFO:tasks.rados.rados.0.smithi064.stdout:3464: read oid 3 snap 440 2024-03-04T06:22:39.865 INFO:tasks.rados.rados.0.smithi064.stdout:3464: expect (ObjNum 1452 snap 406 seq_num 1452) 2024-03-04T06:22:39.865 INFO:tasks.rados.rados.0.smithi064.stdout:3465: write oid 44 current snap is 443 2024-03-04T06:22:39.865 INFO:tasks.rados.rados.0.smithi064.stdout:3465: seq_num 1568 ranges {753376=759150,2188942=164434} 2024-03-04T06:22:39.875 INFO:tasks.rados.rados.0.smithi064.stdout:3465: writing smithi06431743-44 from 753376 to 1512526 tid 1 2024-03-04T06:22:39.880 INFO:tasks.rados.rados.0.smithi064.stdout:3465: writing smithi06431743-44 from 2188942 to 2353376 tid 2 2024-03-04T06:22:39.880 INFO:tasks.rados.rados.0.smithi064.stdout:3466: delete oid 50 current snap is 443 2024-03-04T06:22:39.880 INFO:tasks.rados.rados.0.smithi064.stdout:3464: expect (ObjNum 1452 snap 406 seq_num 1452) 2024-03-04T06:22:39.887 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:22:39.887 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:22:39.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:22:40.007 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:40.003+0000 7f481886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:40.010 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:40.010+0000 7f481886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:40.018 INFO:tasks.rados.rados.0.smithi064.stdout:3465: finishing write tid 1 to smithi06431743-44 2024-03-04T06:22:40.018 INFO:tasks.rados.rados.0.smithi064.stdout:3465: finishing write tid 2 to smithi06431743-44 2024-03-04T06:22:40.018 INFO:tasks.rados.rados.0.smithi064.stdout:3465: finishing write tid 3 to smithi06431743-44 2024-03-04T06:22:40.018 INFO:tasks.rados.rados.0.smithi064.stdout:3464: done (2 left) 2024-03-04T06:22:40.018 INFO:tasks.rados.rados.0.smithi064.stdout:3466: done (1 left) 2024-03-04T06:22:40.019 INFO:tasks.rados.rados.0.smithi064.stdout:3467: read oid 45 snap 415 2024-03-04T06:22:40.019 INFO:tasks.rados.rados.0.smithi064.stdout:3467: expect (ObjNum 1454 snap 406 seq_num 1454) 2024-03-04T06:22:40.019 INFO:tasks.rados.rados.0.smithi064.stdout:3468: write oid 47 current snap is 443 2024-03-04T06:22:40.019 INFO:tasks.rados.rados.0.smithi064.stdout:3465: finishing write tid 4 to smithi06431743-44 2024-03-04T06:22:40.019 INFO:tasks.rados.rados.0.smithi064.stdout:3465: oid 44 updating version 0 to 933 2024-03-04T06:22:40.019 INFO:tasks.rados.rados.0.smithi064.stdout:3465: oid 44 version 933 is already newer than 933 2024-03-04T06:22:40.019 INFO:tasks.rados.rados.0.smithi064.stdout:3465: oid 44 version 933 is already newer than 931 2024-03-04T06:22:40.019 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 933 (ObjNum 1568 snap 443 seq_num 1568) dirty exists 2024-03-04T06:22:40.019 INFO:tasks.rados.rados.0.smithi064.stdout:3465: left oid 44 (ObjNum 1568 snap 443 seq_num 1568) 2024-03-04T06:22:40.019 INFO:tasks.rados.rados.0.smithi064.stdout:3468: seq_num 1570 ranges {786990=509648,1905906=481084} 2024-03-04T06:22:40.027 INFO:tasks.rados.rados.0.smithi064.stdout:3468: writing smithi06431743-47 from 786990 to 1296638 tid 1 2024-03-04T06:22:40.035 INFO:tasks.rados.rados.0.smithi064.stdout:3468: writing smithi06431743-47 from 1905906 to 2386990 tid 2 2024-03-04T06:22:40.035 INFO:tasks.rados.rados.0.smithi064.stdout:3465: done (2 left) 2024-03-04T06:22:40.035 INFO:tasks.rados.rados.0.smithi064.stdout:3469: write (excl) oid 15 current snap is 443 2024-03-04T06:22:40.035 INFO:tasks.rados.rados.0.smithi064.stdout:3469: seq_num 1571 ranges {403797=584897,1564388=660878} 2024-03-04T06:22:40.041 INFO:tasks.rados.rados.0.smithi064.stdout:3469: writing smithi06431743-15 from 403797 to 988694 tid 1 2024-03-04T06:22:40.050 INFO:tasks.rados.rados.0.smithi064.stdout:3469: writing smithi06431743-15 from 1564388 to 2225266 tid 2 2024-03-04T06:22:40.050 INFO:tasks.rados.rados.0.smithi064.stdout:3470: read oid 36 snap -1 2024-03-04T06:22:40.050 INFO:tasks.rados.rados.0.smithi064.stdout:3470: expect (ObjNum 1541 snap 437 seq_num 1541) 2024-03-04T06:22:40.050 INFO:tasks.rados.rados.0.smithi064.stdout:3471: read oid 39 snap -1 2024-03-04T06:22:40.051 INFO:tasks.rados.rados.0.smithi064.stdout:3471: expect deleted 2024-03-04T06:22:40.051 INFO:tasks.rados.rados.0.smithi064.stdout:3472: delete oid 43 current snap is 443 2024-03-04T06:22:40.051 INFO:tasks.rados.rados.0.smithi064.stdout:3467: expect (ObjNum 1454 snap 406 seq_num 1454) 2024-03-04T06:22:40.174 INFO:tasks.rados.rados.0.smithi064.stdout:3468: finishing write tid 1 to smithi06431743-47 2024-03-04T06:22:40.174 INFO:tasks.rados.rados.0.smithi064.stdout:3470: expect (ObjNum 1541 snap 437 seq_num 1541) 2024-03-04T06:22:40.247 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:22:40.248 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:22:40.259 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1326,"stamp":"2024-03-04T06:22:39.464077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122353740,"num_objects":1144,"num_object_clones":934,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":171,"num_read":6252,"num_read_kb":4891803,"num_write":5323,"num_write_kb":1418607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14973,"ondisk_log_size":14973,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":614064,"kb_used_data":379088,"kb_used_omap":0,"kb_used_meta":234944,"kb_avail":376873296,"statfs":{"total":386547056640,"available":385918255104,"internally_reserved":0,"allocated":388186112,"data_stored":422501195,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":240582656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12539078,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":10,"num_read_kb":7909,"num_write":11,"num_write_kb":1866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018241"},"pg_stats":[{"pgid":"3.f","version":"872'809","reported_seq":3008,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-04T06:22:33.542172+0000","last_change":"2024-03-04T06:22:33.525336+0000","last_active":"2024-03-04T06:22:33.542172+0000","last_peered":"2024-03-04T06:22:33.542172+0000","last_clean":"2024-03-04T06:22:33.542172+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:22:33.542172+0000","last_undegraded":"2024-03-04T06:22:33.542172+0000","last_fullsized":"2024-03-04T06:22:33.542172+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7378999,"num_objects":72,"num_object_clones":57,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":337,"num_read_kb":266550,"num_write":299,"num_write_kb":78674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"882'783","reported_seq":3004,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-03-04T06:22:34.460252+0000","last_change":"2024-03-04T06:22:33.576449+0000","last_active":"2024-03-04T06:22:34.460252+0000","last_peered":"2024-03-04T06:22:34.460252+0000","last_clean":"2024-03-04T06:22:34.460252+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:22:34.460252+0000","last_undegraded":"2024-03-04T06:22:34.460252+0000","last_fullsized":"2024-03-04T06:22:34.460252+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":783,"log_dups_size":0,"ondisk_log_size":783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3781905,"num_objects":61,"num_object_clones":45,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":232,"num_read_kb":180836,"num_write":251,"num_write_kb":66356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"884'1096","reported_seq":3188,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.592435+0000","last_change":"2024-03-04T06:22:33.599675+0000","last_active":"2024-03-04T06:22:35.592435+0000","last_peered":"2024-03-04T06:22:35.592435+0000","last_clean":"2024-03-04T06:22:35.592435+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:22:35.592435+0000","last_undegraded":"2024-03-04T06:22:35.592435+0000","last_fullsized":"2024-03-04T06:22:35.592435+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1096,"log_dups_size":0,"ondisk_log_size":1096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7950833,"num_objects":90,"num_object_clones":75,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":516,"num_read_kb":411113,"num_write":404,"num_write_kb":110315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"882'928","reported_seq":2995,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-04T06:22:33.475378+0000","last_change":"2024-03-04T06:22:33.657152+0000","last_active":"2024-03-04T06:22:33.475378+0000","last_peered":"2024-03-04T06:22:33.475378+0000","last_clean":"2024-03-04T06:22:33.475378+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:22:33.475378+0000","last_undegraded":"2024-03-04T06:22:33.475378+0000","last_fullsized":"2024-03-04T06:22:33.475378+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":928,"log_dups_size":0,"ondisk_log_size":928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10707148,"num_objects":61,"num_object_clones":51,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":310,"num_read_kb":237435,"num_write":296,"num_write_kb":82903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"882'369","reported_seq":2804,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.472414+0000","last_change":"2024-03-04T06:22:33.557942+0000","last_active":"2024-03-04T06:22:35.472414+0000","last_peered":"2024-03-04T06:22:35.472414+0000","last_clean":"2024-03-04T06:22:35.472414+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:22:35.472414+0000","last_undegraded":"2024-03-04T06:22:35.472414+0000","last_fullsized":"2024-03-04T06:22:35.472414+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":106,"num_read_kb":79818,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"882'924","reported_seq":3053,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-04T06:22:33.490542+0000","last_change":"2024-03-04T06:22:33.567750+0000","last_active":"2024-03-04T06:22:33.490542+0000","last_peered":"2024-03-04T06:22:33.490542+0000","last_clean":"2024-03-04T06:22:33.490542+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:22:33.490542+0000","last_undegraded":"2024-03-04T06:22:33.490542+0000","last_fullsized":"2024-03-04T06:22:33.490542+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":924,"log_dups_size":0,"ondisk_log_size":924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":66,"num_object_clones":56,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":396,"num_read_kb":322264,"num_write":316,"num_write_kb":91407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2589,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-04T06:22:33.491945+0000","last_change":"2024-03-04T06:22:33.512290+0000","last_active":"2024-03-04T06:22:33.491945+0000","last_peered":"2024-03-04T06:22:33.491945+0000","last_clean":"2024-03-04T06:22:33.491945+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:22:33.491945+0000","last_undegraded":"2024-03-04T06:22:33.491945+0000","last_fullsized":"2024-03-04T06:22:33.491945+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"882'590","reported_seq":2890,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.525175+0000","last_change":"2024-03-04T06:22:33.535294+0000","last_active":"2024-03-04T06:22:35.525175+0000","last_peered":"2024-03-04T06:22:35.525175+0000","last_clean":"2024-03-04T06:22:35.525175+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:22:35.525175+0000","last_undegraded":"2024-03-04T06:22:35.525175+0000","last_fullsized":"2024-03-04T06:22:35.525175+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":590,"log_dups_size":0,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6807561,"num_objects":40,"num_object_clones":31,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":281,"num_read_kb":255605,"num_write":181,"num_write_kb":49146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2375,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.455554+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:22:35.455554+0000","last_peered":"2024-03-04T06:22:35.455554+0000","last_clean":"2024-03-04T06:22:35.455554+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:22:35.455554+0000","last_undegraded":"2024-03-04T06:22:35.455554+0000","last_fullsized":"2024-03-04T06:22:35.455554+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2186,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.537343+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:22:35.537343+0000","last_peered":"2024-03-04T06:22:35.537343+0000","last_clean":"2024-03-04T06:22:35.537343+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:22:35.537343+0000","last_undegraded":"2024-03-04T06:22:35.537343+0000","last_fullsized":"2024-03-04T06:22:35.537343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"882'945","reported_seq":3077,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.539961+0000","last_change":"2024-03-04T06:22:33.584810+0000","last_active":"2024-03-04T06:22:35.539961+0000","last_peered":"2024-03-04T06:22:35.539961+0000","last_clean":"2024-03-04T06:22:35.539961+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:22:35.539961+0000","last_undegraded":"2024-03-04T06:22:35.539961+0000","last_fullsized":"2024-03-04T06:22:35.539961+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":945,"log_dups_size":0,"ondisk_log_size":945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":417,"num_read_kb":336563,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2176,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-04T06:22:33.493400+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:22:33.493400+0000","last_peered":"2024-03-04T06:22:33.493400+0000","last_clean":"2024-03-04T06:22:33.493400+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:22:33.493400+0000","last_undegraded":"2024-03-04T06:22:33.493400+0000","last_fullsized":"2024-03-04T06:22:33.493400+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"882'616","reported_seq":2980,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.462299+0000","last_change":"2024-03-04T06:22:33.591427+0000","last_active":"2024-03-04T06:22:35.462299+0000","last_peered":"2024-03-04T06:22:35.462299+0000","last_clean":"2024-03-04T06:22:35.462299+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:22:35.462299+0000","last_undegraded":"2024-03-04T06:22:35.462299+0000","last_fullsized":"2024-03-04T06:22:35.462299+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3580254,"num_objects":49,"num_object_clones":40,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":306,"num_read_kb":250020,"num_write":206,"num_write_kb":58469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2186,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.529974+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:22:35.529974+0000","last_peered":"2024-03-04T06:22:35.529974+0000","last_clean":"2024-03-04T06:22:35.529974+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:22:35.529974+0000","last_undegraded":"2024-03-04T06:22:35.529974+0000","last_fullsized":"2024-03-04T06:22:35.529974+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"883'1641","reported_seq":3565,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.468111+0000","last_change":"2024-03-04T06:22:33.657171+0000","last_active":"2024-03-04T06:22:35.468111+0000","last_peered":"2024-03-04T06:22:35.468111+0000","last_clean":"2024-03-04T06:22:35.468111+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:22:35.468111+0000","last_undegraded":"2024-03-04T06:22:35.468111+0000","last_fullsized":"2024-03-04T06:22:35.468111+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1641,"log_dups_size":0,"ondisk_log_size":1641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17901270,"num_objects":133,"num_object_clones":107,"num_object_copies":399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":771,"num_read_kb":618689,"num_write":623,"num_write_kb":167821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2176,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-03-04T06:22:33.477643+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:22:33.477643+0000","last_peered":"2024-03-04T06:22:33.477643+0000","last_clean":"2024-03-04T06:22:33.477643+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:22:33.477643+0000","last_undegraded":"2024-03-04T06:22:33.477643+0000","last_fullsized":"2024-03-04T06:22:33.477643+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"882'1888","reported_seq":3549,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-03-04T06:22:34.495081+0000","last_change":"2024-03-04T06:22:33.647056+0000","last_active":"2024-03-04T06:22:34.495081+0000","last_peered":"2024-03-04T06:22:34.495081+0000","last_clean":"2024-03-04T06:22:34.495081+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:22:34.495081+0000","last_undegraded":"2024-03-04T06:22:34.495081+0000","last_fullsized":"2024-03-04T06:22:34.495081+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1888,"log_dups_size":0,"ondisk_log_size":1888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15834009,"num_objects":142,"num_object_clones":118,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":19,"num_read":662,"num_read_kb":495713,"num_write":654,"num_write_kb":176564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"882'874","reported_seq":3074,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.536045+0000","last_change":"2024-03-04T06:22:33.557412+0000","last_active":"2024-03-04T06:22:35.536045+0000","last_peered":"2024-03-04T06:22:35.536045+0000","last_clean":"2024-03-04T06:22:35.536045+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:22:35.536045+0000","last_undegraded":"2024-03-04T06:22:35.536045+0000","last_fullsized":"2024-03-04T06:22:35.536045+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7244543,"num_objects":66,"num_object_clones":53,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":424,"num_read_kb":349869,"num_write":313,"num_write_kb":85291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2261,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-03-04T06:22:34.472613+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:22:34.472613+0000","last_peered":"2024-03-04T06:22:34.472613+0000","last_clean":"2024-03-04T06:22:34.472613+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:22:34.472613+0000","last_undegraded":"2024-03-04T06:22:34.472613+0000","last_fullsized":"2024-03-04T06:22:34.472613+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"882'981","reported_seq":3140,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-03-04T06:22:34.464753+0000","last_change":"2024-03-04T06:22:33.627115+0000","last_active":"2024-03-04T06:22:34.464753+0000","last_peered":"2024-03-04T06:22:34.464753+0000","last_clean":"2024-03-04T06:22:34.464753+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:22:34.464753+0000","last_undegraded":"2024-03-04T06:22:34.464753+0000","last_fullsized":"2024-03-04T06:22:34.464753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":981,"log_dups_size":0,"ondisk_log_size":981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9413042,"num_objects":79,"num_object_clones":63,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":366,"num_read_kb":259104,"num_write":344,"num_write_kb":92772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2242,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.469123+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:22:35.469123+0000","last_peered":"2024-03-04T06:22:35.469123+0000","last_clean":"2024-03-04T06:22:35.469123+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:22:35.469123+0000","last_undegraded":"2024-03-04T06:22:35.469123+0000","last_fullsized":"2024-03-04T06:22:35.469123+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"882'1333","reported_seq":3265,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.461419+0000","last_change":"2024-03-04T06:22:33.621653+0000","last_active":"2024-03-04T06:22:35.461419+0000","last_peered":"2024-03-04T06:22:35.461419+0000","last_clean":"2024-03-04T06:22:35.461419+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:22:35.461419+0000","last_undegraded":"2024-03-04T06:22:35.461419+0000","last_fullsized":"2024-03-04T06:22:35.461419+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1333,"log_dups_size":0,"ondisk_log_size":1333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7748720,"num_objects":95,"num_object_clones":81,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":481,"num_read_kb":372501,"num_write":446,"num_write_kb":124420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2261,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-03-04T06:22:34.454963+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:22:34.454963+0000","last_peered":"2024-03-04T06:22:34.454963+0000","last_clean":"2024-03-04T06:22:34.454963+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:22:34.454963+0000","last_undegraded":"2024-03-04T06:22:34.454963+0000","last_fullsized":"2024-03-04T06:22:34.454963+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"882'1099","reported_seq":3295,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-03-04T06:22:34.453019+0000","last_change":"2024-03-04T06:22:33.667787+0000","last_active":"2024-03-04T06:22:34.453019+0000","last_peered":"2024-03-04T06:22:34.453019+0000","last_clean":"2024-03-04T06:22:34.453019+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:22:34.453019+0000","last_undegraded":"2024-03-04T06:22:34.453019+0000","last_fullsized":"2024-03-04T06:22:34.453019+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11665543,"num_objects":92,"num_object_clones":78,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":544,"num_read_kb":455632,"num_write":417,"num_write_kb":114545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2186,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-03-04T06:22:35.541939+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:22:35.541939+0000","last_peered":"2024-03-04T06:22:35.541939+0000","last_clean":"2024-03-04T06:22:35.541939+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:22:35.541939+0000","last_undegraded":"2024-03-04T06:22:35.541939+0000","last_fullsized":"2024-03-04T06:22:35.541939+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":121763372,"num_objects":1142,"num_object_clones":934,"num_object_copies":3426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":171,"num_read":6149,"num_read_kb":4891712,"num_write":5186,"num_write_kb":1416713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14876,"ondisk_log_size":14876,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706554,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":191912,"kb_used_data":124832,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94179928,"statfs":{"total":96636764160,"available":96440246272,"internally_reserved":0,"allocated":127827968,"data_stored":139894074,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706554,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":144440,"kb_used_data":87088,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94227400,"statfs":{"total":96636764160,"available":96488857600,"internally_reserved":0,"allocated":89178112,"data_stored":96739703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706553,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":141828,"kb_used_data":84476,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94230012,"statfs":{"total":96636764160,"available":96491532288,"internally_reserved":0,"allocated":86503424,"data_stored":95180517,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706553,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":135884,"kb_used_data":82692,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94235956,"statfs":{"total":96636764160,"available":96497618944,"internally_reserved":0,"allocated":84676608,"data_stored":90686901,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:22:40.395 INFO:tasks.rados.rados.0.smithi064.stdout:3469: finishing write tid 1 to smithi06431743-15 2024-03-04T06:22:40.395 INFO:tasks.rados.rados.0.smithi064.stdout:3468: finishing write tid 2 to smithi06431743-47 2024-03-04T06:22:40.395 INFO:tasks.rados.rados.0.smithi064.stdout:3468: finishing write tid 3 to smithi06431743-47 2024-03-04T06:22:40.395 INFO:tasks.rados.rados.0.smithi064.stdout:3468: finishing write tid 4 to smithi06431743-47 2024-03-04T06:22:40.395 INFO:tasks.rados.rados.0.smithi064.stdout:3468: oid 47 updating version 0 to 931 2024-03-04T06:22:40.395 INFO:tasks.rados.rados.0.smithi064.stdout:3468: oid 47 updating version 931 to 932 2024-03-04T06:22:40.395 INFO:tasks.rados.rados.0.smithi064.stdout:3468: oid 47 version 932 is already newer than 930 2024-03-04T06:22:40.395 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 932 (ObjNum 1570 snap 443 seq_num 1570) dirty exists 2024-03-04T06:22:40.395 INFO:tasks.rados.rados.0.smithi064.stdout:3468: left oid 47 (ObjNum 1570 snap 443 seq_num 1570) 2024-03-04T06:22:40.395 INFO:tasks.rados.rados.0.smithi064.stdout:3469: finishing write tid 2 to smithi06431743-15 2024-03-04T06:22:40.395 INFO:tasks.rados.rados.0.smithi064.stdout:3467: done (5 left) 2024-03-04T06:22:40.396 INFO:tasks.rados.rados.0.smithi064.stdout:3468: done (4 left) 2024-03-04T06:22:40.396 INFO:tasks.rados.rados.0.smithi064.stdout:3470: done (3 left) 2024-03-04T06:22:40.396 INFO:tasks.rados.rados.0.smithi064.stdout:3471: done (2 left) 2024-03-04T06:22:40.396 INFO:tasks.rados.rados.0.smithi064.stdout:3472: done (1 left) 2024-03-04T06:22:40.396 INFO:tasks.rados.rados.0.smithi064.stdout:3473: snap_remove snap 422 2024-03-04T06:22:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:3473: done (1 left) 2024-03-04T06:22:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:3474: write oid 48 current snap is 443 2024-03-04T06:22:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:3469: finishing write tid 3 to smithi06431743-15 2024-03-04T06:22:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:3469: finishing write tid 4 to smithi06431743-15 2024-03-04T06:22:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:3469: oid 15 updating version 0 to 880 2024-03-04T06:22:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:3469: oid 15 updating version 880 to 882 2024-03-04T06:22:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:3469: oid 15 version 882 is already newer than 881 2024-03-04T06:22:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 882 (ObjNum 1571 snap 443 seq_num 1571) dirty exists 2024-03-04T06:22:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:3469: left oid 15 (ObjNum 1571 snap 443 seq_num 1571) 2024-03-04T06:22:40.465 INFO:tasks.rados.rados.0.smithi064.stdout:3474: seq_num 1573 ranges {437411=735395,1964999=472412} 2024-03-04T06:22:40.473 INFO:tasks.rados.rados.0.smithi064.stdout:3474: writing smithi06431743-48 from 437411 to 1172806 tid 1 2024-03-04T06:22:40.482 INFO:tasks.rados.rados.0.smithi064.stdout:3474: writing smithi06431743-48 from 1964999 to 2437411 tid 2 2024-03-04T06:22:40.482 INFO:tasks.rados.rados.0.smithi064.stdout:3469: done (1 left) 2024-03-04T06:22:40.482 INFO:tasks.rados.rados.0.smithi064.stdout:3475: read oid 5 snap -1 2024-03-04T06:22:40.482 INFO:tasks.rados.rados.0.smithi064.stdout:3475: expect (ObjNum 1558 snap 442 seq_num 1558) 2024-03-04T06:22:40.483 INFO:tasks.rados.rados.0.smithi064.stdout:3476: read oid 44 snap -1 2024-03-04T06:22:40.483 INFO:tasks.rados.rados.0.smithi064.stdout:3476: expect (ObjNum 1568 snap 443 seq_num 1568) 2024-03-04T06:22:40.483 INFO:tasks.rados.rados.0.smithi064.stdout:3477: snap_remove snap 429 2024-03-04T06:22:41.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:41.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:41.271+0000 7fd603f14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:41.275 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:41.275+0000 7fd603f14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:41.467 INFO:tasks.rados.rados.0.smithi064.stdout:3477: done (3 left) 2024-03-04T06:22:41.468 INFO:tasks.rados.rados.0.smithi064.stdout:3478: snap_remove snap 426 2024-03-04T06:22:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:3478: done (3 left) 2024-03-04T06:22:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:3479: write oid 45 current snap is 443 2024-03-04T06:22:42.469 INFO:tasks.rados.rados.0.smithi064.stdout:3479: seq_num 1574 ranges {454218=526997,1739834=408205} 2024-03-04T06:22:42.474 INFO:tasks.rados.rados.0.smithi064.stdout:3479: writing smithi06431743-45 from 454218 to 981215 tid 1 2024-03-04T06:22:42.482 INFO:tasks.rados.rados.0.smithi064.stdout:3479: writing smithi06431743-45 from 1739834 to 2148039 tid 2 2024-03-04T06:22:42.482 INFO:tasks.rados.rados.0.smithi064.stdout:3480: snap_remove snap 423 2024-03-04T06:22:43.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:43.461 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:43.459+0000 7f3c5022d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:43.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:43.464+0000 7f3c5022d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:43.476 INFO:tasks.rados.rados.0.smithi064.stdout:3480: done (4 left) 2024-03-04T06:22:43.476 INFO:tasks.rados.rados.0.smithi064.stdout:3481: delete oid 36 current snap is 443 2024-03-04T06:22:43.476 INFO:tasks.rados.rados.0.smithi064.stdout:3476: expect (ObjNum 1568 snap 443 seq_num 1568) 2024-03-04T06:22:43.581 INFO:tasks.rados.rados.0.smithi064.stdout:3475: expect (ObjNum 1558 snap 442 seq_num 1558) 2024-03-04T06:22:43.662 INFO:tasks.rados.rados.0.smithi064.stdout:3474: finishing write tid 1 to smithi06431743-48 2024-03-04T06:22:43.662 INFO:tasks.rados.rados.0.smithi064.stdout:3474: finishing write tid 2 to smithi06431743-48 2024-03-04T06:22:43.662 INFO:tasks.rados.rados.0.smithi064.stdout:3474: finishing write tid 3 to smithi06431743-48 2024-03-04T06:22:43.662 INFO:tasks.rados.rados.0.smithi064.stdout:3474: finishing write tid 4 to smithi06431743-48 2024-03-04T06:22:43.662 INFO:tasks.rados.rados.0.smithi064.stdout:3474: oid 48 updating version 0 to 619 2024-03-04T06:22:43.662 INFO:tasks.rados.rados.0.smithi064.stdout:3474: oid 48 version 619 is already newer than 617 2024-03-04T06:22:43.662 INFO:tasks.rados.rados.0.smithi064.stdout:3474: oid 48 version 619 is already newer than 619 2024-03-04T06:22:43.662 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 619 (ObjNum 1573 snap 443 seq_num 1573) dirty exists 2024-03-04T06:22:43.662 INFO:tasks.rados.rados.0.smithi064.stdout:3474: left oid 48 (ObjNum 1573 snap 443 seq_num 1573) 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3479: finishing write tid 1 to smithi06431743-45 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3479: finishing write tid 2 to smithi06431743-45 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3479: finishing write tid 3 to smithi06431743-45 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3479: finishing write tid 4 to smithi06431743-45 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3479: oid 45 updating version 0 to 1123 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3479: oid 45 updating version 1123 to 1124 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3479: oid 45 updating version 1124 to 1125 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 1125 (ObjNum 1574 snap 443 seq_num 1574) dirty exists 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3479: left oid 45 (ObjNum 1574 snap 443 seq_num 1574) 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3474: done (4 left) 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3475: done (3 left) 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3476: done (2 left) 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3479: done (1 left) 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3481: done (0 left) 2024-03-04T06:22:43.663 INFO:tasks.rados.rados.0.smithi064.stdout:3482: snap_create 2024-03-04T06:22:44.479 INFO:tasks.rados.rados.0.smithi064.stdout:3482: done (0 left) 2024-03-04T06:22:44.479 INFO:tasks.rados.rados.0.smithi064.stdout:3483: write oid 29 current snap is 444 2024-03-04T06:22:44.479 INFO:tasks.rados.rados.0.smithi064.stdout:3483: seq_num 1576 ranges {487832=677495,1740445=411874} 2024-03-04T06:22:44.485 INFO:tasks.rados.rados.0.smithi064.stdout:3483: writing smithi06431743-29 from 487832 to 1165327 tid 1 2024-03-04T06:22:44.491 INFO:tasks.rados.rados.0.smithi064.stdout:3483: writing smithi06431743-29 from 1740445 to 2152319 tid 2 2024-03-04T06:22:44.491 INFO:tasks.rados.rados.0.smithi064.stdout:3484: write (excl) oid 49 current snap is 444 2024-03-04T06:22:44.491 INFO:tasks.rados.rados.0.smithi064.stdout:3484: seq_num 1577 ranges {504639=752744,1798927=671885} 2024-03-04T06:22:44.497 INFO:tasks.rados.rados.0.smithi064.stdout:3484: writing smithi06431743-49 from 504639 to 1257383 tid 1 2024-03-04T06:22:44.504 INFO:tasks.rados.rados.0.smithi064.stdout:3484: writing smithi06431743-49 from 1798927 to 2470812 tid 2 2024-03-04T06:22:44.504 INFO:tasks.rados.rados.0.smithi064.stdout:3485: snap_remove snap 436 2024-03-04T06:22:45.262 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:22:45.263 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:22:45.263 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:22:45.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:22:45.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:45.375+0000 7f367e4d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:45.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:45.383+0000 7f367e4d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:45.396 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:45.397 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:45.399 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:45.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:22:45.487 INFO:tasks.rados.rados.0.smithi064.stdout:3485: done (2 left) 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3486: delete oid 10 current snap is 444 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3483: finishing write tid 1 to smithi06431743-29 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3483: finishing write tid 2 to smithi06431743-29 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3484: finishing write tid 1 to smithi06431743-49 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3483: finishing write tid 3 to smithi06431743-29 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3484: finishing write tid 2 to smithi06431743-49 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3483: finishing write tid 4 to smithi06431743-29 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3483: oid 29 updating version 0 to 1932 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3483: oid 29 updating version 1932 to 1933 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3483: oid 29 updating version 1933 to 1934 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 1934 (ObjNum 1576 snap 444 seq_num 1576) dirty exists 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3483: left oid 29 (ObjNum 1576 snap 444 seq_num 1576) 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3484: finishing write tid 3 to smithi06431743-49 2024-03-04T06:22:45.488 INFO:tasks.rados.rados.0.smithi064.stdout:3484: finishing write tid 4 to smithi06431743-49 2024-03-04T06:22:45.489 INFO:tasks.rados.rados.0.smithi064.stdout:3484: oid 49 updating version 0 to 606 2024-03-04T06:22:45.489 INFO:tasks.rados.rados.0.smithi064.stdout:3484: oid 49 version 606 is already newer than 604 2024-03-04T06:22:45.489 INFO:tasks.rados.rados.0.smithi064.stdout:3484: oid 49 version 606 is already newer than 606 2024-03-04T06:22:45.489 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 606 (ObjNum 1577 snap 444 seq_num 1577) dirty exists 2024-03-04T06:22:45.489 INFO:tasks.rados.rados.0.smithi064.stdout:3484: left oid 49 (ObjNum 1577 snap 444 seq_num 1577) 2024-03-04T06:22:45.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:45.510+0000 7f2d3e82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:45.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:45.522+0000 7f2d3e82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:45.548 INFO:tasks.rados.rados.0.smithi064.stdout:3483: done (2 left) 2024-03-04T06:22:45.548 INFO:tasks.rados.rados.0.smithi064.stdout:3484: done (1 left) 2024-03-04T06:22:45.548 INFO:tasks.rados.rados.0.smithi064.stdout:3486: done (0 left) 2024-03-04T06:22:45.548 INFO:tasks.rados.rados.0.smithi064.stdout:3487: snap_create 2024-03-04T06:22:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:45.551 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:45.552 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:45.554 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:45.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:22:45.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:45.664 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:45.660+0000 7f3dc6d14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:45.668 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:45.668+0000 7f3dc6d14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:45.682 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:45.682 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:45.682 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:45.682 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:45.682 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:45.682 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:45.682 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:45.682 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:45.682 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:45.682 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:45.683 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:45.683 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:45.686 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:45.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:22:45.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:45.702+0000 7fd18655d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:45.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:45.706+0000 7fd18655d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:45.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:45.792+0000 7fe79d664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:45.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:45.799+0000 7fe79d664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:45.812 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:22:45.813 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:22:45.813 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:22:45.813 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:22:45.813 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:22:45.813 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:22:45.813 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:22:45.813 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:22:45.813 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:22:45.813 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:22:45.813 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:22:45.813 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:22:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:22:45.816 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:22:46.489 INFO:tasks.rados.rados.0.smithi064.stdout:3487: done (0 left) 2024-03-04T06:22:46.489 INFO:tasks.rados.rados.0.smithi064.stdout:3488: write (excl) oid 19 current snap is 445 2024-03-04T06:22:46.490 INFO:tasks.rados.rados.0.smithi064.stdout:3488: seq_num 1579 ranges {538253=503242,1515891=675554} 2024-03-04T06:22:46.495 INFO:tasks.rados.rados.0.smithi064.stdout:3488: writing smithi06431743-19 from 538253 to 1041495 tid 1 2024-03-04T06:22:46.501 INFO:tasks.rados.rados.0.smithi064.stdout:3488: writing smithi06431743-19 from 1515891 to 2191445 tid 2 2024-03-04T06:22:46.501 INFO:tasks.rados.rados.0.smithi064.stdout:3489: snap_remove snap 415 2024-03-04T06:22:47.492 INFO:tasks.rados.rados.0.smithi064.stdout:3489: done (1 left) 2024-03-04T06:22:47.492 INFO:tasks.rados.rados.0.smithi064.stdout:3490: snap_create 2024-03-04T06:22:47.492 INFO:tasks.rados.rados.0.smithi064.stdout:3488: finishing write tid 1 to smithi06431743-19 2024-03-04T06:22:47.492 INFO:tasks.rados.rados.0.smithi064.stdout:3488: finishing write tid 2 to smithi06431743-19 2024-03-04T06:22:47.492 INFO:tasks.rados.rados.0.smithi064.stdout:3488: finishing write tid 3 to smithi06431743-19 2024-03-04T06:22:47.492 INFO:tasks.rados.rados.0.smithi064.stdout:3488: finishing write tid 4 to smithi06431743-19 2024-03-04T06:22:47.492 INFO:tasks.rados.rados.0.smithi064.stdout:3488: oid 19 updating version 0 to 1944 2024-03-04T06:22:47.492 INFO:tasks.rados.rados.0.smithi064.stdout:3488: oid 19 updating version 1944 to 1945 2024-03-04T06:22:47.492 INFO:tasks.rados.rados.0.smithi064.stdout:3488: oid 19 updating version 1945 to 1946 2024-03-04T06:22:47.492 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 1946 (ObjNum 1579 snap 445 seq_num 1579) dirty exists 2024-03-04T06:22:47.492 INFO:tasks.rados.rados.0.smithi064.stdout:3488: left oid 19 (ObjNum 1579 snap 445 seq_num 1579) 2024-03-04T06:22:47.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:47.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:47.890+0000 7fdb0a15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:47.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:47.895+0000 7fdb0a15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:48.327 INFO:tasks.rados.rados.0.smithi064.stdout:3488: done (1 left) 2024-03-04T06:22:48.327 INFO:tasks.rados.rados.0.smithi064.stdout:3490: done (0 left) 2024-03-04T06:22:48.327 INFO:tasks.rados.rados.0.smithi064.stdout:3491: snap_create 2024-03-04T06:22:49.331 INFO:tasks.rados.rados.0.smithi064.stdout:3491: done (0 left) 2024-03-04T06:22:49.331 INFO:tasks.rados.rados.0.smithi064.stdout:3492: write oid 9 current snap is 447 2024-03-04T06:22:49.331 INFO:tasks.rados.rados.0.smithi064.stdout:3492: seq_num 1580 ranges {555060=578491,1858020=419212} 2024-03-04T06:22:49.337 INFO:tasks.rados.rados.0.smithi064.stdout:3492: writing smithi06431743-9 from 555060 to 1133551 tid 1 2024-03-04T06:22:49.343 INFO:tasks.rados.rados.0.smithi064.stdout:3492: writing smithi06431743-9 from 1858020 to 2277232 tid 2 2024-03-04T06:22:49.343 INFO:tasks.rados.rados.0.smithi064.stdout:3493: write (excl) oid 13 current snap is 447 2024-03-04T06:22:49.343 INFO:tasks.rados.rados.0.smithi064.stdout:3493: seq_num 1581 ranges {571867=653740,1916502=655365} 2024-03-04T06:22:49.350 INFO:tasks.rados.rados.0.smithi064.stdout:3493: writing smithi06431743-13 from 571867 to 1225607 tid 1 2024-03-04T06:22:49.357 INFO:tasks.rados.rados.0.smithi064.stdout:3493: writing smithi06431743-13 from 1916502 to 2571867 tid 2 2024-03-04T06:22:49.357 INFO:tasks.rados.rados.0.smithi064.stdout:3494: write oid 47 current snap is 447 2024-03-04T06:22:49.357 INFO:tasks.rados.rados.0.smithi064.stdout:3494: seq_num 1582 ranges {588674=445342,1691337=422881} 2024-03-04T06:22:49.363 INFO:tasks.rados.rados.0.smithi064.stdout:3494: writing smithi06431743-47 from 588674 to 1034016 tid 1 2024-03-04T06:22:49.369 INFO:tasks.rados.rados.0.smithi064.stdout:3494: writing smithi06431743-47 from 1691337 to 2114218 tid 2 2024-03-04T06:22:49.369 INFO:tasks.rados.rados.0.smithi064.stdout:3495: snap_create 2024-03-04T06:22:49.468 INFO:tasks.rados.rados.0.smithi064.stdout:3492: finishing write tid 1 to smithi06431743-9 2024-03-04T06:22:49.477 INFO:tasks.rados.rados.0.smithi064.stdout:3494: finishing write tid 1 to smithi06431743-47 2024-03-04T06:22:49.485 INFO:tasks.rados.rados.0.smithi064.stdout:3493: finishing write tid 1 to smithi06431743-13 2024-03-04T06:22:49.491 INFO:tasks.rados.rados.0.smithi064.stdout:3492: finishing write tid 2 to smithi06431743-9 2024-03-04T06:22:49.496 INFO:tasks.rados.rados.0.smithi064.stdout:3494: finishing write tid 2 to smithi06431743-47 2024-03-04T06:22:49.499 INFO:tasks.rados.rados.0.smithi064.stdout:3493: finishing write tid 2 to smithi06431743-13 2024-03-04T06:22:49.508 INFO:tasks.rados.rados.0.smithi064.stdout:3492: finishing write tid 3 to smithi06431743-9 2024-03-04T06:22:49.511 INFO:tasks.rados.rados.0.smithi064.stdout:3492: finishing write tid 4 to smithi06431743-9 2024-03-04T06:22:49.511 INFO:tasks.rados.rados.0.smithi064.stdout:3492: oid 9 updating version 0 to 1133 2024-03-04T06:22:49.512 INFO:tasks.rados.rados.0.smithi064.stdout:3492: oid 9 version 1133 is already newer than 1132 2024-03-04T06:22:49.512 INFO:tasks.rados.rados.0.smithi064.stdout:3492: oid 9 version 1133 is already newer than 1131 2024-03-04T06:22:49.512 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 1133 (ObjNum 1580 snap 447 seq_num 1580) dirty exists 2024-03-04T06:22:49.512 INFO:tasks.rados.rados.0.smithi064.stdout:3492: left oid 9 (ObjNum 1580 snap 447 seq_num 1580) 2024-03-04T06:22:49.514 INFO:tasks.rados.rados.0.smithi064.stdout:3493: finishing write tid 3 to smithi06431743-13 2024-03-04T06:22:49.516 INFO:tasks.rados.rados.0.smithi064.stdout:3493: finishing write tid 4 to smithi06431743-13 2024-03-04T06:22:49.516 INFO:tasks.rados.rados.0.smithi064.stdout:3493: oid 13 updating version 0 to 1952 2024-03-04T06:22:49.516 INFO:tasks.rados.rados.0.smithi064.stdout:3493: oid 13 updating version 1952 to 1954 2024-03-04T06:22:49.516 INFO:tasks.rados.rados.0.smithi064.stdout:3493: oid 13 version 1954 is already newer than 1953 2024-03-04T06:22:49.516 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 1954 (ObjNum 1581 snap 447 seq_num 1581) dirty exists 2024-03-04T06:22:49.516 INFO:tasks.rados.rados.0.smithi064.stdout:3493: left oid 13 (ObjNum 1581 snap 447 seq_num 1581) 2024-03-04T06:22:49.516 INFO:tasks.rados.rados.0.smithi064.stdout:3494: finishing write tid 3 to smithi06431743-47 2024-03-04T06:22:49.518 INFO:tasks.rados.rados.0.smithi064.stdout:3494: finishing write tid 4 to smithi06431743-47 2024-03-04T06:22:49.518 INFO:tasks.rados.rados.0.smithi064.stdout:3494: oid 47 updating version 0 to 944 2024-03-04T06:22:49.518 INFO:tasks.rados.rados.0.smithi064.stdout:3494: oid 47 version 944 is already newer than 943 2024-03-04T06:22:49.518 INFO:tasks.rados.rados.0.smithi064.stdout:3494: oid 47 version 944 is already newer than 943 2024-03-04T06:22:49.518 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 944 (ObjNum 1582 snap 447 seq_num 1582) dirty exists 2024-03-04T06:22:49.518 INFO:tasks.rados.rados.0.smithi064.stdout:3494: left oid 47 (ObjNum 1582 snap 447 seq_num 1582) 2024-03-04T06:22:49.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:50.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:50.072+0000 7ffbc1d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:50.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:50.076+0000 7ffbc1d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:50.331 INFO:tasks.rados.rados.0.smithi064.stdout:3492: done (3 left) 2024-03-04T06:22:50.331 INFO:tasks.rados.rados.0.smithi064.stdout:3493: done (2 left) 2024-03-04T06:22:50.331 INFO:tasks.rados.rados.0.smithi064.stdout:3494: done (1 left) 2024-03-04T06:22:50.332 INFO:tasks.rados.rados.0.smithi064.stdout:3495: done (0 left) 2024-03-04T06:22:50.332 INFO:tasks.rados.rados.0.smithi064.stdout:3496: read oid 11 snap -1 2024-03-04T06:22:50.332 INFO:tasks.rados.rados.0.smithi064.stdout:3496: expect (ObjNum 1514 snap 424 seq_num 1514) 2024-03-04T06:22:50.332 INFO:tasks.rados.rados.0.smithi064.stdout:3497: write (excl) oid 18 current snap is 448 2024-03-04T06:22:50.332 INFO:tasks.rados.rados.0.smithi064.stdout:3497: seq_num 1583 ranges {605481=520591,1749819=682892} 2024-03-04T06:22:50.339 INFO:tasks.rados.rados.0.smithi064.stdout:3497: writing smithi06431743-18 from 605481 to 1126072 tid 1 2024-03-04T06:22:50.348 INFO:tasks.rados.rados.0.smithi064.stdout:3497: writing smithi06431743-18 from 1749819 to 2432711 tid 2 2024-03-04T06:22:50.348 INFO:tasks.rados.rados.0.smithi064.stdout:3498: snap_create 2024-03-04T06:22:50.381 INFO:tasks.rados.rados.0.smithi064.stdout:3496: expect (ObjNum 1514 snap 424 seq_num 1514) 2024-03-04T06:22:50.555 INFO:tasks.rados.rados.0.smithi064.stdout:3497: finishing write tid 1 to smithi06431743-18 2024-03-04T06:22:50.556 INFO:tasks.rados.rados.0.smithi064.stdout:3497: finishing write tid 2 to smithi06431743-18 2024-03-04T06:22:50.556 INFO:tasks.rados.rados.0.smithi064.stdout:3497: finishing write tid 3 to smithi06431743-18 2024-03-04T06:22:50.556 INFO:tasks.rados.rados.0.smithi064.stdout:3497: finishing write tid 4 to smithi06431743-18 2024-03-04T06:22:50.556 INFO:tasks.rados.rados.0.smithi064.stdout:3497: oid 18 updating version 0 to 786 2024-03-04T06:22:50.556 INFO:tasks.rados.rados.0.smithi064.stdout:3497: oid 18 version 786 is already newer than 785 2024-03-04T06:22:50.556 INFO:tasks.rados.rados.0.smithi064.stdout:3497: oid 18 updating version 786 to 787 2024-03-04T06:22:50.556 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 18 v 787 (ObjNum 1583 snap 448 seq_num 1583) dirty exists 2024-03-04T06:22:50.556 INFO:tasks.rados.rados.0.smithi064.stdout:3497: left oid 18 (ObjNum 1583 snap 448 seq_num 1583) 2024-03-04T06:22:50.816 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:22:50.816 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:22:50.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:22:50.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:50.921+0000 7efc1df5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:50.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:50.927+0000 7efc1df5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:22:51.118 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:22:51.129 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1343,"stamp":"2024-03-04T06:22:50.897041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":106398834,"num_objects":1162,"num_object_clones":951,"num_object_copies":3486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":174,"num_read":6356,"num_read_kb":4979526,"num_write":5392,"num_write_kb":1435482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15309,"ondisk_log_size":15309,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":644096,"kb_used_data":404960,"kb_used_omap":0,"kb_used_meta":239104,"kb_avail":376843264,"statfs":{"total":386547056640,"available":385887502336,"internally_reserved":0,"allocated":414679040,"data_stored":453956960,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":244842496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2365516,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":18,"num_write_kb":5431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000983"},"pg_stats":[{"pgid":"3.f","version":"898'829","reported_seq":3058,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.366163+0000","last_change":"2024-03-04T06:22:49.428004+0000","last_active":"2024-03-04T06:22:50.366163+0000","last_peered":"2024-03-04T06:22:50.366163+0000","last_clean":"2024-03-04T06:22:50.366163+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:22:50.366163+0000","last_undegraded":"2024-03-04T06:22:50.366163+0000","last_fullsized":"2024-03-04T06:22:50.366163+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":829,"log_dups_size":0,"ondisk_log_size":829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3529833,"num_objects":73,"num_object_clones":58,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":343,"num_read_kb":273127,"num_write":300,"num_write_kb":78674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"899'785","reported_seq":3057,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.374108+0000","last_change":"2024-03-04T06:22:49.393299+0000","last_active":"2024-03-04T06:22:50.374108+0000","last_peered":"2024-03-04T06:22:50.374108+0000","last_clean":"2024-03-04T06:22:50.374108+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:22:50.374108+0000","last_undegraded":"2024-03-04T06:22:50.374108+0000","last_fullsized":"2024-03-04T06:22:50.374108+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":785,"log_dups_size":0,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6214616,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":233,"num_read_kb":180836,"num_write":253,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"894'1141","reported_seq":3236,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-03-04T06:22:45.565522+0000","last_change":"2024-03-04T06:22:45.565522+0000","last_active":"2024-03-04T06:22:45.565522+0000","last_peered":"2024-03-04T06:22:45.565522+0000","last_clean":"2024-03-04T06:22:45.565522+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:22:45.565522+0000","last_undegraded":"2024-03-04T06:22:45.565522+0000","last_fullsized":"2024-03-04T06:22:45.565522+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1141,"log_dups_size":0,"ondisk_log_size":1141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5900016,"num_objects":92,"num_object_clones":77,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":532,"num_read_kb":423196,"num_write":413,"num_write_kb":112065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"898'944","reported_seq":3049,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.359254+0000","last_change":"2024-03-04T06:22:49.519544+0000","last_active":"2024-03-04T06:22:50.359254+0000","last_peered":"2024-03-04T06:22:50.359254+0000","last_clean":"2024-03-04T06:22:50.359254+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:22:50.359254+0000","last_undegraded":"2024-03-04T06:22:50.359254+0000","last_fullsized":"2024-03-04T06:22:50.359254+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":944,"log_dups_size":0,"ondisk_log_size":944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9631533,"num_objects":63,"num_object_clones":53,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":312,"num_read_kb":237437,"num_write":304,"num_write_kb":84719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"882'369","reported_seq":2832,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-04T06:22:44.400233+0000","last_change":"2024-03-04T06:22:44.407027+0000","last_active":"2024-03-04T06:22:44.400233+0000","last_peered":"2024-03-04T06:22:44.400233+0000","last_clean":"2024-03-04T06:22:44.400233+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:22:44.400233+0000","last_undegraded":"2024-03-04T06:22:44.400233+0000","last_fullsized":"2024-03-04T06:22:44.400233+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":112,"num_read_kb":87747,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"894'940","reported_seq":3106,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.368271+0000","last_change":"2024-03-04T06:22:49.390929+0000","last_active":"2024-03-04T06:22:50.368271+0000","last_peered":"2024-03-04T06:22:50.368271+0000","last_clean":"2024-03-04T06:22:50.368271+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:22:50.368271+0000","last_undegraded":"2024-03-04T06:22:50.368271+0000","last_fullsized":"2024-03-04T06:22:50.368271+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":940,"log_dups_size":0,"ondisk_log_size":940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353376,"num_objects":66,"num_object_clones":56,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":403,"num_read_kb":327765,"num_write":320,"num_write_kb":92310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2635,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.370146+0000","last_change":"2024-03-04T06:22:49.392222+0000","last_active":"2024-03-04T06:22:50.370146+0000","last_peered":"2024-03-04T06:22:50.370146+0000","last_clean":"2024-03-04T06:22:50.370146+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:22:50.370146+0000","last_undegraded":"2024-03-04T06:22:50.370146+0000","last_fullsized":"2024-03-04T06:22:50.370146+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"894'608","reported_seq":2928,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-03-04T06:22:45.502275+0000","last_change":"2024-03-04T06:22:45.542054+0000","last_active":"2024-03-04T06:22:45.502275+0000","last_peered":"2024-03-04T06:22:45.502275+0000","last_clean":"2024-03-04T06:22:45.502275+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:22:45.502275+0000","last_undegraded":"2024-03-04T06:22:45.502275+0000","last_fullsized":"2024-03-04T06:22:45.502275+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5362192,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":283,"num_read_kb":255606,"num_write":185,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2393,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-04T06:22:44.371057+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:22:44.371057+0000","last_peered":"2024-03-04T06:22:44.371057+0000","last_clean":"2024-03-04T06:22:44.371057+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:22:44.371057+0000","last_undegraded":"2024-03-04T06:22:44.371057+0000","last_fullsized":"2024-03-04T06:22:44.371057+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2212,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-03-04T06:22:45.523720+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:22:45.523720+0000","last_peered":"2024-03-04T06:22:45.523720+0000","last_clean":"2024-03-04T06:22:45.523720+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:22:45.523720+0000","last_undegraded":"2024-03-04T06:22:45.523720+0000","last_fullsized":"2024-03-04T06:22:45.523720+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"882'945","reported_seq":3114,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-03-04T06:22:45.521491+0000","last_change":"2024-03-04T06:22:45.531770+0000","last_active":"2024-03-04T06:22:45.521491+0000","last_peered":"2024-03-04T06:22:45.521491+0000","last_clean":"2024-03-04T06:22:45.521491+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:22:45.521491+0000","last_undegraded":"2024-03-04T06:22:45.521491+0000","last_fullsized":"2024-03-04T06:22:45.521491+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":945,"log_dups_size":0,"ondisk_log_size":945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":423,"num_read_kb":341165,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2213,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.370534+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:22:50.370534+0000","last_peered":"2024-03-04T06:22:50.370534+0000","last_clean":"2024-03-04T06:22:50.370534+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:22:50.370534+0000","last_undegraded":"2024-03-04T06:22:50.370534+0000","last_fullsized":"2024-03-04T06:22:50.370534+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"889'619","reported_seq":3009,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-04T06:22:44.389463+0000","last_change":"2024-03-04T06:22:44.409811+0000","last_active":"2024-03-04T06:22:44.389463+0000","last_peered":"2024-03-04T06:22:44.389463+0000","last_clean":"2024-03-04T06:22:44.389463+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:22:44.389463+0000","last_undegraded":"2024-03-04T06:22:44.389463+0000","last_fullsized":"2024-03-04T06:22:44.389463+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":619,"log_dups_size":0,"ondisk_log_size":619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017665,"num_objects":49,"num_object_clones":40,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":307,"num_read_kb":250021,"num_write":210,"num_write_kb":59650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2212,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-03-04T06:22:45.503347+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:22:45.503347+0000","last_peered":"2024-03-04T06:22:45.503347+0000","last_clean":"2024-03-04T06:22:45.503347+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:22:45.503347+0000","last_undegraded":"2024-03-04T06:22:45.503347+0000","last_fullsized":"2024-03-04T06:22:45.503347+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"892'1671","reported_seq":3593,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-04T06:22:44.590897+0000","last_change":"2024-03-04T06:22:44.621245+0000","last_active":"2024-03-04T06:22:44.590897+0000","last_peered":"2024-03-04T06:22:44.590897+0000","last_clean":"2024-03-04T06:22:44.590897+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:22:44.590897+0000","last_undegraded":"2024-03-04T06:22:44.590897+0000","last_fullsized":"2024-03-04T06:22:44.590897+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17027339,"num_objects":134,"num_object_clones":108,"num_object_copies":402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":772,"num_read_kb":618690,"num_write":627,"num_write_kb":168870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2213,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.357404+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:22:50.357404+0000","last_peered":"2024-03-04T06:22:50.357404+0000","last_clean":"2024-03-04T06:22:50.357404+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:22:50.357404+0000","last_undegraded":"2024-03-04T06:22:50.357404+0000","last_fullsized":"2024-03-04T06:22:50.357404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"898'1960","reported_seq":3630,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.351556+0000","last_change":"2024-03-04T06:22:49.561488+0000","last_active":"2024-03-04T06:22:50.351556+0000","last_peered":"2024-03-04T06:22:50.351556+0000","last_clean":"2024-03-04T06:22:50.351556+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:22:50.351556+0000","last_undegraded":"2024-03-04T06:22:50.351556+0000","last_fullsized":"2024-03-04T06:22:50.351556+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1960,"log_dups_size":0,"ondisk_log_size":1960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15296680,"num_objects":146,"num_object_clones":122,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":689,"num_read_kb":513489,"num_write":675,"num_write_kb":181978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"894'896","reported_seq":3115,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-03-04T06:22:45.516018+0000","last_change":"2024-03-04T06:22:45.548717+0000","last_active":"2024-03-04T06:22:45.516018+0000","last_peered":"2024-03-04T06:22:45.516018+0000","last_clean":"2024-03-04T06:22:45.516018+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:22:45.516018+0000","last_undegraded":"2024-03-04T06:22:45.516018+0000","last_fullsized":"2024-03-04T06:22:45.516018+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269770,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":432,"num_read_kb":357436,"num_write":317,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2303,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.366752+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:22:50.366752+0000","last_peered":"2024-03-04T06:22:50.366752+0000","last_clean":"2024-03-04T06:22:50.366752+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:22:50.366752+0000","last_undegraded":"2024-03-04T06:22:50.366752+0000","last_fullsized":"2024-03-04T06:22:50.366752+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"898'1017","reported_seq":3196,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.367416+0000","last_change":"2024-03-04T06:22:49.469616+0000","last_active":"2024-03-04T06:22:50.367416+0000","last_peered":"2024-03-04T06:22:50.367416+0000","last_clean":"2024-03-04T06:22:50.367416+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:22:50.367416+0000","last_undegraded":"2024-03-04T06:22:50.367416+0000","last_fullsized":"2024-03-04T06:22:50.367416+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1017,"log_dups_size":0,"ondisk_log_size":1017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3412184,"num_objects":81,"num_object_clones":65,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":373,"num_read_kb":269282,"num_write":346,"num_write_kb":92772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2260,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-04T06:22:44.384799+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:22:44.384799+0000","last_peered":"2024-03-04T06:22:44.384799+0000","last_clean":"2024-03-04T06:22:44.384799+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:22:44.384799+0000","last_undegraded":"2024-03-04T06:22:44.384799+0000","last_fullsized":"2024-03-04T06:22:44.384799+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"892'1355","reported_seq":3290,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-04T06:22:44.570575+0000","last_change":"2024-03-04T06:22:44.601932+0000","last_active":"2024-03-04T06:22:44.570575+0000","last_peered":"2024-03-04T06:22:44.570575+0000","last_clean":"2024-03-04T06:22:44.570575+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:22:44.570575+0000","last_undegraded":"2024-03-04T06:22:44.570575+0000","last_fullsized":"2024-03-04T06:22:44.570575+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3916394,"num_objects":96,"num_object_clones":82,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":481,"num_read_kb":372501,"num_write":447,"num_write_kb":124420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2303,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.356299+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:22:50.356299+0000","last_peered":"2024-03-04T06:22:50.356299+0000","last_clean":"2024-03-04T06:22:50.356299+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:22:50.356299+0000","last_undegraded":"2024-03-04T06:22:50.356299+0000","last_fullsized":"2024-03-04T06:22:50.356299+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"898'1133","reported_seq":3357,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.361609+0000","last_change":"2024-03-04T06:22:49.514573+0000","last_active":"2024-03-04T06:22:50.361609+0000","last_peered":"2024-03-04T06:22:50.361609+0000","last_clean":"2024-03-04T06:22:50.361609+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:22:50.361609+0000","last_undegraded":"2024-03-04T06:22:50.361609+0000","last_fullsized":"2024-03-04T06:22:50.361609+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1133,"log_dups_size":0,"ondisk_log_size":1133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9127323,"num_objects":94,"num_object_clones":80,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":558,"num_read_kb":471137,"num_write":422,"num_write_kb":115520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2212,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-03-04T06:22:45.525553+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:22:45.525553+0000","last_peered":"2024-03-04T06:22:45.525553+0000","last_clean":"2024-03-04T06:22:45.525553+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:22:45.525553+0000","last_undegraded":"2024-03-04T06:22:45.525553+0000","last_fullsized":"2024-03-04T06:22:45.525553+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":105808466,"num_objects":1160,"num_object_clones":951,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":174,"num_read":6253,"num_read_kb":4979435,"num_write":5255,"num_write_kb":1433588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15212,"ondisk_log_size":15212,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706564,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":198596,"kb_used_data":131516,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94173244,"statfs":{"total":96636764160,"available":96433401856,"internally_reserved":0,"allocated":134672384,"data_stored":147515361,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706564,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":149096,"kb_used_data":91744,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94222744,"statfs":{"total":96636764160,"available":96484089856,"internally_reserved":0,"allocated":93945856,"data_stored":100974506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":149152,"kb_used_data":91800,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94222688,"statfs":{"total":96636764160,"available":96484032512,"internally_reserved":0,"allocated":94003200,"data_stored":104661748,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":147252,"kb_used_data":89900,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94224588,"statfs":{"total":96636764160,"available":96485978112,"internally_reserved":0,"allocated":92057600,"data_stored":100805345,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:22:51.336 INFO:tasks.rados.rados.0.smithi064.stdout:3496: done (2 left) 2024-03-04T06:22:51.336 INFO:tasks.rados.rados.0.smithi064.stdout:3497: done (1 left) 2024-03-04T06:22:51.336 INFO:tasks.rados.rados.0.smithi064.stdout:3498: done (0 left) 2024-03-04T06:22:51.336 INFO:tasks.rados.rados.0.smithi064.stdout:3499: snap_create 2024-03-04T06:22:52.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:52.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:52.257+0000 7f871ef9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:52.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:52.261+0000 7f871ef9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:52.336 INFO:tasks.rados.rados.0.smithi064.stdout:3499: done (0 left) 2024-03-04T06:22:52.337 INFO:tasks.rados.rados.0.smithi064.stdout:3500: write (excl) oid 2 current snap is 450 2024-03-04T06:22:52.337 INFO:tasks.rados.rados.0.smithi064.stdout:3500: seq_num 1584 ranges {622288=595840,1691948=426550} 2024-03-04T06:22:52.343 INFO:tasks.rados.rados.0.smithi064.stdout:3500: writing smithi06431743-2 from 622288 to 1218128 tid 1 2024-03-04T06:22:52.348 INFO:tasks.rados.rados.0.smithi064.stdout:3500: writing smithi06431743-2 from 1691948 to 2118498 tid 2 2024-03-04T06:22:52.349 INFO:tasks.rados.rados.0.smithi064.stdout:3501: snap_create 2024-03-04T06:22:52.404 INFO:tasks.rados.rados.0.smithi064.stdout:3500: finishing write tid 1 to smithi06431743-2 2024-03-04T06:22:52.416 INFO:tasks.rados.rados.0.smithi064.stdout:3500: finishing write tid 2 to smithi06431743-2 2024-03-04T06:22:52.423 INFO:tasks.rados.rados.0.smithi064.stdout:3500: finishing write tid 3 to smithi06431743-2 2024-03-04T06:22:52.424 INFO:tasks.rados.rados.0.smithi064.stdout:3500: finishing write tid 4 to smithi06431743-2 2024-03-04T06:22:52.424 INFO:tasks.rados.rados.0.smithi064.stdout:3500: oid 2 updating version 0 to 947 2024-03-04T06:22:52.424 INFO:tasks.rados.rados.0.smithi064.stdout:3500: oid 2 version 947 is already newer than 946 2024-03-04T06:22:52.424 INFO:tasks.rados.rados.0.smithi064.stdout:3500: oid 2 updating version 947 to 948 2024-03-04T06:22:52.424 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 948 (ObjNum 1584 snap 450 seq_num 1584) dirty exists 2024-03-04T06:22:52.424 INFO:tasks.rados.rados.0.smithi064.stdout:3500: left oid 2 (ObjNum 1584 snap 450 seq_num 1584) 2024-03-04T06:22:53.348 INFO:tasks.rados.rados.0.smithi064.stdout:3500: done (1 left) 2024-03-04T06:22:53.348 INFO:tasks.rados.rados.0.smithi064.stdout:3501: done (0 left) 2024-03-04T06:22:53.349 INFO:tasks.rados.rados.0.smithi064.stdout:3502: write (excl) oid 30 current snap is 451 2024-03-04T06:22:53.349 INFO:tasks.rados.rados.0.smithi064.stdout:3502: seq_num 1585 ranges {639095=671089,1750430=570208} 2024-03-04T06:22:53.355 INFO:tasks.rados.rados.0.smithi064.stdout:3502: writing smithi06431743-30 from 639095 to 1310184 tid 1 2024-03-04T06:22:53.362 INFO:tasks.rados.rados.0.smithi064.stdout:3502: writing smithi06431743-30 from 1750430 to 2320638 tid 2 2024-03-04T06:22:53.362 INFO:tasks.rados.rados.0.smithi064.stdout:3503: delete oid 20 current snap is 451 2024-03-04T06:22:53.401 INFO:tasks.rados.rados.0.smithi064.stdout:3503: done (1 left) 2024-03-04T06:22:53.401 INFO:tasks.rados.rados.0.smithi064.stdout:3504: read oid 24 snap -1 2024-03-04T06:22:53.401 INFO:tasks.rados.rados.0.smithi064.stdout:3504: expect (ObjNum 1542 snap 437 seq_num 1542) 2024-03-04T06:22:53.401 INFO:tasks.rados.rados.0.smithi064.stdout:3505: snap_remove snap 416 2024-03-04T06:22:54.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:3505: done (2 left) 2024-03-04T06:22:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:3506: delete oid 11 current snap is 451 2024-03-04T06:22:54.352 INFO:tasks.rados.rados.0.smithi064.stdout:3504: expect (ObjNum 1542 snap 437 seq_num 1542) 2024-03-04T06:22:54.482 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:54.480+0000 7f0c86f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:54.486 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:54.486+0000 7f0c86f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:54.563 INFO:tasks.rados.rados.0.smithi064.stdout:3502: finishing write tid 1 to smithi06431743-30 2024-03-04T06:22:54.563 INFO:tasks.rados.rados.0.smithi064.stdout:3502: finishing write tid 2 to smithi06431743-30 2024-03-04T06:22:54.563 INFO:tasks.rados.rados.0.smithi064.stdout:3502: finishing write tid 3 to smithi06431743-30 2024-03-04T06:22:54.564 INFO:tasks.rados.rados.0.smithi064.stdout:3502: finishing write tid 4 to smithi06431743-30 2024-03-04T06:22:54.564 INFO:tasks.rados.rados.0.smithi064.stdout:3502: oid 30 updating version 0 to 1963 2024-03-04T06:22:54.564 INFO:tasks.rados.rados.0.smithi064.stdout:3502: oid 30 version 1963 is already newer than 1962 2024-03-04T06:22:54.564 INFO:tasks.rados.rados.0.smithi064.stdout:3502: oid 30 updating version 1963 to 1964 2024-03-04T06:22:54.564 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 1964 (ObjNum 1585 snap 451 seq_num 1585) dirty exists 2024-03-04T06:22:54.564 INFO:tasks.rados.rados.0.smithi064.stdout:3502: left oid 30 (ObjNum 1585 snap 451 seq_num 1585) 2024-03-04T06:22:54.564 INFO:tasks.rados.rados.0.smithi064.stdout:3502: done (2 left) 2024-03-04T06:22:54.564 INFO:tasks.rados.rados.0.smithi064.stdout:3504: done (1 left) 2024-03-04T06:22:54.564 INFO:tasks.rados.rados.0.smithi064.stdout:3506: done (0 left) 2024-03-04T06:22:54.564 INFO:tasks.rados.rados.0.smithi064.stdout:3507: write (excl) oid 33 current snap is 451 2024-03-04T06:22:54.564 INFO:tasks.rados.rados.0.smithi064.stdout:3507: seq_num 1588 ranges {689516=496836,1809523=433888} 2024-03-04T06:22:54.571 INFO:tasks.rados.rados.0.smithi064.stdout:3507: writing smithi06431743-33 from 689516 to 1186352 tid 1 2024-03-04T06:22:54.577 INFO:tasks.rados.rados.0.smithi064.stdout:3507: writing smithi06431743-33 from 1809523 to 2243411 tid 2 2024-03-04T06:22:54.577 INFO:tasks.rados.rados.0.smithi064.stdout:3508: snap_create 2024-03-04T06:22:54.629 INFO:tasks.rados.rados.0.smithi064.stdout:3507: finishing write tid 1 to smithi06431743-33 2024-03-04T06:22:54.646 INFO:tasks.rados.rados.0.smithi064.stdout:3507: finishing write tid 2 to smithi06431743-33 2024-03-04T06:22:54.658 INFO:tasks.rados.rados.0.smithi064.stdout:3507: finishing write tid 3 to smithi06431743-33 2024-03-04T06:22:54.659 INFO:tasks.rados.rados.0.smithi064.stdout:3507: finishing write tid 4 to smithi06431743-33 2024-03-04T06:22:54.659 INFO:tasks.rados.rados.0.smithi064.stdout:3507: oid 33 updating version 0 to 1140 2024-03-04T06:22:54.659 INFO:tasks.rados.rados.0.smithi064.stdout:3507: oid 33 version 1140 is already newer than 1139 2024-03-04T06:22:54.659 INFO:tasks.rados.rados.0.smithi064.stdout:3507: oid 33 updating version 1140 to 1141 2024-03-04T06:22:54.659 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 1141 (ObjNum 1588 snap 451 seq_num 1588) dirty exists 2024-03-04T06:22:54.659 INFO:tasks.rados.rados.0.smithi064.stdout:3507: left oid 33 (ObjNum 1588 snap 451 seq_num 1588) 2024-03-04T06:22:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:3507: done (1 left) 2024-03-04T06:22:55.354 INFO:tasks.rados.rados.0.smithi064.stdout:3508: done (0 left) 2024-03-04T06:22:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:3509: write (excl) oid 31 current snap is 452 2024-03-04T06:22:55.355 INFO:tasks.rados.rados.0.smithi064.stdout:3509: seq_num 1589 ranges {706323=688438,1984358=577546} 2024-03-04T06:22:55.362 INFO:tasks.rados.rados.0.smithi064.stdout:3509: writing smithi06431743-31 from 706323 to 1394761 tid 1 2024-03-04T06:22:55.370 INFO:tasks.rados.rados.0.smithi064.stdout:3509: writing smithi06431743-31 from 1984358 to 2561904 tid 2 2024-03-04T06:22:55.370 INFO:tasks.rados.rados.0.smithi064.stdout:3510: snap_remove snap 445 2024-03-04T06:22:56.131 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:22:56.131 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:22:56.132 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:22:56.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:56.243+0000 7f0863e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:56.250 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:56.249+0000 7f0863e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:56.356 INFO:tasks.rados.rados.0.smithi064.stdout:3510: done (1 left) 2024-03-04T06:22:56.356 INFO:tasks.rados.rados.0.smithi064.stdout:3511: snap_remove snap 418 2024-03-04T06:22:56.480 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:22:56.480 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:22:56.492 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1351,"stamp":"2024-03-04T06:22:56.356354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103815388,"num_objects":1165,"num_object_clones":954,"num_object_copies":3495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":174,"num_read":6369,"num_read_kb":4986874,"num_write":5406,"num_write_kb":1438604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15387,"ondisk_log_size":15387,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":641056,"kb_used_data":401592,"kb_used_omap":0,"kb_used_meta":239104,"kb_avail":376846304,"statfs":{"total":386547056640,"available":385890615296,"internally_reserved":0,"allocated":411230208,"data_stored":451896887,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":244842496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2583446,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":14,"num_write_kb":3122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020581"},"pg_stats":[{"pgid":"3.f","version":"903'833","reported_seq":3070,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-03-04T06:22:55.376389+0000","last_change":"2024-03-04T06:22:54.577430+0000","last_active":"2024-03-04T06:22:55.376389+0000","last_peered":"2024-03-04T06:22:55.376389+0000","last_clean":"2024-03-04T06:22:55.376389+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:22:55.376389+0000","last_undegraded":"2024-03-04T06:22:55.376389+0000","last_fullsized":"2024-03-04T06:22:55.376389+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":833,"log_dups_size":0,"ondisk_log_size":833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3529833,"num_objects":73,"num_object_clones":58,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":343,"num_read_kb":273127,"num_write":300,"num_write_kb":78674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"899'787","reported_seq":3070,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-03-04T06:22:54.503087+0000","last_change":"2024-03-04T06:22:54.515064+0000","last_active":"2024-03-04T06:22:54.503087+0000","last_peered":"2024-03-04T06:22:54.503087+0000","last_clean":"2024-03-04T06:22:54.503087+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:22:54.503087+0000","last_undegraded":"2024-03-04T06:22:54.503087+0000","last_fullsized":"2024-03-04T06:22:54.503087+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":234,"num_read_kb":180837,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"897'1149","reported_seq":3254,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.376337+0000","last_change":"2024-03-04T06:22:48.387389+0000","last_active":"2024-03-04T06:22:50.376337+0000","last_peered":"2024-03-04T06:22:50.376337+0000","last_clean":"2024-03-04T06:22:50.376337+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:22:50.376337+0000","last_undegraded":"2024-03-04T06:22:50.376337+0000","last_fullsized":"2024-03-04T06:22:50.376337+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1149,"log_dups_size":0,"ondisk_log_size":1149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5900016,"num_objects":92,"num_object_clones":77,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":538,"num_read_kb":430540,"num_write":413,"num_write_kb":112065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"903'952","reported_seq":3065,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-03-04T06:22:55.363572+0000","last_change":"2024-03-04T06:22:54.585039+0000","last_active":"2024-03-04T06:22:55.363572+0000","last_peered":"2024-03-04T06:22:55.363572+0000","last_clean":"2024-03-04T06:22:55.363572+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:22:55.363572+0000","last_undegraded":"2024-03-04T06:22:55.363572+0000","last_fullsized":"2024-03-04T06:22:55.363572+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":952,"log_dups_size":0,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8438269,"num_objects":64,"num_object_clones":54,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":314,"num_read_kb":237438,"num_write":308,"num_write_kb":85718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"882'369","reported_seq":2854,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-03-04T06:22:51.358780+0000","last_change":"2024-03-04T06:22:47.520275+0000","last_active":"2024-03-04T06:22:51.358780+0000","last_peered":"2024-03-04T06:22:51.358780+0000","last_clean":"2024-03-04T06:22:51.358780+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:22:51.358780+0000","last_undegraded":"2024-03-04T06:22:51.358780+0000","last_fullsized":"2024-03-04T06:22:51.358780+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706290,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":112,"num_read_kb":87747,"num_write":115,"num_write_kb":31854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"894'940","reported_seq":3118,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-03-04T06:22:55.375655+0000","last_change":"2024-03-04T06:22:54.520068+0000","last_active":"2024-03-04T06:22:55.375655+0000","last_peered":"2024-03-04T06:22:55.375655+0000","last_clean":"2024-03-04T06:22:55.375655+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:22:55.375655+0000","last_undegraded":"2024-03-04T06:22:55.375655+0000","last_fullsized":"2024-03-04T06:22:55.375655+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":940,"log_dups_size":0,"ondisk_log_size":940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353376,"num_objects":66,"num_object_clones":56,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":403,"num_read_kb":327765,"num_write":320,"num_write_kb":92310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2647,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-03-04T06:22:55.378187+0000","last_change":"2024-03-04T06:22:54.540980+0000","last_active":"2024-03-04T06:22:55.378187+0000","last_peered":"2024-03-04T06:22:55.378187+0000","last_clean":"2024-03-04T06:22:55.378187+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:22:55.378187+0000","last_undegraded":"2024-03-04T06:22:55.378187+0000","last_fullsized":"2024-03-04T06:22:55.378187+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"894'608","reported_seq":2942,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.348600+0000","last_change":"2024-03-04T06:22:48.349399+0000","last_active":"2024-03-04T06:22:50.348600+0000","last_peered":"2024-03-04T06:22:50.348600+0000","last_clean":"2024-03-04T06:22:50.348600+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:22:50.348600+0000","last_undegraded":"2024-03-04T06:22:50.348600+0000","last_fullsized":"2024-03-04T06:22:50.348600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":608,"log_dups_size":0,"ondisk_log_size":608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5362192,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":283,"num_read_kb":255606,"num_write":185,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2413,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-03-04T06:22:51.349084+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:22:51.349084+0000","last_peered":"2024-03-04T06:22:51.349084+0000","last_clean":"2024-03-04T06:22:51.349084+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:22:51.349084+0000","last_undegraded":"2024-03-04T06:22:51.349084+0000","last_fullsized":"2024-03-04T06:22:51.349084+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2225,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.368398+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:22:50.368398+0000","last_peered":"2024-03-04T06:22:50.368398+0000","last_clean":"2024-03-04T06:22:50.368398+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:22:50.368398+0000","last_undegraded":"2024-03-04T06:22:50.368398+0000","last_fullsized":"2024-03-04T06:22:50.368398+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"897'947","reported_seq":3128,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.357553+0000","last_change":"2024-03-04T06:22:48.373224+0000","last_active":"2024-03-04T06:22:50.357553+0000","last_peered":"2024-03-04T06:22:50.357553+0000","last_clean":"2024-03-04T06:22:50.357553+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:22:50.357553+0000","last_undegraded":"2024-03-04T06:22:50.357553+0000","last_fullsized":"2024-03-04T06:22:50.357553+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":423,"num_read_kb":341165,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2224,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-03-04T06:22:55.373092+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:22:55.373092+0000","last_peered":"2024-03-04T06:22:55.373092+0000","last_clean":"2024-03-04T06:22:55.373092+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:22:55.373092+0000","last_undegraded":"2024-03-04T06:22:55.373092+0000","last_fullsized":"2024-03-04T06:22:55.373092+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"896'621","reported_seq":3031,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-03-04T06:22:51.347574+0000","last_change":"2024-03-04T06:22:47.541660+0000","last_active":"2024-03-04T06:22:51.347574+0000","last_peered":"2024-03-04T06:22:51.347574+0000","last_clean":"2024-03-04T06:22:51.347574+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:22:51.347574+0000","last_undegraded":"2024-03-04T06:22:51.347574+0000","last_fullsized":"2024-03-04T06:22:51.347574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017665,"num_objects":49,"num_object_clones":40,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":307,"num_read_kb":250021,"num_write":210,"num_write_kb":59650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2225,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.351145+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:22:50.351145+0000","last_peered":"2024-03-04T06:22:50.351145+0000","last_clean":"2024-03-04T06:22:50.351145+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:22:50.351145+0000","last_undegraded":"2024-03-04T06:22:50.351145+0000","last_fullsized":"2024-03-04T06:22:50.351145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"896'1683","reported_seq":3616,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-03-04T06:22:51.360199+0000","last_change":"2024-03-04T06:22:47.574547+0000","last_active":"2024-03-04T06:22:51.360199+0000","last_peered":"2024-03-04T06:22:51.360199+0000","last_clean":"2024-03-04T06:22:51.360199+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:22:51.360199+0000","last_undegraded":"2024-03-04T06:22:51.360199+0000","last_fullsized":"2024-03-04T06:22:51.360199+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1683,"log_dups_size":0,"ondisk_log_size":1683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17027339,"num_objects":134,"num_object_clones":108,"num_object_copies":402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":772,"num_read_kb":618690,"num_write":627,"num_write_kb":168870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2224,"reported_epoch":904,"state":"active+clean","last_fresh":"2024-03-04T06:22:55.366823+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:22:55.366823+0000","last_peered":"2024-03-04T06:22:55.366823+0000","last_clean":"2024-03-04T06:22:55.366823+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:22:55.366823+0000","last_undegraded":"2024-03-04T06:22:55.366823+0000","last_fullsized":"2024-03-04T06:22:55.366823+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"903'1974","reported_seq":3645,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-03-04T06:22:54.504738+0000","last_change":"2024-03-04T06:22:54.655691+0000","last_active":"2024-03-04T06:22:54.504738+0000","last_peered":"2024-03-04T06:22:54.504738+0000","last_clean":"2024-03-04T06:22:54.504738+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:22:54.504738+0000","last_undegraded":"2024-03-04T06:22:54.504738+0000","last_fullsized":"2024-03-04T06:22:54.504738+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1974,"log_dups_size":0,"ondisk_log_size":1974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14691694,"num_objects":147,"num_object_clones":123,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":691,"num_read_kb":513490,"num_write":679,"num_write_kb":183191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"897'898","reported_seq":3129,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.355992+0000","last_change":"2024-03-04T06:22:48.375394+0000","last_active":"2024-03-04T06:22:50.355992+0000","last_peered":"2024-03-04T06:22:50.355992+0000","last_clean":"2024-03-04T06:22:50.355992+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:22:50.355992+0000","last_undegraded":"2024-03-04T06:22:50.355992+0000","last_fullsized":"2024-03-04T06:22:50.355992+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":898,"log_dups_size":0,"ondisk_log_size":898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269770,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":432,"num_read_kb":357436,"num_write":317,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2313,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-03-04T06:22:54.510383+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:22:54.510383+0000","last_peered":"2024-03-04T06:22:54.510383+0000","last_clean":"2024-03-04T06:22:54.510383+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:22:54.510383+0000","last_undegraded":"2024-03-04T06:22:54.510383+0000","last_fullsized":"2024-03-04T06:22:54.510383+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"903'1021","reported_seq":3207,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-03-04T06:22:54.511379+0000","last_change":"2024-03-04T06:22:54.574441+0000","last_active":"2024-03-04T06:22:54.511379+0000","last_peered":"2024-03-04T06:22:54.511379+0000","last_clean":"2024-03-04T06:22:54.511379+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:22:54.511379+0000","last_undegraded":"2024-03-04T06:22:54.511379+0000","last_fullsized":"2024-03-04T06:22:54.511379+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1021,"log_dups_size":0,"ondisk_log_size":1021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3412184,"num_objects":81,"num_object_clones":65,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":373,"num_read_kb":269282,"num_write":346,"num_write_kb":92772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2280,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-03-04T06:22:51.358854+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:22:51.358854+0000","last_peered":"2024-03-04T06:22:51.358854+0000","last_clean":"2024-03-04T06:22:51.358854+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:22:51.358854+0000","last_undegraded":"2024-03-04T06:22:51.358854+0000","last_fullsized":"2024-03-04T06:22:51.358854+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"896'1367","reported_seq":3313,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-03-04T06:22:51.346030+0000","last_change":"2024-03-04T06:22:47.570197+0000","last_active":"2024-03-04T06:22:51.346030+0000","last_peered":"2024-03-04T06:22:51.346030+0000","last_clean":"2024-03-04T06:22:51.346030+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:22:51.346030+0000","last_undegraded":"2024-03-04T06:22:51.346030+0000","last_fullsized":"2024-03-04T06:22:51.346030+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1367,"log_dups_size":0,"ondisk_log_size":1367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3916394,"num_objects":96,"num_object_clones":82,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":481,"num_read_kb":372501,"num_write":447,"num_write_kb":124420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2313,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-03-04T06:22:54.507422+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:22:54.507422+0000","last_peered":"2024-03-04T06:22:54.507422+0000","last_clean":"2024-03-04T06:22:54.507422+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:22:54.507422+0000","last_undegraded":"2024-03-04T06:22:54.507422+0000","last_fullsized":"2024-03-04T06:22:54.507422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"903'1141","reported_seq":3372,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-03-04T06:22:54.647912+0000","last_change":"2024-03-04T06:22:54.580779+0000","last_active":"2024-03-04T06:22:54.647912+0000","last_peered":"2024-03-04T06:22:54.647912+0000","last_clean":"2024-03-04T06:22:54.647912+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:22:54.647912+0000","last_undegraded":"2024-03-04T06:22:54.647912+0000","last_fullsized":"2024-03-04T06:22:54.647912+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1141,"log_dups_size":0,"ondisk_log_size":1141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8169357,"num_objects":95,"num_object_clones":81,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":560,"num_read_kb":471138,"num_write":426,"num_write_kb":116430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2225,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-03-04T06:22:50.354828+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:22:50.354828+0000","last_peered":"2024-03-04T06:22:50.354828+0000","last_clean":"2024-03-04T06:22:50.354828+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:22:50.354828+0000","last_undegraded":"2024-03-04T06:22:50.354828+0000","last_fullsized":"2024-03-04T06:22:50.354828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":103225020,"num_objects":1163,"num_object_clones":954,"num_object_copies":3489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":174,"num_read":6266,"num_read_kb":4986783,"num_write":5269,"num_write_kb":1436710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15290,"ondisk_log_size":15290,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706569,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":199640,"kb_used_data":132560,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94172200,"statfs":{"total":96636764160,"available":96432332800,"internally_reserved":0,"allocated":135741440,"data_stored":149589295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706569,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":150088,"kb_used_data":92736,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94221752,"statfs":{"total":96636764160,"available":96483074048,"internally_reserved":0,"allocated":94961664,"data_stored":101818911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":148212,"kb_used_data":90636,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94223628,"statfs":{"total":96636764160,"available":96484995072,"internally_reserved":0,"allocated":92811264,"data_stored":103984507,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":143116,"kb_used_data":85660,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94228724,"statfs":{"total":96636764160,"available":96490213376,"internally_reserved":0,"allocated":87715840,"data_stored":96504174,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:22:56.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:22:56.753 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:56.751+0000 7f362842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:56.755 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:56.755+0000 7f362842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:57.361 INFO:tasks.rados.rados.0.smithi064.stdout:3511: done (1 left) 2024-03-04T06:22:57.361 INFO:tasks.rados.rados.0.smithi064.stdout:3512: snap_create 2024-03-04T06:22:57.361 INFO:tasks.rados.rados.0.smithi064.stdout:3509: finishing write tid 1 to smithi06431743-31 2024-03-04T06:22:57.362 INFO:tasks.rados.rados.0.smithi064.stdout:3509: finishing write tid 2 to smithi06431743-31 2024-03-04T06:22:57.362 INFO:tasks.rados.rados.0.smithi064.stdout:3509: finishing write tid 3 to smithi06431743-31 2024-03-04T06:22:57.362 INFO:tasks.rados.rados.0.smithi064.stdout:3509: finishing write tid 4 to smithi06431743-31 2024-03-04T06:22:57.362 INFO:tasks.rados.rados.0.smithi064.stdout:3509: oid 31 updating version 0 to 373 2024-03-04T06:22:57.362 INFO:tasks.rados.rados.0.smithi064.stdout:3509: oid 31 version 373 is already newer than 372 2024-03-04T06:22:57.362 INFO:tasks.rados.rados.0.smithi064.stdout:3509: oid 31 version 373 is already newer than 371 2024-03-04T06:22:57.362 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 373 (ObjNum 1589 snap 452 seq_num 1589) dirty exists 2024-03-04T06:22:57.362 INFO:tasks.rados.rados.0.smithi064.stdout:3509: left oid 31 (ObjNum 1589 snap 452 seq_num 1589) 2024-03-04T06:22:58.362 INFO:tasks.rados.rados.0.smithi064.stdout:3509: done (1 left) 2024-03-04T06:22:58.362 INFO:tasks.rados.rados.0.smithi064.stdout:3512: done (0 left) 2024-03-04T06:22:58.362 INFO:tasks.rados.rados.0.smithi064.stdout:3513: read oid 37 snap -1 2024-03-04T06:22:58.362 INFO:tasks.rados.rados.0.smithi064.stdout:3513: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:22:58.362 INFO:tasks.rados.rados.0.smithi064.stdout:3514: write (excl) oid 10 current snap is 453 2024-03-04T06:22:58.362 INFO:tasks.rados.rados.0.smithi064.stdout:3514: seq_num 1590 ranges {723130=763687,2042840=437557} 2024-03-04T06:22:58.370 INFO:tasks.rados.rados.0.smithi064.stdout:3514: writing smithi06431743-10 from 723130 to 1486817 tid 1 2024-03-04T06:22:58.375 INFO:tasks.rados.rados.0.smithi064.stdout:3514: writing smithi06431743-10 from 2042840 to 2480397 tid 2 2024-03-04T06:22:58.376 INFO:tasks.rados.rados.0.smithi064.stdout:3515: write (excl) oid 46 current snap is 453 2024-03-04T06:22:58.376 INFO:tasks.rados.rados.0.smithi064.stdout:3515: seq_num 1591 ranges {739937=438936,1701322=581215} 2024-03-04T06:22:58.382 INFO:tasks.rados.rados.0.smithi064.stdout:3515: writing smithi06431743-46 from 739937 to 1178873 tid 1 2024-03-04T06:22:58.389 INFO:tasks.rados.rados.0.smithi064.stdout:3515: writing smithi06431743-46 from 1701322 to 2282537 tid 2 2024-03-04T06:22:58.389 INFO:tasks.rados.rados.0.smithi064.stdout:3516: write oid 28 current snap is 453 2024-03-04T06:22:58.389 INFO:tasks.rados.rados.0.smithi064.stdout:3516: seq_num 1592 ranges {756744=514185,2043451=441226} 2024-03-04T06:22:58.395 INFO:tasks.rados.rados.0.smithi064.stdout:3516: writing smithi06431743-28 from 756744 to 1270929 tid 1 2024-03-04T06:22:58.403 INFO:tasks.rados.rados.0.smithi064.stdout:3516: writing smithi06431743-28 from 2043451 to 2484677 tid 2 2024-03-04T06:22:58.404 INFO:tasks.rados.rados.0.smithi064.stdout:3517: read oid 31 snap -1 2024-03-04T06:22:58.404 INFO:tasks.rados.rados.0.smithi064.stdout:3517: expect (ObjNum 1589 snap 452 seq_num 1589) 2024-03-04T06:22:58.404 INFO:tasks.rados.rados.0.smithi064.stdout:3518: write oid 25 current snap is 453 2024-03-04T06:22:58.404 INFO:tasks.rados.rados.0.smithi064.stdout:3518: seq_num 1593 ranges {773551=589434,2101933=584884} 2024-03-04T06:22:58.413 INFO:tasks.rados.rados.0.smithi064.stdout:3518: writing smithi06431743-25 from 773551 to 1362985 tid 1 2024-03-04T06:22:58.422 INFO:tasks.rados.rados.0.smithi064.stdout:3518: writing smithi06431743-25 from 2101933 to 2686817 tid 2 2024-03-04T06:22:58.422 INFO:tasks.rados.rados.0.smithi064.stdout:3519: write (excl) oid 26 current snap is 453 2024-03-04T06:22:58.422 INFO:tasks.rados.rados.0.smithi064.stdout:3519: seq_num 1594 ranges {790358=664683,2160415=444895} 2024-03-04T06:22:58.432 INFO:tasks.rados.rados.0.smithi064.stdout:3519: writing smithi06431743-26 from 790358 to 1455041 tid 1 2024-03-04T06:22:58.439 INFO:tasks.rados.rados.0.smithi064.stdout:3519: writing smithi06431743-26 from 2160415 to 2605310 tid 2 2024-03-04T06:22:58.439 INFO:tasks.rados.rados.0.smithi064.stdout:3520: read oid 35 snap -1 2024-03-04T06:22:58.439 INFO:tasks.rados.rados.0.smithi064.stdout:3520: expect deleted 2024-03-04T06:22:58.439 INFO:tasks.rados.rados.0.smithi064.stdout:3521: read oid 15 snap 444 2024-03-04T06:22:58.439 INFO:tasks.rados.rados.0.smithi064.stdout:3521: expect (ObjNum 1571 snap 443 seq_num 1571) 2024-03-04T06:22:58.440 INFO:tasks.rados.rados.0.smithi064.stdout:3522: snap_remove snap 451 2024-03-04T06:22:58.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:22:58.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:58.930+0000 7f985d19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:58.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:22:58.934+0000 7f985d19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:22:59.367 INFO:tasks.rados.rados.0.smithi064.stdout:3522: done (9 left) 2024-03-04T06:22:59.367 INFO:tasks.rados.rados.0.smithi064.stdout:3523: snap_remove snap 441 2024-03-04T06:23:00.369 INFO:tasks.rados.rados.0.smithi064.stdout:3523: done (9 left) 2024-03-04T06:23:00.369 INFO:tasks.rados.rados.0.smithi064.stdout:3524: write (excl) oid 5 current snap is 453 2024-03-04T06:23:00.369 INFO:tasks.rados.rados.0.smithi064.stdout:3524: seq_num 1595 ranges {407165=739932,1818897=588553} 2024-03-04T06:23:00.374 INFO:tasks.rados.rados.0.smithi064.stdout:3524: writing smithi06431743-5 from 407165 to 1147097 tid 1 2024-03-04T06:23:00.382 INFO:tasks.rados.rados.0.smithi064.stdout:3524: writing smithi06431743-5 from 1818897 to 2407450 tid 2 2024-03-04T06:23:00.382 INFO:tasks.rados.rados.0.smithi064.stdout:3525: write oid 14 current snap is 453 2024-03-04T06:23:00.382 INFO:tasks.rados.rados.0.smithi064.stdout:3525: seq_num 1596 ranges {423972=415181,1361026=448564,2561920=262052} 2024-03-04T06:23:00.386 INFO:tasks.rados.rados.0.smithi064.stdout:3525: writing smithi06431743-14 from 423972 to 839153 tid 1 2024-03-04T06:23:00.392 INFO:tasks.rados.rados.0.smithi064.stdout:3525: writing smithi06431743-14 from 1361026 to 1809590 tid 2 2024-03-04T06:23:00.398 INFO:tasks.rados.rados.0.smithi064.stdout:3525: writing smithi06431743-14 from 2561920 to 2823972 tid 3 2024-03-04T06:23:00.398 INFO:tasks.rados.rados.0.smithi064.stdout:3526: write (excl) oid 38 current snap is 453 2024-03-04T06:23:00.398 INFO:tasks.rados.rados.0.smithi064.stdout:3526: seq_num 1597 ranges {440779=606783,1535861=592222,2589343=251436} 2024-03-04T06:23:00.404 INFO:tasks.rados.rados.0.smithi064.stdout:3526: writing smithi06431743-38 from 440779 to 1047562 tid 1 2024-03-04T06:23:00.411 INFO:tasks.rados.rados.0.smithi064.stdout:3526: writing smithi06431743-38 from 1535861 to 2128083 tid 2 2024-03-04T06:23:00.416 INFO:tasks.rados.rados.0.smithi064.stdout:3526: writing smithi06431743-38 from 2589343 to 2840779 tid 3 2024-03-04T06:23:00.416 INFO:tasks.rados.rados.0.smithi064.stdout:3527: write (excl) oid 45 current snap is 453 2024-03-04T06:23:00.417 INFO:tasks.rados.rados.0.smithi064.stdout:3527: seq_num 1598 ranges {457586=682032,1594343=735880} 2024-03-04T06:23:00.424 INFO:tasks.rados.rados.0.smithi064.stdout:3527: writing smithi06431743-45 from 457586 to 1139618 tid 1 2024-03-04T06:23:00.434 INFO:tasks.rados.rados.0.smithi064.stdout:3527: writing smithi06431743-45 from 1594343 to 2330223 tid 2 2024-03-04T06:23:00.434 INFO:tasks.rados.rados.0.smithi064.stdout:3528: delete oid 1 current snap is 453 2024-03-04T06:23:00.434 INFO:tasks.rados.rados.0.smithi064.stdout:3513: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:23:00.604 INFO:tasks.rados.rados.0.smithi064.stdout:3514: finishing write tid 1 to smithi06431743-10 2024-03-04T06:23:00.604 INFO:tasks.rados.rados.0.smithi064.stdout:3518: finishing write tid 1 to smithi06431743-25 2024-03-04T06:23:00.604 INFO:tasks.rados.rados.0.smithi064.stdout:3517: expect (ObjNum 1589 snap 452 seq_num 1589) 2024-03-04T06:23:00.750 INFO:tasks.rados.rados.0.smithi064.stdout:3515: finishing write tid 1 to smithi06431743-46 2024-03-04T06:23:00.750 INFO:tasks.rados.rados.0.smithi064.stdout:3518: finishing write tid 2 to smithi06431743-25 2024-03-04T06:23:00.750 INFO:tasks.rados.rados.0.smithi064.stdout:3514: finishing write tid 2 to smithi06431743-10 2024-03-04T06:23:00.750 INFO:tasks.rados.rados.0.smithi064.stdout:3521: expect (ObjNum 1571 snap 443 seq_num 1571) 2024-03-04T06:23:00.874 INFO:tasks.rados.rados.0.smithi064.stdout:3518: finishing write tid 3 to smithi06431743-25 2024-03-04T06:23:00.874 INFO:tasks.rados.rados.0.smithi064.stdout:3515: finishing write tid 2 to smithi06431743-46 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3514: finishing write tid 3 to smithi06431743-10 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3518: finishing write tid 4 to smithi06431743-25 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3518: oid 25 updating version 0 to 1171 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3518: oid 25 updating version 1171 to 1172 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3518: oid 25 version 1172 is already newer than 1170 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 1172 (ObjNum 1593 snap 453 seq_num 1593) dirty exists 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3518: left oid 25 (ObjNum 1593 snap 453 seq_num 1593) 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3514: finishing write tid 4 to smithi06431743-10 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3514: oid 10 updating version 0 to 942 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3514: oid 10 version 942 is already newer than 941 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3514: oid 10 updating version 942 to 943 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 943 (ObjNum 1590 snap 453 seq_num 1590) dirty exists 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3514: left oid 10 (ObjNum 1590 snap 453 seq_num 1590) 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3515: finishing write tid 3 to smithi06431743-46 2024-03-04T06:23:00.875 INFO:tasks.rados.rados.0.smithi064.stdout:3513: done (13 left) 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3514: done (12 left) 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3517: done (11 left) 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3518: done (10 left) 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3520: done (9 left) 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3521: done (8 left) 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3528: done (7 left) 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3529: write (excl) oid 13 current snap is 453 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3515: finishing write tid 4 to smithi06431743-46 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3515: oid 46 updating version 0 to 1701 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3515: oid 46 updating version 1701 to 1703 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3515: oid 46 version 1703 is already newer than 1703 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 1703 (ObjNum 1591 snap 453 seq_num 1591) dirty exists 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3515: left oid 46 (ObjNum 1591 snap 453 seq_num 1591) 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3516: finishing write tid 1 to smithi06431743-28 2024-03-04T06:23:00.876 INFO:tasks.rados.rados.0.smithi064.stdout:3516: finishing write tid 2 to smithi06431743-28 2024-03-04T06:23:00.877 INFO:tasks.rados.rados.0.smithi064.stdout:3516: finishing write tid 3 to smithi06431743-28 2024-03-04T06:23:00.877 INFO:tasks.rados.rados.0.smithi064.stdout:3516: finishing write tid 4 to smithi06431743-28 2024-03-04T06:23:00.877 INFO:tasks.rados.rados.0.smithi064.stdout:3516: oid 28 updating version 0 to 1705 2024-03-04T06:23:00.877 INFO:tasks.rados.rados.0.smithi064.stdout:3516: oid 28 updating version 1705 to 1706 2024-03-04T06:23:00.877 INFO:tasks.rados.rados.0.smithi064.stdout:3516: oid 28 version 1706 is already newer than 1705 2024-03-04T06:23:00.877 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 1706 (ObjNum 1592 snap 453 seq_num 1592) dirty exists 2024-03-04T06:23:00.877 INFO:tasks.rados.rados.0.smithi064.stdout:3516: left oid 28 (ObjNum 1592 snap 453 seq_num 1592) 2024-03-04T06:23:00.877 INFO:tasks.rados.rados.0.smithi064.stdout:3519: finishing write tid 1 to smithi06431743-26 2024-03-04T06:23:00.877 INFO:tasks.rados.rados.0.smithi064.stdout:3529: seq_num 1600 ranges {491200=432530,1594954=739549} 2024-03-04T06:23:00.879 INFO:tasks.rados.rados.0.smithi064.stdout:3529: writing smithi06431743-13 from 491200 to 923730 tid 1 2024-03-04T06:23:00.887 INFO:tasks.rados.rados.0.smithi064.stdout:3529: writing smithi06431743-13 from 1594954 to 2334503 tid 2 2024-03-04T06:23:00.887 INFO:tasks.rados.rados.0.smithi064.stdout:3515: done (7 left) 2024-03-04T06:23:00.887 INFO:tasks.rados.rados.0.smithi064.stdout:3516: done (6 left) 2024-03-04T06:23:00.887 INFO:tasks.rados.rados.0.smithi064.stdout:3530: write oid 42 current snap is 453 2024-03-04T06:23:00.887 INFO:tasks.rados.rados.0.smithi064.stdout:3519: finishing write tid 2 to smithi06431743-26 2024-03-04T06:23:00.887 INFO:tasks.rados.rados.0.smithi064.stdout:3519: finishing write tid 3 to smithi06431743-26 2024-03-04T06:23:00.887 INFO:tasks.rados.rados.0.smithi064.stdout:3519: finishing write tid 4 to smithi06431743-26 2024-03-04T06:23:00.887 INFO:tasks.rados.rados.0.smithi064.stdout:3519: oid 26 updating version 0 to 1710 2024-03-04T06:23:00.888 INFO:tasks.rados.rados.0.smithi064.stdout:3519: oid 26 version 1710 is already newer than 1710 2024-03-04T06:23:00.888 INFO:tasks.rados.rados.0.smithi064.stdout:3519: oid 26 version 1710 is already newer than 1708 2024-03-04T06:23:00.888 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 1710 (ObjNum 1594 snap 453 seq_num 1594) dirty exists 2024-03-04T06:23:00.888 INFO:tasks.rados.rados.0.smithi064.stdout:3519: left oid 26 (ObjNum 1594 snap 453 seq_num 1594) 2024-03-04T06:23:00.888 INFO:tasks.rados.rados.0.smithi064.stdout:3530: seq_num 1601 ranges {508007=507779,1653436=599560} 2024-03-04T06:23:00.892 INFO:tasks.rados.rados.0.smithi064.stdout:3530: writing smithi06431743-42 from 508007 to 1015786 tid 1 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3530: writing smithi06431743-42 from 1653436 to 2252996 tid 2 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3519: done (6 left) 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3531: write (excl) oid 40 current snap is 453 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3525: finishing write tid 1 to smithi06431743-14 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3526: finishing write tid 1 to smithi06431743-38 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3524: finishing write tid 1 to smithi06431743-5 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3524: finishing write tid 2 to smithi06431743-5 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3525: finishing write tid 2 to smithi06431743-14 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3524: finishing write tid 3 to smithi06431743-5 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3524: finishing write tid 4 to smithi06431743-5 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3524: oid 5 updating version 0 to 1996 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3524: oid 5 version 1996 is already newer than 1994 2024-03-04T06:23:00.900 INFO:tasks.rados.rados.0.smithi064.stdout:3524: oid 5 version 1996 is already newer than 1996 2024-03-04T06:23:00.901 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 1996 (ObjNum 1595 snap 453 seq_num 1595) dirty exists 2024-03-04T06:23:00.901 INFO:tasks.rados.rados.0.smithi064.stdout:3524: left oid 5 (ObjNum 1595 snap 453 seq_num 1595) 2024-03-04T06:23:00.901 INFO:tasks.rados.rados.0.smithi064.stdout:3531: seq_num 1602 ranges {524814=583028,1711918=743218} 2024-03-04T06:23:00.905 INFO:tasks.rados.rados.0.smithi064.stdout:3531: writing smithi06431743-40 from 524814 to 1107842 tid 1 2024-03-04T06:23:00.913 INFO:tasks.rados.rados.0.smithi064.stdout:3531: writing smithi06431743-40 from 1711918 to 2455136 tid 2 2024-03-04T06:23:00.913 INFO:tasks.rados.rados.0.smithi064.stdout:3524: done (6 left) 2024-03-04T06:23:00.913 INFO:tasks.rados.rados.0.smithi064.stdout:3532: snap_remove snap 437 2024-03-04T06:23:01.002 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:01.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:01.105+0000 7f57f311d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:01.110 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:01.110+0000 7f57f311d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:01.383 INFO:tasks.rados.rados.0.smithi064.stdout:3532: done (6 left) 2024-03-04T06:23:01.384 INFO:tasks.rados.rados.0.smithi064.stdout:3533: read oid 27 snap -1 2024-03-04T06:23:01.384 INFO:tasks.rados.rados.0.smithi064.stdout:3533: expect deleted 2024-03-04T06:23:01.384 INFO:tasks.rados.rados.0.smithi064.stdout:3534: write (excl) oid 7 current snap is 453 2024-03-04T06:23:01.384 INFO:tasks.rados.rados.0.smithi064.stdout:3526: finishing write tid 2 to smithi06431743-38 2024-03-04T06:23:01.384 INFO:tasks.rados.rados.0.smithi064.stdout:3526: finishing write tid 3 to smithi06431743-38 2024-03-04T06:23:01.384 INFO:tasks.rados.rados.0.smithi064.stdout:3527: finishing write tid 1 to smithi06431743-45 2024-03-04T06:23:01.384 INFO:tasks.rados.rados.0.smithi064.stdout:3525: finishing write tid 3 to smithi06431743-14 2024-03-04T06:23:01.384 INFO:tasks.rados.rados.0.smithi064.stdout:3534: seq_num 1603 ranges {541621=658277,1770400=603229} 2024-03-04T06:23:01.389 INFO:tasks.rados.rados.0.smithi064.stdout:3534: writing smithi06431743-7 from 541621 to 1199898 tid 1 2024-03-04T06:23:01.398 INFO:tasks.rados.rados.0.smithi064.stdout:3534: writing smithi06431743-7 from 1770400 to 2373629 tid 2 2024-03-04T06:23:01.398 INFO:tasks.rados.rados.0.smithi064.stdout:3535: snap_create 2024-03-04T06:23:01.398 INFO:tasks.rados.rados.0.smithi064.stdout:3525: finishing write tid 4 to smithi06431743-14 2024-03-04T06:23:01.398 INFO:tasks.rados.rados.0.smithi064.stdout:3525: finishing write tid 5 to smithi06431743-14 2024-03-04T06:23:01.398 INFO:tasks.rados.rados.0.smithi064.stdout:3525: oid 14 updating version 0 to 1385 2024-03-04T06:23:01.398 INFO:tasks.rados.rados.0.smithi064.stdout:3525: oid 14 version 1385 is already newer than 1382 2024-03-04T06:23:01.398 INFO:tasks.rados.rados.0.smithi064.stdout:3525: oid 14 version 1385 is already newer than 1384 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3525: oid 14 version 1385 is already newer than 1383 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 1385 (ObjNum 1596 snap 453 seq_num 1596) dirty exists 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3525: left oid 14 (ObjNum 1596 snap 453 seq_num 1596) 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3526: finishing write tid 4 to smithi06431743-38 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3526: finishing write tid 5 to smithi06431743-38 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3526: oid 38 updating version 0 to 944 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3526: oid 38 updating version 944 to 945 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3526: oid 38 updating version 945 to 947 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3526: oid 38 version 947 is already newer than 946 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 947 (ObjNum 1597 snap 453 seq_num 1597) dirty exists 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3526: left oid 38 (ObjNum 1597 snap 453 seq_num 1597) 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3527: finishing write tid 2 to smithi06431743-45 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3527: finishing write tid 3 to smithi06431743-45 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3527: finishing write tid 4 to smithi06431743-45 2024-03-04T06:23:01.399 INFO:tasks.rados.rados.0.smithi064.stdout:3527: oid 45 updating version 0 to 1179 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:3527: oid 45 updating version 1179 to 1180 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:3527: oid 45 version 1180 is already newer than 1178 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 1180 (ObjNum 1598 snap 453 seq_num 1598) dirty exists 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:3527: left oid 45 (ObjNum 1598 snap 453 seq_num 1598) 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:3529: finishing write tid 1 to smithi06431743-13 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:3530: finishing write tid 1 to smithi06431743-42 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:3529: finishing write tid 2 to smithi06431743-13 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:3529: finishing write tid 3 to smithi06431743-13 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:3529: finishing write tid 4 to smithi06431743-13 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:3529: oid 13 updating version 0 to 2000 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:3529: oid 13 version 2000 is already newer than 1998 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:3529: oid 13 version 2000 is already newer than 2000 2024-03-04T06:23:01.400 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 13 v 2000 (ObjNum 1600 snap 453 seq_num 1600) dirty exists 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3529: left oid 13 (ObjNum 1600 snap 453 seq_num 1600) 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3530: finishing write tid 2 to smithi06431743-42 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3530: finishing write tid 3 to smithi06431743-42 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3530: finishing write tid 4 to smithi06431743-42 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3530: oid 42 updating version 0 to 841 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3530: oid 42 updating version 841 to 843 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3530: oid 42 version 843 is already newer than 842 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 42 v 843 (ObjNum 1601 snap 453 seq_num 1601) dirty exists 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3530: left oid 42 (ObjNum 1601 snap 453 seq_num 1601) 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3531: finishing write tid 1 to smithi06431743-40 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3531: finishing write tid 2 to smithi06431743-40 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3531: finishing write tid 3 to smithi06431743-40 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3531: finishing write tid 4 to smithi06431743-40 2024-03-04T06:23:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3531: oid 40 updating version 0 to 844 2024-03-04T06:23:01.402 INFO:tasks.rados.rados.0.smithi064.stdout:3531: oid 40 updating version 844 to 846 2024-03-04T06:23:01.402 INFO:tasks.rados.rados.0.smithi064.stdout:3531: oid 40 version 846 is already newer than 846 2024-03-04T06:23:01.402 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 846 (ObjNum 1602 snap 453 seq_num 1602) dirty exists 2024-03-04T06:23:01.402 INFO:tasks.rados.rados.0.smithi064.stdout:3531: left oid 40 (ObjNum 1602 snap 453 seq_num 1602) 2024-03-04T06:23:01.476 INFO:tasks.rados.rados.0.smithi064.stdout:3534: finishing write tid 1 to smithi06431743-7 2024-03-04T06:23:01.495 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:23:01.495 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:23:01.495 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:01.496 INFO:tasks.rados.rados.0.smithi064.stdout:3534: finishing write tid 2 to smithi06431743-7 2024-03-04T06:23:01.503 INFO:tasks.rados.rados.0.smithi064.stdout:3534: finishing write tid 3 to smithi06431743-7 2024-03-04T06:23:01.503 INFO:tasks.rados.rados.0.smithi064.stdout:3534: finishing write tid 4 to smithi06431743-7 2024-03-04T06:23:01.504 INFO:tasks.rados.rados.0.smithi064.stdout:3534: oid 7 updating version 0 to 628 2024-03-04T06:23:01.504 INFO:tasks.rados.rados.0.smithi064.stdout:3534: oid 7 version 628 is already newer than 627 2024-03-04T06:23:01.504 INFO:tasks.rados.rados.0.smithi064.stdout:3534: oid 7 updating version 628 to 629 2024-03-04T06:23:01.504 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 629 (ObjNum 1603 snap 453 seq_num 1603) dirty exists 2024-03-04T06:23:01.504 INFO:tasks.rados.rados.0.smithi064.stdout:3534: left oid 7 (ObjNum 1603 snap 453 seq_num 1603) 2024-03-04T06:23:01.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:01.599+0000 7f640039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:01.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:01.606+0000 7f640039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:01.795 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:01.807 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1359,"stamp":"2024-03-04T06:23:01.381825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103092753,"num_objects":1168,"num_object_clones":956,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":175,"num_read":6385,"num_read_kb":5005270,"num_write":5415,"num_write_kb":1441016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15460,"ondisk_log_size":15460,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":623552,"kb_used_data":384096,"kb_used_omap":0,"kb_used_meta":239104,"kb_avail":376863808,"statfs":{"total":386547056640,"available":385908539392,"internally_reserved":0,"allocated":393314304,"data_stored":432630420,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":244842496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2723130,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":9045,"num_write":4,"num_write_kb":1174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.022467"},"pg_stats":[{"pgid":"3.f","version":"906'837","reported_seq":3082,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-04T06:22:58.411524+0000","last_change":"2024-03-04T06:22:58.061507+0000","last_active":"2024-03-04T06:22:58.411524+0000","last_peered":"2024-03-04T06:22:58.411524+0000","last_clean":"2024-03-04T06:22:58.411524+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:22:58.411524+0000","last_undegraded":"2024-03-04T06:22:58.411524+0000","last_fullsized":"2024-03-04T06:22:58.411524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3529833,"num_objects":73,"num_object_clones":58,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":343,"num_read_kb":273127,"num_write":300,"num_write_kb":78674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"899'787","reported_seq":3085,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-04T06:22:58.406484+0000","last_change":"2024-03-04T06:22:57.955867+0000","last_active":"2024-03-04T06:22:58.406484+0000","last_peered":"2024-03-04T06:22:58.406484+0000","last_clean":"2024-03-04T06:22:58.406484+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:22:58.406484+0000","last_undegraded":"2024-03-04T06:22:58.406484+0000","last_fullsized":"2024-03-04T06:22:58.406484+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":240,"num_read_kb":189881,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"903'1159","reported_seq":3270,"reported_epoch":905,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:56.425381+0000","last_change":"2024-03-04T06:22:56.425381+0000","last_active":"2024-03-04T06:22:56.425381+0000","last_peered":"2024-03-04T06:22:56.425381+0000","last_clean":"2024-03-04T06:22:56.425381+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:22:56.425381+0000","last_undegraded":"2024-03-04T06:22:56.425381+0000","last_fullsized":"2024-03-04T06:22:56.425381+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1159,"log_dups_size":0,"ondisk_log_size":1159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2454218,"num_objects":93,"num_object_clones":78,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":538,"num_read_kb":430540,"num_write":414,"num_write_kb":112065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"905'956","reported_seq":3080,"reported_epoch":909,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:00.405537+0000","last_change":"2024-03-04T06:23:00.405537+0000","last_active":"2024-03-04T06:23:00.405537+0000","last_peered":"2024-03-04T06:23:00.405537+0000","last_clean":"2024-03-04T06:23:00.405537+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:00.405537+0000","last_undegraded":"2024-03-04T06:23:00.405537+0000","last_fullsized":"2024-03-04T06:23:00.405537+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8438269,"num_objects":64,"num_object_clones":54,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":314,"num_read_kb":237438,"num_write":308,"num_write_kb":85718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"904'373","reported_seq":2871,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-03-04T06:22:55.437131+0000","last_change":"2024-03-04T06:22:54.382502+0000","last_active":"2024-03-04T06:22:55.437131+0000","last_peered":"2024-03-04T06:22:55.437131+0000","last_clean":"2024-03-04T06:22:55.437131+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:22:55.437131+0000","last_undegraded":"2024-03-04T06:22:55.437131+0000","last_fullsized":"2024-03-04T06:22:55.437131+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2706323,"num_objects":26,"num_object_clones":23,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":114,"num_read_kb":87748,"num_write":119,"num_write_kb":33092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"907'943","reported_seq":3134,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-04T06:22:58.496249+0000","last_change":"2024-03-04T06:22:58.017850+0000","last_active":"2024-03-04T06:22:58.496249+0000","last_peered":"2024-03-04T06:22:58.496249+0000","last_clean":"2024-03-04T06:22:58.496249+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:22:58.496249+0000","last_undegraded":"2024-03-04T06:22:58.496249+0000","last_fullsized":"2024-03-04T06:22:58.496249+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":943,"log_dups_size":0,"ondisk_log_size":943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5076506,"num_objects":67,"num_object_clones":56,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":405,"num_read_kb":327766,"num_write":324,"num_write_kb":93484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2659,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-04T06:22:58.412823+0000","last_change":"2024-03-04T06:22:58.017699+0000","last_active":"2024-03-04T06:22:58.412823+0000","last_peered":"2024-03-04T06:22:58.412823+0000","last_clean":"2024-03-04T06:22:58.412823+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:22:58.412823+0000","last_undegraded":"2024-03-04T06:22:58.412823+0000","last_fullsized":"2024-03-04T06:22:58.412823+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"903'610","reported_seq":2957,"reported_epoch":905,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:56.422254+0000","last_change":"2024-03-04T06:22:56.422254+0000","last_active":"2024-03-04T06:22:56.422254+0000","last_peered":"2024-03-04T06:22:56.422254+0000","last_clean":"2024-03-04T06:22:56.422254+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:22:56.422254+0000","last_undegraded":"2024-03-04T06:22:56.422254+0000","last_fullsized":"2024-03-04T06:22:56.422254+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5362192,"num_objects":41,"num_object_clones":32,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":283,"num_read_kb":255606,"num_write":185,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2428,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-03-04T06:22:56.415318+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:22:56.415318+0000","last_peered":"2024-03-04T06:22:56.415318+0000","last_clean":"2024-03-04T06:22:56.415318+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:22:56.415318+0000","last_undegraded":"2024-03-04T06:22:56.415318+0000","last_fullsized":"2024-03-04T06:22:56.415318+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2239,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-03-04T06:22:56.424228+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:22:56.424228+0000","last_peered":"2024-03-04T06:22:56.424228+0000","last_clean":"2024-03-04T06:22:56.424228+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:22:56.424228+0000","last_undegraded":"2024-03-04T06:22:56.424228+0000","last_fullsized":"2024-03-04T06:22:56.424228+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"903'949","reported_seq":3144,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-03-04T06:22:56.424604+0000","last_change":"2024-03-04T06:22:56.429293+0000","last_active":"2024-03-04T06:22:56.424604+0000","last_peered":"2024-03-04T06:22:56.424604+0000","last_clean":"2024-03-04T06:22:56.424604+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:22:56.424604+0000","last_undegraded":"2024-03-04T06:22:56.424604+0000","last_fullsized":"2024-03-04T06:22:56.424604+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":949,"log_dups_size":0,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":423,"num_read_kb":341165,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2237,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-04T06:23:00.414259+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:00.414259+0000","last_peered":"2024-03-04T06:23:00.414259+0000","last_clean":"2024-03-04T06:23:00.414259+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:00.414259+0000","last_undegraded":"2024-03-04T06:23:00.414259+0000","last_fullsized":"2024-03-04T06:23:00.414259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"903'623","reported_seq":3045,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-03-04T06:22:55.381289+0000","last_change":"2024-03-04T06:22:54.477157+0000","last_active":"2024-03-04T06:22:55.381289+0000","last_peered":"2024-03-04T06:22:55.381289+0000","last_clean":"2024-03-04T06:22:55.381289+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:22:55.381289+0000","last_undegraded":"2024-03-04T06:22:55.381289+0000","last_fullsized":"2024-03-04T06:22:55.381289+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":623,"log_dups_size":0,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017665,"num_objects":49,"num_object_clones":40,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":307,"num_read_kb":250021,"num_write":210,"num_write_kb":59650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2239,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-03-04T06:22:56.418064+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:22:56.418064+0000","last_peered":"2024-03-04T06:22:56.418064+0000","last_clean":"2024-03-04T06:22:56.418064+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:22:56.418064+0000","last_undegraded":"2024-03-04T06:22:56.418064+0000","last_fullsized":"2024-03-04T06:22:56.418064+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"903'1691","reported_seq":3629,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-03-04T06:22:55.396846+0000","last_change":"2024-03-04T06:22:54.527253+0000","last_active":"2024-03-04T06:22:55.396846+0000","last_peered":"2024-03-04T06:22:55.396846+0000","last_clean":"2024-03-04T06:22:55.396846+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:22:55.396846+0000","last_undegraded":"2024-03-04T06:22:55.396846+0000","last_fullsized":"2024-03-04T06:22:55.396846+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1691,"log_dups_size":0,"ondisk_log_size":1691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17027339,"num_objects":134,"num_object_clones":108,"num_object_copies":402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":772,"num_read_kb":618690,"num_write":627,"num_write_kb":168870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2237,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-04T06:23:00.403387+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:00.403387+0000","last_peered":"2024-03-04T06:23:00.403387+0000","last_clean":"2024-03-04T06:23:00.403387+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:00.403387+0000","last_undegraded":"2024-03-04T06:23:00.403387+0000","last_fullsized":"2024-03-04T06:23:00.403387+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"906'1988","reported_seq":3657,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-04T06:22:58.383772+0000","last_change":"2024-03-04T06:22:58.099052+0000","last_active":"2024-03-04T06:22:58.383772+0000","last_peered":"2024-03-04T06:22:58.383772+0000","last_clean":"2024-03-04T06:22:58.383772+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:22:58.383772+0000","last_undegraded":"2024-03-04T06:22:58.383772+0000","last_fullsized":"2024-03-04T06:22:58.383772+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1988,"log_dups_size":0,"ondisk_log_size":1988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14691694,"num_objects":147,"num_object_clones":123,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":691,"num_read_kb":513490,"num_write":679,"num_write_kb":183191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"903'900","reported_seq":3144,"reported_epoch":905,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:22:56.423309+0000","last_change":"2024-03-04T06:22:56.423309+0000","last_active":"2024-03-04T06:22:56.423309+0000","last_peered":"2024-03-04T06:22:56.423309+0000","last_clean":"2024-03-04T06:22:56.423309+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:22:56.423309+0000","last_undegraded":"2024-03-04T06:22:56.423309+0000","last_fullsized":"2024-03-04T06:22:56.423309+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269770,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":432,"num_read_kb":357436,"num_write":317,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2323,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-04T06:22:58.389108+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:22:58.389108+0000","last_peered":"2024-03-04T06:22:58.389108+0000","last_clean":"2024-03-04T06:22:58.389108+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:22:58.389108+0000","last_undegraded":"2024-03-04T06:22:58.389108+0000","last_fullsized":"2024-03-04T06:22:58.389108+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"906'1025","reported_seq":3219,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-04T06:22:58.393514+0000","last_change":"2024-03-04T06:22:58.050727+0000","last_active":"2024-03-04T06:22:58.393514+0000","last_peered":"2024-03-04T06:22:58.393514+0000","last_clean":"2024-03-04T06:22:58.393514+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:22:58.393514+0000","last_undegraded":"2024-03-04T06:22:58.393514+0000","last_fullsized":"2024-03-04T06:22:58.393514+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1025,"log_dups_size":0,"ondisk_log_size":1025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3412184,"num_objects":81,"num_object_clones":65,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":373,"num_read_kb":269282,"num_write":346,"num_write_kb":92772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2292,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-03-04T06:22:55.393080+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:22:55.393080+0000","last_peered":"2024-03-04T06:22:55.393080+0000","last_clean":"2024-03-04T06:22:55.393080+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:22:55.393080+0000","last_undegraded":"2024-03-04T06:22:55.393080+0000","last_fullsized":"2024-03-04T06:22:55.393080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"903'1373","reported_seq":3331,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-03-04T06:22:55.387506+0000","last_change":"2024-03-04T06:22:54.532115+0000","last_active":"2024-03-04T06:22:55.387506+0000","last_peered":"2024-03-04T06:22:55.387506+0000","last_clean":"2024-03-04T06:22:55.387506+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:22:55.387506+0000","last_undegraded":"2024-03-04T06:22:55.387506+0000","last_fullsized":"2024-03-04T06:22:55.387506+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1373,"log_dups_size":0,"ondisk_log_size":1373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3916394,"num_objects":96,"num_object_clones":82,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":487,"num_read_kb":381851,"num_write":447,"num_write_kb":124420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2323,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-04T06:22:58.384518+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:22:58.384518+0000","last_peered":"2024-03-04T06:22:58.384518+0000","last_clean":"2024-03-04T06:22:58.384518+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:22:58.384518+0000","last_undegraded":"2024-03-04T06:22:58.384518+0000","last_fullsized":"2024-03-04T06:22:58.384518+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"906'1149","reported_seq":3384,"reported_epoch":909,"state":"active+clean","last_fresh":"2024-03-04T06:22:58.378857+0000","last_change":"2024-03-04T06:22:58.019461+0000","last_active":"2024-03-04T06:22:58.378857+0000","last_peered":"2024-03-04T06:22:58.378857+0000","last_clean":"2024-03-04T06:22:58.378857+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:22:58.378857+0000","last_undegraded":"2024-03-04T06:22:58.378857+0000","last_fullsized":"2024-03-04T06:22:58.378857+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1149,"log_dups_size":0,"ondisk_log_size":1149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8169357,"num_objects":95,"num_object_clones":81,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":560,"num_read_kb":471138,"num_write":426,"num_write_kb":116430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2239,"reported_epoch":905,"state":"active+clean","last_fresh":"2024-03-04T06:22:56.423750+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:22:56.423750+0000","last_peered":"2024-03-04T06:22:56.423750+0000","last_clean":"2024-03-04T06:22:56.423750+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:22:56.423750+0000","last_undegraded":"2024-03-04T06:22:56.423750+0000","last_fullsized":"2024-03-04T06:22:56.423750+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":102502385,"num_objects":1166,"num_object_clones":956,"num_object_copies":3498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":175,"num_read":6282,"num_read_kb":5005179,"num_write":5278,"num_write_kb":1439122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15363,"ondisk_log_size":15363,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":196664,"kb_used_data":129472,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94175176,"statfs":{"total":96636764160,"available":96435380224,"internally_reserved":0,"allocated":132579328,"data_stored":146861176,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":148384,"kb_used_data":90824,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94223456,"statfs":{"total":96636764160,"available":96484818944,"internally_reserved":0,"allocated":93003776,"data_stored":99802755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706578,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":146228,"kb_used_data":88876,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94225612,"statfs":{"total":96636764160,"available":96487026688,"internally_reserved":0,"allocated":91009024,"data_stored":102243468,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706578,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":132276,"kb_used_data":74924,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94239564,"statfs":{"total":96636764160,"available":96501313536,"internally_reserved":0,"allocated":76722176,"data_stored":83723021,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:02.386 INFO:tasks.rados.rados.0.smithi064.stdout:3525: done (8 left) 2024-03-04T06:23:02.387 INFO:tasks.rados.rados.0.smithi064.stdout:3526: done (7 left) 2024-03-04T06:23:02.387 INFO:tasks.rados.rados.0.smithi064.stdout:3527: done (6 left) 2024-03-04T06:23:02.387 INFO:tasks.rados.rados.0.smithi064.stdout:3529: done (5 left) 2024-03-04T06:23:02.387 INFO:tasks.rados.rados.0.smithi064.stdout:3530: done (4 left) 2024-03-04T06:23:02.387 INFO:tasks.rados.rados.0.smithi064.stdout:3531: done (3 left) 2024-03-04T06:23:02.387 INFO:tasks.rados.rados.0.smithi064.stdout:3533: done (2 left) 2024-03-04T06:23:02.387 INFO:tasks.rados.rados.0.smithi064.stdout:3534: done (1 left) 2024-03-04T06:23:02.387 INFO:tasks.rados.rados.0.smithi064.stdout:3535: done (0 left) 2024-03-04T06:23:02.387 INFO:tasks.rados.rados.0.smithi064.stdout:3536: write (excl) oid 10 current snap is 454 2024-03-04T06:23:02.387 INFO:tasks.rados.rados.0.smithi064.stdout:3536: seq_num 1604 ranges {558428=733526,1712529=746887,2948598=9830} 2024-03-04T06:23:02.395 INFO:tasks.rados.rados.0.smithi064.stdout:3536: writing smithi06431743-10 from 558428 to 1291954 tid 1 2024-03-04T06:23:02.403 INFO:tasks.rados.rados.0.smithi064.stdout:3536: writing smithi06431743-10 from 1712529 to 2459416 tid 2 2024-03-04T06:23:02.407 INFO:tasks.rados.rados.0.smithi064.stdout:3536: writing smithi06431743-10 from 2948598 to 2958428 tid 3 2024-03-04T06:23:02.407 INFO:tasks.rados.rados.0.smithi064.stdout:3537: snap_create 2024-03-04T06:23:02.470 INFO:tasks.rados.rados.0.smithi064.stdout:3536: finishing write tid 1 to smithi06431743-10 2024-03-04T06:23:02.503 INFO:tasks.rados.rados.0.smithi064.stdout:3536: finishing write tid 2 to smithi06431743-10 2024-03-04T06:23:02.509 INFO:tasks.rados.rados.0.smithi064.stdout:3536: finishing write tid 3 to smithi06431743-10 2024-03-04T06:23:02.513 INFO:tasks.rados.rados.0.smithi064.stdout:3536: finishing write tid 4 to smithi06431743-10 2024-03-04T06:23:02.514 INFO:tasks.rados.rados.0.smithi064.stdout:3536: finishing write tid 5 to smithi06431743-10 2024-03-04T06:23:02.514 INFO:tasks.rados.rados.0.smithi064.stdout:3536: oid 10 updating version 0 to 952 2024-03-04T06:23:02.514 INFO:tasks.rados.rados.0.smithi064.stdout:3536: oid 10 version 952 is already newer than 951 2024-03-04T06:23:02.514 INFO:tasks.rados.rados.0.smithi064.stdout:3536: oid 10 updating version 952 to 954 2024-03-04T06:23:02.514 INFO:tasks.rados.rados.0.smithi064.stdout:3536: oid 10 version 954 is already newer than 954 2024-03-04T06:23:02.515 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 954 (ObjNum 1604 snap 454 seq_num 1604) dirty exists 2024-03-04T06:23:02.515 INFO:tasks.rados.rados.0.smithi064.stdout:3536: left oid 10 (ObjNum 1604 snap 454 seq_num 1604) 2024-03-04T06:23:03.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:03.295 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:03.294+0000 7fb4aa99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:03.298 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:03.298+0000 7fb4aa99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:03.396 INFO:tasks.rados.rados.0.smithi064.stdout:3536: done (1 left) 2024-03-04T06:23:03.396 INFO:tasks.rados.rados.0.smithi064.stdout:3537: done (0 left) 2024-03-04T06:23:03.396 INFO:tasks.rados.rados.0.smithi064.stdout:3538: read oid 21 snap 449 2024-03-04T06:23:03.397 INFO:tasks.rados.rados.0.smithi064.stdout:3538: expect deleted 2024-03-04T06:23:03.397 INFO:tasks.rados.rados.0.smithi064.stdout:3539: read oid 44 snap -1 2024-03-04T06:23:03.397 INFO:tasks.rados.rados.0.smithi064.stdout:3539: expect (ObjNum 1568 snap 443 seq_num 1568) 2024-03-04T06:23:03.397 INFO:tasks.rados.rados.0.smithi064.stdout:3540: delete oid 17 current snap is 455 2024-03-04T06:23:03.434 INFO:tasks.rados.rados.0.smithi064.stdout:3539: expect (ObjNum 1568 snap 443 seq_num 1568) 2024-03-04T06:23:03.524 INFO:tasks.rados.rados.0.smithi064.stdout:3538: done (2 left) 2024-03-04T06:23:03.524 INFO:tasks.rados.rados.0.smithi064.stdout:3539: done (1 left) 2024-03-04T06:23:03.524 INFO:tasks.rados.rados.0.smithi064.stdout:3540: done (0 left) 2024-03-04T06:23:03.524 INFO:tasks.rados.rados.0.smithi064.stdout:3541: snap_create 2024-03-04T06:23:04.398 INFO:tasks.rados.rados.0.smithi064.stdout:3541: done (0 left) 2024-03-04T06:23:04.398 INFO:tasks.rados.rados.0.smithi064.stdout:3542: read oid 33 snap -1 2024-03-04T06:23:04.398 INFO:tasks.rados.rados.0.smithi064.stdout:3542: expect (ObjNum 1588 snap 451 seq_num 1588) 2024-03-04T06:23:04.398 INFO:tasks.rados.rados.0.smithi064.stdout:3543: write (excl) oid 22 current snap is 456 2024-03-04T06:23:04.399 INFO:tasks.rados.rados.0.smithi064.stdout:3543: seq_num 1606 ranges {592042=600377,1945846=750556} 2024-03-04T06:23:04.404 INFO:tasks.rados.rados.0.smithi064.stdout:3543: writing smithi06431743-22 from 592042 to 1192419 tid 1 2024-03-04T06:23:04.412 INFO:tasks.rados.rados.0.smithi064.stdout:3543: writing smithi06431743-22 from 1945846 to 2696402 tid 2 2024-03-04T06:23:04.412 INFO:tasks.rados.rados.0.smithi064.stdout:3544: snap_create 2024-03-04T06:23:04.470 INFO:tasks.rados.rados.0.smithi064.stdout:3542: expect (ObjNum 1588 snap 451 seq_num 1588) 2024-03-04T06:23:04.647 INFO:tasks.rados.rados.0.smithi064.stdout:3543: finishing write tid 1 to smithi06431743-22 2024-03-04T06:23:04.647 INFO:tasks.rados.rados.0.smithi064.stdout:3543: finishing write tid 2 to smithi06431743-22 2024-03-04T06:23:04.647 INFO:tasks.rados.rados.0.smithi064.stdout:3543: finishing write tid 3 to smithi06431743-22 2024-03-04T06:23:04.647 INFO:tasks.rados.rados.0.smithi064.stdout:3543: finishing write tid 4 to smithi06431743-22 2024-03-04T06:23:04.647 INFO:tasks.rados.rados.0.smithi064.stdout:3543: oid 22 updating version 0 to 1161 2024-03-04T06:23:04.647 INFO:tasks.rados.rados.0.smithi064.stdout:3543: oid 22 updating version 1161 to 1162 2024-03-04T06:23:04.647 INFO:tasks.rados.rados.0.smithi064.stdout:3543: oid 22 updating version 1162 to 1163 2024-03-04T06:23:04.647 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 1163 (ObjNum 1606 snap 456 seq_num 1606) dirty exists 2024-03-04T06:23:04.647 INFO:tasks.rados.rados.0.smithi064.stdout:3543: left oid 22 (ObjNum 1606 snap 456 seq_num 1606) 2024-03-04T06:23:05.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:05.403 INFO:tasks.rados.rados.0.smithi064.stdout:3542: done (2 left) 2024-03-04T06:23:05.403 INFO:tasks.rados.rados.0.smithi064.stdout:3543: done (1 left) 2024-03-04T06:23:05.403 INFO:tasks.rados.rados.0.smithi064.stdout:3544: done (0 left) 2024-03-04T06:23:05.403 INFO:tasks.rados.rados.0.smithi064.stdout:3545: read oid 20 snap 440 2024-03-04T06:23:05.403 INFO:tasks.rados.rados.0.smithi064.stdout:3545: expect deleted 2024-03-04T06:23:05.403 INFO:tasks.rados.rados.0.smithi064.stdout:3546: delete oid 27 current snap is 457 2024-03-04T06:23:05.449 INFO:tasks.rados.rados.0.smithi064.stdout:3546: done (1 left) 2024-03-04T06:23:05.449 INFO:tasks.rados.rados.0.smithi064.stdout:3547: write oid 40 current snap is 457 2024-03-04T06:23:05.449 INFO:tasks.rados.rados.0.smithi064.stdout:3547: seq_num 1608 ranges {625656=750875,2062810=754225} 2024-03-04T06:23:05.457 INFO:tasks.rados.rados.0.smithi064.stdout:3547: writing smithi06431743-40 from 625656 to 1376531 tid 1 2024-03-04T06:23:05.465 INFO:tasks.rados.rados.0.smithi064.stdout:3547: writing smithi06431743-40 from 2062810 to 2817035 tid 2 2024-03-04T06:23:05.466 INFO:tasks.rados.rados.0.smithi064.stdout:3548: read oid 10 snap -1 2024-03-04T06:23:05.466 INFO:tasks.rados.rados.0.smithi064.stdout:3548: expect (ObjNum 1604 snap 454 seq_num 1604) 2024-03-04T06:23:05.466 INFO:tasks.rados.rados.0.smithi064.stdout:3549: read oid 29 snap -1 2024-03-04T06:23:05.466 INFO:tasks.rados.rados.0.smithi064.stdout:3549: expect (ObjNum 1576 snap 444 seq_num 1576) 2024-03-04T06:23:05.466 INFO:tasks.rados.rados.0.smithi064.stdout:3550: delete oid 31 current snap is 457 2024-03-04T06:23:05.484 INFO:tasks.rados.rados.0.smithi064.stdout:3549: expect (ObjNum 1576 snap 444 seq_num 1576) 2024-03-04T06:23:05.499 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:05.497+0000 7fe726a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:05.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:05.503+0000 7fe726a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:05.596 INFO:tasks.rados.rados.0.smithi064.stdout:3547: finishing write tid 1 to smithi06431743-40 2024-03-04T06:23:05.596 INFO:tasks.rados.rados.0.smithi064.stdout:3548: expect (ObjNum 1604 snap 454 seq_num 1604) 2024-03-04T06:23:05.747 INFO:tasks.rados.rados.0.smithi064.stdout:3547: finishing write tid 2 to smithi06431743-40 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3547: finishing write tid 3 to smithi06431743-40 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3547: finishing write tid 4 to smithi06431743-40 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3547: oid 40 updating version 0 to 852 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3547: oid 40 updating version 852 to 854 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3547: oid 40 version 854 is already newer than 853 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 854 (ObjNum 1608 snap 457 seq_num 1608) dirty exists 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3547: left oid 40 (ObjNum 1608 snap 457 seq_num 1608) 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3545: done (4 left) 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3547: done (3 left) 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3548: done (2 left) 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3549: done (1 left) 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3550: done (0 left) 2024-03-04T06:23:05.748 INFO:tasks.rados.rados.0.smithi064.stdout:3551: write (excl) oid 10 current snap is 457 2024-03-04T06:23:05.749 INFO:tasks.rados.rados.0.smithi064.stdout:3551: seq_num 1610 ranges {659270=501373,1663421=757894} 2024-03-04T06:23:05.754 INFO:tasks.rados.rados.0.smithi064.stdout:3551: writing smithi06431743-10 from 659270 to 1160643 tid 1 2024-03-04T06:23:05.761 INFO:tasks.rados.rados.0.smithi064.stdout:3551: writing smithi06431743-10 from 1663421 to 2421315 tid 2 2024-03-04T06:23:05.761 INFO:tasks.rados.rados.0.smithi064.stdout:3552: read oid 25 snap -1 2024-03-04T06:23:05.761 INFO:tasks.rados.rados.0.smithi064.stdout:3552: expect (ObjNum 1593 snap 453 seq_num 1593) 2024-03-04T06:23:05.761 INFO:tasks.rados.rados.0.smithi064.stdout:3553: write oid 40 current snap is 457 2024-03-04T06:23:05.761 INFO:tasks.rados.rados.0.smithi064.stdout:3553: seq_num 1611 ranges {676077=576622,1721903=501552,2740559=335518} 2024-03-04T06:23:05.771 INFO:tasks.rados.rados.0.smithi064.stdout:3553: writing smithi06431743-40 from 676077 to 1252699 tid 1 2024-03-04T06:23:05.777 INFO:tasks.rados.rados.0.smithi064.stdout:3553: writing smithi06431743-40 from 1721903 to 2223455 tid 2 2024-03-04T06:23:05.783 INFO:tasks.rados.rados.0.smithi064.stdout:3553: writing smithi06431743-40 from 2740559 to 3076077 tid 3 2024-03-04T06:23:05.783 INFO:tasks.rados.rados.0.smithi064.stdout:3554: read oid 37 snap -1 2024-03-04T06:23:05.784 INFO:tasks.rados.rados.0.smithi064.stdout:3554: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:23:05.784 INFO:tasks.rados.rados.0.smithi064.stdout:3555: snap_create 2024-03-04T06:23:05.787 INFO:tasks.rados.rados.0.smithi064.stdout:3551: finishing write tid 1 to smithi06431743-10 2024-03-04T06:23:05.801 INFO:tasks.rados.rados.0.smithi064.stdout:3554: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:23:05.958 INFO:tasks.rados.rados.0.smithi064.stdout:3553: finishing write tid 1 to smithi06431743-40 2024-03-04T06:23:05.958 INFO:tasks.rados.rados.0.smithi064.stdout:3551: finishing write tid 2 to smithi06431743-10 2024-03-04T06:23:05.958 INFO:tasks.rados.rados.0.smithi064.stdout:3552: expect (ObjNum 1593 snap 453 seq_num 1593) 2024-03-04T06:23:06.071 INFO:tasks.rados.rados.0.smithi064.stdout:3553: finishing write tid 2 to smithi06431743-40 2024-03-04T06:23:06.071 INFO:tasks.rados.rados.0.smithi064.stdout:3551: finishing write tid 3 to smithi06431743-10 2024-03-04T06:23:06.071 INFO:tasks.rados.rados.0.smithi064.stdout:3551: finishing write tid 4 to smithi06431743-10 2024-03-04T06:23:06.071 INFO:tasks.rados.rados.0.smithi064.stdout:3551: oid 10 updating version 0 to 956 2024-03-04T06:23:06.071 INFO:tasks.rados.rados.0.smithi064.stdout:3551: oid 10 updating version 956 to 958 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:3551: oid 10 version 958 is already newer than 957 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 958 (ObjNum 1610 snap 457 seq_num 1610) dirty exists 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:3551: left oid 10 (ObjNum 1610 snap 457 seq_num 1610) 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:3553: finishing write tid 3 to smithi06431743-40 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:3553: finishing write tid 4 to smithi06431743-40 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:3553: finishing write tid 5 to smithi06431743-40 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:3553: oid 40 updating version 0 to 856 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:3553: oid 40 version 856 is already newer than 856 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:3553: oid 40 updating version 856 to 858 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:3553: oid 40 version 858 is already newer than 857 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 858 (ObjNum 1611 snap 457 seq_num 1611) dirty exists 2024-03-04T06:23:06.072 INFO:tasks.rados.rados.0.smithi064.stdout:3553: left oid 40 (ObjNum 1611 snap 457 seq_num 1611) 2024-03-04T06:23:06.403 INFO:tasks.rados.rados.0.smithi064.stdout:3551: done (4 left) 2024-03-04T06:23:06.403 INFO:tasks.rados.rados.0.smithi064.stdout:3552: done (3 left) 2024-03-04T06:23:06.403 INFO:tasks.rados.rados.0.smithi064.stdout:3553: done (2 left) 2024-03-04T06:23:06.403 INFO:tasks.rados.rados.0.smithi064.stdout:3554: done (1 left) 2024-03-04T06:23:06.403 INFO:tasks.rados.rados.0.smithi064.stdout:3555: done (0 left) 2024-03-04T06:23:06.403 INFO:tasks.rados.rados.0.smithi064.stdout:3556: snap_remove snap 455 2024-03-04T06:23:06.808 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:23:06.809 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:23:06.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:06.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:06.919+0000 7fd8d2d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:06.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:06.925+0000 7fd8d2d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:07.115 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:07.115 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:07.127 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1367,"stamp":"2024-03-04T06:23:06.900280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":106757702,"num_objects":1182,"num_object_clones":968,"num_object_copies":3546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":175,"num_read":6440,"num_read_kb":5035323,"num_write":5477,"num_write_kb":1457917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15654,"ondisk_log_size":15654,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":641640,"kb_used_data":401616,"kb_used_omap":0,"kb_used_meta":239104,"kb_avail":376845720,"statfs":{"total":386547056640,"available":385890017280,"internally_reserved":0,"allocated":411254784,"data_stored":457843245,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":244842496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":185009,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":33,"num_read_kb":17604,"num_write":36,"num_write_kb":10250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000776"},"pg_stats":[{"pgid":"3.f","version":"910'850","reported_seq":3105,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:03.417875+0000","last_change":"2024-03-04T06:23:02.439343+0000","last_active":"2024-03-04T06:23:03.417875+0000","last_peered":"2024-03-04T06:23:03.417875+0000","last_clean":"2024-03-04T06:23:03.417875+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:03.417875+0000","last_undegraded":"2024-03-04T06:23:03.417875+0000","last_fullsized":"2024-03-04T06:23:03.417875+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":850,"log_dups_size":0,"ondisk_log_size":850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832821,"num_objects":74,"num_object_clones":59,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":346,"num_read_kb":273129,"num_write":308,"num_write_kb":81052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"899'787","reported_seq":3104,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-03-04T06:23:04.427497+0000","last_change":"2024-03-04T06:23:01.404724+0000","last_active":"2024-03-04T06:23:04.427497+0000","last_peered":"2024-03-04T06:23:04.427497+0000","last_clean":"2024-03-04T06:23:04.427497+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:04.427497+0000","last_undegraded":"2024-03-04T06:23:04.427497+0000","last_fullsized":"2024-03-04T06:23:04.427497+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":240,"num_read_kb":189881,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"912'1188","reported_seq":3308,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.843835+0000","last_change":"2024-03-04T06:23:02.453808+0000","last_active":"2024-03-04T06:23:05.843835+0000","last_peered":"2024-03-04T06:23:05.843835+0000","last_clean":"2024-03-04T06:23:05.843835+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:05.843835+0000","last_undegraded":"2024-03-04T06:23:05.843835+0000","last_fullsized":"2024-03-04T06:23:05.843835+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1188,"log_dups_size":0,"ondisk_log_size":1188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5631137,"num_objects":94,"num_object_clones":79,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":547,"num_read_kb":437107,"num_write":422,"num_write_kb":114599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"910'964","reported_seq":3092,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:03.412064+0000","last_change":"2024-03-04T06:23:02.430082+0000","last_active":"2024-03-04T06:23:03.412064+0000","last_peered":"2024-03-04T06:23:03.412064+0000","last_clean":"2024-03-04T06:23:03.412064+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:03.412064+0000","last_undegraded":"2024-03-04T06:23:03.412064+0000","last_fullsized":"2024-03-04T06:23:03.412064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":964,"log_dups_size":0,"ondisk_log_size":964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8438269,"num_objects":64,"num_object_clones":54,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":314,"num_read_kb":237438,"num_write":308,"num_write_kb":85718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"910'383","reported_seq":2908,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.439592+0000","last_change":"2024-03-04T06:23:01.470167+0000","last_active":"2024-03-04T06:23:05.439592+0000","last_peered":"2024-03-04T06:23:05.439592+0000","last_clean":"2024-03-04T06:23:05.439592+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:05.439592+0000","last_undegraded":"2024-03-04T06:23:05.439592+0000","last_fullsized":"2024-03-04T06:23:05.439592+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":120,"num_read_kb":93147,"num_write":120,"num_write_kb":33092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"911'954","reported_seq":3162,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:03.432330+0000","last_change":"2024-03-04T06:23:02.411406+0000","last_active":"2024-03-04T06:23:03.432330+0000","last_peered":"2024-03-04T06:23:03.432330+0000","last_clean":"2024-03-04T06:23:03.432330+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:03.432330+0000","last_undegraded":"2024-03-04T06:23:03.432330+0000","last_fullsized":"2024-03-04T06:23:03.432330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":954,"log_dups_size":0,"ondisk_log_size":954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8152583,"num_objects":69,"num_object_clones":57,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":415,"num_read_kb":331871,"num_write":334,"num_write_kb":96359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2674,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:03.419551+0000","last_change":"2024-03-04T06:23:02.375545+0000","last_active":"2024-03-04T06:23:03.419551+0000","last_peered":"2024-03-04T06:23:03.419551+0000","last_clean":"2024-03-04T06:23:03.419551+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:03.419551+0000","last_undegraded":"2024-03-04T06:23:03.419551+0000","last_fullsized":"2024-03-04T06:23:03.419551+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"912'618","reported_seq":2986,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.802924+0000","last_change":"2024-03-04T06:23:02.451055+0000","last_active":"2024-03-04T06:23:05.802924+0000","last_peered":"2024-03-04T06:23:05.802924+0000","last_clean":"2024-03-04T06:23:05.802924+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:05.802924+0000","last_undegraded":"2024-03-04T06:23:05.802924+0000","last_fullsized":"2024-03-04T06:23:05.802924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":618,"log_dups_size":0,"ondisk_log_size":618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2453,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.421315+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:05.421315+0000","last_peered":"2024-03-04T06:23:05.421315+0000","last_clean":"2024-03-04T06:23:05.421315+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:05.421315+0000","last_undegraded":"2024-03-04T06:23:05.421315+0000","last_fullsized":"2024-03-04T06:23:05.421315+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2262,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.814324+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:05.814324+0000","last_peered":"2024-03-04T06:23:05.814324+0000","last_clean":"2024-03-04T06:23:05.814324+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:05.814324+0000","last_undegraded":"2024-03-04T06:23:05.814324+0000","last_fullsized":"2024-03-04T06:23:05.814324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"906'951","reported_seq":3171,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.811309+0000","last_change":"2024-03-04T06:23:02.401981+0000","last_active":"2024-03-04T06:23:05.811309+0000","last_peered":"2024-03-04T06:23:05.811309+0000","last_clean":"2024-03-04T06:23:05.811309+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:05.811309+0000","last_undegraded":"2024-03-04T06:23:05.811309+0000","last_fullsized":"2024-03-04T06:23:05.811309+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":423,"num_read_kb":341165,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2246,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:03.421516+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:03.421516+0000","last_peered":"2024-03-04T06:23:03.421516+0000","last_clean":"2024-03-04T06:23:03.421516+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:03.421516+0000","last_undegraded":"2024-03-04T06:23:03.421516+0000","last_fullsized":"2024-03-04T06:23:03.421516+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"910'629","reported_seq":3081,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.420590+0000","last_change":"2024-03-04T06:23:01.423043+0000","last_active":"2024-03-04T06:23:05.420590+0000","last_peered":"2024-03-04T06:23:05.420590+0000","last_clean":"2024-03-04T06:23:05.420590+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:05.420590+0000","last_undegraded":"2024-03-04T06:23:05.420590+0000","last_fullsized":"2024-03-04T06:23:05.420590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5379032,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":309,"num_read_kb":250022,"num_write":214,"num_write_kb":60883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2262,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.799636+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:05.799636+0000","last_peered":"2024-03-04T06:23:05.799636+0000","last_clean":"2024-03-04T06:23:05.799636+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:05.799636+0000","last_undegraded":"2024-03-04T06:23:05.799636+0000","last_fullsized":"2024-03-04T06:23:05.799636+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"910'1732","reported_seq":3675,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.439064+0000","last_change":"2024-03-04T06:23:01.513407+0000","last_active":"2024-03-04T06:23:05.439064+0000","last_peered":"2024-03-04T06:23:05.439064+0000","last_clean":"2024-03-04T06:23:05.439064+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:05.439064+0000","last_undegraded":"2024-03-04T06:23:05.439064+0000","last_fullsized":"2024-03-04T06:23:05.439064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1732,"log_dups_size":0,"ondisk_log_size":1732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14506388,"num_objects":137,"num_object_clones":111,"num_object_copies":411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":778,"num_read_kb":618693,"num_write":640,"num_write_kb":171886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2247,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:03.413722+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:03.413722+0000","last_peered":"2024-03-04T06:23:03.413722+0000","last_clean":"2024-03-04T06:23:03.413722+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:03.413722+0000","last_undegraded":"2024-03-04T06:23:03.413722+0000","last_fullsized":"2024-03-04T06:23:03.413722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"912'2010","reported_seq":3683,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-03-04T06:23:04.430339+0000","last_change":"2024-03-04T06:23:02.485624+0000","last_active":"2024-03-04T06:23:04.430339+0000","last_peered":"2024-03-04T06:23:04.430339+0000","last_clean":"2024-03-04T06:23:04.430339+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:04.430339+0000","last_undegraded":"2024-03-04T06:23:04.430339+0000","last_fullsized":"2024-03-04T06:23:04.430339+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2010,"log_dups_size":0,"ondisk_log_size":2010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15632886,"num_objects":149,"num_object_clones":125,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":695,"num_read_kb":513492,"num_write":687,"num_write_kb":185635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"910'906","reported_seq":3175,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.812109+0000","last_change":"2024-03-04T06:23:02.458576+0000","last_active":"2024-03-04T06:23:05.812109+0000","last_peered":"2024-03-04T06:23:05.812109+0000","last_clean":"2024-03-04T06:23:05.812109+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:05.812109+0000","last_undegraded":"2024-03-04T06:23:05.812109+0000","last_fullsized":"2024-03-04T06:23:05.812109+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":906,"log_dups_size":0,"ondisk_log_size":906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269770,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":438,"num_read_kb":364480,"num_write":317,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2339,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-03-04T06:23:04.442569+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:04.442569+0000","last_peered":"2024-03-04T06:23:04.442569+0000","last_clean":"2024-03-04T06:23:04.442569+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:04.442569+0000","last_undegraded":"2024-03-04T06:23:04.442569+0000","last_fullsized":"2024-03-04T06:23:04.442569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"910'1033","reported_seq":3238,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-03-04T06:23:04.437715+0000","last_change":"2024-03-04T06:23:02.490252+0000","last_active":"2024-03-04T06:23:04.437715+0000","last_peered":"2024-03-04T06:23:04.437715+0000","last_clean":"2024-03-04T06:23:04.437715+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:04.437715+0000","last_undegraded":"2024-03-04T06:23:04.437715+0000","last_fullsized":"2024-03-04T06:23:04.437715+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1033,"log_dups_size":0,"ondisk_log_size":1033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3412184,"num_objects":81,"num_object_clones":65,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":373,"num_read_kb":269282,"num_write":346,"num_write_kb":92772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2320,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.432523+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:05.432523+0000","last_peered":"2024-03-04T06:23:05.432523+0000","last_clean":"2024-03-04T06:23:05.432523+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:05.432523+0000","last_undegraded":"2024-03-04T06:23:05.432523+0000","last_fullsized":"2024-03-04T06:23:05.432523+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"910'1389","reported_seq":3368,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.429085+0000","last_change":"2024-03-04T06:23:01.477272+0000","last_active":"2024-03-04T06:23:05.429085+0000","last_peered":"2024-03-04T06:23:05.429085+0000","last_clean":"2024-03-04T06:23:05.429085+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:05.429085+0000","last_undegraded":"2024-03-04T06:23:05.429085+0000","last_fullsized":"2024-03-04T06:23:05.429085+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1389,"log_dups_size":0,"ondisk_log_size":1389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6740366,"num_objects":97,"num_object_clones":82,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":488,"num_read_kb":381852,"num_write":452,"num_write_kb":125521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2339,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-03-04T06:23:04.415647+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:04.415647+0000","last_peered":"2024-03-04T06:23:04.415647+0000","last_clean":"2024-03-04T06:23:04.415647+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:04.415647+0000","last_undegraded":"2024-03-04T06:23:04.415647+0000","last_fullsized":"2024-03-04T06:23:04.415647+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"913'1163","reported_seq":3410,"reported_epoch":913,"state":"active+clean","last_fresh":"2024-03-04T06:23:04.512994+0000","last_change":"2024-03-04T06:23:02.485491+0000","last_active":"2024-03-04T06:23:04.512994+0000","last_peered":"2024-03-04T06:23:04.512994+0000","last_clean":"2024-03-04T06:23:04.512994+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:04.512994+0000","last_undegraded":"2024-03-04T06:23:04.512994+0000","last_fullsized":"2024-03-04T06:23:04.512994+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8236618,"num_objects":96,"num_object_clones":82,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":568,"num_read_kb":478067,"num_write":430,"num_write_kb":117750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2262,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-04T06:23:05.808503+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:05.808503+0000","last_peered":"2024-03-04T06:23:05.808503+0000","last_clean":"2024-03-04T06:23:05.808503+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:05.808503+0000","last_undegraded":"2024-03-04T06:23:05.808503+0000","last_fullsized":"2024-03-04T06:23:05.808503+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":106167334,"num_objects":1180,"num_object_clones":968,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":175,"num_read":6337,"num_read_kb":5035232,"num_write":5340,"num_write_kb":1456023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15557,"ondisk_log_size":15557,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706584,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":194228,"kb_used_data":126932,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94177612,"statfs":{"total":96636764160,"available":96437874688,"internally_reserved":0,"allocated":129978368,"data_stored":145871473,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706584,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":150980,"kb_used_data":93628,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94220860,"statfs":{"total":96636764160,"available":96482160640,"internally_reserved":0,"allocated":95875072,"data_stored":104290067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706583,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":152740,"kb_used_data":95052,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94219100,"statfs":{"total":96636764160,"available":96480358400,"internally_reserved":0,"allocated":97333248,"data_stored":109917147,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706583,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":143692,"kb_used_data":86004,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94228148,"statfs":{"total":96636764160,"available":96489623552,"internally_reserved":0,"allocated":88068096,"data_stored":97764558,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:07.405 INFO:tasks.rados.rados.0.smithi064.stdout:3556: done (0 left) 2024-03-04T06:23:07.406 INFO:tasks.rados.rados.0.smithi064.stdout:3557: read oid 8 snap -1 2024-03-04T06:23:07.406 INFO:tasks.rados.rados.0.smithi064.stdout:3557: expect (ObjNum 1512 snap 423 seq_num 1512) 2024-03-04T06:23:07.406 INFO:tasks.rados.rados.0.smithi064.stdout:3558: snap_create 2024-03-04T06:23:07.472 INFO:tasks.rados.rados.0.smithi064.stdout:3557: expect (ObjNum 1512 snap 423 seq_num 1512) 2024-03-04T06:23:07.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:07.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:07.688+0000 7f757c19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:07.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:07.692+0000 7f757c19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:08.409 INFO:tasks.rados.rados.0.smithi064.stdout:3557: done (1 left) 2024-03-04T06:23:08.409 INFO:tasks.rados.rados.0.smithi064.stdout:3558: done (0 left) 2024-03-04T06:23:08.409 INFO:tasks.rados.rados.0.smithi064.stdout:3559: write (excl) oid 29 current snap is 459 2024-03-04T06:23:08.409 INFO:tasks.rados.rados.0.smithi064.stdout:3559: seq_num 1612 ranges {692884=768224,1896738=761563} 2024-03-04T06:23:08.417 INFO:tasks.rados.rados.0.smithi064.stdout:3559: writing smithi06431743-29 from 692884 to 1461108 tid 1 2024-03-04T06:23:08.424 INFO:tasks.rados.rados.0.smithi064.stdout:3559: writing smithi06431743-29 from 1896738 to 2658301 tid 2 2024-03-04T06:23:08.424 INFO:tasks.rados.rados.0.smithi064.stdout:3560: write (excl) oid 30 current snap is 459 2024-03-04T06:23:08.424 INFO:tasks.rados.rados.0.smithi064.stdout:3560: seq_num 1613 ranges {709691=443473,1838867=505221,2795405=314286} 2024-03-04T06:23:08.430 INFO:tasks.rados.rados.0.smithi064.stdout:3560: writing smithi06431743-30 from 709691 to 1153164 tid 1 2024-03-04T06:23:08.436 INFO:tasks.rados.rados.0.smithi064.stdout:3560: writing smithi06431743-30 from 1838867 to 2344088 tid 2 2024-03-04T06:23:08.441 INFO:tasks.rados.rados.0.smithi064.stdout:3560: writing smithi06431743-30 from 2795405 to 3109691 tid 3 2024-03-04T06:23:08.441 INFO:tasks.rados.rados.0.smithi064.stdout:3561: read oid 14 snap -1 2024-03-04T06:23:08.441 INFO:tasks.rados.rados.0.smithi064.stdout:3561: expect (ObjNum 1596 snap 453 seq_num 1596) 2024-03-04T06:23:08.441 INFO:tasks.rados.rados.0.smithi064.stdout:3562: write (excl) oid 26 current snap is 459 2024-03-04T06:23:08.441 INFO:tasks.rados.rados.0.smithi064.stdout:3562: seq_num 1614 ranges {726498=518722,1897349=765232} 2024-03-04T06:23:08.447 INFO:tasks.rados.rados.0.smithi064.stdout:3562: writing smithi06431743-26 from 726498 to 1245220 tid 1 2024-03-04T06:23:08.455 INFO:tasks.rados.rados.0.smithi064.stdout:3562: writing smithi06431743-26 from 1897349 to 2662581 tid 2 2024-03-04T06:23:08.455 INFO:tasks.rados.rados.0.smithi064.stdout:3563: snap_create 2024-03-04T06:23:08.492 INFO:tasks.rados.rados.0.smithi064.stdout:3561: expect (ObjNum 1596 snap 453 seq_num 1596) 2024-03-04T06:23:08.605 INFO:tasks.rados.rados.0.smithi064.stdout:3562: finishing write tid 1 to smithi06431743-26 2024-03-04T06:23:08.605 INFO:tasks.rados.rados.0.smithi064.stdout:3559: finishing write tid 1 to smithi06431743-29 2024-03-04T06:23:08.605 INFO:tasks.rados.rados.0.smithi064.stdout:3562: finishing write tid 2 to smithi06431743-26 2024-03-04T06:23:08.605 INFO:tasks.rados.rados.0.smithi064.stdout:3562: finishing write tid 3 to smithi06431743-26 2024-03-04T06:23:08.605 INFO:tasks.rados.rados.0.smithi064.stdout:3562: finishing write tid 4 to smithi06431743-26 2024-03-04T06:23:08.605 INFO:tasks.rados.rados.0.smithi064.stdout:3562: oid 26 updating version 0 to 1736 2024-03-04T06:23:08.605 INFO:tasks.rados.rados.0.smithi064.stdout:3562: oid 26 version 1736 is already newer than 1734 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3562: oid 26 version 1736 is already newer than 1736 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 1736 (ObjNum 1614 snap 459 seq_num 1614) dirty exists 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3562: left oid 26 (ObjNum 1614 snap 459 seq_num 1614) 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3559: finishing write tid 2 to smithi06431743-29 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3559: finishing write tid 3 to smithi06431743-29 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3559: finishing write tid 4 to smithi06431743-29 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3559: oid 29 updating version 0 to 2014 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3559: oid 29 version 2014 is already newer than 2013 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3559: oid 29 version 2014 is already newer than 2012 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 2014 (ObjNum 1612 snap 459 seq_num 1612) dirty exists 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3559: left oid 29 (ObjNum 1612 snap 459 seq_num 1612) 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3560: finishing write tid 1 to smithi06431743-30 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3560: finishing write tid 2 to smithi06431743-30 2024-03-04T06:23:08.606 INFO:tasks.rados.rados.0.smithi064.stdout:3560: finishing write tid 3 to smithi06431743-30 2024-03-04T06:23:08.607 INFO:tasks.rados.rados.0.smithi064.stdout:3560: finishing write tid 4 to smithi06431743-30 2024-03-04T06:23:08.607 INFO:tasks.rados.rados.0.smithi064.stdout:3560: finishing write tid 5 to smithi06431743-30 2024-03-04T06:23:08.607 INFO:tasks.rados.rados.0.smithi064.stdout:3560: oid 30 updating version 0 to 2019 2024-03-04T06:23:08.607 INFO:tasks.rados.rados.0.smithi064.stdout:3560: oid 30 version 2019 is already newer than 2019 2024-03-04T06:23:08.607 INFO:tasks.rados.rados.0.smithi064.stdout:3560: oid 30 version 2019 is already newer than 2019 2024-03-04T06:23:08.607 INFO:tasks.rados.rados.0.smithi064.stdout:3560: oid 30 version 2019 is already newer than 2016 2024-03-04T06:23:08.607 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 30 v 2019 (ObjNum 1613 snap 459 seq_num 1613) dirty exists 2024-03-04T06:23:08.607 INFO:tasks.rados.rados.0.smithi064.stdout:3560: left oid 30 (ObjNum 1613 snap 459 seq_num 1613) 2024-03-04T06:23:09.414 INFO:tasks.rados.rados.0.smithi064.stdout:3559: done (4 left) 2024-03-04T06:23:09.414 INFO:tasks.rados.rados.0.smithi064.stdout:3560: done (3 left) 2024-03-04T06:23:09.414 INFO:tasks.rados.rados.0.smithi064.stdout:3561: done (2 left) 2024-03-04T06:23:09.414 INFO:tasks.rados.rados.0.smithi064.stdout:3562: done (1 left) 2024-03-04T06:23:09.414 INFO:tasks.rados.rados.0.smithi064.stdout:3563: done (0 left) 2024-03-04T06:23:09.415 INFO:tasks.rados.rados.0.smithi064.stdout:3564: write oid 29 current snap is 460 2024-03-04T06:23:09.415 INFO:tasks.rados.rados.0.smithi064.stdout:3564: seq_num 1615 ranges {743305=593971,1955831=508890} 2024-03-04T06:23:09.421 INFO:tasks.rados.rados.0.smithi064.stdout:3564: writing smithi06431743-29 from 743305 to 1337276 tid 1 2024-03-04T06:23:09.427 INFO:tasks.rados.rados.0.smithi064.stdout:3564: writing smithi06431743-29 from 1955831 to 2464721 tid 2 2024-03-04T06:23:09.428 INFO:tasks.rados.rados.0.smithi064.stdout:3565: read oid 45 snap -1 2024-03-04T06:23:09.428 INFO:tasks.rados.rados.0.smithi064.stdout:3565: expect (ObjNum 1598 snap 453 seq_num 1598) 2024-03-04T06:23:09.428 INFO:tasks.rados.rados.0.smithi064.stdout:3566: delete oid 47 current snap is 460 2024-03-04T06:23:09.484 INFO:tasks.rados.rados.0.smithi064.stdout:3565: expect (ObjNum 1598 snap 453 seq_num 1598) 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:3564: finishing write tid 1 to smithi06431743-29 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:3564: finishing write tid 2 to smithi06431743-29 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:3564: finishing write tid 3 to smithi06431743-29 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:3564: finishing write tid 4 to smithi06431743-29 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:3564: oid 29 updating version 0 to 2023 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:3564: oid 29 version 2023 is already newer than 2022 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:3564: oid 29 version 2023 is already newer than 2021 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 2023 (ObjNum 1615 snap 460 seq_num 1615) dirty exists 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:3564: left oid 29 (ObjNum 1615 snap 460 seq_num 1615) 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:3564: done (2 left) 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:3565: done (1 left) 2024-03-04T06:23:09.719 INFO:tasks.rados.rados.0.smithi064.stdout:3566: done (0 left) 2024-03-04T06:23:09.720 INFO:tasks.rados.rados.0.smithi064.stdout:3567: read oid 16 snap -1 2024-03-04T06:23:09.720 INFO:tasks.rados.rados.0.smithi064.stdout:3567: expect deleted 2024-03-04T06:23:09.720 INFO:tasks.rados.rados.0.smithi064.stdout:3568: read oid 33 snap -1 2024-03-04T06:23:09.720 INFO:tasks.rados.rados.0.smithi064.stdout:3568: expect (ObjNum 1588 snap 451 seq_num 1588) 2024-03-04T06:23:09.720 INFO:tasks.rados.rados.0.smithi064.stdout:3569: snap_remove snap 453 2024-03-04T06:23:09.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:09.867 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:09.865+0000 7fcd886dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:09.870 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:09.870+0000 7fcd886dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:10.416 INFO:tasks.rados.rados.0.smithi064.stdout:3569: done (2 left) 2024-03-04T06:23:10.416 INFO:tasks.rados.rados.0.smithi064.stdout:3570: write (excl) oid 1 current snap is 460 2024-03-04T06:23:10.416 INFO:tasks.rados.rados.0.smithi064.stdout:3570: seq_num 1617 ranges {776919=744469,1956442=512559} 2024-03-04T06:23:10.424 INFO:tasks.rados.rados.0.smithi064.stdout:3570: writing smithi06431743-1 from 776919 to 1521388 tid 1 2024-03-04T06:23:10.430 INFO:tasks.rados.rados.0.smithi064.stdout:3570: writing smithi06431743-1 from 1956442 to 2469001 tid 2 2024-03-04T06:23:10.430 INFO:tasks.rados.rados.0.smithi064.stdout:3571: delete oid 8 current snap is 460 2024-03-04T06:23:10.430 INFO:tasks.rados.rados.0.smithi064.stdout:3568: expect (ObjNum 1588 snap 451 seq_num 1588) 2024-03-04T06:23:10.622 INFO:tasks.rados.rados.0.smithi064.stdout:3570: finishing write tid 1 to smithi06431743-1 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:3570: finishing write tid 2 to smithi06431743-1 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:3567: done (3 left) 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:3568: done (2 left) 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:3571: done (1 left) 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:3572: snap_create 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:3570: finishing write tid 3 to smithi06431743-1 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:3570: finishing write tid 4 to smithi06431743-1 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:3570: oid 1 updating version 0 to 1739 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:3570: oid 1 version 1739 is already newer than 1738 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:3570: oid 1 version 1739 is already newer than 1737 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 1739 (ObjNum 1617 snap 460 seq_num 1617) dirty exists 2024-03-04T06:23:10.623 INFO:tasks.rados.rados.0.smithi064.stdout:3570: left oid 1 (ObjNum 1617 snap 460 seq_num 1617) 2024-03-04T06:23:11.419 INFO:tasks.rados.rados.0.smithi064.stdout:3570: done (1 left) 2024-03-04T06:23:11.420 INFO:tasks.rados.rados.0.smithi064.stdout:3572: done (0 left) 2024-03-04T06:23:11.420 INFO:tasks.rados.rados.0.smithi064.stdout:3573: snap_create 2024-03-04T06:23:11.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:12.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:12.055+0000 7f5c8f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:12.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:12.060+0000 7f5c8f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:12.128 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:23:12.128 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:23:12.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:12.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:12.232+0000 7f181195d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:12.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:12.238+0000 7f181195d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:12.420 INFO:tasks.rados.rados.0.smithi064.stdout:3573: done (0 left) 2024-03-04T06:23:12.420 INFO:tasks.rados.rados.0.smithi064.stdout:3574: read oid 4 snap 434 2024-03-04T06:23:12.421 INFO:tasks.rados.rados.0.smithi064.stdout:3574: expect (ObjNum 1506 snap 421 seq_num 1506) 2024-03-04T06:23:12.421 INFO:tasks.rados.rados.0.smithi064.stdout:3575: snap_remove snap 450 2024-03-04T06:23:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:12.428 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:12.441 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1375,"stamp":"2024-03-04T06:23:12.419366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109060261,"num_objects":1189,"num_object_clones":975,"num_object_copies":3567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":175,"num_read":6496,"num_read_kb":5079702,"num_write":5512,"num_write_kb":1468293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15703,"ondisk_log_size":15703,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":663584,"kb_used_data":423680,"kb_used_omap":0,"kb_used_meta":239104,"kb_avail":376823776,"statfs":{"total":386547056640,"available":385867546624,"internally_reserved":0,"allocated":433848320,"data_stored":488716420,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":244842496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2302559,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":56,"num_read_kb":44379,"num_write":35,"num_write_kb":10376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011695"},"pg_stats":[{"pgid":"3.f","version":"916'860","reported_seq":3129,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.450859+0000","last_change":"2024-03-04T06:23:08.404520+0000","last_active":"2024-03-04T06:23:09.450859+0000","last_peered":"2024-03-04T06:23:09.450859+0000","last_clean":"2024-03-04T06:23:09.450859+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:09.450859+0000","last_undegraded":"2024-03-04T06:23:09.450859+0000","last_fullsized":"2024-03-04T06:23:09.450859+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":860,"log_dups_size":0,"ondisk_log_size":860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5984084,"num_objects":75,"num_object_clones":60,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":348,"num_read_kb":273131,"num_write":317,"num_write_kb":83905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"914'788","reported_seq":3123,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.425715+0000","last_change":"2024-03-04T06:23:07.440337+0000","last_active":"2024-03-04T06:23:09.425715+0000","last_peered":"2024-03-04T06:23:09.425715+0000","last_clean":"2024-03-04T06:23:09.425715+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:09.425715+0000","last_undegraded":"2024-03-04T06:23:09.425715+0000","last_fullsized":"2024-03-04T06:23:09.425715+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"912'1188","reported_seq":3327,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.430977+0000","last_change":"2024-03-04T06:23:10.436246+0000","last_active":"2024-03-04T06:23:10.430977+0000","last_peered":"2024-03-04T06:23:10.430977+0000","last_clean":"2024-03-04T06:23:10.430977+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:10.430977+0000","last_undegraded":"2024-03-04T06:23:10.430977+0000","last_fullsized":"2024-03-04T06:23:10.430977+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1188,"log_dups_size":0,"ondisk_log_size":1188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5631137,"num_objects":94,"num_object_clones":79,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":553,"num_read_kb":444395,"num_write":422,"num_write_kb":114599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"910'964","reported_seq":3108,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.440812+0000","last_change":"2024-03-04T06:23:08.377600+0000","last_active":"2024-03-04T06:23:09.440812+0000","last_peered":"2024-03-04T06:23:09.440812+0000","last_clean":"2024-03-04T06:23:09.440812+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:09.440812+0000","last_undegraded":"2024-03-04T06:23:09.440812+0000","last_fullsized":"2024-03-04T06:23:09.440812+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":964,"log_dups_size":0,"ondisk_log_size":964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849595,"num_objects":65,"num_object_clones":55,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":315,"num_read_kb":237438,"num_write":309,"num_write_kb":85718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"919'389","reported_seq":2921,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.474345+0000","last_change":"2024-03-04T06:23:10.560665+0000","last_active":"2024-03-04T06:23:10.474345+0000","last_peered":"2024-03-04T06:23:10.474345+0000","last_clean":"2024-03-04T06:23:10.474345+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:10.474345+0000","last_undegraded":"2024-03-04T06:23:10.474345+0000","last_fullsized":"2024-03-04T06:23:10.474345+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":120,"num_read_kb":93147,"num_write":120,"num_write_kb":33092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"916'960","reported_seq":3184,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.452223+0000","last_change":"2024-03-04T06:23:08.414224+0000","last_active":"2024-03-04T06:23:09.452223+0000","last_peered":"2024-03-04T06:23:09.452223+0000","last_clean":"2024-03-04T06:23:09.452223+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:09.452223+0000","last_undegraded":"2024-03-04T06:23:09.452223+0000","last_fullsized":"2024-03-04T06:23:09.452223+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":960,"log_dups_size":0,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8253425,"num_objects":70,"num_object_clones":58,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":423,"num_read_kb":338096,"num_write":338,"num_write_kb":97590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2689,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.460774+0000","last_change":"2024-03-04T06:23:08.389567+0000","last_active":"2024-03-04T06:23:09.460774+0000","last_peered":"2024-03-04T06:23:09.460774+0000","last_clean":"2024-03-04T06:23:09.460774+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:09.460774+0000","last_undegraded":"2024-03-04T06:23:09.460774+0000","last_fullsized":"2024-03-04T06:23:09.460774+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"919'620","reported_seq":3001,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.433444+0000","last_change":"2024-03-04T06:23:10.467729+0000","last_active":"2024-03-04T06:23:10.433444+0000","last_peered":"2024-03-04T06:23:10.433444+0000","last_clean":"2024-03-04T06:23:10.433444+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:10.433444+0000","last_undegraded":"2024-03-04T06:23:10.433444+0000","last_fullsized":"2024-03-04T06:23:10.433444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2464,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.470507+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:10.470507+0000","last_peered":"2024-03-04T06:23:10.470507+0000","last_clean":"2024-03-04T06:23:10.470507+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:10.470507+0000","last_undegraded":"2024-03-04T06:23:10.470507+0000","last_fullsized":"2024-03-04T06:23:10.470507+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2275,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.455605+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:10.455605+0000","last_peered":"2024-03-04T06:23:10.455605+0000","last_clean":"2024-03-04T06:23:10.455605+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:10.455605+0000","last_undegraded":"2024-03-04T06:23:10.455605+0000","last_fullsized":"2024-03-04T06:23:10.455605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"906'951","reported_seq":3186,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.438838+0000","last_change":"2024-03-04T06:23:10.441945+0000","last_active":"2024-03-04T06:23:10.438838+0000","last_peered":"2024-03-04T06:23:10.438838+0000","last_clean":"2024-03-04T06:23:10.438838+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:10.438838+0000","last_undegraded":"2024-03-04T06:23:10.438838+0000","last_fullsized":"2024-03-04T06:23:10.438838+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":423,"num_read_kb":341165,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2260,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.453860+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:09.453860+0000","last_peered":"2024-03-04T06:23:09.453860+0000","last_clean":"2024-03-04T06:23:09.453860+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:09.453860+0000","last_undegraded":"2024-03-04T06:23:09.453860+0000","last_fullsized":"2024-03-04T06:23:09.453860+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"910'629","reported_seq":3094,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.458485+0000","last_change":"2024-03-04T06:23:10.477399+0000","last_active":"2024-03-04T06:23:10.458485+0000","last_peered":"2024-03-04T06:23:10.458485+0000","last_clean":"2024-03-04T06:23:10.458485+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:10.458485+0000","last_undegraded":"2024-03-04T06:23:10.458485+0000","last_fullsized":"2024-03-04T06:23:10.458485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5379032,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":309,"num_read_kb":250022,"num_write":214,"num_write_kb":60883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2275,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.428583+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:10.428583+0000","last_peered":"2024-03-04T06:23:10.428583+0000","last_clean":"2024-03-04T06:23:10.428583+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:10.428583+0000","last_undegraded":"2024-03-04T06:23:10.428583+0000","last_fullsized":"2024-03-04T06:23:10.428583+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"919'1741","reported_seq":3696,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.567986+0000","last_change":"2024-03-04T06:23:10.573262+0000","last_active":"2024-03-04T06:23:10.567986+0000","last_peered":"2024-03-04T06:23:10.567986+0000","last_clean":"2024-03-04T06:23:10.567986+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:10.567986+0000","last_undegraded":"2024-03-04T06:23:10.567986+0000","last_fullsized":"2024-03-04T06:23:10.567986+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1741,"log_dups_size":0,"ondisk_log_size":1741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18019447,"num_objects":138,"num_object_clones":112,"num_object_copies":414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":782,"num_read_kb":618695,"num_write":648,"num_write_kb":174370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2260,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.449042+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:09.449042+0000","last_peered":"2024-03-04T06:23:09.449042+0000","last_clean":"2024-03-04T06:23:09.449042+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:09.449042+0000","last_undegraded":"2024-03-04T06:23:09.449042+0000","last_fullsized":"2024-03-04T06:23:09.449042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"918'2023","reported_seq":3716,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.503988+0000","last_change":"2024-03-04T06:23:07.438959+0000","last_active":"2024-03-04T06:23:09.503988+0000","last_peered":"2024-03-04T06:23:09.503988+0000","last_clean":"2024-03-04T06:23:09.503988+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:09.503988+0000","last_undegraded":"2024-03-04T06:23:09.503988+0000","last_fullsized":"2024-03-04T06:23:09.503988+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2023,"log_dups_size":0,"ondisk_log_size":2023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16758955,"num_objects":152,"num_object_clones":128,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":706,"num_read_kb":519419,"num_write":700,"num_write_kb":189443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"910'906","reported_seq":3190,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.454868+0000","last_change":"2024-03-04T06:23:10.458656+0000","last_active":"2024-03-04T06:23:10.454868+0000","last_peered":"2024-03-04T06:23:10.454868+0000","last_clean":"2024-03-04T06:23:10.454868+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:10.454868+0000","last_undegraded":"2024-03-04T06:23:10.454868+0000","last_fullsized":"2024-03-04T06:23:10.454868+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":906,"log_dups_size":0,"ondisk_log_size":906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269770,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":438,"num_read_kb":364480,"num_write":317,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2354,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.444727+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:09.444727+0000","last_peered":"2024-03-04T06:23:09.444727+0000","last_clean":"2024-03-04T06:23:09.444727+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:09.444727+0000","last_undegraded":"2024-03-04T06:23:09.444727+0000","last_fullsized":"2024-03-04T06:23:09.444727+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"910'1033","reported_seq":3257,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.437530+0000","last_change":"2024-03-04T06:23:07.445264+0000","last_active":"2024-03-04T06:23:09.437530+0000","last_peered":"2024-03-04T06:23:09.437530+0000","last_clean":"2024-03-04T06:23:09.437530+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:09.437530+0000","last_undegraded":"2024-03-04T06:23:09.437530+0000","last_fullsized":"2024-03-04T06:23:09.437530+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1033,"log_dups_size":0,"ondisk_log_size":1033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3412184,"num_objects":81,"num_object_clones":65,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":379,"num_read_kb":277363,"num_write":346,"num_write_kb":92772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2331,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.471144+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:10.471144+0000","last_peered":"2024-03-04T06:23:10.471144+0000","last_clean":"2024-03-04T06:23:10.471144+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:10.471144+0000","last_undegraded":"2024-03-04T06:23:10.471144+0000","last_fullsized":"2024-03-04T06:23:10.471144+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"910'1389","reported_seq":3384,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.467718+0000","last_change":"2024-03-04T06:23:10.476848+0000","last_active":"2024-03-04T06:23:10.467718+0000","last_peered":"2024-03-04T06:23:10.467718+0000","last_clean":"2024-03-04T06:23:10.467718+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:10.467718+0000","last_undegraded":"2024-03-04T06:23:10.467718+0000","last_fullsized":"2024-03-04T06:23:10.467718+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1389,"log_dups_size":0,"ondisk_log_size":1389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6740366,"num_objects":97,"num_object_clones":82,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":494,"num_read_kb":386810,"num_write":452,"num_write_kb":125521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2354,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.432644+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:09.432644+0000","last_peered":"2024-03-04T06:23:09.432644+0000","last_clean":"2024-03-04T06:23:09.432644+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:09.432644+0000","last_undegraded":"2024-03-04T06:23:09.432644+0000","last_fullsized":"2024-03-04T06:23:09.432644+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"916'1165","reported_seq":3429,"reported_epoch":918,"state":"active+clean","last_fresh":"2024-03-04T06:23:09.730298+0000","last_change":"2024-03-04T06:23:07.456324+0000","last_active":"2024-03-04T06:23:09.730298+0000","last_peered":"2024-03-04T06:23:09.730298+0000","last_clean":"2024-03-04T06:23:09.730298+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:09.730298+0000","last_undegraded":"2024-03-04T06:23:09.730298+0000","last_fullsized":"2024-03-04T06:23:09.730298+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1165,"log_dups_size":0,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8236618,"num_objects":96,"num_object_clones":82,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":574,"num_read_kb":484995,"num_write":430,"num_write_kb":117750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2275,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-03-04T06:23:10.439486+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:10.439486+0000","last_peered":"2024-03-04T06:23:10.439486+0000","last_clean":"2024-03-04T06:23:10.439486+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:10.439486+0000","last_undegraded":"2024-03-04T06:23:10.439486+0000","last_fullsized":"2024-03-04T06:23:10.439486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":108469893,"num_objects":1187,"num_object_clones":975,"num_object_copies":3561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":175,"num_read":6393,"num_read_kb":5079611,"num_write":5375,"num_write_kb":1466399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15606,"ondisk_log_size":15606,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":200748,"kb_used_data":133564,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94171092,"statfs":{"total":96636764160,"available":96431198208,"internally_reserved":0,"allocated":136769536,"data_stored":154704997,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153964,"kb_used_data":96500,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94217876,"statfs":{"total":96636764160,"available":96479105024,"internally_reserved":0,"allocated":98816000,"data_stored":107695825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706588,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":158440,"kb_used_data":100864,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94213400,"statfs":{"total":96636764160,"available":96474521600,"internally_reserved":0,"allocated":103284736,"data_stored":118877006,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706588,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":150432,"kb_used_data":92752,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94221408,"statfs":{"total":96636764160,"available":96482721792,"internally_reserved":0,"allocated":94978048,"data_stored":107438592,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:13.422 INFO:tasks.rados.rados.0.smithi064.stdout:3575: done (1 left) 2024-03-04T06:23:13.422 INFO:tasks.rados.rados.0.smithi064.stdout:3576: write oid 5 current snap is 462 2024-03-04T06:23:13.422 INFO:tasks.rados.rados.0.smithi064.stdout:3576: seq_num 1619 ranges {410533=494967,1673406=516228,2843590=366943} 2024-03-04T06:23:13.426 INFO:tasks.rados.rados.0.smithi064.stdout:3576: writing smithi06431743-5 from 410533 to 905500 tid 1 2024-03-04T06:23:13.433 INFO:tasks.rados.rados.0.smithi064.stdout:3576: writing smithi06431743-5 from 1673406 to 2189634 tid 2 2024-03-04T06:23:13.439 INFO:tasks.rados.rados.0.smithi064.stdout:3576: writing smithi06431743-5 from 2843590 to 3210533 tid 3 2024-03-04T06:23:13.439 INFO:tasks.rados.rados.0.smithi064.stdout:3577: snap_remove snap 440 2024-03-04T06:23:14.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:14.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:14.235+0000 7ff81691d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:14.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:14.239+0000 7ff81691d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:14.426 INFO:tasks.rados.rados.0.smithi064.stdout:3577: done (2 left) 2024-03-04T06:23:14.427 INFO:tasks.rados.rados.0.smithi064.stdout:3578: snap_create 2024-03-04T06:23:14.427 INFO:tasks.rados.rados.0.smithi064.stdout:3574: expect (ObjNum 1506 snap 421 seq_num 1506) 2024-03-04T06:23:14.592 INFO:tasks.rados.rados.0.smithi064.stdout:3576: finishing write tid 1 to smithi06431743-5 2024-03-04T06:23:14.592 INFO:tasks.rados.rados.0.smithi064.stdout:3576: finishing write tid 2 to smithi06431743-5 2024-03-04T06:23:14.593 INFO:tasks.rados.rados.0.smithi064.stdout:3576: finishing write tid 3 to smithi06431743-5 2024-03-04T06:23:14.593 INFO:tasks.rados.rados.0.smithi064.stdout:3576: finishing write tid 4 to smithi06431743-5 2024-03-04T06:23:14.593 INFO:tasks.rados.rados.0.smithi064.stdout:3576: finishing write tid 5 to smithi06431743-5 2024-03-04T06:23:14.593 INFO:tasks.rados.rados.0.smithi064.stdout:3576: oid 5 updating version 0 to 2033 2024-03-04T06:23:14.593 INFO:tasks.rados.rados.0.smithi064.stdout:3576: oid 5 updating version 2033 to 2035 2024-03-04T06:23:14.593 INFO:tasks.rados.rados.0.smithi064.stdout:3576: oid 5 updating version 2035 to 2036 2024-03-04T06:23:14.593 INFO:tasks.rados.rados.0.smithi064.stdout:3576: oid 5 version 2036 is already newer than 2035 2024-03-04T06:23:14.593 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 2036 (ObjNum 1619 snap 462 seq_num 1619) dirty exists 2024-03-04T06:23:14.593 INFO:tasks.rados.rados.0.smithi064.stdout:3576: left oid 5 (ObjNum 1619 snap 462 seq_num 1619) 2024-03-04T06:23:15.438 INFO:tasks.rados.rados.0.smithi064.stdout:3574: done (2 left) 2024-03-04T06:23:15.438 INFO:tasks.rados.rados.0.smithi064.stdout:3576: done (1 left) 2024-03-04T06:23:15.438 INFO:tasks.rados.rados.0.smithi064.stdout:3578: done (0 left) 2024-03-04T06:23:15.438 INFO:tasks.rados.rados.0.smithi064.stdout:3579: delete oid 28 current snap is 463 2024-03-04T06:23:15.521 INFO:tasks.rados.rados.0.smithi064.stdout:3579: done (0 left) 2024-03-04T06:23:15.521 INFO:tasks.rados.rados.0.smithi064.stdout:3580: write oid 25 current snap is 463 2024-03-04T06:23:15.521 INFO:tasks.rados.rados.0.smithi064.stdout:3580: seq_num 1621 ranges {444147=761818,1790370=519897,2898436=345711} 2024-03-04T06:23:15.527 INFO:tasks.rados.rados.0.smithi064.stdout:3580: writing smithi06431743-25 from 444147 to 1205965 tid 1 2024-03-04T06:23:15.533 INFO:tasks.rados.rados.0.smithi064.stdout:3580: writing smithi06431743-25 from 1790370 to 2310267 tid 2 2024-03-04T06:23:15.539 INFO:tasks.rados.rados.0.smithi064.stdout:3580: writing smithi06431743-25 from 2898436 to 3244147 tid 3 2024-03-04T06:23:15.539 INFO:tasks.rados.rados.0.smithi064.stdout:3581: write oid 33 current snap is 463 2024-03-04T06:23:15.539 INFO:tasks.rados.rados.0.smithi064.stdout:3581: seq_num 1622 ranges {460954=437067,1448852=663555,2525859=735095} 2024-03-04T06:23:15.543 INFO:tasks.rados.rados.0.smithi064.stdout:3581: writing smithi06431743-33 from 460954 to 898021 tid 1 2024-03-04T06:23:15.550 INFO:tasks.rados.rados.0.smithi064.stdout:3581: writing smithi06431743-33 from 1448852 to 2112407 tid 2 2024-03-04T06:23:15.557 INFO:tasks.rados.rados.0.smithi064.stdout:3581: writing smithi06431743-33 from 2525859 to 3260954 tid 3 2024-03-04T06:23:15.557 INFO:tasks.rados.rados.0.smithi064.stdout:3582: write (excl) oid 44 current snap is 463 2024-03-04T06:23:15.557 INFO:tasks.rados.rados.0.smithi064.stdout:3582: seq_num 1623 ranges {477761=512316,1507334=523566,2553282=599771} 2024-03-04T06:23:15.562 INFO:tasks.rados.rados.0.smithi064.stdout:3582: writing smithi06431743-44 from 477761 to 990077 tid 1 2024-03-04T06:23:15.569 INFO:tasks.rados.rados.0.smithi064.stdout:3582: writing smithi06431743-44 from 1507334 to 2030900 tid 2 2024-03-04T06:23:15.578 INFO:tasks.rados.rados.0.smithi064.stdout:3582: writing smithi06431743-44 from 2553282 to 3153053 tid 3 2024-03-04T06:23:15.578 INFO:tasks.rados.rados.0.smithi064.stdout:3583: write (excl) oid 40 current snap is 463 2024-03-04T06:23:15.578 INFO:tasks.rados.rados.0.smithi064.stdout:3583: seq_num 1624 ranges {494568=587565,1565816=667224,2980705=313863} 2024-03-04T06:23:15.586 INFO:tasks.rados.rados.0.smithi064.stdout:3583: writing smithi06431743-40 from 494568 to 1082133 tid 1 2024-03-04T06:23:15.594 INFO:tasks.rados.rados.0.smithi064.stdout:3583: writing smithi06431743-40 from 1565816 to 2233040 tid 2 2024-03-04T06:23:15.602 INFO:tasks.rados.rados.0.smithi064.stdout:3583: writing smithi06431743-40 from 2980705 to 3294568 tid 3 2024-03-04T06:23:15.602 INFO:tasks.rados.rados.0.smithi064.stdout:3584: snap_create 2024-03-04T06:23:15.639 INFO:tasks.rados.rados.0.smithi064.stdout:3582: finishing write tid 1 to smithi06431743-44 2024-03-04T06:23:15.640 INFO:tasks.rados.rados.0.smithi064.stdout:3581: finishing write tid 1 to smithi06431743-33 2024-03-04T06:23:15.645 INFO:tasks.rados.rados.0.smithi064.stdout:3583: finishing write tid 1 to smithi06431743-40 2024-03-04T06:23:15.673 INFO:tasks.rados.rados.0.smithi064.stdout:3582: finishing write tid 2 to smithi06431743-44 2024-03-04T06:23:15.673 INFO:tasks.rados.rados.0.smithi064.stdout:3581: finishing write tid 2 to smithi06431743-33 2024-03-04T06:23:15.674 INFO:tasks.rados.rados.0.smithi064.stdout:3580: finishing write tid 1 to smithi06431743-25 2024-03-04T06:23:15.676 INFO:tasks.rados.rados.0.smithi064.stdout:3583: finishing write tid 2 to smithi06431743-40 2024-03-04T06:23:15.698 INFO:tasks.rados.rados.0.smithi064.stdout:3582: finishing write tid 3 to smithi06431743-44 2024-03-04T06:23:15.698 INFO:tasks.rados.rados.0.smithi064.stdout:3581: finishing write tid 3 to smithi06431743-33 2024-03-04T06:23:15.699 INFO:tasks.rados.rados.0.smithi064.stdout:3580: finishing write tid 2 to smithi06431743-25 2024-03-04T06:23:15.702 INFO:tasks.rados.rados.0.smithi064.stdout:3583: finishing write tid 3 to smithi06431743-40 2024-03-04T06:23:15.706 INFO:tasks.rados.rados.0.smithi064.stdout:3582: finishing write tid 4 to smithi06431743-44 2024-03-04T06:23:15.710 INFO:tasks.rados.rados.0.smithi064.stdout:3582: finishing write tid 5 to smithi06431743-44 2024-03-04T06:23:15.710 INFO:tasks.rados.rados.0.smithi064.stdout:3582: oid 44 updating version 0 to 964 2024-03-04T06:23:15.710 INFO:tasks.rados.rados.0.smithi064.stdout:3582: oid 44 updating version 964 to 967 2024-03-04T06:23:15.710 INFO:tasks.rados.rados.0.smithi064.stdout:3582: oid 44 version 967 is already newer than 967 2024-03-04T06:23:15.710 INFO:tasks.rados.rados.0.smithi064.stdout:3582: oid 44 version 967 is already newer than 965 2024-03-04T06:23:15.710 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 967 (ObjNum 1623 snap 463 seq_num 1623) dirty exists 2024-03-04T06:23:15.710 INFO:tasks.rados.rados.0.smithi064.stdout:3582: left oid 44 (ObjNum 1623 snap 463 seq_num 1623) 2024-03-04T06:23:15.710 INFO:tasks.rados.rados.0.smithi064.stdout:3581: finishing write tid 4 to smithi06431743-33 2024-03-04T06:23:15.711 INFO:tasks.rados.rados.0.smithi064.stdout:3581: finishing write tid 5 to smithi06431743-33 2024-03-04T06:23:15.712 INFO:tasks.rados.rados.0.smithi064.stdout:3581: oid 33 updating version 0 to 1184 2024-03-04T06:23:15.712 INFO:tasks.rados.rados.0.smithi064.stdout:3581: oid 33 version 1184 is already newer than 1182 2024-03-04T06:23:15.712 INFO:tasks.rados.rados.0.smithi064.stdout:3581: oid 33 version 1184 is already newer than 1184 2024-03-04T06:23:15.712 INFO:tasks.rados.rados.0.smithi064.stdout:3581: oid 33 version 1184 is already newer than 1182 2024-03-04T06:23:15.712 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 1184 (ObjNum 1622 snap 463 seq_num 1622) dirty exists 2024-03-04T06:23:15.712 INFO:tasks.rados.rados.0.smithi064.stdout:3581: left oid 33 (ObjNum 1622 snap 463 seq_num 1622) 2024-03-04T06:23:15.715 INFO:tasks.rados.rados.0.smithi064.stdout:3580: finishing write tid 3 to smithi06431743-25 2024-03-04T06:23:15.716 INFO:tasks.rados.rados.0.smithi064.stdout:3583: finishing write tid 4 to smithi06431743-40 2024-03-04T06:23:15.716 INFO:tasks.rados.rados.0.smithi064.stdout:3580: finishing write tid 4 to smithi06431743-25 2024-03-04T06:23:15.717 INFO:tasks.rados.rados.0.smithi064.stdout:3583: finishing write tid 5 to smithi06431743-40 2024-03-04T06:23:15.717 INFO:tasks.rados.rados.0.smithi064.stdout:3583: oid 40 updating version 0 to 873 2024-03-04T06:23:15.717 INFO:tasks.rados.rados.0.smithi064.stdout:3583: oid 40 version 873 is already newer than 872 2024-03-04T06:23:15.717 INFO:tasks.rados.rados.0.smithi064.stdout:3583: oid 40 version 873 is already newer than 871 2024-03-04T06:23:15.717 INFO:tasks.rados.rados.0.smithi064.stdout:3583: oid 40 version 873 is already newer than 870 2024-03-04T06:23:15.717 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 873 (ObjNum 1624 snap 463 seq_num 1624) dirty exists 2024-03-04T06:23:15.717 INFO:tasks.rados.rados.0.smithi064.stdout:3583: left oid 40 (ObjNum 1624 snap 463 seq_num 1624) 2024-03-04T06:23:15.719 INFO:tasks.rados.rados.0.smithi064.stdout:3580: finishing write tid 5 to smithi06431743-25 2024-03-04T06:23:15.719 INFO:tasks.rados.rados.0.smithi064.stdout:3580: oid 25 updating version 0 to 1201 2024-03-04T06:23:15.719 INFO:tasks.rados.rados.0.smithi064.stdout:3580: oid 25 version 1201 is already newer than 1198 2024-03-04T06:23:15.719 INFO:tasks.rados.rados.0.smithi064.stdout:3580: oid 25 version 1201 is already newer than 1199 2024-03-04T06:23:15.719 INFO:tasks.rados.rados.0.smithi064.stdout:3580: oid 25 version 1201 is already newer than 1201 2024-03-04T06:23:15.719 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 1201 (ObjNum 1621 snap 463 seq_num 1621) dirty exists 2024-03-04T06:23:15.719 INFO:tasks.rados.rados.0.smithi064.stdout:3580: left oid 25 (ObjNum 1621 snap 463 seq_num 1621) 2024-03-04T06:23:16.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:16.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:16.411+0000 7f0c5c4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:16.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:16.415+0000 7f0c5c4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:16.443 INFO:tasks.rados.rados.0.smithi064.stdout:3580: done (4 left) 2024-03-04T06:23:16.443 INFO:tasks.rados.rados.0.smithi064.stdout:3581: done (3 left) 2024-03-04T06:23:16.443 INFO:tasks.rados.rados.0.smithi064.stdout:3582: done (2 left) 2024-03-04T06:23:16.443 INFO:tasks.rados.rados.0.smithi064.stdout:3583: done (1 left) 2024-03-04T06:23:16.443 INFO:tasks.rados.rados.0.smithi064.stdout:3584: done (0 left) 2024-03-04T06:23:16.443 INFO:tasks.rados.rados.0.smithi064.stdout:3585: delete oid 39 current snap is 464 2024-03-04T06:23:16.479 INFO:tasks.rados.rados.0.smithi064.stdout:3585: done (0 left) 2024-03-04T06:23:16.479 INFO:tasks.rados.rados.0.smithi064.stdout:3586: write oid 44 current snap is 464 2024-03-04T06:23:16.479 INFO:tasks.rados.rados.0.smithi064.stdout:3586: seq_num 1626 ranges {528182=738063,1966427=670893,3319198=8984} 2024-03-04T06:23:16.486 INFO:tasks.rados.rados.0.smithi064.stdout:3586: writing smithi06431743-44 from 528182 to 1266245 tid 1 2024-03-04T06:23:16.493 INFO:tasks.rados.rados.0.smithi064.stdout:3586: writing smithi06431743-44 from 1966427 to 2637320 tid 2 2024-03-04T06:23:16.498 INFO:tasks.rados.rados.0.smithi064.stdout:3586: writing smithi06431743-44 from 3319198 to 3328182 tid 3 2024-03-04T06:23:16.498 INFO:tasks.rados.rados.0.smithi064.stdout:3587: read oid 12 snap -1 2024-03-04T06:23:16.498 INFO:tasks.rados.rados.0.smithi064.stdout:3587: expect (ObjNum 1501 snap 418 seq_num 1501) 2024-03-04T06:23:16.498 INFO:tasks.rados.rados.0.smithi064.stdout:3588: write (excl) oid 26 current snap is 464 2024-03-04T06:23:16.498 INFO:tasks.rados.rados.0.smithi064.stdout:3588: seq_num 1627 ranges {544989=529665,1741262=530904,3062974=282015} 2024-03-04T06:23:16.503 INFO:tasks.rados.rados.0.smithi064.stdout:3588: writing smithi06431743-26 from 544989 to 1074654 tid 1 2024-03-04T06:23:16.510 INFO:tasks.rados.rados.0.smithi064.stdout:3588: writing smithi06431743-26 from 1741262 to 2272166 tid 2 2024-03-04T06:23:16.516 INFO:tasks.rados.rados.0.smithi064.stdout:3588: writing smithi06431743-26 from 3062974 to 3344989 tid 3 2024-03-04T06:23:16.516 INFO:tasks.rados.rados.0.smithi064.stdout:3589: write (excl) oid 29 current snap is 464 2024-03-04T06:23:16.516 INFO:tasks.rados.rados.0.smithi064.stdout:3589: seq_num 1628 ranges {561796=604914,1799744=674562,3090397=271399} 2024-03-04T06:23:16.523 INFO:tasks.rados.rados.0.smithi064.stdout:3589: writing smithi06431743-29 from 561796 to 1166710 tid 1 2024-03-04T06:23:16.530 INFO:tasks.rados.rados.0.smithi064.stdout:3589: writing smithi06431743-29 from 1799744 to 2474306 tid 2 2024-03-04T06:23:16.535 INFO:tasks.rados.rados.0.smithi064.stdout:3589: writing smithi06431743-29 from 3090397 to 3361796 tid 3 2024-03-04T06:23:16.535 INFO:tasks.rados.rados.0.smithi064.stdout:3590: snap_create 2024-03-04T06:23:16.535 INFO:tasks.rados.rados.0.smithi064.stdout:3587: expect (ObjNum 1501 snap 418 seq_num 1501) 2024-03-04T06:23:16.759 INFO:tasks.rados.rados.0.smithi064.stdout:3586: finishing write tid 1 to smithi06431743-44 2024-03-04T06:23:16.759 INFO:tasks.rados.rados.0.smithi064.stdout:3588: finishing write tid 1 to smithi06431743-26 2024-03-04T06:23:16.759 INFO:tasks.rados.rados.0.smithi064.stdout:3586: finishing write tid 2 to smithi06431743-44 2024-03-04T06:23:16.759 INFO:tasks.rados.rados.0.smithi064.stdout:3588: finishing write tid 2 to smithi06431743-26 2024-03-04T06:23:16.759 INFO:tasks.rados.rados.0.smithi064.stdout:3586: finishing write tid 3 to smithi06431743-44 2024-03-04T06:23:16.759 INFO:tasks.rados.rados.0.smithi064.stdout:3588: finishing write tid 3 to smithi06431743-26 2024-03-04T06:23:16.759 INFO:tasks.rados.rados.0.smithi064.stdout:3589: finishing write tid 1 to smithi06431743-29 2024-03-04T06:23:16.759 INFO:tasks.rados.rados.0.smithi064.stdout:3588: finishing write tid 4 to smithi06431743-26 2024-03-04T06:23:16.759 INFO:tasks.rados.rados.0.smithi064.stdout:3588: finishing write tid 5 to smithi06431743-26 2024-03-04T06:23:16.759 INFO:tasks.rados.rados.0.smithi064.stdout:3588: oid 26 updating version 0 to 1764 2024-03-04T06:23:16.759 INFO:tasks.rados.rados.0.smithi064.stdout:3588: oid 26 version 1764 is already newer than 1762 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:3588: oid 26 version 1764 is already newer than 1761 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:3588: oid 26 version 1764 is already newer than 1764 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 1764 (ObjNum 1627 snap 464 seq_num 1627) dirty exists 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:3588: left oid 26 (ObjNum 1627 snap 464 seq_num 1627) 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:3586: finishing write tid 4 to smithi06431743-44 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:3586: finishing write tid 5 to smithi06431743-44 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:3586: oid 44 updating version 0 to 972 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:3586: oid 44 version 972 is already newer than 971 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:3586: oid 44 version 972 is already newer than 969 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:3586: oid 44 version 972 is already newer than 970 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 972 (ObjNum 1626 snap 464 seq_num 1626) dirty exists 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:3586: left oid 44 (ObjNum 1626 snap 464 seq_num 1626) 2024-03-04T06:23:16.760 INFO:tasks.rados.rados.0.smithi064.stdout:3589: finishing write tid 2 to smithi06431743-29 2024-03-04T06:23:16.761 INFO:tasks.rados.rados.0.smithi064.stdout:3589: finishing write tid 3 to smithi06431743-29 2024-03-04T06:23:16.761 INFO:tasks.rados.rados.0.smithi064.stdout:3589: finishing write tid 4 to smithi06431743-29 2024-03-04T06:23:16.761 INFO:tasks.rados.rados.0.smithi064.stdout:3589: finishing write tid 5 to smithi06431743-29 2024-03-04T06:23:16.761 INFO:tasks.rados.rados.0.smithi064.stdout:3589: oid 29 updating version 0 to 2054 2024-03-04T06:23:16.761 INFO:tasks.rados.rados.0.smithi064.stdout:3589: oid 29 updating version 2054 to 2055 2024-03-04T06:23:16.761 INFO:tasks.rados.rados.0.smithi064.stdout:3589: oid 29 version 2055 is already newer than 2054 2024-03-04T06:23:16.761 INFO:tasks.rados.rados.0.smithi064.stdout:3589: oid 29 version 2055 is already newer than 2052 2024-03-04T06:23:16.761 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 2055 (ObjNum 1628 snap 464 seq_num 1628) dirty exists 2024-03-04T06:23:16.761 INFO:tasks.rados.rados.0.smithi064.stdout:3589: left oid 29 (ObjNum 1628 snap 464 seq_num 1628) 2024-03-04T06:23:17.442 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:23:17.443 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:23:17.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:17.444 INFO:tasks.rados.rados.0.smithi064.stdout:3586: done (4 left) 2024-03-04T06:23:17.444 INFO:tasks.rados.rados.0.smithi064.stdout:3587: done (3 left) 2024-03-04T06:23:17.444 INFO:tasks.rados.rados.0.smithi064.stdout:3588: done (2 left) 2024-03-04T06:23:17.444 INFO:tasks.rados.rados.0.smithi064.stdout:3589: done (1 left) 2024-03-04T06:23:17.444 INFO:tasks.rados.rados.0.smithi064.stdout:3590: done (0 left) 2024-03-04T06:23:17.444 INFO:tasks.rados.rados.0.smithi064.stdout:3591: snap_create 2024-03-04T06:23:17.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:17.553+0000 7fb12182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:17.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:17.559+0000 7fb12182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:17.749 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:17.750 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:17.762 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1383,"stamp":"2024-03-04T06:23:17.442492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103765297,"num_objects":1193,"num_object_clones":979,"num_object_copies":3579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":177,"num_read":6506,"num_read_kb":5088254,"num_write":5524,"num_write_kb":1471231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15817,"ondisk_log_size":15817,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":673460,"kb_used_data":433812,"kb_used_omap":0,"kb_used_meta":239104,"kb_avail":376813900,"statfs":{"total":386547056640,"available":385857433600,"internally_reserved":0,"allocated":444223488,"data_stored":505402390,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":244842496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5294964,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":10,"num_read_kb":8552,"num_write":12,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.022201"},"pg_stats":[{"pgid":"3.f","version":"922'864","reported_seq":3139,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-04T06:23:13.932943+0000","last_change":"2024-03-04T06:23:13.954959+0000","last_active":"2024-03-04T06:23:13.932943+0000","last_peered":"2024-03-04T06:23:13.932943+0000","last_clean":"2024-03-04T06:23:13.932943+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:13.932943+0000","last_undegraded":"2024-03-04T06:23:13.932943+0000","last_fullsized":"2024-03-04T06:23:13.932943+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5984084,"num_objects":75,"num_object_clones":60,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":348,"num_read_kb":273131,"num_write":317,"num_write_kb":83905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"914'788","reported_seq":3141,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-04T06:23:14.441419+0000","last_change":"2024-03-04T06:23:14.446091+0000","last_active":"2024-03-04T06:23:14.441419+0000","last_peered":"2024-03-04T06:23:14.441419+0000","last_clean":"2024-03-04T06:23:14.441419+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:14.441419+0000","last_undegraded":"2024-03-04T06:23:14.441419+0000","last_fullsized":"2024-03-04T06:23:14.441419+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"924'1203","reported_seq":3343,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.717915+0000","last_change":"2024-03-04T06:23:15.729474+0000","last_active":"2024-03-04T06:23:15.717915+0000","last_peered":"2024-03-04T06:23:15.717915+0000","last_clean":"2024-03-04T06:23:15.717915+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:15.717915+0000","last_undegraded":"2024-03-04T06:23:15.717915+0000","last_fullsized":"2024-03-04T06:23:15.717915+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1203,"log_dups_size":0,"ondisk_log_size":1203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6101733,"num_objects":95,"num_object_clones":80,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":554,"num_read_kb":444396,"num_write":427,"num_write_kb":116189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"922'970","reported_seq":3118,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-04T06:23:13.908214+0000","last_change":"2024-03-04T06:23:13.934144+0000","last_active":"2024-03-04T06:23:13.908214+0000","last_peered":"2024-03-04T06:23:13.908214+0000","last_clean":"2024-03-04T06:23:13.908214+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:13.908214+0000","last_undegraded":"2024-03-04T06:23:13.908214+0000","last_fullsized":"2024-03-04T06:23:13.908214+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849595,"num_objects":65,"num_object_clones":55,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":315,"num_read_kb":237438,"num_write":309,"num_write_kb":85718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"924'393","reported_seq":2932,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.470773+0000","last_change":"2024-03-04T06:23:15.503091+0000","last_active":"2024-03-04T06:23:15.470773+0000","last_peered":"2024-03-04T06:23:15.470773+0000","last_clean":"2024-03-04T06:23:15.470773+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:15.470773+0000","last_undegraded":"2024-03-04T06:23:15.470773+0000","last_fullsized":"2024-03-04T06:23:15.470773+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":120,"num_read_kb":93147,"num_write":120,"num_write_kb":33092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"919'962","reported_seq":3194,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-04T06:23:13.932086+0000","last_change":"2024-03-04T06:23:13.936659+0000","last_active":"2024-03-04T06:23:13.932086+0000","last_peered":"2024-03-04T06:23:13.932086+0000","last_clean":"2024-03-04T06:23:13.932086+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:13.932086+0000","last_undegraded":"2024-03-04T06:23:13.932086+0000","last_fullsized":"2024-03-04T06:23:13.932086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8253425,"num_objects":70,"num_object_clones":58,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":423,"num_read_kb":338096,"num_write":338,"num_write_kb":97590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2699,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-04T06:23:13.928393+0000","last_change":"2024-03-04T06:23:13.936355+0000","last_active":"2024-03-04T06:23:13.928393+0000","last_peered":"2024-03-04T06:23:13.928393+0000","last_clean":"2024-03-04T06:23:13.928393+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:13.928393+0000","last_undegraded":"2024-03-04T06:23:13.928393+0000","last_fullsized":"2024-03-04T06:23:13.928393+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"924'626","reported_seq":3012,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.560894+0000","last_change":"2024-03-04T06:23:15.624852+0000","last_active":"2024-03-04T06:23:15.560894+0000","last_peered":"2024-03-04T06:23:15.560894+0000","last_clean":"2024-03-04T06:23:15.560894+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:15.560894+0000","last_undegraded":"2024-03-04T06:23:15.560894+0000","last_fullsized":"2024-03-04T06:23:15.560894+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2473,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.461140+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:15.461140+0000","last_peered":"2024-03-04T06:23:15.461140+0000","last_clean":"2024-03-04T06:23:15.461140+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:15.461140+0000","last_undegraded":"2024-03-04T06:23:15.461140+0000","last_fullsized":"2024-03-04T06:23:15.461140+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2284,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.593875+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:15.593875+0000","last_peered":"2024-03-04T06:23:15.593875+0000","last_clean":"2024-03-04T06:23:15.593875+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:15.593875+0000","last_undegraded":"2024-03-04T06:23:15.593875+0000","last_fullsized":"2024-03-04T06:23:15.593875+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"906'951","reported_seq":3197,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.572662+0000","last_change":"2024-03-04T06:23:15.575935+0000","last_active":"2024-03-04T06:23:15.572662+0000","last_peered":"2024-03-04T06:23:15.572662+0000","last_clean":"2024-03-04T06:23:15.572662+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:15.572662+0000","last_undegraded":"2024-03-04T06:23:15.572662+0000","last_fullsized":"2024-03-04T06:23:15.572662+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":423,"num_read_kb":341165,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2268,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-04T06:23:13.930482+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:13.930482+0000","last_peered":"2024-03-04T06:23:13.930482+0000","last_clean":"2024-03-04T06:23:13.930482+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:13.930482+0000","last_undegraded":"2024-03-04T06:23:13.930482+0000","last_fullsized":"2024-03-04T06:23:13.930482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"924'633","reported_seq":3105,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.461993+0000","last_change":"2024-03-04T06:23:15.497119+0000","last_active":"2024-03-04T06:23:15.461993+0000","last_peered":"2024-03-04T06:23:15.461993+0000","last_clean":"2024-03-04T06:23:15.461993+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:15.461993+0000","last_undegraded":"2024-03-04T06:23:15.461993+0000","last_fullsized":"2024-03-04T06:23:15.461993+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":633,"log_dups_size":0,"ondisk_log_size":633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5379032,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":309,"num_read_kb":250022,"num_write":214,"num_write_kb":60883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2284,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.563785+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:15.563785+0000","last_peered":"2024-03-04T06:23:15.563785+0000","last_clean":"2024-03-04T06:23:15.563785+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:15.563785+0000","last_undegraded":"2024-03-04T06:23:15.563785+0000","last_fullsized":"2024-03-04T06:23:15.563785+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"924'1759","reported_seq":3708,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.502775+0000","last_change":"2024-03-04T06:23:15.554958+0000","last_active":"2024-03-04T06:23:15.502775+0000","last_peered":"2024-03-04T06:23:15.502775+0000","last_clean":"2024-03-04T06:23:15.502775+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:15.502775+0000","last_undegraded":"2024-03-04T06:23:15.502775+0000","last_fullsized":"2024-03-04T06:23:15.502775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1759,"log_dups_size":0,"ondisk_log_size":1759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15262703,"num_objects":139,"num_object_clones":113,"num_object_copies":417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":783,"num_read_kb":618695,"num_write":649,"num_write_kb":174370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2268,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-04T06:23:13.917912+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:13.917912+0000","last_peered":"2024-03-04T06:23:13.917912+0000","last_clean":"2024-03-04T06:23:13.917912+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:13.917912+0000","last_undegraded":"2024-03-04T06:23:13.917912+0000","last_fullsized":"2024-03-04T06:23:13.917912+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"923'2050","reported_seq":3742,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-04T06:23:14.435373+0000","last_change":"2024-03-04T06:23:14.542232+0000","last_active":"2024-03-04T06:23:14.435373+0000","last_peered":"2024-03-04T06:23:14.435373+0000","last_clean":"2024-03-04T06:23:14.435373+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:14.435373+0000","last_undegraded":"2024-03-04T06:23:14.435373+0000","last_fullsized":"2024-03-04T06:23:14.435373+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2050,"log_dups_size":0,"ondisk_log_size":2050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17162323,"num_objects":153,"num_object_clones":129,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":18,"num_read":713,"num_read_kb":527970,"num_write":705,"num_write_kb":190791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"924'908","reported_seq":3201,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.598497+0000","last_change":"2024-03-04T06:23:15.637181+0000","last_active":"2024-03-04T06:23:15.598497+0000","last_peered":"2024-03-04T06:23:15.598497+0000","last_clean":"2024-03-04T06:23:15.598497+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:15.598497+0000","last_undegraded":"2024-03-04T06:23:15.598497+0000","last_fullsized":"2024-03-04T06:23:15.598497+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":908,"log_dups_size":0,"ondisk_log_size":908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269770,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":438,"num_read_kb":364480,"num_write":317,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2369,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-04T06:23:14.445572+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:14.445572+0000","last_peered":"2024-03-04T06:23:14.445572+0000","last_clean":"2024-03-04T06:23:14.445572+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:14.445572+0000","last_undegraded":"2024-03-04T06:23:14.445572+0000","last_fullsized":"2024-03-04T06:23:14.445572+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"923'1043","reported_seq":3276,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-04T06:23:14.446544+0000","last_change":"2024-03-04T06:23:14.519400+0000","last_active":"2024-03-04T06:23:14.446544+0000","last_peered":"2024-03-04T06:23:14.446544+0000","last_clean":"2024-03-04T06:23:14.446544+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:14.446544+0000","last_undegraded":"2024-03-04T06:23:14.446544+0000","last_fullsized":"2024-03-04T06:23:14.446544+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1043,"log_dups_size":0,"ondisk_log_size":1043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":380,"num_read_kb":277363,"num_write":347,"num_write_kb":92772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2340,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.471979+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:15.471979+0000","last_peered":"2024-03-04T06:23:15.471979+0000","last_clean":"2024-03-04T06:23:15.471979+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:15.471979+0000","last_undegraded":"2024-03-04T06:23:15.471979+0000","last_fullsized":"2024-03-04T06:23:15.471979+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"924'1391","reported_seq":3395,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.466716+0000","last_change":"2024-03-04T06:23:15.492287+0000","last_active":"2024-03-04T06:23:15.466716+0000","last_peered":"2024-03-04T06:23:15.466716+0000","last_clean":"2024-03-04T06:23:15.466716+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:15.466716+0000","last_undegraded":"2024-03-04T06:23:15.466716+0000","last_fullsized":"2024-03-04T06:23:15.466716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1391,"log_dups_size":0,"ondisk_log_size":1391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6740366,"num_objects":97,"num_object_clones":82,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":494,"num_read_kb":386810,"num_write":452,"num_write_kb":125521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2369,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-04T06:23:14.439347+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:14.439347+0000","last_peered":"2024-03-04T06:23:14.439347+0000","last_clean":"2024-03-04T06:23:14.439347+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:14.439347+0000","last_undegraded":"2024-03-04T06:23:14.439347+0000","last_fullsized":"2024-03-04T06:23:14.439347+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"923'1179","reported_seq":3447,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-04T06:23:14.438723+0000","last_change":"2024-03-04T06:23:14.519061+0000","last_active":"2024-03-04T06:23:14.438723+0000","last_peered":"2024-03-04T06:23:14.438723+0000","last_clean":"2024-03-04T06:23:14.438723+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:14.438723+0000","last_undegraded":"2024-03-04T06:23:14.438723+0000","last_fullsized":"2024-03-04T06:23:14.438723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1179,"log_dups_size":0,"ondisk_log_size":1179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8236618,"num_objects":96,"num_object_clones":82,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":574,"num_read_kb":484995,"num_write":430,"num_write_kb":117750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2284,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-04T06:23:15.573325+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:15.573325+0000","last_peered":"2024-03-04T06:23:15.573325+0000","last_clean":"2024-03-04T06:23:15.573325+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:15.573325+0000","last_undegraded":"2024-03-04T06:23:15.573325+0000","last_fullsized":"2024-03-04T06:23:15.573325+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":103174929,"num_objects":1191,"num_object_clones":979,"num_object_copies":3573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":177,"num_read":6403,"num_read_kb":5088163,"num_write":5387,"num_write_kb":1469337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15720,"ondisk_log_size":15720,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706594,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":205768,"kb_used_data":138584,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94166072,"statfs":{"total":96636764160,"available":96426057728,"internally_reserved":0,"allocated":141910016,"data_stored":162449235,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706594,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":160112,"kb_used_data":102560,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94211728,"statfs":{"total":96636764160,"available":96472809472,"internally_reserved":0,"allocated":105021440,"data_stored":116712976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":158300,"kb_used_data":100836,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94213540,"statfs":{"total":96636764160,"available":96474664960,"internally_reserved":0,"allocated":103256064,"data_stored":119386917,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":149280,"kb_used_data":91832,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94222560,"statfs":{"total":96636764160,"available":96483901440,"internally_reserved":0,"allocated":94035968,"data_stored":106853262,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:18.448 INFO:tasks.rados.rados.0.smithi064.stdout:3591: done (0 left) 2024-03-04T06:23:18.448 INFO:tasks.rados.rados.0.smithi064.stdout:3592: snap_remove snap 434 2024-03-04T06:23:18.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:18.624 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:18.622+0000 7fc35f75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:18.627 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:18.626+0000 7fc35f75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:19.448 INFO:tasks.rados.rados.0.smithi064.stdout:3592: done (0 left) 2024-03-04T06:23:19.448 INFO:tasks.rados.rados.0.smithi064.stdout:3593: write oid 14 current snap is 466 2024-03-04T06:23:19.448 INFO:tasks.rados.rados.0.smithi064.stdout:3593: seq_num 1629 ranges {578603=680163,1741873=534573,3001467=377136} 2024-03-04T06:23:19.454 INFO:tasks.rados.rados.0.smithi064.stdout:3593: writing smithi06431743-14 from 578603 to 1258766 tid 1 2024-03-04T06:23:19.460 INFO:tasks.rados.rados.0.smithi064.stdout:3593: writing smithi06431743-14 from 1741873 to 2276446 tid 2 2024-03-04T06:23:19.467 INFO:tasks.rados.rados.0.smithi064.stdout:3593: writing smithi06431743-14 from 3001467 to 3378603 tid 3 2024-03-04T06:23:19.467 INFO:tasks.rados.rados.0.smithi064.stdout:3594: write oid 24 current snap is 466 2024-03-04T06:23:19.467 INFO:tasks.rados.rados.0.smithi064.stdout:3594: seq_num 1630 ranges {595410=755412,1800355=678231,3028890=366520} 2024-03-04T06:23:19.476 INFO:tasks.rados.rados.0.smithi064.stdout:3594: writing smithi06431743-24 from 595410 to 1350822 tid 1 2024-03-04T06:23:19.484 INFO:tasks.rados.rados.0.smithi064.stdout:3594: writing smithi06431743-24 from 1800355 to 2478586 tid 2 2024-03-04T06:23:19.490 INFO:tasks.rados.rados.0.smithi064.stdout:3594: writing smithi06431743-24 from 3028890 to 3395410 tid 3 2024-03-04T06:23:19.490 INFO:tasks.rados.rados.0.smithi064.stdout:3595: delete oid 11 current snap is 466 2024-03-04T06:23:19.556 INFO:tasks.rados.rados.0.smithi064.stdout:3595: done (2 left) 2024-03-04T06:23:19.556 INFO:tasks.rados.rados.0.smithi064.stdout:3596: delete oid 46 current snap is 466 2024-03-04T06:23:19.563 INFO:tasks.rados.rados.0.smithi064.stdout:3593: finishing write tid 1 to smithi06431743-14 2024-03-04T06:23:19.589 INFO:tasks.rados.rados.0.smithi064.stdout:3593: finishing write tid 2 to smithi06431743-14 2024-03-04T06:23:19.597 INFO:tasks.rados.rados.0.smithi064.stdout:3596: done (2 left) 2024-03-04T06:23:19.597 INFO:tasks.rados.rados.0.smithi064.stdout:3597: read oid 5 snap -1 2024-03-04T06:23:19.597 INFO:tasks.rados.rados.0.smithi064.stdout:3597: expect (ObjNum 1619 snap 462 seq_num 1619) 2024-03-04T06:23:19.597 INFO:tasks.rados.rados.0.smithi064.stdout:3598: delete oid 27 current snap is 466 2024-03-04T06:23:19.612 INFO:tasks.rados.rados.0.smithi064.stdout:3593: finishing write tid 3 to smithi06431743-14 2024-03-04T06:23:19.623 INFO:tasks.rados.rados.0.smithi064.stdout:3593: finishing write tid 4 to smithi06431743-14 2024-03-04T06:23:19.626 INFO:tasks.rados.rados.0.smithi064.stdout:3593: finishing write tid 5 to smithi06431743-14 2024-03-04T06:23:19.626 INFO:tasks.rados.rados.0.smithi064.stdout:3593: oid 14 updating version 0 to 1399 2024-03-04T06:23:19.626 INFO:tasks.rados.rados.0.smithi064.stdout:3593: oid 14 updating version 1399 to 1400 2024-03-04T06:23:19.626 INFO:tasks.rados.rados.0.smithi064.stdout:3593: oid 14 version 1400 is already newer than 1398 2024-03-04T06:23:19.626 INFO:tasks.rados.rados.0.smithi064.stdout:3593: oid 14 version 1400 is already newer than 1397 2024-03-04T06:23:19.626 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 1400 (ObjNum 1629 snap 466 seq_num 1629) dirty exists 2024-03-04T06:23:19.626 INFO:tasks.rados.rados.0.smithi064.stdout:3593: left oid 14 (ObjNum 1629 snap 466 seq_num 1629) 2024-03-04T06:23:19.657 INFO:tasks.rados.rados.0.smithi064.stdout:3593: done (3 left) 2024-03-04T06:23:19.657 INFO:tasks.rados.rados.0.smithi064.stdout:3598: done (2 left) 2024-03-04T06:23:19.657 INFO:tasks.rados.rados.0.smithi064.stdout:3599: read oid 3 snap 442 2024-03-04T06:23:19.657 INFO:tasks.rados.rados.0.smithi064.stdout:3599: expect (ObjNum 1452 snap 406 seq_num 1452) 2024-03-04T06:23:19.657 INFO:tasks.rados.rados.0.smithi064.stdout:3600: read oid 20 snap 457 2024-03-04T06:23:19.657 INFO:tasks.rados.rados.0.smithi064.stdout:3600: expect deleted 2024-03-04T06:23:19.657 INFO:tasks.rados.rados.0.smithi064.stdout:3601: snap_remove snap 456 2024-03-04T06:23:20.462 INFO:tasks.rados.rados.0.smithi064.stdout:3601: done (4 left) 2024-03-04T06:23:20.462 INFO:tasks.rados.rados.0.smithi064.stdout:3602: delete oid 4 current snap is 466 2024-03-04T06:23:20.462 INFO:tasks.rados.rados.0.smithi064.stdout:3594: finishing write tid 1 to smithi06431743-24 2024-03-04T06:23:20.462 INFO:tasks.rados.rados.0.smithi064.stdout:3597: expect (ObjNum 1619 snap 462 seq_num 1619) 2024-03-04T06:23:20.668 INFO:tasks.rados.rados.0.smithi064.stdout:3594: finishing write tid 2 to smithi06431743-24 2024-03-04T06:23:20.669 INFO:tasks.rados.rados.0.smithi064.stdout:3594: finishing write tid 3 to smithi06431743-24 2024-03-04T06:23:20.669 INFO:tasks.rados.rados.0.smithi064.stdout:3599: expect (ObjNum 1452 snap 406 seq_num 1452) 2024-03-04T06:23:20.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:20.795 INFO:tasks.rados.rados.0.smithi064.stdout:3594: finishing write tid 4 to smithi06431743-24 2024-03-04T06:23:20.795 INFO:tasks.rados.rados.0.smithi064.stdout:3594: finishing write tid 5 to smithi06431743-24 2024-03-04T06:23:20.795 INFO:tasks.rados.rados.0.smithi064.stdout:3594: oid 24 updating version 0 to 1405 2024-03-04T06:23:20.795 INFO:tasks.rados.rados.0.smithi064.stdout:3594: oid 24 version 1405 is already newer than 1405 2024-03-04T06:23:20.795 INFO:tasks.rados.rados.0.smithi064.stdout:3594: oid 24 version 1405 is already newer than 1405 2024-03-04T06:23:20.796 INFO:tasks.rados.rados.0.smithi064.stdout:3594: oid 24 version 1405 is already newer than 1405 2024-03-04T06:23:20.796 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 1405 (ObjNum 1630 snap 466 seq_num 1630) dirty exists 2024-03-04T06:23:20.796 INFO:tasks.rados.rados.0.smithi064.stdout:3594: left oid 24 (ObjNum 1630 snap 466 seq_num 1630) 2024-03-04T06:23:20.796 INFO:tasks.rados.rados.0.smithi064.stdout:3594: done (4 left) 2024-03-04T06:23:20.796 INFO:tasks.rados.rados.0.smithi064.stdout:3597: done (3 left) 2024-03-04T06:23:20.796 INFO:tasks.rados.rados.0.smithi064.stdout:3599: done (2 left) 2024-03-04T06:23:20.796 INFO:tasks.rados.rados.0.smithi064.stdout:3600: done (1 left) 2024-03-04T06:23:20.796 INFO:tasks.rados.rados.0.smithi064.stdout:3602: done (0 left) 2024-03-04T06:23:20.796 INFO:tasks.rados.rados.0.smithi064.stdout:3603: read oid 11 snap 428 2024-03-04T06:23:20.796 INFO:tasks.rados.rados.0.smithi064.stdout:3603: expect (ObjNum 1514 snap 424 seq_num 1514) 2024-03-04T06:23:20.796 INFO:tasks.rados.rados.0.smithi064.stdout:3604: delete oid 30 current snap is 466 2024-03-04T06:23:20.800 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:20.798+0000 7f141895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:20.804 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:20.804+0000 7f141895d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:20.820 INFO:tasks.rados.rados.0.smithi064.stdout:3604: done (1 left) 2024-03-04T06:23:20.820 INFO:tasks.rados.rados.0.smithi064.stdout:3605: snap_remove snap 459 2024-03-04T06:23:21.465 INFO:tasks.rados.rados.0.smithi064.stdout:3605: done (1 left) 2024-03-04T06:23:21.466 INFO:tasks.rados.rados.0.smithi064.stdout:3606: snap_remove snap 465 2024-03-04T06:23:22.466 INFO:tasks.rados.rados.0.smithi064.stdout:3606: done (1 left) 2024-03-04T06:23:22.466 INFO:tasks.rados.rados.0.smithi064.stdout:3607: delete oid 16 current snap is 466 2024-03-04T06:23:22.466 INFO:tasks.rados.rados.0.smithi064.stdout:3603: expect (ObjNum 1514 snap 424 seq_num 1514) 2024-03-04T06:23:22.650 INFO:tasks.rados.rados.0.smithi064.stdout:3603: done (1 left) 2024-03-04T06:23:22.650 INFO:tasks.rados.rados.0.smithi064.stdout:3607: done (0 left) 2024-03-04T06:23:22.651 INFO:tasks.rados.rados.0.smithi064.stdout:3608: snap_remove snap 443 2024-03-04T06:23:22.763 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:23:22.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:23:22.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:22.867+0000 7f0c8ab9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:22.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:22.876+0000 7f0c8ab9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:22.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:23.002 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:23.000+0000 7f4c6d310640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:23.004 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:23.004+0000 7f4c6d310640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:23.472 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:23:23.478 INFO:tasks.rados.rados.0.smithi064.stdout:3608: done (0 left) 2024-03-04T06:23:23.478 INFO:tasks.rados.rados.0.smithi064.stdout:3609: write (excl) oid 19 current snap is 466 2024-03-04T06:23:23.478 INFO:tasks.rados.rados.0.smithi064.stdout:3609: seq_num 1637 ranges {713059=598508,2093376=432896,3104498=408561} 2024-03-04T06:23:23.485 INFO:tasks.rados.rados.0.smithi064.stdout:3609: writing smithi06431743-19 from 713059 to 1311567 tid 1 2024-03-04T06:23:23.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:23:23.491 INFO:tasks.rados.rados.0.smithi064.stdout:3609: writing smithi06431743-19 from 2093376 to 2526272 tid 2 2024-03-04T06:23:23.497 INFO:tasks.rados.rados.0.smithi064.stdout:3609: writing smithi06431743-19 from 3104498 to 3513059 tid 3 2024-03-04T06:23:23.497 INFO:tasks.rados.rados.0.smithi064.stdout:3610: snap_remove snap 461 2024-03-04T06:23:23.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:23.593+0000 7f81cba2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:23.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:23.602+0000 7f81cba2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:24.473 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:23:24.476 INFO:tasks.rados.rados.0.smithi064.stdout:3610: done (1 left) 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3611: read oid 47 snap -1 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3611: expect deleted 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3612: read oid 16 snap 460 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3612: expect deleted 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3613: snap_create 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3609: finishing write tid 1 to smithi06431743-19 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3609: finishing write tid 2 to smithi06431743-19 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3609: finishing write tid 3 to smithi06431743-19 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3609: finishing write tid 4 to smithi06431743-19 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3609: finishing write tid 5 to smithi06431743-19 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3609: oid 19 updating version 0 to 2096 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3609: oid 19 version 2096 is already newer than 2093 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3609: oid 19 version 2096 is already newer than 2096 2024-03-04T06:23:24.477 INFO:tasks.rados.rados.0.smithi064.stdout:3609: oid 19 version 2096 is already newer than 2094 2024-03-04T06:23:24.478 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 2096 (ObjNum 1637 snap 466 seq_num 1637) dirty exists 2024-03-04T06:23:24.478 INFO:tasks.rados.rados.0.smithi064.stdout:3609: left oid 19 (ObjNum 1637 snap 466 seq_num 1637) 2024-03-04T06:23:24.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:23:24.644 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:24.638+0000 7f272a55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:24.647 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:24.647+0000 7f272a55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:25.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:25.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:25.186+0000 7f8a07c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:25.190 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:25.190+0000 7f8a07c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:25.479 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:23:25.482 INFO:tasks.rados.rados.0.smithi064.stdout:3609: done (3 left) 2024-03-04T06:23:25.483 INFO:tasks.rados.rados.0.smithi064.stdout:3611: done (2 left) 2024-03-04T06:23:25.483 INFO:tasks.rados.rados.0.smithi064.stdout:3612: done (1 left) 2024-03-04T06:23:25.483 INFO:tasks.rados.rados.0.smithi064.stdout:3613: done (0 left) 2024-03-04T06:23:25.483 INFO:tasks.rados.rados.0.smithi064.stdout:3614: read oid 42 snap -1 2024-03-04T06:23:25.483 INFO:tasks.rados.rados.0.smithi064.stdout:3614: expect (ObjNum 1601 snap 453 seq_num 1601) 2024-03-04T06:23:25.483 INFO:tasks.rados.rados.0.smithi064.stdout:3615: write (excl) oid 26 current snap is 467 2024-03-04T06:23:25.483 INFO:tasks.rados.rados.0.smithi064.stdout:3615: seq_num 1638 ranges {729866=673757,2151858=692907} 2024-03-04T06:23:25.490 INFO:tasks.rados.rados.0.smithi064.stdout:3615: writing smithi06431743-26 from 729866 to 1403623 tid 1 2024-03-04T06:23:25.495 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:23:25.499 INFO:tasks.rados.rados.0.smithi064.stdout:3615: writing smithi06431743-26 from 2151858 to 2844765 tid 2 2024-03-04T06:23:25.499 INFO:tasks.rados.rados.0.smithi064.stdout:3616: read oid 21 snap 438 2024-03-04T06:23:25.499 INFO:tasks.rados.rados.0.smithi064.stdout:3616: expect deleted 2024-03-04T06:23:25.499 INFO:tasks.rados.rados.0.smithi064.stdout:3617: snap_remove snap 439 2024-03-04T06:23:25.623 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:25.617+0000 7fdff94d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:25.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:25.626+0000 7fdff94d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:26.482 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:23:26.485 INFO:tasks.rados.rados.0.smithi064.stdout:3617: done (3 left) 2024-03-04T06:23:26.485 INFO:tasks.rados.rados.0.smithi064.stdout:3618: write oid 48 current snap is 467 2024-03-04T06:23:26.485 INFO:tasks.rados.rados.0.smithi064.stdout:3618: seq_num 1639 ranges {746673=749006,2210340=436565,3159344=387329} 2024-03-04T06:23:26.495 INFO:tasks.rados.rados.0.smithi064.stdout:3618: writing smithi06431743-48 from 746673 to 1495679 tid 1 2024-03-04T06:23:26.500 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:23:26.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:26.501 INFO:tasks.rados.rados.0.smithi064.stdout:3618: writing smithi06431743-48 from 2210340 to 2646905 tid 2 2024-03-04T06:23:26.506 INFO:tasks.rados.rados.0.smithi064.stdout:3618: writing smithi06431743-48 from 3159344 to 3546673 tid 3 2024-03-04T06:23:26.506 INFO:tasks.rados.rados.0.smithi064.stdout:3619: read oid 16 snap -1 2024-03-04T06:23:26.507 INFO:tasks.rados.rados.0.smithi064.stdout:3619: expect deleted 2024-03-04T06:23:26.507 INFO:tasks.rados.rados.0.smithi064.stdout:3620: write (excl) oid 1 current snap is 467 2024-03-04T06:23:26.507 INFO:tasks.rados.rados.0.smithi064.stdout:3620: seq_num 1640 ranges {763480=424255,1868822=696576,3186767=376713} 2024-03-04T06:23:26.513 INFO:tasks.rados.rados.0.smithi064.stdout:3620: writing smithi06431743-1 from 763480 to 1187735 tid 1 2024-03-04T06:23:26.520 INFO:tasks.rados.rados.0.smithi064.stdout:3620: writing smithi06431743-1 from 1868822 to 2565398 tid 2 2024-03-04T06:23:26.526 INFO:tasks.rados.rados.0.smithi064.stdout:3620: writing smithi06431743-1 from 3186767 to 3563480 tid 3 2024-03-04T06:23:26.526 INFO:tasks.rados.rados.0.smithi064.stdout:3621: snap_create 2024-03-04T06:23:26.526 INFO:tasks.rados.rados.0.smithi064.stdout:3614: expect (ObjNum 1601 snap 453 seq_num 1601) 2024-03-04T06:23:26.642 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:26.637+0000 7f4d82b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:26.651 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:26.650+0000 7f4d82b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:26.744 INFO:tasks.rados.rados.0.smithi064.stdout:3615: finishing write tid 1 to smithi06431743-26 2024-03-04T06:23:26.744 INFO:tasks.rados.rados.0.smithi064.stdout:3615: finishing write tid 2 to smithi06431743-26 2024-03-04T06:23:26.744 INFO:tasks.rados.rados.0.smithi064.stdout:3615: finishing write tid 3 to smithi06431743-26 2024-03-04T06:23:26.744 INFO:tasks.rados.rados.0.smithi064.stdout:3615: finishing write tid 4 to smithi06431743-26 2024-03-04T06:23:26.744 INFO:tasks.rados.rados.0.smithi064.stdout:3615: oid 26 updating version 0 to 1790 2024-03-04T06:23:26.744 INFO:tasks.rados.rados.0.smithi064.stdout:3615: oid 26 updating version 1790 to 1792 2024-03-04T06:23:26.744 INFO:tasks.rados.rados.0.smithi064.stdout:3615: oid 26 version 1792 is already newer than 1792 2024-03-04T06:23:26.744 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 26 v 1792 (ObjNum 1638 snap 467 seq_num 1638) dirty exists 2024-03-04T06:23:26.744 INFO:tasks.rados.rados.0.smithi064.stdout:3615: left oid 26 (ObjNum 1638 snap 467 seq_num 1638) 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3620: finishing write tid 1 to smithi06431743-1 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3618: finishing write tid 1 to smithi06431743-48 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3620: finishing write tid 2 to smithi06431743-1 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3618: finishing write tid 2 to smithi06431743-48 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3620: finishing write tid 3 to smithi06431743-1 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3618: finishing write tid 3 to smithi06431743-48 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3620: finishing write tid 4 to smithi06431743-1 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3620: finishing write tid 5 to smithi06431743-1 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3620: oid 1 updating version 0 to 1802 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3620: oid 1 updating version 1802 to 1804 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3620: oid 1 version 1804 is already newer than 1804 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3620: oid 1 updating version 1804 to 1805 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 1805 (ObjNum 1640 snap 467 seq_num 1640) dirty exists 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3620: left oid 1 (ObjNum 1640 snap 467 seq_num 1640) 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3618: finishing write tid 4 to smithi06431743-48 2024-03-04T06:23:26.745 INFO:tasks.rados.rados.0.smithi064.stdout:3618: finishing write tid 5 to smithi06431743-48 2024-03-04T06:23:26.746 INFO:tasks.rados.rados.0.smithi064.stdout:3618: oid 48 updating version 0 to 643 2024-03-04T06:23:26.746 INFO:tasks.rados.rados.0.smithi064.stdout:3618: oid 48 version 643 is already newer than 643 2024-03-04T06:23:26.746 INFO:tasks.rados.rados.0.smithi064.stdout:3618: oid 48 updating version 643 to 644 2024-03-04T06:23:26.746 INFO:tasks.rados.rados.0.smithi064.stdout:3618: oid 48 version 644 is already newer than 641 2024-03-04T06:23:26.746 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 644 (ObjNum 1639 snap 467 seq_num 1639) dirty exists 2024-03-04T06:23:26.746 INFO:tasks.rados.rados.0.smithi064.stdout:3618: left oid 48 (ObjNum 1639 snap 467 seq_num 1639) 2024-03-04T06:23:26.870 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:26.870 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:26.882 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1396,"stamp":"2024-03-04T06:23:26.483377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99041375,"num_objects":1206,"num_object_clones":992,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":180,"num_read":6547,"num_read_kb":5118708,"num_write":5576,"num_write_kb":1486430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16045,"ondisk_log_size":16045,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":706708,"kb_used_data":467332,"kb_used_omap":0,"kb_used_meta":239104,"kb_avail":376780652,"statfs":{"total":386547056640,"available":385823387648,"internally_reserved":0,"allocated":478547968,"data_stored":557929502,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":244842496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4219349,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":4,"num_read_kb":2,"num_write":11,"num_write_kb":2742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019120"},"pg_stats":[{"pgid":"3.f","version":"930'881","reported_seq":3175,"reported_epoch":933,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:24.523034+0000","last_change":"2024-03-04T06:23:24.523034+0000","last_active":"2024-03-04T06:23:24.523034+0000","last_peered":"2024-03-04T06:23:24.523034+0000","last_clean":"2024-03-04T06:23:24.523034+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:24.523034+0000","last_undegraded":"2024-03-04T06:23:24.523034+0000","last_fullsized":"2024-03-04T06:23:24.523034+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6202575,"num_objects":76,"num_object_clones":61,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":350,"num_read_kb":273132,"num_write":322,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3175,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.500340+0000","last_change":"2024-03-04T06:23:24.507535+0000","last_active":"2024-03-04T06:23:24.500340+0000","last_peered":"2024-03-04T06:23:24.500340+0000","last_clean":"2024-03-04T06:23:24.500340+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:24.500340+0000","last_undegraded":"2024-03-04T06:23:24.500340+0000","last_fullsized":"2024-03-04T06:23:24.500340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"930'1215","reported_seq":3376,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.528947+0000","last_change":"2024-03-04T06:23:25.528947+0000","last_active":"2024-03-04T06:23:25.528947+0000","last_peered":"2024-03-04T06:23:25.528947+0000","last_clean":"2024-03-04T06:23:25.528947+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:25.528947+0000","last_undegraded":"2024-03-04T06:23:25.528947+0000","last_fullsized":"2024-03-04T06:23:25.528947+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1215,"log_dups_size":0,"ondisk_log_size":1215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6101733,"num_objects":95,"num_object_clones":80,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":561,"num_read_kb":451740,"num_write":427,"num_write_kb":116189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"930'982","reported_seq":3152,"reported_epoch":933,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:24.514647+0000","last_change":"2024-03-04T06:23:24.514647+0000","last_active":"2024-03-04T06:23:24.514647+0000","last_peered":"2024-03-04T06:23:24.514647+0000","last_clean":"2024-03-04T06:23:24.514647+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:24.514647+0000","last_undegraded":"2024-03-04T06:23:24.514647+0000","last_fullsized":"2024-03-04T06:23:24.514647+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":982,"log_dups_size":0,"ondisk_log_size":982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849595,"num_objects":65,"num_object_clones":55,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":321,"num_read_kb":245412,"num_write":309,"num_write_kb":85718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"932'399","reported_seq":2959,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.541726+0000","last_change":"2024-03-04T06:23:25.544800+0000","last_active":"2024-03-04T06:23:25.541726+0000","last_peered":"2024-03-04T06:23:25.541726+0000","last_clean":"2024-03-04T06:23:25.541726+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:25.541726+0000","last_undegraded":"2024-03-04T06:23:25.541726+0000","last_fullsized":"2024-03-04T06:23:25.541726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":120,"num_read_kb":93147,"num_write":120,"num_write_kb":33092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"930'980","reported_seq":3235,"reported_epoch":933,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:24.524489+0000","last_change":"2024-03-04T06:23:24.524489+0000","last_active":"2024-03-04T06:23:24.524489+0000","last_peered":"2024-03-04T06:23:24.524489+0000","last_clean":"2024-03-04T06:23:24.524489+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:24.524489+0000","last_undegraded":"2024-03-04T06:23:24.524489+0000","last_fullsized":"2024-03-04T06:23:24.524489+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":980,"log_dups_size":0,"ondisk_log_size":980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9228231,"num_objects":72,"num_object_clones":60,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":426,"num_read_kb":338098,"num_write":348,"num_write_kb":100575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2733,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.548244+0000","last_change":"2024-03-04T06:23:24.550420+0000","last_active":"2024-03-04T06:23:24.548244+0000","last_peered":"2024-03-04T06:23:24.548244+0000","last_clean":"2024-03-04T06:23:24.548244+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:24.548244+0000","last_undegraded":"2024-03-04T06:23:24.548244+0000","last_fullsized":"2024-03-04T06:23:24.548244+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"928'630","reported_seq":3040,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.520607+0000","last_change":"2024-03-04T06:23:25.520607+0000","last_active":"2024-03-04T06:23:25.520607+0000","last_peered":"2024-03-04T06:23:25.520607+0000","last_clean":"2024-03-04T06:23:25.520607+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:25.520607+0000","last_undegraded":"2024-03-04T06:23:25.520607+0000","last_fullsized":"2024-03-04T06:23:25.520607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2495,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.512664+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:25.512664+0000","last_peered":"2024-03-04T06:23:25.512664+0000","last_clean":"2024-03-04T06:23:25.512664+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:25.512664+0000","last_undegraded":"2024-03-04T06:23:25.512664+0000","last_fullsized":"2024-03-04T06:23:25.512664+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2308,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.550565+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:25.550565+0000","last_peered":"2024-03-04T06:23:25.550565+0000","last_clean":"2024-03-04T06:23:25.550565+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:25.550565+0000","last_undegraded":"2024-03-04T06:23:25.550565+0000","last_fullsized":"2024-03-04T06:23:25.550565+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"906'951","reported_seq":3230,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.553725+0000","last_change":"2024-03-04T06:23:25.556015+0000","last_active":"2024-03-04T06:23:25.553725+0000","last_peered":"2024-03-04T06:23:25.553725+0000","last_clean":"2024-03-04T06:23:25.553725+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:25.553725+0000","last_undegraded":"2024-03-04T06:23:25.553725+0000","last_fullsized":"2024-03-04T06:23:25.553725+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":429,"num_read_kb":348209,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2295,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.522882+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:24.522882+0000","last_peered":"2024-03-04T06:23:24.522882+0000","last_clean":"2024-03-04T06:23:24.522882+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:24.522882+0000","last_undegraded":"2024-03-04T06:23:24.522882+0000","last_fullsized":"2024-03-04T06:23:24.522882+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"932'637","reported_seq":3132,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.535170+0000","last_change":"2024-03-04T06:23:25.550042+0000","last_active":"2024-03-04T06:23:25.535170+0000","last_peered":"2024-03-04T06:23:25.535170+0000","last_clean":"2024-03-04T06:23:25.535170+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:25.535170+0000","last_undegraded":"2024-03-04T06:23:25.535170+0000","last_fullsized":"2024-03-04T06:23:25.535170+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5379032,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":309,"num_read_kb":250022,"num_write":214,"num_write_kb":60883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2308,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.526299+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:25.526299+0000","last_peered":"2024-03-04T06:23:25.526299+0000","last_clean":"2024-03-04T06:23:25.526299+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:25.526299+0000","last_undegraded":"2024-03-04T06:23:25.526299+0000","last_fullsized":"2024-03-04T06:23:25.526299+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"934'1792","reported_seq":3742,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.632086+0000","last_change":"2024-03-04T06:23:21.496105+0000","last_active":"2024-03-04T06:23:25.632086+0000","last_peered":"2024-03-04T06:23:25.632086+0000","last_clean":"2024-03-04T06:23:25.632086+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:25.632086+0000","last_undegraded":"2024-03-04T06:23:25.632086+0000","last_fullsized":"2024-03-04T06:23:25.632086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1792,"log_dups_size":0,"ondisk_log_size":1792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12926134,"num_objects":142,"num_object_clones":116,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":787,"num_read_kb":618697,"num_write":659,"num_write_kb":177018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2295,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.505308+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:24.505308+0000","last_peered":"2024-03-04T06:23:24.505308+0000","last_clean":"2024-03-04T06:23:24.505308+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:24.505308+0000","last_undegraded":"2024-03-04T06:23:24.505308+0000","last_fullsized":"2024-03-04T06:23:24.505308+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"933'2112","reported_seq":3791,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.494129+0000","last_change":"2024-03-04T06:23:24.593330+0000","last_active":"2024-03-04T06:23:24.494129+0000","last_peered":"2024-03-04T06:23:24.494129+0000","last_clean":"2024-03-04T06:23:24.494129+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:24.494129+0000","last_undegraded":"2024-03-04T06:23:24.494129+0000","last_fullsized":"2024-03-04T06:23:24.494129+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2112,"log_dups_size":0,"ondisk_log_size":2112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976588,"num_objects":157,"num_object_clones":133,"num_object_copies":471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":723,"num_read_kb":536054,"num_write":717,"num_write_kb":193714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"928'910","reported_seq":3231,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.556061+0000","last_change":"2024-03-04T06:23:25.558586+0000","last_active":"2024-03-04T06:23:25.556061+0000","last_peered":"2024-03-04T06:23:25.556061+0000","last_clean":"2024-03-04T06:23:25.556061+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:25.556061+0000","last_undegraded":"2024-03-04T06:23:25.556061+0000","last_fullsized":"2024-03-04T06:23:25.556061+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269770,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":438,"num_read_kb":364480,"num_write":317,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2397,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.513064+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:24.513064+0000","last_peered":"2024-03-04T06:23:24.513064+0000","last_clean":"2024-03-04T06:23:24.513064+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:24.513064+0000","last_undegraded":"2024-03-04T06:23:24.513064+0000","last_fullsized":"2024-03-04T06:23:24.513064+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"932'1055","reported_seq":3311,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.505756+0000","last_change":"2024-03-04T06:23:24.517335+0000","last_active":"2024-03-04T06:23:24.505756+0000","last_peered":"2024-03-04T06:23:24.505756+0000","last_clean":"2024-03-04T06:23:24.505756+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:24.505756+0000","last_undegraded":"2024-03-04T06:23:24.505756+0000","last_fullsized":"2024-03-04T06:23:24.505756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1055,"log_dups_size":0,"ondisk_log_size":1055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":380,"num_read_kb":277363,"num_write":347,"num_write_kb":92772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2362,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.542934+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:25.542934+0000","last_peered":"2024-03-04T06:23:25.542934+0000","last_clean":"2024-03-04T06:23:25.542934+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:25.542934+0000","last_undegraded":"2024-03-04T06:23:25.542934+0000","last_fullsized":"2024-03-04T06:23:25.542934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"932'1411","reported_seq":3430,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.514615+0000","last_change":"2024-03-04T06:23:24.484017+0000","last_active":"2024-03-04T06:23:25.514615+0000","last_peered":"2024-03-04T06:23:25.514615+0000","last_clean":"2024-03-04T06:23:25.514615+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:25.514615+0000","last_undegraded":"2024-03-04T06:23:25.514615+0000","last_fullsized":"2024-03-04T06:23:25.514615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1411,"log_dups_size":0,"ondisk_log_size":1411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6774013,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":496,"num_read_kb":386812,"num_write":462,"num_write_kb":128837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2397,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.491927+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:24.491927+0000","last_peered":"2024-03-04T06:23:24.491927+0000","last_clean":"2024-03-04T06:23:24.491927+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:24.491927+0000","last_undegraded":"2024-03-04T06:23:24.491927+0000","last_fullsized":"2024-03-04T06:23:24.491927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"933'1204","reported_seq":3486,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.502670+0000","last_change":"2024-03-04T06:23:24.537983+0000","last_active":"2024-03-04T06:23:24.502670+0000","last_peered":"2024-03-04T06:23:24.502670+0000","last_clean":"2024-03-04T06:23:24.502670+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:24.502670+0000","last_undegraded":"2024-03-04T06:23:24.502670+0000","last_fullsized":"2024-03-04T06:23:24.502670+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8808056,"num_objects":97,"num_object_clones":83,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":575,"num_read_kb":484996,"num_write":435,"num_write_kb":119544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2308,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.537068+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:25.537068+0000","last_peered":"2024-03-04T06:23:25.537068+0000","last_clean":"2024-03-04T06:23:25.537068+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:25.537068+0000","last_undegraded":"2024-03-04T06:23:25.537068+0000","last_fullsized":"2024-03-04T06:23:25.537068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98451007,"num_objects":1204,"num_object_clones":992,"num_object_copies":3612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":180,"num_read":6444,"num_read_kb":5118617,"num_write":5439,"num_write_kb":1484536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15948,"ondisk_log_size":15948,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":213136,"kb_used_data":146056,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94158704,"statfs":{"total":96636764160,"available":96418512896,"internally_reserved":0,"allocated":149561344,"data_stored":174045322,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":167316,"kb_used_data":109884,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94204524,"statfs":{"total":96636764160,"available":96465432576,"internally_reserved":0,"allocated":112521216,"data_stored":128576063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706603,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166244,"kb_used_data":108812,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94205596,"statfs":{"total":96636764160,"available":96466530304,"internally_reserved":0,"allocated":111423488,"data_stored":131597752,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706603,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":160012,"kb_used_data":102580,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94211828,"statfs":{"total":96636764160,"available":96472911872,"internally_reserved":0,"allocated":105041920,"data_stored":123710365,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:26.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:27.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:27.026+0000 7f03014dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:27.032 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:27.032+0000 7f03014dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:27.226 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:27.227 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:27.238 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1397,"stamp":"2024-03-04T06:23:26.904571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99041375,"num_objects":1206,"num_object_clones":992,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":180,"num_read":6547,"num_read_kb":5118708,"num_write":5576,"num_write_kb":1486430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16045,"ondisk_log_size":16045,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":706708,"kb_used_data":467332,"kb_used_omap":0,"kb_used_meta":239104,"kb_avail":376780652,"statfs":{"total":386547056640,"available":385823387648,"internally_reserved":0,"allocated":478547968,"data_stored":557929502,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":244842496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4219349,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":4,"num_read_kb":2,"num_write":11,"num_write_kb":2742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000871"},"pg_stats":[{"pgid":"3.f","version":"930'881","reported_seq":3175,"reported_epoch":933,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:24.523034+0000","last_change":"2024-03-04T06:23:24.523034+0000","last_active":"2024-03-04T06:23:24.523034+0000","last_peered":"2024-03-04T06:23:24.523034+0000","last_clean":"2024-03-04T06:23:24.523034+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:24.523034+0000","last_undegraded":"2024-03-04T06:23:24.523034+0000","last_fullsized":"2024-03-04T06:23:24.523034+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6202575,"num_objects":76,"num_object_clones":61,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":350,"num_read_kb":273132,"num_write":322,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3175,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.500340+0000","last_change":"2024-03-04T06:23:24.507535+0000","last_active":"2024-03-04T06:23:24.500340+0000","last_peered":"2024-03-04T06:23:24.500340+0000","last_clean":"2024-03-04T06:23:24.500340+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:24.500340+0000","last_undegraded":"2024-03-04T06:23:24.500340+0000","last_fullsized":"2024-03-04T06:23:24.500340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"930'1215","reported_seq":3376,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.528947+0000","last_change":"2024-03-04T06:23:25.528947+0000","last_active":"2024-03-04T06:23:25.528947+0000","last_peered":"2024-03-04T06:23:25.528947+0000","last_clean":"2024-03-04T06:23:25.528947+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:25.528947+0000","last_undegraded":"2024-03-04T06:23:25.528947+0000","last_fullsized":"2024-03-04T06:23:25.528947+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1215,"log_dups_size":0,"ondisk_log_size":1215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6101733,"num_objects":95,"num_object_clones":80,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":561,"num_read_kb":451740,"num_write":427,"num_write_kb":116189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"930'982","reported_seq":3152,"reported_epoch":933,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:24.514647+0000","last_change":"2024-03-04T06:23:24.514647+0000","last_active":"2024-03-04T06:23:24.514647+0000","last_peered":"2024-03-04T06:23:24.514647+0000","last_clean":"2024-03-04T06:23:24.514647+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:24.514647+0000","last_undegraded":"2024-03-04T06:23:24.514647+0000","last_fullsized":"2024-03-04T06:23:24.514647+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":982,"log_dups_size":0,"ondisk_log_size":982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849595,"num_objects":65,"num_object_clones":55,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":321,"num_read_kb":245412,"num_write":309,"num_write_kb":85718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"932'399","reported_seq":2959,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.541726+0000","last_change":"2024-03-04T06:23:25.544800+0000","last_active":"2024-03-04T06:23:25.541726+0000","last_peered":"2024-03-04T06:23:25.541726+0000","last_clean":"2024-03-04T06:23:25.541726+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:25.541726+0000","last_undegraded":"2024-03-04T06:23:25.541726+0000","last_fullsized":"2024-03-04T06:23:25.541726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":120,"num_read_kb":93147,"num_write":120,"num_write_kb":33092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"930'980","reported_seq":3235,"reported_epoch":933,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:24.524489+0000","last_change":"2024-03-04T06:23:24.524489+0000","last_active":"2024-03-04T06:23:24.524489+0000","last_peered":"2024-03-04T06:23:24.524489+0000","last_clean":"2024-03-04T06:23:24.524489+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:24.524489+0000","last_undegraded":"2024-03-04T06:23:24.524489+0000","last_fullsized":"2024-03-04T06:23:24.524489+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":980,"log_dups_size":0,"ondisk_log_size":980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9228231,"num_objects":72,"num_object_clones":60,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":426,"num_read_kb":338098,"num_write":348,"num_write_kb":100575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2733,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.548244+0000","last_change":"2024-03-04T06:23:24.550420+0000","last_active":"2024-03-04T06:23:24.548244+0000","last_peered":"2024-03-04T06:23:24.548244+0000","last_clean":"2024-03-04T06:23:24.548244+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:24.548244+0000","last_undegraded":"2024-03-04T06:23:24.548244+0000","last_fullsized":"2024-03-04T06:23:24.548244+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"928'630","reported_seq":3040,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.520607+0000","last_change":"2024-03-04T06:23:25.520607+0000","last_active":"2024-03-04T06:23:25.520607+0000","last_peered":"2024-03-04T06:23:25.520607+0000","last_clean":"2024-03-04T06:23:25.520607+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:25.520607+0000","last_undegraded":"2024-03-04T06:23:25.520607+0000","last_fullsized":"2024-03-04T06:23:25.520607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2495,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.512664+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:25.512664+0000","last_peered":"2024-03-04T06:23:25.512664+0000","last_clean":"2024-03-04T06:23:25.512664+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:25.512664+0000","last_undegraded":"2024-03-04T06:23:25.512664+0000","last_fullsized":"2024-03-04T06:23:25.512664+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2308,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.550565+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:25.550565+0000","last_peered":"2024-03-04T06:23:25.550565+0000","last_clean":"2024-03-04T06:23:25.550565+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:25.550565+0000","last_undegraded":"2024-03-04T06:23:25.550565+0000","last_fullsized":"2024-03-04T06:23:25.550565+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"906'951","reported_seq":3230,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.553725+0000","last_change":"2024-03-04T06:23:25.556015+0000","last_active":"2024-03-04T06:23:25.553725+0000","last_peered":"2024-03-04T06:23:25.553725+0000","last_clean":"2024-03-04T06:23:25.553725+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:25.553725+0000","last_undegraded":"2024-03-04T06:23:25.553725+0000","last_fullsized":"2024-03-04T06:23:25.553725+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":429,"num_read_kb":348209,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2295,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.522882+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:24.522882+0000","last_peered":"2024-03-04T06:23:24.522882+0000","last_clean":"2024-03-04T06:23:24.522882+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:24.522882+0000","last_undegraded":"2024-03-04T06:23:24.522882+0000","last_fullsized":"2024-03-04T06:23:24.522882+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"932'637","reported_seq":3132,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.535170+0000","last_change":"2024-03-04T06:23:25.550042+0000","last_active":"2024-03-04T06:23:25.535170+0000","last_peered":"2024-03-04T06:23:25.535170+0000","last_clean":"2024-03-04T06:23:25.535170+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:25.535170+0000","last_undegraded":"2024-03-04T06:23:25.535170+0000","last_fullsized":"2024-03-04T06:23:25.535170+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5379032,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":309,"num_read_kb":250022,"num_write":214,"num_write_kb":60883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2308,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.526299+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:25.526299+0000","last_peered":"2024-03-04T06:23:25.526299+0000","last_clean":"2024-03-04T06:23:25.526299+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:25.526299+0000","last_undegraded":"2024-03-04T06:23:25.526299+0000","last_fullsized":"2024-03-04T06:23:25.526299+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"934'1792","reported_seq":3742,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.632086+0000","last_change":"2024-03-04T06:23:21.496105+0000","last_active":"2024-03-04T06:23:25.632086+0000","last_peered":"2024-03-04T06:23:25.632086+0000","last_clean":"2024-03-04T06:23:25.632086+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:25.632086+0000","last_undegraded":"2024-03-04T06:23:25.632086+0000","last_fullsized":"2024-03-04T06:23:25.632086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1792,"log_dups_size":0,"ondisk_log_size":1792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12926134,"num_objects":142,"num_object_clones":116,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":787,"num_read_kb":618697,"num_write":659,"num_write_kb":177018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2295,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.505308+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:24.505308+0000","last_peered":"2024-03-04T06:23:24.505308+0000","last_clean":"2024-03-04T06:23:24.505308+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:24.505308+0000","last_undegraded":"2024-03-04T06:23:24.505308+0000","last_fullsized":"2024-03-04T06:23:24.505308+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"933'2112","reported_seq":3791,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.494129+0000","last_change":"2024-03-04T06:23:24.593330+0000","last_active":"2024-03-04T06:23:24.494129+0000","last_peered":"2024-03-04T06:23:24.494129+0000","last_clean":"2024-03-04T06:23:24.494129+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:24.494129+0000","last_undegraded":"2024-03-04T06:23:24.494129+0000","last_fullsized":"2024-03-04T06:23:24.494129+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2112,"log_dups_size":0,"ondisk_log_size":2112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976588,"num_objects":157,"num_object_clones":133,"num_object_copies":471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":723,"num_read_kb":536054,"num_write":717,"num_write_kb":193714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"928'910","reported_seq":3231,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.556061+0000","last_change":"2024-03-04T06:23:25.558586+0000","last_active":"2024-03-04T06:23:25.556061+0000","last_peered":"2024-03-04T06:23:25.556061+0000","last_clean":"2024-03-04T06:23:25.556061+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:25.556061+0000","last_undegraded":"2024-03-04T06:23:25.556061+0000","last_fullsized":"2024-03-04T06:23:25.556061+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269770,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":438,"num_read_kb":364480,"num_write":317,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2397,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.513064+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:24.513064+0000","last_peered":"2024-03-04T06:23:24.513064+0000","last_clean":"2024-03-04T06:23:24.513064+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:24.513064+0000","last_undegraded":"2024-03-04T06:23:24.513064+0000","last_fullsized":"2024-03-04T06:23:24.513064+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"932'1055","reported_seq":3311,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.505756+0000","last_change":"2024-03-04T06:23:24.517335+0000","last_active":"2024-03-04T06:23:24.505756+0000","last_peered":"2024-03-04T06:23:24.505756+0000","last_clean":"2024-03-04T06:23:24.505756+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:24.505756+0000","last_undegraded":"2024-03-04T06:23:24.505756+0000","last_fullsized":"2024-03-04T06:23:24.505756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1055,"log_dups_size":0,"ondisk_log_size":1055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":380,"num_read_kb":277363,"num_write":347,"num_write_kb":92772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2362,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.542934+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:25.542934+0000","last_peered":"2024-03-04T06:23:25.542934+0000","last_clean":"2024-03-04T06:23:25.542934+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:25.542934+0000","last_undegraded":"2024-03-04T06:23:25.542934+0000","last_fullsized":"2024-03-04T06:23:25.542934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"932'1411","reported_seq":3430,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.514615+0000","last_change":"2024-03-04T06:23:24.484017+0000","last_active":"2024-03-04T06:23:25.514615+0000","last_peered":"2024-03-04T06:23:25.514615+0000","last_clean":"2024-03-04T06:23:25.514615+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:25.514615+0000","last_undegraded":"2024-03-04T06:23:25.514615+0000","last_fullsized":"2024-03-04T06:23:25.514615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1411,"log_dups_size":0,"ondisk_log_size":1411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6774013,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":496,"num_read_kb":386812,"num_write":462,"num_write_kb":128837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2397,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.491927+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:24.491927+0000","last_peered":"2024-03-04T06:23:24.491927+0000","last_clean":"2024-03-04T06:23:24.491927+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:24.491927+0000","last_undegraded":"2024-03-04T06:23:24.491927+0000","last_fullsized":"2024-03-04T06:23:24.491927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"933'1204","reported_seq":3486,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.502670+0000","last_change":"2024-03-04T06:23:24.537983+0000","last_active":"2024-03-04T06:23:24.502670+0000","last_peered":"2024-03-04T06:23:24.502670+0000","last_clean":"2024-03-04T06:23:24.502670+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:24.502670+0000","last_undegraded":"2024-03-04T06:23:24.502670+0000","last_fullsized":"2024-03-04T06:23:24.502670+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8808056,"num_objects":97,"num_object_clones":83,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":575,"num_read_kb":484996,"num_write":435,"num_write_kb":119544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2308,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.537068+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:25.537068+0000","last_peered":"2024-03-04T06:23:25.537068+0000","last_clean":"2024-03-04T06:23:25.537068+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:25.537068+0000","last_undegraded":"2024-03-04T06:23:25.537068+0000","last_fullsized":"2024-03-04T06:23:25.537068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98451007,"num_objects":1204,"num_object_clones":992,"num_object_copies":3612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":180,"num_read":6444,"num_read_kb":5118617,"num_write":5439,"num_write_kb":1484536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15948,"ondisk_log_size":15948,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":213136,"kb_used_data":146056,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94158704,"statfs":{"total":96636764160,"available":96418512896,"internally_reserved":0,"allocated":149561344,"data_stored":174045322,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":167316,"kb_used_data":109884,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94204524,"statfs":{"total":96636764160,"available":96465432576,"internally_reserved":0,"allocated":112521216,"data_stored":128576063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706603,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166244,"kb_used_data":108812,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94205596,"statfs":{"total":96636764160,"available":96466530304,"internally_reserved":0,"allocated":111423488,"data_stored":131597752,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706603,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":160012,"kb_used_data":102580,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94211828,"statfs":{"total":96636764160,"available":96472911872,"internally_reserved":0,"allocated":105041920,"data_stored":123710365,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:27.239 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:23:27.240 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:23:27.240 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:27.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:27.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:27.344+0000 7fe05bc24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:27.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:27.351+0000 7fe05bc24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:27.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:27.387+0000 7f084446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:27.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:27.391+0000 7f084446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:27.488 INFO:tasks.rados.rados.0.smithi064.stdout:3614: done (6 left) 2024-03-04T06:23:27.488 INFO:tasks.rados.rados.0.smithi064.stdout:3615: done (5 left) 2024-03-04T06:23:27.488 INFO:tasks.rados.rados.0.smithi064.stdout:3616: done (4 left) 2024-03-04T06:23:27.488 INFO:tasks.rados.rados.0.smithi064.stdout:3618: done (3 left) 2024-03-04T06:23:27.488 INFO:tasks.rados.rados.0.smithi064.stdout:3619: done (2 left) 2024-03-04T06:23:27.488 INFO:tasks.rados.rados.0.smithi064.stdout:3620: done (1 left) 2024-03-04T06:23:27.488 INFO:tasks.rados.rados.0.smithi064.stdout:3621: done (0 left) 2024-03-04T06:23:27.488 INFO:tasks.rados.rados.0.smithi064.stdout:3622: snap_create 2024-03-04T06:23:27.552 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:27.553 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:27.564 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1398,"stamp":"2024-03-04T06:23:27.487170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99041375,"num_objects":1206,"num_object_clones":992,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":180,"num_read":6547,"num_read_kb":5118708,"num_write":5576,"num_write_kb":1486430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16045,"ondisk_log_size":16045,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":706708,"kb_used_data":467332,"kb_used_omap":0,"kb_used_meta":239104,"kb_avail":376780652,"statfs":{"total":386547056640,"available":385823387648,"internally_reserved":0,"allocated":478547968,"data_stored":557929502,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":244842496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4219349,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":4,"num_read_kb":2,"num_write":11,"num_write_kb":2742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011258"},"pg_stats":[{"pgid":"3.f","version":"930'881","reported_seq":3175,"reported_epoch":933,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:24.523034+0000","last_change":"2024-03-04T06:23:24.523034+0000","last_active":"2024-03-04T06:23:24.523034+0000","last_peered":"2024-03-04T06:23:24.523034+0000","last_clean":"2024-03-04T06:23:24.523034+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:24.523034+0000","last_undegraded":"2024-03-04T06:23:24.523034+0000","last_fullsized":"2024-03-04T06:23:24.523034+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6202575,"num_objects":76,"num_object_clones":61,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":350,"num_read_kb":273132,"num_write":322,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3175,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.500340+0000","last_change":"2024-03-04T06:23:24.507535+0000","last_active":"2024-03-04T06:23:24.500340+0000","last_peered":"2024-03-04T06:23:24.500340+0000","last_clean":"2024-03-04T06:23:24.500340+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:24.500340+0000","last_undegraded":"2024-03-04T06:23:24.500340+0000","last_fullsized":"2024-03-04T06:23:24.500340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"930'1215","reported_seq":3376,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.528947+0000","last_change":"2024-03-04T06:23:25.528947+0000","last_active":"2024-03-04T06:23:25.528947+0000","last_peered":"2024-03-04T06:23:25.528947+0000","last_clean":"2024-03-04T06:23:25.528947+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:25.528947+0000","last_undegraded":"2024-03-04T06:23:25.528947+0000","last_fullsized":"2024-03-04T06:23:25.528947+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1215,"log_dups_size":0,"ondisk_log_size":1215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6101733,"num_objects":95,"num_object_clones":80,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":561,"num_read_kb":451740,"num_write":427,"num_write_kb":116189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"930'982","reported_seq":3152,"reported_epoch":933,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:24.514647+0000","last_change":"2024-03-04T06:23:24.514647+0000","last_active":"2024-03-04T06:23:24.514647+0000","last_peered":"2024-03-04T06:23:24.514647+0000","last_clean":"2024-03-04T06:23:24.514647+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:24.514647+0000","last_undegraded":"2024-03-04T06:23:24.514647+0000","last_fullsized":"2024-03-04T06:23:24.514647+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":982,"log_dups_size":0,"ondisk_log_size":982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849595,"num_objects":65,"num_object_clones":55,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":321,"num_read_kb":245412,"num_write":309,"num_write_kb":85718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"932'399","reported_seq":2959,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.541726+0000","last_change":"2024-03-04T06:23:25.544800+0000","last_active":"2024-03-04T06:23:25.541726+0000","last_peered":"2024-03-04T06:23:25.541726+0000","last_clean":"2024-03-04T06:23:25.541726+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:25.541726+0000","last_undegraded":"2024-03-04T06:23:25.541726+0000","last_fullsized":"2024-03-04T06:23:25.541726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":120,"num_read_kb":93147,"num_write":120,"num_write_kb":33092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"930'980","reported_seq":3235,"reported_epoch":933,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:24.524489+0000","last_change":"2024-03-04T06:23:24.524489+0000","last_active":"2024-03-04T06:23:24.524489+0000","last_peered":"2024-03-04T06:23:24.524489+0000","last_clean":"2024-03-04T06:23:24.524489+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:24.524489+0000","last_undegraded":"2024-03-04T06:23:24.524489+0000","last_fullsized":"2024-03-04T06:23:24.524489+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":980,"log_dups_size":0,"ondisk_log_size":980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9228231,"num_objects":72,"num_object_clones":60,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":426,"num_read_kb":338098,"num_write":348,"num_write_kb":100575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2733,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.548244+0000","last_change":"2024-03-04T06:23:24.550420+0000","last_active":"2024-03-04T06:23:24.548244+0000","last_peered":"2024-03-04T06:23:24.548244+0000","last_clean":"2024-03-04T06:23:24.548244+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:24.548244+0000","last_undegraded":"2024-03-04T06:23:24.548244+0000","last_fullsized":"2024-03-04T06:23:24.548244+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"928'630","reported_seq":3040,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.520607+0000","last_change":"2024-03-04T06:23:25.520607+0000","last_active":"2024-03-04T06:23:25.520607+0000","last_peered":"2024-03-04T06:23:25.520607+0000","last_clean":"2024-03-04T06:23:25.520607+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:25.520607+0000","last_undegraded":"2024-03-04T06:23:25.520607+0000","last_fullsized":"2024-03-04T06:23:25.520607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2495,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.512664+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:25.512664+0000","last_peered":"2024-03-04T06:23:25.512664+0000","last_clean":"2024-03-04T06:23:25.512664+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:25.512664+0000","last_undegraded":"2024-03-04T06:23:25.512664+0000","last_fullsized":"2024-03-04T06:23:25.512664+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2308,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.550565+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:25.550565+0000","last_peered":"2024-03-04T06:23:25.550565+0000","last_clean":"2024-03-04T06:23:25.550565+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:25.550565+0000","last_undegraded":"2024-03-04T06:23:25.550565+0000","last_fullsized":"2024-03-04T06:23:25.550565+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"906'951","reported_seq":3230,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.553725+0000","last_change":"2024-03-04T06:23:25.556015+0000","last_active":"2024-03-04T06:23:25.553725+0000","last_peered":"2024-03-04T06:23:25.553725+0000","last_clean":"2024-03-04T06:23:25.553725+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:25.553725+0000","last_undegraded":"2024-03-04T06:23:25.553725+0000","last_fullsized":"2024-03-04T06:23:25.553725+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":429,"num_read_kb":348209,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2295,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.522882+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:24.522882+0000","last_peered":"2024-03-04T06:23:24.522882+0000","last_clean":"2024-03-04T06:23:24.522882+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:24.522882+0000","last_undegraded":"2024-03-04T06:23:24.522882+0000","last_fullsized":"2024-03-04T06:23:24.522882+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"932'637","reported_seq":3132,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.535170+0000","last_change":"2024-03-04T06:23:25.550042+0000","last_active":"2024-03-04T06:23:25.535170+0000","last_peered":"2024-03-04T06:23:25.535170+0000","last_clean":"2024-03-04T06:23:25.535170+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:25.535170+0000","last_undegraded":"2024-03-04T06:23:25.535170+0000","last_fullsized":"2024-03-04T06:23:25.535170+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5379032,"num_objects":50,"num_object_clones":41,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":309,"num_read_kb":250022,"num_write":214,"num_write_kb":60883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2308,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.526299+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:25.526299+0000","last_peered":"2024-03-04T06:23:25.526299+0000","last_clean":"2024-03-04T06:23:25.526299+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:25.526299+0000","last_undegraded":"2024-03-04T06:23:25.526299+0000","last_fullsized":"2024-03-04T06:23:25.526299+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"934'1792","reported_seq":3742,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.632086+0000","last_change":"2024-03-04T06:23:21.496105+0000","last_active":"2024-03-04T06:23:25.632086+0000","last_peered":"2024-03-04T06:23:25.632086+0000","last_clean":"2024-03-04T06:23:25.632086+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:25.632086+0000","last_undegraded":"2024-03-04T06:23:25.632086+0000","last_fullsized":"2024-03-04T06:23:25.632086+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1792,"log_dups_size":0,"ondisk_log_size":1792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12926134,"num_objects":142,"num_object_clones":116,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":787,"num_read_kb":618697,"num_write":659,"num_write_kb":177018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2295,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.505308+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:24.505308+0000","last_peered":"2024-03-04T06:23:24.505308+0000","last_clean":"2024-03-04T06:23:24.505308+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:24.505308+0000","last_undegraded":"2024-03-04T06:23:24.505308+0000","last_fullsized":"2024-03-04T06:23:24.505308+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"933'2112","reported_seq":3791,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.494129+0000","last_change":"2024-03-04T06:23:24.593330+0000","last_active":"2024-03-04T06:23:24.494129+0000","last_peered":"2024-03-04T06:23:24.494129+0000","last_clean":"2024-03-04T06:23:24.494129+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:24.494129+0000","last_undegraded":"2024-03-04T06:23:24.494129+0000","last_fullsized":"2024-03-04T06:23:24.494129+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2112,"log_dups_size":0,"ondisk_log_size":2112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976588,"num_objects":157,"num_object_clones":133,"num_object_copies":471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":723,"num_read_kb":536054,"num_write":717,"num_write_kb":193714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"928'910","reported_seq":3231,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.556061+0000","last_change":"2024-03-04T06:23:25.558586+0000","last_active":"2024-03-04T06:23:25.556061+0000","last_peered":"2024-03-04T06:23:25.556061+0000","last_clean":"2024-03-04T06:23:25.556061+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:25.556061+0000","last_undegraded":"2024-03-04T06:23:25.556061+0000","last_fullsized":"2024-03-04T06:23:25.556061+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269770,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":438,"num_read_kb":364480,"num_write":317,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2397,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.513064+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:24.513064+0000","last_peered":"2024-03-04T06:23:24.513064+0000","last_clean":"2024-03-04T06:23:24.513064+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:24.513064+0000","last_undegraded":"2024-03-04T06:23:24.513064+0000","last_fullsized":"2024-03-04T06:23:24.513064+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"932'1055","reported_seq":3311,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.505756+0000","last_change":"2024-03-04T06:23:24.517335+0000","last_active":"2024-03-04T06:23:24.505756+0000","last_peered":"2024-03-04T06:23:24.505756+0000","last_clean":"2024-03-04T06:23:24.505756+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:24.505756+0000","last_undegraded":"2024-03-04T06:23:24.505756+0000","last_fullsized":"2024-03-04T06:23:24.505756+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1055,"log_dups_size":0,"ondisk_log_size":1055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":380,"num_read_kb":277363,"num_write":347,"num_write_kb":92772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2362,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.542934+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:25.542934+0000","last_peered":"2024-03-04T06:23:25.542934+0000","last_clean":"2024-03-04T06:23:25.542934+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:25.542934+0000","last_undegraded":"2024-03-04T06:23:25.542934+0000","last_fullsized":"2024-03-04T06:23:25.542934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"932'1411","reported_seq":3430,"reported_epoch":934,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:25.514615+0000","last_change":"2024-03-04T06:23:24.484017+0000","last_active":"2024-03-04T06:23:25.514615+0000","last_peered":"2024-03-04T06:23:25.514615+0000","last_clean":"2024-03-04T06:23:25.514615+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:25.514615+0000","last_undegraded":"2024-03-04T06:23:25.514615+0000","last_fullsized":"2024-03-04T06:23:25.514615+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1411,"log_dups_size":0,"ondisk_log_size":1411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6774013,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":496,"num_read_kb":386812,"num_write":462,"num_write_kb":128837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2397,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.491927+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:24.491927+0000","last_peered":"2024-03-04T06:23:24.491927+0000","last_clean":"2024-03-04T06:23:24.491927+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:24.491927+0000","last_undegraded":"2024-03-04T06:23:24.491927+0000","last_fullsized":"2024-03-04T06:23:24.491927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"933'1204","reported_seq":3486,"reported_epoch":933,"state":"active+clean","last_fresh":"2024-03-04T06:23:24.502670+0000","last_change":"2024-03-04T06:23:24.537983+0000","last_active":"2024-03-04T06:23:24.502670+0000","last_peered":"2024-03-04T06:23:24.502670+0000","last_clean":"2024-03-04T06:23:24.502670+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:24.502670+0000","last_undegraded":"2024-03-04T06:23:24.502670+0000","last_fullsized":"2024-03-04T06:23:24.502670+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8808056,"num_objects":97,"num_object_clones":83,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":575,"num_read_kb":484996,"num_write":435,"num_write_kb":119544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2308,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-03-04T06:23:25.537068+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:25.537068+0000","last_peered":"2024-03-04T06:23:25.537068+0000","last_clean":"2024-03-04T06:23:25.537068+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:25.537068+0000","last_undegraded":"2024-03-04T06:23:25.537068+0000","last_fullsized":"2024-03-04T06:23:25.537068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98451007,"num_objects":1204,"num_object_clones":992,"num_object_copies":3612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":180,"num_read":6444,"num_read_kb":5118617,"num_write":5439,"num_write_kb":1484536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15948,"ondisk_log_size":15948,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":213136,"kb_used_data":146056,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94158704,"statfs":{"total":96636764160,"available":96418512896,"internally_reserved":0,"allocated":149561344,"data_stored":174045322,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706604,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":167316,"kb_used_data":109884,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94204524,"statfs":{"total":96636764160,"available":96465432576,"internally_reserved":0,"allocated":112521216,"data_stored":128576063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706603,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166244,"kb_used_data":108812,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94205596,"statfs":{"total":96636764160,"available":96466530304,"internally_reserved":0,"allocated":111423488,"data_stored":131597752,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706603,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":160012,"kb_used_data":102580,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94211828,"statfs":{"total":96636764160,"available":96472911872,"internally_reserved":0,"allocated":105041920,"data_stored":123710365,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:28.490 INFO:tasks.rados.rados.0.smithi064.stdout:3622: done (0 left) 2024-03-04T06:23:28.490 INFO:tasks.rados.rados.0.smithi064.stdout:3623: snap_remove snap 467 2024-03-04T06:23:29.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:29.495 INFO:tasks.rados.rados.0.smithi064.stdout:3623: done (0 left) 2024-03-04T06:23:29.496 INFO:tasks.rados.rados.0.smithi064.stdout:3624: snap_remove snap 428 2024-03-04T06:23:29.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:29.594+0000 7ffb9582d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:29.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:29.599+0000 7ffb9582d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:30.496 INFO:tasks.rados.rados.0.smithi064.stdout:3624: done (0 left) 2024-03-04T06:23:30.496 INFO:tasks.rados.rados.0.smithi064.stdout:3625: snap_remove snap 442 2024-03-04T06:23:31.503 INFO:tasks.rados.rados.0.smithi064.stdout:3625: done (0 left) 2024-03-04T06:23:31.503 INFO:tasks.rados.rados.0.smithi064.stdout:3626: write (excl) oid 32 current snap is 469 2024-03-04T06:23:31.503 INFO:tasks.rados.rados.0.smithi064.stdout:3626: seq_num 1641 ranges {780287=499504,1810951=440234,2697837=468079} 2024-03-04T06:23:31.511 INFO:tasks.rados.rados.0.smithi064.stdout:3626: writing smithi06431743-32 from 780287 to 1279791 tid 1 2024-03-04T06:23:31.517 INFO:tasks.rados.rados.0.smithi064.stdout:3626: writing smithi06431743-32 from 1810951 to 2251185 tid 2 2024-03-04T06:23:31.524 INFO:tasks.rados.rados.0.smithi064.stdout:3626: writing smithi06431743-32 from 2697837 to 3165916 tid 3 2024-03-04T06:23:31.524 INFO:tasks.rados.rados.0.smithi064.stdout:3627: snap_remove snap 446 2024-03-04T06:23:31.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:31.778 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:31.777+0000 7f44de35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:31.781 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:31.781+0000 7f44de35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:32.503 INFO:tasks.rados.rados.0.smithi064.stdout:3627: done (1 left) 2024-03-04T06:23:32.503 INFO:tasks.rados.rados.0.smithi064.stdout:3628: snap_create 2024-03-04T06:23:32.503 INFO:tasks.rados.rados.0.smithi064.stdout:3626: finishing write tid 1 to smithi06431743-32 2024-03-04T06:23:32.503 INFO:tasks.rados.rados.0.smithi064.stdout:3626: finishing write tid 2 to smithi06431743-32 2024-03-04T06:23:32.503 INFO:tasks.rados.rados.0.smithi064.stdout:3626: finishing write tid 3 to smithi06431743-32 2024-03-04T06:23:32.503 INFO:tasks.rados.rados.0.smithi064.stdout:3626: finishing write tid 4 to smithi06431743-32 2024-03-04T06:23:32.503 INFO:tasks.rados.rados.0.smithi064.stdout:3626: finishing write tid 5 to smithi06431743-32 2024-03-04T06:23:32.503 INFO:tasks.rados.rados.0.smithi064.stdout:3626: oid 32 updating version 0 to 1819 2024-03-04T06:23:32.503 INFO:tasks.rados.rados.0.smithi064.stdout:3626: oid 32 updating version 1819 to 1820 2024-03-04T06:23:32.503 INFO:tasks.rados.rados.0.smithi064.stdout:3626: oid 32 version 1820 is already newer than 1818 2024-03-04T06:23:32.503 INFO:tasks.rados.rados.0.smithi064.stdout:3626: oid 32 version 1820 is already newer than 1817 2024-03-04T06:23:32.504 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1820 (ObjNum 1641 snap 469 seq_num 1641) dirty exists 2024-03-04T06:23:32.504 INFO:tasks.rados.rados.0.smithi064.stdout:3626: left oid 32 (ObjNum 1641 snap 469 seq_num 1641) 2024-03-04T06:23:32.566 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:23:32.566 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:23:32.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:32.675 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:32.672+0000 7fc16e35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:32.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:32.678+0000 7fc16e35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:32.869 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:32.869 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:32.880 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1405,"stamp":"2024-03-04T06:23:32.501325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":100537198,"num_objects":1208,"num_object_clones":994,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":180,"num_read":6556,"num_read_kb":5125422,"num_write":5586,"num_write_kb":1489432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16145,"ondisk_log_size":16145,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":714300,"kb_used_data":475164,"kb_used_omap":0,"kb_used_meta":239104,"kb_avail":376773060,"statfs":{"total":386547056640,"available":385815613440,"internally_reserved":0,"allocated":486567936,"data_stored":567954502,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":244842496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1495823,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":6714,"num_write":10,"num_write_kb":3002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013806"},"pg_stats":[{"pgid":"3.f","version":"935'889","reported_seq":3194,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.542899+0000","last_change":"2024-03-04T06:23:29.554016+0000","last_active":"2024-03-04T06:23:29.542899+0000","last_peered":"2024-03-04T06:23:29.542899+0000","last_clean":"2024-03-04T06:23:29.542899+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:29.542899+0000","last_undegraded":"2024-03-04T06:23:29.542899+0000","last_fullsized":"2024-03-04T06:23:29.542899+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":889,"log_dups_size":0,"ondisk_log_size":889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6202575,"num_objects":76,"num_object_clones":61,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":356,"num_read_kb":279844,"num_write":322,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3188,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.927176+0000","last_change":"2024-03-04T06:23:29.939771+0000","last_active":"2024-03-04T06:23:29.927176+0000","last_peered":"2024-03-04T06:23:29.927176+0000","last_clean":"2024-03-04T06:23:29.927176+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:29.927176+0000","last_undegraded":"2024-03-04T06:23:29.927176+0000","last_fullsized":"2024-03-04T06:23:29.927176+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"937'1227","reported_seq":3390,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.952487+0000","last_change":"2024-03-04T06:23:29.952487+0000","last_active":"2024-03-04T06:23:29.952487+0000","last_peered":"2024-03-04T06:23:29.952487+0000","last_clean":"2024-03-04T06:23:29.952487+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:29.952487+0000","last_undegraded":"2024-03-04T06:23:29.952487+0000","last_fullsized":"2024-03-04T06:23:29.952487+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1227,"log_dups_size":0,"ondisk_log_size":1227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6101733,"num_objects":95,"num_object_clones":80,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":561,"num_read_kb":451740,"num_write":427,"num_write_kb":116189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"935'990","reported_seq":3168,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.516238+0000","last_change":"2024-03-04T06:23:29.527177+0000","last_active":"2024-03-04T06:23:29.516238+0000","last_peered":"2024-03-04T06:23:29.516238+0000","last_clean":"2024-03-04T06:23:29.516238+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:29.516238+0000","last_undegraded":"2024-03-04T06:23:29.516238+0000","last_fullsized":"2024-03-04T06:23:29.516238+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":990,"log_dups_size":0,"ondisk_log_size":990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849595,"num_objects":65,"num_object_clones":55,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":321,"num_read_kb":245412,"num_write":309,"num_write_kb":85718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"935'401","reported_seq":2972,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.926976+0000","last_change":"2024-03-04T06:23:29.935371+0000","last_active":"2024-03-04T06:23:29.926976+0000","last_peered":"2024-03-04T06:23:29.926976+0000","last_clean":"2024-03-04T06:23:29.926976+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:29.926976+0000","last_undegraded":"2024-03-04T06:23:29.926976+0000","last_fullsized":"2024-03-04T06:23:29.926976+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":401,"log_dups_size":0,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":120,"num_read_kb":93147,"num_write":120,"num_write_kb":33092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"933'982","reported_seq":3251,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.541605+0000","last_change":"2024-03-04T06:23:29.553558+0000","last_active":"2024-03-04T06:23:29.541605+0000","last_peered":"2024-03-04T06:23:29.541605+0000","last_clean":"2024-03-04T06:23:29.541605+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:29.541605+0000","last_undegraded":"2024-03-04T06:23:29.541605+0000","last_fullsized":"2024-03-04T06:23:29.541605+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":982,"log_dups_size":0,"ondisk_log_size":982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9228231,"num_objects":72,"num_object_clones":60,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":426,"num_read_kb":338098,"num_write":348,"num_write_kb":100575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2746,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.549710+0000","last_change":"2024-03-04T06:23:29.554201+0000","last_active":"2024-03-04T06:23:29.549710+0000","last_peered":"2024-03-04T06:23:29.549710+0000","last_clean":"2024-03-04T06:23:29.549710+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:29.549710+0000","last_undegraded":"2024-03-04T06:23:29.549710+0000","last_fullsized":"2024-03-04T06:23:29.549710+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"935'638","reported_seq":3054,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.944262+0000","last_change":"2024-03-04T06:23:29.952013+0000","last_active":"2024-03-04T06:23:29.944262+0000","last_peered":"2024-03-04T06:23:29.944262+0000","last_clean":"2024-03-04T06:23:29.944262+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:29.944262+0000","last_undegraded":"2024-03-04T06:23:29.944262+0000","last_fullsized":"2024-03-04T06:23:29.944262+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2505,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.914913+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:29.914913+0000","last_peered":"2024-03-04T06:23:29.914913+0000","last_clean":"2024-03-04T06:23:29.914913+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:29.914913+0000","last_undegraded":"2024-03-04T06:23:29.914913+0000","last_fullsized":"2024-03-04T06:23:29.914913+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2318,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.953835+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:29.953835+0000","last_peered":"2024-03-04T06:23:29.953835+0000","last_clean":"2024-03-04T06:23:29.953835+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:29.953835+0000","last_undegraded":"2024-03-04T06:23:29.953835+0000","last_fullsized":"2024-03-04T06:23:29.953835+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"906'951","reported_seq":3242,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.953890+0000","last_change":"2024-03-04T06:23:29.956394+0000","last_active":"2024-03-04T06:23:29.953890+0000","last_peered":"2024-03-04T06:23:29.953890+0000","last_clean":"2024-03-04T06:23:29.953890+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:29.953890+0000","last_undegraded":"2024-03-04T06:23:29.953890+0000","last_fullsized":"2024-03-04T06:23:29.953890+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":429,"num_read_kb":348209,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2306,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.538824+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:29.538824+0000","last_peered":"2024-03-04T06:23:29.538824+0000","last_clean":"2024-03-04T06:23:29.538824+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:29.538824+0000","last_undegraded":"2024-03-04T06:23:29.538824+0000","last_fullsized":"2024-03-04T06:23:29.538824+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"935'644","reported_seq":3150,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.907358+0000","last_change":"2024-03-04T06:23:29.920125+0000","last_active":"2024-03-04T06:23:29.907358+0000","last_peered":"2024-03-04T06:23:29.907358+0000","last_clean":"2024-03-04T06:23:29.907358+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:29.907358+0000","last_undegraded":"2024-03-04T06:23:29.907358+0000","last_fullsized":"2024-03-04T06:23:29.907358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6488294,"num_objects":51,"num_object_clones":42,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":310,"num_read_kb":250023,"num_write":219,"num_write_kb":62421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2318,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.948029+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:29.948029+0000","last_peered":"2024-03-04T06:23:29.948029+0000","last_clean":"2024-03-04T06:23:29.948029+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:29.948029+0000","last_undegraded":"2024-03-04T06:23:29.948029+0000","last_fullsized":"2024-03-04T06:23:29.948029+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"936'1811","reported_seq":3763,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.935967+0000","last_change":"2024-03-04T06:23:29.935967+0000","last_active":"2024-03-04T06:23:29.935967+0000","last_peered":"2024-03-04T06:23:29.935967+0000","last_clean":"2024-03-04T06:23:29.935967+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:29.935967+0000","last_undegraded":"2024-03-04T06:23:29.935967+0000","last_fullsized":"2024-03-04T06:23:29.935967+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1811,"log_dups_size":0,"ondisk_log_size":1811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13312695,"num_objects":143,"num_object_clones":117,"num_object_copies":429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":789,"num_read_kb":618698,"num_write":664,"num_write_kb":178482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2306,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.523353+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:29.523353+0000","last_peered":"2024-03-04T06:23:29.523353+0000","last_clean":"2024-03-04T06:23:29.523353+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:29.523353+0000","last_undegraded":"2024-03-04T06:23:29.523353+0000","last_fullsized":"2024-03-04T06:23:29.523353+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"935'2122","reported_seq":3804,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.925679+0000","last_change":"2024-03-04T06:23:29.939594+0000","last_active":"2024-03-04T06:23:29.925679+0000","last_peered":"2024-03-04T06:23:29.925679+0000","last_clean":"2024-03-04T06:23:29.925679+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:29.925679+0000","last_undegraded":"2024-03-04T06:23:29.925679+0000","last_fullsized":"2024-03-04T06:23:29.925679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2122,"log_dups_size":0,"ondisk_log_size":2122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976588,"num_objects":157,"num_object_clones":133,"num_object_copies":471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":723,"num_read_kb":536054,"num_write":717,"num_write_kb":193714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"935'912","reported_seq":3243,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.954607+0000","last_change":"2024-03-04T06:23:29.963622+0000","last_active":"2024-03-04T06:23:29.954607+0000","last_peered":"2024-03-04T06:23:29.954607+0000","last_clean":"2024-03-04T06:23:29.954607+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:29.954607+0000","last_undegraded":"2024-03-04T06:23:29.954607+0000","last_fullsized":"2024-03-04T06:23:29.954607+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":912,"log_dups_size":0,"ondisk_log_size":912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269770,"num_objects":67,"num_object_clones":54,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":438,"num_read_kb":364480,"num_write":317,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2408,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.938525+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:29.938525+0000","last_peered":"2024-03-04T06:23:29.938525+0000","last_clean":"2024-03-04T06:23:29.938525+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:29.938525+0000","last_undegraded":"2024-03-04T06:23:29.938525+0000","last_fullsized":"2024-03-04T06:23:29.938525+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"935'1059","reported_seq":3324,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.937733+0000","last_change":"2024-03-04T06:23:29.946328+0000","last_active":"2024-03-04T06:23:29.937733+0000","last_peered":"2024-03-04T06:23:29.937733+0000","last_clean":"2024-03-04T06:23:29.937733+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:29.937733+0000","last_undegraded":"2024-03-04T06:23:29.937733+0000","last_fullsized":"2024-03-04T06:23:29.937733+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1059,"log_dups_size":0,"ondisk_log_size":1059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":380,"num_read_kb":277363,"num_write":347,"num_write_kb":92772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2372,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.927505+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:29.927505+0000","last_peered":"2024-03-04T06:23:29.927505+0000","last_clean":"2024-03-04T06:23:29.927505+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:29.927505+0000","last_undegraded":"2024-03-04T06:23:29.927505+0000","last_fullsized":"2024-03-04T06:23:29.927505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"936'1421","reported_seq":3445,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.920583+0000","last_change":"2024-03-04T06:23:29.920583+0000","last_active":"2024-03-04T06:23:29.920583+0000","last_peered":"2024-03-04T06:23:29.920583+0000","last_clean":"2024-03-04T06:23:29.920583+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:29.920583+0000","last_undegraded":"2024-03-04T06:23:29.920583+0000","last_fullsized":"2024-03-04T06:23:29.920583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1421,"log_dups_size":0,"ondisk_log_size":1421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6774013,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":496,"num_read_kb":386812,"num_write":462,"num_write_kb":128837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2408,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.929834+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:29.929834+0000","last_peered":"2024-03-04T06:23:29.929834+0000","last_clean":"2024-03-04T06:23:29.929834+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:29.929834+0000","last_undegraded":"2024-03-04T06:23:29.929834+0000","last_fullsized":"2024-03-04T06:23:29.929834+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"935'1212","reported_seq":3499,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.915858+0000","last_change":"2024-03-04T06:23:29.939953+0000","last_active":"2024-03-04T06:23:29.915858+0000","last_peered":"2024-03-04T06:23:29.915858+0000","last_clean":"2024-03-04T06:23:29.915858+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:29.915858+0000","last_undegraded":"2024-03-04T06:23:29.915858+0000","last_fullsized":"2024-03-04T06:23:29.915858+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1212,"log_dups_size":0,"ondisk_log_size":1212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8808056,"num_objects":97,"num_object_clones":83,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":575,"num_read_kb":484996,"num_write":435,"num_write_kb":119544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2318,"reported_epoch":938,"state":"active+clean","last_fresh":"2024-03-04T06:23:29.949968+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:29.949968+0000","last_peered":"2024-03-04T06:23:29.949968+0000","last_clean":"2024-03-04T06:23:29.949968+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:29.949968+0000","last_undegraded":"2024-03-04T06:23:29.949968+0000","last_fullsized":"2024-03-04T06:23:29.949968+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":99946830,"num_objects":1206,"num_object_clones":994,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":180,"num_read":6453,"num_read_kb":5125331,"num_write":5449,"num_write_kb":1487538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16048,"ondisk_log_size":16048,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706609,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":215292,"kb_used_data":148212,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94156548,"statfs":{"total":96636764160,"available":96416305152,"internally_reserved":0,"allocated":151769088,"data_stored":176617991,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706609,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":169392,"kb_used_data":112040,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94202448,"statfs":{"total":96636764160,"available":96463306752,"internally_reserved":0,"allocated":114728960,"data_stored":131148732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706608,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":168096,"kb_used_data":110744,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94203744,"statfs":{"total":96636764160,"available":96464633856,"internally_reserved":0,"allocated":113401856,"data_stored":134330862,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706608,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":161520,"kb_used_data":104168,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94210320,"statfs":{"total":96636764160,"available":96471367680,"internally_reserved":0,"allocated":106668032,"data_stored":125856917,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:33.509 INFO:tasks.rados.rados.0.smithi064.stdout:3626: done (1 left) 2024-03-04T06:23:33.509 INFO:tasks.rados.rados.0.smithi064.stdout:3628: done (0 left) 2024-03-04T06:23:33.509 INFO:tasks.rados.rados.0.smithi064.stdout:3629: delete oid 15 current snap is 470 2024-03-04T06:23:33.593 INFO:tasks.rados.rados.0.smithi064.stdout:3629: done (0 left) 2024-03-04T06:23:33.593 INFO:tasks.rados.rados.0.smithi064.stdout:3630: read oid 41 snap -1 2024-03-04T06:23:33.593 INFO:tasks.rados.rados.0.smithi064.stdout:3630: expect (ObjNum 1505 snap 421 seq_num 1505) 2024-03-04T06:23:33.593 INFO:tasks.rados.rados.0.smithi064.stdout:3631: write oid 10 current snap is 470 2024-03-04T06:23:33.593 INFO:tasks.rados.rados.0.smithi064.stdout:3631: seq_num 1643 ranges {413901=766355,1644268=443903,2869036=490623} 2024-03-04T06:23:33.601 INFO:tasks.rados.rados.0.smithi064.stdout:3631: writing smithi06431743-10 from 413901 to 1180256 tid 1 2024-03-04T06:23:33.606 INFO:tasks.rados.rados.0.smithi064.stdout:3631: writing smithi06431743-10 from 1644268 to 2088171 tid 2 2024-03-04T06:23:33.614 INFO:tasks.rados.rados.0.smithi064.stdout:3631: writing smithi06431743-10 from 2869036 to 3359659 tid 3 2024-03-04T06:23:33.614 INFO:tasks.rados.rados.0.smithi064.stdout:3632: write (excl) oid 16 current snap is 470 2024-03-04T06:23:33.614 INFO:tasks.rados.rados.0.smithi064.stdout:3632: seq_num 1644 ranges {430708=441604,1302750=703914,2496459=701895} 2024-03-04T06:23:33.620 INFO:tasks.rados.rados.0.smithi064.stdout:3632: writing smithi06431743-16 from 430708 to 872312 tid 1 2024-03-04T06:23:33.629 INFO:tasks.rados.rados.0.smithi064.stdout:3632: writing smithi06431743-16 from 1302750 to 2006664 tid 2 2024-03-04T06:23:33.636 INFO:tasks.rados.rados.0.smithi064.stdout:3632: writing smithi06431743-16 from 2496459 to 3198354 tid 3 2024-03-04T06:23:33.636 INFO:tasks.rados.rados.0.smithi064.stdout:3633: write oid 29 current snap is 470 2024-03-04T06:23:33.636 INFO:tasks.rados.rados.0.smithi064.stdout:3630: expect (ObjNum 1505 snap 421 seq_num 1505) 2024-03-04T06:23:33.793 INFO:tasks.rados.rados.0.smithi064.stdout:3632: finishing write tid 1 to smithi06431743-16 2024-03-04T06:23:33.793 INFO:tasks.rados.rados.0.smithi064.stdout:3631: finishing write tid 1 to smithi06431743-10 2024-03-04T06:23:33.793 INFO:tasks.rados.rados.0.smithi064.stdout:3632: finishing write tid 2 to smithi06431743-16 2024-03-04T06:23:33.793 INFO:tasks.rados.rados.0.smithi064.stdout:3631: finishing write tid 2 to smithi06431743-10 2024-03-04T06:23:33.793 INFO:tasks.rados.rados.0.smithi064.stdout:3632: finishing write tid 3 to smithi06431743-16 2024-03-04T06:23:33.793 INFO:tasks.rados.rados.0.smithi064.stdout:3631: finishing write tid 3 to smithi06431743-10 2024-03-04T06:23:33.794 INFO:tasks.rados.rados.0.smithi064.stdout:3632: finishing write tid 4 to smithi06431743-16 2024-03-04T06:23:33.794 INFO:tasks.rados.rados.0.smithi064.stdout:3632: finishing write tid 5 to smithi06431743-16 2024-03-04T06:23:33.794 INFO:tasks.rados.rados.0.smithi064.stdout:3632: oid 16 updating version 0 to 1074 2024-03-04T06:23:33.794 INFO:tasks.rados.rados.0.smithi064.stdout:3632: oid 16 version 1074 is already newer than 1072 2024-03-04T06:23:33.794 INFO:tasks.rados.rados.0.smithi064.stdout:3632: oid 16 updating version 1074 to 1075 2024-03-04T06:23:33.794 INFO:tasks.rados.rados.0.smithi064.stdout:3632: oid 16 version 1075 is already newer than 1074 2024-03-04T06:23:33.794 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 1075 (ObjNum 1644 snap 470 seq_num 1644) dirty exists 2024-03-04T06:23:33.794 INFO:tasks.rados.rados.0.smithi064.stdout:3632: left oid 16 (ObjNum 1644 snap 470 seq_num 1644) 2024-03-04T06:23:33.794 INFO:tasks.rados.rados.0.smithi064.stdout:3631: finishing write tid 4 to smithi06431743-10 2024-03-04T06:23:33.794 INFO:tasks.rados.rados.0.smithi064.stdout:3633: seq_num 1645 ranges {447515=516853,1644879=447572,2807529=629520} 2024-03-04T06:23:33.798 INFO:tasks.rados.rados.0.smithi064.stdout:3633: writing smithi06431743-29 from 447515 to 964368 tid 1 2024-03-04T06:23:33.804 INFO:tasks.rados.rados.0.smithi064.stdout:3633: writing smithi06431743-29 from 1644879 to 2092451 tid 2 2024-03-04T06:23:33.812 INFO:tasks.rados.rados.0.smithi064.stdout:3633: writing smithi06431743-29 from 2807529 to 3437049 tid 3 2024-03-04T06:23:33.812 INFO:tasks.rados.rados.0.smithi064.stdout:3630: done (3 left) 2024-03-04T06:23:33.812 INFO:tasks.rados.rados.0.smithi064.stdout:3632: done (2 left) 2024-03-04T06:23:33.812 INFO:tasks.rados.rados.0.smithi064.stdout:3634: write (excl) oid 7 current snap is 470 2024-03-04T06:23:33.812 INFO:tasks.rados.rados.0.smithi064.stdout:3631: finishing write tid 5 to smithi06431743-10 2024-03-04T06:23:33.812 INFO:tasks.rados.rados.0.smithi064.stdout:3631: oid 10 updating version 0 to 988 2024-03-04T06:23:33.812 INFO:tasks.rados.rados.0.smithi064.stdout:3631: oid 10 version 988 is already newer than 986 2024-03-04T06:23:33.812 INFO:tasks.rados.rados.0.smithi064.stdout:3631: oid 10 version 988 is already newer than 987 2024-03-04T06:23:33.812 INFO:tasks.rados.rados.0.smithi064.stdout:3631: oid 10 updating version 988 to 989 2024-03-04T06:23:33.813 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 989 (ObjNum 1643 snap 470 seq_num 1643) dirty exists 2024-03-04T06:23:33.813 INFO:tasks.rados.rados.0.smithi064.stdout:3631: left oid 10 (ObjNum 1643 snap 470 seq_num 1643) 2024-03-04T06:23:33.813 INFO:tasks.rados.rados.0.smithi064.stdout:3634: seq_num 1646 ranges {464322=592102,1703361=591230,2718599=440792} 2024-03-04T06:23:33.817 INFO:tasks.rados.rados.0.smithi064.stdout:3634: writing smithi06431743-7 from 464322 to 1056424 tid 1 2024-03-04T06:23:33.824 INFO:tasks.rados.rados.0.smithi064.stdout:3634: writing smithi06431743-7 from 1703361 to 2294591 tid 2 2024-03-04T06:23:33.829 INFO:tasks.rados.rados.0.smithi064.stdout:3634: writing smithi06431743-7 from 2718599 to 3159391 tid 3 2024-03-04T06:23:33.829 INFO:tasks.rados.rados.0.smithi064.stdout:3631: done (2 left) 2024-03-04T06:23:33.830 INFO:tasks.rados.rados.0.smithi064.stdout:3635: read oid 33 snap 438 2024-03-04T06:23:33.830 INFO:tasks.rados.rados.0.smithi064.stdout:3635: expect (ObjNum 1526 snap 427 seq_num 1526) 2024-03-04T06:23:33.830 INFO:tasks.rados.rados.0.smithi064.stdout:3636: read oid 17 snap 468 2024-03-04T06:23:33.830 INFO:tasks.rados.rados.0.smithi064.stdout:3636: expect deleted 2024-03-04T06:23:33.830 INFO:tasks.rados.rados.0.smithi064.stdout:3637: write (excl) oid 39 current snap is 470 2024-03-04T06:23:33.830 INFO:tasks.rados.rados.0.smithi064.stdout:3637: seq_num 1647 ranges {481129=667351,1761843=451241,2862375=652064} 2024-03-04T06:23:33.837 INFO:tasks.rados.rados.0.smithi064.stdout:3637: writing smithi06431743-39 from 481129 to 1148480 tid 1 2024-03-04T06:23:33.845 INFO:tasks.rados.rados.0.smithi064.stdout:3637: writing smithi06431743-39 from 1761843 to 2213084 tid 2 2024-03-04T06:23:33.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:33.854 INFO:tasks.rados.rados.0.smithi064.stdout:3637: writing smithi06431743-39 from 2862375 to 3514439 tid 3 2024-03-04T06:23:33.854 INFO:tasks.rados.rados.0.smithi064.stdout:3638: snap_remove snap 427 2024-03-04T06:23:33.968 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:33.966+0000 7fcda2b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:33.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:33.970+0000 7fcda2b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:34.512 INFO:tasks.rados.rados.0.smithi064.stdout:3638: done (5 left) 2024-03-04T06:23:34.512 INFO:tasks.rados.rados.0.smithi064.stdout:3639: snap_create 2024-03-04T06:23:34.512 INFO:tasks.rados.rados.0.smithi064.stdout:3633: finishing write tid 1 to smithi06431743-29 2024-03-04T06:23:34.512 INFO:tasks.rados.rados.0.smithi064.stdout:3634: finishing write tid 1 to smithi06431743-7 2024-03-04T06:23:34.513 INFO:tasks.rados.rados.0.smithi064.stdout:3633: finishing write tid 2 to smithi06431743-29 2024-03-04T06:23:34.513 INFO:tasks.rados.rados.0.smithi064.stdout:3635: expect (ObjNum 1526 snap 427 seq_num 1526) 2024-03-04T06:23:34.721 INFO:tasks.rados.rados.0.smithi064.stdout:3634: finishing write tid 2 to smithi06431743-7 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3637: finishing write tid 1 to smithi06431743-39 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3633: finishing write tid 3 to smithi06431743-29 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3634: finishing write tid 3 to smithi06431743-7 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3633: finishing write tid 4 to smithi06431743-29 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3633: finishing write tid 5 to smithi06431743-29 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3633: oid 29 updating version 0 to 2154 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3633: oid 29 updating version 2154 to 2156 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3633: oid 29 version 2156 is already newer than 2156 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3633: oid 29 updating version 2156 to 2157 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 2157 (ObjNum 1645 snap 470 seq_num 1645) dirty exists 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3633: left oid 29 (ObjNum 1645 snap 470 seq_num 1645) 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3637: finishing write tid 2 to smithi06431743-39 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3634: finishing write tid 4 to smithi06431743-7 2024-03-04T06:23:34.722 INFO:tasks.rados.rados.0.smithi064.stdout:3634: finishing write tid 5 to smithi06431743-7 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3634: oid 7 updating version 0 to 652 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3634: oid 7 updating version 652 to 655 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3634: oid 7 version 655 is already newer than 654 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3634: oid 7 version 655 is already newer than 654 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 655 (ObjNum 1646 snap 470 seq_num 1646) dirty exists 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3634: left oid 7 (ObjNum 1646 snap 470 seq_num 1646) 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3637: finishing write tid 3 to smithi06431743-39 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3637: finishing write tid 4 to smithi06431743-39 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3637: finishing write tid 5 to smithi06431743-39 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3637: oid 39 updating version 0 to 1239 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3637: oid 39 updating version 1239 to 1241 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3637: oid 39 version 1241 is already newer than 1240 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3637: oid 39 version 1241 is already newer than 1238 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 1241 (ObjNum 1647 snap 470 seq_num 1647) dirty exists 2024-03-04T06:23:34.723 INFO:tasks.rados.rados.0.smithi064.stdout:3637: left oid 39 (ObjNum 1647 snap 470 seq_num 1647) 2024-03-04T06:23:35.517 INFO:tasks.rados.rados.0.smithi064.stdout:3633: done (5 left) 2024-03-04T06:23:35.518 INFO:tasks.rados.rados.0.smithi064.stdout:3634: done (4 left) 2024-03-04T06:23:35.518 INFO:tasks.rados.rados.0.smithi064.stdout:3635: done (3 left) 2024-03-04T06:23:35.518 INFO:tasks.rados.rados.0.smithi064.stdout:3636: done (2 left) 2024-03-04T06:23:35.518 INFO:tasks.rados.rados.0.smithi064.stdout:3637: done (1 left) 2024-03-04T06:23:35.518 INFO:tasks.rados.rados.0.smithi064.stdout:3639: done (0 left) 2024-03-04T06:23:35.518 INFO:tasks.rados.rados.0.smithi064.stdout:3640: snap_remove snap 457 2024-03-04T06:23:36.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:36.156 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:36.154+0000 7f61ff75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:36.159 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:36.159+0000 7f61ff75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:36.520 INFO:tasks.rados.rados.0.smithi064.stdout:3640: done (0 left) 2024-03-04T06:23:36.520 INFO:tasks.rados.rados.0.smithi064.stdout:3641: read oid 23 snap -1 2024-03-04T06:23:36.520 INFO:tasks.rados.rados.0.smithi064.stdout:3641: expect (ObjNum 1508 snap 421 seq_num 1508) 2024-03-04T06:23:36.521 INFO:tasks.rados.rados.0.smithi064.stdout:3642: write (excl) oid 44 current snap is 471 2024-03-04T06:23:36.521 INFO:tasks.rados.rados.0.smithi064.stdout:3642: seq_num 1648 ranges {497936=742600,1820325=594899,2889798=463336} 2024-03-04T06:23:36.529 INFO:tasks.rados.rados.0.smithi064.stdout:3642: writing smithi06431743-44 from 497936 to 1240536 tid 1 2024-03-04T06:23:36.537 INFO:tasks.rados.rados.0.smithi064.stdout:3642: writing smithi06431743-44 from 1820325 to 2415224 tid 2 2024-03-04T06:23:36.544 INFO:tasks.rados.rados.0.smithi064.stdout:3642: writing smithi06431743-44 from 2889798 to 3353134 tid 3 2024-03-04T06:23:36.544 INFO:tasks.rados.rados.0.smithi064.stdout:3643: write (excl) oid 22 current snap is 471 2024-03-04T06:23:36.544 INFO:tasks.rados.rados.0.smithi064.stdout:3643: seq_num 1649 ranges {514743=417849,1362454=454910,2400868=790961} 2024-03-04T06:23:36.551 INFO:tasks.rados.rados.0.smithi064.stdout:3643: writing smithi06431743-22 from 514743 to 932592 tid 1 2024-03-04T06:23:36.558 INFO:tasks.rados.rados.0.smithi064.stdout:3643: writing smithi06431743-22 from 1362454 to 1817364 tid 2 2024-03-04T06:23:36.567 INFO:tasks.rados.rados.0.smithi064.stdout:3643: writing smithi06431743-22 from 2400868 to 3191829 tid 3 2024-03-04T06:23:36.568 INFO:tasks.rados.rados.0.smithi064.stdout:3644: snap_remove snap 433 2024-03-04T06:23:37.530 INFO:tasks.rados.rados.0.smithi064.stdout:3644: done (3 left) 2024-03-04T06:23:37.531 INFO:tasks.rados.rados.0.smithi064.stdout:3645: write oid 8 current snap is 471 2024-03-04T06:23:37.531 INFO:tasks.rados.rados.0.smithi064.stdout:3645: seq_num 1650 ranges {531550=609451,1937289=598568,2944644=602233} 2024-03-04T06:23:37.536 INFO:tasks.rados.rados.0.smithi064.stdout:3645: writing smithi06431743-8 from 531550 to 1141001 tid 1 2024-03-04T06:23:37.544 INFO:tasks.rados.rados.0.smithi064.stdout:3645: writing smithi06431743-8 from 1937289 to 2535857 tid 2 2024-03-04T06:23:37.550 INFO:tasks.rados.rados.0.smithi064.stdout:3645: writing smithi06431743-8 from 2944644 to 3546877 tid 3 2024-03-04T06:23:37.550 INFO:tasks.rados.rados.0.smithi064.stdout:3646: read oid 28 snap -1 2024-03-04T06:23:37.550 INFO:tasks.rados.rados.0.smithi064.stdout:3646: expect deleted 2024-03-04T06:23:37.550 INFO:tasks.rados.rados.0.smithi064.stdout:3647: write (excl) oid 31 current snap is 471 2024-03-04T06:23:37.550 INFO:tasks.rados.rados.0.smithi064.stdout:3647: seq_num 1651 ranges {548357=684700,1995771=458579,2972067=413505} 2024-03-04T06:23:37.556 INFO:tasks.rados.rados.0.smithi064.stdout:3647: writing smithi06431743-31 from 548357 to 1233057 tid 1 2024-03-04T06:23:37.563 INFO:tasks.rados.rados.0.smithi064.stdout:3647: writing smithi06431743-31 from 1995771 to 2454350 tid 2 2024-03-04T06:23:37.568 INFO:tasks.rados.rados.0.smithi064.stdout:3647: writing smithi06431743-31 from 2972067 to 3385572 tid 3 2024-03-04T06:23:37.568 INFO:tasks.rados.rados.0.smithi064.stdout:3648: snap_create 2024-03-04T06:23:37.568 INFO:tasks.rados.rados.0.smithi064.stdout:3641: expect (ObjNum 1508 snap 421 seq_num 1508) 2024-03-04T06:23:37.746 INFO:tasks.rados.rados.0.smithi064.stdout:3643: finishing write tid 1 to smithi06431743-22 2024-03-04T06:23:37.746 INFO:tasks.rados.rados.0.smithi064.stdout:3643: finishing write tid 2 to smithi06431743-22 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3642: finishing write tid 1 to smithi06431743-44 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3643: finishing write tid 3 to smithi06431743-22 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3642: finishing write tid 2 to smithi06431743-44 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3642: finishing write tid 3 to smithi06431743-44 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3643: finishing write tid 4 to smithi06431743-22 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3643: finishing write tid 5 to smithi06431743-22 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3643: oid 22 updating version 0 to 1248 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3643: oid 22 version 1248 is already newer than 1247 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3643: oid 22 updating version 1248 to 1249 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3643: oid 22 version 1249 is already newer than 1246 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 1249 (ObjNum 1649 snap 471 seq_num 1649) dirty exists 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3643: left oid 22 (ObjNum 1649 snap 471 seq_num 1649) 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3642: finishing write tid 4 to smithi06431743-44 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3642: finishing write tid 5 to smithi06431743-44 2024-03-04T06:23:37.747 INFO:tasks.rados.rados.0.smithi064.stdout:3642: oid 44 updating version 0 to 1002 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3642: oid 44 version 1002 is already newer than 1001 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3642: oid 44 version 1002 is already newer than 1001 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3642: oid 44 version 1002 is already newer than 999 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 44 v 1002 (ObjNum 1648 snap 471 seq_num 1648) dirty exists 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3642: left oid 44 (ObjNum 1648 snap 471 seq_num 1648) 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3645: finishing write tid 1 to smithi06431743-8 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3647: finishing write tid 1 to smithi06431743-31 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3647: finishing write tid 2 to smithi06431743-31 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3645: finishing write tid 2 to smithi06431743-8 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3647: finishing write tid 3 to smithi06431743-31 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3645: finishing write tid 3 to smithi06431743-8 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3647: finishing write tid 4 to smithi06431743-31 2024-03-04T06:23:37.748 INFO:tasks.rados.rados.0.smithi064.stdout:3647: finishing write tid 5 to smithi06431743-31 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:3647: oid 31 updating version 0 to 411 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:3647: oid 31 updating version 411 to 412 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:3647: oid 31 updating version 412 to 413 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:3647: oid 31 version 413 is already newer than 410 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 413 (ObjNum 1651 snap 471 seq_num 1651) dirty exists 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:3647: left oid 31 (ObjNum 1651 snap 471 seq_num 1651) 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:3645: finishing write tid 4 to smithi06431743-8 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:3645: finishing write tid 5 to smithi06431743-8 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:3645: oid 8 updating version 0 to 1090 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:3645: oid 8 version 1090 is already newer than 1088 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:3645: oid 8 version 1090 is already newer than 1090 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:3645: oid 8 updating version 1090 to 1091 2024-03-04T06:23:37.749 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 8 v 1091 (ObjNum 1650 snap 471 seq_num 1650) dirty exists 2024-03-04T06:23:37.750 INFO:tasks.rados.rados.0.smithi064.stdout:3645: left oid 8 (ObjNum 1650 snap 471 seq_num 1650) 2024-03-04T06:23:37.883 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:23:37.883 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:23:37.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:38.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:38.012+0000 7f5454b07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:38.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:38.019+0000 7f5454b07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:38.212 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:38.212 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:38.224 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1413,"stamp":"2024-03-04T06:23:37.529252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":106655375,"num_objects":1213,"num_object_clones":999,"num_object_copies":3639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":179,"num_read":6578,"num_read_kb":5142407,"num_write":5617,"num_write_kb":1499153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16365,"ondisk_log_size":16365,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":727172,"kb_used_data":483876,"kb_used_omap":0,"kb_used_meta":243264,"kb_avail":376760188,"statfs":{"total":386547056640,"available":385802432512,"internally_reserved":0,"allocated":495489024,"data_stored":575876666,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":249102336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6118177,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":22,"num_read_kb":16985,"num_write":31,"num_write_kb":9721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.021129"},"pg_stats":[{"pgid":"3.f","version":"943'901","reported_seq":3208,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.535239+0000","last_change":"2024-03-04T06:23:34.643064+0000","last_active":"2024-03-04T06:23:34.535239+0000","last_peered":"2024-03-04T06:23:34.535239+0000","last_clean":"2024-03-04T06:23:34.535239+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:34.535239+0000","last_undegraded":"2024-03-04T06:23:34.535239+0000","last_fullsized":"2024-03-04T06:23:34.535239+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6202575,"num_objects":76,"num_object_clones":61,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":356,"num_read_kb":279844,"num_write":322,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3207,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.550303+0000","last_change":"2024-03-04T06:23:34.559248+0000","last_active":"2024-03-04T06:23:34.550303+0000","last_peered":"2024-03-04T06:23:34.550303+0000","last_clean":"2024-03-04T06:23:34.550303+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:34.550303+0000","last_undegraded":"2024-03-04T06:23:34.550303+0000","last_fullsized":"2024-03-04T06:23:34.550303+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"944'1247","reported_seq":3412,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-03-04T06:23:35.590421+0000","last_change":"2024-03-04T06:23:35.553085+0000","last_active":"2024-03-04T06:23:35.590421+0000","last_peered":"2024-03-04T06:23:35.590421+0000","last_clean":"2024-03-04T06:23:35.590421+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:35.590421+0000","last_undegraded":"2024-03-04T06:23:35.590421+0000","last_fullsized":"2024-03-04T06:23:35.590421+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1247,"log_dups_size":0,"ondisk_log_size":1247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9782862,"num_objects":95,"num_object_clones":80,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":563,"num_read_kb":451741,"num_write":432,"num_write_kb":117919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"943'998","reported_seq":3182,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.531633+0000","last_change":"2024-03-04T06:23:34.608750+0000","last_active":"2024-03-04T06:23:34.531633+0000","last_peered":"2024-03-04T06:23:34.531633+0000","last_clean":"2024-03-04T06:23:34.531633+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:34.531633+0000","last_undegraded":"2024-03-04T06:23:34.531633+0000","last_fullsized":"2024-03-04T06:23:34.531633+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5849595,"num_objects":65,"num_object_clones":55,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":321,"num_read_kb":245412,"num_write":309,"num_write_kb":85718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"943'407","reported_seq":2987,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.578383+0000","last_change":"2024-03-04T06:23:34.639201+0000","last_active":"2024-03-04T06:23:34.578383+0000","last_peered":"2024-03-04T06:23:34.578383+0000","last_clean":"2024-03-04T06:23:34.578383+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:34.578383+0000","last_undegraded":"2024-03-04T06:23:34.578383+0000","last_fullsized":"2024-03-04T06:23:34.578383+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":120,"num_read_kb":93147,"num_write":120,"num_write_kb":33092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"943'993","reported_seq":3271,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.544294+0000","last_change":"2024-03-04T06:23:34.613472+0000","last_active":"2024-03-04T06:23:34.544294+0000","last_peered":"2024-03-04T06:23:34.544294+0000","last_clean":"2024-03-04T06:23:34.544294+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:34.544294+0000","last_undegraded":"2024-03-04T06:23:34.544294+0000","last_fullsized":"2024-03-04T06:23:34.544294+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9782862,"num_objects":73,"num_object_clones":61,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":427,"num_read_kb":338099,"num_write":353,"num_write_kb":102238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2761,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.545878+0000","last_change":"2024-03-04T06:23:34.556183+0000","last_active":"2024-03-04T06:23:34.545878+0000","last_peered":"2024-03-04T06:23:34.545878+0000","last_clean":"2024-03-04T06:23:34.545878+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:34.545878+0000","last_undegraded":"2024-03-04T06:23:34.545878+0000","last_fullsized":"2024-03-04T06:23:34.545878+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"943'650","reported_seq":3072,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-03-04T06:23:35.528918+0000","last_change":"2024-03-04T06:23:34.647301+0000","last_active":"2024-03-04T06:23:35.528918+0000","last_peered":"2024-03-04T06:23:35.528918+0000","last_clean":"2024-03-04T06:23:35.528918+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:35.528918+0000","last_undegraded":"2024-03-04T06:23:35.528918+0000","last_fullsized":"2024-03-04T06:23:35.528918+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2516,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.569111+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:34.569111+0000","last_peered":"2024-03-04T06:23:34.569111+0000","last_clean":"2024-03-04T06:23:34.569111+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:34.569111+0000","last_undegraded":"2024-03-04T06:23:34.569111+0000","last_fullsized":"2024-03-04T06:23:34.569111+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2332,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-03-04T06:23:35.537253+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:35.537253+0000","last_peered":"2024-03-04T06:23:35.537253+0000","last_clean":"2024-03-04T06:23:35.537253+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:35.537253+0000","last_undegraded":"2024-03-04T06:23:35.537253+0000","last_fullsized":"2024-03-04T06:23:35.537253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"906'951","reported_seq":3263,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-03-04T06:23:35.535342+0000","last_change":"2024-03-04T06:23:34.569449+0000","last_active":"2024-03-04T06:23:35.535342+0000","last_peered":"2024-03-04T06:23:35.535342+0000","last_clean":"2024-03-04T06:23:35.535342+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:35.535342+0000","last_undegraded":"2024-03-04T06:23:35.535342+0000","last_fullsized":"2024-03-04T06:23:35.535342+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043255,"num_objects":71,"num_object_clones":57,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":435,"num_read_kb":356021,"num_write":321,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2317,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.544044+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:34.544044+0000","last_peered":"2024-03-04T06:23:34.544044+0000","last_clean":"2024-03-04T06:23:34.544044+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:34.544044+0000","last_undegraded":"2024-03-04T06:23:34.544044+0000","last_fullsized":"2024-03-04T06:23:34.544044+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"943'657","reported_seq":3170,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.570123+0000","last_change":"2024-03-04T06:23:34.648077+0000","last_active":"2024-03-04T06:23:34.570123+0000","last_peered":"2024-03-04T06:23:34.570123+0000","last_clean":"2024-03-04T06:23:34.570123+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:34.570123+0000","last_undegraded":"2024-03-04T06:23:34.570123+0000","last_fullsized":"2024-03-04T06:23:34.570123+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210995,"num_objects":52,"num_object_clones":43,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":312,"num_read_kb":250024,"num_write":224,"num_write_kb":64009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2332,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-03-04T06:23:35.528186+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:35.528186+0000","last_peered":"2024-03-04T06:23:35.528186+0000","last_clean":"2024-03-04T06:23:35.528186+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:35.528186+0000","last_undegraded":"2024-03-04T06:23:35.528186+0000","last_fullsized":"2024-03-04T06:23:35.528186+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"943'1834","reported_seq":3781,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:35.634951+0000","last_change":"2024-03-04T06:23:35.653255+0000","last_active":"2024-03-04T06:23:35.634951+0000","last_peered":"2024-03-04T06:23:35.634951+0000","last_clean":"2024-03-04T06:23:35.634951+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:35.634951+0000","last_undegraded":"2024-03-04T06:23:35.634951+0000","last_fullsized":"2024-03-04T06:23:35.634951+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1834,"log_dups_size":0,"ondisk_log_size":1834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12959781,"num_objects":144,"num_object_clones":118,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":791,"num_read_kb":618699,"num_write":669,"num_write_kb":179858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2317,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.523122+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:34.523122+0000","last_peered":"2024-03-04T06:23:34.523122+0000","last_clean":"2024-03-04T06:23:34.523122+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:34.523122+0000","last_undegraded":"2024-03-04T06:23:34.523122+0000","last_fullsized":"2024-03-04T06:23:34.523122+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"943'2165","reported_seq":3828,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.547225+0000","last_change":"2024-03-04T06:23:34.694634+0000","last_active":"2024-03-04T06:23:34.547225+0000","last_peered":"2024-03-04T06:23:34.547225+0000","last_clean":"2024-03-04T06:23:34.547225+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:34.547225+0000","last_undegraded":"2024-03-04T06:23:34.547225+0000","last_fullsized":"2024-03-04T06:23:34.547225+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2165,"log_dups_size":0,"ondisk_log_size":2165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13262307,"num_objects":158,"num_object_clones":134,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":724,"num_read_kb":536055,"num_write":722,"num_write_kb":195272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"943'922","reported_seq":3262,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-03-04T06:23:35.534760+0000","last_change":"2024-03-04T06:23:34.661942+0000","last_active":"2024-03-04T06:23:35.534760+0000","last_peered":"2024-03-04T06:23:35.534760+0000","last_clean":"2024-03-04T06:23:35.534760+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:35.534760+0000","last_undegraded":"2024-03-04T06:23:35.534760+0000","last_fullsized":"2024-03-04T06:23:35.534760+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":922,"log_dups_size":0,"ondisk_log_size":922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865973,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":438,"num_read_kb":364480,"num_write":318,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2423,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.557468+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:34.557468+0000","last_peered":"2024-03-04T06:23:34.557468+0000","last_clean":"2024-03-04T06:23:34.557468+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:34.557468+0000","last_undegraded":"2024-03-04T06:23:34.557468+0000","last_fullsized":"2024-03-04T06:23:34.557468+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"943'1081","reported_seq":3348,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.561934+0000","last_change":"2024-03-04T06:23:34.673005+0000","last_active":"2024-03-04T06:23:34.561934+0000","last_peered":"2024-03-04T06:23:34.561934+0000","last_clean":"2024-03-04T06:23:34.561934+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:34.561934+0000","last_undegraded":"2024-03-04T06:23:34.561934+0000","last_fullsized":"2024-03-04T06:23:34.561934+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1081,"log_dups_size":0,"ondisk_log_size":1081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3630708,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":382,"num_read_kb":277364,"num_write":352,"num_write_kb":94578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2383,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.581343+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:34.581343+0000","last_peered":"2024-03-04T06:23:34.581343+0000","last_clean":"2024-03-04T06:23:34.581343+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:34.581343+0000","last_undegraded":"2024-03-04T06:23:34.581343+0000","last_fullsized":"2024-03-04T06:23:34.581343+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"943'1431","reported_seq":3460,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.664811+0000","last_change":"2024-03-04T06:23:34.664811+0000","last_active":"2024-03-04T06:23:34.664811+0000","last_peered":"2024-03-04T06:23:34.664811+0000","last_clean":"2024-03-04T06:23:34.664811+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:34.664811+0000","last_undegraded":"2024-03-04T06:23:34.664811+0000","last_fullsized":"2024-03-04T06:23:34.664811+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1431,"log_dups_size":0,"ondisk_log_size":1431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6774013,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":496,"num_read_kb":386812,"num_write":462,"num_write_kb":128837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2423,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.548510+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:34.548510+0000","last_peered":"2024-03-04T06:23:34.548510+0000","last_clean":"2024-03-04T06:23:34.548510+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:34.548510+0000","last_undegraded":"2024-03-04T06:23:34.548510+0000","last_fullsized":"2024-03-04T06:23:34.548510+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"943'1242","reported_seq":3521,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-03-04T06:23:34.551204+0000","last_change":"2024-03-04T06:23:34.669634+0000","last_active":"2024-03-04T06:23:34.551204+0000","last_peered":"2024-03-04T06:23:34.551204+0000","last_clean":"2024-03-04T06:23:34.551204+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:34.551204+0000","last_undegraded":"2024-03-04T06:23:34.551204+0000","last_fullsized":"2024-03-04T06:23:34.551204+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1242,"log_dups_size":0,"ondisk_log_size":1242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8808056,"num_objects":97,"num_object_clones":83,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":581,"num_read_kb":494163,"num_write":435,"num_write_kb":119544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2332,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-03-04T06:23:35.534486+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:35.534486+0000","last_peered":"2024-03-04T06:23:35.534486+0000","last_clean":"2024-03-04T06:23:35.534486+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:35.534486+0000","last_undegraded":"2024-03-04T06:23:35.534486+0000","last_fullsized":"2024-03-04T06:23:35.534486+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":106065007,"num_objects":1211,"num_object_clones":999,"num_object_copies":3633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":179,"num_read":6475,"num_read_kb":5142316,"num_write":5480,"num_write_kb":1497259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16268,"ondisk_log_size":16268,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706614,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":220660,"kb_used_data":149420,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94151180,"statfs":{"total":96636764160,"available":96410808320,"internally_reserved":0,"allocated":153006080,"data_stored":176847608,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706614,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":170224,"kb_used_data":112872,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94201616,"statfs":{"total":96636764160,"available":96462454784,"internally_reserved":0,"allocated":115580928,"data_stored":131406269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706613,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":170396,"kb_used_data":113044,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94201444,"statfs":{"total":96636764160,"available":96462278656,"internally_reserved":0,"allocated":115757056,"data_stored":137218999,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706613,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":165892,"kb_used_data":108540,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94205948,"statfs":{"total":96636764160,"available":96466890752,"internally_reserved":0,"allocated":111144960,"data_stored":130403790,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:38.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:38.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:38.333+0000 7f982d86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:38.341 INFO:tasks.rados.rados.0.smithi064.stdout:3641: done (6 left) 2024-03-04T06:23:38.341 INFO:tasks.rados.rados.0.smithi064.stdout:3642: done (5 left) 2024-03-04T06:23:38.341 INFO:tasks.rados.rados.0.smithi064.stdout:3643: done (4 left) 2024-03-04T06:23:38.341 INFO:tasks.rados.rados.0.smithi064.stdout:3645: done (3 left) 2024-03-04T06:23:38.342 INFO:tasks.rados.rados.0.smithi064.stdout:3646: done (2 left) 2024-03-04T06:23:38.342 INFO:tasks.rados.rados.0.smithi064.stdout:3647: done (1 left) 2024-03-04T06:23:38.342 INFO:tasks.rados.rados.0.smithi064.stdout:3648: done (0 left) 2024-03-04T06:23:38.342 INFO:tasks.rados.rados.0.smithi064.stdout:3649: snap_remove snap 438 2024-03-04T06:23:38.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:38.342+0000 7f982d86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:39.344 INFO:tasks.rados.rados.0.smithi064.stdout:3649: done (0 left) 2024-03-04T06:23:39.344 INFO:tasks.rados.rados.0.smithi064.stdout:3650: read oid 15 snap -1 2024-03-04T06:23:39.344 INFO:tasks.rados.rados.0.smithi064.stdout:3650: expect deleted 2024-03-04T06:23:39.344 INFO:tasks.rados.rados.0.smithi064.stdout:3651: snap_remove snap 447 2024-03-04T06:23:40.345 INFO:tasks.rados.rados.0.smithi064.stdout:3651: done (1 left) 2024-03-04T06:23:40.345 INFO:tasks.rados.rados.0.smithi064.stdout:3652: write oid 16 current snap is 472 2024-03-04T06:23:40.345 INFO:tasks.rados.rados.0.smithi064.stdout:3652: seq_num 1652 ranges {565164=759949,2054253=602237,3399490=365674} 2024-03-04T06:23:40.352 INFO:tasks.rados.rados.0.smithi064.stdout:3652: writing smithi06431743-16 from 565164 to 1325113 tid 1 2024-03-04T06:23:40.359 INFO:tasks.rados.rados.0.smithi064.stdout:3652: writing smithi06431743-16 from 2054253 to 2656490 tid 2 2024-03-04T06:23:40.365 INFO:tasks.rados.rados.0.smithi064.stdout:3652: writing smithi06431743-16 from 3399490 to 3765164 tid 3 2024-03-04T06:23:40.365 INFO:tasks.rados.rados.0.smithi064.stdout:3653: snap_remove snap 468 2024-03-04T06:23:40.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:40.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:40.552+0000 7fcb0855d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:40.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:40.556+0000 7fcb0855d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:41.350 INFO:tasks.rados.rados.0.smithi064.stdout:3653: done (2 left) 2024-03-04T06:23:41.350 INFO:tasks.rados.rados.0.smithi064.stdout:3654: write oid 33 current snap is 472 2024-03-04T06:23:41.350 INFO:tasks.rados.rados.0.smithi064.stdout:3654: seq_num 1653 ranges {581971=435198,1712735=462248,2626913=436049,3752375=29596} 2024-03-04T06:23:41.355 INFO:tasks.rados.rados.0.smithi064.stdout:3654: writing smithi06431743-33 from 581971 to 1017169 tid 1 2024-03-04T06:23:41.362 INFO:tasks.rados.rados.0.smithi064.stdout:3654: writing smithi06431743-33 from 1712735 to 2174983 tid 2 2024-03-04T06:23:41.367 INFO:tasks.rados.rados.0.smithi064.stdout:3654: writing smithi06431743-33 from 2626913 to 3062962 tid 3 2024-03-04T06:23:41.371 INFO:tasks.rados.rados.0.smithi064.stdout:3654: writing smithi06431743-33 from 3752375 to 3781971 tid 4 2024-03-04T06:23:41.371 INFO:tasks.rados.rados.0.smithi064.stdout:3655: write (excl) oid 39 current snap is 472 2024-03-04T06:23:41.371 INFO:tasks.rados.rados.0.smithi064.stdout:3655: seq_num 1654 ranges {598778=510447,1654864=605906,2937983=763674} 2024-03-04T06:23:41.377 INFO:tasks.rados.rados.0.smithi064.stdout:3655: writing smithi06431743-39 from 598778 to 1109225 tid 1 2024-03-04T06:23:41.385 INFO:tasks.rados.rados.0.smithi064.stdout:3655: writing smithi06431743-39 from 1654864 to 2260770 tid 2 2024-03-04T06:23:41.396 INFO:tasks.rados.rados.0.smithi064.stdout:3655: writing smithi06431743-39 from 2937983 to 3701657 tid 3 2024-03-04T06:23:41.396 INFO:tasks.rados.rados.0.smithi064.stdout:3656: read oid 42 snap -1 2024-03-04T06:23:41.396 INFO:tasks.rados.rados.0.smithi064.stdout:3656: expect (ObjNum 1601 snap 453 seq_num 1601) 2024-03-04T06:23:41.396 INFO:tasks.rados.rados.0.smithi064.stdout:3657: snap_create 2024-03-04T06:23:41.396 INFO:tasks.rados.rados.0.smithi064.stdout:3652: finishing write tid 1 to smithi06431743-16 2024-03-04T06:23:41.396 INFO:tasks.rados.rados.0.smithi064.stdout:3652: finishing write tid 2 to smithi06431743-16 2024-03-04T06:23:41.396 INFO:tasks.rados.rados.0.smithi064.stdout:3652: finishing write tid 3 to smithi06431743-16 2024-03-04T06:23:41.396 INFO:tasks.rados.rados.0.smithi064.stdout:3652: finishing write tid 4 to smithi06431743-16 2024-03-04T06:23:41.396 INFO:tasks.rados.rados.0.smithi064.stdout:3652: finishing write tid 5 to smithi06431743-16 2024-03-04T06:23:41.396 INFO:tasks.rados.rados.0.smithi064.stdout:3652: oid 16 updating version 0 to 1101 2024-03-04T06:23:41.397 INFO:tasks.rados.rados.0.smithi064.stdout:3652: oid 16 version 1101 is already newer than 1101 2024-03-04T06:23:41.397 INFO:tasks.rados.rados.0.smithi064.stdout:3652: oid 16 updating version 1101 to 1102 2024-03-04T06:23:41.397 INFO:tasks.rados.rados.0.smithi064.stdout:3652: oid 16 version 1102 is already newer than 1099 2024-03-04T06:23:41.397 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 1102 (ObjNum 1652 snap 472 seq_num 1652) dirty exists 2024-03-04T06:23:41.397 INFO:tasks.rados.rados.0.smithi064.stdout:3652: left oid 16 (ObjNum 1652 snap 472 seq_num 1652) 2024-03-04T06:23:41.476 INFO:tasks.rados.rados.0.smithi064.stdout:3656: expect (ObjNum 1601 snap 453 seq_num 1601) 2024-03-04T06:23:41.666 INFO:tasks.rados.rados.0.smithi064.stdout:3655: finishing write tid 1 to smithi06431743-39 2024-03-04T06:23:41.666 INFO:tasks.rados.rados.0.smithi064.stdout:3654: finishing write tid 1 to smithi06431743-33 2024-03-04T06:23:41.666 INFO:tasks.rados.rados.0.smithi064.stdout:3655: finishing write tid 2 to smithi06431743-39 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3654: finishing write tid 2 to smithi06431743-33 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3655: finishing write tid 3 to smithi06431743-39 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3654: finishing write tid 3 to smithi06431743-33 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3655: finishing write tid 4 to smithi06431743-39 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3655: finishing write tid 5 to smithi06431743-39 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3655: oid 39 updating version 0 to 1275 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3655: oid 39 updating version 1275 to 1278 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3655: oid 39 version 1278 is already newer than 1276 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3655: oid 39 version 1278 is already newer than 1277 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 1278 (ObjNum 1654 snap 472 seq_num 1654) dirty exists 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3655: left oid 39 (ObjNum 1654 snap 472 seq_num 1654) 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3654: finishing write tid 4 to smithi06431743-33 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3654: finishing write tid 5 to smithi06431743-33 2024-03-04T06:23:41.667 INFO:tasks.rados.rados.0.smithi064.stdout:3654: finishing write tid 6 to smithi06431743-33 2024-03-04T06:23:41.668 INFO:tasks.rados.rados.0.smithi064.stdout:3654: oid 33 updating version 0 to 1268 2024-03-04T06:23:41.668 INFO:tasks.rados.rados.0.smithi064.stdout:3654: oid 33 version 1268 is already newer than 1265 2024-03-04T06:23:41.668 INFO:tasks.rados.rados.0.smithi064.stdout:3654: oid 33 version 1268 is already newer than 1267 2024-03-04T06:23:41.668 INFO:tasks.rados.rados.0.smithi064.stdout:3654: oid 33 updating version 1268 to 1269 2024-03-04T06:23:41.668 INFO:tasks.rados.rados.0.smithi064.stdout:3654: oid 33 version 1269 is already newer than 1266 2024-03-04T06:23:41.668 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 1269 (ObjNum 1653 snap 472 seq_num 1653) dirty exists 2024-03-04T06:23:41.668 INFO:tasks.rados.rados.0.smithi064.stdout:3654: left oid 33 (ObjNum 1653 snap 472 seq_num 1653) 2024-03-04T06:23:42.353 INFO:tasks.rados.rados.0.smithi064.stdout:3650: done (5 left) 2024-03-04T06:23:42.353 INFO:tasks.rados.rados.0.smithi064.stdout:3652: done (4 left) 2024-03-04T06:23:42.353 INFO:tasks.rados.rados.0.smithi064.stdout:3654: done (3 left) 2024-03-04T06:23:42.354 INFO:tasks.rados.rados.0.smithi064.stdout:3655: done (2 left) 2024-03-04T06:23:42.354 INFO:tasks.rados.rados.0.smithi064.stdout:3656: done (1 left) 2024-03-04T06:23:42.354 INFO:tasks.rados.rados.0.smithi064.stdout:3657: done (0 left) 2024-03-04T06:23:42.354 INFO:tasks.rados.rados.0.smithi064.stdout:3658: read oid 44 snap -1 2024-03-04T06:23:42.354 INFO:tasks.rados.rados.0.smithi064.stdout:3658: expect (ObjNum 1648 snap 471 seq_num 1648) 2024-03-04T06:23:42.354 INFO:tasks.rados.rados.0.smithi064.stdout:3659: snap_remove snap 469 2024-03-04T06:23:42.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:42.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:42.729+0000 7fe7d5d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:42.733 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:42.733+0000 7fe7d5d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:43.227 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:23:43.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:23:43.340 INFO:tasks.rados.rados.0.smithi064.stdout:3659: done (1 left) 2024-03-04T06:23:43.341 INFO:tasks.rados.rados.0.smithi064.stdout:3660: snap_create 2024-03-04T06:23:43.341 INFO:tasks.rados.rados.0.smithi064.stdout:3658: expect (ObjNum 1648 snap 471 seq_num 1648) 2024-03-04T06:23:43.341 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:43.335+0000 7f663439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:43.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:43.343+0000 7f663439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:44.340 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:23:44.342 INFO:tasks.rados.rados.0.smithi064.stdout:3658: done (1 left) 2024-03-04T06:23:44.342 INFO:tasks.rados.rados.0.smithi064.stdout:3660: done (0 left) 2024-03-04T06:23:44.342 INFO:tasks.rados.rados.0.smithi064.stdout:3661: write (excl) oid 11 current snap is 474 2024-03-04T06:23:44.343 INFO:tasks.rados.rados.0.smithi064.stdout:3661: seq_num 1655 ranges {615585=585696,1713346=465917,2965406=574946} 2024-03-04T06:23:44.349 INFO:tasks.rados.rados.0.smithi064.stdout:3661: writing smithi06431743-11 from 615585 to 1201281 tid 1 2024-03-04T06:23:44.355 INFO:tasks.rados.rados.0.smithi064.stdout:3661: writing smithi06431743-11 from 1713346 to 2179263 tid 2 2024-03-04T06:23:44.357 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:23:44.364 INFO:tasks.rados.rados.0.smithi064.stdout:3661: writing smithi06431743-11 from 2965406 to 3540352 tid 3 2024-03-04T06:23:44.365 INFO:tasks.rados.rados.0.smithi064.stdout:3662: delete oid 42 current snap is 474 2024-03-04T06:23:44.426 INFO:tasks.rados.rados.0.smithi064.stdout:3661: finishing write tid 1 to smithi06431743-11 2024-03-04T06:23:44.451 INFO:tasks.rados.rados.0.smithi064.stdout:3661: finishing write tid 2 to smithi06431743-11 2024-03-04T06:23:44.457 INFO:tasks.rados.rados.0.smithi064.stdout:3661: finishing write tid 3 to smithi06431743-11 2024-03-04T06:23:44.475 INFO:tasks.rados.rados.0.smithi064.stdout:3661: finishing write tid 4 to smithi06431743-11 2024-03-04T06:23:44.475 INFO:tasks.rados.rados.0.smithi064.stdout:3661: finishing write tid 5 to smithi06431743-11 2024-03-04T06:23:44.475 INFO:tasks.rados.rados.0.smithi064.stdout:3661: oid 11 updating version 0 to 1281 2024-03-04T06:23:44.476 INFO:tasks.rados.rados.0.smithi064.stdout:3661: oid 11 version 1281 is already newer than 1279 2024-03-04T06:23:44.476 INFO:tasks.rados.rados.0.smithi064.stdout:3661: oid 11 updating version 1281 to 1282 2024-03-04T06:23:44.476 INFO:tasks.rados.rados.0.smithi064.stdout:3661: oid 11 version 1282 is already newer than 1281 2024-03-04T06:23:44.476 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 1282 (ObjNum 1655 snap 474 seq_num 1655) dirty exists 2024-03-04T06:23:44.476 INFO:tasks.rados.rados.0.smithi064.stdout:3661: left oid 11 (ObjNum 1655 snap 474 seq_num 1655) 2024-03-04T06:23:44.476 INFO:tasks.rados.rados.0.smithi064.stdout:3661: done (1 left) 2024-03-04T06:23:44.476 INFO:tasks.rados.rados.0.smithi064.stdout:3662: done (0 left) 2024-03-04T06:23:44.477 INFO:tasks.rados.rados.0.smithi064.stdout:3663: delete oid 33 current snap is 474 2024-03-04T06:23:44.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:44.491+0000 7f533746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:44.500 INFO:tasks.rados.rados.0.smithi064.stdout:3663: done (0 left) 2024-03-04T06:23:44.500 INFO:tasks.rados.rados.0.smithi064.stdout:3664: write oid 2 current snap is 474 2024-03-04T06:23:44.500 INFO:tasks.rados.rados.0.smithi064.stdout:3664: seq_num 1658 ranges {666006=527796,1888792=613244,3047675=525115} 2024-03-04T06:23:44.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:44.500+0000 7f533746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:44.506 INFO:tasks.rados.rados.0.smithi064.stdout:3664: writing smithi06431743-2 from 666006 to 1193802 tid 1 2024-03-04T06:23:44.514 INFO:tasks.rados.rados.0.smithi064.stdout:3664: writing smithi06431743-2 from 1888792 to 2502036 tid 2 2024-03-04T06:23:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:3664: writing smithi06431743-2 from 3047675 to 3572790 tid 3 2024-03-04T06:23:44.520 INFO:tasks.rados.rados.0.smithi064.stdout:3665: delete oid 15 current snap is 474 2024-03-04T06:23:44.524 INFO:tasks.rados.rados.0.smithi064.stdout:3665: done (1 left) 2024-03-04T06:23:44.524 INFO:tasks.rados.rados.0.smithi064.stdout:3666: read oid 1 snap -1 2024-03-04T06:23:44.524 INFO:tasks.rados.rados.0.smithi064.stdout:3666: expect (ObjNum 1640 snap 467 seq_num 1640) 2024-03-04T06:23:44.524 INFO:tasks.rados.rados.0.smithi064.stdout:3667: read oid 8 snap -1 2024-03-04T06:23:44.525 INFO:tasks.rados.rados.0.smithi064.stdout:3667: expect (ObjNum 1650 snap 471 seq_num 1650) 2024-03-04T06:23:44.525 INFO:tasks.rados.rados.0.smithi064.stdout:3668: read oid 25 snap -1 2024-03-04T06:23:44.525 INFO:tasks.rados.rados.0.smithi064.stdout:3668: expect (ObjNum 1621 snap 463 seq_num 1621) 2024-03-04T06:23:44.525 INFO:tasks.rados.rados.0.smithi064.stdout:3669: read oid 46 snap 472 2024-03-04T06:23:44.525 INFO:tasks.rados.rados.0.smithi064.stdout:3669: expect deleted 2024-03-04T06:23:44.525 INFO:tasks.rados.rados.0.smithi064.stdout:3670: snap_create 2024-03-04T06:23:44.542 INFO:tasks.rados.rados.0.smithi064.stdout:3668: expect (ObjNum 1621 snap 463 seq_num 1621) 2024-03-04T06:23:44.706 INFO:tasks.rados.rados.0.smithi064.stdout:3667: expect (ObjNum 1650 snap 471 seq_num 1650) 2024-03-04T06:23:44.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:44.843 INFO:tasks.rados.rados.0.smithi064.stdout:3664: finishing write tid 1 to smithi06431743-2 2024-03-04T06:23:44.843 INFO:tasks.rados.rados.0.smithi064.stdout:3664: finishing write tid 2 to smithi06431743-2 2024-03-04T06:23:44.843 INFO:tasks.rados.rados.0.smithi064.stdout:3666: expect (ObjNum 1640 snap 467 seq_num 1640) 2024-03-04T06:23:44.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:44.904+0000 7fb4eae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:44.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:44.908+0000 7fb4eae2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:45.025 INFO:tasks.rados.rados.0.smithi064.stdout:3664: finishing write tid 3 to smithi06431743-2 2024-03-04T06:23:45.025 INFO:tasks.rados.rados.0.smithi064.stdout:3664: finishing write tid 4 to smithi06431743-2 2024-03-04T06:23:45.025 INFO:tasks.rados.rados.0.smithi064.stdout:3664: finishing write tid 5 to smithi06431743-2 2024-03-04T06:23:45.025 INFO:tasks.rados.rados.0.smithi064.stdout:3664: oid 2 updating version 0 to 1012 2024-03-04T06:23:45.025 INFO:tasks.rados.rados.0.smithi064.stdout:3664: oid 2 updating version 1012 to 1015 2024-03-04T06:23:45.025 INFO:tasks.rados.rados.0.smithi064.stdout:3664: oid 2 version 1015 is already newer than 1013 2024-03-04T06:23:45.025 INFO:tasks.rados.rados.0.smithi064.stdout:3664: oid 2 version 1015 is already newer than 1015 2024-03-04T06:23:45.025 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 1015 (ObjNum 1658 snap 474 seq_num 1658) dirty exists 2024-03-04T06:23:45.025 INFO:tasks.rados.rados.0.smithi064.stdout:3664: left oid 2 (ObjNum 1658 snap 474 seq_num 1658) 2024-03-04T06:23:45.360 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:23:45.363 INFO:tasks.rados.rados.0.smithi064.stdout:3664: done (5 left) 2024-03-04T06:23:45.364 INFO:tasks.rados.rados.0.smithi064.stdout:3666: done (4 left) 2024-03-04T06:23:45.364 INFO:tasks.rados.rados.0.smithi064.stdout:3667: done (3 left) 2024-03-04T06:23:45.364 INFO:tasks.rados.rados.0.smithi064.stdout:3668: done (2 left) 2024-03-04T06:23:45.364 INFO:tasks.rados.rados.0.smithi064.stdout:3669: done (1 left) 2024-03-04T06:23:45.364 INFO:tasks.rados.rados.0.smithi064.stdout:3670: done (0 left) 2024-03-04T06:23:45.364 INFO:tasks.rados.rados.0.smithi064.stdout:3671: delete oid 23 current snap is 475 2024-03-04T06:23:45.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:23:45.414 INFO:tasks.rados.rados.0.smithi064.stdout:3671: done (0 left) 2024-03-04T06:23:45.414 INFO:tasks.rados.rados.0.smithi064.stdout:3672: snap_create 2024-03-04T06:23:45.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:45.481+0000 7f4d05507640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:45.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:45.489+0000 7f4d05507640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:46.360 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:23:46.363 INFO:tasks.rados.rados.0.smithi064.stdout:3672: done (0 left) 2024-03-04T06:23:46.363 INFO:tasks.rados.rados.0.smithi064.stdout:3673: snap_create 2024-03-04T06:23:46.374 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:23:46.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:46.518+0000 7feab4f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:46.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:46.526+0000 7feab4f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:46.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:47.082 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:47.080+0000 7fbd8bf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:47.084 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:47.084+0000 7fbd8bf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:47.366 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:23:47.368 INFO:tasks.rados.rados.0.smithi064.stdout:3673: done (0 left) 2024-03-04T06:23:47.368 INFO:tasks.rados.rados.0.smithi064.stdout:3674: read oid 2 snap 452 2024-03-04T06:23:47.369 INFO:tasks.rados.rados.0.smithi064.stdout:3674: expect (ObjNum 1584 snap 450 seq_num 1584) 2024-03-04T06:23:47.369 INFO:tasks.rados.rados.0.smithi064.stdout:3675: write oid 1 current snap is 477 2024-03-04T06:23:47.369 INFO:tasks.rados.rados.0.smithi064.stdout:3675: seq_num 1661 ranges {716427=753543,2064238=760571,3413591=502836} 2024-03-04T06:23:47.378 INFO:tasks.rados.rados.0.smithi064.stdout:3675: writing smithi06431743-1 from 716427 to 1469970 tid 1 2024-03-04T06:23:47.379 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:23:47.380 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:47.387 INFO:tasks.rados.rados.0.smithi064.stdout:3675: writing smithi06431743-1 from 2064238 to 2824809 tid 2 2024-03-04T06:23:47.395 INFO:tasks.rados.rados.0.smithi064.stdout:3675: writing smithi06431743-1 from 3413591 to 3916427 tid 3 2024-03-04T06:23:47.395 INFO:tasks.rados.rados.0.smithi064.stdout:3676: delete oid 12 current snap is 477 2024-03-04T06:23:47.433 INFO:tasks.rados.rados.0.smithi064.stdout:3674: expect (ObjNum 1584 snap 450 seq_num 1584) 2024-03-04T06:23:47.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:47.501+0000 7f166a710640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:47.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:47.507+0000 7f166a710640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:47.580 INFO:tasks.rados.rados.0.smithi064.stdout:3675: finishing write tid 1 to smithi06431743-1 2024-03-04T06:23:47.580 INFO:tasks.rados.rados.0.smithi064.stdout:3675: finishing write tid 2 to smithi06431743-1 2024-03-04T06:23:47.580 INFO:tasks.rados.rados.0.smithi064.stdout:3675: finishing write tid 3 to smithi06431743-1 2024-03-04T06:23:47.581 INFO:tasks.rados.rados.0.smithi064.stdout:3675: finishing write tid 4 to smithi06431743-1 2024-03-04T06:23:47.581 INFO:tasks.rados.rados.0.smithi064.stdout:3675: finishing write tid 5 to smithi06431743-1 2024-03-04T06:23:47.581 INFO:tasks.rados.rados.0.smithi064.stdout:3675: oid 1 updating version 0 to 1875 2024-03-04T06:23:47.581 INFO:tasks.rados.rados.0.smithi064.stdout:3675: oid 1 version 1875 is already newer than 1873 2024-03-04T06:23:47.581 INFO:tasks.rados.rados.0.smithi064.stdout:3675: oid 1 version 1875 is already newer than 1874 2024-03-04T06:23:47.581 INFO:tasks.rados.rados.0.smithi064.stdout:3675: oid 1 version 1875 is already newer than 1873 2024-03-04T06:23:47.581 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 1 v 1875 (ObjNum 1661 snap 477 seq_num 1661) dirty exists 2024-03-04T06:23:47.581 INFO:tasks.rados.rados.0.smithi064.stdout:3675: left oid 1 (ObjNum 1661 snap 477 seq_num 1661) 2024-03-04T06:23:47.581 INFO:tasks.rados.rados.0.smithi064.stdout:3674: done (2 left) 2024-03-04T06:23:47.581 INFO:tasks.rados.rados.0.smithi064.stdout:3675: done (1 left) 2024-03-04T06:23:47.581 INFO:tasks.rados.rados.0.smithi064.stdout:3676: done (0 left) 2024-03-04T06:23:47.582 INFO:tasks.rados.rados.0.smithi064.stdout:3677: read oid 36 snap -1 2024-03-04T06:23:47.582 INFO:tasks.rados.rados.0.smithi064.stdout:3677: expect deleted 2024-03-04T06:23:47.582 INFO:tasks.rados.rados.0.smithi064.stdout:3678: snap_remove snap 452 2024-03-04T06:23:47.699 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:47.699 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:47.711 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1428,"stamp":"2024-03-04T06:23:47.367283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":111025228,"num_objects":1222,"num_object_clones":1008,"num_object_copies":3666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":179,"num_read":6629,"num_read_kb":5189503,"num_write":5666,"num_write_kb":1513907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16638,"ondisk_log_size":16638,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":657796,"kb_used_data":414500,"kb_used_omap":0,"kb_used_meta":243264,"kb_avail":376829564,"statfs":{"total":386547056640,"available":385873473536,"internally_reserved":0,"allocated":424448000,"data_stored":505485455,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":249102336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4202509,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":38,"num_read_kb":39251,"num_write":29,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027457"},"pg_stats":[{"pgid":"3.f","version":"949'909","reported_seq":3245,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.429418+0000","last_change":"2024-03-04T06:23:44.398415+0000","last_active":"2024-03-04T06:23:44.429418+0000","last_peered":"2024-03-04T06:23:44.429418+0000","last_clean":"2024-03-04T06:23:44.429418+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:44.429418+0000","last_undegraded":"2024-03-04T06:23:44.429418+0000","last_fullsized":"2024-03-04T06:23:44.429418+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294568,"num_objects":77,"num_object_clones":62,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":362,"num_read_kb":286556,"num_write":323,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3239,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.398954+0000","last_change":"2024-03-04T06:23:43.361299+0000","last_active":"2024-03-04T06:23:44.398954+0000","last_peered":"2024-03-04T06:23:44.398954+0000","last_clean":"2024-03-04T06:23:44.398954+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:44.398954+0000","last_undegraded":"2024-03-04T06:23:44.398954+0000","last_fullsized":"2024-03-04T06:23:44.398954+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"953'1282","reported_seq":3458,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.375347+0000","last_change":"2024-03-04T06:23:44.380147+0000","last_active":"2024-03-04T06:23:45.375347+0000","last_peered":"2024-03-04T06:23:45.375347+0000","last_clean":"2024-03-04T06:23:45.375347+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:45.375347+0000","last_undegraded":"2024-03-04T06:23:45.375347+0000","last_fullsized":"2024-03-04T06:23:45.375347+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13716096,"num_objects":96,"num_object_clones":81,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":573,"num_read_kb":460192,"num_write":442,"num_write_kb":121345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"953'1015","reported_seq":3220,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.576674+0000","last_change":"2024-03-04T06:23:44.387390+0000","last_active":"2024-03-04T06:23:44.576674+0000","last_peered":"2024-03-04T06:23:44.576674+0000","last_clean":"2024-03-04T06:23:44.576674+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:44.576674+0000","last_undegraded":"2024-03-04T06:23:44.576674+0000","last_fullsized":"2024-03-04T06:23:44.576674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1015,"log_dups_size":0,"ondisk_log_size":1015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7093313,"num_objects":66,"num_object_clones":56,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":322,"num_read_kb":245413,"num_write":314,"num_write_kb":87346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"949'415","reported_seq":3028,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.386301+0000","last_change":"2024-03-04T06:23:44.401552+0000","last_active":"2024-03-04T06:23:46.386301+0000","last_peered":"2024-03-04T06:23:46.386301+0000","last_clean":"2024-03-04T06:23:46.386301+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:46.386301+0000","last_undegraded":"2024-03-04T06:23:46.386301+0000","last_fullsized":"2024-03-04T06:23:46.386301+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3748357,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":122,"num_read_kb":93148,"num_write":125,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"952'1014","reported_seq":3311,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.429260+0000","last_change":"2024-03-04T06:23:44.429260+0000","last_active":"2024-03-04T06:23:44.429260+0000","last_peered":"2024-03-04T06:23:44.429260+0000","last_clean":"2024-03-04T06:23:44.429260+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:44.429260+0000","last_undegraded":"2024-03-04T06:23:44.429260+0000","last_fullsized":"2024-03-04T06:23:44.429260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1014,"log_dups_size":0,"ondisk_log_size":1014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10152616,"num_objects":74,"num_object_clones":62,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":435,"num_read_kb":348132,"num_write":358,"num_write_kb":103998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2793,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.424384+0000","last_change":"2024-03-04T06:23:44.398237+0000","last_active":"2024-03-04T06:23:44.424384+0000","last_peered":"2024-03-04T06:23:44.424384+0000","last_clean":"2024-03-04T06:23:44.424384+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:44.424384+0000","last_undegraded":"2024-03-04T06:23:44.424384+0000","last_fullsized":"2024-03-04T06:23:44.424384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"950'660","reported_seq":3104,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.373600+0000","last_change":"2024-03-04T06:23:44.381630+0000","last_active":"2024-03-04T06:23:45.373600+0000","last_peered":"2024-03-04T06:23:45.373600+0000","last_clean":"2024-03-04T06:23:45.373600+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:45.373600+0000","last_undegraded":"2024-03-04T06:23:45.373600+0000","last_fullsized":"2024-03-04T06:23:45.373600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2546,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.380176+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:46.380176+0000","last_peered":"2024-03-04T06:23:46.380176+0000","last_clean":"2024-03-04T06:23:46.380176+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:46.380176+0000","last_undegraded":"2024-03-04T06:23:46.380176+0000","last_fullsized":"2024-03-04T06:23:46.380176+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2358,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.381934+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:45.381934+0000","last_peered":"2024-03-04T06:23:45.381934+0000","last_clean":"2024-03-04T06:23:45.381934+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:45.381934+0000","last_undegraded":"2024-03-04T06:23:45.381934+0000","last_fullsized":"2024-03-04T06:23:45.381934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"906'951","reported_seq":3299,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.382106+0000","last_change":"2024-03-04T06:23:44.383722+0000","last_active":"2024-03-04T06:23:45.382106+0000","last_peered":"2024-03-04T06:23:45.382106+0000","last_clean":"2024-03-04T06:23:45.382106+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:45.382106+0000","last_undegraded":"2024-03-04T06:23:45.382106+0000","last_fullsized":"2024-03-04T06:23:45.382106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698299,"num_objects":72,"num_object_clones":58,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":441,"num_read_kb":363862,"num_write":322,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2343,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.441005+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:44.441005+0000","last_peered":"2024-03-04T06:23:44.441005+0000","last_clean":"2024-03-04T06:23:44.441005+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:44.441005+0000","last_undegraded":"2024-03-04T06:23:44.441005+0000","last_fullsized":"2024-03-04T06:23:44.441005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"952'671","reported_seq":3206,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.379185+0000","last_change":"2024-03-04T06:23:44.431429+0000","last_active":"2024-03-04T06:23:46.379185+0000","last_peered":"2024-03-04T06:23:46.379185+0000","last_clean":"2024-03-04T06:23:46.379185+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:46.379185+0000","last_undegraded":"2024-03-04T06:23:46.379185+0000","last_fullsized":"2024-03-04T06:23:46.379185+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210995,"num_objects":52,"num_object_clones":43,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":312,"num_read_kb":250024,"num_write":224,"num_write_kb":64009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2358,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.369991+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:45.369991+0000","last_peered":"2024-03-04T06:23:45.369991+0000","last_clean":"2024-03-04T06:23:45.369991+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:45.369991+0000","last_undegraded":"2024-03-04T06:23:45.369991+0000","last_fullsized":"2024-03-04T06:23:45.369991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"952'1870","reported_seq":3821,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.388552+0000","last_change":"2024-03-04T06:23:44.434727+0000","last_active":"2024-03-04T06:23:46.388552+0000","last_peered":"2024-03-04T06:23:46.388552+0000","last_clean":"2024-03-04T06:23:46.388552+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:46.388552+0000","last_undegraded":"2024-03-04T06:23:46.388552+0000","last_fullsized":"2024-03-04T06:23:46.388552+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1870,"log_dups_size":0,"ondisk_log_size":1870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12959781,"num_objects":144,"num_object_clones":118,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":797,"num_read_kb":627448,"num_write":669,"num_write_kb":179858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2343,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.423505+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:44.423505+0000","last_peered":"2024-03-04T06:23:44.423505+0000","last_clean":"2024-03-04T06:23:44.423505+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:44.423505+0000","last_undegraded":"2024-03-04T06:23:44.423505+0000","last_fullsized":"2024-03-04T06:23:44.423505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"952'2205","reported_seq":3858,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.401718+0000","last_change":"2024-03-04T06:23:43.391976+0000","last_active":"2024-03-04T06:23:44.401718+0000","last_peered":"2024-03-04T06:23:44.401718+0000","last_clean":"2024-03-04T06:23:44.401718+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:44.401718+0000","last_undegraded":"2024-03-04T06:23:44.401718+0000","last_fullsized":"2024-03-04T06:23:44.401718+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2205,"log_dups_size":0,"ondisk_log_size":2205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13262307,"num_objects":158,"num_object_clones":134,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":724,"num_read_kb":536055,"num_write":722,"num_write_kb":195272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"953'934","reported_seq":3295,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.380025+0000","last_change":"2024-03-04T06:23:44.421745+0000","last_active":"2024-03-04T06:23:45.380025+0000","last_peered":"2024-03-04T06:23:45.380025+0000","last_clean":"2024-03-04T06:23:45.380025+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:45.380025+0000","last_undegraded":"2024-03-04T06:23:45.380025+0000","last_fullsized":"2024-03-04T06:23:45.380025+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865973,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":438,"num_read_kb":364480,"num_write":318,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2449,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.412376+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:44.412376+0000","last_peered":"2024-03-04T06:23:44.412376+0000","last_clean":"2024-03-04T06:23:44.412376+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:44.412376+0000","last_undegraded":"2024-03-04T06:23:44.412376+0000","last_fullsized":"2024-03-04T06:23:44.412376+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"949'1102","reported_seq":3393,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.539284+0000","last_change":"2024-03-04T06:23:43.371182+0000","last_active":"2024-03-04T06:23:44.539284+0000","last_peered":"2024-03-04T06:23:44.539284+0000","last_clean":"2024-03-04T06:23:44.539284+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:44.539284+0000","last_undegraded":"2024-03-04T06:23:44.539284+0000","last_fullsized":"2024-03-04T06:23:44.539284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1102,"log_dups_size":0,"ondisk_log_size":1102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7496714,"num_objects":83,"num_object_clones":67,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":390,"num_read_kb":282670,"num_write":362,"num_write_kb":98038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2413,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.388555+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:46.388555+0000","last_peered":"2024-03-04T06:23:46.388555+0000","last_clean":"2024-03-04T06:23:46.388555+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:46.388555+0000","last_undegraded":"2024-03-04T06:23:46.388555+0000","last_fullsized":"2024-03-04T06:23:46.388555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"951'1445","reported_seq":3496,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.378464+0000","last_change":"2024-03-04T06:23:44.390581+0000","last_active":"2024-03-04T06:23:46.378464+0000","last_peered":"2024-03-04T06:23:46.378464+0000","last_clean":"2024-03-04T06:23:46.378464+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:46.378464+0000","last_undegraded":"2024-03-04T06:23:46.378464+0000","last_fullsized":"2024-03-04T06:23:46.378464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6774013,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":496,"num_read_kb":386812,"num_write":462,"num_write_kb":128837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2449,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.396390+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:44.396390+0000","last_peered":"2024-03-04T06:23:44.396390+0000","last_clean":"2024-03-04T06:23:44.396390+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:44.396390+0000","last_undegraded":"2024-03-04T06:23:44.396390+0000","last_fullsized":"2024-03-04T06:23:44.396390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"953'1279","reported_seq":3565,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.479912+0000","last_change":"2024-03-04T06:23:43.393113+0000","last_active":"2024-03-04T06:23:44.479912+0000","last_peered":"2024-03-04T06:23:44.479912+0000","last_clean":"2024-03-04T06:23:44.479912+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:44.479912+0000","last_undegraded":"2024-03-04T06:23:44.479912+0000","last_fullsized":"2024-03-04T06:23:44.479912+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1279,"log_dups_size":0,"ondisk_log_size":1279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269803,"num_objects":100,"num_object_clones":86,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":585,"num_read_kb":494165,"num_write":447,"num_write_kb":122503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2358,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.381831+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:45.381831+0000","last_peered":"2024-03-04T06:23:45.381831+0000","last_clean":"2024-03-04T06:23:45.381831+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:45.381831+0000","last_undegraded":"2024-03-04T06:23:45.381831+0000","last_fullsized":"2024-03-04T06:23:45.381831+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":110434860,"num_objects":1220,"num_object_clones":1008,"num_object_copies":3660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":179,"num_read":6526,"num_read_kb":5189412,"num_write":5529,"num_write_kb":1512013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16541,"ondisk_log_size":16541,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706624,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":197504,"kb_used_data":126264,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94174336,"statfs":{"total":96636764160,"available":96434520064,"internally_reserved":0,"allocated":129294336,"data_stored":152290758,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706624,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":151276,"kb_used_data":93924,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94220564,"statfs":{"total":96636764160,"available":96481857536,"internally_reserved":0,"allocated":96178176,"data_stored":112728420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706623,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":155676,"kb_used_data":98324,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94216164,"statfs":{"total":96636764160,"available":96477351936,"internally_reserved":0,"allocated":100683776,"data_stored":122432189,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706623,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153340,"kb_used_data":95988,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94218500,"statfs":{"total":96636764160,"available":96479744000,"internally_reserved":0,"allocated":98291712,"data_stored":118034088,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:47.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:47.859 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:47.854+0000 7f9b5a06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:47.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:47.860+0000 7f9b5a06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:48.052 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:48.052 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:48.064 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1428,"stamp":"2024-03-04T06:23:47.367283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":111025228,"num_objects":1222,"num_object_clones":1008,"num_object_copies":3666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":179,"num_read":6629,"num_read_kb":5189503,"num_write":5666,"num_write_kb":1513907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16638,"ondisk_log_size":16638,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":657796,"kb_used_data":414500,"kb_used_omap":0,"kb_used_meta":243264,"kb_avail":376829564,"statfs":{"total":386547056640,"available":385873473536,"internally_reserved":0,"allocated":424448000,"data_stored":505485455,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":249102336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4202509,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":38,"num_read_kb":39251,"num_write":29,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027457"},"pg_stats":[{"pgid":"3.f","version":"949'909","reported_seq":3245,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.429418+0000","last_change":"2024-03-04T06:23:44.398415+0000","last_active":"2024-03-04T06:23:44.429418+0000","last_peered":"2024-03-04T06:23:44.429418+0000","last_clean":"2024-03-04T06:23:44.429418+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:44.429418+0000","last_undegraded":"2024-03-04T06:23:44.429418+0000","last_fullsized":"2024-03-04T06:23:44.429418+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294568,"num_objects":77,"num_object_clones":62,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":362,"num_read_kb":286556,"num_write":323,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3239,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.398954+0000","last_change":"2024-03-04T06:23:43.361299+0000","last_active":"2024-03-04T06:23:44.398954+0000","last_peered":"2024-03-04T06:23:44.398954+0000","last_clean":"2024-03-04T06:23:44.398954+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:44.398954+0000","last_undegraded":"2024-03-04T06:23:44.398954+0000","last_fullsized":"2024-03-04T06:23:44.398954+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"953'1282","reported_seq":3458,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.375347+0000","last_change":"2024-03-04T06:23:44.380147+0000","last_active":"2024-03-04T06:23:45.375347+0000","last_peered":"2024-03-04T06:23:45.375347+0000","last_clean":"2024-03-04T06:23:45.375347+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:45.375347+0000","last_undegraded":"2024-03-04T06:23:45.375347+0000","last_fullsized":"2024-03-04T06:23:45.375347+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13716096,"num_objects":96,"num_object_clones":81,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":573,"num_read_kb":460192,"num_write":442,"num_write_kb":121345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"953'1015","reported_seq":3220,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.576674+0000","last_change":"2024-03-04T06:23:44.387390+0000","last_active":"2024-03-04T06:23:44.576674+0000","last_peered":"2024-03-04T06:23:44.576674+0000","last_clean":"2024-03-04T06:23:44.576674+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:44.576674+0000","last_undegraded":"2024-03-04T06:23:44.576674+0000","last_fullsized":"2024-03-04T06:23:44.576674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1015,"log_dups_size":0,"ondisk_log_size":1015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7093313,"num_objects":66,"num_object_clones":56,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":322,"num_read_kb":245413,"num_write":314,"num_write_kb":87346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"949'415","reported_seq":3028,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.386301+0000","last_change":"2024-03-04T06:23:44.401552+0000","last_active":"2024-03-04T06:23:46.386301+0000","last_peered":"2024-03-04T06:23:46.386301+0000","last_clean":"2024-03-04T06:23:46.386301+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:46.386301+0000","last_undegraded":"2024-03-04T06:23:46.386301+0000","last_fullsized":"2024-03-04T06:23:46.386301+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3748357,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":122,"num_read_kb":93148,"num_write":125,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"952'1014","reported_seq":3311,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.429260+0000","last_change":"2024-03-04T06:23:44.429260+0000","last_active":"2024-03-04T06:23:44.429260+0000","last_peered":"2024-03-04T06:23:44.429260+0000","last_clean":"2024-03-04T06:23:44.429260+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:44.429260+0000","last_undegraded":"2024-03-04T06:23:44.429260+0000","last_fullsized":"2024-03-04T06:23:44.429260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1014,"log_dups_size":0,"ondisk_log_size":1014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10152616,"num_objects":74,"num_object_clones":62,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":435,"num_read_kb":348132,"num_write":358,"num_write_kb":103998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2793,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.424384+0000","last_change":"2024-03-04T06:23:44.398237+0000","last_active":"2024-03-04T06:23:44.424384+0000","last_peered":"2024-03-04T06:23:44.424384+0000","last_clean":"2024-03-04T06:23:44.424384+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:44.424384+0000","last_undegraded":"2024-03-04T06:23:44.424384+0000","last_fullsized":"2024-03-04T06:23:44.424384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"950'660","reported_seq":3104,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.373600+0000","last_change":"2024-03-04T06:23:44.381630+0000","last_active":"2024-03-04T06:23:45.373600+0000","last_peered":"2024-03-04T06:23:45.373600+0000","last_clean":"2024-03-04T06:23:45.373600+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:45.373600+0000","last_undegraded":"2024-03-04T06:23:45.373600+0000","last_fullsized":"2024-03-04T06:23:45.373600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2546,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.380176+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:46.380176+0000","last_peered":"2024-03-04T06:23:46.380176+0000","last_clean":"2024-03-04T06:23:46.380176+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:46.380176+0000","last_undegraded":"2024-03-04T06:23:46.380176+0000","last_fullsized":"2024-03-04T06:23:46.380176+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2358,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.381934+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:45.381934+0000","last_peered":"2024-03-04T06:23:45.381934+0000","last_clean":"2024-03-04T06:23:45.381934+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:45.381934+0000","last_undegraded":"2024-03-04T06:23:45.381934+0000","last_fullsized":"2024-03-04T06:23:45.381934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"906'951","reported_seq":3299,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.382106+0000","last_change":"2024-03-04T06:23:44.383722+0000","last_active":"2024-03-04T06:23:45.382106+0000","last_peered":"2024-03-04T06:23:45.382106+0000","last_clean":"2024-03-04T06:23:45.382106+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:45.382106+0000","last_undegraded":"2024-03-04T06:23:45.382106+0000","last_fullsized":"2024-03-04T06:23:45.382106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698299,"num_objects":72,"num_object_clones":58,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":441,"num_read_kb":363862,"num_write":322,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2343,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.441005+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:44.441005+0000","last_peered":"2024-03-04T06:23:44.441005+0000","last_clean":"2024-03-04T06:23:44.441005+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:44.441005+0000","last_undegraded":"2024-03-04T06:23:44.441005+0000","last_fullsized":"2024-03-04T06:23:44.441005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"952'671","reported_seq":3206,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.379185+0000","last_change":"2024-03-04T06:23:44.431429+0000","last_active":"2024-03-04T06:23:46.379185+0000","last_peered":"2024-03-04T06:23:46.379185+0000","last_clean":"2024-03-04T06:23:46.379185+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:46.379185+0000","last_undegraded":"2024-03-04T06:23:46.379185+0000","last_fullsized":"2024-03-04T06:23:46.379185+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210995,"num_objects":52,"num_object_clones":43,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":312,"num_read_kb":250024,"num_write":224,"num_write_kb":64009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2358,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.369991+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:45.369991+0000","last_peered":"2024-03-04T06:23:45.369991+0000","last_clean":"2024-03-04T06:23:45.369991+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:45.369991+0000","last_undegraded":"2024-03-04T06:23:45.369991+0000","last_fullsized":"2024-03-04T06:23:45.369991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"952'1870","reported_seq":3821,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.388552+0000","last_change":"2024-03-04T06:23:44.434727+0000","last_active":"2024-03-04T06:23:46.388552+0000","last_peered":"2024-03-04T06:23:46.388552+0000","last_clean":"2024-03-04T06:23:46.388552+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:46.388552+0000","last_undegraded":"2024-03-04T06:23:46.388552+0000","last_fullsized":"2024-03-04T06:23:46.388552+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1870,"log_dups_size":0,"ondisk_log_size":1870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12959781,"num_objects":144,"num_object_clones":118,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":797,"num_read_kb":627448,"num_write":669,"num_write_kb":179858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2343,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.423505+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:44.423505+0000","last_peered":"2024-03-04T06:23:44.423505+0000","last_clean":"2024-03-04T06:23:44.423505+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:44.423505+0000","last_undegraded":"2024-03-04T06:23:44.423505+0000","last_fullsized":"2024-03-04T06:23:44.423505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"952'2205","reported_seq":3858,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.401718+0000","last_change":"2024-03-04T06:23:43.391976+0000","last_active":"2024-03-04T06:23:44.401718+0000","last_peered":"2024-03-04T06:23:44.401718+0000","last_clean":"2024-03-04T06:23:44.401718+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:44.401718+0000","last_undegraded":"2024-03-04T06:23:44.401718+0000","last_fullsized":"2024-03-04T06:23:44.401718+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2205,"log_dups_size":0,"ondisk_log_size":2205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13262307,"num_objects":158,"num_object_clones":134,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":724,"num_read_kb":536055,"num_write":722,"num_write_kb":195272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"953'934","reported_seq":3295,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.380025+0000","last_change":"2024-03-04T06:23:44.421745+0000","last_active":"2024-03-04T06:23:45.380025+0000","last_peered":"2024-03-04T06:23:45.380025+0000","last_clean":"2024-03-04T06:23:45.380025+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:45.380025+0000","last_undegraded":"2024-03-04T06:23:45.380025+0000","last_fullsized":"2024-03-04T06:23:45.380025+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865973,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":438,"num_read_kb":364480,"num_write":318,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2449,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.412376+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:44.412376+0000","last_peered":"2024-03-04T06:23:44.412376+0000","last_clean":"2024-03-04T06:23:44.412376+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:44.412376+0000","last_undegraded":"2024-03-04T06:23:44.412376+0000","last_fullsized":"2024-03-04T06:23:44.412376+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"949'1102","reported_seq":3393,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.539284+0000","last_change":"2024-03-04T06:23:43.371182+0000","last_active":"2024-03-04T06:23:44.539284+0000","last_peered":"2024-03-04T06:23:44.539284+0000","last_clean":"2024-03-04T06:23:44.539284+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:44.539284+0000","last_undegraded":"2024-03-04T06:23:44.539284+0000","last_fullsized":"2024-03-04T06:23:44.539284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1102,"log_dups_size":0,"ondisk_log_size":1102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7496714,"num_objects":83,"num_object_clones":67,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":390,"num_read_kb":282670,"num_write":362,"num_write_kb":98038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2413,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.388555+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:46.388555+0000","last_peered":"2024-03-04T06:23:46.388555+0000","last_clean":"2024-03-04T06:23:46.388555+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:46.388555+0000","last_undegraded":"2024-03-04T06:23:46.388555+0000","last_fullsized":"2024-03-04T06:23:46.388555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"951'1445","reported_seq":3496,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.378464+0000","last_change":"2024-03-04T06:23:44.390581+0000","last_active":"2024-03-04T06:23:46.378464+0000","last_peered":"2024-03-04T06:23:46.378464+0000","last_clean":"2024-03-04T06:23:46.378464+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:46.378464+0000","last_undegraded":"2024-03-04T06:23:46.378464+0000","last_fullsized":"2024-03-04T06:23:46.378464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6774013,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":496,"num_read_kb":386812,"num_write":462,"num_write_kb":128837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2449,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.396390+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:44.396390+0000","last_peered":"2024-03-04T06:23:44.396390+0000","last_clean":"2024-03-04T06:23:44.396390+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:44.396390+0000","last_undegraded":"2024-03-04T06:23:44.396390+0000","last_fullsized":"2024-03-04T06:23:44.396390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"953'1279","reported_seq":3565,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.479912+0000","last_change":"2024-03-04T06:23:43.393113+0000","last_active":"2024-03-04T06:23:44.479912+0000","last_peered":"2024-03-04T06:23:44.479912+0000","last_clean":"2024-03-04T06:23:44.479912+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:44.479912+0000","last_undegraded":"2024-03-04T06:23:44.479912+0000","last_fullsized":"2024-03-04T06:23:44.479912+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1279,"log_dups_size":0,"ondisk_log_size":1279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269803,"num_objects":100,"num_object_clones":86,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":585,"num_read_kb":494165,"num_write":447,"num_write_kb":122503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2358,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.381831+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:45.381831+0000","last_peered":"2024-03-04T06:23:45.381831+0000","last_clean":"2024-03-04T06:23:45.381831+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:45.381831+0000","last_undegraded":"2024-03-04T06:23:45.381831+0000","last_fullsized":"2024-03-04T06:23:45.381831+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":110434860,"num_objects":1220,"num_object_clones":1008,"num_object_copies":3660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":179,"num_read":6526,"num_read_kb":5189412,"num_write":5529,"num_write_kb":1512013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16541,"ondisk_log_size":16541,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706624,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":197504,"kb_used_data":126264,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94174336,"statfs":{"total":96636764160,"available":96434520064,"internally_reserved":0,"allocated":129294336,"data_stored":152290758,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706624,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":151276,"kb_used_data":93924,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94220564,"statfs":{"total":96636764160,"available":96481857536,"internally_reserved":0,"allocated":96178176,"data_stored":112728420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706623,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":155676,"kb_used_data":98324,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94216164,"statfs":{"total":96636764160,"available":96477351936,"internally_reserved":0,"allocated":100683776,"data_stored":122432189,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706623,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153340,"kb_used_data":95988,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94218500,"statfs":{"total":96636764160,"available":96479744000,"internally_reserved":0,"allocated":98291712,"data_stored":118034088,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:48.065 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:23:48.065 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:23:48.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:48.173 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:48.168+0000 7fd8af99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:48.176 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:48.175+0000 7fd8af99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:48.367 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:48.367 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:48.371 INFO:tasks.rados.rados.0.smithi064.stdout:3678: done (1 left) 2024-03-04T06:23:48.371 INFO:tasks.rados.rados.0.smithi064.stdout:3679: delete oid 23 current snap is 477 2024-03-04T06:23:48.380 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1428,"stamp":"2024-03-04T06:23:47.367283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":111025228,"num_objects":1222,"num_object_clones":1008,"num_object_copies":3666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":179,"num_read":6629,"num_read_kb":5189503,"num_write":5666,"num_write_kb":1513907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16638,"ondisk_log_size":16638,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":657796,"kb_used_data":414500,"kb_used_omap":0,"kb_used_meta":243264,"kb_avail":376829564,"statfs":{"total":386547056640,"available":385873473536,"internally_reserved":0,"allocated":424448000,"data_stored":505485455,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":249102336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4202509,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":38,"num_read_kb":39251,"num_write":29,"num_write_kb":8076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027457"},"pg_stats":[{"pgid":"3.f","version":"949'909","reported_seq":3245,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.429418+0000","last_change":"2024-03-04T06:23:44.398415+0000","last_active":"2024-03-04T06:23:44.429418+0000","last_peered":"2024-03-04T06:23:44.429418+0000","last_clean":"2024-03-04T06:23:44.429418+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:44.429418+0000","last_undegraded":"2024-03-04T06:23:44.429418+0000","last_fullsized":"2024-03-04T06:23:44.429418+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294568,"num_objects":77,"num_object_clones":62,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":362,"num_read_kb":286556,"num_write":323,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3239,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.398954+0000","last_change":"2024-03-04T06:23:43.361299+0000","last_active":"2024-03-04T06:23:44.398954+0000","last_peered":"2024-03-04T06:23:44.398954+0000","last_clean":"2024-03-04T06:23:44.398954+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:44.398954+0000","last_undegraded":"2024-03-04T06:23:44.398954+0000","last_fullsized":"2024-03-04T06:23:44.398954+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"953'1282","reported_seq":3458,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.375347+0000","last_change":"2024-03-04T06:23:44.380147+0000","last_active":"2024-03-04T06:23:45.375347+0000","last_peered":"2024-03-04T06:23:45.375347+0000","last_clean":"2024-03-04T06:23:45.375347+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:45.375347+0000","last_undegraded":"2024-03-04T06:23:45.375347+0000","last_fullsized":"2024-03-04T06:23:45.375347+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13716096,"num_objects":96,"num_object_clones":81,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":573,"num_read_kb":460192,"num_write":442,"num_write_kb":121345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"953'1015","reported_seq":3220,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.576674+0000","last_change":"2024-03-04T06:23:44.387390+0000","last_active":"2024-03-04T06:23:44.576674+0000","last_peered":"2024-03-04T06:23:44.576674+0000","last_clean":"2024-03-04T06:23:44.576674+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:44.576674+0000","last_undegraded":"2024-03-04T06:23:44.576674+0000","last_fullsized":"2024-03-04T06:23:44.576674+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1015,"log_dups_size":0,"ondisk_log_size":1015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7093313,"num_objects":66,"num_object_clones":56,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":322,"num_read_kb":245413,"num_write":314,"num_write_kb":87346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"949'415","reported_seq":3028,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.386301+0000","last_change":"2024-03-04T06:23:44.401552+0000","last_active":"2024-03-04T06:23:46.386301+0000","last_peered":"2024-03-04T06:23:46.386301+0000","last_clean":"2024-03-04T06:23:46.386301+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:46.386301+0000","last_undegraded":"2024-03-04T06:23:46.386301+0000","last_fullsized":"2024-03-04T06:23:46.386301+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3748357,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":122,"num_read_kb":93148,"num_write":125,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"952'1014","reported_seq":3311,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.429260+0000","last_change":"2024-03-04T06:23:44.429260+0000","last_active":"2024-03-04T06:23:44.429260+0000","last_peered":"2024-03-04T06:23:44.429260+0000","last_clean":"2024-03-04T06:23:44.429260+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:44.429260+0000","last_undegraded":"2024-03-04T06:23:44.429260+0000","last_fullsized":"2024-03-04T06:23:44.429260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1014,"log_dups_size":0,"ondisk_log_size":1014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10152616,"num_objects":74,"num_object_clones":62,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":435,"num_read_kb":348132,"num_write":358,"num_write_kb":103998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2793,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.424384+0000","last_change":"2024-03-04T06:23:44.398237+0000","last_active":"2024-03-04T06:23:44.424384+0000","last_peered":"2024-03-04T06:23:44.424384+0000","last_clean":"2024-03-04T06:23:44.424384+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:44.424384+0000","last_undegraded":"2024-03-04T06:23:44.424384+0000","last_fullsized":"2024-03-04T06:23:44.424384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"950'660","reported_seq":3104,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.373600+0000","last_change":"2024-03-04T06:23:44.381630+0000","last_active":"2024-03-04T06:23:45.373600+0000","last_peered":"2024-03-04T06:23:45.373600+0000","last_clean":"2024-03-04T06:23:45.373600+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:45.373600+0000","last_undegraded":"2024-03-04T06:23:45.373600+0000","last_fullsized":"2024-03-04T06:23:45.373600+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2546,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.380176+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:46.380176+0000","last_peered":"2024-03-04T06:23:46.380176+0000","last_clean":"2024-03-04T06:23:46.380176+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:46.380176+0000","last_undegraded":"2024-03-04T06:23:46.380176+0000","last_fullsized":"2024-03-04T06:23:46.380176+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2358,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.381934+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:45.381934+0000","last_peered":"2024-03-04T06:23:45.381934+0000","last_clean":"2024-03-04T06:23:45.381934+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:45.381934+0000","last_undegraded":"2024-03-04T06:23:45.381934+0000","last_fullsized":"2024-03-04T06:23:45.381934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"906'951","reported_seq":3299,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.382106+0000","last_change":"2024-03-04T06:23:44.383722+0000","last_active":"2024-03-04T06:23:45.382106+0000","last_peered":"2024-03-04T06:23:45.382106+0000","last_clean":"2024-03-04T06:23:45.382106+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:45.382106+0000","last_undegraded":"2024-03-04T06:23:45.382106+0000","last_fullsized":"2024-03-04T06:23:45.382106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":951,"log_dups_size":0,"ondisk_log_size":951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698299,"num_objects":72,"num_object_clones":58,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":441,"num_read_kb":363862,"num_write":322,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2343,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.441005+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:44.441005+0000","last_peered":"2024-03-04T06:23:44.441005+0000","last_clean":"2024-03-04T06:23:44.441005+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:44.441005+0000","last_undegraded":"2024-03-04T06:23:44.441005+0000","last_fullsized":"2024-03-04T06:23:44.441005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"952'671","reported_seq":3206,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.379185+0000","last_change":"2024-03-04T06:23:44.431429+0000","last_active":"2024-03-04T06:23:46.379185+0000","last_peered":"2024-03-04T06:23:46.379185+0000","last_clean":"2024-03-04T06:23:46.379185+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:46.379185+0000","last_undegraded":"2024-03-04T06:23:46.379185+0000","last_fullsized":"2024-03-04T06:23:46.379185+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210995,"num_objects":52,"num_object_clones":43,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":312,"num_read_kb":250024,"num_write":224,"num_write_kb":64009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2358,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.369991+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:45.369991+0000","last_peered":"2024-03-04T06:23:45.369991+0000","last_clean":"2024-03-04T06:23:45.369991+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:45.369991+0000","last_undegraded":"2024-03-04T06:23:45.369991+0000","last_fullsized":"2024-03-04T06:23:45.369991+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"952'1870","reported_seq":3821,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.388552+0000","last_change":"2024-03-04T06:23:44.434727+0000","last_active":"2024-03-04T06:23:46.388552+0000","last_peered":"2024-03-04T06:23:46.388552+0000","last_clean":"2024-03-04T06:23:46.388552+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:46.388552+0000","last_undegraded":"2024-03-04T06:23:46.388552+0000","last_fullsized":"2024-03-04T06:23:46.388552+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1870,"log_dups_size":0,"ondisk_log_size":1870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12959781,"num_objects":144,"num_object_clones":118,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":797,"num_read_kb":627448,"num_write":669,"num_write_kb":179858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2343,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.423505+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:44.423505+0000","last_peered":"2024-03-04T06:23:44.423505+0000","last_clean":"2024-03-04T06:23:44.423505+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:44.423505+0000","last_undegraded":"2024-03-04T06:23:44.423505+0000","last_fullsized":"2024-03-04T06:23:44.423505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"952'2205","reported_seq":3858,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.401718+0000","last_change":"2024-03-04T06:23:43.391976+0000","last_active":"2024-03-04T06:23:44.401718+0000","last_peered":"2024-03-04T06:23:44.401718+0000","last_clean":"2024-03-04T06:23:44.401718+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:44.401718+0000","last_undegraded":"2024-03-04T06:23:44.401718+0000","last_fullsized":"2024-03-04T06:23:44.401718+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2205,"log_dups_size":0,"ondisk_log_size":2205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13262307,"num_objects":158,"num_object_clones":134,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":724,"num_read_kb":536055,"num_write":722,"num_write_kb":195272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"953'934","reported_seq":3295,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.380025+0000","last_change":"2024-03-04T06:23:44.421745+0000","last_active":"2024-03-04T06:23:45.380025+0000","last_peered":"2024-03-04T06:23:45.380025+0000","last_clean":"2024-03-04T06:23:45.380025+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:45.380025+0000","last_undegraded":"2024-03-04T06:23:45.380025+0000","last_fullsized":"2024-03-04T06:23:45.380025+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":934,"log_dups_size":0,"ondisk_log_size":934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865973,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":438,"num_read_kb":364480,"num_write":318,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2449,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.412376+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:44.412376+0000","last_peered":"2024-03-04T06:23:44.412376+0000","last_clean":"2024-03-04T06:23:44.412376+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:44.412376+0000","last_undegraded":"2024-03-04T06:23:44.412376+0000","last_fullsized":"2024-03-04T06:23:44.412376+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"949'1102","reported_seq":3393,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.539284+0000","last_change":"2024-03-04T06:23:43.371182+0000","last_active":"2024-03-04T06:23:44.539284+0000","last_peered":"2024-03-04T06:23:44.539284+0000","last_clean":"2024-03-04T06:23:44.539284+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:44.539284+0000","last_undegraded":"2024-03-04T06:23:44.539284+0000","last_fullsized":"2024-03-04T06:23:44.539284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1102,"log_dups_size":0,"ondisk_log_size":1102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7496714,"num_objects":83,"num_object_clones":67,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":390,"num_read_kb":282670,"num_write":362,"num_write_kb":98038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2413,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.388555+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:46.388555+0000","last_peered":"2024-03-04T06:23:46.388555+0000","last_clean":"2024-03-04T06:23:46.388555+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:46.388555+0000","last_undegraded":"2024-03-04T06:23:46.388555+0000","last_fullsized":"2024-03-04T06:23:46.388555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"951'1445","reported_seq":3496,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-03-04T06:23:46.378464+0000","last_change":"2024-03-04T06:23:44.390581+0000","last_active":"2024-03-04T06:23:46.378464+0000","last_peered":"2024-03-04T06:23:46.378464+0000","last_clean":"2024-03-04T06:23:46.378464+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:46.378464+0000","last_undegraded":"2024-03-04T06:23:46.378464+0000","last_fullsized":"2024-03-04T06:23:46.378464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6774013,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":496,"num_read_kb":386812,"num_write":462,"num_write_kb":128837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2449,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.396390+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:44.396390+0000","last_peered":"2024-03-04T06:23:44.396390+0000","last_clean":"2024-03-04T06:23:44.396390+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:44.396390+0000","last_undegraded":"2024-03-04T06:23:44.396390+0000","last_fullsized":"2024-03-04T06:23:44.396390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"953'1279","reported_seq":3565,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-03-04T06:23:44.479912+0000","last_change":"2024-03-04T06:23:43.393113+0000","last_active":"2024-03-04T06:23:44.479912+0000","last_peered":"2024-03-04T06:23:44.479912+0000","last_clean":"2024-03-04T06:23:44.479912+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:44.479912+0000","last_undegraded":"2024-03-04T06:23:44.479912+0000","last_fullsized":"2024-03-04T06:23:44.479912+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1279,"log_dups_size":0,"ondisk_log_size":1279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269803,"num_objects":100,"num_object_clones":86,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":585,"num_read_kb":494165,"num_write":447,"num_write_kb":122503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2358,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-03-04T06:23:45.381831+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:45.381831+0000","last_peered":"2024-03-04T06:23:45.381831+0000","last_clean":"2024-03-04T06:23:45.381831+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:45.381831+0000","last_undegraded":"2024-03-04T06:23:45.381831+0000","last_fullsized":"2024-03-04T06:23:45.381831+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":110434860,"num_objects":1220,"num_object_clones":1008,"num_object_copies":3660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":179,"num_read":6526,"num_read_kb":5189412,"num_write":5529,"num_write_kb":1512013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16541,"ondisk_log_size":16541,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706624,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":197504,"kb_used_data":126264,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94174336,"statfs":{"total":96636764160,"available":96434520064,"internally_reserved":0,"allocated":129294336,"data_stored":152290758,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706624,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":151276,"kb_used_data":93924,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94220564,"statfs":{"total":96636764160,"available":96481857536,"internally_reserved":0,"allocated":96178176,"data_stored":112728420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706623,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":155676,"kb_used_data":98324,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94216164,"statfs":{"total":96636764160,"available":96477351936,"internally_reserved":0,"allocated":100683776,"data_stored":122432189,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706623,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153340,"kb_used_data":95988,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94218500,"statfs":{"total":96636764160,"available":96479744000,"internally_reserved":0,"allocated":98291712,"data_stored":118034088,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:48.421 INFO:tasks.rados.rados.0.smithi064.stdout:3677: done (1 left) 2024-03-04T06:23:48.421 INFO:tasks.rados.rados.0.smithi064.stdout:3679: done (0 left) 2024-03-04T06:23:48.421 INFO:tasks.rados.rados.0.smithi064.stdout:3680: snap_create 2024-03-04T06:23:49.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:49.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:49.266+0000 7ffad9c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:49.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:49.270+0000 7ffad9c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:49.373 INFO:tasks.rados.rados.0.smithi064.stdout:3680: done (0 left) 2024-03-04T06:23:49.373 INFO:tasks.rados.rados.0.smithi064.stdout:3681: delete oid 31 current snap is 478 2024-03-04T06:23:49.444 INFO:tasks.rados.rados.0.smithi064.stdout:3681: done (0 left) 2024-03-04T06:23:49.444 INFO:tasks.rados.rados.0.smithi064.stdout:3682: delete oid 33 current snap is 478 2024-03-04T06:23:49.510 INFO:tasks.rados.rados.0.smithi064.stdout:3682: done (0 left) 2024-03-04T06:23:49.511 INFO:tasks.rados.rados.0.smithi064.stdout:3683: snap_remove snap 462 2024-03-04T06:23:50.376 INFO:tasks.rados.rados.0.smithi064.stdout:3683: done (0 left) 2024-03-04T06:23:50.376 INFO:tasks.rados.rados.0.smithi064.stdout:3684: write (excl) oid 24 current snap is 478 2024-03-04T06:23:50.376 INFO:tasks.rados.rados.0.smithi064.stdout:3684: seq_num 1666 ranges {400462=446141,1440295=560167} 2024-03-04T06:23:50.382 INFO:tasks.rados.rados.0.smithi064.stdout:3684: writing smithi06431743-24 from 400462 to 846603 tid 1 2024-03-04T06:23:50.390 INFO:tasks.rados.rados.0.smithi064.stdout:3684: writing smithi06431743-24 from 1440295 to 2000462 tid 2 2024-03-04T06:23:50.390 INFO:tasks.rados.rados.0.smithi064.stdout:3685: write (excl) oid 12 current snap is 478 2024-03-04T06:23:50.391 INFO:tasks.rados.rados.0.smithi064.stdout:3685: seq_num 1667 ranges {417269=521390,1498777=518492} 2024-03-04T06:23:50.397 INFO:tasks.rados.rados.0.smithi064.stdout:3685: writing smithi06431743-12 from 417269 to 938659 tid 1 2024-03-04T06:23:50.403 INFO:tasks.rados.rados.0.smithi064.stdout:3685: writing smithi06431743-12 from 1498777 to 2017269 tid 2 2024-03-04T06:23:50.403 INFO:tasks.rados.rados.0.smithi064.stdout:3686: snap_remove snap 473 2024-03-04T06:23:51.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:51.382 INFO:tasks.rados.rados.0.smithi064.stdout:3686: done (2 left) 2024-03-04T06:23:51.382 INFO:tasks.rados.rados.0.smithi064.stdout:3687: delete oid 11 current snap is 478 2024-03-04T06:23:51.382 INFO:tasks.rados.rados.0.smithi064.stdout:3684: finishing write tid 1 to smithi06431743-24 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3685: finishing write tid 1 to smithi06431743-12 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3684: finishing write tid 2 to smithi06431743-24 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3685: finishing write tid 2 to smithi06431743-12 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3684: finishing write tid 3 to smithi06431743-24 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3685: finishing write tid 3 to smithi06431743-12 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3685: finishing write tid 4 to smithi06431743-12 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3685: oid 12 updating version 0 to 1026 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3685: oid 12 version 1026 is already newer than 1026 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3685: oid 12 version 1026 is already newer than 1024 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 1026 (ObjNum 1667 snap 478 seq_num 1667) dirty exists 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3685: left oid 12 (ObjNum 1667 snap 478 seq_num 1667) 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3684: finishing write tid 4 to smithi06431743-24 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3684: oid 24 updating version 0 to 1453 2024-03-04T06:23:51.383 INFO:tasks.rados.rados.0.smithi064.stdout:3684: oid 24 updating version 1453 to 1455 2024-03-04T06:23:51.384 INFO:tasks.rados.rados.0.smithi064.stdout:3684: oid 24 version 1455 is already newer than 1455 2024-03-04T06:23:51.384 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 1455 (ObjNum 1666 snap 478 seq_num 1666) dirty exists 2024-03-04T06:23:51.384 INFO:tasks.rados.rados.0.smithi064.stdout:3684: left oid 24 (ObjNum 1666 snap 478 seq_num 1666) 2024-03-04T06:23:51.458 INFO:tasks.rados.rados.0.smithi064.stdout:3684: done (2 left) 2024-03-04T06:23:51.459 INFO:tasks.rados.rados.0.smithi064.stdout:3685: done (1 left) 2024-03-04T06:23:51.459 INFO:tasks.rados.rados.0.smithi064.stdout:3687: done (0 left) 2024-03-04T06:23:51.459 INFO:tasks.rados.rados.0.smithi064.stdout:3688: read oid 30 snap -1 2024-03-04T06:23:51.459 INFO:tasks.rados.rados.0.smithi064.stdout:3688: expect deleted 2024-03-04T06:23:51.459 INFO:tasks.rados.rados.0.smithi064.stdout:3689: delete oid 46 current snap is 478 2024-03-04T06:23:51.473 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:51.472+0000 7fb65f82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:51.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:51.476+0000 7fb65f82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:51.508 INFO:tasks.rados.rados.0.smithi064.stdout:3688: done (1 left) 2024-03-04T06:23:51.508 INFO:tasks.rados.rados.0.smithi064.stdout:3689: done (0 left) 2024-03-04T06:23:51.508 INFO:tasks.rados.rados.0.smithi064.stdout:3690: snap_create 2024-03-04T06:23:52.383 INFO:tasks.rados.rados.0.smithi064.stdout:3690: done (0 left) 2024-03-04T06:23:52.383 INFO:tasks.rados.rados.0.smithi064.stdout:3691: snap_create 2024-03-04T06:23:53.344 INFO:tasks.rados.rados.0.smithi064.stdout:3691: done (0 left) 2024-03-04T06:23:53.344 INFO:tasks.rados.rados.0.smithi064.stdout:3692: read oid 9 snap -1 2024-03-04T06:23:53.344 INFO:tasks.rados.rados.0.smithi064.stdout:3692: expect (ObjNum 1580 snap 447 seq_num 1580) 2024-03-04T06:23:53.344 INFO:tasks.rados.rados.0.smithi064.stdout:3693: snap_remove snap 448 2024-03-04T06:23:53.382 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:23:53.382 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:23:53.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:23:53.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:53.494+0000 7fe6edb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:53.500+0000 7fe6edb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:53.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:53.655 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:53.652+0000 7f443be24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:53.657 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:53.656+0000 7f443be24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:53.700 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:23:53.701 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:23:53.713 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1437,"stamp":"2024-03-04T06:23:53.342455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99191978,"num_objects":1227,"num_object_clones":1013,"num_object_copies":3681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":182,"num_read":6639,"num_read_kb":5196139,"num_write":5678,"num_write_kb":1516862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16729,"ondisk_log_size":16729,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":666912,"kb_used_data":423616,"kb_used_omap":0,"kb_used_meta":243264,"kb_avail":376820448,"statfs":{"total":386547056640,"available":385864138752,"internally_reserved":0,"allocated":433782784,"data_stored":517268328,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":249102336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11833250,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":10,"num_read_kb":6636,"num_write":12,"num_write_kb":2955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.969748"},"pg_stats":[{"pgid":"3.f","version":"953'911","reported_seq":3255,"reported_epoch":958,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:50.375735+0000","last_change":"2024-03-04T06:23:50.375735+0000","last_active":"2024-03-04T06:23:50.375735+0000","last_peered":"2024-03-04T06:23:50.375735+0000","last_clean":"2024-03-04T06:23:50.375735+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:50.375735+0000","last_undegraded":"2024-03-04T06:23:50.375735+0000","last_fullsized":"2024-03-04T06:23:50.375735+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":911,"log_dups_size":0,"ondisk_log_size":911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294568,"num_objects":77,"num_object_clones":62,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":362,"num_read_kb":286556,"num_write":323,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3251,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-04T06:23:49.465000+0000","last_change":"2024-03-04T06:23:49.472640+0000","last_active":"2024-03-04T06:23:49.465000+0000","last_peered":"2024-03-04T06:23:49.465000+0000","last_clean":"2024-03-04T06:23:49.465000+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:49.465000+0000","last_undegraded":"2024-03-04T06:23:49.465000+0000","last_fullsized":"2024-03-04T06:23:49.465000+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"959'1286","reported_seq":3476,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-03-04T06:23:51.406859+0000","last_change":"2024-03-04T06:23:51.406859+0000","last_active":"2024-03-04T06:23:51.406859+0000","last_peered":"2024-03-04T06:23:51.406859+0000","last_clean":"2024-03-04T06:23:51.406859+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:51.406859+0000","last_undegraded":"2024-03-04T06:23:51.406859+0000","last_fullsized":"2024-03-04T06:23:51.406859+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1286,"log_dups_size":0,"ondisk_log_size":1286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9900511,"num_objects":97,"num_object_clones":82,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":573,"num_read_kb":460192,"num_write":443,"num_write_kb":121345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"956'1017","reported_seq":3234,"reported_epoch":958,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:50.364568+0000","last_change":"2024-03-04T06:23:50.364568+0000","last_active":"2024-03-04T06:23:50.364568+0000","last_peered":"2024-03-04T06:23:50.364568+0000","last_clean":"2024-03-04T06:23:50.364568+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:50.364568+0000","last_undegraded":"2024-03-04T06:23:50.364568+0000","last_fullsized":"2024-03-04T06:23:50.364568+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1017,"log_dups_size":0,"ondisk_log_size":1017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866006,"num_objects":67,"num_object_clones":57,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":328,"num_read_kb":252047,"num_write":315,"num_write_kb":87346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"958'419","reported_seq":3041,"reported_epoch":959,"state":"active+clean","last_fresh":"2024-03-04T06:23:50.410106+0000","last_change":"2024-03-04T06:23:50.413577+0000","last_active":"2024-03-04T06:23:50.410106+0000","last_peered":"2024-03-04T06:23:50.410106+0000","last_clean":"2024-03-04T06:23:50.410106+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:50.410106+0000","last_undegraded":"2024-03-04T06:23:50.410106+0000","last_fullsized":"2024-03-04T06:23:50.410106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":122,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"957'1016","reported_seq":3322,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-04T06:23:50.372652+0000","last_change":"2024-03-04T06:23:50.412669+0000","last_active":"2024-03-04T06:23:50.372652+0000","last_peered":"2024-03-04T06:23:50.372652+0000","last_clean":"2024-03-04T06:23:50.372652+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:50.372652+0000","last_undegraded":"2024-03-04T06:23:50.372652+0000","last_fullsized":"2024-03-04T06:23:50.372652+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1016,"log_dups_size":0,"ondisk_log_size":1016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10152616,"num_objects":74,"num_object_clones":62,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":435,"num_read_kb":348132,"num_write":358,"num_write_kb":103998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2804,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-04T06:23:50.386603+0000","last_change":"2024-03-04T06:23:50.397108+0000","last_active":"2024-03-04T06:23:50.386603+0000","last_peered":"2024-03-04T06:23:50.386603+0000","last_clean":"2024-03-04T06:23:50.386603+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:50.386603+0000","last_undegraded":"2024-03-04T06:23:50.386603+0000","last_fullsized":"2024-03-04T06:23:50.386603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"957'662","reported_seq":3121,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-03-04T06:23:51.402940+0000","last_change":"2024-03-04T06:23:51.407154+0000","last_active":"2024-03-04T06:23:51.402940+0000","last_peered":"2024-03-04T06:23:51.402940+0000","last_clean":"2024-03-04T06:23:51.402940+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:51.402940+0000","last_undegraded":"2024-03-04T06:23:51.402940+0000","last_fullsized":"2024-03-04T06:23:51.402940+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2556,"reported_epoch":959,"state":"active+clean","last_fresh":"2024-03-04T06:23:50.388628+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:50.388628+0000","last_peered":"2024-03-04T06:23:50.388628+0000","last_clean":"2024-03-04T06:23:50.388628+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:50.388628+0000","last_undegraded":"2024-03-04T06:23:50.388628+0000","last_fullsized":"2024-03-04T06:23:50.388628+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2372,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-03-04T06:23:51.411044+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:51.411044+0000","last_peered":"2024-03-04T06:23:51.411044+0000","last_clean":"2024-03-04T06:23:51.411044+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:51.411044+0000","last_undegraded":"2024-03-04T06:23:51.411044+0000","last_fullsized":"2024-03-04T06:23:51.411044+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"960'959","reported_seq":3317,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-03-04T06:23:51.409911+0000","last_change":"2024-03-04T06:23:51.425878+0000","last_active":"2024-03-04T06:23:51.409911+0000","last_peered":"2024-03-04T06:23:51.409911+0000","last_clean":"2024-03-04T06:23:51.409911+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:51.409911+0000","last_undegraded":"2024-03-04T06:23:51.409911+0000","last_fullsized":"2024-03-04T06:23:51.409911+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698299,"num_objects":72,"num_object_clones":58,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":441,"num_read_kb":363862,"num_write":322,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2353,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-04T06:23:50.378053+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:50.378053+0000","last_peered":"2024-03-04T06:23:50.378053+0000","last_clean":"2024-03-04T06:23:50.378053+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:50.378053+0000","last_undegraded":"2024-03-04T06:23:50.378053+0000","last_fullsized":"2024-03-04T06:23:50.378053+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"959'679","reported_seq":3218,"reported_epoch":959,"state":"active+clean","last_fresh":"2024-03-04T06:23:50.384230+0000","last_change":"2024-03-04T06:23:50.489806+0000","last_active":"2024-03-04T06:23:50.384230+0000","last_peered":"2024-03-04T06:23:50.384230+0000","last_clean":"2024-03-04T06:23:50.384230+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:23:50.384230+0000","last_undegraded":"2024-03-04T06:23:50.384230+0000","last_fullsized":"2024-03-04T06:23:50.384230+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210995,"num_objects":52,"num_object_clones":43,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":312,"num_read_kb":250024,"num_write":224,"num_write_kb":64009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2372,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-03-04T06:23:51.400508+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:51.400508+0000","last_peered":"2024-03-04T06:23:51.400508+0000","last_clean":"2024-03-04T06:23:51.400508+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:51.400508+0000","last_undegraded":"2024-03-04T06:23:51.400508+0000","last_fullsized":"2024-03-04T06:23:51.400508+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"959'1893","reported_seq":3839,"reported_epoch":959,"state":"active+clean","last_fresh":"2024-03-04T06:23:50.399574+0000","last_change":"2024-03-04T06:23:50.563729+0000","last_active":"2024-03-04T06:23:50.399574+0000","last_peered":"2024-03-04T06:23:50.399574+0000","last_clean":"2024-03-04T06:23:50.399574+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:50.399574+0000","last_undegraded":"2024-03-04T06:23:50.399574+0000","last_fullsized":"2024-03-04T06:23:50.399574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1893,"log_dups_size":0,"ondisk_log_size":1893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13312728,"num_objects":145,"num_object_clones":119,"num_object_copies":435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":798,"num_read_kb":627449,"num_write":674,"num_write_kb":181829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2353,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-04T06:23:50.369805+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:50.369805+0000","last_peered":"2024-03-04T06:23:50.369805+0000","last_clean":"2024-03-04T06:23:50.369805+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:50.369805+0000","last_undegraded":"2024-03-04T06:23:50.369805+0000","last_fullsized":"2024-03-04T06:23:50.369805+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"958'2221","reported_seq":3872,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-04T06:23:49.461626+0000","last_change":"2024-03-04T06:23:49.567884+0000","last_active":"2024-03-04T06:23:49.461626+0000","last_peered":"2024-03-04T06:23:49.461626+0000","last_clean":"2024-03-04T06:23:49.461626+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:49.461626+0000","last_undegraded":"2024-03-04T06:23:49.461626+0000","last_fullsized":"2024-03-04T06:23:49.461626+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2221,"log_dups_size":0,"ondisk_log_size":2221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13262307,"num_objects":158,"num_object_clones":134,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":724,"num_read_kb":536055,"num_write":722,"num_write_kb":195272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"959'938","reported_seq":3312,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-03-04T06:23:51.413242+0000","last_change":"2024-03-04T06:23:51.439474+0000","last_active":"2024-03-04T06:23:51.413242+0000","last_peered":"2024-03-04T06:23:51.413242+0000","last_clean":"2024-03-04T06:23:51.413242+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:51.413242+0000","last_undegraded":"2024-03-04T06:23:51.413242+0000","last_fullsized":"2024-03-04T06:23:51.413242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865973,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":438,"num_read_kb":364480,"num_write":318,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2460,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-04T06:23:49.474591+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:49.474591+0000","last_peered":"2024-03-04T06:23:49.474591+0000","last_clean":"2024-03-04T06:23:49.474591+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:49.474591+0000","last_undegraded":"2024-03-04T06:23:49.474591+0000","last_fullsized":"2024-03-04T06:23:49.474591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"958'1104","reported_seq":3405,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-04T06:23:49.479405+0000","last_change":"2024-03-04T06:23:49.514703+0000","last_active":"2024-03-04T06:23:49.479405+0000","last_peered":"2024-03-04T06:23:49.479405+0000","last_clean":"2024-03-04T06:23:49.479405+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:49.479405+0000","last_undegraded":"2024-03-04T06:23:49.479405+0000","last_fullsized":"2024-03-04T06:23:49.479405+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1104,"log_dups_size":0,"ondisk_log_size":1104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7496714,"num_objects":83,"num_object_clones":67,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":390,"num_read_kb":282670,"num_write":362,"num_write_kb":98038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2423,"reported_epoch":959,"state":"active+clean","last_fresh":"2024-03-04T06:23:50.400432+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:50.400432+0000","last_peered":"2024-03-04T06:23:50.400432+0000","last_clean":"2024-03-04T06:23:50.400432+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:50.400432+0000","last_undegraded":"2024-03-04T06:23:50.400432+0000","last_fullsized":"2024-03-04T06:23:50.400432+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"959'1455","reported_seq":3512,"reported_epoch":959,"state":"active+clean","last_fresh":"2024-03-04T06:23:50.557863+0000","last_change":"2024-03-04T06:23:50.561409+0000","last_active":"2024-03-04T06:23:50.557863+0000","last_peered":"2024-03-04T06:23:50.557863+0000","last_clean":"2024-03-04T06:23:50.557863+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:23:50.557863+0000","last_undegraded":"2024-03-04T06:23:50.557863+0000","last_fullsized":"2024-03-04T06:23:50.557863+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1455,"log_dups_size":0,"ondisk_log_size":1455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5379065,"num_objects":100,"num_object_clones":85,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":498,"num_read_kb":386813,"num_write":466,"num_write_kb":129821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2460,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-04T06:23:49.458571+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:49.458571+0000","last_peered":"2024-03-04T06:23:49.458571+0000","last_clean":"2024-03-04T06:23:49.458571+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:49.458571+0000","last_undegraded":"2024-03-04T06:23:49.458571+0000","last_fullsized":"2024-03-04T06:23:49.458571+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"958'1283","reported_seq":3578,"reported_epoch":958,"state":"active+clean","last_fresh":"2024-03-04T06:23:49.510165+0000","last_change":"2024-03-04T06:23:49.517543+0000","last_active":"2024-03-04T06:23:49.510165+0000","last_peered":"2024-03-04T06:23:49.510165+0000","last_clean":"2024-03-04T06:23:49.510165+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:49.510165+0000","last_undegraded":"2024-03-04T06:23:49.510165+0000","last_fullsized":"2024-03-04T06:23:49.510165+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1283,"log_dups_size":0,"ondisk_log_size":1283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6269803,"num_objects":100,"num_object_clones":86,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":586,"num_read_kb":494165,"num_write":447,"num_write_kb":122503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2372,"reported_epoch":960,"state":"active+clean","last_fresh":"2024-03-04T06:23:51.409072+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:51.409072+0000","last_peered":"2024-03-04T06:23:51.409072+0000","last_clean":"2024-03-04T06:23:51.409072+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:51.409072+0000","last_undegraded":"2024-03-04T06:23:51.409072+0000","last_fullsized":"2024-03-04T06:23:51.409072+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":98601610,"num_objects":1225,"num_object_clones":1013,"num_object_copies":3675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":182,"num_read":6536,"num_read_kb":5196048,"num_write":5541,"num_write_kb":1514968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16632,"ondisk_log_size":16632,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706629,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":201556,"kb_used_data":130316,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94170284,"statfs":{"total":96636764160,"available":96430370816,"internally_reserved":0,"allocated":133443584,"data_stored":157443734,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706629,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154292,"kb_used_data":96940,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94217548,"statfs":{"total":96636764160,"available":96478769152,"internally_reserved":0,"allocated":99266560,"data_stored":116837773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":157700,"kb_used_data":100348,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94214140,"statfs":{"total":96636764160,"available":96475279360,"internally_reserved":0,"allocated":102756352,"data_stored":124941510,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706628,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153364,"kb_used_data":96012,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94218476,"statfs":{"total":96636764160,"available":96479719424,"internally_reserved":0,"allocated":98316288,"data_stored":118045311,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:23:54.344 INFO:tasks.rados.rados.0.smithi064.stdout:3693: done (1 left) 2024-03-04T06:23:54.344 INFO:tasks.rados.rados.0.smithi064.stdout:3694: snap_create 2024-03-04T06:23:54.344 INFO:tasks.rados.rados.0.smithi064.stdout:3692: expect (ObjNum 1580 snap 447 seq_num 1580) 2024-03-04T06:23:55.345 INFO:tasks.rados.rados.0.smithi064.stdout:3692: done (1 left) 2024-03-04T06:23:55.345 INFO:tasks.rados.rados.0.smithi064.stdout:3694: done (0 left) 2024-03-04T06:23:55.345 INFO:tasks.rados.rados.0.smithi064.stdout:3695: snap_create 2024-03-04T06:23:55.733 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:23:55.853 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:55.852+0000 7f6652e2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:55.856 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:55.856+0000 7f6652e2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:56.346 INFO:tasks.rados.rados.0.smithi064.stdout:3695: done (0 left) 2024-03-04T06:23:56.346 INFO:tasks.rados.rados.0.smithi064.stdout:3696: read oid 28 snap -1 2024-03-04T06:23:56.346 INFO:tasks.rados.rados.0.smithi064.stdout:3696: expect deleted 2024-03-04T06:23:56.346 INFO:tasks.rados.rados.0.smithi064.stdout:3697: write (excl) oid 32 current snap is 482 2024-03-04T06:23:56.346 INFO:tasks.rados.rados.0.smithi064.stdout:3697: seq_num 1670 ranges {467690=747137,1957870=109820} 2024-03-04T06:23:56.353 INFO:tasks.rados.rados.0.smithi064.stdout:3697: writing smithi06431743-32 from 467690 to 1214827 tid 1 2024-03-04T06:23:56.358 INFO:tasks.rados.rados.0.smithi064.stdout:3697: writing smithi06431743-32 from 1957870 to 2067690 tid 2 2024-03-04T06:23:56.358 INFO:tasks.rados.rados.0.smithi064.stdout:3698: write oid 9 current snap is 482 2024-03-04T06:23:56.358 INFO:tasks.rados.rados.0.smithi064.stdout:3698: seq_num 1671 ranges {484497=422386,1616352=468145} 2024-03-04T06:23:56.363 INFO:tasks.rados.rados.0.smithi064.stdout:3698: writing smithi06431743-9 from 484497 to 906883 tid 1 2024-03-04T06:23:56.370 INFO:tasks.rados.rados.0.smithi064.stdout:3698: writing smithi06431743-9 from 1616352 to 2084497 tid 2 2024-03-04T06:23:56.370 INFO:tasks.rados.rados.0.smithi064.stdout:3699: write (excl) oid 24 current snap is 482 2024-03-04T06:23:56.371 INFO:tasks.rados.rados.0.smithi064.stdout:3699: seq_num 1672 ranges {501304=497635,1674834=426470} 2024-03-04T06:23:56.377 INFO:tasks.rados.rados.0.smithi064.stdout:3699: writing smithi06431743-24 from 501304 to 998939 tid 1 2024-03-04T06:23:56.385 INFO:tasks.rados.rados.0.smithi064.stdout:3699: writing smithi06431743-24 from 1674834 to 2101304 tid 2 2024-03-04T06:23:56.386 INFO:tasks.rados.rados.0.smithi064.stdout:3700: write oid 22 current snap is 482 2024-03-04T06:23:56.386 INFO:tasks.rados.rados.0.smithi064.stdout:3700: seq_num 1673 ranges {518111=689237,1849669=268442} 2024-03-04T06:23:56.394 INFO:tasks.rados.rados.0.smithi064.stdout:3700: writing smithi06431743-22 from 518111 to 1207348 tid 1 2024-03-04T06:23:56.400 INFO:tasks.rados.rados.0.smithi064.stdout:3700: writing smithi06431743-22 from 1849669 to 2118111 tid 2 2024-03-04T06:23:56.400 INFO:tasks.rados.rados.0.smithi064.stdout:3701: write oid 12 current snap is 482 2024-03-04T06:23:56.400 INFO:tasks.rados.rados.0.smithi064.stdout:3701: seq_num 1674 ranges {534918=764486,1791798=343120} 2024-03-04T06:23:56.409 INFO:tasks.rados.rados.0.smithi064.stdout:3701: writing smithi06431743-12 from 534918 to 1299404 tid 1 2024-03-04T06:23:56.414 INFO:tasks.rados.rados.0.smithi064.stdout:3701: writing smithi06431743-12 from 1791798 to 2134918 tid 2 2024-03-04T06:23:56.414 INFO:tasks.rados.rados.0.smithi064.stdout:3696: done (5 left) 2024-03-04T06:23:56.414 INFO:tasks.rados.rados.0.smithi064.stdout:3702: delete oid 3 current snap is 482 2024-03-04T06:23:56.475 INFO:tasks.rados.rados.0.smithi064.stdout:3699: finishing write tid 1 to smithi06431743-24 2024-03-04T06:23:56.476 INFO:tasks.rados.rados.0.smithi064.stdout:3701: finishing write tid 1 to smithi06431743-12 2024-03-04T06:23:56.479 INFO:tasks.rados.rados.0.smithi064.stdout:3697: finishing write tid 1 to smithi06431743-32 2024-03-04T06:23:56.494 INFO:tasks.rados.rados.0.smithi064.stdout:3701: finishing write tid 2 to smithi06431743-12 2024-03-04T06:23:56.494 INFO:tasks.rados.rados.0.smithi064.stdout:3698: finishing write tid 1 to smithi06431743-9 2024-03-04T06:23:56.512 INFO:tasks.rados.rados.0.smithi064.stdout:3701: finishing write tid 3 to smithi06431743-12 2024-03-04T06:23:56.516 INFO:tasks.rados.rados.0.smithi064.stdout:3701: finishing write tid 4 to smithi06431743-12 2024-03-04T06:23:56.516 INFO:tasks.rados.rados.0.smithi064.stdout:3701: oid 12 updating version 0 to 1042 2024-03-04T06:23:56.516 INFO:tasks.rados.rados.0.smithi064.stdout:3701: oid 12 version 1042 is already newer than 1040 2024-03-04T06:23:56.516 INFO:tasks.rados.rados.0.smithi064.stdout:3701: oid 12 version 1042 is already newer than 1041 2024-03-04T06:23:56.516 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 12 v 1042 (ObjNum 1674 snap 482 seq_num 1674) dirty exists 2024-03-04T06:23:56.516 INFO:tasks.rados.rados.0.smithi064.stdout:3701: left oid 12 (ObjNum 1674 snap 482 seq_num 1674) 2024-03-04T06:23:56.521 INFO:tasks.rados.rados.0.smithi064.stdout:3699: finishing write tid 2 to smithi06431743-24 2024-03-04T06:23:56.526 INFO:tasks.rados.rados.0.smithi064.stdout:3701: done (5 left) 2024-03-04T06:23:56.526 INFO:tasks.rados.rados.0.smithi064.stdout:3702: done (4 left) 2024-03-04T06:23:56.526 INFO:tasks.rados.rados.0.smithi064.stdout:3703: write oid 33 current snap is 482 2024-03-04T06:23:56.527 INFO:tasks.rados.rados.0.smithi064.stdout:3703: seq_num 1676 ranges {568532=514984,1508762=529912} 2024-03-04T06:23:56.533 INFO:tasks.rados.rados.0.smithi064.stdout:3703: writing smithi06431743-33 from 568532 to 1083516 tid 1 2024-03-04T06:23:56.539 INFO:tasks.rados.rados.0.smithi064.stdout:3703: writing smithi06431743-33 from 1508762 to 2038674 tid 2 2024-03-04T06:23:56.539 INFO:tasks.rados.rados.0.smithi064.stdout:3704: delete oid 31 current snap is 482 2024-03-04T06:23:56.539 INFO:tasks.rados.rados.0.smithi064.stdout:3697: finishing write tid 2 to smithi06431743-32 2024-03-04T06:23:56.539 INFO:tasks.rados.rados.0.smithi064.stdout:3697: finishing write tid 3 to smithi06431743-32 2024-03-04T06:23:56.539 INFO:tasks.rados.rados.0.smithi064.stdout:3697: finishing write tid 4 to smithi06431743-32 2024-03-04T06:23:56.539 INFO:tasks.rados.rados.0.smithi064.stdout:3697: oid 32 updating version 0 to 1907 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:3697: oid 32 version 1907 is already newer than 1907 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:3697: oid 32 version 1907 is already newer than 1905 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 32 v 1907 (ObjNum 1670 snap 482 seq_num 1670) dirty exists 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:3697: left oid 32 (ObjNum 1670 snap 482 seq_num 1670) 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:3698: finishing write tid 2 to smithi06431743-9 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:3699: finishing write tid 3 to smithi06431743-24 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:3699: finishing write tid 4 to smithi06431743-24 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:3699: oid 24 updating version 0 to 1461 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:3699: oid 24 updating version 1461 to 1463 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:3699: oid 24 version 1463 is already newer than 1462 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 1463 (ObjNum 1672 snap 482 seq_num 1672) dirty exists 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:3699: left oid 24 (ObjNum 1672 snap 482 seq_num 1672) 2024-03-04T06:23:56.540 INFO:tasks.rados.rados.0.smithi064.stdout:3698: finishing write tid 3 to smithi06431743-9 2024-03-04T06:23:56.541 INFO:tasks.rados.rados.0.smithi064.stdout:3697: done (5 left) 2024-03-04T06:23:56.541 INFO:tasks.rados.rados.0.smithi064.stdout:3699: done (4 left) 2024-03-04T06:23:56.542 INFO:tasks.rados.rados.0.smithi064.stdout:3704: done (3 left) 2024-03-04T06:23:56.542 INFO:tasks.rados.rados.0.smithi064.stdout:3705: snap_remove snap 479 2024-03-04T06:23:57.397 INFO:tasks.rados.rados.0.smithi064.stdout:3705: done (3 left) 2024-03-04T06:23:57.397 INFO:tasks.rados.rados.0.smithi064.stdout:3706: delete oid 16 current snap is 482 2024-03-04T06:23:57.397 INFO:tasks.rados.rados.0.smithi064.stdout:3698: finishing write tid 4 to smithi06431743-9 2024-03-04T06:23:57.397 INFO:tasks.rados.rados.0.smithi064.stdout:3698: oid 9 updating version 0 to 1297 2024-03-04T06:23:57.397 INFO:tasks.rados.rados.0.smithi064.stdout:3698: oid 9 version 1297 is already newer than 1296 2024-03-04T06:23:57.397 INFO:tasks.rados.rados.0.smithi064.stdout:3698: oid 9 version 1297 is already newer than 1295 2024-03-04T06:23:57.397 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 9 v 1297 (ObjNum 1671 snap 482 seq_num 1671) dirty exists 2024-03-04T06:23:57.397 INFO:tasks.rados.rados.0.smithi064.stdout:3698: left oid 9 (ObjNum 1671 snap 482 seq_num 1671) 2024-03-04T06:23:57.397 INFO:tasks.rados.rados.0.smithi064.stdout:3700: finishing write tid 1 to smithi06431743-22 2024-03-04T06:23:57.397 INFO:tasks.rados.rados.0.smithi064.stdout:3700: finishing write tid 2 to smithi06431743-22 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3700: finishing write tid 3 to smithi06431743-22 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3700: finishing write tid 4 to smithi06431743-22 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3700: oid 22 updating version 0 to 1301 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3700: oid 22 version 1301 is already newer than 1301 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3700: oid 22 version 1301 is already newer than 1301 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 1301 (ObjNum 1673 snap 482 seq_num 1673) dirty exists 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3700: left oid 22 (ObjNum 1673 snap 482 seq_num 1673) 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3703: finishing write tid 1 to smithi06431743-33 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3703: finishing write tid 2 to smithi06431743-33 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3703: finishing write tid 3 to smithi06431743-33 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3703: finishing write tid 4 to smithi06431743-33 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3703: oid 33 updating version 0 to 1304 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3703: oid 33 version 1304 is already newer than 1304 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:3703: oid 33 version 1304 is already newer than 1304 2024-03-04T06:23:57.398 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 1304 (ObjNum 1676 snap 482 seq_num 1676) dirty exists 2024-03-04T06:23:57.399 INFO:tasks.rados.rados.0.smithi064.stdout:3703: left oid 33 (ObjNum 1676 snap 482 seq_num 1676) 2024-03-04T06:23:57.458 INFO:tasks.rados.rados.0.smithi064.stdout:3698: done (3 left) 2024-03-04T06:23:57.458 INFO:tasks.rados.rados.0.smithi064.stdout:3700: done (2 left) 2024-03-04T06:23:57.458 INFO:tasks.rados.rados.0.smithi064.stdout:3703: done (1 left) 2024-03-04T06:23:57.458 INFO:tasks.rados.rados.0.smithi064.stdout:3706: done (0 left) 2024-03-04T06:23:57.458 INFO:tasks.rados.rados.0.smithi064.stdout:3707: snap_remove snap 466 2024-03-04T06:23:57.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:23:58.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:58.042+0000 7fd2a686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:58.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:58.047+0000 7fd2a686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:58.398 INFO:tasks.rados.rados.0.smithi064.stdout:3707: done (0 left) 2024-03-04T06:23:58.398 INFO:tasks.rados.rados.0.smithi064.stdout:3708: delete oid 39 current snap is 482 2024-03-04T06:23:58.459 INFO:tasks.rados.rados.0.smithi064.stdout:3708: done (0 left) 2024-03-04T06:23:58.459 INFO:tasks.rados.rados.0.smithi064.stdout:3709: read oid 34 snap 463 2024-03-04T06:23:58.459 INFO:tasks.rados.rados.0.smithi064.stdout:3709: expect (ObjNum 1564 snap 443 seq_num 1564) 2024-03-04T06:23:58.459 INFO:tasks.rados.rados.0.smithi064.stdout:3710: delete oid 26 current snap is 482 2024-03-04T06:23:58.545 INFO:tasks.rados.rados.0.smithi064.stdout:3709: expect (ObjNum 1564 snap 443 seq_num 1564) 2024-03-04T06:23:58.661 INFO:tasks.rados.rados.0.smithi064.stdout:3709: done (1 left) 2024-03-04T06:23:58.661 INFO:tasks.rados.rados.0.smithi064.stdout:3710: done (0 left) 2024-03-04T06:23:58.661 INFO:tasks.rados.rados.0.smithi064.stdout:3711: read oid 14 snap -1 2024-03-04T06:23:58.661 INFO:tasks.rados.rados.0.smithi064.stdout:3711: expect (ObjNum 1629 snap 466 seq_num 1629) 2024-03-04T06:23:58.661 INFO:tasks.rados.rados.0.smithi064.stdout:3712: write (excl) oid 25 current snap is 482 2024-03-04T06:23:58.661 INFO:tasks.rados.rados.0.smithi064.stdout:3712: seq_num 1681 ranges {652567=607582,1801172=451395} 2024-03-04T06:23:58.667 INFO:tasks.rados.rados.0.smithi064.stdout:3712: writing smithi06431743-25 from 652567 to 1260149 tid 1 2024-03-04T06:23:58.673 INFO:tasks.rados.rados.0.smithi064.stdout:3712: writing smithi06431743-25 from 1801172 to 2252567 tid 2 2024-03-04T06:23:58.673 INFO:tasks.rados.rados.0.smithi064.stdout:3713: read oid 6 snap 477 2024-03-04T06:23:58.673 INFO:tasks.rados.rados.0.smithi064.stdout:3713: expect (ObjNum 1539 snap 434 seq_num 1539) 2024-03-04T06:23:58.673 INFO:tasks.rados.rados.0.smithi064.stdout:3714: read oid 1 snap 460 2024-03-04T06:23:58.673 INFO:tasks.rados.rados.0.smithi064.stdout:3714: expect (ObjNum 1617 snap 460 seq_num 1617) 2024-03-04T06:23:58.673 INFO:tasks.rados.rados.0.smithi064.stdout:3715: snap_create 2024-03-04T06:23:58.689 INFO:tasks.rados.rados.0.smithi064.stdout:3711: expect (ObjNum 1629 snap 466 seq_num 1629) 2024-03-04T06:23:58.715 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:23:58.715 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:23:58.715 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:23:58.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:23:58.826 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:58.822+0000 7fefda4d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:58.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:58.829+0000 7fefda4d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:58.842 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:23:58.842 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:23:58.842 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:23:58.842 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:23:58.842 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:23:58.842 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:23:58.843 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:23:58.843 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:23:58.843 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:23:58.843 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:23:58.843 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:23:58.843 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:23:58.845 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:23:58.845 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:23:58.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:23:58.857 INFO:tasks.rados.rados.0.smithi064.stdout:3713: expect (ObjNum 1539 snap 434 seq_num 1539) 2024-03-04T06:23:58.953 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:58.949+0000 7f6925f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:58.956 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:58.956+0000 7f6925f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:58.968 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:23:58.968 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:23:58.968 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:23:58.968 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:23:58.969 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:23:58.969 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:23:58.969 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:23:58.969 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:23:58.969 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:23:58.969 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:23:58.969 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:23:58.969 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:23:58.971 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:23:58.971 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:23:58.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:23:59.063 INFO:tasks.rados.rados.0.smithi064.stdout:3714: expect (ObjNum 1617 snap 460 seq_num 1617) 2024-03-04T06:23:59.120 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:59.114+0000 7fb3c395d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:59.122 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:59.121+0000 7fb3c395d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:59.134 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:23:59.134 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:23:59.134 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:23:59.134 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:23:59.134 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:23:59.134 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:23:59.134 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:23:59.134 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:23:59.135 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:23:59.135 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:23:59.135 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:23:59.135 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:23:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:23:59.137 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:23:59.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:23:59.184 INFO:tasks.rados.rados.0.smithi064.stdout:3712: finishing write tid 1 to smithi06431743-25 2024-03-04T06:23:59.184 INFO:tasks.rados.rados.0.smithi064.stdout:3712: finishing write tid 2 to smithi06431743-25 2024-03-04T06:23:59.184 INFO:tasks.rados.rados.0.smithi064.stdout:3712: finishing write tid 3 to smithi06431743-25 2024-03-04T06:23:59.185 INFO:tasks.rados.rados.0.smithi064.stdout:3712: finishing write tid 4 to smithi06431743-25 2024-03-04T06:23:59.185 INFO:tasks.rados.rados.0.smithi064.stdout:3712: oid 25 updating version 0 to 1298 2024-03-04T06:23:59.185 INFO:tasks.rados.rados.0.smithi064.stdout:3712: oid 25 version 1298 is already newer than 1297 2024-03-04T06:23:59.185 INFO:tasks.rados.rados.0.smithi064.stdout:3712: oid 25 version 1298 is already newer than 1296 2024-03-04T06:23:59.185 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 1298 (ObjNum 1681 snap 482 seq_num 1681) dirty exists 2024-03-04T06:23:59.185 INFO:tasks.rados.rados.0.smithi064.stdout:3712: left oid 25 (ObjNum 1681 snap 482 seq_num 1681) 2024-03-04T06:23:59.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:59.241+0000 7f796df5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:59.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:23:59.249+0000 7f796df5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:23:59.262 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:23:59.262 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:23:59.262 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:23:59.262 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:23:59.262 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:23:59.262 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:23:59.262 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:23:59.262 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:23:59.262 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:23:59.262 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:23:59.262 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:23:59.263 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:23:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:23:59.265 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:23:59.403 INFO:tasks.rados.rados.0.smithi064.stdout:3711: done (4 left) 2024-03-04T06:23:59.403 INFO:tasks.rados.rados.0.smithi064.stdout:3712: done (3 left) 2024-03-04T06:23:59.403 INFO:tasks.rados.rados.0.smithi064.stdout:3713: done (2 left) 2024-03-04T06:23:59.403 INFO:tasks.rados.rados.0.smithi064.stdout:3714: done (1 left) 2024-03-04T06:23:59.403 INFO:tasks.rados.rados.0.smithi064.stdout:3715: done (0 left) 2024-03-04T06:23:59.403 INFO:tasks.rados.rados.0.smithi064.stdout:3716: snap_create 2024-03-04T06:24:00.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:00.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:00.224+0000 7fcb0b2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:00.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:00.229+0000 7fcb0b2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:00.404 INFO:tasks.rados.rados.0.smithi064.stdout:3716: done (0 left) 2024-03-04T06:24:00.404 INFO:tasks.rados.rados.0.smithi064.stdout:3717: read oid 33 snap -1 2024-03-04T06:24:00.404 INFO:tasks.rados.rados.0.smithi064.stdout:3717: expect (ObjNum 1676 snap 482 seq_num 1676) 2024-03-04T06:24:00.404 INFO:tasks.rados.rados.0.smithi064.stdout:3718: snap_create 2024-03-04T06:24:00.454 INFO:tasks.rados.rados.0.smithi064.stdout:3717: expect (ObjNum 1676 snap 482 seq_num 1676) 2024-03-04T06:24:01.413 INFO:tasks.rados.rados.0.smithi064.stdout:3717: done (1 left) 2024-03-04T06:24:01.413 INFO:tasks.rados.rados.0.smithi064.stdout:3718: done (0 left) 2024-03-04T06:24:01.413 INFO:tasks.rados.rados.0.smithi064.stdout:3719: snap_create 2024-03-04T06:24:02.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:02.411 INFO:tasks.rados.rados.0.smithi064.stdout:3719: done (0 left) 2024-03-04T06:24:02.412 INFO:tasks.rados.rados.0.smithi064.stdout:3720: snap_create 2024-03-04T06:24:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:02.414+0000 7fab4a35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:02.419 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:02.419+0000 7fab4a35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:03.351 INFO:tasks.rados.rados.0.smithi064.stdout:3720: done (0 left) 2024-03-04T06:24:03.351 INFO:tasks.rados.rados.0.smithi064.stdout:3721: snap_create 2024-03-04T06:24:04.266 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:24:04.266 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:24:04.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:24:04.347 INFO:tasks.rados.rados.0.smithi064.stdout:3721: done (0 left) 2024-03-04T06:24:04.347 INFO:tasks.rados.rados.0.smithi064.stdout:3722: snap_remove snap 472 2024-03-04T06:24:04.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:04.384+0000 7f85e811d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:04.390 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:04.390+0000 7f85e811d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:04.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:04.587 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:24:04.589 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:24:04.601 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1453,"stamp":"2024-03-04T06:24:04.345511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":85527326,"num_objects":1237,"num_object_clones":1023,"num_object_copies":3711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":184,"num_read":6683,"num_read_kb":5234239,"num_write":5714,"num_write_kb":1524567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16899,"ondisk_log_size":16899,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":694624,"kb_used_data":447168,"kb_used_omap":0,"kb_used_meta":247424,"kb_avail":376792736,"statfs":{"total":386547056640,"available":385835761664,"internally_reserved":0,"allocated":457900032,"data_stored":549681999,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":253362176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12018094,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":31,"num_read_kb":32652,"num_write":17,"num_write_kb":2643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.942640"},"pg_stats":[{"pgid":"3.f","version":"965'921","reported_seq":3279,"reported_epoch":967,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:58.484123+0000","last_change":"2024-03-04T06:23:58.484123+0000","last_active":"2024-03-04T06:23:58.484123+0000","last_peered":"2024-03-04T06:23:58.484123+0000","last_clean":"2024-03-04T06:23:58.484123+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:23:58.484123+0000","last_undegraded":"2024-03-04T06:23:58.484123+0000","last_fullsized":"2024-03-04T06:23:58.484123+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294568,"num_objects":77,"num_object_clones":62,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":362,"num_read_kb":286556,"num_write":323,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3281,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.510485+0000","last_change":"2024-03-04T06:23:58.524137+0000","last_active":"2024-03-04T06:23:58.510485+0000","last_peered":"2024-03-04T06:23:58.510485+0000","last_clean":"2024-03-04T06:23:58.510485+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:23:58.510485+0000","last_undegraded":"2024-03-04T06:23:58.510485+0000","last_fullsized":"2024-03-04T06:23:58.510485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"967'1298","reported_seq":3502,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-03-04T06:23:59.412664+0000","last_change":"2024-03-04T06:23:58.421859+0000","last_active":"2024-03-04T06:23:59.412664+0000","last_peered":"2024-03-04T06:23:59.412664+0000","last_clean":"2024-03-04T06:23:59.412664+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:23:59.412664+0000","last_undegraded":"2024-03-04T06:23:59.412664+0000","last_fullsized":"2024-03-04T06:23:59.412664+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1298,"log_dups_size":0,"ondisk_log_size":1298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5110153,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":575,"num_read_kb":460193,"num_write":448,"num_write_kb":122380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"965'1044","reported_seq":3266,"reported_epoch":967,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:58.480829+0000","last_change":"2024-03-04T06:23:58.480829+0000","last_active":"2024-03-04T06:23:58.480829+0000","last_peered":"2024-03-04T06:23:58.480829+0000","last_clean":"2024-03-04T06:23:58.480829+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:23:58.480829+0000","last_undegraded":"2024-03-04T06:23:58.480829+0000","last_fullsized":"2024-03-04T06:23:58.480829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1044,"log_dups_size":0,"ondisk_log_size":1044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6000924,"num_objects":68,"num_object_clones":58,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":331,"num_read_kb":252049,"num_write":323,"num_write_kb":89446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"963'423","reported_seq":3064,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.496336+0000","last_change":"2024-03-04T06:23:58.502086+0000","last_active":"2024-03-04T06:23:58.496336+0000","last_peered":"2024-03-04T06:23:58.496336+0000","last_clean":"2024-03-04T06:23:58.496336+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:23:58.496336+0000","last_undegraded":"2024-03-04T06:23:58.496336+0000","last_fullsized":"2024-03-04T06:23:58.496336+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"965'1022","reported_seq":3345,"reported_epoch":967,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:58.490403+0000","last_change":"2024-03-04T06:23:58.490403+0000","last_active":"2024-03-04T06:23:58.490403+0000","last_peered":"2024-03-04T06:23:58.490403+0000","last_clean":"2024-03-04T06:23:58.490403+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:23:58.490403+0000","last_undegraded":"2024-03-04T06:23:58.490403+0000","last_fullsized":"2024-03-04T06:23:58.490403+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1022,"log_dups_size":0,"ondisk_log_size":1022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10152616,"num_objects":74,"num_object_clones":62,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":435,"num_read_kb":348132,"num_write":358,"num_write_kb":103998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2829,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.498088+0000","last_change":"2024-03-04T06:23:58.509420+0000","last_active":"2024-03-04T06:23:58.498088+0000","last_peered":"2024-03-04T06:23:58.498088+0000","last_clean":"2024-03-04T06:23:58.498088+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:23:58.498088+0000","last_undegraded":"2024-03-04T06:23:58.498088+0000","last_fullsized":"2024-03-04T06:23:58.498088+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"964'664","reported_seq":3142,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-03-04T06:23:59.411356+0000","last_change":"2024-03-04T06:23:58.421420+0000","last_active":"2024-03-04T06:23:59.411356+0000","last_peered":"2024-03-04T06:23:59.411356+0000","last_clean":"2024-03-04T06:23:59.411356+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:23:59.411356+0000","last_undegraded":"2024-03-04T06:23:59.411356+0000","last_fullsized":"2024-03-04T06:23:59.411356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":664,"log_dups_size":0,"ondisk_log_size":664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2574,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.480571+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:23:58.480571+0000","last_peered":"2024-03-04T06:23:58.480571+0000","last_clean":"2024-03-04T06:23:58.480571+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:23:58.480571+0000","last_undegraded":"2024-03-04T06:23:58.480571+0000","last_fullsized":"2024-03-04T06:23:58.480571+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2390,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-03-04T06:23:59.425932+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:23:59.425932+0000","last_peered":"2024-03-04T06:23:59.425932+0000","last_clean":"2024-03-04T06:23:59.425932+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:23:59.425932+0000","last_undegraded":"2024-03-04T06:23:59.425932+0000","last_fullsized":"2024-03-04T06:23:59.425932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"967'967","reported_seq":3339,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-03-04T06:23:59.426851+0000","last_change":"2024-03-04T06:23:59.426851+0000","last_active":"2024-03-04T06:23:59.426851+0000","last_peered":"2024-03-04T06:23:59.426851+0000","last_clean":"2024-03-04T06:23:59.426851+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:23:59.426851+0000","last_undegraded":"2024-03-04T06:23:59.426851+0000","last_fullsized":"2024-03-04T06:23:59.426851+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294535,"num_objects":73,"num_object_clones":59,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":442,"num_read_kb":363862,"num_write":323,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2373,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.483172+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:23:58.483172+0000","last_peered":"2024-03-04T06:23:58.483172+0000","last_clean":"2024-03-04T06:23:58.483172+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:23:58.483172+0000","last_undegraded":"2024-03-04T06:23:58.483172+0000","last_fullsized":"2024-03-04T06:23:58.483172+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"967'687","reported_seq":3240,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:24:00.524199+0000","last_change":"2024-03-04T06:24:00.531239+0000","last_active":"2024-03-04T06:24:00.524199+0000","last_peered":"2024-03-04T06:24:00.524199+0000","last_clean":"2024-03-04T06:24:00.524199+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:24:00.524199+0000","last_undegraded":"2024-03-04T06:24:00.524199+0000","last_fullsized":"2024-03-04T06:24:00.524199+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210995,"num_objects":52,"num_object_clones":43,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":312,"num_read_kb":250024,"num_write":224,"num_write_kb":64009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2390,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-03-04T06:23:59.417670+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:23:59.417670+0000","last_peered":"2024-03-04T06:23:59.417670+0000","last_clean":"2024-03-04T06:23:59.417670+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:23:59.417670+0000","last_undegraded":"2024-03-04T06:23:59.417670+0000","last_fullsized":"2024-03-04T06:23:59.417670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"967'1913","reported_seq":3871,"reported_epoch":967,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:23:58.694347+0000","last_change":"2024-03-04T06:23:58.491944+0000","last_active":"2024-03-04T06:23:58.694347+0000","last_peered":"2024-03-04T06:23:58.694347+0000","last_clean":"2024-03-04T06:23:58.694347+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:23:58.694347+0000","last_undegraded":"2024-03-04T06:23:58.694347+0000","last_fullsized":"2024-03-04T06:23:58.694347+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1913,"log_dups_size":0,"ondisk_log_size":1913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8270265,"num_objects":147,"num_object_clones":121,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":23,"num_read":812,"num_read_kb":640751,"num_write":679,"num_write_kb":182667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2373,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.470226+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:23:58.470226+0000","last_peered":"2024-03-04T06:23:58.470226+0000","last_clean":"2024-03-04T06:23:58.470226+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:23:58.470226+0000","last_undegraded":"2024-03-04T06:23:58.470226+0000","last_fullsized":"2024-03-04T06:23:58.470226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"967'2243","reported_seq":3902,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.507778+0000","last_change":"2024-03-04T06:23:58.561749+0000","last_active":"2024-03-04T06:23:58.507778+0000","last_peered":"2024-03-04T06:23:58.507778+0000","last_clean":"2024-03-04T06:23:58.507778+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:23:58.507778+0000","last_undegraded":"2024-03-04T06:23:58.507778+0000","last_fullsized":"2024-03-04T06:23:58.507778+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2243,"log_dups_size":0,"ondisk_log_size":2243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13262307,"num_objects":158,"num_object_clones":134,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":724,"num_read_kb":536055,"num_write":722,"num_write_kb":195272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"967'942","reported_seq":3336,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-03-04T06:23:59.428159+0000","last_change":"2024-03-04T06:23:59.428159+0000","last_active":"2024-03-04T06:23:59.428159+0000","last_peered":"2024-03-04T06:23:59.428159+0000","last_clean":"2024-03-04T06:23:59.428159+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:23:59.428159+0000","last_undegraded":"2024-03-04T06:23:59.428159+0000","last_fullsized":"2024-03-04T06:23:59.428159+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":942,"log_dups_size":0,"ondisk_log_size":942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865973,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":444,"num_read_kb":373927,"num_write":318,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2485,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.518976+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:23:58.518976+0000","last_peered":"2024-03-04T06:23:58.518976+0000","last_clean":"2024-03-04T06:23:58.518976+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:23:58.518976+0000","last_undegraded":"2024-03-04T06:23:58.518976+0000","last_fullsized":"2024-03-04T06:23:58.518976+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"966'1108","reported_seq":3436,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.525571+0000","last_change":"2024-03-04T06:23:58.541251+0000","last_active":"2024-03-04T06:23:58.525571+0000","last_peered":"2024-03-04T06:23:58.525571+0000","last_clean":"2024-03-04T06:23:58.525571+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:23:58.525571+0000","last_undegraded":"2024-03-04T06:23:58.525571+0000","last_fullsized":"2024-03-04T06:23:58.525571+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731550,"num_objects":84,"num_object_clones":68,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":390,"num_read_kb":282670,"num_write":363,"num_write_kb":98038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2441,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.492479+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:23:58.492479+0000","last_peered":"2024-03-04T06:23:58.492479+0000","last_clean":"2024-03-04T06:23:58.492479+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:23:58.492479+0000","last_undegraded":"2024-03-04T06:23:58.492479+0000","last_fullsized":"2024-03-04T06:23:58.492479+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"967'1467","reported_seq":3541,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:24:00.519863+0000","last_change":"2024-03-04T06:24:00.538811+0000","last_active":"2024-03-04T06:24:00.519863+0000","last_peered":"2024-03-04T06:24:00.519863+0000","last_clean":"2024-03-04T06:24:00.519863+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:24:00.519863+0000","last_undegraded":"2024-03-04T06:24:00.519863+0000","last_fullsized":"2024-03-04T06:24:00.519863+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1467,"log_dups_size":0,"ondisk_log_size":1467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5479907,"num_objects":101,"num_object_clones":86,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":506,"num_read_kb":396714,"num_write":470,"num_write_kb":130724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2485,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.505539+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:23:58.505539+0000","last_peered":"2024-03-04T06:23:58.505539+0000","last_clean":"2024-03-04T06:23:58.505539+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:23:58.505539+0000","last_undegraded":"2024-03-04T06:23:58.505539+0000","last_fullsized":"2024-03-04T06:23:58.505539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"967'1314","reported_seq":3623,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-03-04T06:23:58.516022+0000","last_change":"2024-03-04T06:23:58.587722+0000","last_active":"2024-03-04T06:23:58.516022+0000","last_peered":"2024-03-04T06:23:58.516022+0000","last_clean":"2024-03-04T06:23:58.516022+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:23:58.516022+0000","last_undegraded":"2024-03-04T06:23:58.516022+0000","last_fullsized":"2024-03-04T06:23:58.516022+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1314,"log_dups_size":0,"ondisk_log_size":1314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6371140,"num_objects":102,"num_object_clones":88,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":595,"num_read_kb":499612,"num_write":459,"num_write_kb":125332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2390,"reported_epoch":968,"state":"active+clean","last_fresh":"2024-03-04T06:23:59.422359+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:23:59.422359+0000","last_peered":"2024-03-04T06:23:59.422359+0000","last_clean":"2024-03-04T06:23:59.422359+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:23:59.422359+0000","last_undegraded":"2024-03-04T06:23:59.422359+0000","last_fullsized":"2024-03-04T06:23:59.422359+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":84936958,"num_objects":1235,"num_object_clones":1023,"num_object_copies":3705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":184,"num_read":6580,"num_read_kb":5234148,"num_write":5577,"num_write_kb":1522673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16802,"ondisk_log_size":16802,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706639,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208356,"kb_used_data":137116,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94163484,"statfs":{"total":96636764160,"available":96423407616,"internally_reserved":0,"allocated":140406784,"data_stored":167005412,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706639,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":164168,"kb_used_data":102656,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94207672,"statfs":{"total":96636764160,"available":96468656128,"internally_reserved":0,"allocated":105119744,"data_stored":124884908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":163584,"kb_used_data":106232,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94208256,"statfs":{"total":96636764160,"available":96469254144,"internally_reserved":0,"allocated":108781568,"data_stored":132464966,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":158516,"kb_used_data":101164,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94213324,"statfs":{"total":96636764160,"available":96474443776,"internally_reserved":0,"allocated":103591936,"data_stored":125326713,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:24:04.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:04.601+0000 7f019db5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:04.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:04.605+0000 7f019db5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:05.349 INFO:tasks.rados.rados.0.smithi064.stdout:3722: done (0 left) 2024-03-04T06:24:05.349 INFO:tasks.rados.rados.0.smithi064.stdout:3723: read oid 50 snap -1 2024-03-04T06:24:05.349 INFO:tasks.rados.rados.0.smithi064.stdout:3723: expect deleted 2024-03-04T06:24:05.349 INFO:tasks.rados.rados.0.smithi064.stdout:3724: read oid 48 snap -1 2024-03-04T06:24:05.350 INFO:tasks.rados.rados.0.smithi064.stdout:3724: expect (ObjNum 1639 snap 467 seq_num 1639) 2024-03-04T06:24:05.350 INFO:tasks.rados.rados.0.smithi064.stdout:3725: read oid 15 snap -1 2024-03-04T06:24:05.350 INFO:tasks.rados.rados.0.smithi064.stdout:3725: expect deleted 2024-03-04T06:24:05.350 INFO:tasks.rados.rados.0.smithi064.stdout:3726: write (excl) oid 11 current snap is 488 2024-03-04T06:24:05.350 INFO:tasks.rados.rados.0.smithi064.stdout:3726: seq_num 1682 ranges {669374=682831,2143301=126073} 2024-03-04T06:24:05.356 INFO:tasks.rados.rados.0.smithi064.stdout:3726: writing smithi06431743-11 from 669374 to 1352205 tid 1 2024-03-04T06:24:05.361 INFO:tasks.rados.rados.0.smithi064.stdout:3726: writing smithi06431743-11 from 2143301 to 2269374 tid 2 2024-03-04T06:24:05.361 INFO:tasks.rados.rados.0.smithi064.stdout:3727: read oid 42 snap -1 2024-03-04T06:24:05.362 INFO:tasks.rados.rados.0.smithi064.stdout:3727: expect deleted 2024-03-04T06:24:05.362 INFO:tasks.rados.rados.0.smithi064.stdout:3728: read oid 22 snap -1 2024-03-04T06:24:05.362 INFO:tasks.rados.rados.0.smithi064.stdout:3728: expect (ObjNum 1673 snap 482 seq_num 1673) 2024-03-04T06:24:05.362 INFO:tasks.rados.rados.0.smithi064.stdout:3729: snap_remove snap 454 2024-03-04T06:24:06.352 INFO:tasks.rados.rados.0.smithi064.stdout:3729: done (6 left) 2024-03-04T06:24:06.352 INFO:tasks.rados.rados.0.smithi064.stdout:3730: read oid 23 snap -1 2024-03-04T06:24:06.352 INFO:tasks.rados.rados.0.smithi064.stdout:3730: expect deleted 2024-03-04T06:24:06.352 INFO:tasks.rados.rados.0.smithi064.stdout:3731: snap_remove snap 485 2024-03-04T06:24:06.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:06.782 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:06.780+0000 7f023531d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:06.784 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:06.784+0000 7f023531d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:07.355 INFO:tasks.rados.rados.0.smithi064.stdout:3731: done (7 left) 2024-03-04T06:24:07.355 INFO:tasks.rados.rados.0.smithi064.stdout:3732: read oid 8 snap -1 2024-03-04T06:24:07.355 INFO:tasks.rados.rados.0.smithi064.stdout:3732: expect (ObjNum 1650 snap 471 seq_num 1650) 2024-03-04T06:24:07.355 INFO:tasks.rados.rados.0.smithi064.stdout:3733: write oid 36 current snap is 488 2024-03-04T06:24:07.355 INFO:tasks.rados.rados.0.smithi064.stdout:3733: seq_num 1683 ranges {686181=758080,2201783=84398} 2024-03-04T06:24:07.362 INFO:tasks.rados.rados.0.smithi064.stdout:3733: writing smithi06431743-36 from 686181 to 1444261 tid 1 2024-03-04T06:24:07.367 INFO:tasks.rados.rados.0.smithi064.stdout:3733: writing smithi06431743-36 from 2201783 to 2286181 tid 2 2024-03-04T06:24:07.367 INFO:tasks.rados.rados.0.smithi064.stdout:3734: read oid 6 snap -1 2024-03-04T06:24:07.367 INFO:tasks.rados.rados.0.smithi064.stdout:3734: expect (ObjNum 1539 snap 434 seq_num 1539) 2024-03-04T06:24:07.368 INFO:tasks.rados.rados.0.smithi064.stdout:3735: read oid 26 snap 486 2024-03-04T06:24:07.368 INFO:tasks.rados.rados.0.smithi064.stdout:3735: expect deleted 2024-03-04T06:24:07.368 INFO:tasks.rados.rados.0.smithi064.stdout:3736: read oid 32 snap -1 2024-03-04T06:24:07.368 INFO:tasks.rados.rados.0.smithi064.stdout:3736: expect (ObjNum 1670 snap 482 seq_num 1670) 2024-03-04T06:24:07.368 INFO:tasks.rados.rados.0.smithi064.stdout:3723: done (11 left) 2024-03-04T06:24:07.368 INFO:tasks.rados.rados.0.smithi064.stdout:3737: read oid 37 snap 480 2024-03-04T06:24:07.369 INFO:tasks.rados.rados.0.smithi064.stdout:3737: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:24:07.369 INFO:tasks.rados.rados.0.smithi064.stdout:3738: snap_remove snap 470 2024-03-04T06:24:08.363 INFO:tasks.rados.rados.0.smithi064.stdout:3738: done (12 left) 2024-03-04T06:24:08.363 INFO:tasks.rados.rados.0.smithi064.stdout:3739: snap_create 2024-03-04T06:24:08.363 INFO:tasks.rados.rados.0.smithi064.stdout:3728: expect (ObjNum 1673 snap 482 seq_num 1673) 2024-03-04T06:24:08.504 INFO:tasks.rados.rados.0.smithi064.stdout:3724: expect (ObjNum 1639 snap 467 seq_num 1639) 2024-03-04T06:24:08.676 INFO:tasks.rados.rados.0.smithi064.stdout:3726: finishing write tid 1 to smithi06431743-11 2024-03-04T06:24:08.676 INFO:tasks.rados.rados.0.smithi064.stdout:3726: finishing write tid 2 to smithi06431743-11 2024-03-04T06:24:08.676 INFO:tasks.rados.rados.0.smithi064.stdout:3726: finishing write tid 3 to smithi06431743-11 2024-03-04T06:24:08.676 INFO:tasks.rados.rados.0.smithi064.stdout:3726: finishing write tid 4 to smithi06431743-11 2024-03-04T06:24:08.676 INFO:tasks.rados.rados.0.smithi064.stdout:3726: oid 11 updating version 0 to 1304 2024-03-04T06:24:08.677 INFO:tasks.rados.rados.0.smithi064.stdout:3726: oid 11 updating version 1304 to 1305 2024-03-04T06:24:08.677 INFO:tasks.rados.rados.0.smithi064.stdout:3726: oid 11 version 1305 is already newer than 1303 2024-03-04T06:24:08.677 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 1305 (ObjNum 1682 snap 488 seq_num 1682) dirty exists 2024-03-04T06:24:08.677 INFO:tasks.rados.rados.0.smithi064.stdout:3726: left oid 11 (ObjNum 1682 snap 488 seq_num 1682) 2024-03-04T06:24:08.677 INFO:tasks.rados.rados.0.smithi064.stdout:3734: expect (ObjNum 1539 snap 434 seq_num 1539) 2024-03-04T06:24:08.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:08.879 INFO:tasks.rados.rados.0.smithi064.stdout:3736: expect (ObjNum 1670 snap 482 seq_num 1670) 2024-03-04T06:24:08.958 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:08.956+0000 7f94be91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:08.960 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:08.960+0000 7f94be91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:09.016 INFO:tasks.rados.rados.0.smithi064.stdout:3737: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:24:09.157 INFO:tasks.rados.rados.0.smithi064.stdout:3732: expect (ObjNum 1650 snap 471 seq_num 1650) 2024-03-04T06:24:09.295 INFO:tasks.rados.rados.0.smithi064.stdout:3733: finishing write tid 1 to smithi06431743-36 2024-03-04T06:24:09.295 INFO:tasks.rados.rados.0.smithi064.stdout:3733: finishing write tid 2 to smithi06431743-36 2024-03-04T06:24:09.295 INFO:tasks.rados.rados.0.smithi064.stdout:3733: finishing write tid 3 to smithi06431743-36 2024-03-04T06:24:09.295 INFO:tasks.rados.rados.0.smithi064.stdout:3733: finishing write tid 4 to smithi06431743-36 2024-03-04T06:24:09.295 INFO:tasks.rados.rados.0.smithi064.stdout:3733: oid 36 updating version 0 to 1113 2024-03-04T06:24:09.295 INFO:tasks.rados.rados.0.smithi064.stdout:3733: oid 36 updating version 1113 to 1115 2024-03-04T06:24:09.295 INFO:tasks.rados.rados.0.smithi064.stdout:3733: oid 36 version 1115 is already newer than 1115 2024-03-04T06:24:09.295 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 36 v 1115 (ObjNum 1683 snap 488 seq_num 1683) dirty exists 2024-03-04T06:24:09.296 INFO:tasks.rados.rados.0.smithi064.stdout:3733: left oid 36 (ObjNum 1683 snap 488 seq_num 1683) 2024-03-04T06:24:09.365 INFO:tasks.rados.rados.0.smithi064.stdout:3724: done (12 left) 2024-03-04T06:24:09.365 INFO:tasks.rados.rados.0.smithi064.stdout:3725: done (11 left) 2024-03-04T06:24:09.365 INFO:tasks.rados.rados.0.smithi064.stdout:3726: done (10 left) 2024-03-04T06:24:09.365 INFO:tasks.rados.rados.0.smithi064.stdout:3727: done (9 left) 2024-03-04T06:24:09.365 INFO:tasks.rados.rados.0.smithi064.stdout:3728: done (8 left) 2024-03-04T06:24:09.365 INFO:tasks.rados.rados.0.smithi064.stdout:3730: done (7 left) 2024-03-04T06:24:09.366 INFO:tasks.rados.rados.0.smithi064.stdout:3732: done (6 left) 2024-03-04T06:24:09.366 INFO:tasks.rados.rados.0.smithi064.stdout:3733: done (5 left) 2024-03-04T06:24:09.366 INFO:tasks.rados.rados.0.smithi064.stdout:3734: done (4 left) 2024-03-04T06:24:09.366 INFO:tasks.rados.rados.0.smithi064.stdout:3735: done (3 left) 2024-03-04T06:24:09.366 INFO:tasks.rados.rados.0.smithi064.stdout:3736: done (2 left) 2024-03-04T06:24:09.366 INFO:tasks.rados.rados.0.smithi064.stdout:3737: done (1 left) 2024-03-04T06:24:09.366 INFO:tasks.rados.rados.0.smithi064.stdout:3739: done (0 left) 2024-03-04T06:24:09.366 INFO:tasks.rados.rados.0.smithi064.stdout:3740: write oid 2 current snap is 489 2024-03-04T06:24:09.366 INFO:tasks.rados.rados.0.smithi064.stdout:3740: seq_num 1684 ranges {702988=433329,1860265=442723} 2024-03-04T06:24:09.370 INFO:tasks.rados.rados.0.smithi064.stdout:3740: writing smithi06431743-2 from 702988 to 1136317 tid 1 2024-03-04T06:24:09.378 INFO:tasks.rados.rados.0.smithi064.stdout:3740: writing smithi06431743-2 from 1860265 to 2302988 tid 2 2024-03-04T06:24:09.378 INFO:tasks.rados.rados.0.smithi064.stdout:3741: snap_create 2024-03-04T06:24:09.430 INFO:tasks.rados.rados.0.smithi064.stdout:3740: finishing write tid 1 to smithi06431743-2 2024-03-04T06:24:09.444 INFO:tasks.rados.rados.0.smithi064.stdout:3740: finishing write tid 2 to smithi06431743-2 2024-03-04T06:24:09.452 INFO:tasks.rados.rados.0.smithi064.stdout:3740: finishing write tid 3 to smithi06431743-2 2024-03-04T06:24:09.452 INFO:tasks.rados.rados.0.smithi064.stdout:3740: finishing write tid 4 to smithi06431743-2 2024-03-04T06:24:09.452 INFO:tasks.rados.rados.0.smithi064.stdout:3740: oid 2 updating version 0 to 1066 2024-03-04T06:24:09.453 INFO:tasks.rados.rados.0.smithi064.stdout:3740: oid 2 updating version 1066 to 1067 2024-03-04T06:24:09.453 INFO:tasks.rados.rados.0.smithi064.stdout:3740: oid 2 updating version 1067 to 1068 2024-03-04T06:24:09.453 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 1068 (ObjNum 1684 snap 489 seq_num 1684) dirty exists 2024-03-04T06:24:09.453 INFO:tasks.rados.rados.0.smithi064.stdout:3740: left oid 2 (ObjNum 1684 snap 489 seq_num 1684) 2024-03-04T06:24:09.604 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:24:09.604 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:24:09.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:24:09.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:09.708+0000 7f845079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:09.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:09.714+0000 7f845079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:09.904 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:24:09.904 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:24:09.916 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1461,"stamp":"2024-03-04T06:24:09.363822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87796700,"num_objects":1237,"num_object_clones":1023,"num_object_copies":3711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":183,"num_read":6697,"num_read_kb":5248503,"num_write":5718,"num_write_kb":1525358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16946,"ondisk_log_size":16946,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":693736,"kb_used_data":446280,"kb_used_omap":0,"kb_used_meta":247424,"kb_avail":376793624,"statfs":{"total":386547056640,"available":385836670976,"internally_reserved":0,"allocated":456990720,"data_stored":546632254,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":253362176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2269374,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":14,"num_read_kb":14264,"num_write":4,"num_write_kb":791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016008"},"pg_stats":[{"pgid":"3.f","version":"967'923","reported_seq":3296,"reported_epoch":974,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:05.388569+0000","last_change":"2024-03-04T06:24:05.388569+0000","last_active":"2024-03-04T06:24:05.388569+0000","last_peered":"2024-03-04T06:24:05.388569+0000","last_clean":"2024-03-04T06:24:05.388569+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:24:05.388569+0000","last_undegraded":"2024-03-04T06:24:05.388569+0000","last_fullsized":"2024-03-04T06:24:05.388569+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":923,"log_dups_size":0,"ondisk_log_size":923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294568,"num_objects":77,"num_object_clones":62,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":362,"num_read_kb":286556,"num_write":323,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3297,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.363858+0000","last_change":"2024-03-04T06:24:05.380173+0000","last_active":"2024-03-04T06:24:05.363858+0000","last_peered":"2024-03-04T06:24:05.363858+0000","last_clean":"2024-03-04T06:24:05.363858+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:24:05.363858+0000","last_undegraded":"2024-03-04T06:24:05.363858+0000","last_fullsized":"2024-03-04T06:24:05.363858+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":246,"num_read_kb":194849,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"975'1307","reported_seq":3523,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-03-04T06:24:06.361348+0000","last_change":"2024-03-04T06:24:06.392410+0000","last_active":"2024-03-04T06:24:06.361348+0000","last_peered":"2024-03-04T06:24:06.361348+0000","last_clean":"2024-03-04T06:24:06.361348+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:24:06.361348+0000","last_undegraded":"2024-03-04T06:24:06.361348+0000","last_fullsized":"2024-03-04T06:24:06.361348+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7379527,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":577,"num_read_kb":460194,"num_write":452,"num_write_kb":123171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"967'1050","reported_seq":3283,"reported_epoch":974,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:05.378739+0000","last_change":"2024-03-04T06:24:05.378739+0000","last_active":"2024-03-04T06:24:05.378739+0000","last_peered":"2024-03-04T06:24:05.378739+0000","last_clean":"2024-03-04T06:24:05.378739+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:24:05.378739+0000","last_undegraded":"2024-03-04T06:24:05.378739+0000","last_fullsized":"2024-03-04T06:24:05.378739+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6000924,"num_objects":68,"num_object_clones":58,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":331,"num_read_kb":252049,"num_write":323,"num_write_kb":89446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"974'425","reported_seq":3080,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.376795+0000","last_change":"2024-03-04T06:24:05.446078+0000","last_active":"2024-03-04T06:24:05.376795+0000","last_peered":"2024-03-04T06:24:05.376795+0000","last_clean":"2024-03-04T06:24:05.376795+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:24:05.376795+0000","last_undegraded":"2024-03-04T06:24:05.376795+0000","last_fullsized":"2024-03-04T06:24:05.376795+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"967'1026","reported_seq":3363,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.384768+0000","last_change":"2024-03-04T06:24:05.395693+0000","last_active":"2024-03-04T06:24:05.384768+0000","last_peered":"2024-03-04T06:24:05.384768+0000","last_clean":"2024-03-04T06:24:05.384768+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:24:05.384768+0000","last_undegraded":"2024-03-04T06:24:05.384768+0000","last_fullsized":"2024-03-04T06:24:05.384768+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1026,"log_dups_size":0,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10152616,"num_objects":74,"num_object_clones":62,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":435,"num_read_kb":348132,"num_write":358,"num_write_kb":103998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2845,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.385580+0000","last_change":"2024-03-04T06:24:05.393177+0000","last_active":"2024-03-04T06:24:05.385580+0000","last_peered":"2024-03-04T06:24:05.385580+0000","last_clean":"2024-03-04T06:24:05.385580+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:24:05.385580+0000","last_undegraded":"2024-03-04T06:24:05.385580+0000","last_fullsized":"2024-03-04T06:24:05.385580+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"975'666","reported_seq":3159,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-03-04T06:24:06.363491+0000","last_change":"2024-03-04T06:24:06.389142+0000","last_active":"2024-03-04T06:24:06.363491+0000","last_peered":"2024-03-04T06:24:06.363491+0000","last_clean":"2024-03-04T06:24:06.363491+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:24:06.363491+0000","last_undegraded":"2024-03-04T06:24:06.363491+0000","last_fullsized":"2024-03-04T06:24:06.363491+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2504639,"num_objects":42,"num_object_clones":33,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":283,"num_read_kb":255606,"num_write":186,"num_write_kb":50539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2589,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.363583+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:24:05.363583+0000","last_peered":"2024-03-04T06:24:05.363583+0000","last_clean":"2024-03-04T06:24:05.363583+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:24:05.363583+0000","last_undegraded":"2024-03-04T06:24:05.363583+0000","last_fullsized":"2024-03-04T06:24:05.363583+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2405,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-03-04T06:24:06.369676+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:24:06.369676+0000","last_peered":"2024-03-04T06:24:06.369676+0000","last_clean":"2024-03-04T06:24:06.369676+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:24:06.369676+0000","last_undegraded":"2024-03-04T06:24:06.369676+0000","last_fullsized":"2024-03-04T06:24:06.369676+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"975'975","reported_seq":3356,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-03-04T06:24:06.370091+0000","last_change":"2024-03-04T06:24:06.399544+0000","last_active":"2024-03-04T06:24:06.370091+0000","last_peered":"2024-03-04T06:24:06.370091+0000","last_clean":"2024-03-04T06:24:06.370091+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:24:06.370091+0000","last_undegraded":"2024-03-04T06:24:06.370091+0000","last_fullsized":"2024-03-04T06:24:06.370091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294535,"num_objects":73,"num_object_clones":59,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":442,"num_read_kb":363862,"num_write":323,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2388,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.391559+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:24:05.391559+0000","last_peered":"2024-03-04T06:24:05.391559+0000","last_clean":"2024-03-04T06:24:05.391559+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:24:05.391559+0000","last_undegraded":"2024-03-04T06:24:05.391559+0000","last_fullsized":"2024-03-04T06:24:05.391559+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"967'687","reported_seq":3259,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.422645+0000","last_change":"2024-03-04T06:24:05.371410+0000","last_active":"2024-03-04T06:24:05.422645+0000","last_peered":"2024-03-04T06:24:05.422645+0000","last_clean":"2024-03-04T06:24:05.422645+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:24:05.422645+0000","last_undegraded":"2024-03-04T06:24:05.422645+0000","last_fullsized":"2024-03-04T06:24:05.422645+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":687,"log_dups_size":0,"ondisk_log_size":687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210995,"num_objects":52,"num_object_clones":43,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":318,"num_read_kb":259030,"num_write":224,"num_write_kb":64009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2405,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-03-04T06:24:06.364465+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:24:06.364465+0000","last_peered":"2024-03-04T06:24:06.364465+0000","last_clean":"2024-03-04T06:24:06.364465+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:24:06.364465+0000","last_undegraded":"2024-03-04T06:24:06.364465+0000","last_fullsized":"2024-03-04T06:24:06.364465+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"974'1923","reported_seq":3889,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.378389+0000","last_change":"2024-03-04T06:24:05.471694+0000","last_active":"2024-03-04T06:24:05.378389+0000","last_peered":"2024-03-04T06:24:05.378389+0000","last_clean":"2024-03-04T06:24:05.378389+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:24:05.378389+0000","last_undegraded":"2024-03-04T06:24:05.378389+0000","last_fullsized":"2024-03-04T06:24:05.378389+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1923,"log_dups_size":0,"ondisk_log_size":1923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8270265,"num_objects":147,"num_object_clones":121,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":23,"num_read":812,"num_read_kb":640751,"num_write":679,"num_write_kb":182667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2388,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.381269+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:24:05.381269+0000","last_peered":"2024-03-04T06:24:05.381269+0000","last_clean":"2024-03-04T06:24:05.381269+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:24:05.381269+0000","last_undegraded":"2024-03-04T06:24:05.381269+0000","last_fullsized":"2024-03-04T06:24:05.381269+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"967'2243","reported_seq":3918,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.365385+0000","last_change":"2024-03-04T06:24:05.379198+0000","last_active":"2024-03-04T06:24:05.365385+0000","last_peered":"2024-03-04T06:24:05.365385+0000","last_clean":"2024-03-04T06:24:05.365385+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:24:05.365385+0000","last_undegraded":"2024-03-04T06:24:05.365385+0000","last_fullsized":"2024-03-04T06:24:05.365385+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2243,"log_dups_size":0,"ondisk_log_size":2243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13262307,"num_objects":158,"num_object_clones":134,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":724,"num_read_kb":536055,"num_write":722,"num_write_kb":195272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"975'944","reported_seq":3353,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-03-04T06:24:06.372201+0000","last_change":"2024-03-04T06:24:06.397447+0000","last_active":"2024-03-04T06:24:06.372201+0000","last_peered":"2024-03-04T06:24:06.372201+0000","last_clean":"2024-03-04T06:24:06.372201+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:24:06.372201+0000","last_undegraded":"2024-03-04T06:24:06.372201+0000","last_fullsized":"2024-03-04T06:24:06.372201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":944,"log_dups_size":0,"ondisk_log_size":944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865973,"num_objects":68,"num_object_clones":55,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":444,"num_read_kb":373927,"num_write":318,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2500,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.372229+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:24:05.372229+0000","last_peered":"2024-03-04T06:24:05.372229+0000","last_clean":"2024-03-04T06:24:05.372229+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:24:05.372229+0000","last_undegraded":"2024-03-04T06:24:05.372229+0000","last_fullsized":"2024-03-04T06:24:05.372229+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"966'1108","reported_seq":3451,"reported_epoch":974,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:05.379147+0000","last_change":"2024-03-04T06:24:05.379147+0000","last_active":"2024-03-04T06:24:05.379147+0000","last_peered":"2024-03-04T06:24:05.379147+0000","last_clean":"2024-03-04T06:24:05.379147+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:24:05.379147+0000","last_undegraded":"2024-03-04T06:24:05.379147+0000","last_fullsized":"2024-03-04T06:24:05.379147+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731550,"num_objects":84,"num_object_clones":68,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":390,"num_read_kb":282670,"num_write":363,"num_write_kb":98038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2456,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.376245+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:24:05.376245+0000","last_peered":"2024-03-04T06:24:05.376245+0000","last_clean":"2024-03-04T06:24:05.376245+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:24:05.376245+0000","last_undegraded":"2024-03-04T06:24:05.376245+0000","last_fullsized":"2024-03-04T06:24:05.376245+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"974'1469","reported_seq":3557,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.362269+0000","last_change":"2024-03-04T06:24:05.441652+0000","last_active":"2024-03-04T06:24:05.362269+0000","last_peered":"2024-03-04T06:24:05.362269+0000","last_clean":"2024-03-04T06:24:05.362269+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:24:05.362269+0000","last_undegraded":"2024-03-04T06:24:05.362269+0000","last_fullsized":"2024-03-04T06:24:05.362269+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1469,"log_dups_size":0,"ondisk_log_size":1469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5479907,"num_objects":101,"num_object_clones":86,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":506,"num_read_kb":396714,"num_write":470,"num_write_kb":130724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2500,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-03-04T06:24:05.367558+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:24:05.367558+0000","last_peered":"2024-03-04T06:24:05.367558+0000","last_clean":"2024-03-04T06:24:05.367558+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:24:05.367558+0000","last_undegraded":"2024-03-04T06:24:05.367558+0000","last_fullsized":"2024-03-04T06:24:05.367558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"967'1314","reported_seq":3641,"reported_epoch":974,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:05.366074+0000","last_change":"2024-03-04T06:24:05.366074+0000","last_active":"2024-03-04T06:24:05.366074+0000","last_peered":"2024-03-04T06:24:05.366074+0000","last_clean":"2024-03-04T06:24:05.366074+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:24:05.366074+0000","last_undegraded":"2024-03-04T06:24:05.366074+0000","last_fullsized":"2024-03-04T06:24:05.366074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1314,"log_dups_size":0,"ondisk_log_size":1314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6371140,"num_objects":102,"num_object_clones":88,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":601,"num_read_kb":504869,"num_write":459,"num_write_kb":125332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2405,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-03-04T06:24:06.368494+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:24:06.368494+0000","last_peered":"2024-03-04T06:24:06.368494+0000","last_clean":"2024-03-04T06:24:06.368494+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:24:06.368494+0000","last_undegraded":"2024-03-04T06:24:06.368494+0000","last_fullsized":"2024-03-04T06:24:06.368494+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":87206332,"num_objects":1235,"num_object_clones":1023,"num_object_copies":3705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":183,"num_read":6594,"num_read_kb":5248412,"num_write":5581,"num_write_kb":1523464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16849,"ondisk_log_size":16849,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":207868,"kb_used_data":136628,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94163972,"statfs":{"total":96636764160,"available":96423907328,"internally_reserved":0,"allocated":139907072,"data_stored":165460008,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":163672,"kb_used_data":102160,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94208168,"statfs":{"total":96636764160,"available":96469164032,"internally_reserved":0,"allocated":104611840,"data_stored":123335771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":163632,"kb_used_data":106280,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94208208,"statfs":{"total":96636764160,"available":96469204992,"internally_reserved":0,"allocated":108830720,"data_stored":132487364,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706643,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":158564,"kb_used_data":101212,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94213276,"statfs":{"total":96636764160,"available":96474394624,"internally_reserved":0,"allocated":103641088,"data_stored":125349111,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:24:10.368 INFO:tasks.rados.rados.0.smithi064.stdout:3740: done (1 left) 2024-03-04T06:24:10.368 INFO:tasks.rados.rados.0.smithi064.stdout:3741: done (0 left) 2024-03-04T06:24:10.368 INFO:tasks.rados.rados.0.smithi064.stdout:3742: read oid 17 snap -1 2024-03-04T06:24:10.368 INFO:tasks.rados.rados.0.smithi064.stdout:3742: expect deleted 2024-03-04T06:24:10.368 INFO:tasks.rados.rados.0.smithi064.stdout:3743: read oid 22 snap -1 2024-03-04T06:24:10.368 INFO:tasks.rados.rados.0.smithi064.stdout:3743: expect (ObjNum 1673 snap 482 seq_num 1673) 2024-03-04T06:24:10.368 INFO:tasks.rados.rados.0.smithi064.stdout:3744: write (excl) oid 49 current snap is 490 2024-03-04T06:24:10.368 INFO:tasks.rados.rados.0.smithi064.stdout:3744: seq_num 1685 ranges {719795=508578,1918747=401048} 2024-03-04T06:24:10.374 INFO:tasks.rados.rados.0.smithi064.stdout:3744: writing smithi06431743-49 from 719795 to 1228373 tid 1 2024-03-04T06:24:10.380 INFO:tasks.rados.rados.0.smithi064.stdout:3744: writing smithi06431743-49 from 1918747 to 2319795 tid 2 2024-03-04T06:24:10.381 INFO:tasks.rados.rados.0.smithi064.stdout:3745: delete oid 6 current snap is 490 2024-03-04T06:24:10.415 INFO:tasks.rados.rados.0.smithi064.stdout:3743: expect (ObjNum 1673 snap 482 seq_num 1673) 2024-03-04T06:24:10.541 INFO:tasks.rados.rados.0.smithi064.stdout:3744: finishing write tid 1 to smithi06431743-49 2024-03-04T06:24:10.541 INFO:tasks.rados.rados.0.smithi064.stdout:3744: finishing write tid 2 to smithi06431743-49 2024-03-04T06:24:10.541 INFO:tasks.rados.rados.0.smithi064.stdout:3744: finishing write tid 3 to smithi06431743-49 2024-03-04T06:24:10.541 INFO:tasks.rados.rados.0.smithi064.stdout:3744: finishing write tid 4 to smithi06431743-49 2024-03-04T06:24:10.541 INFO:tasks.rados.rados.0.smithi064.stdout:3744: oid 49 updating version 0 to 668 2024-03-04T06:24:10.541 INFO:tasks.rados.rados.0.smithi064.stdout:3744: oid 49 updating version 668 to 670 2024-03-04T06:24:10.541 INFO:tasks.rados.rados.0.smithi064.stdout:3744: oid 49 version 670 is already newer than 670 2024-03-04T06:24:10.541 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 670 (ObjNum 1685 snap 490 seq_num 1685) dirty exists 2024-03-04T06:24:10.542 INFO:tasks.rados.rados.0.smithi064.stdout:3744: left oid 49 (ObjNum 1685 snap 490 seq_num 1685) 2024-03-04T06:24:10.542 INFO:tasks.rados.rados.0.smithi064.stdout:3742: done (3 left) 2024-03-04T06:24:10.542 INFO:tasks.rados.rados.0.smithi064.stdout:3743: done (2 left) 2024-03-04T06:24:10.542 INFO:tasks.rados.rados.0.smithi064.stdout:3744: done (1 left) 2024-03-04T06:24:10.542 INFO:tasks.rados.rados.0.smithi064.stdout:3745: done (0 left) 2024-03-04T06:24:10.542 INFO:tasks.rados.rados.0.smithi064.stdout:3746: snap_remove snap 449 2024-03-04T06:24:11.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:11.135 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:11.134+0000 7ff890b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:11.138 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:11.138+0000 7ff890b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:11.370 INFO:tasks.rados.rados.0.smithi064.stdout:3746: done (0 left) 2024-03-04T06:24:11.370 INFO:tasks.rados.rados.0.smithi064.stdout:3747: write (excl) oid 39 current snap is 490 2024-03-04T06:24:11.371 INFO:tasks.rados.rados.0.smithi064.stdout:3747: seq_num 1687 ranges {753409=659076,1919358=434051} 2024-03-04T06:24:11.378 INFO:tasks.rados.rados.0.smithi064.stdout:3747: writing smithi06431743-39 from 753409 to 1412485 tid 1 2024-03-04T06:24:11.383 INFO:tasks.rados.rados.0.smithi064.stdout:3747: writing smithi06431743-39 from 1919358 to 2353409 tid 2 2024-03-04T06:24:11.383 INFO:tasks.rados.rados.0.smithi064.stdout:3748: snap_remove snap 464 2024-03-04T06:24:12.372 INFO:tasks.rados.rados.0.smithi064.stdout:3748: done (1 left) 2024-03-04T06:24:12.372 INFO:tasks.rados.rados.0.smithi064.stdout:3749: write (excl) oid 24 current snap is 490 2024-03-04T06:24:12.372 INFO:tasks.rados.rados.0.smithi064.stdout:3747: finishing write tid 1 to smithi06431743-39 2024-03-04T06:24:12.372 INFO:tasks.rados.rados.0.smithi064.stdout:3749: seq_num 1688 ranges {770216=450678,1694193=551926} 2024-03-04T06:24:12.378 INFO:tasks.rados.rados.0.smithi064.stdout:3749: writing smithi06431743-24 from 770216 to 1220894 tid 1 2024-03-04T06:24:12.385 INFO:tasks.rados.rados.0.smithi064.stdout:3749: writing smithi06431743-24 from 1694193 to 2246119 tid 2 2024-03-04T06:24:12.385 INFO:tasks.rados.rados.0.smithi064.stdout:3750: read oid 50 snap 475 2024-03-04T06:24:12.385 INFO:tasks.rados.rados.0.smithi064.stdout:3750: expect deleted 2024-03-04T06:24:12.385 INFO:tasks.rados.rados.0.smithi064.stdout:3751: write (excl) oid 16 current snap is 490 2024-03-04T06:24:12.385 INFO:tasks.rados.rados.0.smithi064.stdout:3747: finishing write tid 2 to smithi06431743-39 2024-03-04T06:24:12.385 INFO:tasks.rados.rados.0.smithi064.stdout:3747: finishing write tid 3 to smithi06431743-39 2024-03-04T06:24:12.385 INFO:tasks.rados.rados.0.smithi064.stdout:3747: finishing write tid 4 to smithi06431743-39 2024-03-04T06:24:12.385 INFO:tasks.rados.rados.0.smithi064.stdout:3747: oid 39 updating version 0 to 1318 2024-03-04T06:24:12.385 INFO:tasks.rados.rados.0.smithi064.stdout:3747: oid 39 version 1318 is already newer than 1316 2024-03-04T06:24:12.385 INFO:tasks.rados.rados.0.smithi064.stdout:3747: oid 39 version 1318 is already newer than 1317 2024-03-04T06:24:12.385 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 39 v 1318 (ObjNum 1687 snap 490 seq_num 1687) dirty exists 2024-03-04T06:24:12.386 INFO:tasks.rados.rados.0.smithi064.stdout:3747: left oid 39 (ObjNum 1687 snap 490 seq_num 1687) 2024-03-04T06:24:12.386 INFO:tasks.rados.rados.0.smithi064.stdout:3751: seq_num 1689 ranges {787023=525927,1752675=634348} 2024-03-04T06:24:12.392 INFO:tasks.rados.rados.0.smithi064.stdout:3751: writing smithi06431743-16 from 787023 to 1312950 tid 1 2024-03-04T06:24:12.399 INFO:tasks.rados.rados.0.smithi064.stdout:3751: writing smithi06431743-16 from 1752675 to 2387023 tid 2 2024-03-04T06:24:12.399 INFO:tasks.rados.rados.0.smithi064.stdout:3747: done (3 left) 2024-03-04T06:24:12.399 INFO:tasks.rados.rados.0.smithi064.stdout:3752: write (excl) oid 3 current snap is 490 2024-03-04T06:24:12.399 INFO:tasks.rados.rados.0.smithi064.stdout:3752: seq_num 1690 ranges {403830=601176,1694804=555595} 2024-03-04T06:24:12.405 INFO:tasks.rados.rados.0.smithi064.stdout:3752: writing smithi06431743-3 from 403830 to 1005006 tid 1 2024-03-04T06:24:12.413 INFO:tasks.rados.rados.0.smithi064.stdout:3752: writing smithi06431743-3 from 1694804 to 2250399 tid 2 2024-03-04T06:24:12.413 INFO:tasks.rados.rados.0.smithi064.stdout:3753: delete oid 10 current snap is 490 2024-03-04T06:24:12.514 INFO:tasks.rados.rados.0.smithi064.stdout:3750: done (4 left) 2024-03-04T06:24:12.514 INFO:tasks.rados.rados.0.smithi064.stdout:3753: done (3 left) 2024-03-04T06:24:12.514 INFO:tasks.rados.rados.0.smithi064.stdout:3754: write oid 33 current snap is 490 2024-03-04T06:24:12.514 INFO:tasks.rados.rados.0.smithi064.stdout:3754: seq_num 1692 ranges {437444=751674,1811768=559264} 2024-03-04T06:24:12.520 INFO:tasks.rados.rados.0.smithi064.stdout:3754: writing smithi06431743-33 from 437444 to 1189118 tid 1 2024-03-04T06:24:12.527 INFO:tasks.rados.rados.0.smithi064.stdout:3754: writing smithi06431743-33 from 1811768 to 2371032 tid 2 2024-03-04T06:24:12.527 INFO:tasks.rados.rados.0.smithi064.stdout:3755: write (excl) oid 15 current snap is 490 2024-03-04T06:24:12.528 INFO:tasks.rados.rados.0.smithi064.stdout:3751: finishing write tid 1 to smithi06431743-16 2024-03-04T06:24:12.528 INFO:tasks.rados.rados.0.smithi064.stdout:3755: seq_num 1693 ranges {454251=426923,1470250=702922} 2024-03-04T06:24:12.533 INFO:tasks.rados.rados.0.smithi064.stdout:3755: writing smithi06431743-15 from 454251 to 881174 tid 1 2024-03-04T06:24:12.542 INFO:tasks.rados.rados.0.smithi064.stdout:3755: writing smithi06431743-15 from 1470250 to 2173172 tid 2 2024-03-04T06:24:12.543 INFO:tasks.rados.rados.0.smithi064.stdout:3756: write (excl) oid 22 current snap is 490 2024-03-04T06:24:12.543 INFO:tasks.rados.rados.0.smithi064.stdout:3749: finishing write tid 1 to smithi06431743-24 2024-03-04T06:24:12.543 INFO:tasks.rados.rados.0.smithi064.stdout:3756: seq_num 1694 ranges {471058=502172,1412379=446580} 2024-03-04T06:24:12.549 INFO:tasks.rados.rados.0.smithi064.stdout:3756: writing smithi06431743-22 from 471058 to 973230 tid 1 2024-03-04T06:24:12.555 INFO:tasks.rados.rados.0.smithi064.stdout:3756: writing smithi06431743-22 from 1412379 to 1858959 tid 2 2024-03-04T06:24:12.555 INFO:tasks.rados.rados.0.smithi064.stdout:3757: snap_remove snap 458 2024-03-04T06:24:13.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:13.318 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:13.316+0000 7f01e2d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:13.321 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:13.321+0000 7f01e2d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:13.348 INFO:tasks.rados.rados.0.smithi064.stdout:3757: done (6 left) 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3758: delete oid 35 current snap is 490 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3751: finishing write tid 2 to smithi06431743-16 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3752: finishing write tid 1 to smithi06431743-3 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3754: finishing write tid 1 to smithi06431743-33 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3755: finishing write tid 1 to smithi06431743-15 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3749: finishing write tid 2 to smithi06431743-24 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3751: finishing write tid 3 to smithi06431743-16 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3751: finishing write tid 4 to smithi06431743-16 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3751: oid 16 updating version 0 to 1121 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3751: oid 16 version 1121 is already newer than 1120 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3751: oid 16 updating version 1121 to 1122 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 1122 (ObjNum 1689 snap 490 seq_num 1689) dirty exists 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3751: left oid 16 (ObjNum 1689 snap 490 seq_num 1689) 2024-03-04T06:24:13.349 INFO:tasks.rados.rados.0.smithi064.stdout:3752: finishing write tid 2 to smithi06431743-3 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3749: finishing write tid 3 to smithi06431743-24 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3749: finishing write tid 4 to smithi06431743-24 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3749: oid 24 updating version 0 to 1481 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3749: oid 24 version 1481 is already newer than 1480 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3749: oid 24 version 1481 is already newer than 1479 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 1481 (ObjNum 1688 snap 490 seq_num 1688) dirty exists 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3749: left oid 24 (ObjNum 1688 snap 490 seq_num 1688) 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3755: finishing write tid 2 to smithi06431743-15 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3754: finishing write tid 2 to smithi06431743-33 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3752: finishing write tid 3 to smithi06431743-3 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3752: finishing write tid 4 to smithi06431743-3 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3752: oid 3 updating version 0 to 990 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3752: oid 3 version 990 is already newer than 988 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:3752: oid 3 version 990 is already newer than 989 2024-03-04T06:24:13.350 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 990 (ObjNum 1690 snap 490 seq_num 1690) dirty exists 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3752: left oid 3 (ObjNum 1690 snap 490 seq_num 1690) 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3754: finishing write tid 3 to smithi06431743-33 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3755: finishing write tid 3 to smithi06431743-15 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3755: finishing write tid 4 to smithi06431743-15 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3755: oid 15 updating version 0 to 957 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3755: oid 15 version 957 is already newer than 956 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3755: oid 15 version 957 is already newer than 955 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 957 (ObjNum 1693 snap 490 seq_num 1693) dirty exists 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3755: left oid 15 (ObjNum 1693 snap 490 seq_num 1693) 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3754: finishing write tid 4 to smithi06431743-33 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3754: oid 33 updating version 0 to 1338 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3754: oid 33 updating version 1338 to 1339 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3754: oid 33 updating version 1339 to 1340 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 1340 (ObjNum 1692 snap 490 seq_num 1692) dirty exists 2024-03-04T06:24:13.351 INFO:tasks.rados.rados.0.smithi064.stdout:3754: left oid 33 (ObjNum 1692 snap 490 seq_num 1692) 2024-03-04T06:24:13.352 INFO:tasks.rados.rados.0.smithi064.stdout:3756: finishing write tid 1 to smithi06431743-22 2024-03-04T06:24:13.352 INFO:tasks.rados.rados.0.smithi064.stdout:3756: finishing write tid 2 to smithi06431743-22 2024-03-04T06:24:13.352 INFO:tasks.rados.rados.0.smithi064.stdout:3756: finishing write tid 3 to smithi06431743-22 2024-03-04T06:24:13.352 INFO:tasks.rados.rados.0.smithi064.stdout:3756: finishing write tid 4 to smithi06431743-22 2024-03-04T06:24:13.352 INFO:tasks.rados.rados.0.smithi064.stdout:3756: oid 22 updating version 0 to 1344 2024-03-04T06:24:13.352 INFO:tasks.rados.rados.0.smithi064.stdout:3756: oid 22 version 1344 is already newer than 1342 2024-03-04T06:24:13.352 INFO:tasks.rados.rados.0.smithi064.stdout:3756: oid 22 version 1344 is already newer than 1344 2024-03-04T06:24:13.352 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 1344 (ObjNum 1694 snap 490 seq_num 1694) dirty exists 2024-03-04T06:24:13.352 INFO:tasks.rados.rados.0.smithi064.stdout:3756: left oid 22 (ObjNum 1694 snap 490 seq_num 1694) 2024-03-04T06:24:13.399 INFO:tasks.rados.rados.0.smithi064.stdout:3749: done (6 left) 2024-03-04T06:24:13.399 INFO:tasks.rados.rados.0.smithi064.stdout:3751: done (5 left) 2024-03-04T06:24:13.400 INFO:tasks.rados.rados.0.smithi064.stdout:3752: done (4 left) 2024-03-04T06:24:13.400 INFO:tasks.rados.rados.0.smithi064.stdout:3754: done (3 left) 2024-03-04T06:24:13.400 INFO:tasks.rados.rados.0.smithi064.stdout:3755: done (2 left) 2024-03-04T06:24:13.400 INFO:tasks.rados.rados.0.smithi064.stdout:3756: done (1 left) 2024-03-04T06:24:13.400 INFO:tasks.rados.rados.0.smithi064.stdout:3758: done (0 left) 2024-03-04T06:24:13.400 INFO:tasks.rados.rados.0.smithi064.stdout:3759: delete oid 47 current snap is 490 2024-03-04T06:24:13.437 INFO:tasks.rados.rados.0.smithi064.stdout:3759: done (0 left) 2024-03-04T06:24:13.437 INFO:tasks.rados.rados.0.smithi064.stdout:3760: snap_remove snap 463 2024-03-04T06:24:14.383 INFO:tasks.rados.rados.0.smithi064.stdout:3760: done (0 left) 2024-03-04T06:24:14.383 INFO:tasks.rados.rados.0.smithi064.stdout:3761: read oid 37 snap -1 2024-03-04T06:24:14.384 INFO:tasks.rados.rados.0.smithi064.stdout:3761: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:24:14.384 INFO:tasks.rados.rados.0.smithi064.stdout:3762: snap_create 2024-03-04T06:24:14.539 INFO:tasks.rados.rados.0.smithi064.stdout:3761: expect (ObjNum 1415 snap 401 seq_num 1415) 2024-03-04T06:24:14.918 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:24:14.918 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:24:14.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:24:15.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:15.023+0000 7f8af795d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:15.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:15.029+0000 7f8af795d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:15.218 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:24:15.218 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:24:15.230 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1469,"stamp":"2024-03-04T06:24:14.915761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":85881531,"num_objects":1241,"num_object_clones":1026,"num_object_copies":3723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":183,"num_read":6732,"num_read_kb":5280915,"num_write":5732,"num_write_kb":1528572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17071,"ondisk_log_size":17071,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":693548,"kb_used_data":441932,"kb_used_omap":0,"kb_used_meta":251584,"kb_avail":376793812,"statfs":{"total":386547056640,"available":385836863488,"internally_reserved":0,"allocated":452538368,"data_stored":540962670,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":257622016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2638332,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":12701,"num_write":6,"num_write_kb":1533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001065"},"pg_stats":[{"pgid":"3.f","version":"977'931","reported_seq":3315,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.386384+0000","last_change":"2024-03-04T06:24:08.503935+0000","last_active":"2024-03-04T06:24:10.386384+0000","last_peered":"2024-03-04T06:24:10.386384+0000","last_clean":"2024-03-04T06:24:10.386384+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:24:10.386384+0000","last_undegraded":"2024-03-04T06:24:10.386384+0000","last_fullsized":"2024-03-04T06:24:10.386384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294568,"num_objects":77,"num_object_clones":62,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":362,"num_read_kb":286556,"num_write":323,"num_write_kb":85438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3318,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.382716+0000","last_change":"2024-03-04T06:24:08.398090+0000","last_active":"2024-03-04T06:24:10.382716+0000","last_peered":"2024-03-04T06:24:10.382716+0000","last_clean":"2024-03-04T06:24:10.382716+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:24:10.382716+0000","last_undegraded":"2024-03-04T06:24:10.382716+0000","last_fullsized":"2024-03-04T06:24:10.382716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":252,"num_read_kb":201855,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"980'1315","reported_seq":3541,"reported_epoch":980,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:11.375483+0000","last_change":"2024-03-04T06:24:11.375339+0000","last_active":"2024-03-04T06:24:11.375483+0000","last_peered":"2024-03-04T06:24:11.375483+0000","last_clean":"2024-03-04T06:24:11.375483+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:24:11.375483+0000","last_undegraded":"2024-03-04T06:24:11.375483+0000","last_fullsized":"2024-03-04T06:24:11.375483+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1315,"log_dups_size":0,"ondisk_log_size":1315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8792012,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":578,"num_read_kb":460194,"num_write":453,"num_write_kb":123815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"978'1068","reported_seq":3306,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.377691+0000","last_change":"2024-03-04T06:24:08.501576+0000","last_active":"2024-03-04T06:24:10.377691+0000","last_peered":"2024-03-04T06:24:10.377691+0000","last_clean":"2024-03-04T06:24:10.377691+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:24:10.377691+0000","last_undegraded":"2024-03-04T06:24:10.377691+0000","last_fullsized":"2024-03-04T06:24:10.377691+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1068,"log_dups_size":0,"ondisk_log_size":1068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4437906,"num_objects":69,"num_object_clones":59,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":332,"num_read_kb":252050,"num_write":327,"num_write_kb":90303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"975'427","reported_seq":3095,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-03-04T06:24:09.912663+0000","last_change":"2024-03-04T06:24:08.438465+0000","last_active":"2024-03-04T06:24:09.912663+0000","last_peered":"2024-03-04T06:24:09.912663+0000","last_clean":"2024-03-04T06:24:09.912663+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:24:09.912663+0000","last_undegraded":"2024-03-04T06:24:09.912663+0000","last_fullsized":"2024-03-04T06:24:09.912663+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"977'1032","reported_seq":3381,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.387154+0000","last_change":"2024-03-04T06:24:08.518668+0000","last_active":"2024-03-04T06:24:10.387154+0000","last_peered":"2024-03-04T06:24:10.387154+0000","last_clean":"2024-03-04T06:24:10.387154+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:24:10.387154+0000","last_undegraded":"2024-03-04T06:24:10.387154+0000","last_fullsized":"2024-03-04T06:24:10.387154+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1032,"log_dups_size":0,"ondisk_log_size":1032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10152616,"num_objects":74,"num_object_clones":62,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":435,"num_read_kb":348132,"num_write":358,"num_write_kb":103998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2863,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.385029+0000","last_change":"2024-03-04T06:24:08.491218+0000","last_active":"2024-03-04T06:24:10.385029+0000","last_peered":"2024-03-04T06:24:10.385029+0000","last_clean":"2024-03-04T06:24:10.385029+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:24:10.385029+0000","last_undegraded":"2024-03-04T06:24:10.385029+0000","last_fullsized":"2024-03-04T06:24:10.385029+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"980'674","reported_seq":3181,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-03-04T06:24:11.380200+0000","last_change":"2024-03-04T06:24:11.434029+0000","last_active":"2024-03-04T06:24:11.380200+0000","last_peered":"2024-03-04T06:24:11.380200+0000","last_clean":"2024-03-04T06:24:11.380200+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:24:11.380200+0000","last_undegraded":"2024-03-04T06:24:11.380200+0000","last_fullsized":"2024-03-04T06:24:11.380200+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":674,"log_dups_size":0,"ondisk_log_size":674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2319795,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":285,"num_read_kb":255607,"num_write":190,"num_write_kb":51428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2601,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-03-04T06:24:09.905015+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:24:09.905015+0000","last_peered":"2024-03-04T06:24:09.905015+0000","last_clean":"2024-03-04T06:24:09.905015+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:24:09.905015+0000","last_undegraded":"2024-03-04T06:24:09.905015+0000","last_fullsized":"2024-03-04T06:24:09.905015+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2420,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-03-04T06:24:11.387052+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:24:11.387052+0000","last_peered":"2024-03-04T06:24:11.387052+0000","last_clean":"2024-03-04T06:24:11.387052+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:24:11.387052+0000","last_undegraded":"2024-03-04T06:24:11.387052+0000","last_fullsized":"2024-03-04T06:24:11.387052+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"980'983","reported_seq":3374,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-03-04T06:24:11.390506+0000","last_change":"2024-03-04T06:24:11.431829+0000","last_active":"2024-03-04T06:24:11.390506+0000","last_peered":"2024-03-04T06:24:11.390506+0000","last_clean":"2024-03-04T06:24:11.390506+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:24:11.390506+0000","last_undegraded":"2024-03-04T06:24:11.390506+0000","last_fullsized":"2024-03-04T06:24:11.390506+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":983,"log_dups_size":0,"ondisk_log_size":983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3294535,"num_objects":73,"num_object_clones":59,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":442,"num_read_kb":363862,"num_write":323,"num_write_kb":86176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2403,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.388379+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:24:10.388379+0000","last_peered":"2024-03-04T06:24:10.388379+0000","last_clean":"2024-03-04T06:24:10.388379+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:24:10.388379+0000","last_undegraded":"2024-03-04T06:24:10.388379+0000","last_fullsized":"2024-03-04T06:24:10.388379+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"975'691","reported_seq":3274,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-03-04T06:24:09.907723+0000","last_change":"2024-03-04T06:24:08.438381+0000","last_active":"2024-03-04T06:24:09.907723+0000","last_peered":"2024-03-04T06:24:09.907723+0000","last_clean":"2024-03-04T06:24:09.907723+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:24:09.907723+0000","last_undegraded":"2024-03-04T06:24:09.907723+0000","last_fullsized":"2024-03-04T06:24:09.907723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":691,"log_dups_size":0,"ondisk_log_size":691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210995,"num_objects":52,"num_object_clones":43,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":318,"num_read_kb":259030,"num_write":224,"num_write_kb":64009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2420,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-03-04T06:24:11.381509+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:24:11.381509+0000","last_peered":"2024-03-04T06:24:11.381509+0000","last_clean":"2024-03-04T06:24:11.381509+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:24:11.381509+0000","last_undegraded":"2024-03-04T06:24:11.381509+0000","last_fullsized":"2024-03-04T06:24:11.381509+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"977'1937","reported_seq":3907,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-03-04T06:24:09.915291+0000","last_change":"2024-03-04T06:24:08.485962+0000","last_active":"2024-03-04T06:24:09.915291+0000","last_peered":"2024-03-04T06:24:09.915291+0000","last_clean":"2024-03-04T06:24:09.915291+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:24:09.915291+0000","last_undegraded":"2024-03-04T06:24:09.915291+0000","last_fullsized":"2024-03-04T06:24:09.915291+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1937,"log_dups_size":0,"ondisk_log_size":1937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8270265,"num_objects":147,"num_object_clones":121,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":23,"num_read":818,"num_read_kb":645885,"num_write":679,"num_write_kb":182667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2403,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.378505+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:24:10.378505+0000","last_peered":"2024-03-04T06:24:10.378505+0000","last_clean":"2024-03-04T06:24:10.378505+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:24:10.378505+0000","last_undegraded":"2024-03-04T06:24:10.378505+0000","last_fullsized":"2024-03-04T06:24:10.378505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"975'2253","reported_seq":3936,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.378212+0000","last_change":"2024-03-04T06:24:08.396713+0000","last_active":"2024-03-04T06:24:10.378212+0000","last_peered":"2024-03-04T06:24:10.378212+0000","last_clean":"2024-03-04T06:24:10.378212+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:24:10.378212+0000","last_undegraded":"2024-03-04T06:24:10.378212+0000","last_fullsized":"2024-03-04T06:24:10.378212+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2253,"log_dups_size":0,"ondisk_log_size":2253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13262307,"num_objects":158,"num_object_clones":134,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":724,"num_read_kb":536055,"num_write":722,"num_write_kb":195272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"980'950","reported_seq":3375,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-03-04T06:24:11.390443+0000","last_change":"2024-03-04T06:24:11.431929+0000","last_active":"2024-03-04T06:24:11.390443+0000","last_peered":"2024-03-04T06:24:11.390443+0000","last_clean":"2024-03-04T06:24:11.390443+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:24:11.390443+0000","last_undegraded":"2024-03-04T06:24:11.390443+0000","last_fullsized":"2024-03-04T06:24:11.390443+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":950,"log_dups_size":0,"ondisk_log_size":950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":69,"num_object_clones":56,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":450,"num_read_kb":381493,"num_write":319,"num_write_kb":86509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2515,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.392770+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:24:10.392770+0000","last_peered":"2024-03-04T06:24:10.392770+0000","last_clean":"2024-03-04T06:24:10.392770+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:24:10.392770+0000","last_undegraded":"2024-03-04T06:24:10.392770+0000","last_fullsized":"2024-03-04T06:24:10.392770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"977'1117","reported_seq":3477,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.395460+0000","last_change":"2024-03-04T06:24:08.460662+0000","last_active":"2024-03-04T06:24:10.395460+0000","last_peered":"2024-03-04T06:24:10.395460+0000","last_clean":"2024-03-04T06:24:10.395460+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:24:10.395460+0000","last_undegraded":"2024-03-04T06:24:10.395460+0000","last_fullsized":"2024-03-04T06:24:10.395460+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1117,"log_dups_size":0,"ondisk_log_size":1117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6017731,"num_objects":85,"num_object_clones":68,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":397,"num_read_kb":289852,"num_write":367,"num_write_kb":98862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2468,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-03-04T06:24:09.915167+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:24:09.915167+0000","last_peered":"2024-03-04T06:24:09.915167+0000","last_clean":"2024-03-04T06:24:09.915167+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:24:09.915167+0000","last_undegraded":"2024-03-04T06:24:09.915167+0000","last_fullsized":"2024-03-04T06:24:09.915167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"977'1475","reported_seq":3572,"reported_epoch":978,"state":"active+clean","last_fresh":"2024-03-04T06:24:09.903637+0000","last_change":"2024-03-04T06:24:08.480433+0000","last_active":"2024-03-04T06:24:09.903637+0000","last_peered":"2024-03-04T06:24:09.903637+0000","last_clean":"2024-03-04T06:24:09.903637+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:24:09.903637+0000","last_undegraded":"2024-03-04T06:24:09.903637+0000","last_fullsized":"2024-03-04T06:24:09.903637+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1475,"log_dups_size":0,"ondisk_log_size":1475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5479907,"num_objects":101,"num_object_clones":86,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":506,"num_read_kb":396714,"num_write":470,"num_write_kb":130724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2515,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.381268+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:24:10.381268+0000","last_peered":"2024-03-04T06:24:10.381268+0000","last_clean":"2024-03-04T06:24:10.381268+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:24:10.381268+0000","last_undegraded":"2024-03-04T06:24:10.381268+0000","last_fullsized":"2024-03-04T06:24:10.381268+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"977'1332","reported_seq":3663,"reported_epoch":979,"state":"active+clean","last_fresh":"2024-03-04T06:24:10.384320+0000","last_change":"2024-03-04T06:24:08.463419+0000","last_active":"2024-03-04T06:24:10.384320+0000","last_peered":"2024-03-04T06:24:10.384320+0000","last_clean":"2024-03-04T06:24:10.384320+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:24:10.384320+0000","last_undegraded":"2024-03-04T06:24:10.384320+0000","last_fullsized":"2024-03-04T06:24:10.384320+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1332,"log_dups_size":0,"ondisk_log_size":1332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6371140,"num_objects":102,"num_object_clones":88,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":607,"num_read_kb":510391,"num_write":459,"num_write_kb":125332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2420,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-03-04T06:24:11.389185+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:24:11.389185+0000","last_peered":"2024-03-04T06:24:11.389185+0000","last_clean":"2024-03-04T06:24:11.389185+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:24:11.389185+0000","last_undegraded":"2024-03-04T06:24:11.389185+0000","last_fullsized":"2024-03-04T06:24:11.389185+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":85291163,"num_objects":1239,"num_object_clones":1026,"num_object_copies":3717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":183,"num_read":6629,"num_read_kb":5280824,"num_write":5595,"num_write_kb":1526678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16974,"ondisk_log_size":16974,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706649,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208060,"kb_used_data":136820,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94163780,"statfs":{"total":96636764160,"available":96423710720,"internally_reserved":0,"allocated":140103680,"data_stored":166248083,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706649,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162240,"kb_used_data":100728,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94209600,"statfs":{"total":96636764160,"available":96470630400,"internally_reserved":0,"allocated":103145472,"data_stored":121692607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706648,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":165760,"kb_used_data":104248,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94206080,"statfs":{"total":96636764160,"available":96467025920,"internally_reserved":0,"allocated":106749952,"data_stored":129148540,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706648,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":157488,"kb_used_data":100136,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94214352,"statfs":{"total":96636764160,"available":96475496448,"internally_reserved":0,"allocated":102539264,"data_stored":123873440,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:24:15.393 INFO:tasks.rados.rados.0.smithi064.stdout:3761: done (1 left) 2024-03-04T06:24:15.393 INFO:tasks.rados.rados.0.smithi064.stdout:3762: done (0 left) 2024-03-04T06:24:15.393 INFO:tasks.rados.rados.0.smithi064.stdout:3763: write oid 15 current snap is 491 2024-03-04T06:24:15.393 INFO:tasks.rados.rados.0.smithi064.stdout:3763: seq_num 1697 ranges {521479=444272,1704178=710260} 2024-03-04T06:24:15.398 INFO:tasks.rados.rados.0.smithi064.stdout:3763: writing smithi06431743-15 from 521479 to 965751 tid 1 2024-03-04T06:24:15.406 INFO:tasks.rados.rados.0.smithi064.stdout:3763: writing smithi06431743-15 from 1704178 to 2414438 tid 2 2024-03-04T06:24:15.406 INFO:tasks.rados.rados.0.smithi064.stdout:3764: write (excl) oid 43 current snap is 491 2024-03-04T06:24:15.406 INFO:tasks.rados.rados.0.smithi064.stdout:3764: seq_num 1698 ranges {538286=519521,1762660=453918} 2024-03-04T06:24:15.412 INFO:tasks.rados.rados.0.smithi064.stdout:3764: writing smithi06431743-43 from 538286 to 1057807 tid 1 2024-03-04T06:24:15.419 INFO:tasks.rados.rados.0.smithi064.stdout:3764: writing smithi06431743-43 from 1762660 to 2216578 tid 2 2024-03-04T06:24:15.419 INFO:tasks.rados.rados.0.smithi064.stdout:3765: snap_remove snap 477 2024-03-04T06:24:15.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:15.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:15.552+0000 7f2d8be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:15.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:15.556+0000 7f2d8be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:16.393 INFO:tasks.rados.rados.0.smithi064.stdout:3765: done (2 left) 2024-03-04T06:24:16.393 INFO:tasks.rados.rados.0.smithi064.stdout:3766: write (excl) oid 19 current snap is 491 2024-03-04T06:24:16.393 INFO:tasks.rados.rados.0.smithi064.stdout:3766: seq_num 1699 ranges {555093=594770,1704789=713929} 2024-03-04T06:24:16.401 INFO:tasks.rados.rados.0.smithi064.stdout:3766: writing smithi06431743-19 from 555093 to 1149863 tid 1 2024-03-04T06:24:16.410 INFO:tasks.rados.rados.0.smithi064.stdout:3766: writing smithi06431743-19 from 1704789 to 2418718 tid 2 2024-03-04T06:24:16.410 INFO:tasks.rados.rados.0.smithi064.stdout:3767: snap_remove snap 486 2024-03-04T06:24:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:3767: done (3 left) 2024-03-04T06:24:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:3768: read oid 40 snap -1 2024-03-04T06:24:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:3768: expect (ObjNum 1624 snap 463 seq_num 1624) 2024-03-04T06:24:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:3769: read oid 45 snap -1 2024-03-04T06:24:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:3769: expect (ObjNum 1598 snap 453 seq_num 1598) 2024-03-04T06:24:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:3770: snap_create 2024-03-04T06:24:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:3763: finishing write tid 1 to smithi06431743-15 2024-03-04T06:24:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:3764: finishing write tid 1 to smithi06431743-43 2024-03-04T06:24:17.398 INFO:tasks.rados.rados.0.smithi064.stdout:3764: finishing write tid 2 to smithi06431743-43 2024-03-04T06:24:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:3763: finishing write tid 2 to smithi06431743-15 2024-03-04T06:24:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:3764: finishing write tid 3 to smithi06431743-43 2024-03-04T06:24:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:3764: finishing write tid 4 to smithi06431743-43 2024-03-04T06:24:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:3764: oid 43 updating version 0 to 942 2024-03-04T06:24:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:3764: oid 43 version 942 is already newer than 942 2024-03-04T06:24:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:3764: oid 43 version 942 is already newer than 940 2024-03-04T06:24:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 942 (ObjNum 1698 snap 491 seq_num 1698) dirty exists 2024-03-04T06:24:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:3764: left oid 43 (ObjNum 1698 snap 491 seq_num 1698) 2024-03-04T06:24:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:3763: finishing write tid 3 to smithi06431743-15 2024-03-04T06:24:17.399 INFO:tasks.rados.rados.0.smithi064.stdout:3763: finishing write tid 4 to smithi06431743-15 2024-03-04T06:24:17.400 INFO:tasks.rados.rados.0.smithi064.stdout:3763: oid 15 updating version 0 to 967 2024-03-04T06:24:17.400 INFO:tasks.rados.rados.0.smithi064.stdout:3763: oid 15 updating version 967 to 969 2024-03-04T06:24:17.400 INFO:tasks.rados.rados.0.smithi064.stdout:3763: oid 15 version 969 is already newer than 968 2024-03-04T06:24:17.400 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 969 (ObjNum 1697 snap 491 seq_num 1697) dirty exists 2024-03-04T06:24:17.400 INFO:tasks.rados.rados.0.smithi064.stdout:3763: left oid 15 (ObjNum 1697 snap 491 seq_num 1697) 2024-03-04T06:24:17.400 INFO:tasks.rados.rados.0.smithi064.stdout:3766: finishing write tid 1 to smithi06431743-19 2024-03-04T06:24:17.400 INFO:tasks.rados.rados.0.smithi064.stdout:3766: finishing write tid 2 to smithi06431743-19 2024-03-04T06:24:17.400 INFO:tasks.rados.rados.0.smithi064.stdout:3766: finishing write tid 3 to smithi06431743-19 2024-03-04T06:24:17.403 INFO:tasks.rados.rados.0.smithi064.stdout:3766: finishing write tid 4 to smithi06431743-19 2024-03-04T06:24:17.403 INFO:tasks.rados.rados.0.smithi064.stdout:3766: oid 19 updating version 0 to 2277 2024-03-04T06:24:17.403 INFO:tasks.rados.rados.0.smithi064.stdout:3766: oid 19 version 2277 is already newer than 2276 2024-03-04T06:24:17.403 INFO:tasks.rados.rados.0.smithi064.stdout:3766: oid 19 version 2277 is already newer than 2275 2024-03-04T06:24:17.403 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 19 v 2277 (ObjNum 1699 snap 491 seq_num 1699) dirty exists 2024-03-04T06:24:17.403 INFO:tasks.rados.rados.0.smithi064.stdout:3766: left oid 19 (ObjNum 1699 snap 491 seq_num 1699) 2024-03-04T06:24:17.468 INFO:tasks.rados.rados.0.smithi064.stdout:3768: expect (ObjNum 1624 snap 463 seq_num 1624) 2024-03-04T06:24:17.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:17.676 INFO:tasks.rados.rados.0.smithi064.stdout:3769: expect (ObjNum 1598 snap 453 seq_num 1598) 2024-03-04T06:24:17.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:17.726+0000 7f360862e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:17.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:17.731+0000 7f360862e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:18.398 INFO:tasks.rados.rados.0.smithi064.stdout:3763: done (5 left) 2024-03-04T06:24:18.398 INFO:tasks.rados.rados.0.smithi064.stdout:3764: done (4 left) 2024-03-04T06:24:18.398 INFO:tasks.rados.rados.0.smithi064.stdout:3766: done (3 left) 2024-03-04T06:24:18.398 INFO:tasks.rados.rados.0.smithi064.stdout:3768: done (2 left) 2024-03-04T06:24:18.399 INFO:tasks.rados.rados.0.smithi064.stdout:3769: done (1 left) 2024-03-04T06:24:18.399 INFO:tasks.rados.rados.0.smithi064.stdout:3770: done (0 left) 2024-03-04T06:24:18.399 INFO:tasks.rados.rados.0.smithi064.stdout:3771: write (excl) oid 40 current snap is 492 2024-03-04T06:24:18.399 INFO:tasks.rados.rados.0.smithi064.stdout:3771: seq_num 1700 ranges {571900=670019,1763271=457587} 2024-03-04T06:24:18.407 INFO:tasks.rados.rados.0.smithi064.stdout:3771: writing smithi06431743-40 from 571900 to 1241919 tid 1 2024-03-04T06:24:18.414 INFO:tasks.rados.rados.0.smithi064.stdout:3771: writing smithi06431743-40 from 1763271 to 2220858 tid 2 2024-03-04T06:24:18.414 INFO:tasks.rados.rados.0.smithi064.stdout:3772: write (excl) oid 23 current snap is 492 2024-03-04T06:24:18.414 INFO:tasks.rados.rados.0.smithi064.stdout:3772: seq_num 1701 ranges {588707=745268,1821753=717598} 2024-03-04T06:24:18.423 INFO:tasks.rados.rados.0.smithi064.stdout:3772: writing smithi06431743-23 from 588707 to 1333975 tid 1 2024-03-04T06:24:18.431 INFO:tasks.rados.rados.0.smithi064.stdout:3772: writing smithi06431743-23 from 1821753 to 2539351 tid 2 2024-03-04T06:24:18.431 INFO:tasks.rados.rados.0.smithi064.stdout:3773: delete oid 13 current snap is 492 2024-03-04T06:24:18.475 INFO:tasks.rados.rados.0.smithi064.stdout:3771: finishing write tid 1 to smithi06431743-40 2024-03-04T06:24:18.482 INFO:tasks.rados.rados.0.smithi064.stdout:3772: finishing write tid 1 to smithi06431743-23 2024-03-04T06:24:18.488 INFO:tasks.rados.rados.0.smithi064.stdout:3773: done (2 left) 2024-03-04T06:24:18.488 INFO:tasks.rados.rados.0.smithi064.stdout:3774: write (excl) oid 29 current snap is 492 2024-03-04T06:24:18.488 INFO:tasks.rados.rados.0.smithi064.stdout:3774: seq_num 1703 ranges {622321=612119,1938717=683604} 2024-03-04T06:24:18.495 INFO:tasks.rados.rados.0.smithi064.stdout:3774: writing smithi06431743-29 from 622321 to 1234440 tid 1 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3774: writing smithi06431743-29 from 1938717 to 2622321 tid 2 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3775: read oid 39 snap -1 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3775: expect (ObjNum 1687 snap 490 seq_num 1687) 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3776: snap_create 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3771: finishing write tid 2 to smithi06431743-40 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3771: finishing write tid 3 to smithi06431743-40 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3771: finishing write tid 4 to smithi06431743-40 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3771: oid 40 updating version 0 to 946 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3771: oid 40 version 946 is already newer than 946 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3771: oid 40 version 946 is already newer than 944 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 946 (ObjNum 1700 snap 492 seq_num 1700) dirty exists 2024-03-04T06:24:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3771: left oid 40 (ObjNum 1700 snap 492 seq_num 1700) 2024-03-04T06:24:18.505 INFO:tasks.rados.rados.0.smithi064.stdout:3772: finishing write tid 2 to smithi06431743-23 2024-03-04T06:24:18.516 INFO:tasks.rados.rados.0.smithi064.stdout:3772: finishing write tid 3 to smithi06431743-23 2024-03-04T06:24:18.516 INFO:tasks.rados.rados.0.smithi064.stdout:3772: finishing write tid 4 to smithi06431743-23 2024-03-04T06:24:18.516 INFO:tasks.rados.rados.0.smithi064.stdout:3772: oid 23 updating version 0 to 1003 2024-03-04T06:24:18.516 INFO:tasks.rados.rados.0.smithi064.stdout:3772: oid 23 version 1003 is already newer than 1002 2024-03-04T06:24:18.516 INFO:tasks.rados.rados.0.smithi064.stdout:3772: oid 23 version 1003 is already newer than 1001 2024-03-04T06:24:18.516 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 23 v 1003 (ObjNum 1701 snap 492 seq_num 1701) dirty exists 2024-03-04T06:24:18.516 INFO:tasks.rados.rados.0.smithi064.stdout:3772: left oid 23 (ObjNum 1701 snap 492 seq_num 1701) 2024-03-04T06:24:18.522 INFO:tasks.rados.rados.0.smithi064.stdout:3775: expect (ObjNum 1687 snap 490 seq_num 1687) 2024-03-04T06:24:18.611 INFO:tasks.rados.rados.0.smithi064.stdout:3774: finishing write tid 1 to smithi06431743-29 2024-03-04T06:24:18.611 INFO:tasks.rados.rados.0.smithi064.stdout:3774: finishing write tid 2 to smithi06431743-29 2024-03-04T06:24:18.611 INFO:tasks.rados.rados.0.smithi064.stdout:3774: finishing write tid 3 to smithi06431743-29 2024-03-04T06:24:18.611 INFO:tasks.rados.rados.0.smithi064.stdout:3774: finishing write tid 4 to smithi06431743-29 2024-03-04T06:24:18.611 INFO:tasks.rados.rados.0.smithi064.stdout:3774: oid 29 updating version 0 to 2289 2024-03-04T06:24:18.611 INFO:tasks.rados.rados.0.smithi064.stdout:3774: oid 29 updating version 2289 to 2291 2024-03-04T06:24:18.612 INFO:tasks.rados.rados.0.smithi064.stdout:3774: oid 29 version 2291 is already newer than 2290 2024-03-04T06:24:18.612 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 2291 (ObjNum 1703 snap 492 seq_num 1703) dirty exists 2024-03-04T06:24:18.612 INFO:tasks.rados.rados.0.smithi064.stdout:3774: left oid 29 (ObjNum 1703 snap 492 seq_num 1703) 2024-03-04T06:24:19.411 INFO:tasks.rados.rados.0.smithi064.stdout:3771: done (4 left) 2024-03-04T06:24:19.411 INFO:tasks.rados.rados.0.smithi064.stdout:3772: done (3 left) 2024-03-04T06:24:19.411 INFO:tasks.rados.rados.0.smithi064.stdout:3774: done (2 left) 2024-03-04T06:24:19.411 INFO:tasks.rados.rados.0.smithi064.stdout:3775: done (1 left) 2024-03-04T06:24:19.411 INFO:tasks.rados.rados.0.smithi064.stdout:3776: done (0 left) 2024-03-04T06:24:19.411 INFO:tasks.rados.rados.0.smithi064.stdout:3777: snap_remove snap 490 2024-03-04T06:24:19.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:19.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:19.923+0000 7ff16019d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:19.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:19.928+0000 7ff16019d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:20.232 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:24:20.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:24:20.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:20.337+0000 7fa3b0d14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:20.345 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:20.345+0000 7fa3b0d14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:20.410 INFO:tasks.rados.rados.0.smithi064.stdout:3777: done (0 left) 2024-03-04T06:24:20.410 INFO:tasks.rados.rados.0.smithi064.stdout:3778: snap_remove snap 481 2024-03-04T06:24:21.415 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:24:21.419 INFO:tasks.rados.rados.0.smithi064.stdout:3778: done (0 left) 2024-03-04T06:24:21.419 INFO:tasks.rados.rados.0.smithi064.stdout:3779: write oid 43 current snap is 493 2024-03-04T06:24:21.419 INFO:tasks.rados.rados.0.smithi064.stdout:3779: seq_num 1704 ranges {639128=687368,1997199=464925} 2024-03-04T06:24:21.426 INFO:tasks.rados.rados.0.smithi064.stdout:3779: writing smithi06431743-43 from 639128 to 1326496 tid 1 2024-03-04T06:24:21.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:24:21.433 INFO:tasks.rados.rados.0.smithi064.stdout:3779: writing smithi06431743-43 from 1997199 to 2462124 tid 2 2024-03-04T06:24:21.433 INFO:tasks.rados.rados.0.smithi064.stdout:3780: delete oid 37 current snap is 493 2024-03-04T06:24:21.514 INFO:tasks.rados.rados.0.smithi064.stdout:3779: finishing write tid 1 to smithi06431743-43 2024-03-04T06:24:21.528 INFO:tasks.rados.rados.0.smithi064.stdout:3779: finishing write tid 2 to smithi06431743-43 2024-03-04T06:24:21.533 INFO:tasks.rados.rados.0.smithi064.stdout:3780: done (1 left) 2024-03-04T06:24:21.533 INFO:tasks.rados.rados.0.smithi064.stdout:3781: read oid 3 snap 480 2024-03-04T06:24:21.533 INFO:tasks.rados.rados.0.smithi064.stdout:3781: expect (ObjNum 1452 snap 406 seq_num 1452) 2024-03-04T06:24:21.533 INFO:tasks.rados.rados.0.smithi064.stdout:3782: snap_create 2024-03-04T06:24:21.538 INFO:tasks.rados.rados.0.smithi064.stdout:3779: finishing write tid 3 to smithi06431743-43 2024-03-04T06:24:21.540 INFO:tasks.rados.rados.0.smithi064.stdout:3779: finishing write tid 4 to smithi06431743-43 2024-03-04T06:24:21.540 INFO:tasks.rados.rados.0.smithi064.stdout:3779: oid 43 updating version 0 to 952 2024-03-04T06:24:21.540 INFO:tasks.rados.rados.0.smithi064.stdout:3779: oid 43 version 952 is already newer than 951 2024-03-04T06:24:21.540 INFO:tasks.rados.rados.0.smithi064.stdout:3779: oid 43 version 952 is already newer than 951 2024-03-04T06:24:21.540 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 43 v 952 (ObjNum 1704 snap 493 seq_num 1704) dirty exists 2024-03-04T06:24:21.540 INFO:tasks.rados.rados.0.smithi064.stdout:3779: left oid 43 (ObjNum 1704 snap 493 seq_num 1704) 2024-03-04T06:24:21.583 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:21.577+0000 7f43d3c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:21.588 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:21.588+0000 7f43d3c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:21.605 INFO:tasks.rados.rados.0.smithi064.stdout:3781: expect (ObjNum 1452 snap 406 seq_num 1452) 2024-03-04T06:24:21.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:22.105 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:22.103+0000 7fce88b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:22.108 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:22.108+0000 7fce88b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:22.421 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:24:22.423 INFO:tasks.rados.rados.0.smithi064.stdout:3779: done (2 left) 2024-03-04T06:24:22.423 INFO:tasks.rados.rados.0.smithi064.stdout:3781: done (1 left) 2024-03-04T06:24:22.423 INFO:tasks.rados.rados.0.smithi064.stdout:3782: done (0 left) 2024-03-04T06:24:22.423 INFO:tasks.rados.rados.0.smithi064.stdout:3783: write (excl) oid 49 current snap is 494 2024-03-04T06:24:22.423 INFO:tasks.rados.rados.0.smithi064.stdout:3783: seq_num 1706 ranges {672742=437866,1714163=468594} 2024-03-04T06:24:22.430 INFO:tasks.rados.rados.0.smithi064.stdout:3783: writing smithi06431743-49 from 672742 to 1110608 tid 1 2024-03-04T06:24:22.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:24:22.438 INFO:tasks.rados.rados.0.smithi064.stdout:3783: writing smithi06431743-49 from 1714163 to 2182757 tid 2 2024-03-04T06:24:22.438 INFO:tasks.rados.rados.0.smithi064.stdout:3784: write (excl) oid 25 current snap is 494 2024-03-04T06:24:22.438 INFO:tasks.rados.rados.0.smithi064.stdout:3784: seq_num 1707 ranges {689549=513115,1656292=612252} 2024-03-04T06:24:22.444 INFO:tasks.rados.rados.0.smithi064.stdout:3784: writing smithi06431743-25 from 689549 to 1202664 tid 1 2024-03-04T06:24:22.450 INFO:tasks.rados.rados.0.smithi064.stdout:3784: writing smithi06431743-25 from 1656292 to 2268544 tid 2 2024-03-04T06:24:22.450 INFO:tasks.rados.rados.0.smithi064.stdout:3785: snap_remove snap 476 2024-03-04T06:24:22.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:22.560+0000 7f33d8c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:22.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:22.569+0000 7f33d8c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:23.347 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:24:23.350 INFO:tasks.rados.rados.0.smithi064.stdout:3785: done (2 left) 2024-03-04T06:24:23.350 INFO:tasks.rados.rados.0.smithi064.stdout:3786: delete oid 43 current snap is 494 2024-03-04T06:24:23.350 INFO:tasks.rados.rados.0.smithi064.stdout:3783: finishing write tid 1 to smithi06431743-49 2024-03-04T06:24:23.350 INFO:tasks.rados.rados.0.smithi064.stdout:3784: finishing write tid 1 to smithi06431743-25 2024-03-04T06:24:23.350 INFO:tasks.rados.rados.0.smithi064.stdout:3783: finishing write tid 2 to smithi06431743-49 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3784: finishing write tid 2 to smithi06431743-25 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3783: finishing write tid 3 to smithi06431743-49 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3783: finishing write tid 4 to smithi06431743-49 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3783: oid 49 updating version 0 to 688 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3783: oid 49 version 688 is already newer than 686 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3783: oid 49 version 688 is already newer than 688 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 688 (ObjNum 1706 snap 494 seq_num 1706) dirty exists 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3783: left oid 49 (ObjNum 1706 snap 494 seq_num 1706) 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3784: finishing write tid 3 to smithi06431743-25 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3784: finishing write tid 4 to smithi06431743-25 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3784: oid 25 updating version 0 to 1338 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3784: oid 25 version 1338 is already newer than 1336 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:3784: oid 25 version 1338 is already newer than 1337 2024-03-04T06:24:23.351 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 25 v 1338 (ObjNum 1707 snap 494 seq_num 1707) dirty exists 2024-03-04T06:24:23.352 INFO:tasks.rados.rados.0.smithi064.stdout:3784: left oid 25 (ObjNum 1707 snap 494 seq_num 1707) 2024-03-04T06:24:23.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:24:23.402 INFO:tasks.rados.rados.0.smithi064.stdout:3783: done (2 left) 2024-03-04T06:24:23.402 INFO:tasks.rados.rados.0.smithi064.stdout:3784: done (1 left) 2024-03-04T06:24:23.402 INFO:tasks.rados.rados.0.smithi064.stdout:3786: done (0 left) 2024-03-04T06:24:23.402 INFO:tasks.rados.rados.0.smithi064.stdout:3787: delete oid 24 current snap is 494 2024-03-04T06:24:23.466 INFO:tasks.rados.rados.0.smithi064.stdout:3787: done (0 left) 2024-03-04T06:24:23.466 INFO:tasks.rados.rados.0.smithi064.stdout:3788: read oid 38 snap 489 2024-03-04T06:24:23.466 INFO:tasks.rados.rados.0.smithi064.stdout:3788: expect (ObjNum 1597 snap 453 seq_num 1597) 2024-03-04T06:24:23.467 INFO:tasks.rados.rados.0.smithi064.stdout:3789: read oid 5 snap -1 2024-03-04T06:24:23.467 INFO:tasks.rados.rados.0.smithi064.stdout:3789: expect (ObjNum 1619 snap 462 seq_num 1619) 2024-03-04T06:24:23.467 INFO:tasks.rados.rados.0.smithi064.stdout:3790: write (excl) oid 46 current snap is 494 2024-03-04T06:24:23.467 INFO:tasks.rados.rados.0.smithi064.stdout:3790: seq_num 1710 ranges {739970=738862,2231738=475932} 2024-03-04T06:24:23.474 INFO:tasks.rados.rados.0.smithi064.stdout:3790: writing smithi06431743-46 from 739970 to 1478832 tid 1 2024-03-04T06:24:23.481 INFO:tasks.rados.rados.0.smithi064.stdout:3790: writing smithi06431743-46 from 2231738 to 2707670 tid 2 2024-03-04T06:24:23.481 INFO:tasks.rados.rados.0.smithi064.stdout:3791: delete oid 48 current snap is 494 2024-03-04T06:24:23.486 INFO:tasks.rados.rados.0.smithi064.stdout:3788: expect (ObjNum 1597 snap 453 seq_num 1597) 2024-03-04T06:24:23.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:23.526+0000 7f9ed8e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:23.536 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:23.535+0000 7f9ed8e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:23.609 INFO:tasks.rados.rados.0.smithi064.stdout:3790: finishing write tid 1 to smithi06431743-46 2024-03-04T06:24:23.609 INFO:tasks.rados.rados.0.smithi064.stdout:3788: done (3 left) 2024-03-04T06:24:23.609 INFO:tasks.rados.rados.0.smithi064.stdout:3791: done (2 left) 2024-03-04T06:24:23.609 INFO:tasks.rados.rados.0.smithi064.stdout:3792: snap_create 2024-03-04T06:24:23.609 INFO:tasks.rados.rados.0.smithi064.stdout:3789: expect (ObjNum 1619 snap 462 seq_num 1619) 2024-03-04T06:24:23.809 INFO:tasks.rados.rados.0.smithi064.stdout:3790: finishing write tid 2 to smithi06431743-46 2024-03-04T06:24:23.809 INFO:tasks.rados.rados.0.smithi064.stdout:3790: finishing write tid 3 to smithi06431743-46 2024-03-04T06:24:23.809 INFO:tasks.rados.rados.0.smithi064.stdout:3790: finishing write tid 4 to smithi06431743-46 2024-03-04T06:24:23.809 INFO:tasks.rados.rados.0.smithi064.stdout:3790: oid 46 updating version 0 to 1977 2024-03-04T06:24:23.809 INFO:tasks.rados.rados.0.smithi064.stdout:3790: oid 46 version 1977 is already newer than 1976 2024-03-04T06:24:23.809 INFO:tasks.rados.rados.0.smithi064.stdout:3790: oid 46 updating version 1977 to 1978 2024-03-04T06:24:23.809 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 1978 (ObjNum 1710 snap 494 seq_num 1710) dirty exists 2024-03-04T06:24:23.810 INFO:tasks.rados.rados.0.smithi064.stdout:3790: left oid 46 (ObjNum 1710 snap 494 seq_num 1710) 2024-03-04T06:24:24.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:24.283 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:24.281+0000 7ff941b07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:24.285 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:24.285+0000 7ff941b07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:24.426 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:24:24.428 INFO:tasks.rados.rados.0.smithi064.stdout:3789: done (2 left) 2024-03-04T06:24:24.428 INFO:tasks.rados.rados.0.smithi064.stdout:3790: done (1 left) 2024-03-04T06:24:24.429 INFO:tasks.rados.rados.0.smithi064.stdout:3792: done (0 left) 2024-03-04T06:24:24.429 INFO:tasks.rados.rados.0.smithi064.stdout:3793: read oid 35 snap -1 2024-03-04T06:24:24.429 INFO:tasks.rados.rados.0.smithi064.stdout:3793: expect deleted 2024-03-04T06:24:24.429 INFO:tasks.rados.rados.0.smithi064.stdout:3794: write (excl) oid 4 current snap is 495 2024-03-04T06:24:24.429 INFO:tasks.rados.rados.0.smithi064.stdout:3794: seq_num 1712 ranges {773584=605713,1948702=479601} 2024-03-04T06:24:24.436 INFO:tasks.rados.rados.0.smithi064.stdout:3794: writing smithi06431743-4 from 773584 to 1379297 tid 1 2024-03-04T06:24:24.441 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:24:24.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:24:24.444 INFO:tasks.rados.rados.0.smithi064.stdout:3794: writing smithi06431743-4 from 1948702 to 2428303 tid 2 2024-03-04T06:24:24.444 INFO:tasks.rados.rados.0.smithi064.stdout:3795: delete oid 31 current snap is 495 2024-03-04T06:24:24.472 INFO:tasks.rados.rados.0.smithi064.stdout:3793: done (2 left) 2024-03-04T06:24:24.472 INFO:tasks.rados.rados.0.smithi064.stdout:3795: done (1 left) 2024-03-04T06:24:24.472 INFO:tasks.rados.rados.0.smithi064.stdout:3796: snap_create 2024-03-04T06:24:24.490 INFO:tasks.rados.rados.0.smithi064.stdout:3794: finishing write tid 1 to smithi06431743-4 2024-03-04T06:24:24.509 INFO:tasks.rados.rados.0.smithi064.stdout:3794: finishing write tid 2 to smithi06431743-4 2024-03-04T06:24:24.514 INFO:tasks.rados.rados.0.smithi064.stdout:3794: finishing write tid 3 to smithi06431743-4 2024-03-04T06:24:24.515 INFO:tasks.rados.rados.0.smithi064.stdout:3794: finishing write tid 4 to smithi06431743-4 2024-03-04T06:24:24.515 INFO:tasks.rados.rados.0.smithi064.stdout:3794: oid 4 updating version 0 to 2312 2024-03-04T06:24:24.515 INFO:tasks.rados.rados.0.smithi064.stdout:3794: oid 4 version 2312 is already newer than 2312 2024-03-04T06:24:24.515 INFO:tasks.rados.rados.0.smithi064.stdout:3794: oid 4 version 2312 is already newer than 2310 2024-03-04T06:24:24.515 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 2312 (ObjNum 1712 snap 495 seq_num 1712) dirty exists 2024-03-04T06:24:24.515 INFO:tasks.rados.rados.0.smithi064.stdout:3794: left oid 4 (ObjNum 1712 snap 495 seq_num 1712) 2024-03-04T06:24:24.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:24.551+0000 7f1c5526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:24.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:24.557+0000 7f1c5526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:24.752 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:24:24.752 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:24:24.764 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1483,"stamp":"2024-03-04T06:24:24.427609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":90941622,"num_objects":1251,"num_object_clones":1035,"num_object_copies":3753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":181,"num_read":6785,"num_read_kb":5317580,"num_write":5785,"num_write_kb":1542711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17332,"ondisk_log_size":17332,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":700260,"kb_used_data":465284,"kb_used_omap":0,"kb_used_meta":234944,"kb_avail":376787100,"statfs":{"total":386547056640,"available":385829990400,"internally_reserved":0,"allocated":476450816,"data_stored":549737580,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":240582656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-470035,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":22091,"num_write":21,"num_write_kb":6028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018114"},"pg_stats":[{"pgid":"3.f","version":"987'946","reported_seq":3357,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.383464+0000","last_change":"2024-03-04T06:24:17.429988+0000","last_active":"2024-03-04T06:24:20.383464+0000","last_peered":"2024-03-04T06:24:20.383464+0000","last_clean":"2024-03-04T06:24:20.383464+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:24:20.383464+0000","last_undegraded":"2024-03-04T06:24:20.383464+0000","last_fullsized":"2024-03-04T06:24:20.383464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5110186,"num_objects":79,"num_object_clones":63,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":372,"num_read_kb":295544,"num_write":331,"num_write_kb":87492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3352,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.389504+0000","last_change":"2024-03-04T06:24:17.911839+0000","last_active":"2024-03-04T06:24:20.389504+0000","last_peered":"2024-03-04T06:24:20.389504+0000","last_clean":"2024-03-04T06:24:20.389504+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:24:20.389504+0000","last_undegraded":"2024-03-04T06:24:20.389504+0000","last_fullsized":"2024-03-04T06:24:20.389504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":258,"num_read_kb":210899,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"988'1330","reported_seq":3581,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.454280+0000","last_change":"2024-03-04T06:24:20.461395+0000","last_active":"2024-03-04T06:24:20.454280+0000","last_peered":"2024-03-04T06:24:20.454280+0000","last_clean":"2024-03-04T06:24:20.454280+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:24:20.454280+0000","last_undegraded":"2024-03-04T06:24:20.454280+0000","last_fullsized":"2024-03-04T06:24:20.454280+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1330,"log_dups_size":0,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9732936,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":591,"num_read_kb":473295,"num_write":456,"num_write_kb":124239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"986'1086","reported_seq":3338,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.369277+0000","last_change":"2024-03-04T06:24:17.448885+0000","last_active":"2024-03-04T06:24:20.369277+0000","last_peered":"2024-03-04T06:24:20.369277+0000","last_clean":"2024-03-04T06:24:20.369277+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:24:20.369277+0000","last_undegraded":"2024-03-04T06:24:20.369277+0000","last_fullsized":"2024-03-04T06:24:20.369277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1086,"log_dups_size":0,"ondisk_log_size":1086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4437906,"num_objects":69,"num_object_clones":59,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":332,"num_read_kb":252050,"num_write":327,"num_write_kb":90303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"986'431","reported_seq":3127,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.487068+0000","last_change":"2024-03-04T06:24:20.491366+0000","last_active":"2024-03-04T06:24:20.487068+0000","last_peered":"2024-03-04T06:24:20.487068+0000","last_clean":"2024-03-04T06:24:20.487068+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:24:20.487068+0000","last_undegraded":"2024-03-04T06:24:20.487068+0000","last_fullsized":"2024-03-04T06:24:20.487068+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"986'1050","reported_seq":3413,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.385012+0000","last_change":"2024-03-04T06:24:17.469951+0000","last_active":"2024-03-04T06:24:20.385012+0000","last_peered":"2024-03-04T06:24:20.385012+0000","last_clean":"2024-03-04T06:24:20.385012+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:24:20.385012+0000","last_undegraded":"2024-03-04T06:24:20.385012+0000","last_fullsized":"2024-03-04T06:24:20.385012+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6538715,"num_objects":75,"num_object_clones":63,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":435,"num_read_kb":348132,"num_write":359,"num_write_kb":103998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2894,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.385742+0000","last_change":"2024-03-04T06:24:17.428907+0000","last_active":"2024-03-04T06:24:20.385742+0000","last_peered":"2024-03-04T06:24:20.385742+0000","last_clean":"2024-03-04T06:24:20.385742+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:24:20.385742+0000","last_undegraded":"2024-03-04T06:24:20.385742+0000","last_fullsized":"2024-03-04T06:24:20.385742+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"986'682","reported_seq":3212,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.449851+0000","last_change":"2024-03-04T06:24:20.461610+0000","last_active":"2024-03-04T06:24:20.449851+0000","last_peered":"2024-03-04T06:24:20.449851+0000","last_clean":"2024-03-04T06:24:20.449851+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:24:20.449851+0000","last_undegraded":"2024-03-04T06:24:20.449851+0000","last_fullsized":"2024-03-04T06:24:20.449851+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2319795,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":285,"num_read_kb":255607,"num_write":190,"num_write_kb":51428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2626,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.476390+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:24:20.476390+0000","last_peered":"2024-03-04T06:24:20.476390+0000","last_clean":"2024-03-04T06:24:20.476390+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:24:20.476390+0000","last_undegraded":"2024-03-04T06:24:20.476390+0000","last_fullsized":"2024-03-04T06:24:20.476390+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2445,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.464983+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:24:20.464983+0000","last_peered":"2024-03-04T06:24:20.464983+0000","last_clean":"2024-03-04T06:24:20.464983+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:24:20.464983+0000","last_undegraded":"2024-03-04T06:24:20.464983+0000","last_fullsized":"2024-03-04T06:24:20.464983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"987'1003","reported_seq":3413,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.463197+0000","last_change":"2024-03-04T06:24:20.465704+0000","last_active":"2024-03-04T06:24:20.463197+0000","last_peered":"2024-03-04T06:24:20.463197+0000","last_clean":"2024-03-04T06:24:20.463197+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:24:20.463197+0000","last_undegraded":"2024-03-04T06:24:20.463197+0000","last_fullsized":"2024-03-04T06:24:20.463197+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1003,"log_dups_size":0,"ondisk_log_size":1003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8287072,"num_objects":73,"num_object_clones":59,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":446,"num_read_kb":363864,"num_write":331,"num_write_kb":88736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2428,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.382224+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:24:20.382224+0000","last_peered":"2024-03-04T06:24:20.382224+0000","last_clean":"2024-03-04T06:24:20.382224+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:24:20.382224+0000","last_undegraded":"2024-03-04T06:24:20.382224+0000","last_fullsized":"2024-03-04T06:24:20.382224+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"983'703","reported_seq":3306,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.475749+0000","last_change":"2024-03-04T06:24:20.485821+0000","last_active":"2024-03-04T06:24:20.475749+0000","last_peered":"2024-03-04T06:24:20.475749+0000","last_clean":"2024-03-04T06:24:20.475749+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:24:20.475749+0000","last_undegraded":"2024-03-04T06:24:20.475749+0000","last_fullsized":"2024-03-04T06:24:20.475749+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210995,"num_objects":52,"num_object_clones":43,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":318,"num_read_kb":259030,"num_write":224,"num_write_kb":64009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2445,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.453066+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:24:20.453066+0000","last_peered":"2024-03-04T06:24:20.453066+0000","last_clean":"2024-03-04T06:24:20.453066+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:24:20.453066+0000","last_undegraded":"2024-03-04T06:24:20.453066+0000","last_fullsized":"2024-03-04T06:24:20.453066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"986'1965","reported_seq":3939,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.488531+0000","last_change":"2024-03-04T06:24:20.491104+0000","last_active":"2024-03-04T06:24:20.488531+0000","last_peered":"2024-03-04T06:24:20.488531+0000","last_clean":"2024-03-04T06:24:20.488531+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:24:20.488531+0000","last_undegraded":"2024-03-04T06:24:20.488531+0000","last_fullsized":"2024-03-04T06:24:20.488531+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1965,"log_dups_size":0,"ondisk_log_size":1965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8270265,"num_objects":147,"num_object_clones":121,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":23,"num_read":818,"num_read_kb":645885,"num_write":679,"num_write_kb":182667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2428,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.372272+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:24:20.372272+0000","last_peered":"2024-03-04T06:24:20.372272+0000","last_clean":"2024-03-04T06:24:20.372272+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:24:20.372272+0000","last_undegraded":"2024-03-04T06:24:20.372272+0000","last_fullsized":"2024-03-04T06:24:20.372272+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"987'2291","reported_seq":3976,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.393032+0000","last_change":"2024-03-04T06:24:17.941746+0000","last_active":"2024-03-04T06:24:20.393032+0000","last_peered":"2024-03-04T06:24:20.393032+0000","last_clean":"2024-03-04T06:24:20.393032+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:24:20.393032+0000","last_undegraded":"2024-03-04T06:24:20.393032+0000","last_fullsized":"2024-03-04T06:24:20.393032+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2291,"log_dups_size":0,"ondisk_log_size":2291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8387947,"num_objects":161,"num_object_clones":137,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":728,"num_read_kb":536057,"num_write":731,"num_write_kb":197817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"989'973","reported_seq":3414,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.465730+0000","last_change":"2024-03-04T06:24:20.498982+0000","last_active":"2024-03-04T06:24:20.465730+0000","last_peered":"2024-03-04T06:24:20.465730+0000","last_clean":"2024-03-04T06:24:20.465730+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:24:20.465730+0000","last_undegraded":"2024-03-04T06:24:20.465730+0000","last_fullsized":"2024-03-04T06:24:20.465730+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521479,"num_objects":70,"num_object_clones":57,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":453,"num_read_kb":381495,"num_write":327,"num_write_kb":88741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2540,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.403884+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:24:20.403884+0000","last_peered":"2024-03-04T06:24:20.403884+0000","last_clean":"2024-03-04T06:24:20.403884+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:24:20.403884+0000","last_undegraded":"2024-03-04T06:24:20.403884+0000","last_fullsized":"2024-03-04T06:24:20.403884+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"985'1126","reported_seq":3509,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:18.456574+0000","last_change":"2024-03-04T06:24:17.919490+0000","last_active":"2024-03-04T06:24:18.456574+0000","last_peered":"2024-03-04T06:24:18.456574+0000","last_clean":"2024-03-04T06:24:18.456574+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:24:18.456574+0000","last_undegraded":"2024-03-04T06:24:18.456574+0000","last_fullsized":"2024-03-04T06:24:18.456574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1126,"log_dups_size":0,"ondisk_log_size":1126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8404754,"num_objects":85,"num_object_clones":68,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":399,"num_read_kb":289853,"num_write":371,"num_write_kb":99996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2493,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.485179+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:24:20.485179+0000","last_peered":"2024-03-04T06:24:20.485179+0000","last_clean":"2024-03-04T06:24:20.485179+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:24:20.485179+0000","last_undegraded":"2024-03-04T06:24:20.485179+0000","last_fullsized":"2024-03-04T06:24:20.485179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"986'1494","reported_seq":3608,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.478833+0000","last_change":"2024-03-04T06:24:20.486205+0000","last_active":"2024-03-04T06:24:20.478833+0000","last_peered":"2024-03-04T06:24:20.478833+0000","last_clean":"2024-03-04T06:24:20.478833+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:24:20.478833+0000","last_undegraded":"2024-03-04T06:24:20.478833+0000","last_fullsized":"2024-03-04T06:24:20.478833+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1494,"log_dups_size":0,"ondisk_log_size":1494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5748819,"num_objects":102,"num_object_clones":87,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":508,"num_read_kb":396715,"num_write":474,"num_write_kb":131704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2538,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:18.441476+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:24:18.441476+0000","last_peered":"2024-03-04T06:24:18.441476+0000","last_clean":"2024-03-04T06:24:18.441476+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:24:18.441476+0000","last_undegraded":"2024-03-04T06:24:18.441476+0000","last_fullsized":"2024-03-04T06:24:18.441476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"986'1366","reported_seq":3705,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.394729+0000","last_change":"2024-03-04T06:24:17.942012+0000","last_active":"2024-03-04T06:24:20.394729+0000","last_peered":"2024-03-04T06:24:20.394729+0000","last_clean":"2024-03-04T06:24:20.394729+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:24:20.394729+0000","last_undegraded":"2024-03-04T06:24:20.394729+0000","last_fullsized":"2024-03-04T06:24:20.394729+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1366,"log_dups_size":0,"ondisk_log_size":1366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6992999,"num_objects":104,"num_object_clones":90,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":616,"num_read_kb":515915,"num_write":467,"num_write_kb":127542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2445,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.461734+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:24:20.461734+0000","last_peered":"2024-03-04T06:24:20.461734+0000","last_clean":"2024-03-04T06:24:20.461734+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:24:20.461734+0000","last_undegraded":"2024-03-04T06:24:20.461734+0000","last_fullsized":"2024-03-04T06:24:20.461734+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":90351254,"num_objects":1249,"num_object_clones":1035,"num_object_copies":3747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":181,"num_read":6682,"num_read_kb":5317489,"num_write":5648,"num_write_kb":1540817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17235,"ondisk_log_size":17235,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706659,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":217424,"kb_used_data":146184,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94154416,"statfs":{"total":96636764160,"available":96414121984,"internally_reserved":0,"allocated":149692416,"data_stored":173333784,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706659,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":152548,"kb_used_data":107676,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94219292,"statfs":{"total":96636764160,"available":96480555008,"internally_reserved":0,"allocated":110260224,"data_stored":126139601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706658,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":167936,"kb_used_data":106424,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94203904,"statfs":{"total":96636764160,"available":96464797696,"internally_reserved":0,"allocated":108978176,"data_stored":125305437,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706658,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162352,"kb_used_data":105000,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94209488,"statfs":{"total":96636764160,"available":96470515712,"internally_reserved":0,"allocated":107520000,"data_stored":124958758,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:24:24.766 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:24:24.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:24.907+0000 7f83a68dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:24.914 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:24.913+0000 7f83a68dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:25.103 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:24:25.104 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:24:25.115 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1484,"stamp":"2024-03-04T06:24:24.917909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":90941622,"num_objects":1251,"num_object_clones":1035,"num_object_copies":3753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":181,"num_read":6785,"num_read_kb":5317580,"num_write":5785,"num_write_kb":1542711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17332,"ondisk_log_size":17332,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":700260,"kb_used_data":465284,"kb_used_omap":0,"kb_used_meta":234944,"kb_avail":376787100,"statfs":{"total":386547056640,"available":385829990400,"internally_reserved":0,"allocated":476450816,"data_stored":549737580,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":240582656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2588707,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":14,"num_read_kb":13101,"num_write":4,"num_write_kb":1429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000887"},"pg_stats":[{"pgid":"3.f","version":"987'946","reported_seq":3357,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.383464+0000","last_change":"2024-03-04T06:24:17.429988+0000","last_active":"2024-03-04T06:24:20.383464+0000","last_peered":"2024-03-04T06:24:20.383464+0000","last_clean":"2024-03-04T06:24:20.383464+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:24:20.383464+0000","last_undegraded":"2024-03-04T06:24:20.383464+0000","last_fullsized":"2024-03-04T06:24:20.383464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5110186,"num_objects":79,"num_object_clones":63,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":372,"num_read_kb":295544,"num_write":331,"num_write_kb":87492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"928'789","reported_seq":3352,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.389504+0000","last_change":"2024-03-04T06:24:17.911839+0000","last_active":"2024-03-04T06:24:20.389504+0000","last_peered":"2024-03-04T06:24:20.389504+0000","last_clean":"2024-03-04T06:24:20.389504+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:24:20.389504+0000","last_undegraded":"2024-03-04T06:24:20.389504+0000","last_fullsized":"2024-03-04T06:24:20.389504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6387386,"num_objects":62,"num_object_clones":45,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":258,"num_read_kb":210899,"num_write":255,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"988'1330","reported_seq":3581,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.454280+0000","last_change":"2024-03-04T06:24:20.461395+0000","last_active":"2024-03-04T06:24:20.454280+0000","last_peered":"2024-03-04T06:24:20.454280+0000","last_clean":"2024-03-04T06:24:20.454280+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:24:20.454280+0000","last_undegraded":"2024-03-04T06:24:20.454280+0000","last_fullsized":"2024-03-04T06:24:20.454280+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1330,"log_dups_size":0,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9732936,"num_objects":99,"num_object_clones":84,"num_object_copies":297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":591,"num_read_kb":473295,"num_write":456,"num_write_kb":124239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"986'1086","reported_seq":3338,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.369277+0000","last_change":"2024-03-04T06:24:17.448885+0000","last_active":"2024-03-04T06:24:20.369277+0000","last_peered":"2024-03-04T06:24:20.369277+0000","last_clean":"2024-03-04T06:24:20.369277+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:24:20.369277+0000","last_undegraded":"2024-03-04T06:24:20.369277+0000","last_fullsized":"2024-03-04T06:24:20.369277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1086,"log_dups_size":0,"ondisk_log_size":1086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4437906,"num_objects":69,"num_object_clones":59,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":332,"num_read_kb":252050,"num_write":327,"num_write_kb":90303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"986'431","reported_seq":3127,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.487068+0000","last_change":"2024-03-04T06:24:20.491366+0000","last_active":"2024-03-04T06:24:20.487068+0000","last_peered":"2024-03-04T06:24:20.487068+0000","last_clean":"2024-03-04T06:24:20.487068+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:24:20.487068+0000","last_undegraded":"2024-03-04T06:24:20.487068+0000","last_fullsized":"2024-03-04T06:24:20.487068+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"986'1050","reported_seq":3413,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.385012+0000","last_change":"2024-03-04T06:24:17.469951+0000","last_active":"2024-03-04T06:24:20.385012+0000","last_peered":"2024-03-04T06:24:20.385012+0000","last_clean":"2024-03-04T06:24:20.385012+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:24:20.385012+0000","last_undegraded":"2024-03-04T06:24:20.385012+0000","last_fullsized":"2024-03-04T06:24:20.385012+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1050,"log_dups_size":0,"ondisk_log_size":1050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6538715,"num_objects":75,"num_object_clones":63,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":435,"num_read_kb":348132,"num_write":359,"num_write_kb":103998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2894,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.385742+0000","last_change":"2024-03-04T06:24:17.428907+0000","last_active":"2024-03-04T06:24:20.385742+0000","last_peered":"2024-03-04T06:24:20.385742+0000","last_clean":"2024-03-04T06:24:20.385742+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:24:20.385742+0000","last_undegraded":"2024-03-04T06:24:20.385742+0000","last_fullsized":"2024-03-04T06:24:20.385742+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"986'682","reported_seq":3212,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.449851+0000","last_change":"2024-03-04T06:24:20.461610+0000","last_active":"2024-03-04T06:24:20.449851+0000","last_peered":"2024-03-04T06:24:20.449851+0000","last_clean":"2024-03-04T06:24:20.449851+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:24:20.449851+0000","last_undegraded":"2024-03-04T06:24:20.449851+0000","last_fullsized":"2024-03-04T06:24:20.449851+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2319795,"num_objects":43,"num_object_clones":34,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":285,"num_read_kb":255607,"num_write":190,"num_write_kb":51428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2626,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.476390+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:24:20.476390+0000","last_peered":"2024-03-04T06:24:20.476390+0000","last_clean":"2024-03-04T06:24:20.476390+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:24:20.476390+0000","last_undegraded":"2024-03-04T06:24:20.476390+0000","last_fullsized":"2024-03-04T06:24:20.476390+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2445,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.464983+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:24:20.464983+0000","last_peered":"2024-03-04T06:24:20.464983+0000","last_clean":"2024-03-04T06:24:20.464983+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:24:20.464983+0000","last_undegraded":"2024-03-04T06:24:20.464983+0000","last_fullsized":"2024-03-04T06:24:20.464983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"987'1003","reported_seq":3413,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.463197+0000","last_change":"2024-03-04T06:24:20.465704+0000","last_active":"2024-03-04T06:24:20.463197+0000","last_peered":"2024-03-04T06:24:20.463197+0000","last_clean":"2024-03-04T06:24:20.463197+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:24:20.463197+0000","last_undegraded":"2024-03-04T06:24:20.463197+0000","last_fullsized":"2024-03-04T06:24:20.463197+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1003,"log_dups_size":0,"ondisk_log_size":1003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8287072,"num_objects":73,"num_object_clones":59,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":446,"num_read_kb":363864,"num_write":331,"num_write_kb":88736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2428,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.382224+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:24:20.382224+0000","last_peered":"2024-03-04T06:24:20.382224+0000","last_clean":"2024-03-04T06:24:20.382224+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:24:20.382224+0000","last_undegraded":"2024-03-04T06:24:20.382224+0000","last_fullsized":"2024-03-04T06:24:20.382224+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"983'703","reported_seq":3306,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.475749+0000","last_change":"2024-03-04T06:24:20.485821+0000","last_active":"2024-03-04T06:24:20.475749+0000","last_peered":"2024-03-04T06:24:20.475749+0000","last_clean":"2024-03-04T06:24:20.475749+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:24:20.475749+0000","last_undegraded":"2024-03-04T06:24:20.475749+0000","last_fullsized":"2024-03-04T06:24:20.475749+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7210995,"num_objects":52,"num_object_clones":43,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":318,"num_read_kb":259030,"num_write":224,"num_write_kb":64009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2445,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.453066+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:24:20.453066+0000","last_peered":"2024-03-04T06:24:20.453066+0000","last_clean":"2024-03-04T06:24:20.453066+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:24:20.453066+0000","last_undegraded":"2024-03-04T06:24:20.453066+0000","last_fullsized":"2024-03-04T06:24:20.453066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"986'1965","reported_seq":3939,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.488531+0000","last_change":"2024-03-04T06:24:20.491104+0000","last_active":"2024-03-04T06:24:20.488531+0000","last_peered":"2024-03-04T06:24:20.488531+0000","last_clean":"2024-03-04T06:24:20.488531+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:24:20.488531+0000","last_undegraded":"2024-03-04T06:24:20.488531+0000","last_fullsized":"2024-03-04T06:24:20.488531+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1965,"log_dups_size":0,"ondisk_log_size":1965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8270265,"num_objects":147,"num_object_clones":121,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":23,"num_read":818,"num_read_kb":645885,"num_write":679,"num_write_kb":182667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2428,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.372272+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:24:20.372272+0000","last_peered":"2024-03-04T06:24:20.372272+0000","last_clean":"2024-03-04T06:24:20.372272+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:24:20.372272+0000","last_undegraded":"2024-03-04T06:24:20.372272+0000","last_fullsized":"2024-03-04T06:24:20.372272+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"987'2291","reported_seq":3976,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.393032+0000","last_change":"2024-03-04T06:24:17.941746+0000","last_active":"2024-03-04T06:24:20.393032+0000","last_peered":"2024-03-04T06:24:20.393032+0000","last_clean":"2024-03-04T06:24:20.393032+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:24:20.393032+0000","last_undegraded":"2024-03-04T06:24:20.393032+0000","last_fullsized":"2024-03-04T06:24:20.393032+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2291,"log_dups_size":0,"ondisk_log_size":2291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8387947,"num_objects":161,"num_object_clones":137,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":728,"num_read_kb":536057,"num_write":731,"num_write_kb":197817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"989'973","reported_seq":3414,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.465730+0000","last_change":"2024-03-04T06:24:20.498982+0000","last_active":"2024-03-04T06:24:20.465730+0000","last_peered":"2024-03-04T06:24:20.465730+0000","last_clean":"2024-03-04T06:24:20.465730+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:24:20.465730+0000","last_undegraded":"2024-03-04T06:24:20.465730+0000","last_fullsized":"2024-03-04T06:24:20.465730+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521479,"num_objects":70,"num_object_clones":57,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":453,"num_read_kb":381495,"num_write":327,"num_write_kb":88741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2540,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.403884+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:24:20.403884+0000","last_peered":"2024-03-04T06:24:20.403884+0000","last_clean":"2024-03-04T06:24:20.403884+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:24:20.403884+0000","last_undegraded":"2024-03-04T06:24:20.403884+0000","last_fullsized":"2024-03-04T06:24:20.403884+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"985'1126","reported_seq":3509,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:18.456574+0000","last_change":"2024-03-04T06:24:17.919490+0000","last_active":"2024-03-04T06:24:18.456574+0000","last_peered":"2024-03-04T06:24:18.456574+0000","last_clean":"2024-03-04T06:24:18.456574+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:24:18.456574+0000","last_undegraded":"2024-03-04T06:24:18.456574+0000","last_fullsized":"2024-03-04T06:24:18.456574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1126,"log_dups_size":0,"ondisk_log_size":1126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8404754,"num_objects":85,"num_object_clones":68,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":399,"num_read_kb":289853,"num_write":371,"num_write_kb":99996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2493,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.485179+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:24:20.485179+0000","last_peered":"2024-03-04T06:24:20.485179+0000","last_clean":"2024-03-04T06:24:20.485179+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:24:20.485179+0000","last_undegraded":"2024-03-04T06:24:20.485179+0000","last_fullsized":"2024-03-04T06:24:20.485179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"986'1494","reported_seq":3608,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.478833+0000","last_change":"2024-03-04T06:24:20.486205+0000","last_active":"2024-03-04T06:24:20.478833+0000","last_peered":"2024-03-04T06:24:20.478833+0000","last_clean":"2024-03-04T06:24:20.478833+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:24:20.478833+0000","last_undegraded":"2024-03-04T06:24:20.478833+0000","last_fullsized":"2024-03-04T06:24:20.478833+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1494,"log_dups_size":0,"ondisk_log_size":1494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5748819,"num_objects":102,"num_object_clones":87,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":508,"num_read_kb":396715,"num_write":474,"num_write_kb":131704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2538,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:18.441476+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:24:18.441476+0000","last_peered":"2024-03-04T06:24:18.441476+0000","last_clean":"2024-03-04T06:24:18.441476+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:24:18.441476+0000","last_undegraded":"2024-03-04T06:24:18.441476+0000","last_fullsized":"2024-03-04T06:24:18.441476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"986'1366","reported_seq":3705,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.394729+0000","last_change":"2024-03-04T06:24:17.942012+0000","last_active":"2024-03-04T06:24:20.394729+0000","last_peered":"2024-03-04T06:24:20.394729+0000","last_clean":"2024-03-04T06:24:20.394729+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:24:20.394729+0000","last_undegraded":"2024-03-04T06:24:20.394729+0000","last_fullsized":"2024-03-04T06:24:20.394729+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1366,"log_dups_size":0,"ondisk_log_size":1366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6992999,"num_objects":104,"num_object_clones":90,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":616,"num_read_kb":515915,"num_write":467,"num_write_kb":127542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2445,"reported_epoch":989,"state":"active+clean","last_fresh":"2024-03-04T06:24:20.461734+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:24:20.461734+0000","last_peered":"2024-03-04T06:24:20.461734+0000","last_clean":"2024-03-04T06:24:20.461734+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:24:20.461734+0000","last_undegraded":"2024-03-04T06:24:20.461734+0000","last_fullsized":"2024-03-04T06:24:20.461734+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":90351254,"num_objects":1249,"num_object_clones":1035,"num_object_copies":3747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":181,"num_read":6682,"num_read_kb":5317489,"num_write":5648,"num_write_kb":1540817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17235,"ondisk_log_size":17235,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706659,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":217424,"kb_used_data":146184,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94154416,"statfs":{"total":96636764160,"available":96414121984,"internally_reserved":0,"allocated":149692416,"data_stored":173333784,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706659,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":152548,"kb_used_data":107676,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94219292,"statfs":{"total":96636764160,"available":96480555008,"internally_reserved":0,"allocated":110260224,"data_stored":126139601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706658,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":167936,"kb_used_data":106424,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94203904,"statfs":{"total":96636764160,"available":96464797696,"internally_reserved":0,"allocated":108978176,"data_stored":125305437,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706658,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":162352,"kb_used_data":105000,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94209488,"statfs":{"total":96636764160,"available":96470515712,"internally_reserved":0,"allocated":107520000,"data_stored":124958758,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:24:25.116 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:24:25.116 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:24:25.117 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:24:25.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:24:25.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:25.220+0000 7f67c8a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:25.227 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:25.227+0000 7f67c8a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:25.240 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:25.240 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:25.240 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:25.240 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:25.240 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:25.240 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:25.240 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:25.240 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:25.240 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:25.241 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:25.241 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:25.241 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:25.242 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:25.243 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:24:25.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:25.345+0000 7fddbd06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:25.353 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:25.352+0000 7fddbd06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:25.365 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:25.366 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:25.366 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:25.366 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:25.366 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:25.366 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:25.366 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:25.367 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:25.367 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:25.367 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:25.367 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:25.367 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:25.369 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:25.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:24:25.440 INFO:tasks.rados.rados.0.smithi064.stdout:3794: done (1 left) 2024-03-04T06:24:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3796: done (0 left) 2024-03-04T06:24:25.441 INFO:tasks.rados.rados.0.smithi064.stdout:3797: snap_create 2024-03-04T06:24:25.488 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:25.479+0000 7fe8dff5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:25.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:25.492+0000 7fe8dff5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:25.508 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:25.509 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:25.511 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:25.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:24:25.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:25.615+0000 7eff37e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:25.622 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:25.622+0000 7eff37e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:25.635 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:25.635 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:25.635 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:25.635 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:25.636 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:25.636 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:25.636 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:25.636 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:25.636 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:25.636 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:25.636 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:25.636 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:25.638 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:25.638 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:26.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:26.439 INFO:tasks.rados.rados.0.smithi064.stdout:3797: done (0 left) 2024-03-04T06:24:26.439 INFO:tasks.rados.rados.0.smithi064.stdout:3798: snap_create 2024-03-04T06:24:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:26.460+0000 7f982199d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:26.464+0000 7f982199d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:27.439 INFO:tasks.rados.rados.0.smithi064.stdout:3798: done (0 left) 2024-03-04T06:24:27.439 INFO:tasks.rados.rados.0.smithi064.stdout:3799: delete oid 9 current snap is 498 2024-03-04T06:24:27.489 INFO:tasks.rados.rados.0.smithi064.stdout:3799: done (0 left) 2024-03-04T06:24:27.489 INFO:tasks.rados.rados.0.smithi064.stdout:3800: snap_create 2024-03-04T06:24:28.453 INFO:tasks.rados.rados.0.smithi064.stdout:3800: done (0 left) 2024-03-04T06:24:28.454 INFO:tasks.rados.rados.0.smithi064.stdout:3801: read oid 7 snap 484 2024-03-04T06:24:28.454 INFO:tasks.rados.rados.0.smithi064.stdout:3801: expect (ObjNum 1646 snap 470 seq_num 1646) 2024-03-04T06:24:28.454 INFO:tasks.rados.rados.0.smithi064.stdout:3802: snap_create 2024-03-04T06:24:28.526 INFO:tasks.rados.rados.0.smithi064.stdout:3801: expect (ObjNum 1646 snap 470 seq_num 1646) 2024-03-04T06:24:28.534 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:28.649 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:28.647+0000 7fb20a42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:28.652 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:28.652+0000 7fb20a42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:29.455 INFO:tasks.rados.rados.0.smithi064.stdout:3801: done (1 left) 2024-03-04T06:24:29.455 INFO:tasks.rados.rados.0.smithi064.stdout:3802: done (0 left) 2024-03-04T06:24:29.456 INFO:tasks.rados.rados.0.smithi064.stdout:3803: write (excl) oid 10 current snap is 500 2024-03-04T06:24:29.456 INFO:tasks.rados.rados.0.smithi064.stdout:3803: seq_num 1715 ranges {424005=431460,1607795=626928} 2024-03-04T06:24:29.460 INFO:tasks.rados.rados.0.smithi064.stdout:3803: writing smithi06431743-10 from 424005 to 855465 tid 1 2024-03-04T06:24:29.468 INFO:tasks.rados.rados.0.smithi064.stdout:3803: writing smithi06431743-10 from 1607795 to 2234723 tid 2 2024-03-04T06:24:29.468 INFO:tasks.rados.rados.0.smithi064.stdout:3804: delete oid 40 current snap is 500 2024-03-04T06:24:29.525 INFO:tasks.rados.rados.0.smithi064.stdout:3803: finishing write tid 1 to smithi06431743-10 2024-03-04T06:24:29.543 INFO:tasks.rados.rados.0.smithi064.stdout:3804: done (1 left) 2024-03-04T06:24:29.544 INFO:tasks.rados.rados.0.smithi064.stdout:3805: snap_remove snap 474 2024-03-04T06:24:30.457 INFO:tasks.rados.rados.0.smithi064.stdout:3805: done (1 left) 2024-03-04T06:24:30.457 INFO:tasks.rados.rados.0.smithi064.stdout:3806: delete oid 46 current snap is 500 2024-03-04T06:24:30.457 INFO:tasks.rados.rados.0.smithi064.stdout:3803: finishing write tid 2 to smithi06431743-10 2024-03-04T06:24:30.457 INFO:tasks.rados.rados.0.smithi064.stdout:3803: finishing write tid 3 to smithi06431743-10 2024-03-04T06:24:30.458 INFO:tasks.rados.rados.0.smithi064.stdout:3803: finishing write tid 4 to smithi06431743-10 2024-03-04T06:24:30.458 INFO:tasks.rados.rados.0.smithi064.stdout:3803: oid 10 updating version 0 to 1056 2024-03-04T06:24:30.458 INFO:tasks.rados.rados.0.smithi064.stdout:3803: oid 10 updating version 1056 to 1057 2024-03-04T06:24:30.458 INFO:tasks.rados.rados.0.smithi064.stdout:3803: oid 10 version 1057 is already newer than 1055 2024-03-04T06:24:30.458 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 10 v 1057 (ObjNum 1715 snap 500 seq_num 1715) dirty exists 2024-03-04T06:24:30.458 INFO:tasks.rados.rados.0.smithi064.stdout:3803: left oid 10 (ObjNum 1715 snap 500 seq_num 1715) 2024-03-04T06:24:30.593 INFO:tasks.rados.rados.0.smithi064.stdout:3803: done (1 left) 2024-03-04T06:24:30.594 INFO:tasks.rados.rados.0.smithi064.stdout:3806: done (0 left) 2024-03-04T06:24:30.594 INFO:tasks.rados.rados.0.smithi064.stdout:3807: read oid 37 snap -1 2024-03-04T06:24:30.594 INFO:tasks.rados.rados.0.smithi064.stdout:3807: expect deleted 2024-03-04T06:24:30.594 INFO:tasks.rados.rados.0.smithi064.stdout:3808: write (excl) oid 2 current snap is 500 2024-03-04T06:24:30.594 INFO:tasks.rados.rados.0.smithi064.stdout:3808: seq_num 1718 ranges {474426=657207,1783241=774255} 2024-03-04T06:24:30.600 INFO:tasks.rados.rados.0.smithi064.stdout:3808: writing smithi06431743-2 from 474426 to 1131633 tid 1 2024-03-04T06:24:30.609 INFO:tasks.rados.rados.0.smithi064.stdout:3808: writing smithi06431743-2 from 1783241 to 2557496 tid 2 2024-03-04T06:24:30.609 INFO:tasks.rados.rados.0.smithi064.stdout:3809: delete oid 26 current snap is 500 2024-03-04T06:24:30.617 INFO:tasks.rados.rados.0.smithi064.stdout:3807: done (2 left) 2024-03-04T06:24:30.618 INFO:tasks.rados.rados.0.smithi064.stdout:3809: done (1 left) 2024-03-04T06:24:30.618 INFO:tasks.rados.rados.0.smithi064.stdout:3810: write oid 46 current snap is 500 2024-03-04T06:24:30.618 INFO:tasks.rados.rados.0.smithi064.stdout:3810: seq_num 1720 ranges {508040=524058,1500205=777924} 2024-03-04T06:24:30.623 INFO:tasks.rados.rados.0.smithi064.stdout:3810: writing smithi06431743-46 from 508040 to 1032098 tid 1 2024-03-04T06:24:30.630 INFO:tasks.rados.rados.0.smithi064.stdout:3810: writing smithi06431743-46 from 1500205 to 2278129 tid 2 2024-03-04T06:24:30.630 INFO:tasks.rados.rados.0.smithi064.stdout:3811: write oid 16 current snap is 500 2024-03-04T06:24:30.630 INFO:tasks.rados.rados.0.smithi064.stdout:3811: seq_num 1721 ranges {524847=599307,1558687=637935,2644500=280347} 2024-03-04T06:24:30.636 INFO:tasks.rados.rados.0.smithi064.stdout:3811: writing smithi06431743-16 from 524847 to 1124154 tid 1 2024-03-04T06:24:30.639 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:24:30.640 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:24:30.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:24:30.643 INFO:tasks.rados.rados.0.smithi064.stdout:3811: writing smithi06431743-16 from 1558687 to 2196622 tid 2 2024-03-04T06:24:30.646 INFO:tasks.rados.rados.0.smithi064.stdout:3811: writing smithi06431743-16 from 2644500 to 2924847 tid 3 2024-03-04T06:24:30.647 INFO:tasks.rados.rados.0.smithi064.stdout:3812: write (excl) oid 35 current snap is 500 2024-03-04T06:24:30.647 INFO:tasks.rados.rados.0.smithi064.stdout:3808: finishing write tid 1 to smithi06431743-2 2024-03-04T06:24:30.647 INFO:tasks.rados.rados.0.smithi064.stdout:3812: seq_num 1722 ranges {541654=674556,1617169=781593} 2024-03-04T06:24:30.653 INFO:tasks.rados.rados.0.smithi064.stdout:3812: writing smithi06431743-35 from 541654 to 1216210 tid 1 2024-03-04T06:24:30.660 INFO:tasks.rados.rados.0.smithi064.stdout:3812: writing smithi06431743-35 from 1617169 to 2398762 tid 2 2024-03-04T06:24:30.660 INFO:tasks.rados.rados.0.smithi064.stdout:3813: write oid 33 current snap is 500 2024-03-04T06:24:30.660 INFO:tasks.rados.rados.0.smithi064.stdout:3813: seq_num 1723 ranges {558461=749805,1959298=641604} 2024-03-04T06:24:30.667 INFO:tasks.rados.rados.0.smithi064.stdout:3813: writing smithi06431743-33 from 558461 to 1308266 tid 1 2024-03-04T06:24:30.676 INFO:tasks.rados.rados.0.smithi064.stdout:3813: writing smithi06431743-33 from 1959298 to 2600902 tid 2 2024-03-04T06:24:30.676 INFO:tasks.rados.rados.0.smithi064.stdout:3814: read oid 12 snap -1 2024-03-04T06:24:30.676 INFO:tasks.rados.rados.0.smithi064.stdout:3814: expect (ObjNum 1674 snap 482 seq_num 1674) 2024-03-04T06:24:30.676 INFO:tasks.rados.rados.0.smithi064.stdout:3815: snap_remove snap 499 2024-03-04T06:24:30.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:30.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:30.766+0000 7f0f40c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:30.773 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:30.773+0000 7f0f40c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:30.840 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:30.838+0000 7f6297ed4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:30.842 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:30.842+0000 7f6297ed4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:30.977 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:24:30.978 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:24:30.990 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1493,"stamp":"2024-03-04T06:24:30.919189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83175633,"num_objects":1260,"num_object_clones":1044,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":184,"num_read":6816,"num_read_kb":5336213,"num_write":5815,"num_write_kb":1549107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17447,"ondisk_log_size":17447,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":701312,"kb_used_data":482976,"kb_used_omap":0,"kb_used_meta":218304,"kb_avail":376786048,"statfs":{"total":386547056640,"available":385828913152,"internally_reserved":0,"allocated":494567424,"data_stored":571155324,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":223543296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1832392,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":1035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000881"},"pg_stats":[{"pgid":"3.f","version":"992'956","reported_seq":3388,"reported_epoch":998,"state":"active+clean","last_fresh":"2024-03-04T06:24:29.505361+0000","last_change":"2024-03-04T06:24:23.394752+0000","last_active":"2024-03-04T06:24:29.505361+0000","last_peered":"2024-03-04T06:24:29.505361+0000","last_clean":"2024-03-04T06:24:29.505361+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:24:29.505361+0000","last_undegraded":"2024-03-04T06:24:29.505361+0000","last_fullsized":"2024-03-04T06:24:29.505361+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":374,"num_read_kb":295545,"num_write":337,"num_write_kb":88619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"992'793","reported_seq":3380,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-03-04T06:24:27.456914+0000","last_change":"2024-03-04T06:24:23.541553+0000","last_active":"2024-03-04T06:24:27.456914+0000","last_peered":"2024-03-04T06:24:27.456914+0000","last_clean":"2024-03-04T06:24:27.456914+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:24:27.456914+0000","last_undegraded":"2024-03-04T06:24:27.456914+0000","last_fullsized":"2024-03-04T06:24:27.456914+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":793,"log_dups_size":0,"ondisk_log_size":793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605481,"num_objects":63,"num_object_clones":46,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":258,"num_read_kb":210899,"num_write":256,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"994'1344","reported_seq":3599,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.456630+0000","last_change":"2024-03-04T06:24:25.517856+0000","last_active":"2024-03-04T06:24:25.456630+0000","last_peered":"2024-03-04T06:24:25.456630+0000","last_clean":"2024-03-04T06:24:25.456630+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:24:25.456630+0000","last_undegraded":"2024-03-04T06:24:25.456630+0000","last_fullsized":"2024-03-04T06:24:25.456630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1344,"log_dups_size":0,"ondisk_log_size":1344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10169918,"num_objects":100,"num_object_clones":85,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":593,"num_read_kb":473296,"num_write":460,"num_write_kb":125339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"992'1094","reported_seq":3363,"reported_epoch":998,"state":"active+clean","last_fresh":"2024-03-04T06:24:29.491351+0000","last_change":"2024-03-04T06:24:23.389236+0000","last_active":"2024-03-04T06:24:29.491351+0000","last_peered":"2024-03-04T06:24:29.491351+0000","last_clean":"2024-03-04T06:24:29.491351+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:24:29.491351+0000","last_undegraded":"2024-03-04T06:24:29.491351+0000","last_fullsized":"2024-03-04T06:24:29.491351+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1094,"log_dups_size":0,"ondisk_log_size":1094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4437906,"num_objects":69,"num_object_clones":59,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":332,"num_read_kb":252050,"num_write":327,"num_write_kb":90303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"992'433","reported_seq":3143,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.946335+0000","last_change":"2024-03-04T06:24:23.458923+0000","last_active":"2024-03-04T06:24:25.946335+0000","last_peered":"2024-03-04T06:24:25.946335+0000","last_clean":"2024-03-04T06:24:25.946335+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:24:25.946335+0000","last_undegraded":"2024-03-04T06:24:25.946335+0000","last_fullsized":"2024-03-04T06:24:25.946335+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"998'1057","reported_seq":3445,"reported_epoch":998,"state":"active+clean","last_fresh":"2024-03-04T06:24:29.545998+0000","last_change":"2024-03-04T06:24:23.397717+0000","last_active":"2024-03-04T06:24:29.545998+0000","last_peered":"2024-03-04T06:24:29.545998+0000","last_clean":"2024-03-04T06:24:29.545998+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:24:29.545998+0000","last_undegraded":"2024-03-04T06:24:29.545998+0000","last_fullsized":"2024-03-04T06:24:29.545998+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1057,"log_dups_size":0,"ondisk_log_size":1057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9362720,"num_objects":75,"num_object_clones":63,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":9,"num_read":443,"num_read_kb":355100,"num_write":363,"num_write_kb":105033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2919,"reported_epoch":998,"state":"active+clean","last_fresh":"2024-03-04T06:24:29.499812+0000","last_change":"2024-03-04T06:24:23.382807+0000","last_active":"2024-03-04T06:24:29.499812+0000","last_peered":"2024-03-04T06:24:29.499812+0000","last_clean":"2024-03-04T06:24:29.499812+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:24:29.499812+0000","last_undegraded":"2024-03-04T06:24:29.499812+0000","last_fullsized":"2024-03-04T06:24:29.499812+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"994'690","reported_seq":3229,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.461941+0000","last_change":"2024-03-04T06:24:25.492079+0000","last_active":"2024-03-04T06:24:25.461941+0000","last_peered":"2024-03-04T06:24:25.461941+0000","last_clean":"2024-03-04T06:24:25.461941+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:24:25.461941+0000","last_undegraded":"2024-03-04T06:24:25.461941+0000","last_fullsized":"2024-03-04T06:24:25.461941+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672742,"num_objects":44,"num_object_clones":35,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":287,"num_read_kb":255608,"num_write":194,"num_write_kb":52314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2639,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.925483+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:24:25.925483+0000","last_peered":"2024-03-04T06:24:25.925483+0000","last_clean":"2024-03-04T06:24:25.925483+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:24:25.925483+0000","last_undegraded":"2024-03-04T06:24:25.925483+0000","last_fullsized":"2024-03-04T06:24:25.925483+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2456,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.482153+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:24:25.482153+0000","last_peered":"2024-03-04T06:24:25.482153+0000","last_clean":"2024-03-04T06:24:25.482153+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:24:25.482153+0000","last_undegraded":"2024-03-04T06:24:25.482153+0000","last_fullsized":"2024-03-04T06:24:25.482153+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"994'1007","reported_seq":3429,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.485388+0000","last_change":"2024-03-04T06:24:25.503583+0000","last_active":"2024-03-04T06:24:25.485388+0000","last_peered":"2024-03-04T06:24:25.485388+0000","last_clean":"2024-03-04T06:24:25.485388+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:24:25.485388+0000","last_undegraded":"2024-03-04T06:24:25.485388+0000","last_fullsized":"2024-03-04T06:24:25.485388+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1007,"log_dups_size":0,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8287072,"num_objects":73,"num_object_clones":59,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":452,"num_read_kb":370094,"num_write":331,"num_write_kb":88736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2450,"reported_epoch":998,"state":"active+clean","last_fresh":"2024-03-04T06:24:29.502458+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:24:29.502458+0000","last_peered":"2024-03-04T06:24:29.502458+0000","last_clean":"2024-03-04T06:24:29.502458+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:24:29.502458+0000","last_undegraded":"2024-03-04T06:24:29.502458+0000","last_fullsized":"2024-03-04T06:24:29.502458+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"992'705","reported_seq":3322,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.932081+0000","last_change":"2024-03-04T06:24:23.426144+0000","last_active":"2024-03-04T06:24:25.932081+0000","last_peered":"2024-03-04T06:24:25.932081+0000","last_clean":"2024-03-04T06:24:25.932081+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:24:25.932081+0000","last_undegraded":"2024-03-04T06:24:25.932081+0000","last_fullsized":"2024-03-04T06:24:25.932081+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3664322,"num_objects":53,"num_object_clones":44,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":318,"num_read_kb":259030,"num_write":225,"num_write_kb":64009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2456,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.464504+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:24:25.464504+0000","last_peered":"2024-03-04T06:24:25.464504+0000","last_clean":"2024-03-04T06:24:25.464504+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:24:25.464504+0000","last_undegraded":"2024-03-04T06:24:25.464504+0000","last_fullsized":"2024-03-04T06:24:25.464504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"992'1978","reported_seq":3958,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.943285+0000","last_change":"2024-03-04T06:24:23.573542+0000","last_active":"2024-03-04T06:24:25.943285+0000","last_peered":"2024-03-04T06:24:25.943285+0000","last_clean":"2024-03-04T06:24:25.943285+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:24:25.943285+0000","last_undegraded":"2024-03-04T06:24:25.943285+0000","last_fullsized":"2024-03-04T06:24:25.943285+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1978,"log_dups_size":0,"ondisk_log_size":1978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11010235,"num_objects":147,"num_object_clones":121,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":820,"num_read_kb":645886,"num_write":683,"num_write_kb":183854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2450,"reported_epoch":998,"state":"active+clean","last_fresh":"2024-03-04T06:24:29.476479+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:24:29.476479+0000","last_peered":"2024-03-04T06:24:29.476479+0000","last_clean":"2024-03-04T06:24:29.476479+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:24:29.476479+0000","last_undegraded":"2024-03-04T06:24:29.476479+0000","last_fullsized":"2024-03-04T06:24:29.476479+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"993'2312","reported_seq":4010,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-03-04T06:24:27.455221+0000","last_change":"2024-03-04T06:24:23.567107+0000","last_active":"2024-03-04T06:24:27.455221+0000","last_peered":"2024-03-04T06:24:27.455221+0000","last_clean":"2024-03-04T06:24:27.455221+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:24:27.455221+0000","last_undegraded":"2024-03-04T06:24:27.455221+0000","last_fullsized":"2024-03-04T06:24:27.455221+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2312,"log_dups_size":0,"ondisk_log_size":2312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11161531,"num_objects":161,"num_object_clones":137,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":736,"num_read_kb":541488,"num_write":735,"num_write_kb":198878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"994'977","reported_seq":3427,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.483064+0000","last_change":"2024-03-04T06:24:25.503407+0000","last_active":"2024-03-04T06:24:25.483064+0000","last_peered":"2024-03-04T06:24:25.483064+0000","last_clean":"2024-03-04T06:24:25.483064+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:24:25.483064+0000","last_undegraded":"2024-03-04T06:24:25.483064+0000","last_fullsized":"2024-03-04T06:24:25.483064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":977,"log_dups_size":0,"ondisk_log_size":977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521479,"num_objects":70,"num_object_clones":57,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":453,"num_read_kb":381495,"num_write":327,"num_write_kb":88741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2564,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-03-04T06:24:27.462335+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:24:27.462335+0000","last_peered":"2024-03-04T06:24:27.462335+0000","last_clean":"2024-03-04T06:24:27.462335+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:24:27.462335+0000","last_undegraded":"2024-03-04T06:24:27.462335+0000","last_fullsized":"2024-03-04T06:24:27.462335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"992'1130","reported_seq":3539,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-03-04T06:24:27.465863+0000","last_change":"2024-03-04T06:24:23.543845+0000","last_active":"2024-03-04T06:24:27.465863+0000","last_peered":"2024-03-04T06:24:27.465863+0000","last_clean":"2024-03-04T06:24:27.465863+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:24:27.465863+0000","last_undegraded":"2024-03-04T06:24:27.465863+0000","last_fullsized":"2024-03-04T06:24:27.465863+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1130,"log_dups_size":0,"ondisk_log_size":1130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8404754,"num_objects":85,"num_object_clones":68,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":399,"num_read_kb":289853,"num_write":371,"num_write_kb":99996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2506,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.941648+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:24:25.941648+0000","last_peered":"2024-03-04T06:24:25.941648+0000","last_clean":"2024-03-04T06:24:25.941648+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:24:25.941648+0000","last_undegraded":"2024-03-04T06:24:25.941648+0000","last_fullsized":"2024-03-04T06:24:25.941648+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"992'1500","reported_seq":3624,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.930277+0000","last_change":"2024-03-04T06:24:23.459695+0000","last_active":"2024-03-04T06:24:25.930277+0000","last_peered":"2024-03-04T06:24:25.930277+0000","last_clean":"2024-03-04T06:24:25.930277+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:24:25.930277+0000","last_undegraded":"2024-03-04T06:24:25.930277+0000","last_fullsized":"2024-03-04T06:24:25.930277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1500,"log_dups_size":0,"ondisk_log_size":1500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3378603,"num_objects":103,"num_object_clones":88,"num_object_copies":309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":509,"num_read_kb":396715,"num_write":475,"num_write_kb":131704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2564,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-03-04T06:24:27.458604+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:24:27.458604+0000","last_peered":"2024-03-04T06:24:27.458604+0000","last_clean":"2024-03-04T06:24:27.458604+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:24:27.458604+0000","last_undegraded":"2024-03-04T06:24:27.458604+0000","last_fullsized":"2024-03-04T06:24:27.458604+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"992'1374","reported_seq":3733,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-03-04T06:24:27.453512+0000","last_change":"2024-03-04T06:24:23.548854+0000","last_active":"2024-03-04T06:24:27.453512+0000","last_peered":"2024-03-04T06:24:27.453512+0000","last_clean":"2024-03-04T06:24:27.453512+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:24:27.453512+0000","last_undegraded":"2024-03-04T06:24:27.453512+0000","last_fullsized":"2024-03-04T06:24:27.453512+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1374,"log_dups_size":0,"ondisk_log_size":1374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4908502,"num_objects":105,"num_object_clones":91,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":616,"num_read_kb":515915,"num_write":468,"num_write_kb":127542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2456,"reported_epoch":994,"state":"active+clean","last_fresh":"2024-03-04T06:24:25.484040+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:24:25.484040+0000","last_peered":"2024-03-04T06:24:25.484040+0000","last_clean":"2024-03-04T06:24:25.484040+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:24:25.484040+0000","last_undegraded":"2024-03-04T06:24:25.484040+0000","last_fullsized":"2024-03-04T06:24:25.484040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":82585265,"num_objects":1258,"num_object_clones":1044,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":184,"num_read":6713,"num_read_kb":5336122,"num_write":5678,"num_write_kb":1547213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17350,"ondisk_log_size":17350,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706664,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":222852,"kb_used_data":151612,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94148988,"statfs":{"total":96636764160,"available":96408563712,"internally_reserved":0,"allocated":155250688,"data_stored":180122848,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706664,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":156916,"kb_used_data":112044,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94214924,"statfs":{"total":96636764160,"available":96476082176,"internally_reserved":0,"allocated":114733056,"data_stored":131847084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154760,"kb_used_data":109888,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94217080,"statfs":{"total":96636764160,"available":96478289920,"internally_reserved":0,"allocated":112525312,"data_stored":129022953,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166784,"kb_used_data":109432,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94205056,"statfs":{"total":96636764160,"available":96465977344,"internally_reserved":0,"allocated":112058368,"data_stored":130162439,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:24:31.463 INFO:tasks.rados.rados.0.smithi064.stdout:3815: done (6 left) 2024-03-04T06:24:31.463 INFO:tasks.rados.rados.0.smithi064.stdout:3816: read oid 8 snap -1 2024-03-04T06:24:31.463 INFO:tasks.rados.rados.0.smithi064.stdout:3816: expect (ObjNum 1650 snap 471 seq_num 1650) 2024-03-04T06:24:31.463 INFO:tasks.rados.rados.0.smithi064.stdout:3817: read oid 9 snap -1 2024-03-04T06:24:31.463 INFO:tasks.rados.rados.0.smithi064.stdout:3817: expect deleted 2024-03-04T06:24:31.463 INFO:tasks.rados.rados.0.smithi064.stdout:3818: write (excl) oid 5 current snap is 500 2024-03-04T06:24:31.463 INFO:tasks.rados.rados.0.smithi064.stdout:3810: finishing write tid 1 to smithi06431743-46 2024-03-04T06:24:31.463 INFO:tasks.rados.rados.0.smithi064.stdout:3818: seq_num 1724 ranges {575268=425054,1617780=785262} 2024-03-04T06:24:31.468 INFO:tasks.rados.rados.0.smithi064.stdout:3818: writing smithi06431743-5 from 575268 to 1000322 tid 1 2024-03-04T06:24:31.478 INFO:tasks.rados.rados.0.smithi064.stdout:3818: writing smithi06431743-5 from 1617780 to 2403042 tid 2 2024-03-04T06:24:31.478 INFO:tasks.rados.rados.0.smithi064.stdout:3819: read oid 4 snap 471 2024-03-04T06:24:31.478 INFO:tasks.rados.rados.0.smithi064.stdout:3819: expect deleted 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3820: read oid 14 snap -1 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3820: expect (ObjNum 1629 snap 466 seq_num 1629) 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3808: finishing write tid 2 to smithi06431743-2 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3810: finishing write tid 2 to smithi06431743-46 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3811: finishing write tid 1 to smithi06431743-16 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3821: read oid 43 snap -1 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3821: expect deleted 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3822: snap_create 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3808: finishing write tid 3 to smithi06431743-2 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3808: finishing write tid 4 to smithi06431743-2 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3808: oid 2 updating version 0 to 1102 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3808: oid 2 version 1102 is already newer than 1101 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3808: oid 2 version 1102 is already newer than 1100 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 1102 (ObjNum 1718 snap 500 seq_num 1718) dirty exists 2024-03-04T06:24:31.479 INFO:tasks.rados.rados.0.smithi064.stdout:3808: left oid 2 (ObjNum 1718 snap 500 seq_num 1718) 2024-03-04T06:24:31.480 INFO:tasks.rados.rados.0.smithi064.stdout:3812: finishing write tid 1 to smithi06431743-35 2024-03-04T06:24:31.480 INFO:tasks.rados.rados.0.smithi064.stdout:3813: finishing write tid 1 to smithi06431743-33 2024-03-04T06:24:31.480 INFO:tasks.rados.rados.0.smithi064.stdout:3811: finishing write tid 2 to smithi06431743-16 2024-03-04T06:24:31.480 INFO:tasks.rados.rados.0.smithi064.stdout:3812: finishing write tid 2 to smithi06431743-35 2024-03-04T06:24:31.480 INFO:tasks.rados.rados.0.smithi064.stdout:3814: expect (ObjNum 1674 snap 482 seq_num 1674) 2024-03-04T06:24:31.599 INFO:tasks.rados.rados.0.smithi064.stdout:3810: finishing write tid 3 to smithi06431743-46 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:3810: finishing write tid 4 to smithi06431743-46 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:3810: oid 46 updating version 0 to 1989 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:3810: oid 46 version 1989 is already newer than 1988 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:3810: oid 46 version 1989 is already newer than 1987 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 46 v 1989 (ObjNum 1720 snap 500 seq_num 1720) dirty exists 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:3810: left oid 46 (ObjNum 1720 snap 500 seq_num 1720) 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:3813: finishing write tid 2 to smithi06431743-33 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:3811: finishing write tid 3 to smithi06431743-16 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:3812: finishing write tid 3 to smithi06431743-35 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:3813: finishing write tid 3 to smithi06431743-33 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:3813: finishing write tid 4 to smithi06431743-33 2024-03-04T06:24:31.600 INFO:tasks.rados.rados.0.smithi064.stdout:3813: oid 33 updating version 0 to 1383 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:3813: oid 33 version 1383 is already newer than 1382 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:3813: oid 33 updating version 1383 to 1384 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 33 v 1384 (ObjNum 1723 snap 500 seq_num 1723) dirty exists 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:3813: left oid 33 (ObjNum 1723 snap 500 seq_num 1723) 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:3812: finishing write tid 4 to smithi06431743-35 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:3812: oid 35 updating version 0 to 1505 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:3812: oid 35 version 1505 is already newer than 1504 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:3812: oid 35 version 1505 is already newer than 1503 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 35 v 1505 (ObjNum 1722 snap 500 seq_num 1722) dirty exists 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:3812: left oid 35 (ObjNum 1722 snap 500 seq_num 1722) 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:3811: finishing write tid 4 to smithi06431743-16 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:3811: finishing write tid 5 to smithi06431743-16 2024-03-04T06:24:31.601 INFO:tasks.rados.rados.0.smithi064.stdout:3811: oid 16 updating version 0 to 1135 2024-03-04T06:24:31.602 INFO:tasks.rados.rados.0.smithi064.stdout:3811: oid 16 updating version 1135 to 1136 2024-03-04T06:24:31.602 INFO:tasks.rados.rados.0.smithi064.stdout:3811: oid 16 updating version 1136 to 1137 2024-03-04T06:24:31.602 INFO:tasks.rados.rados.0.smithi064.stdout:3811: oid 16 version 1137 is already newer than 1134 2024-03-04T06:24:31.602 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 16 v 1137 (ObjNum 1721 snap 500 seq_num 1721) dirty exists 2024-03-04T06:24:31.602 INFO:tasks.rados.rados.0.smithi064.stdout:3811: left oid 16 (ObjNum 1721 snap 500 seq_num 1721) 2024-03-04T06:24:31.602 INFO:tasks.rados.rados.0.smithi064.stdout:3816: expect (ObjNum 1650 snap 471 seq_num 1650) 2024-03-04T06:24:31.740 INFO:tasks.rados.rados.0.smithi064.stdout:3820: expect (ObjNum 1629 snap 466 seq_num 1629) 2024-03-04T06:24:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:3818: finishing write tid 1 to smithi06431743-5 2024-03-04T06:24:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:3818: finishing write tid 2 to smithi06431743-5 2024-03-04T06:24:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:3818: finishing write tid 3 to smithi06431743-5 2024-03-04T06:24:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:3818: finishing write tid 4 to smithi06431743-5 2024-03-04T06:24:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:3818: oid 5 updating version 0 to 2320 2024-03-04T06:24:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:3818: oid 5 updating version 2320 to 2322 2024-03-04T06:24:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:3818: oid 5 version 2322 is already newer than 2321 2024-03-04T06:24:31.901 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 5 v 2322 (ObjNum 1724 snap 500 seq_num 1724) dirty exists 2024-03-04T06:24:31.902 INFO:tasks.rados.rados.0.smithi064.stdout:3818: left oid 5 (ObjNum 1724 snap 500 seq_num 1724) 2024-03-04T06:24:32.464 INFO:tasks.rados.rados.0.smithi064.stdout:3808: done (12 left) 2024-03-04T06:24:32.464 INFO:tasks.rados.rados.0.smithi064.stdout:3810: done (11 left) 2024-03-04T06:24:32.464 INFO:tasks.rados.rados.0.smithi064.stdout:3811: done (10 left) 2024-03-04T06:24:32.464 INFO:tasks.rados.rados.0.smithi064.stdout:3812: done (9 left) 2024-03-04T06:24:32.464 INFO:tasks.rados.rados.0.smithi064.stdout:3813: done (8 left) 2024-03-04T06:24:32.465 INFO:tasks.rados.rados.0.smithi064.stdout:3814: done (7 left) 2024-03-04T06:24:32.465 INFO:tasks.rados.rados.0.smithi064.stdout:3816: done (6 left) 2024-03-04T06:24:32.465 INFO:tasks.rados.rados.0.smithi064.stdout:3817: done (5 left) 2024-03-04T06:24:32.465 INFO:tasks.rados.rados.0.smithi064.stdout:3818: done (4 left) 2024-03-04T06:24:32.465 INFO:tasks.rados.rados.0.smithi064.stdout:3819: done (3 left) 2024-03-04T06:24:32.465 INFO:tasks.rados.rados.0.smithi064.stdout:3820: done (2 left) 2024-03-04T06:24:32.465 INFO:tasks.rados.rados.0.smithi064.stdout:3821: done (1 left) 2024-03-04T06:24:32.465 INFO:tasks.rados.rados.0.smithi064.stdout:3822: done (0 left) 2024-03-04T06:24:32.465 INFO:tasks.rados.rados.0.smithi064.stdout:3823: snap_remove snap 492 2024-03-04T06:24:32.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:33.031 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:33.029+0000 7f1b0406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:33.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:33.033+0000 7f1b0406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:33.353 INFO:tasks.rados.rados.0.smithi064.stdout:3823: done (0 left) 2024-03-04T06:24:33.353 INFO:tasks.rados.rados.0.smithi064.stdout:3824: delete oid 10 current snap is 501 2024-03-04T06:24:33.407 INFO:tasks.rados.rados.0.smithi064.stdout:3824: done (0 left) 2024-03-04T06:24:33.407 INFO:tasks.rados.rados.0.smithi064.stdout:3825: read oid 30 snap -1 2024-03-04T06:24:33.408 INFO:tasks.rados.rados.0.smithi064.stdout:3825: expect deleted 2024-03-04T06:24:33.408 INFO:tasks.rados.rados.0.smithi064.stdout:3826: write (excl) oid 41 current snap is 501 2024-03-04T06:24:33.408 INFO:tasks.rados.rados.0.smithi064.stdout:3826: seq_num 1726 ranges {608882=691905,1851097=788931} 2024-03-04T06:24:33.414 INFO:tasks.rados.rados.0.smithi064.stdout:3826: writing smithi06431743-41 from 608882 to 1300787 tid 1 2024-03-04T06:24:33.424 INFO:tasks.rados.rados.0.smithi064.stdout:3826: writing smithi06431743-41 from 1851097 to 2640028 tid 2 2024-03-04T06:24:33.424 INFO:tasks.rados.rados.0.smithi064.stdout:3827: snap_create 2024-03-04T06:24:33.449 INFO:tasks.rados.rados.0.smithi064.stdout:3826: finishing write tid 1 to smithi06431743-41 2024-03-04T06:24:33.476 INFO:tasks.rados.rados.0.smithi064.stdout:3826: finishing write tid 2 to smithi06431743-41 2024-03-04T06:24:33.486 INFO:tasks.rados.rados.0.smithi064.stdout:3826: finishing write tid 3 to smithi06431743-41 2024-03-04T06:24:33.490 INFO:tasks.rados.rados.0.smithi064.stdout:3826: finishing write tid 4 to smithi06431743-41 2024-03-04T06:24:33.490 INFO:tasks.rados.rados.0.smithi064.stdout:3826: oid 41 updating version 0 to 1014 2024-03-04T06:24:33.490 INFO:tasks.rados.rados.0.smithi064.stdout:3826: oid 41 version 1014 is already newer than 1013 2024-03-04T06:24:33.490 INFO:tasks.rados.rados.0.smithi064.stdout:3826: oid 41 updating version 1014 to 1015 2024-03-04T06:24:33.491 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 1015 (ObjNum 1726 snap 501 seq_num 1726) dirty exists 2024-03-04T06:24:33.491 INFO:tasks.rados.rados.0.smithi064.stdout:3826: left oid 41 (ObjNum 1726 snap 501 seq_num 1726) 2024-03-04T06:24:34.352 INFO:tasks.rados.rados.0.smithi064.stdout:3825: done (2 left) 2024-03-04T06:24:34.352 INFO:tasks.rados.rados.0.smithi064.stdout:3826: done (1 left) 2024-03-04T06:24:34.353 INFO:tasks.rados.rados.0.smithi064.stdout:3827: done (0 left) 2024-03-04T06:24:34.353 INFO:tasks.rados.rados.0.smithi064.stdout:3828: write (excl) oid 7 current snap is 502 2024-03-04T06:24:34.353 INFO:tasks.rados.rados.0.smithi064.stdout:3828: seq_num 1727 ranges {625689=767154,1793226=648942} 2024-03-04T06:24:34.359 INFO:tasks.rados.rados.0.smithi064.stdout:3828: writing smithi06431743-7 from 625689 to 1392843 tid 1 2024-03-04T06:24:34.366 INFO:tasks.rados.rados.0.smithi064.stdout:3828: writing smithi06431743-7 from 1793226 to 2442168 tid 2 2024-03-04T06:24:34.366 INFO:tasks.rados.rados.0.smithi064.stdout:3829: delete oid 18 current snap is 502 2024-03-04T06:24:34.423 INFO:tasks.rados.rados.0.smithi064.stdout:3829: done (1 left) 2024-03-04T06:24:34.424 INFO:tasks.rados.rados.0.smithi064.stdout:3830: write oid 2 current snap is 502 2024-03-04T06:24:34.424 INFO:tasks.rados.rados.0.smithi064.stdout:3830: seq_num 1729 ranges {659303=517652,1910190=652611} 2024-03-04T06:24:34.429 INFO:tasks.rados.rados.0.smithi064.stdout:3830: writing smithi06431743-2 from 659303 to 1176955 tid 1 2024-03-04T06:24:34.441 INFO:tasks.rados.rados.0.smithi064.stdout:3830: writing smithi06431743-2 from 1910190 to 2562801 tid 2 2024-03-04T06:24:34.441 INFO:tasks.rados.rados.0.smithi064.stdout:3831: read oid 42 snap -1 2024-03-04T06:24:34.441 INFO:tasks.rados.rados.0.smithi064.stdout:3831: expect deleted 2024-03-04T06:24:34.441 INFO:tasks.rados.rados.0.smithi064.stdout:3832: write (excl) oid 49 current snap is 502 2024-03-04T06:24:34.441 INFO:tasks.rados.rados.0.smithi064.stdout:3828: finishing write tid 1 to smithi06431743-7 2024-03-04T06:24:34.442 INFO:tasks.rados.rados.0.smithi064.stdout:3832: seq_num 1730 ranges {676110=592901,1968672=796269} 2024-03-04T06:24:34.448 INFO:tasks.rados.rados.0.smithi064.stdout:3832: writing smithi06431743-49 from 676110 to 1269011 tid 1 2024-03-04T06:24:34.460 INFO:tasks.rados.rados.0.smithi064.stdout:3832: writing smithi06431743-49 from 1968672 to 2764941 tid 2 2024-03-04T06:24:34.460 INFO:tasks.rados.rados.0.smithi064.stdout:3833: read oid 26 snap 495 2024-03-04T06:24:34.460 INFO:tasks.rados.rados.0.smithi064.stdout:3833: expect deleted 2024-03-04T06:24:34.460 INFO:tasks.rados.rados.0.smithi064.stdout:3834: snap_create 2024-03-04T06:24:34.460 INFO:tasks.rados.rados.0.smithi064.stdout:3828: finishing write tid 2 to smithi06431743-7 2024-03-04T06:24:34.474 INFO:tasks.rados.rados.0.smithi064.stdout:3828: finishing write tid 3 to smithi06431743-7 2024-03-04T06:24:34.475 INFO:tasks.rados.rados.0.smithi064.stdout:3828: finishing write tid 4 to smithi06431743-7 2024-03-04T06:24:34.475 INFO:tasks.rados.rados.0.smithi064.stdout:3828: oid 7 updating version 0 to 713 2024-03-04T06:24:34.475 INFO:tasks.rados.rados.0.smithi064.stdout:3828: oid 7 version 713 is already newer than 712 2024-03-04T06:24:34.476 INFO:tasks.rados.rados.0.smithi064.stdout:3828: oid 7 version 713 is already newer than 711 2024-03-04T06:24:34.476 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 713 (ObjNum 1727 snap 502 seq_num 1727) dirty exists 2024-03-04T06:24:34.476 INFO:tasks.rados.rados.0.smithi064.stdout:3828: left oid 7 (ObjNum 1727 snap 502 seq_num 1727) 2024-03-04T06:24:34.478 INFO:tasks.rados.rados.0.smithi064.stdout:3832: finishing write tid 1 to smithi06431743-49 2024-03-04T06:24:34.499 INFO:tasks.rados.rados.0.smithi064.stdout:3830: finishing write tid 1 to smithi06431743-2 2024-03-04T06:24:34.506 INFO:tasks.rados.rados.0.smithi064.stdout:3832: finishing write tid 2 to smithi06431743-49 2024-03-04T06:24:34.515 INFO:tasks.rados.rados.0.smithi064.stdout:3832: finishing write tid 3 to smithi06431743-49 2024-03-04T06:24:34.516 INFO:tasks.rados.rados.0.smithi064.stdout:3832: finishing write tid 4 to smithi06431743-49 2024-03-04T06:24:34.516 INFO:tasks.rados.rados.0.smithi064.stdout:3832: oid 49 updating version 0 to 696 2024-03-04T06:24:34.517 INFO:tasks.rados.rados.0.smithi064.stdout:3832: oid 49 updating version 696 to 697 2024-03-04T06:24:34.517 INFO:tasks.rados.rados.0.smithi064.stdout:3832: oid 49 updating version 697 to 698 2024-03-04T06:24:34.517 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 698 (ObjNum 1730 snap 502 seq_num 1730) dirty exists 2024-03-04T06:24:34.517 INFO:tasks.rados.rados.0.smithi064.stdout:3832: left oid 49 (ObjNum 1730 snap 502 seq_num 1730) 2024-03-04T06:24:34.518 INFO:tasks.rados.rados.0.smithi064.stdout:3830: finishing write tid 2 to smithi06431743-2 2024-03-04T06:24:34.524 INFO:tasks.rados.rados.0.smithi064.stdout:3830: finishing write tid 3 to smithi06431743-2 2024-03-04T06:24:34.525 INFO:tasks.rados.rados.0.smithi064.stdout:3830: finishing write tid 4 to smithi06431743-2 2024-03-04T06:24:34.525 INFO:tasks.rados.rados.0.smithi064.stdout:3830: oid 2 updating version 0 to 1109 2024-03-04T06:24:34.525 INFO:tasks.rados.rados.0.smithi064.stdout:3830: oid 2 version 1109 is already newer than 1109 2024-03-04T06:24:34.525 INFO:tasks.rados.rados.0.smithi064.stdout:3830: oid 2 updating version 1109 to 1110 2024-03-04T06:24:34.525 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 2 v 1110 (ObjNum 1729 snap 502 seq_num 1729) dirty exists 2024-03-04T06:24:34.525 INFO:tasks.rados.rados.0.smithi064.stdout:3830: left oid 2 (ObjNum 1729 snap 502 seq_num 1729) 2024-03-04T06:24:35.104 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:35.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:35.214+0000 7f88b426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:35.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:35.218+0000 7f88b426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:35.355 INFO:tasks.rados.rados.0.smithi064.stdout:3828: done (5 left) 2024-03-04T06:24:35.355 INFO:tasks.rados.rados.0.smithi064.stdout:3830: done (4 left) 2024-03-04T06:24:35.355 INFO:tasks.rados.rados.0.smithi064.stdout:3831: done (3 left) 2024-03-04T06:24:35.355 INFO:tasks.rados.rados.0.smithi064.stdout:3832: done (2 left) 2024-03-04T06:24:35.355 INFO:tasks.rados.rados.0.smithi064.stdout:3833: done (1 left) 2024-03-04T06:24:35.355 INFO:tasks.rados.rados.0.smithi064.stdout:3834: done (0 left) 2024-03-04T06:24:35.355 INFO:tasks.rados.rados.0.smithi064.stdout:3835: read oid 7 snap -1 2024-03-04T06:24:35.355 INFO:tasks.rados.rados.0.smithi064.stdout:3835: expect (ObjNum 1727 snap 502 seq_num 1727) 2024-03-04T06:24:35.355 INFO:tasks.rados.rados.0.smithi064.stdout:3836: snap_remove snap 493 2024-03-04T06:24:35.992 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:24:35.992 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:24:35.992 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:24:35.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:24:36.104 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:36.098+0000 7f20b5b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:36.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:36.106+0000 7f20b5b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:36.119 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:36.119 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:36.119 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:36.119 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:36.119 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:36.119 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:36.119 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:36.119 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:36.120 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:36.120 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:36.120 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:36.120 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:36.122 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:36.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:24:36.231 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:36.226+0000 7f6812b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:36.234 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:36.233+0000 7f6812b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:36.247 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:36.247 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:36.247 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:36.247 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:36.247 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:36.247 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:36.247 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:36.247 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:36.248 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:36.248 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:36.248 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:36.248 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:36.250 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:36.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:24:36.355 INFO:tasks.rados.rados.0.smithi064.stdout:3835: expect (ObjNum 1727 snap 502 seq_num 1727) 2024-03-04T06:24:36.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:36.353+0000 7fc91b99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:36.364 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:36.363+0000 7fc91b99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:36.384 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:36.384 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:36.384 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:36.385 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:36.385 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:36.385 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:36.385 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:36.385 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:36.385 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:36.385 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:36.385 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:36.385 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:36.389 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:36.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:24:36.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:36.493+0000 7f0b80b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:36.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:36.502+0000 7f0b80b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:36.531 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:36.531 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:36.532 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:36.532 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:36.532 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:36.532 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:36.532 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:36.532 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:36.532 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:36.532 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:36.532 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:36.532 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:36.535 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:36.564 INFO:tasks.rados.rados.0.smithi064.stdout:3835: done (1 left) 2024-03-04T06:24:36.564 INFO:tasks.rados.rados.0.smithi064.stdout:3836: done (0 left) 2024-03-04T06:24:36.564 INFO:tasks.rados.rados.0.smithi064.stdout:3837: snap_remove snap 460 2024-03-04T06:24:37.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:37.361 INFO:tasks.rados.rados.0.smithi064.stdout:3837: done (0 left) 2024-03-04T06:24:37.361 INFO:tasks.rados.rados.0.smithi064.stdout:3838: read oid 45 snap -1 2024-03-04T06:24:37.361 INFO:tasks.rados.rados.0.smithi064.stdout:3838: expect (ObjNum 1598 snap 453 seq_num 1598) 2024-03-04T06:24:37.361 INFO:tasks.rados.rados.0.smithi064.stdout:3839: write oid 38 current snap is 503 2024-03-04T06:24:37.361 INFO:tasks.rados.rados.0.smithi064.stdout:3839: seq_num 1731 ranges {692917=668150,1910801=539927,3086024=6893} 2024-03-04T06:24:37.368 INFO:tasks.rados.rados.0.smithi064.stdout:3839: writing smithi06431743-38 from 692917 to 1361067 tid 1 2024-03-04T06:24:37.374 INFO:tasks.rados.rados.0.smithi064.stdout:3839: writing smithi06431743-38 from 1910801 to 2450728 tid 2 2024-03-04T06:24:37.378 INFO:tasks.rados.rados.0.smithi064.stdout:3839: writing smithi06431743-38 from 3086024 to 3092917 tid 3 2024-03-04T06:24:37.378 INFO:tasks.rados.rados.0.smithi064.stdout:3840: write oid 48 current snap is 503 2024-03-04T06:24:37.378 INFO:tasks.rados.rados.0.smithi064.stdout:3840: seq_num 1732 ranges {709724=743399,1969283=799938} 2024-03-04T06:24:37.388 INFO:tasks.rados.rados.0.smithi064.stdout:3840: writing smithi06431743-48 from 709724 to 1453123 tid 1 2024-03-04T06:24:37.399 INFO:tasks.rados.rados.0.smithi064.stdout:3840: writing smithi06431743-48 from 1969283 to 2769221 tid 2 2024-03-04T06:24:37.399 INFO:tasks.rados.rados.0.smithi064.stdout:3841: snap_remove snap 491 2024-03-04T06:24:37.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:37.404+0000 7f19ab66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:37.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:37.412+0000 7f19ab66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:38.365 INFO:tasks.rados.rados.0.smithi064.stdout:3841: done (3 left) 2024-03-04T06:24:38.365 INFO:tasks.rados.rados.0.smithi064.stdout:3842: snap_create 2024-03-04T06:24:38.366 INFO:tasks.rados.rados.0.smithi064.stdout:3839: finishing write tid 1 to smithi06431743-38 2024-03-04T06:24:38.366 INFO:tasks.rados.rados.0.smithi064.stdout:3838: expect (ObjNum 1598 snap 453 seq_num 1598) 2024-03-04T06:24:38.613 INFO:tasks.rados.rados.0.smithi064.stdout:3840: finishing write tid 1 to smithi06431743-48 2024-03-04T06:24:38.613 INFO:tasks.rados.rados.0.smithi064.stdout:3839: finishing write tid 2 to smithi06431743-38 2024-03-04T06:24:38.613 INFO:tasks.rados.rados.0.smithi064.stdout:3839: finishing write tid 3 to smithi06431743-38 2024-03-04T06:24:38.613 INFO:tasks.rados.rados.0.smithi064.stdout:3840: finishing write tid 2 to smithi06431743-48 2024-03-04T06:24:38.613 INFO:tasks.rados.rados.0.smithi064.stdout:3839: finishing write tid 4 to smithi06431743-38 2024-03-04T06:24:38.613 INFO:tasks.rados.rados.0.smithi064.stdout:3839: finishing write tid 5 to smithi06431743-38 2024-03-04T06:24:38.613 INFO:tasks.rados.rados.0.smithi064.stdout:3839: oid 38 updating version 0 to 1070 2024-03-04T06:24:38.613 INFO:tasks.rados.rados.0.smithi064.stdout:3839: oid 38 version 1070 is already newer than 1068 2024-03-04T06:24:38.613 INFO:tasks.rados.rados.0.smithi064.stdout:3839: oid 38 version 1070 is already newer than 1067 2024-03-04T06:24:38.613 INFO:tasks.rados.rados.0.smithi064.stdout:3839: oid 38 version 1070 is already newer than 1069 2024-03-04T06:24:38.614 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 38 v 1070 (ObjNum 1731 snap 503 seq_num 1731) dirty exists 2024-03-04T06:24:38.614 INFO:tasks.rados.rados.0.smithi064.stdout:3839: left oid 38 (ObjNum 1731 snap 503 seq_num 1731) 2024-03-04T06:24:38.614 INFO:tasks.rados.rados.0.smithi064.stdout:3840: finishing write tid 3 to smithi06431743-48 2024-03-04T06:24:38.614 INFO:tasks.rados.rados.0.smithi064.stdout:3840: finishing write tid 4 to smithi06431743-48 2024-03-04T06:24:38.614 INFO:tasks.rados.rados.0.smithi064.stdout:3840: oid 48 updating version 0 to 724 2024-03-04T06:24:38.614 INFO:tasks.rados.rados.0.smithi064.stdout:3840: oid 48 version 724 is already newer than 723 2024-03-04T06:24:38.614 INFO:tasks.rados.rados.0.smithi064.stdout:3840: oid 48 version 724 is already newer than 722 2024-03-04T06:24:38.614 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 48 v 724 (ObjNum 1732 snap 503 seq_num 1732) dirty exists 2024-03-04T06:24:38.614 INFO:tasks.rados.rados.0.smithi064.stdout:3840: left oid 48 (ObjNum 1732 snap 503 seq_num 1732) 2024-03-04T06:24:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:3838: done (3 left) 2024-03-04T06:24:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:3839: done (2 left) 2024-03-04T06:24:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:3840: done (1 left) 2024-03-04T06:24:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:3842: done (0 left) 2024-03-04T06:24:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:3843: read oid 1 snap -1 2024-03-04T06:24:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:3843: expect (ObjNum 1661 snap 477 seq_num 1661) 2024-03-04T06:24:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:3844: read oid 17 snap -1 2024-03-04T06:24:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:3844: expect deleted 2024-03-04T06:24:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:3845: write (excl) oid 4 current snap is 504 2024-03-04T06:24:39.367 INFO:tasks.rados.rados.0.smithi064.stdout:3845: seq_num 1733 ranges {726531=418648,1627765=543596,2740870=385661} 2024-03-04T06:24:39.373 INFO:tasks.rados.rados.0.smithi064.stdout:3845: writing smithi06431743-4 from 726531 to 1145179 tid 1 2024-03-04T06:24:39.378 INFO:tasks.rados.rados.0.smithi064.stdout:3845: writing smithi06431743-4 from 1627765 to 2171361 tid 2 2024-03-04T06:24:39.384 INFO:tasks.rados.rados.0.smithi064.stdout:3845: writing smithi06431743-4 from 2740870 to 3126531 tid 3 2024-03-04T06:24:39.384 INFO:tasks.rados.rados.0.smithi064.stdout:3846: read oid 19 snap -1 2024-03-04T06:24:39.384 INFO:tasks.rados.rados.0.smithi064.stdout:3846: expect (ObjNum 1699 snap 491 seq_num 1699) 2024-03-04T06:24:39.385 INFO:tasks.rados.rados.0.smithi064.stdout:3847: read oid 8 snap -1 2024-03-04T06:24:39.385 INFO:tasks.rados.rados.0.smithi064.stdout:3847: expect (ObjNum 1650 snap 471 seq_num 1650) 2024-03-04T06:24:39.385 INFO:tasks.rados.rados.0.smithi064.stdout:3848: read oid 26 snap -1 2024-03-04T06:24:39.385 INFO:tasks.rados.rados.0.smithi064.stdout:3848: expect deleted 2024-03-04T06:24:39.385 INFO:tasks.rados.rados.0.smithi064.stdout:3849: delete oid 29 current snap is 504 2024-03-04T06:24:39.430 INFO:tasks.rados.rados.0.smithi064.stdout:3843: expect (ObjNum 1661 snap 477 seq_num 1661) 2024-03-04T06:24:39.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:3843: done (6 left) 2024-03-04T06:24:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:3844: done (5 left) 2024-03-04T06:24:39.677 INFO:tasks.rados.rados.0.smithi064.stdout:3848: done (4 left) 2024-03-04T06:24:39.678 INFO:tasks.rados.rados.0.smithi064.stdout:3849: done (3 left) 2024-03-04T06:24:39.678 INFO:tasks.rados.rados.0.smithi064.stdout:3850: read oid 33 snap -1 2024-03-04T06:24:39.678 INFO:tasks.rados.rados.0.smithi064.stdout:3850: expect (ObjNum 1723 snap 500 seq_num 1723) 2024-03-04T06:24:39.678 INFO:tasks.rados.rados.0.smithi064.stdout:3851: delete oid 9 current snap is 504 2024-03-04T06:24:39.678 INFO:tasks.rados.rados.0.smithi064.stdout:3847: expect (ObjNum 1650 snap 471 seq_num 1650) 2024-03-04T06:24:39.678 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:39.676+0000 7fbedd26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:39.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:39.681+0000 7fbedd26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:39.816 INFO:tasks.rados.rados.0.smithi064.stdout:3845: finishing write tid 1 to smithi06431743-4 2024-03-04T06:24:39.816 INFO:tasks.rados.rados.0.smithi064.stdout:3845: finishing write tid 2 to smithi06431743-4 2024-03-04T06:24:39.816 INFO:tasks.rados.rados.0.smithi064.stdout:3845: finishing write tid 3 to smithi06431743-4 2024-03-04T06:24:39.816 INFO:tasks.rados.rados.0.smithi064.stdout:3845: finishing write tid 4 to smithi06431743-4 2024-03-04T06:24:39.816 INFO:tasks.rados.rados.0.smithi064.stdout:3845: finishing write tid 5 to smithi06431743-4 2024-03-04T06:24:39.816 INFO:tasks.rados.rados.0.smithi064.stdout:3845: oid 4 updating version 0 to 2347 2024-03-04T06:24:39.816 INFO:tasks.rados.rados.0.smithi064.stdout:3845: oid 4 version 2347 is already newer than 2346 2024-03-04T06:24:39.816 INFO:tasks.rados.rados.0.smithi064.stdout:3845: oid 4 version 2347 is already newer than 2345 2024-03-04T06:24:39.816 INFO:tasks.rados.rados.0.smithi064.stdout:3845: oid 4 version 2347 is already newer than 2344 2024-03-04T06:24:39.816 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 2347 (ObjNum 1733 snap 504 seq_num 1733) dirty exists 2024-03-04T06:24:39.817 INFO:tasks.rados.rados.0.smithi064.stdout:3845: left oid 4 (ObjNum 1733 snap 504 seq_num 1733) 2024-03-04T06:24:39.817 INFO:tasks.rados.rados.0.smithi064.stdout:3846: expect (ObjNum 1699 snap 491 seq_num 1699) 2024-03-04T06:24:40.088 INFO:tasks.rados.rados.0.smithi064.stdout:3850: expect (ObjNum 1723 snap 500 seq_num 1723) 2024-03-04T06:24:40.264 INFO:tasks.rados.rados.0.smithi064.stdout:3845: done (4 left) 2024-03-04T06:24:40.264 INFO:tasks.rados.rados.0.smithi064.stdout:3846: done (3 left) 2024-03-04T06:24:40.264 INFO:tasks.rados.rados.0.smithi064.stdout:3847: done (2 left) 2024-03-04T06:24:40.264 INFO:tasks.rados.rados.0.smithi064.stdout:3850: done (1 left) 2024-03-04T06:24:40.264 INFO:tasks.rados.rados.0.smithi064.stdout:3851: done (0 left) 2024-03-04T06:24:40.264 INFO:tasks.rados.rados.0.smithi064.stdout:3852: read oid 43 snap 498 2024-03-04T06:24:40.265 INFO:tasks.rados.rados.0.smithi064.stdout:3852: expect deleted 2024-03-04T06:24:40.265 INFO:tasks.rados.rados.0.smithi064.stdout:3853: snap_create 2024-03-04T06:24:40.369 INFO:tasks.rados.rados.0.smithi064.stdout:3852: done (1 left) 2024-03-04T06:24:40.369 INFO:tasks.rados.rados.0.smithi064.stdout:3853: done (0 left) 2024-03-04T06:24:40.370 INFO:tasks.rados.rados.0.smithi064.stdout:3854: snap_create 2024-03-04T06:24:41.375 INFO:tasks.rados.rados.0.smithi064.stdout:3854: done (0 left) 2024-03-04T06:24:41.375 INFO:tasks.rados.rados.0.smithi064.stdout:3855: snap_create 2024-03-04T06:24:41.537 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:24:41.537 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:24:41.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:24:41.658 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:41.654+0000 7f3627e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:41.661 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:41.660+0000 7f3627e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:41.750 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:41.853 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:24:41.854 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:24:41.855 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:41.854+0000 7f9f4ed1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:41.858 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:41.858+0000 7f9f4ed1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:41.867 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1509,"stamp":"2024-03-04T06:24:41.374183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":79897608,"num_objects":1275,"num_object_clones":1058,"num_object_copies":3825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":187,"num_read":6885,"num_read_kb":5399683,"num_write":5870,"num_write_kb":1565065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17660,"ondisk_log_size":17660,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":731988,"kb_used_data":509492,"kb_used_omap":0,"kb_used_meta":222464,"kb_avail":376755372,"statfs":{"total":386547056640,"available":385797500928,"internally_reserved":0,"allocated":521719808,"data_stored":594995058,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":227803136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2017236,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":22,"num_read_kb":22334,"num_write":11,"num_write_kb":2505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014611"},"pg_stats":[{"pgid":"3.f","version":"1007'978","reported_seq":3422,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:40.286323+0000","last_change":"2024-03-04T06:24:38.592101+0000","last_active":"2024-03-04T06:24:40.286323+0000","last_peered":"2024-03-04T06:24:40.286323+0000","last_clean":"2024-03-04T06:24:40.286323+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:24:40.286323+0000","last_undegraded":"2024-03-04T06:24:40.286323+0000","last_fullsized":"2024-03-04T06:24:40.286323+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":374,"num_read_kb":295545,"num_write":337,"num_write_kb":88619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"1007'809","reported_seq":3415,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:39.388848+0000","last_change":"2024-03-04T06:24:38.519952+0000","last_active":"2024-03-04T06:24:39.388848+0000","last_peered":"2024-03-04T06:24:39.388848+0000","last_clean":"2024-03-04T06:24:39.388848+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:24:39.388848+0000","last_undegraded":"2024-03-04T06:24:39.388848+0000","last_fullsized":"2024-03-04T06:24:39.388848+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":64,"num_object_clones":47,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":17,"num_read":258,"num_read_kb":210899,"num_write":257,"num_write_kb":67532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"1002'1352","reported_seq":3621,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-03-04T06:24:34.362325+0000","last_change":"2024-03-04T06:24:33.403802+0000","last_active":"2024-03-04T06:24:34.362325+0000","last_peered":"2024-03-04T06:24:34.362325+0000","last_clean":"2024-03-04T06:24:34.362325+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:24:34.362325+0000","last_undegraded":"2024-03-04T06:24:34.362325+0000","last_fullsized":"2024-03-04T06:24:34.362325+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1352,"log_dups_size":0,"ondisk_log_size":1352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10169918,"num_objects":100,"num_object_clones":85,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":593,"num_read_kb":473296,"num_write":460,"num_write_kb":125339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"1007'1118","reported_seq":3408,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:40.281033+0000","last_change":"2024-03-04T06:24:38.540097+0000","last_active":"2024-03-04T06:24:40.281033+0000","last_peered":"2024-03-04T06:24:40.281033+0000","last_clean":"2024-03-04T06:24:40.281033+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:24:40.281033+0000","last_undegraded":"2024-03-04T06:24:40.281033+0000","last_fullsized":"2024-03-04T06:24:40.281033+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1118,"log_dups_size":0,"ondisk_log_size":1118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5194221,"num_objects":71,"num_object_clones":61,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":341,"num_read_kb":257705,"num_write":335,"num_write_kb":92846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"999'435","reported_seq":3168,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-03-04T06:24:35.379043+0000","last_change":"2024-03-04T06:24:33.936914+0000","last_active":"2024-03-04T06:24:35.379043+0000","last_peered":"2024-03-04T06:24:35.379043+0000","last_clean":"2024-03-04T06:24:35.379043+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:24:35.379043+0000","last_undegraded":"2024-03-04T06:24:35.379043+0000","last_fullsized":"2024-03-04T06:24:35.379043+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"1007'1072","reported_seq":3485,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:40.284354+0000","last_change":"2024-03-04T06:24:38.551681+0000","last_active":"2024-03-04T06:24:40.284354+0000","last_peered":"2024-03-04T06:24:40.284354+0000","last_clean":"2024-03-04T06:24:40.284354+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:24:40.284354+0000","last_undegraded":"2024-03-04T06:24:40.284354+0000","last_fullsized":"2024-03-04T06:24:40.284354+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790853,"num_objects":77,"num_object_clones":65,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":444,"num_read_kb":355101,"num_write":369,"num_write_kb":106221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2953,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:40.286938+0000","last_change":"2024-03-04T06:24:38.518281+0000","last_active":"2024-03-04T06:24:40.286938+0000","last_peered":"2024-03-04T06:24:40.286938+0000","last_clean":"2024-03-04T06:24:40.286938+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:24:40.286938+0000","last_undegraded":"2024-03-04T06:24:40.286938+0000","last_fullsized":"2024-03-04T06:24:40.286938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"1003'698","reported_seq":3255,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-03-04T06:24:34.505455+0000","last_change":"2024-03-04T06:24:33.395862+0000","last_active":"2024-03-04T06:24:34.505455+0000","last_peered":"2024-03-04T06:24:34.505455+0000","last_clean":"2024-03-04T06:24:34.505455+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:24:34.505455+0000","last_undegraded":"2024-03-04T06:24:34.505455+0000","last_fullsized":"2024-03-04T06:24:34.505455+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3076110,"num_objects":45,"num_object_clones":36,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":289,"num_read_kb":255609,"num_write":198,"num_write_kb":53672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2661,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-03-04T06:24:35.368139+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:24:35.368139+0000","last_peered":"2024-03-04T06:24:35.368139+0000","last_clean":"2024-03-04T06:24:35.368139+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:24:35.368139+0000","last_undegraded":"2024-03-04T06:24:35.368139+0000","last_fullsized":"2024-03-04T06:24:35.368139+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2475,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-03-04T06:24:34.370734+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:24:34.370734+0000","last_peered":"2024-03-04T06:24:34.370734+0000","last_clean":"2024-03-04T06:24:34.370734+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:24:34.370734+0000","last_undegraded":"2024-03-04T06:24:34.370734+0000","last_fullsized":"2024-03-04T06:24:34.370734+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"1002'1015","reported_seq":3455,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-03-04T06:24:34.370587+0000","last_change":"2024-03-04T06:24:33.387536+0000","last_active":"2024-03-04T06:24:34.370587+0000","last_peered":"2024-03-04T06:24:34.370587+0000","last_clean":"2024-03-04T06:24:34.370587+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:24:34.370587+0000","last_undegraded":"2024-03-04T06:24:34.370587+0000","last_fullsized":"2024-03-04T06:24:34.370587+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1015,"log_dups_size":0,"ondisk_log_size":1015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8001419,"num_objects":74,"num_object_clones":60,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":454,"num_read_kb":370095,"num_write":335,"num_write_kb":90183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2478,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:40.284737+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:24:40.284737+0000","last_peered":"2024-03-04T06:24:40.284737+0000","last_clean":"2024-03-04T06:24:40.284737+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:24:40.284737+0000","last_undegraded":"2024-03-04T06:24:40.284737+0000","last_fullsized":"2024-03-04T06:24:40.284737+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"1003'713","reported_seq":3357,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-03-04T06:24:35.399400+0000","last_change":"2024-03-04T06:24:33.947758+0000","last_active":"2024-03-04T06:24:35.399400+0000","last_peered":"2024-03-04T06:24:35.399400+0000","last_clean":"2024-03-04T06:24:35.399400+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:24:35.399400+0000","last_undegraded":"2024-03-04T06:24:35.399400+0000","last_fullsized":"2024-03-04T06:24:35.399400+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3025689,"num_objects":54,"num_object_clones":45,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":332,"num_read_kb":275601,"num_write":229,"num_write_kb":65393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2475,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-03-04T06:24:34.363705+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:24:34.363705+0000","last_peered":"2024-03-04T06:24:34.363705+0000","last_clean":"2024-03-04T06:24:34.363705+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:24:34.363705+0000","last_undegraded":"2024-03-04T06:24:34.363705+0000","last_fullsized":"2024-03-04T06:24:34.363705+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"1000'1991","reported_seq":3989,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-03-04T06:24:35.375334+0000","last_change":"2024-03-04T06:24:33.936186+0000","last_active":"2024-03-04T06:24:35.375334+0000","last_peered":"2024-03-04T06:24:35.375334+0000","last_clean":"2024-03-04T06:24:35.375334+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:24:35.375334+0000","last_undegraded":"2024-03-04T06:24:35.375334+0000","last_fullsized":"2024-03-04T06:24:35.375334+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1991,"log_dups_size":0,"ondisk_log_size":1991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11178305,"num_objects":148,"num_object_clones":122,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":822,"num_read_kb":645887,"num_write":688,"num_write_kb":185126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2478,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:40.282598+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:24:40.282598+0000","last_peered":"2024-03-04T06:24:40.282598+0000","last_clean":"2024-03-04T06:24:40.282598+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:24:40.282598+0000","last_undegraded":"2024-03-04T06:24:40.282598+0000","last_fullsized":"2024-03-04T06:24:40.282598+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"1008'2347","reported_seq":4057,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:39.500274+0000","last_change":"2024-03-04T06:24:38.634114+0000","last_active":"2024-03-04T06:24:39.500274+0000","last_peered":"2024-03-04T06:24:39.500274+0000","last_clean":"2024-03-04T06:24:39.500274+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:24:39.500274+0000","last_undegraded":"2024-03-04T06:24:39.500274+0000","last_fullsized":"2024-03-04T06:24:39.500274+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2347,"log_dups_size":0,"ondisk_log_size":2347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8656892,"num_objects":164,"num_object_clones":140,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":747,"num_read_kb":548040,"num_write":745,"num_write_kb":201378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"999'979","reported_seq":3449,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-03-04T06:24:34.366885+0000","last_change":"2024-03-04T06:24:33.396365+0000","last_active":"2024-03-04T06:24:34.366885+0000","last_peered":"2024-03-04T06:24:34.366885+0000","last_clean":"2024-03-04T06:24:34.366885+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:24:34.366885+0000","last_undegraded":"2024-03-04T06:24:34.366885+0000","last_fullsized":"2024-03-04T06:24:34.366885+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":979,"log_dups_size":0,"ondisk_log_size":979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521479,"num_objects":70,"num_object_clones":57,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":453,"num_read_kb":381495,"num_write":327,"num_write_kb":88741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2592,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:39.402381+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:24:39.402381+0000","last_peered":"2024-03-04T06:24:39.402381+0000","last_clean":"2024-03-04T06:24:39.402381+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:24:39.402381+0000","last_undegraded":"2024-03-04T06:24:39.402381+0000","last_fullsized":"2024-03-04T06:24:39.402381+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"1007'1145","reported_seq":3584,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:39.424420+0000","last_change":"2024-03-04T06:24:38.481255+0000","last_active":"2024-03-04T06:24:39.424420+0000","last_peered":"2024-03-04T06:24:39.424420+0000","last_clean":"2024-03-04T06:24:39.424420+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:24:39.424420+0000","last_undegraded":"2024-03-04T06:24:39.424420+0000","last_fullsized":"2024-03-04T06:24:39.424420+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1145,"log_dups_size":0,"ondisk_log_size":1145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8942578,"num_objects":86,"num_object_clones":69,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":412,"num_read_kb":309847,"num_write":376,"num_write_kb":101479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2528,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-03-04T06:24:35.376918+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:24:35.376918+0000","last_peered":"2024-03-04T06:24:35.376918+0000","last_clean":"2024-03-04T06:24:35.376918+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:24:35.376918+0000","last_undegraded":"2024-03-04T06:24:35.376918+0000","last_fullsized":"2024-03-04T06:24:35.376918+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"1002'1507","reported_seq":3656,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-03-04T06:24:35.368883+0000","last_change":"2024-03-04T06:24:33.947258+0000","last_active":"2024-03-04T06:24:35.368883+0000","last_peered":"2024-03-04T06:24:35.368883+0000","last_clean":"2024-03-04T06:24:35.368883+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:24:35.368883+0000","last_undegraded":"2024-03-04T06:24:35.368883+0000","last_fullsized":"2024-03-04T06:24:35.368883+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6320257,"num_objects":104,"num_object_clones":88,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":517,"num_read_kb":404684,"num_write":479,"num_write_kb":133127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2592,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:39.395358+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:24:39.395358+0000","last_peered":"2024-03-04T06:24:39.395358+0000","last_clean":"2024-03-04T06:24:39.395358+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:24:39.395358+0000","last_undegraded":"2024-03-04T06:24:39.395358+0000","last_fullsized":"2024-03-04T06:24:39.395358+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"1007'1404","reported_seq":3775,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-03-04T06:24:39.691624+0000","last_change":"2024-03-04T06:24:38.575677+0000","last_active":"2024-03-04T06:24:39.691624+0000","last_peered":"2024-03-04T06:24:39.691624+0000","last_clean":"2024-03-04T06:24:39.691624+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:24:39.691624+0000","last_undegraded":"2024-03-04T06:24:39.691624+0000","last_fullsized":"2024-03-04T06:24:39.691624+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1404,"log_dups_size":0,"ondisk_log_size":1404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5429519,"num_objects":106,"num_object_clones":92,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":623,"num_read_kb":522640,"num_write":472,"num_write_kb":128902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2475,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-03-04T06:24:34.372041+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:24:34.372041+0000","last_peered":"2024-03-04T06:24:34.372041+0000","last_clean":"2024-03-04T06:24:34.372041+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:24:34.372041+0000","last_undegraded":"2024-03-04T06:24:34.372041+0000","last_fullsized":"2024-03-04T06:24:34.372041+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":79307240,"num_objects":1273,"num_object_clones":1058,"num_object_copies":3819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":187,"num_read":6782,"num_read_kb":5399592,"num_write":5733,"num_write_kb":1563171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17563,"ondisk_log_size":17563,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":236472,"kb_used_data":165232,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94135368,"statfs":{"total":96636764160,"available":96394616832,"internally_reserved":0,"allocated":169197568,"data_stored":197526797,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166312,"kb_used_data":121440,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94205528,"statfs":{"total":96636764160,"available":96466460672,"internally_reserved":0,"allocated":124354560,"data_stored":143341501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706678,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":155708,"kb_used_data":110836,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94216132,"statfs":{"total":96636764160,"available":96477319168,"internally_reserved":0,"allocated":113496064,"data_stored":125802773,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706678,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":173496,"kb_used_data":111984,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94198344,"statfs":{"total":96636764160,"available":96459104256,"internally_reserved":0,"allocated":114671616,"data_stored":128323987,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:24:42.376 INFO:tasks.rados.rados.0.smithi064.stdout:3855: done (0 left) 2024-03-04T06:24:42.376 INFO:tasks.rados.rados.0.smithi064.stdout:3856: write (excl) oid 14 current snap is 507 2024-03-04T06:24:42.376 INFO:tasks.rados.rados.0.smithi064.stdout:3856: seq_num 1736 ranges {776952=760748,2203211=407276} 2024-03-04T06:24:42.384 INFO:tasks.rados.rados.0.smithi064.stdout:3856: writing smithi06431743-14 from 776952 to 1537700 tid 1 2024-03-04T06:24:42.390 INFO:tasks.rados.rados.0.smithi064.stdout:3856: writing smithi06431743-14 from 2203211 to 2610487 tid 2 2024-03-04T06:24:42.390 INFO:tasks.rados.rados.0.smithi064.stdout:3857: write (excl) oid 27 current snap is 507 2024-03-04T06:24:42.390 INFO:tasks.rados.rados.0.smithi064.stdout:3857: seq_num 1737 ranges {793759=435997,1861693=550934,2850562=343197} 2024-03-04T06:24:42.396 INFO:tasks.rados.rados.0.smithi064.stdout:3857: writing smithi06431743-27 from 793759 to 1229756 tid 1 2024-03-04T06:24:42.403 INFO:tasks.rados.rados.0.smithi064.stdout:3857: writing smithi06431743-27 from 1861693 to 2412627 tid 2 2024-03-04T06:24:42.407 INFO:tasks.rados.rados.0.smithi064.stdout:3857: writing smithi06431743-27 from 2850562 to 3193759 tid 3 2024-03-04T06:24:42.408 INFO:tasks.rados.rados.0.smithi064.stdout:3858: read oid 5 snap -1 2024-03-04T06:24:42.408 INFO:tasks.rados.rados.0.smithi064.stdout:3858: expect (ObjNum 1724 snap 500 seq_num 1724) 2024-03-04T06:24:42.408 INFO:tasks.rados.rados.0.smithi064.stdout:3859: write oid 28 current snap is 507 2024-03-04T06:24:42.408 INFO:tasks.rados.rados.0.smithi064.stdout:3859: seq_num 1738 ranges {410566=511246,1520175=410945,2477985=604876} 2024-03-04T06:24:42.412 INFO:tasks.rados.rados.0.smithi064.stdout:3859: writing smithi06431743-28 from 410566 to 921812 tid 1 2024-03-04T06:24:42.418 INFO:tasks.rados.rados.0.smithi064.stdout:3859: writing smithi06431743-28 from 1520175 to 1931120 tid 2 2024-03-04T06:24:42.424 INFO:tasks.rados.rados.0.smithi064.stdout:3859: writing smithi06431743-28 from 2477985 to 3082861 tid 3 2024-03-04T06:24:42.424 INFO:tasks.rados.rados.0.smithi064.stdout:3860: write oid 49 current snap is 507 2024-03-04T06:24:42.424 INFO:tasks.rados.rados.0.smithi064.stdout:3860: seq_num 1739 ranges {427373=586495,1462304=554603,2789055=416148} 2024-03-04T06:24:42.431 INFO:tasks.rados.rados.0.smithi064.stdout:3860: writing smithi06431743-49 from 427373 to 1013868 tid 1 2024-03-04T06:24:42.438 INFO:tasks.rados.rados.0.smithi064.stdout:3860: writing smithi06431743-49 from 1462304 to 2016907 tid 2 2024-03-04T06:24:42.446 INFO:tasks.rados.rados.0.smithi064.stdout:3860: writing smithi06431743-49 from 2789055 to 3205203 tid 3 2024-03-04T06:24:42.446 INFO:tasks.rados.rados.0.smithi064.stdout:3861: read oid 18 snap -1 2024-03-04T06:24:42.446 INFO:tasks.rados.rados.0.smithi064.stdout:3861: expect deleted 2024-03-04T06:24:42.446 INFO:tasks.rados.rados.0.smithi064.stdout:3862: snap_remove snap 506 2024-03-04T06:24:43.354 INFO:tasks.rados.rados.0.smithi064.stdout:3862: done (6 left) 2024-03-04T06:24:43.354 INFO:tasks.rados.rados.0.smithi064.stdout:3863: read oid 43 snap -1 2024-03-04T06:24:43.354 INFO:tasks.rados.rados.0.smithi064.stdout:3863: expect deleted 2024-03-04T06:24:43.354 INFO:tasks.rados.rados.0.smithi064.stdout:3864: delete oid 8 current snap is 507 2024-03-04T06:24:43.355 INFO:tasks.rados.rados.0.smithi064.stdout:3858: expect (ObjNum 1724 snap 500 seq_num 1724) 2024-03-04T06:24:43.596 INFO:tasks.rados.rados.0.smithi064.stdout:3856: finishing write tid 1 to smithi06431743-14 2024-03-04T06:24:43.596 INFO:tasks.rados.rados.0.smithi064.stdout:3857: finishing write tid 1 to smithi06431743-27 2024-03-04T06:24:43.596 INFO:tasks.rados.rados.0.smithi064.stdout:3859: finishing write tid 1 to smithi06431743-28 2024-03-04T06:24:43.596 INFO:tasks.rados.rados.0.smithi064.stdout:3857: finishing write tid 2 to smithi06431743-27 2024-03-04T06:24:43.596 INFO:tasks.rados.rados.0.smithi064.stdout:3859: finishing write tid 2 to smithi06431743-28 2024-03-04T06:24:43.596 INFO:tasks.rados.rados.0.smithi064.stdout:3856: finishing write tid 2 to smithi06431743-14 2024-03-04T06:24:43.596 INFO:tasks.rados.rados.0.smithi064.stdout:3859: finishing write tid 3 to smithi06431743-28 2024-03-04T06:24:43.596 INFO:tasks.rados.rados.0.smithi064.stdout:3856: finishing write tid 3 to smithi06431743-14 2024-03-04T06:24:43.596 INFO:tasks.rados.rados.0.smithi064.stdout:3858: done (7 left) 2024-03-04T06:24:43.596 INFO:tasks.rados.rados.0.smithi064.stdout:3864: done (6 left) 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3865: read oid 39 snap -1 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3865: expect (ObjNum 1687 snap 490 seq_num 1687) 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3866: write oid 41 current snap is 507 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3857: finishing write tid 3 to smithi06431743-27 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3856: finishing write tid 4 to smithi06431743-14 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3856: oid 14 updating version 0 to 1517 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3856: oid 14 updating version 1517 to 1518 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3856: oid 14 updating version 1518 to 1519 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 1519 (ObjNum 1736 snap 507 seq_num 1736) dirty exists 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3856: left oid 14 (ObjNum 1736 snap 507 seq_num 1736) 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3860: finishing write tid 1 to smithi06431743-49 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3859: finishing write tid 4 to smithi06431743-28 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3857: finishing write tid 4 to smithi06431743-27 2024-03-04T06:24:43.597 INFO:tasks.rados.rados.0.smithi064.stdout:3866: seq_num 1741 ranges {460987=453346,1695621=558272,2960254=300733} 2024-03-04T06:24:43.600 INFO:tasks.rados.rados.0.smithi064.stdout:3866: writing smithi06431743-41 from 460987 to 914333 tid 1 2024-03-04T06:24:43.608 INFO:tasks.rados.rados.0.smithi064.stdout:3866: writing smithi06431743-41 from 1695621 to 2253893 tid 2 2024-03-04T06:24:43.614 INFO:tasks.rados.rados.0.smithi064.stdout:3866: writing smithi06431743-41 from 2960254 to 3260987 tid 3 2024-03-04T06:24:43.614 INFO:tasks.rados.rados.0.smithi064.stdout:3856: done (7 left) 2024-03-04T06:24:43.614 INFO:tasks.rados.rados.0.smithi064.stdout:3867: snap_create 2024-03-04T06:24:43.615 INFO:tasks.rados.rados.0.smithi064.stdout:3859: finishing write tid 5 to smithi06431743-28 2024-03-04T06:24:43.615 INFO:tasks.rados.rados.0.smithi064.stdout:3859: oid 28 updating version 0 to 2002 2024-03-04T06:24:43.615 INFO:tasks.rados.rados.0.smithi064.stdout:3859: oid 28 updating version 2002 to 2003 2024-03-04T06:24:43.615 INFO:tasks.rados.rados.0.smithi064.stdout:3859: oid 28 updating version 2003 to 2004 2024-03-04T06:24:43.615 INFO:tasks.rados.rados.0.smithi064.stdout:3859: oid 28 updating version 2004 to 2005 2024-03-04T06:24:43.615 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 2005 (ObjNum 1738 snap 507 seq_num 1738) dirty exists 2024-03-04T06:24:43.615 INFO:tasks.rados.rados.0.smithi064.stdout:3859: left oid 28 (ObjNum 1738 snap 507 seq_num 1738) 2024-03-04T06:24:43.615 INFO:tasks.rados.rados.0.smithi064.stdout:3857: finishing write tid 5 to smithi06431743-27 2024-03-04T06:24:43.615 INFO:tasks.rados.rados.0.smithi064.stdout:3857: oid 27 updating version 0 to 811 2024-03-04T06:24:43.615 INFO:tasks.rados.rados.0.smithi064.stdout:3857: oid 27 updating version 811 to 812 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3857: oid 27 version 812 is already newer than 810 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3857: oid 27 updating version 812 to 813 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 813 (ObjNum 1737 snap 507 seq_num 1737) dirty exists 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3857: left oid 27 (ObjNum 1737 snap 507 seq_num 1737) 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3860: finishing write tid 2 to smithi06431743-49 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3860: finishing write tid 3 to smithi06431743-49 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3860: finishing write tid 4 to smithi06431743-49 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3860: finishing write tid 5 to smithi06431743-49 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3860: oid 49 updating version 0 to 706 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3860: oid 49 version 706 is already newer than 704 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3860: oid 49 updating version 706 to 707 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3860: oid 49 version 707 is already newer than 705 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 707 (ObjNum 1739 snap 507 seq_num 1739) dirty exists 2024-03-04T06:24:43.616 INFO:tasks.rados.rados.0.smithi064.stdout:3860: left oid 49 (ObjNum 1739 snap 507 seq_num 1739) 2024-03-04T06:24:43.643 INFO:tasks.rados.rados.0.smithi064.stdout:3865: expect (ObjNum 1687 snap 490 seq_num 1687) 2024-03-04T06:24:43.737 INFO:tasks.rados.rados.0.smithi064.stdout:3866: finishing write tid 1 to smithi06431743-41 2024-03-04T06:24:43.737 INFO:tasks.rados.rados.0.smithi064.stdout:3866: finishing write tid 2 to smithi06431743-41 2024-03-04T06:24:43.737 INFO:tasks.rados.rados.0.smithi064.stdout:3866: finishing write tid 3 to smithi06431743-41 2024-03-04T06:24:43.737 INFO:tasks.rados.rados.0.smithi064.stdout:3866: finishing write tid 4 to smithi06431743-41 2024-03-04T06:24:43.737 INFO:tasks.rados.rados.0.smithi064.stdout:3866: finishing write tid 5 to smithi06431743-41 2024-03-04T06:24:43.737 INFO:tasks.rados.rados.0.smithi064.stdout:3866: oid 41 updating version 0 to 1028 2024-03-04T06:24:43.737 INFO:tasks.rados.rados.0.smithi064.stdout:3866: oid 41 version 1028 is already newer than 1025 2024-03-04T06:24:43.737 INFO:tasks.rados.rados.0.smithi064.stdout:3866: oid 41 version 1028 is already newer than 1027 2024-03-04T06:24:43.737 INFO:tasks.rados.rados.0.smithi064.stdout:3866: oid 41 version 1028 is already newer than 1026 2024-03-04T06:24:43.737 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 1028 (ObjNum 1741 snap 507 seq_num 1741) dirty exists 2024-03-04T06:24:43.737 INFO:tasks.rados.rados.0.smithi064.stdout:3866: left oid 41 (ObjNum 1741 snap 507 seq_num 1741) 2024-03-04T06:24:43.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:44.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:44.038+0000 7f11a095d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:44.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:44.043+0000 7f11a095d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:44.389 INFO:tasks.rados.rados.0.smithi064.stdout:3857: done (7 left) 2024-03-04T06:24:44.389 INFO:tasks.rados.rados.0.smithi064.stdout:3859: done (6 left) 2024-03-04T06:24:44.389 INFO:tasks.rados.rados.0.smithi064.stdout:3860: done (5 left) 2024-03-04T06:24:44.389 INFO:tasks.rados.rados.0.smithi064.stdout:3861: done (4 left) 2024-03-04T06:24:44.389 INFO:tasks.rados.rados.0.smithi064.stdout:3863: done (3 left) 2024-03-04T06:24:44.389 INFO:tasks.rados.rados.0.smithi064.stdout:3865: done (2 left) 2024-03-04T06:24:44.389 INFO:tasks.rados.rados.0.smithi064.stdout:3866: done (1 left) 2024-03-04T06:24:44.389 INFO:tasks.rados.rados.0.smithi064.stdout:3867: done (0 left) 2024-03-04T06:24:44.389 INFO:tasks.rados.rados.0.smithi064.stdout:3868: delete oid 24 current snap is 508 2024-03-04T06:24:44.427 INFO:tasks.rados.rados.0.smithi064.stdout:3868: done (0 left) 2024-03-04T06:24:44.427 INFO:tasks.rados.rados.0.smithi064.stdout:3869: write (excl) oid 22 current snap is 508 2024-03-04T06:24:44.427 INFO:tasks.rados.rados.0.smithi064.stdout:3869: seq_num 1743 ranges {494601=603844,1812585=561941,3015100=279501} 2024-03-04T06:24:44.432 INFO:tasks.rados.rados.0.smithi064.stdout:3869: writing smithi06431743-22 from 494601 to 1098445 tid 1 2024-03-04T06:24:44.439 INFO:tasks.rados.rados.0.smithi064.stdout:3869: writing smithi06431743-22 from 1812585 to 2374526 tid 2 2024-03-04T06:24:44.444 INFO:tasks.rados.rados.0.smithi064.stdout:3869: writing smithi06431743-22 from 3015100 to 3294601 tid 3 2024-03-04T06:24:44.445 INFO:tasks.rados.rados.0.smithi064.stdout:3870: delete oid 42 current snap is 508 2024-03-04T06:24:44.448 INFO:tasks.rados.rados.0.smithi064.stdout:3870: done (1 left) 2024-03-04T06:24:44.448 INFO:tasks.rados.rados.0.smithi064.stdout:3871: read oid 46 snap -1 2024-03-04T06:24:44.448 INFO:tasks.rados.rados.0.smithi064.stdout:3871: expect (ObjNum 1720 snap 500 seq_num 1720) 2024-03-04T06:24:44.448 INFO:tasks.rados.rados.0.smithi064.stdout:3872: read oid 8 snap -1 2024-03-04T06:24:44.448 INFO:tasks.rados.rados.0.smithi064.stdout:3872: expect deleted 2024-03-04T06:24:44.448 INFO:tasks.rados.rados.0.smithi064.stdout:3873: write oid 37 current snap is 508 2024-03-04T06:24:44.448 INFO:tasks.rados.rados.0.smithi064.stdout:3873: seq_num 1745 ranges {528215=754342,1813196=565610,2953593=374622} 2024-03-04T06:24:44.455 INFO:tasks.rados.rados.0.smithi064.stdout:3873: writing smithi06431743-37 from 528215 to 1282557 tid 1 2024-03-04T06:24:44.461 INFO:tasks.rados.rados.0.smithi064.stdout:3873: writing smithi06431743-37 from 1813196 to 2378806 tid 2 2024-03-04T06:24:44.466 INFO:tasks.rados.rados.0.smithi064.stdout:3873: writing smithi06431743-37 from 2953593 to 3328215 tid 3 2024-03-04T06:24:44.466 INFO:tasks.rados.rados.0.smithi064.stdout:3874: read oid 23 snap 487 2024-03-04T06:24:44.466 INFO:tasks.rados.rados.0.smithi064.stdout:3874: expect deleted 2024-03-04T06:24:44.466 INFO:tasks.rados.rados.0.smithi064.stdout:3875: read oid 13 snap -1 2024-03-04T06:24:44.467 INFO:tasks.rados.rados.0.smithi064.stdout:3875: expect deleted 2024-03-04T06:24:44.467 INFO:tasks.rados.rados.0.smithi064.stdout:3876: snap_create 2024-03-04T06:24:44.470 INFO:tasks.rados.rados.0.smithi064.stdout:3869: finishing write tid 1 to smithi06431743-22 2024-03-04T06:24:44.479 INFO:tasks.rados.rados.0.smithi064.stdout:3871: expect (ObjNum 1720 snap 500 seq_num 1720) 2024-03-04T06:24:44.588 INFO:tasks.rados.rados.0.smithi064.stdout:3873: finishing write tid 1 to smithi06431743-37 2024-03-04T06:24:44.588 INFO:tasks.rados.rados.0.smithi064.stdout:3869: finishing write tid 2 to smithi06431743-22 2024-03-04T06:24:44.588 INFO:tasks.rados.rados.0.smithi064.stdout:3869: finishing write tid 3 to smithi06431743-22 2024-03-04T06:24:44.588 INFO:tasks.rados.rados.0.smithi064.stdout:3873: finishing write tid 2 to smithi06431743-37 2024-03-04T06:24:44.588 INFO:tasks.rados.rados.0.smithi064.stdout:3869: finishing write tid 4 to smithi06431743-22 2024-03-04T06:24:44.588 INFO:tasks.rados.rados.0.smithi064.stdout:3869: finishing write tid 5 to smithi06431743-22 2024-03-04T06:24:44.588 INFO:tasks.rados.rados.0.smithi064.stdout:3869: oid 22 updating version 0 to 1406 2024-03-04T06:24:44.588 INFO:tasks.rados.rados.0.smithi064.stdout:3869: oid 22 updating version 1406 to 1409 2024-03-04T06:24:44.588 INFO:tasks.rados.rados.0.smithi064.stdout:3869: oid 22 version 1409 is already newer than 1408 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:3869: oid 22 version 1409 is already newer than 1408 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 1409 (ObjNum 1743 snap 508 seq_num 1743) dirty exists 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:3869: left oid 22 (ObjNum 1743 snap 508 seq_num 1743) 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:3873: finishing write tid 3 to smithi06431743-37 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:3873: finishing write tid 4 to smithi06431743-37 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:3873: finishing write tid 5 to smithi06431743-37 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:3873: oid 37 updating version 0 to 816 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:3873: oid 37 version 816 is already newer than 814 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:3873: oid 37 version 816 is already newer than 815 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:3873: oid 37 updating version 816 to 817 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 817 (ObjNum 1745 snap 508 seq_num 1745) dirty exists 2024-03-04T06:24:44.589 INFO:tasks.rados.rados.0.smithi064.stdout:3873: left oid 37 (ObjNum 1745 snap 508 seq_num 1745) 2024-03-04T06:24:45.395 INFO:tasks.rados.rados.0.smithi064.stdout:3869: done (6 left) 2024-03-04T06:24:45.395 INFO:tasks.rados.rados.0.smithi064.stdout:3871: done (5 left) 2024-03-04T06:24:45.396 INFO:tasks.rados.rados.0.smithi064.stdout:3872: done (4 left) 2024-03-04T06:24:45.396 INFO:tasks.rados.rados.0.smithi064.stdout:3873: done (3 left) 2024-03-04T06:24:45.396 INFO:tasks.rados.rados.0.smithi064.stdout:3874: done (2 left) 2024-03-04T06:24:45.396 INFO:tasks.rados.rados.0.smithi064.stdout:3875: done (1 left) 2024-03-04T06:24:45.396 INFO:tasks.rados.rados.0.smithi064.stdout:3876: done (0 left) 2024-03-04T06:24:45.396 INFO:tasks.rados.rados.0.smithi064.stdout:3877: read oid 30 snap -1 2024-03-04T06:24:45.396 INFO:tasks.rados.rados.0.smithi064.stdout:3877: expect deleted 2024-03-04T06:24:45.396 INFO:tasks.rados.rados.0.smithi064.stdout:3878: snap_remove snap 475 2024-03-04T06:24:46.112 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:46.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:46.230+0000 7f7bf92ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:46.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:46.234+0000 7f7bf92ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:46.395 INFO:tasks.rados.rados.0.smithi064.stdout:3878: done (1 left) 2024-03-04T06:24:46.396 INFO:tasks.rados.rados.0.smithi064.stdout:3879: delete oid 36 current snap is 509 2024-03-04T06:24:46.494 INFO:tasks.rados.rados.0.smithi064.stdout:3877: done (1 left) 2024-03-04T06:24:46.494 INFO:tasks.rados.rados.0.smithi064.stdout:3879: done (0 left) 2024-03-04T06:24:46.494 INFO:tasks.rados.rados.0.smithi064.stdout:3880: read oid 37 snap 496 2024-03-04T06:24:46.494 INFO:tasks.rados.rados.0.smithi064.stdout:3880: expect deleted 2024-03-04T06:24:46.494 INFO:tasks.rados.rados.0.smithi064.stdout:3881: read oid 32 snap 484 2024-03-04T06:24:46.494 INFO:tasks.rados.rados.0.smithi064.stdout:3881: expect (ObjNum 1670 snap 482 seq_num 1670) 2024-03-04T06:24:46.494 INFO:tasks.rados.rados.0.smithi064.stdout:3882: write oid 29 current snap is 509 2024-03-04T06:24:46.494 INFO:tasks.rados.rados.0.smithi064.stdout:3882: seq_num 1747 ranges {561829=504840,1530160=569279,2608439=739030} 2024-03-04T06:24:46.499 INFO:tasks.rados.rados.0.smithi064.stdout:3882: writing smithi06431743-29 from 561829 to 1066669 tid 1 2024-03-04T06:24:46.506 INFO:tasks.rados.rados.0.smithi064.stdout:3882: writing smithi06431743-29 from 1530160 to 2099439 tid 2 2024-03-04T06:24:46.513 INFO:tasks.rados.rados.0.smithi064.stdout:3882: writing smithi06431743-29 from 2608439 to 3347469 tid 3 2024-03-04T06:24:46.514 INFO:tasks.rados.rados.0.smithi064.stdout:3883: read oid 9 snap -1 2024-03-04T06:24:46.514 INFO:tasks.rados.rados.0.smithi064.stdout:3883: expect deleted 2024-03-04T06:24:46.514 INFO:tasks.rados.rados.0.smithi064.stdout:3884: read oid 38 snap -1 2024-03-04T06:24:46.514 INFO:tasks.rados.rados.0.smithi064.stdout:3884: expect (ObjNum 1731 snap 503 seq_num 1731) 2024-03-04T06:24:46.514 INFO:tasks.rados.rados.0.smithi064.stdout:3885: delete oid 44 current snap is 509 2024-03-04T06:24:46.551 INFO:tasks.rados.rados.0.smithi064.stdout:3884: expect (ObjNum 1731 snap 503 seq_num 1731) 2024-03-04T06:24:46.715 INFO:tasks.rados.rados.0.smithi064.stdout:3881: expect (ObjNum 1670 snap 482 seq_num 1670) 2024-03-04T06:24:46.853 INFO:tasks.rados.rados.0.smithi064.stdout:3882: finishing write tid 1 to smithi06431743-29 2024-03-04T06:24:46.854 INFO:tasks.rados.rados.0.smithi064.stdout:3882: finishing write tid 2 to smithi06431743-29 2024-03-04T06:24:46.854 INFO:tasks.rados.rados.0.smithi064.stdout:3882: finishing write tid 3 to smithi06431743-29 2024-03-04T06:24:46.854 INFO:tasks.rados.rados.0.smithi064.stdout:3880: done (5 left) 2024-03-04T06:24:46.854 INFO:tasks.rados.rados.0.smithi064.stdout:3881: done (4 left) 2024-03-04T06:24:46.854 INFO:tasks.rados.rados.0.smithi064.stdout:3883: done (3 left) 2024-03-04T06:24:46.855 INFO:tasks.rados.rados.0.smithi064.stdout:3884: done (2 left) 2024-03-04T06:24:46.855 INFO:tasks.rados.rados.0.smithi064.stdout:3885: done (1 left) 2024-03-04T06:24:46.855 INFO:tasks.rados.rados.0.smithi064.stdout:3886: read oid 14 snap -1 2024-03-04T06:24:46.855 INFO:tasks.rados.rados.0.smithi064.stdout:3886: expect (ObjNum 1736 snap 507 seq_num 1736) 2024-03-04T06:24:46.855 INFO:tasks.rados.rados.0.smithi064.stdout:3887: write oid 24 current snap is 509 2024-03-04T06:24:46.855 INFO:tasks.rados.rados.0.smithi064.stdout:3887: seq_num 1749 ranges {595443=771691,2047124=572948,3063285=332158} 2024-03-04T06:24:46.859 INFO:tasks.rados.rados.0.smithi064.stdout:3887: writing smithi06431743-24 from 595443 to 1367134 tid 1 2024-03-04T06:24:46.866 INFO:tasks.rados.rados.0.smithi064.stdout:3887: writing smithi06431743-24 from 2047124 to 2620072 tid 2 2024-03-04T06:24:46.870 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:24:46.870 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:24:46.871 INFO:tasks.rados.rados.0.smithi064.stdout:3887: writing smithi06431743-24 from 3063285 to 3395443 tid 3 2024-03-04T06:24:46.871 INFO:tasks.rados.rados.0.smithi064.stdout:3888: snap_remove snap 495 2024-03-04T06:24:46.988 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:46.982+0000 7f4c41864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:46.991 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:46.991+0000 7f4c41864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:47.398 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:24:47.402 INFO:tasks.rados.rados.0.smithi064.stdout:3888: done (3 left) 2024-03-04T06:24:47.402 INFO:tasks.rados.rados.0.smithi064.stdout:3889: delete oid 15 current snap is 509 2024-03-04T06:24:47.402 INFO:tasks.rados.rados.0.smithi064.stdout:3882: finishing write tid 4 to smithi06431743-29 2024-03-04T06:24:47.402 INFO:tasks.rados.rados.0.smithi064.stdout:3882: finishing write tid 5 to smithi06431743-29 2024-03-04T06:24:47.402 INFO:tasks.rados.rados.0.smithi064.stdout:3882: oid 29 updating version 0 to 2360 2024-03-04T06:24:47.402 INFO:tasks.rados.rados.0.smithi064.stdout:3882: oid 29 updating version 2360 to 2361 2024-03-04T06:24:47.403 INFO:tasks.rados.rados.0.smithi064.stdout:3882: oid 29 version 2361 is already newer than 2358 2024-03-04T06:24:47.403 INFO:tasks.rados.rados.0.smithi064.stdout:3882: oid 29 version 2361 is already newer than 2359 2024-03-04T06:24:47.403 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 29 v 2361 (ObjNum 1747 snap 509 seq_num 1747) dirty exists 2024-03-04T06:24:47.403 INFO:tasks.rados.rados.0.smithi064.stdout:3882: left oid 29 (ObjNum 1747 snap 509 seq_num 1747) 2024-03-04T06:24:47.403 INFO:tasks.rados.rados.0.smithi064.stdout:3886: expect (ObjNum 1736 snap 507 seq_num 1736) 2024-03-04T06:24:47.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:24:47.528 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:47.522+0000 7f68c935d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:47.531 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:47.531+0000 7f68c935d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:47.615 INFO:tasks.rados.rados.0.smithi064.stdout:3887: finishing write tid 1 to smithi06431743-24 2024-03-04T06:24:47.615 INFO:tasks.rados.rados.0.smithi064.stdout:3887: finishing write tid 2 to smithi06431743-24 2024-03-04T06:24:47.615 INFO:tasks.rados.rados.0.smithi064.stdout:3887: finishing write tid 3 to smithi06431743-24 2024-03-04T06:24:47.615 INFO:tasks.rados.rados.0.smithi064.stdout:3887: finishing write tid 4 to smithi06431743-24 2024-03-04T06:24:47.615 INFO:tasks.rados.rados.0.smithi064.stdout:3887: finishing write tid 5 to smithi06431743-24 2024-03-04T06:24:47.615 INFO:tasks.rados.rados.0.smithi064.stdout:3887: oid 24 updating version 0 to 1529 2024-03-04T06:24:47.616 INFO:tasks.rados.rados.0.smithi064.stdout:3887: oid 24 version 1529 is already newer than 1526 2024-03-04T06:24:47.616 INFO:tasks.rados.rados.0.smithi064.stdout:3887: oid 24 version 1529 is already newer than 1528 2024-03-04T06:24:47.616 INFO:tasks.rados.rados.0.smithi064.stdout:3887: oid 24 version 1529 is already newer than 1527 2024-03-04T06:24:47.616 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 24 v 1529 (ObjNum 1749 snap 509 seq_num 1749) dirty exists 2024-03-04T06:24:47.616 INFO:tasks.rados.rados.0.smithi064.stdout:3887: left oid 24 (ObjNum 1749 snap 509 seq_num 1749) 2024-03-04T06:24:47.616 INFO:tasks.rados.rados.0.smithi064.stdout:3882: done (3 left) 2024-03-04T06:24:47.616 INFO:tasks.rados.rados.0.smithi064.stdout:3886: done (2 left) 2024-03-04T06:24:47.616 INFO:tasks.rados.rados.0.smithi064.stdout:3887: done (1 left) 2024-03-04T06:24:47.616 INFO:tasks.rados.rados.0.smithi064.stdout:3889: done (0 left) 2024-03-04T06:24:47.616 INFO:tasks.rados.rados.0.smithi064.stdout:3890: delete oid 26 current snap is 509 2024-03-04T06:24:47.657 INFO:tasks.rados.rados.0.smithi064.stdout:3890: done (0 left) 2024-03-04T06:24:47.657 INFO:tasks.rados.rados.0.smithi064.stdout:3891: snap_create 2024-03-04T06:24:48.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:48.401 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:24:48.404 INFO:tasks.rados.rados.0.smithi064.stdout:3891: done (0 left) 2024-03-04T06:24:48.404 INFO:tasks.rados.rados.0.smithi064.stdout:3892: snap_remove snap 496 2024-03-04T06:24:48.416 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:24:48.420 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:48.418+0000 7f8198b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:48.423 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:48.422+0000 7f8198b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:48.529 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:48.524+0000 7ff88c75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:48.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:48.532+0000 7ff88c75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:49.405 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:24:49.408 INFO:tasks.rados.rados.0.smithi064.stdout:3892: done (0 left) 2024-03-04T06:24:49.408 INFO:tasks.rados.rados.0.smithi064.stdout:3893: read oid 36 snap -1 2024-03-04T06:24:49.408 INFO:tasks.rados.rados.0.smithi064.stdout:3893: expect deleted 2024-03-04T06:24:49.408 INFO:tasks.rados.rados.0.smithi064.stdout:3894: snap_create 2024-03-04T06:24:49.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:24:49.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:49.572+0000 7f7e98b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:49.580 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:49.580+0000 7f7e98b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:50.405 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:24:50.408 INFO:tasks.rados.rados.0.smithi064.stdout:3893: done (1 left) 2024-03-04T06:24:50.408 INFO:tasks.rados.rados.0.smithi064.stdout:3894: done (0 left) 2024-03-04T06:24:50.408 INFO:tasks.rados.rados.0.smithi064.stdout:3895: read oid 21 snap -1 2024-03-04T06:24:50.408 INFO:tasks.rados.rados.0.smithi064.stdout:3895: expect deleted 2024-03-04T06:24:50.408 INFO:tasks.rados.rados.0.smithi064.stdout:3896: snap_create 2024-03-04T06:24:50.424 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T06:24:50.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:24:50.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:50.534 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:50.530+0000 7ffa7099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:50.537 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:50.537+0000 7ffa7099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:50.629 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:50.627+0000 7f7dbbb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:50.632 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:50.631+0000 7f7dbbb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:24:50.745 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:24:50.756 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1522,"stamp":"2024-03-04T06:24:50.406857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91109230,"num_objects":1283,"num_object_clones":1064,"num_object_copies":3849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":186,"num_read":6927,"num_read_kb":5436380,"num_write":5915,"num_write_kb":1578153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17818,"ondisk_log_size":17818,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":776064,"kb_used_data":517472,"kb_used_omap":0,"kb_used_meta":258560,"kb_avail":376711296,"statfs":{"total":386547056640,"available":385752367104,"internally_reserved":0,"allocated":529891328,"data_stored":595377211,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":264765440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4487931,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":12960,"num_write":25,"num_write_kb":7211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012203"},"pg_stats":[{"pgid":"3.f","version":"1007'978","reported_seq":3435,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-03-04T06:24:44.422587+0000","last_change":"2024-03-04T06:24:43.412028+0000","last_active":"2024-03-04T06:24:44.422587+0000","last_peered":"2024-03-04T06:24:44.422587+0000","last_clean":"2024-03-04T06:24:44.422587+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:24:44.422587+0000","last_undegraded":"2024-03-04T06:24:44.422587+0000","last_fullsized":"2024-03-04T06:24:44.422587+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":374,"num_read_kb":295545,"num_write":337,"num_write_kb":88619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"1018'825","reported_seq":3455,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.428222+0000","last_change":"2024-03-04T06:24:49.489095+0000","last_active":"2024-03-04T06:24:49.428222+0000","last_peered":"2024-03-04T06:24:49.428222+0000","last_clean":"2024-03-04T06:24:49.428222+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:24:49.428222+0000","last_undegraded":"2024-03-04T06:24:49.428222+0000","last_fullsized":"2024-03-04T06:24:49.428222+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6521974,"num_objects":65,"num_object_clones":47,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":261,"num_read_kb":210901,"num_write":267,"num_write_kb":70489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"1007'1356","reported_seq":3661,"reported_epoch":1015,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:46.410727+0000","last_change":"2024-03-04T06:24:46.410727+0000","last_active":"2024-03-04T06:24:46.410727+0000","last_peered":"2024-03-04T06:24:46.410727+0000","last_clean":"2024-03-04T06:24:46.410727+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:24:46.410727+0000","last_undegraded":"2024-03-04T06:24:46.410727+0000","last_fullsized":"2024-03-04T06:24:46.410727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1356,"log_dups_size":0,"ondisk_log_size":1356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10169918,"num_objects":100,"num_object_clones":85,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":605,"num_read_kb":483934,"num_write":460,"num_write_kb":125339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"1007'1118","reported_seq":3420,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-03-04T06:24:44.415134+0000","last_change":"2024-03-04T06:24:43.389593+0000","last_active":"2024-03-04T06:24:44.415134+0000","last_peered":"2024-03-04T06:24:44.415134+0000","last_clean":"2024-03-04T06:24:44.415134+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:24:44.415134+0000","last_undegraded":"2024-03-04T06:24:44.415134+0000","last_fullsized":"2024-03-04T06:24:44.415134+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1118,"log_dups_size":0,"ondisk_log_size":1118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5194221,"num_objects":71,"num_object_clones":61,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":341,"num_read_kb":257705,"num_write":335,"num_write_kb":92846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"999'435","reported_seq":3200,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.422443+0000","last_change":"2024-03-04T06:24:44.394523+0000","last_active":"2024-03-04T06:24:45.422443+0000","last_peered":"2024-03-04T06:24:45.422443+0000","last_clean":"2024-03-04T06:24:45.422443+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:24:45.422443+0000","last_undegraded":"2024-03-04T06:24:45.422443+0000","last_fullsized":"2024-03-04T06:24:45.422443+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"1007'1072","reported_seq":3497,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-03-04T06:24:44.424296+0000","last_change":"2024-03-04T06:24:43.413226+0000","last_active":"2024-03-04T06:24:44.424296+0000","last_peered":"2024-03-04T06:24:44.424296+0000","last_clean":"2024-03-04T06:24:44.424296+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:24:44.424296+0000","last_undegraded":"2024-03-04T06:24:44.424296+0000","last_fullsized":"2024-03-04T06:24:44.424296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790853,"num_objects":77,"num_object_clones":65,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":10,"num_read":444,"num_read_kb":355101,"num_write":369,"num_write_kb":106221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2965,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-03-04T06:24:44.422362+0000","last_change":"2024-03-04T06:24:43.412570+0000","last_active":"2024-03-04T06:24:44.422362+0000","last_peered":"2024-03-04T06:24:44.422362+0000","last_clean":"2024-03-04T06:24:44.422362+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:24:44.422362+0000","last_undegraded":"2024-03-04T06:24:44.422362+0000","last_fullsized":"2024-03-04T06:24:44.422362+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"1012'709","reported_seq":3294,"reported_epoch":1015,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:46.412129+0000","last_change":"2024-03-04T06:24:46.412129+0000","last_active":"2024-03-04T06:24:46.412129+0000","last_peered":"2024-03-04T06:24:46.412129+0000","last_clean":"2024-03-04T06:24:46.412129+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:24:46.412129+0000","last_undegraded":"2024-03-04T06:24:46.412129+0000","last_fullsized":"2024-03-04T06:24:46.412129+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227373,"num_objects":46,"num_object_clones":37,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":290,"num_read_kb":255610,"num_write":203,"num_write_kb":55194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2689,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.404692+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:24:45.404692+0000","last_peered":"2024-03-04T06:24:45.404692+0000","last_clean":"2024-03-04T06:24:45.404692+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:24:45.404692+0000","last_undegraded":"2024-03-04T06:24:45.404692+0000","last_fullsized":"2024-03-04T06:24:45.404692+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2505,"reported_epoch":1015,"state":"active+clean","last_fresh":"2024-03-04T06:24:46.419295+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:24:46.419295+0000","last_peered":"2024-03-04T06:24:46.419295+0000","last_clean":"2024-03-04T06:24:46.419295+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:24:46.419295+0000","last_undegraded":"2024-03-04T06:24:46.419295+0000","last_fullsized":"2024-03-04T06:24:46.419295+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"1012'1028","reported_seq":3494,"reported_epoch":1015,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:46.426273+0000","last_change":"2024-03-04T06:24:46.426273+0000","last_active":"2024-03-04T06:24:46.426273+0000","last_peered":"2024-03-04T06:24:46.426273+0000","last_clean":"2024-03-04T06:24:46.426273+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:24:46.426273+0000","last_undegraded":"2024-03-04T06:24:46.426273+0000","last_fullsized":"2024-03-04T06:24:46.426273+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1028,"log_dups_size":0,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8253524,"num_objects":75,"num_object_clones":61,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":455,"num_read_kb":370096,"num_write":340,"num_write_kb":91466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2489,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-03-04T06:24:44.423622+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:24:44.423622+0000","last_peered":"2024-03-04T06:24:44.423622+0000","last_clean":"2024-03-04T06:24:44.423622+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:24:44.423622+0000","last_undegraded":"2024-03-04T06:24:44.423622+0000","last_fullsized":"2024-03-04T06:24:44.423622+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"1007'728","reported_seq":3393,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.403979+0000","last_change":"2024-03-04T06:24:44.387071+0000","last_active":"2024-03-04T06:24:45.403979+0000","last_peered":"2024-03-04T06:24:45.403979+0000","last_clean":"2024-03-04T06:24:45.403979+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:24:45.403979+0000","last_undegraded":"2024-03-04T06:24:45.403979+0000","last_fullsized":"2024-03-04T06:24:45.403979+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":728,"log_dups_size":0,"ondisk_log_size":728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135413,"num_objects":54,"num_object_clones":45,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":333,"num_read_kb":275602,"num_write":233,"num_write_kb":66901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2505,"reported_epoch":1015,"state":"active+clean","last_fresh":"2024-03-04T06:24:46.413969+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:24:46.413969+0000","last_peered":"2024-03-04T06:24:46.413969+0000","last_clean":"2024-03-04T06:24:46.413969+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:24:46.413969+0000","last_undegraded":"2024-03-04T06:24:46.413969+0000","last_fullsized":"2024-03-04T06:24:46.413969+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"1011'2005","reported_seq":4032,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.421106+0000","last_change":"2024-03-04T06:24:44.396153+0000","last_active":"2024-03-04T06:24:45.421106+0000","last_peered":"2024-03-04T06:24:45.421106+0000","last_clean":"2024-03-04T06:24:45.421106+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:24:45.421106+0000","last_undegraded":"2024-03-04T06:24:45.421106+0000","last_fullsized":"2024-03-04T06:24:45.421106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2005,"log_dups_size":0,"ondisk_log_size":2005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14388871,"num_objects":149,"num_object_clones":122,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":835,"num_read_kb":664491,"num_write":693,"num_write_kb":186619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2489,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-03-04T06:24:44.416443+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:24:44.416443+0000","last_peered":"2024-03-04T06:24:44.416443+0000","last_clean":"2024-03-04T06:24:44.416443+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:24:44.416443+0000","last_undegraded":"2024-03-04T06:24:44.416443+0000","last_fullsized":"2024-03-04T06:24:44.416443+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"1018'2373","reported_seq":4095,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.423454+0000","last_change":"2024-03-04T06:24:49.520087+0000","last_active":"2024-03-04T06:24:49.423454+0000","last_peered":"2024-03-04T06:24:49.423454+0000","last_clean":"2024-03-04T06:24:49.423454+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:24:49.423454+0000","last_undegraded":"2024-03-04T06:24:49.423454+0000","last_fullsized":"2024-03-04T06:24:49.423454+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2373,"log_dups_size":0,"ondisk_log_size":2373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12018721,"num_objects":164,"num_object_clones":140,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":754,"num_read_kb":555488,"num_write":750,"num_write_kb":203150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"1006'983","reported_seq":3483,"reported_epoch":1015,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:46.422689+0000","last_change":"2024-03-04T06:24:46.422689+0000","last_active":"2024-03-04T06:24:46.422689+0000","last_peered":"2024-03-04T06:24:46.422689+0000","last_clean":"2024-03-04T06:24:46.422689+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:24:46.422689+0000","last_undegraded":"2024-03-04T06:24:46.422689+0000","last_fullsized":"2024-03-04T06:24:46.422689+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":983,"log_dups_size":0,"ondisk_log_size":983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521479,"num_objects":70,"num_object_clones":57,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":453,"num_read_kb":381495,"num_write":327,"num_write_kb":88741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2618,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.437195+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:24:49.437195+0000","last_peered":"2024-03-04T06:24:49.437195+0000","last_clean":"2024-03-04T06:24:49.437195+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:24:49.437195+0000","last_undegraded":"2024-03-04T06:24:49.437195+0000","last_fullsized":"2024-03-04T06:24:49.437195+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"1018'1165","reported_seq":3616,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.436382+0000","last_change":"2024-03-04T06:24:49.518737+0000","last_active":"2024-03-04T06:24:49.436382+0000","last_peered":"2024-03-04T06:24:49.436382+0000","last_clean":"2024-03-04T06:24:49.436382+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:24:49.436382+0000","last_undegraded":"2024-03-04T06:24:49.436382+0000","last_fullsized":"2024-03-04T06:24:49.436382+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1165,"log_dups_size":0,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924847,"num_objects":88,"num_object_clones":71,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":412,"num_read_kb":309847,"num_write":378,"num_write_kb":101479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2556,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.419171+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:24:45.419171+0000","last_peered":"2024-03-04T06:24:45.419171+0000","last_clean":"2024-03-04T06:24:45.419171+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:24:45.419171+0000","last_undegraded":"2024-03-04T06:24:45.419171+0000","last_fullsized":"2024-03-04T06:24:45.419171+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"1012'1521","reported_seq":3693,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.406621+0000","last_change":"2024-03-04T06:24:44.409370+0000","last_active":"2024-03-04T06:24:45.406621+0000","last_peered":"2024-03-04T06:24:45.406621+0000","last_clean":"2024-03-04T06:24:45.406621+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:24:45.406621+0000","last_undegraded":"2024-03-04T06:24:45.406621+0000","last_fullsized":"2024-03-04T06:24:45.406621+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1521,"log_dups_size":0,"ondisk_log_size":1521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6118606,"num_objects":105,"num_object_clones":89,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":519,"num_read_kb":404685,"num_write":483,"num_write_kb":134268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2618,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.424339+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:24:49.424339+0000","last_peered":"2024-03-04T06:24:49.424339+0000","last_clean":"2024-03-04T06:24:49.424339+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:24:49.424339+0000","last_undegraded":"2024-03-04T06:24:49.424339+0000","last_fullsized":"2024-03-04T06:24:49.424339+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"1018'1425","reported_seq":3810,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.418966+0000","last_change":"2024-03-04T06:24:49.514951+0000","last_active":"2024-03-04T06:24:49.418966+0000","last_peered":"2024-03-04T06:24:49.418966+0000","last_clean":"2024-03-04T06:24:49.418966+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:24:49.418966+0000","last_undegraded":"2024-03-04T06:24:49.418966+0000","last_fullsized":"2024-03-04T06:24:49.418966+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1425,"log_dups_size":0,"ondisk_log_size":1425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6253062,"num_objects":107,"num_object_clones":93,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":625,"num_read_kb":522641,"num_write":477,"num_write_kb":130314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2505,"reported_epoch":1015,"state":"active+clean","last_fresh":"2024-03-04T06:24:46.424693+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:24:46.424693+0000","last_peered":"2024-03-04T06:24:46.424693+0000","last_clean":"2024-03-04T06:24:46.424693+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:24:46.424693+0000","last_undegraded":"2024-03-04T06:24:46.424693+0000","last_fullsized":"2024-03-04T06:24:46.424693+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":90518862,"num_objects":1281,"num_object_clones":1064,"num_object_copies":3843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":186,"num_read":6824,"num_read_kb":5436289,"num_write":5778,"num_write_kb":1576259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17721,"ondisk_log_size":17721,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":233216,"kb_used_data":161976,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94138624,"statfs":{"total":96636764160,"available":96397950976,"internally_reserved":0,"allocated":165863424,"data_stored":186883647,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":183788,"kb_used_data":120868,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94188052,"statfs":{"total":96636764160,"available":96448565248,"internally_reserved":0,"allocated":123768832,"data_stored":138593424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":183064,"kb_used_data":120144,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94188776,"statfs":{"total":96636764160,"available":96449306624,"internally_reserved":0,"allocated":123027456,"data_stored":136879298,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":175996,"kb_used_data":114484,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94195844,"statfs":{"total":96636764160,"available":96456544256,"internally_reserved":0,"allocated":117231616,"data_stored":133020842,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:24:50.757 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:24:50.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:50.899+0000 7fd85cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:50.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:50.906+0000 7fd85cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:51.096 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:24:51.097 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:24:51.108 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1523,"stamp":"2024-03-04T06:24:50.923229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87411294,"num_objects":1284,"num_object_clones":1065,"num_object_copies":3852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":187,"num_read":6934,"num_read_kb":5443491,"num_write":5916,"num_write_kb":1578153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17830,"ondisk_log_size":17830,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":777908,"kb_used_data":519316,"kb_used_omap":0,"kb_used_meta":258560,"kb_avail":376709452,"statfs":{"total":386547056640,"available":385750478848,"internally_reserved":0,"allocated":531779584,"data_stored":597248136,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":264765440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2622288,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":8,"num_read_kb":7112,"num_write":7,"num_write_kb":1772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000778"},"pg_stats":[{"pgid":"3.f","version":"1015'980","reported_seq":3445,"reported_epoch":1018,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:49.927169+0000","last_change":"2024-03-04T06:24:49.927169+0000","last_active":"2024-03-04T06:24:49.927169+0000","last_peered":"2024-03-04T06:24:49.927169+0000","last_clean":"2024-03-04T06:24:49.927169+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:24:49.927169+0000","last_undegraded":"2024-03-04T06:24:49.927169+0000","last_fullsized":"2024-03-04T06:24:49.927169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":980,"log_dups_size":0,"ondisk_log_size":980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":374,"num_read_kb":295545,"num_write":337,"num_write_kb":88619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"1018'825","reported_seq":3455,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.428222+0000","last_change":"2024-03-04T06:24:49.489095+0000","last_active":"2024-03-04T06:24:49.428222+0000","last_peered":"2024-03-04T06:24:49.428222+0000","last_clean":"2024-03-04T06:24:49.428222+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:24:49.428222+0000","last_undegraded":"2024-03-04T06:24:49.428222+0000","last_fullsized":"2024-03-04T06:24:49.428222+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6521974,"num_objects":65,"num_object_clones":47,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":261,"num_read_kb":210901,"num_write":267,"num_write_kb":70489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"1007'1356","reported_seq":3661,"reported_epoch":1015,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:46.410727+0000","last_change":"2024-03-04T06:24:46.410727+0000","last_active":"2024-03-04T06:24:46.410727+0000","last_peered":"2024-03-04T06:24:46.410727+0000","last_clean":"2024-03-04T06:24:46.410727+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:24:46.410727+0000","last_undegraded":"2024-03-04T06:24:46.410727+0000","last_fullsized":"2024-03-04T06:24:46.410727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1356,"log_dups_size":0,"ondisk_log_size":1356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10169918,"num_objects":100,"num_object_clones":85,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":605,"num_read_kb":483934,"num_write":460,"num_write_kb":125339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"1015'1122","reported_seq":3430,"reported_epoch":1018,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:49.917252+0000","last_change":"2024-03-04T06:24:49.917252+0000","last_active":"2024-03-04T06:24:49.917252+0000","last_peered":"2024-03-04T06:24:49.917252+0000","last_clean":"2024-03-04T06:24:49.917252+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:24:49.917252+0000","last_undegraded":"2024-03-04T06:24:49.917252+0000","last_fullsized":"2024-03-04T06:24:49.917252+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1122,"log_dups_size":0,"ondisk_log_size":1122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5194221,"num_objects":71,"num_object_clones":61,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":341,"num_read_kb":257705,"num_write":335,"num_write_kb":92846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"999'435","reported_seq":3200,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.422443+0000","last_change":"2024-03-04T06:24:44.394523+0000","last_active":"2024-03-04T06:24:45.422443+0000","last_peered":"2024-03-04T06:24:45.422443+0000","last_clean":"2024-03-04T06:24:45.422443+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:24:45.422443+0000","last_undegraded":"2024-03-04T06:24:45.422443+0000","last_fullsized":"2024-03-04T06:24:45.422443+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"1015'1078","reported_seq":3511,"reported_epoch":1018,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:49.929420+0000","last_change":"2024-03-04T06:24:49.929420+0000","last_active":"2024-03-04T06:24:49.929420+0000","last_peered":"2024-03-04T06:24:49.929420+0000","last_clean":"2024-03-04T06:24:49.929420+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:24:49.929420+0000","last_undegraded":"2024-03-04T06:24:49.929420+0000","last_fullsized":"2024-03-04T06:24:49.929420+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092917,"num_objects":78,"num_object_clones":66,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":451,"num_read_kb":362212,"num_write":370,"num_write_kb":106221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2977,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.932351+0000","last_change":"2024-03-04T06:24:49.938903+0000","last_active":"2024-03-04T06:24:49.932351+0000","last_peered":"2024-03-04T06:24:49.932351+0000","last_clean":"2024-03-04T06:24:49.932351+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:24:49.932351+0000","last_undegraded":"2024-03-04T06:24:49.932351+0000","last_fullsized":"2024-03-04T06:24:49.932351+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"1012'709","reported_seq":3294,"reported_epoch":1015,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:46.412129+0000","last_change":"2024-03-04T06:24:46.412129+0000","last_active":"2024-03-04T06:24:46.412129+0000","last_peered":"2024-03-04T06:24:46.412129+0000","last_clean":"2024-03-04T06:24:46.412129+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:24:46.412129+0000","last_undegraded":"2024-03-04T06:24:46.412129+0000","last_fullsized":"2024-03-04T06:24:46.412129+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227373,"num_objects":46,"num_object_clones":37,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":290,"num_read_kb":255610,"num_write":203,"num_write_kb":55194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2689,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.404692+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:24:45.404692+0000","last_peered":"2024-03-04T06:24:45.404692+0000","last_clean":"2024-03-04T06:24:45.404692+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:24:45.404692+0000","last_undegraded":"2024-03-04T06:24:45.404692+0000","last_fullsized":"2024-03-04T06:24:45.404692+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2505,"reported_epoch":1015,"state":"active+clean","last_fresh":"2024-03-04T06:24:46.419295+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:24:46.419295+0000","last_peered":"2024-03-04T06:24:46.419295+0000","last_clean":"2024-03-04T06:24:46.419295+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:24:46.419295+0000","last_undegraded":"2024-03-04T06:24:46.419295+0000","last_fullsized":"2024-03-04T06:24:46.419295+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"1012'1028","reported_seq":3494,"reported_epoch":1015,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:46.426273+0000","last_change":"2024-03-04T06:24:46.426273+0000","last_active":"2024-03-04T06:24:46.426273+0000","last_peered":"2024-03-04T06:24:46.426273+0000","last_clean":"2024-03-04T06:24:46.426273+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:24:46.426273+0000","last_undegraded":"2024-03-04T06:24:46.426273+0000","last_fullsized":"2024-03-04T06:24:46.426273+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1028,"log_dups_size":0,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8253524,"num_objects":75,"num_object_clones":61,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":455,"num_read_kb":370096,"num_write":340,"num_write_kb":91466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2498,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.929476+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:24:49.929476+0000","last_peered":"2024-03-04T06:24:49.929476+0000","last_clean":"2024-03-04T06:24:49.929476+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:24:49.929476+0000","last_undegraded":"2024-03-04T06:24:49.929476+0000","last_fullsized":"2024-03-04T06:24:49.929476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"1007'728","reported_seq":3393,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.403979+0000","last_change":"2024-03-04T06:24:44.387071+0000","last_active":"2024-03-04T06:24:45.403979+0000","last_peered":"2024-03-04T06:24:45.403979+0000","last_clean":"2024-03-04T06:24:45.403979+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:24:45.403979+0000","last_undegraded":"2024-03-04T06:24:45.403979+0000","last_fullsized":"2024-03-04T06:24:45.403979+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":728,"log_dups_size":0,"ondisk_log_size":728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135413,"num_objects":54,"num_object_clones":45,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":333,"num_read_kb":275602,"num_write":233,"num_write_kb":66901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2505,"reported_epoch":1015,"state":"active+clean","last_fresh":"2024-03-04T06:24:46.413969+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:24:46.413969+0000","last_peered":"2024-03-04T06:24:46.413969+0000","last_clean":"2024-03-04T06:24:46.413969+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:24:46.413969+0000","last_undegraded":"2024-03-04T06:24:46.413969+0000","last_fullsized":"2024-03-04T06:24:46.413969+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"1011'2005","reported_seq":4032,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.421106+0000","last_change":"2024-03-04T06:24:44.396153+0000","last_active":"2024-03-04T06:24:45.421106+0000","last_peered":"2024-03-04T06:24:45.421106+0000","last_clean":"2024-03-04T06:24:45.421106+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:24:45.421106+0000","last_undegraded":"2024-03-04T06:24:45.421106+0000","last_fullsized":"2024-03-04T06:24:45.421106+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2005,"log_dups_size":0,"ondisk_log_size":2005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14388871,"num_objects":149,"num_object_clones":122,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":835,"num_read_kb":664491,"num_write":693,"num_write_kb":186619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2498,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.913084+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:24:49.913084+0000","last_peered":"2024-03-04T06:24:49.913084+0000","last_clean":"2024-03-04T06:24:49.913084+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:24:49.913084+0000","last_undegraded":"2024-03-04T06:24:49.913084+0000","last_fullsized":"2024-03-04T06:24:49.913084+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"1018'2373","reported_seq":4095,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.423454+0000","last_change":"2024-03-04T06:24:49.520087+0000","last_active":"2024-03-04T06:24:49.423454+0000","last_peered":"2024-03-04T06:24:49.423454+0000","last_clean":"2024-03-04T06:24:49.423454+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:24:49.423454+0000","last_undegraded":"2024-03-04T06:24:49.423454+0000","last_fullsized":"2024-03-04T06:24:49.423454+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2373,"log_dups_size":0,"ondisk_log_size":2373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12018721,"num_objects":164,"num_object_clones":140,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":754,"num_read_kb":555488,"num_write":750,"num_write_kb":203150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"1006'983","reported_seq":3483,"reported_epoch":1015,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:46.422689+0000","last_change":"2024-03-04T06:24:46.422689+0000","last_active":"2024-03-04T06:24:46.422689+0000","last_peered":"2024-03-04T06:24:46.422689+0000","last_clean":"2024-03-04T06:24:46.422689+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:24:46.422689+0000","last_undegraded":"2024-03-04T06:24:46.422689+0000","last_fullsized":"2024-03-04T06:24:46.422689+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":983,"log_dups_size":0,"ondisk_log_size":983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521479,"num_objects":70,"num_object_clones":57,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":453,"num_read_kb":381495,"num_write":327,"num_write_kb":88741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2618,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.437195+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:24:49.437195+0000","last_peered":"2024-03-04T06:24:49.437195+0000","last_clean":"2024-03-04T06:24:49.437195+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:24:49.437195+0000","last_undegraded":"2024-03-04T06:24:49.437195+0000","last_fullsized":"2024-03-04T06:24:49.437195+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"1018'1165","reported_seq":3616,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.436382+0000","last_change":"2024-03-04T06:24:49.518737+0000","last_active":"2024-03-04T06:24:49.436382+0000","last_peered":"2024-03-04T06:24:49.436382+0000","last_clean":"2024-03-04T06:24:49.436382+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:24:49.436382+0000","last_undegraded":"2024-03-04T06:24:49.436382+0000","last_fullsized":"2024-03-04T06:24:49.436382+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1165,"log_dups_size":0,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924847,"num_objects":88,"num_object_clones":71,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":412,"num_read_kb":309847,"num_write":378,"num_write_kb":101479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2556,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.419171+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:24:45.419171+0000","last_peered":"2024-03-04T06:24:45.419171+0000","last_clean":"2024-03-04T06:24:45.419171+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:24:45.419171+0000","last_undegraded":"2024-03-04T06:24:45.419171+0000","last_fullsized":"2024-03-04T06:24:45.419171+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"1012'1521","reported_seq":3693,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-03-04T06:24:45.406621+0000","last_change":"2024-03-04T06:24:44.409370+0000","last_active":"2024-03-04T06:24:45.406621+0000","last_peered":"2024-03-04T06:24:45.406621+0000","last_clean":"2024-03-04T06:24:45.406621+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:24:45.406621+0000","last_undegraded":"2024-03-04T06:24:45.406621+0000","last_fullsized":"2024-03-04T06:24:45.406621+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1521,"log_dups_size":0,"ondisk_log_size":1521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6118606,"num_objects":105,"num_object_clones":89,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":519,"num_read_kb":404685,"num_write":483,"num_write_kb":134268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2618,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.424339+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:24:49.424339+0000","last_peered":"2024-03-04T06:24:49.424339+0000","last_clean":"2024-03-04T06:24:49.424339+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:24:49.424339+0000","last_undegraded":"2024-03-04T06:24:49.424339+0000","last_fullsized":"2024-03-04T06:24:49.424339+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"1018'1425","reported_seq":3810,"reported_epoch":1018,"state":"active+clean","last_fresh":"2024-03-04T06:24:49.418966+0000","last_change":"2024-03-04T06:24:49.514951+0000","last_active":"2024-03-04T06:24:49.418966+0000","last_peered":"2024-03-04T06:24:49.418966+0000","last_clean":"2024-03-04T06:24:49.418966+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:24:49.418966+0000","last_undegraded":"2024-03-04T06:24:49.418966+0000","last_fullsized":"2024-03-04T06:24:49.418966+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1425,"log_dups_size":0,"ondisk_log_size":1425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6253062,"num_objects":107,"num_object_clones":93,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":625,"num_read_kb":522641,"num_write":477,"num_write_kb":130314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2505,"reported_epoch":1015,"state":"active+clean","last_fresh":"2024-03-04T06:24:46.424693+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:24:46.424693+0000","last_peered":"2024-03-04T06:24:46.424693+0000","last_clean":"2024-03-04T06:24:46.424693+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:24:46.424693+0000","last_undegraded":"2024-03-04T06:24:46.424693+0000","last_fullsized":"2024-03-04T06:24:46.424693+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":86820926,"num_objects":1282,"num_object_clones":1065,"num_object_copies":3846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":187,"num_read":6831,"num_read_kb":5443400,"num_write":5779,"num_write_kb":1576259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17733,"ondisk_log_size":17733,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":233216,"kb_used_data":161976,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94138624,"statfs":{"total":96636764160,"available":96397950976,"internally_reserved":0,"allocated":165863424,"data_stored":186883647,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":183788,"kb_used_data":120868,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94188052,"statfs":{"total":96636764160,"available":96448565248,"internally_reserved":0,"allocated":123768832,"data_stored":138593424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":183064,"kb_used_data":120144,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94188776,"statfs":{"total":96636764160,"available":96449306624,"internally_reserved":0,"allocated":123027456,"data_stored":136879298,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":177840,"kb_used_data":116328,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94194000,"statfs":{"total":96636764160,"available":96454656000,"internally_reserved":0,"allocated":119119872,"data_stored":134891767,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:24:51.110 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-04T06:24:51.110 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:24:51.110 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:24:51.110 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:24:51.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:51.214+0000 7fcd9ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:51.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:51.221+0000 7fcd9ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:51.238 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:51.238 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:51.238 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:51.238 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:51.238 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:51.238 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:51.239 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:51.239 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:51.239 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:51.239 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:51.239 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:51.239 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:51.241 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:51.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:24:51.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:51.345+0000 7f4e6e55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:51.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:51.352+0000 7f4e6e55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:51.365 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:51.365 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:51.365 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:51.365 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:51.365 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:51.365 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:51.365 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:51.365 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:51.365 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:51.366 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:51.366 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:51.366 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:51.368 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:51.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:24:51.417 INFO:tasks.rados.rados.0.smithi064.stdout:3895: done (1 left) 2024-03-04T06:24:51.417 INFO:tasks.rados.rados.0.smithi064.stdout:3896: done (0 left) 2024-03-04T06:24:51.417 INFO:tasks.rados.rados.0.smithi064.stdout:3897: write (excl) oid 6 current snap is 512 2024-03-04T06:24:51.417 INFO:tasks.rados.rados.0.smithi064.stdout:3897: seq_num 1752 ranges {645864=597438,1706217=720275,3029201=416663} 2024-03-04T06:24:51.424 INFO:tasks.rados.rados.0.smithi064.stdout:3897: writing smithi06431743-6 from 645864 to 1243302 tid 1 2024-03-04T06:24:51.433 INFO:tasks.rados.rados.0.smithi064.stdout:3897: writing smithi06431743-6 from 1706217 to 2426492 tid 2 2024-03-04T06:24:51.441 INFO:tasks.rados.rados.0.smithi064.stdout:3897: writing smithi06431743-6 from 3029201 to 3445864 tid 3 2024-03-04T06:24:51.441 INFO:tasks.rados.rados.0.smithi064.stdout:3898: snap_remove snap 502 2024-03-04T06:24:51.487 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:51.482+0000 7fd6c7f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:51.490 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:51.489+0000 7fd6c7f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:51.505 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:51.505 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:51.505 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:51.505 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:51.505 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:51.505 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:51.505 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:51.506 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:51.506 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:51.506 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:51.506 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:51.506 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:51.508 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:51.508 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:51.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:24:51.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:51.611+0000 7ff914f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:51.619 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:51.618+0000 7ff914f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:51.632 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:24:51.632 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:24:51.632 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:24:51.632 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:24:51.632 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:24:51.632 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:24:51.632 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:24:51.632 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:24:51.632 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:24:51.633 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:24:51.633 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:24:51.633 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:24:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:24:51.635 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:24:52.421 INFO:tasks.rados.rados.0.smithi064.stdout:3898: done (1 left) 2024-03-04T06:24:52.421 INFO:tasks.rados.rados.0.smithi064.stdout:3899: snap_create 2024-03-04T06:24:52.421 INFO:tasks.rados.rados.0.smithi064.stdout:3897: finishing write tid 1 to smithi06431743-6 2024-03-04T06:24:52.421 INFO:tasks.rados.rados.0.smithi064.stdout:3897: finishing write tid 2 to smithi06431743-6 2024-03-04T06:24:52.422 INFO:tasks.rados.rados.0.smithi064.stdout:3897: finishing write tid 3 to smithi06431743-6 2024-03-04T06:24:52.422 INFO:tasks.rados.rados.0.smithi064.stdout:3897: finishing write tid 4 to smithi06431743-6 2024-03-04T06:24:52.422 INFO:tasks.rados.rados.0.smithi064.stdout:3897: finishing write tid 5 to smithi06431743-6 2024-03-04T06:24:52.422 INFO:tasks.rados.rados.0.smithi064.stdout:3897: oid 6 updating version 0 to 992 2024-03-04T06:24:52.422 INFO:tasks.rados.rados.0.smithi064.stdout:3897: oid 6 version 992 is already newer than 991 2024-03-04T06:24:52.422 INFO:tasks.rados.rados.0.smithi064.stdout:3897: oid 6 updating version 992 to 993 2024-03-04T06:24:52.423 INFO:tasks.rados.rados.0.smithi064.stdout:3897: oid 6 version 993 is already newer than 990 2024-03-04T06:24:52.423 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 993 (ObjNum 1752 snap 512 seq_num 1752) dirty exists 2024-03-04T06:24:52.423 INFO:tasks.rados.rados.0.smithi064.stdout:3897: left oid 6 (ObjNum 1752 snap 512 seq_num 1752) 2024-03-04T06:24:52.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:52.827 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:52.825+0000 7f5ddf02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:52.829 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:52.829+0000 7f5ddf02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:53.355 INFO:tasks.rados.rados.0.smithi064.stdout:3897: done (1 left) 2024-03-04T06:24:53.355 INFO:tasks.rados.rados.0.smithi064.stdout:3899: done (0 left) 2024-03-04T06:24:53.355 INFO:tasks.rados.rados.0.smithi064.stdout:3900: read oid 35 snap -1 2024-03-04T06:24:53.355 INFO:tasks.rados.rados.0.smithi064.stdout:3900: expect (ObjNum 1722 snap 500 seq_num 1722) 2024-03-04T06:24:53.355 INFO:tasks.rados.rados.0.smithi064.stdout:3901: snap_create 2024-03-04T06:24:53.405 INFO:tasks.rados.rados.0.smithi064.stdout:3900: expect (ObjNum 1722 snap 500 seq_num 1722) 2024-03-04T06:24:54.355 INFO:tasks.rados.rados.0.smithi064.stdout:3900: done (1 left) 2024-03-04T06:24:54.355 INFO:tasks.rados.rados.0.smithi064.stdout:3901: done (0 left) 2024-03-04T06:24:54.355 INFO:tasks.rados.rados.0.smithi064.stdout:3902: delete oid 20 current snap is 514 2024-03-04T06:24:54.397 INFO:tasks.rados.rados.0.smithi064.stdout:3902: done (0 left) 2024-03-04T06:24:54.397 INFO:tasks.rados.rados.0.smithi064.stdout:3903: delete oid 42 current snap is 514 2024-03-04T06:24:54.399 INFO:tasks.rados.rados.0.smithi064.stdout:3903: done (0 left) 2024-03-04T06:24:54.399 INFO:tasks.rados.rados.0.smithi064.stdout:3904: snap_remove snap 503 2024-03-04T06:24:54.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:55.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:55.010+0000 7f66dfc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:55.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:55.014+0000 7f66dfc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:55.359 INFO:tasks.rados.rados.0.smithi064.stdout:3904: done (0 left) 2024-03-04T06:24:55.359 INFO:tasks.rados.rados.0.smithi064.stdout:3905: read oid 1 snap -1 2024-03-04T06:24:55.359 INFO:tasks.rados.rados.0.smithi064.stdout:3905: expect (ObjNum 1661 snap 477 seq_num 1661) 2024-03-04T06:24:55.359 INFO:tasks.rados.rados.0.smithi064.stdout:3906: snap_create 2024-03-04T06:24:55.419 INFO:tasks.rados.rados.0.smithi064.stdout:3905: expect (ObjNum 1661 snap 477 seq_num 1661) 2024-03-04T06:24:56.362 INFO:tasks.rados.rados.0.smithi064.stdout:3905: done (1 left) 2024-03-04T06:24:56.362 INFO:tasks.rados.rados.0.smithi064.stdout:3906: done (0 left) 2024-03-04T06:24:56.363 INFO:tasks.rados.rados.0.smithi064.stdout:3907: write (excl) oid 50 current snap is 515 2024-03-04T06:24:56.363 INFO:tasks.rados.rados.0.smithi064.stdout:3907: seq_num 1755 ranges {696285=423185,1881663=467602,3111470=384815} 2024-03-04T06:24:56.369 INFO:tasks.rados.rados.0.smithi064.stdout:3907: writing smithi06431743-50 from 696285 to 1119470 tid 1 2024-03-04T06:24:56.378 INFO:tasks.rados.rados.0.smithi064.stdout:3907: writing smithi06431743-50 from 1881663 to 2349265 tid 2 2024-03-04T06:24:56.386 INFO:tasks.rados.rados.0.smithi064.stdout:3907: writing smithi06431743-50 from 3111470 to 3496285 tid 3 2024-03-04T06:24:56.386 INFO:tasks.rados.rados.0.smithi064.stdout:3908: delete oid 9 current snap is 515 2024-03-04T06:24:56.433 INFO:tasks.rados.rados.0.smithi064.stdout:3907: finishing write tid 1 to smithi06431743-50 2024-03-04T06:24:56.458 INFO:tasks.rados.rados.0.smithi064.stdout:3907: finishing write tid 2 to smithi06431743-50 2024-03-04T06:24:56.459 INFO:tasks.rados.rados.0.smithi064.stdout:3907: finishing write tid 3 to smithi06431743-50 2024-03-04T06:24:56.470 INFO:tasks.rados.rados.0.smithi064.stdout:3907: finishing write tid 4 to smithi06431743-50 2024-03-04T06:24:56.471 INFO:tasks.rados.rados.0.smithi064.stdout:3907: finishing write tid 5 to smithi06431743-50 2024-03-04T06:24:56.471 INFO:tasks.rados.rados.0.smithi064.stdout:3907: oid 50 updating version 0 to 1432 2024-03-04T06:24:56.471 INFO:tasks.rados.rados.0.smithi064.stdout:3907: oid 50 updating version 1432 to 1433 2024-03-04T06:24:56.471 INFO:tasks.rados.rados.0.smithi064.stdout:3907: oid 50 version 1433 is already newer than 1430 2024-03-04T06:24:56.471 INFO:tasks.rados.rados.0.smithi064.stdout:3907: oid 50 version 1433 is already newer than 1432 2024-03-04T06:24:56.471 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 1433 (ObjNum 1755 snap 515 seq_num 1755) dirty exists 2024-03-04T06:24:56.471 INFO:tasks.rados.rados.0.smithi064.stdout:3907: left oid 50 (ObjNum 1755 snap 515 seq_num 1755) 2024-03-04T06:24:56.472 INFO:tasks.rados.rados.0.smithi064.stdout:3907: done (1 left) 2024-03-04T06:24:56.472 INFO:tasks.rados.rados.0.smithi064.stdout:3908: done (0 left) 2024-03-04T06:24:56.472 INFO:tasks.rados.rados.0.smithi064.stdout:3909: read oid 24 snap -1 2024-03-04T06:24:56.472 INFO:tasks.rados.rados.0.smithi064.stdout:3909: expect (ObjNum 1749 snap 509 seq_num 1749) 2024-03-04T06:24:56.472 INFO:tasks.rados.rados.0.smithi064.stdout:3910: snap_remove snap 494 2024-03-04T06:24:56.636 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:24:56.636 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:24:56.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:24:56.745 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:56.741+0000 7f898031d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:56.748 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:56.748+0000 7f898031d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:56.935 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:24:56.936 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:24:56.947 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1532,"stamp":"2024-03-04T06:24:56.924613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91731122,"num_objects":1285,"num_object_clones":1066,"num_object_copies":3855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":186,"num_read":6964,"num_read_kb":5474262,"num_write":5927,"num_write_kb":1581487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17929,"ondisk_log_size":17929,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":785168,"kb_used_data":526576,"kb_used_omap":0,"kb_used_meta":258560,"kb_avail":376702192,"statfs":{"total":386547056640,"available":385743044608,"internally_reserved":0,"allocated":539213824,"data_stored":604489614,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":264765440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3445864,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":-1,"num_read":15,"num_read_kb":17253,"num_write":5,"num_write_kb":1695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001006"},"pg_stats":[{"pgid":"3.f","version":"1018'982","reported_seq":3462,"reported_epoch":1023,"state":"active+clean","last_fresh":"2024-03-04T06:24:54.374177+0000","last_change":"2024-03-04T06:24:53.391831+0000","last_active":"2024-03-04T06:24:54.374177+0000","last_peered":"2024-03-04T06:24:54.374177+0000","last_clean":"2024-03-04T06:24:54.374177+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:24:54.374177+0000","last_undegraded":"2024-03-04T06:24:54.374177+0000","last_fullsized":"2024-03-04T06:24:54.374177+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":982,"log_dups_size":0,"ondisk_log_size":982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":375,"num_read_kb":295545,"num_write":337,"num_write_kb":88619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"1018'825","reported_seq":3469,"reported_epoch":1024,"state":"active+clean","last_fresh":"2024-03-04T06:24:55.369988+0000","last_change":"2024-03-04T06:24:55.382606+0000","last_active":"2024-03-04T06:24:55.369988+0000","last_peered":"2024-03-04T06:24:55.369988+0000","last_clean":"2024-03-04T06:24:55.369988+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:24:55.369988+0000","last_undegraded":"2024-03-04T06:24:55.369988+0000","last_fullsized":"2024-03-04T06:24:55.369988+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6521974,"num_objects":65,"num_object_clones":47,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":261,"num_read_kb":210901,"num_write":267,"num_write_kb":70489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"1015'1362","reported_seq":3690,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.409753+0000","last_change":"2024-03-04T06:24:56.416210+0000","last_active":"2024-03-04T06:24:56.409753+0000","last_peered":"2024-03-04T06:24:56.409753+0000","last_clean":"2024-03-04T06:24:56.409753+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:24:56.409753+0000","last_undegraded":"2024-03-04T06:24:56.409753+0000","last_fullsized":"2024-03-04T06:24:56.409753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1362,"log_dups_size":0,"ondisk_log_size":1362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10169918,"num_objects":100,"num_object_clones":85,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":605,"num_read_kb":483934,"num_write":460,"num_write_kb":125339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"1018'1124","reported_seq":3446,"reported_epoch":1023,"state":"active+clean","last_fresh":"2024-03-04T06:24:54.365522+0000","last_change":"2024-03-04T06:24:53.378528+0000","last_active":"2024-03-04T06:24:54.365522+0000","last_peered":"2024-03-04T06:24:54.365522+0000","last_clean":"2024-03-04T06:24:54.365522+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:24:54.365522+0000","last_undegraded":"2024-03-04T06:24:54.365522+0000","last_fullsized":"2024-03-04T06:24:54.365522+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1124,"log_dups_size":0,"ondisk_log_size":1124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5194221,"num_objects":71,"num_object_clones":61,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":341,"num_read_kb":257705,"num_write":335,"num_write_kb":92846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"1015'437","reported_seq":3234,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.402767+0000","last_change":"2024-03-04T06:24:55.396963+0000","last_active":"2024-03-04T06:24:56.402767+0000","last_peered":"2024-03-04T06:24:56.402767+0000","last_clean":"2024-03-04T06:24:56.402767+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:24:56.402767+0000","last_undegraded":"2024-03-04T06:24:56.402767+0000","last_fullsized":"2024-03-04T06:24:56.402767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"1022'1086","reported_seq":3527,"reported_epoch":1023,"state":"active+clean","last_fresh":"2024-03-04T06:24:54.371889+0000","last_change":"2024-03-04T06:24:53.408708+0000","last_active":"2024-03-04T06:24:54.371889+0000","last_peered":"2024-03-04T06:24:54.371889+0000","last_clean":"2024-03-04T06:24:54.371889+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:24:54.371889+0000","last_undegraded":"2024-03-04T06:24:54.371889+0000","last_fullsized":"2024-03-04T06:24:54.371889+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1086,"log_dups_size":0,"ondisk_log_size":1086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092917,"num_objects":78,"num_object_clones":66,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":451,"num_read_kb":362212,"num_write":370,"num_write_kb":106221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":2991,"reported_epoch":1023,"state":"active+clean","last_fresh":"2024-03-04T06:24:54.373837+0000","last_change":"2024-03-04T06:24:53.391183+0000","last_active":"2024-03-04T06:24:54.373837+0000","last_peered":"2024-03-04T06:24:54.373837+0000","last_clean":"2024-03-04T06:24:54.373837+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:24:54.373837+0000","last_undegraded":"2024-03-04T06:24:54.373837+0000","last_fullsized":"2024-03-04T06:24:54.373837+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"1025'719","reported_seq":3323,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.404978+0000","last_change":"2024-03-04T06:24:56.432365+0000","last_active":"2024-03-04T06:24:56.404978+0000","last_peered":"2024-03-04T06:24:56.404978+0000","last_clean":"2024-03-04T06:24:56.404978+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:24:56.404978+0000","last_undegraded":"2024-03-04T06:24:56.404978+0000","last_fullsized":"2024-03-04T06:24:56.404978+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":719,"log_dups_size":0,"ondisk_log_size":719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227373,"num_objects":46,"num_object_clones":37,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":290,"num_read_kb":255610,"num_write":203,"num_write_kb":55194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2718,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.399195+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:24:56.399195+0000","last_peered":"2024-03-04T06:24:56.399195+0000","last_clean":"2024-03-04T06:24:56.399195+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:24:56.399195+0000","last_undegraded":"2024-03-04T06:24:56.399195+0000","last_fullsized":"2024-03-04T06:24:56.399195+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2529,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.425766+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:24:56.425766+0000","last_peered":"2024-03-04T06:24:56.425766+0000","last_clean":"2024-03-04T06:24:56.425766+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:24:56.425766+0000","last_undegraded":"2024-03-04T06:24:56.425766+0000","last_fullsized":"2024-03-04T06:24:56.425766+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"1025'1040","reported_seq":3523,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.419945+0000","last_change":"2024-03-04T06:24:56.441759+0000","last_active":"2024-03-04T06:24:56.419945+0000","last_peered":"2024-03-04T06:24:56.419945+0000","last_clean":"2024-03-04T06:24:56.419945+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:24:56.419945+0000","last_undegraded":"2024-03-04T06:24:56.419945+0000","last_fullsized":"2024-03-04T06:24:56.419945+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8253524,"num_objects":75,"num_object_clones":61,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":455,"num_read_kb":370096,"num_write":340,"num_write_kb":91466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2511,"reported_epoch":1023,"state":"active+clean","last_fresh":"2024-03-04T06:24:54.374378+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:24:54.374378+0000","last_peered":"2024-03-04T06:24:54.374378+0000","last_clean":"2024-03-04T06:24:54.374378+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:24:54.374378+0000","last_undegraded":"2024-03-04T06:24:54.374378+0000","last_fullsized":"2024-03-04T06:24:54.374378+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"1018'736","reported_seq":3427,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.398318+0000","last_change":"2024-03-04T06:24:55.386409+0000","last_active":"2024-03-04T06:24:56.398318+0000","last_peered":"2024-03-04T06:24:56.398318+0000","last_clean":"2024-03-04T06:24:56.398318+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:24:56.398318+0000","last_undegraded":"2024-03-04T06:24:56.398318+0000","last_fullsized":"2024-03-04T06:24:56.398318+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135413,"num_objects":54,"num_object_clones":45,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":333,"num_read_kb":275602,"num_write":233,"num_write_kb":66901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2529,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.408796+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:24:56.408796+0000","last_peered":"2024-03-04T06:24:56.408796+0000","last_clean":"2024-03-04T06:24:56.408796+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:24:56.408796+0000","last_undegraded":"2024-03-04T06:24:56.408796+0000","last_fullsized":"2024-03-04T06:24:56.408796+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"1018'2015","reported_seq":4073,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.399978+0000","last_change":"2024-03-04T06:24:55.394686+0000","last_active":"2024-03-04T06:24:56.399978+0000","last_peered":"2024-03-04T06:24:56.399978+0000","last_clean":"2024-03-04T06:24:56.399978+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:24:56.399978+0000","last_undegraded":"2024-03-04T06:24:56.399978+0000","last_fullsized":"2024-03-04T06:24:56.399978+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2015,"log_dups_size":0,"ondisk_log_size":2015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14388871,"num_objects":149,"num_object_clones":122,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":848,"num_read_kb":678782,"num_write":693,"num_write_kb":186619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2511,"reported_epoch":1023,"state":"active+clean","last_fresh":"2024-03-04T06:24:54.364310+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:24:54.364310+0000","last_peered":"2024-03-04T06:24:54.364310+0000","last_clean":"2024-03-04T06:24:54.364310+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:24:54.364310+0000","last_undegraded":"2024-03-04T06:24:54.364310+0000","last_fullsized":"2024-03-04T06:24:54.364310+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"1023'2377","reported_seq":4108,"reported_epoch":1024,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:55.371723+0000","last_change":"2024-03-04T06:24:55.371723+0000","last_active":"2024-03-04T06:24:55.371723+0000","last_peered":"2024-03-04T06:24:55.371723+0000","last_clean":"2024-03-04T06:24:55.371723+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:24:55.371723+0000","last_undegraded":"2024-03-04T06:24:55.371723+0000","last_fullsized":"2024-03-04T06:24:55.371723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2377,"log_dups_size":0,"ondisk_log_size":2377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12018721,"num_objects":164,"num_object_clones":140,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":754,"num_read_kb":555488,"num_write":750,"num_write_kb":203150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"1021'995","reported_seq":3517,"reported_epoch":1025,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:56.426404+0000","last_change":"2024-03-04T06:24:56.426404+0000","last_active":"2024-03-04T06:24:56.426404+0000","last_peered":"2024-03-04T06:24:56.426404+0000","last_clean":"2024-03-04T06:24:56.426404+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:24:56.426404+0000","last_undegraded":"2024-03-04T06:24:56.426404+0000","last_fullsized":"2024-03-04T06:24:56.426404+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445864,"num_objects":71,"num_object_clones":58,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":456,"num_read_kb":381496,"num_write":333,"num_write_kb":90436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2630,"reported_epoch":1024,"state":"active+clean","last_fresh":"2024-03-04T06:24:55.387573+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:24:55.387573+0000","last_peered":"2024-03-04T06:24:55.387573+0000","last_clean":"2024-03-04T06:24:55.387573+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:24:55.387573+0000","last_undegraded":"2024-03-04T06:24:55.387573+0000","last_fullsized":"2024-03-04T06:24:55.387573+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"1023'1169","reported_seq":3629,"reported_epoch":1024,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:55.384561+0000","last_change":"2024-03-04T06:24:55.384561+0000","last_active":"2024-03-04T06:24:55.384561+0000","last_peered":"2024-03-04T06:24:55.384561+0000","last_clean":"2024-03-04T06:24:55.384561+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:24:55.384561+0000","last_undegraded":"2024-03-04T06:24:55.384561+0000","last_fullsized":"2024-03-04T06:24:55.384561+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1169,"log_dups_size":0,"ondisk_log_size":1169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924847,"num_objects":88,"num_object_clones":71,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":412,"num_read_kb":309847,"num_write":378,"num_write_kb":101479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2585,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.401576+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:24:56.401576+0000","last_peered":"2024-03-04T06:24:56.401576+0000","last_clean":"2024-03-04T06:24:56.401576+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:24:56.401576+0000","last_undegraded":"2024-03-04T06:24:56.401576+0000","last_fullsized":"2024-03-04T06:24:56.401576+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"1024'1538","reported_seq":3738,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.396147+0000","last_change":"2024-03-04T06:24:55.401455+0000","last_active":"2024-03-04T06:24:56.396147+0000","last_peered":"2024-03-04T06:24:56.396147+0000","last_clean":"2024-03-04T06:24:56.396147+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:24:56.396147+0000","last_undegraded":"2024-03-04T06:24:56.396147+0000","last_fullsized":"2024-03-04T06:24:56.396147+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1538,"log_dups_size":0,"ondisk_log_size":1538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9514049,"num_objects":105,"num_object_clones":89,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":532,"num_read_kb":421164,"num_write":488,"num_write_kb":135907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2630,"reported_epoch":1024,"state":"active+clean","last_fresh":"2024-03-04T06:24:55.374609+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:24:55.374609+0000","last_peered":"2024-03-04T06:24:55.374609+0000","last_clean":"2024-03-04T06:24:55.374609+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:24:55.374609+0000","last_undegraded":"2024-03-04T06:24:55.374609+0000","last_fullsized":"2024-03-04T06:24:55.374609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"1023'1427","reported_seq":3823,"reported_epoch":1024,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:55.370741+0000","last_change":"2024-03-04T06:24:55.370741+0000","last_active":"2024-03-04T06:24:55.370741+0000","last_peered":"2024-03-04T06:24:55.370741+0000","last_clean":"2024-03-04T06:24:55.370741+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:24:55.370741+0000","last_undegraded":"2024-03-04T06:24:55.370741+0000","last_fullsized":"2024-03-04T06:24:55.370741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1427,"log_dups_size":0,"ondisk_log_size":1427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6253062,"num_objects":107,"num_object_clones":93,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":625,"num_read_kb":522641,"num_write":477,"num_write_kb":130314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2529,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.417636+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:24:56.417636+0000","last_peered":"2024-03-04T06:24:56.417636+0000","last_clean":"2024-03-04T06:24:56.417636+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:24:56.417636+0000","last_undegraded":"2024-03-04T06:24:56.417636+0000","last_fullsized":"2024-03-04T06:24:56.417636+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":91140754,"num_objects":1283,"num_object_clones":1066,"num_object_copies":3849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":186,"num_read":6861,"num_read_kb":5474171,"num_write":5790,"num_write_kb":1579593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17832,"ondisk_log_size":17832,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706694,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":236832,"kb_used_data":165592,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94135008,"statfs":{"total":96636764160,"available":96394248192,"internally_reserved":0,"allocated":169566208,"data_stored":190519260,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706694,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":185620,"kb_used_data":122700,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94186220,"statfs":{"total":96636764160,"available":96446689280,"internally_reserved":0,"allocated":125644800,"data_stored":140415816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":184812,"kb_used_data":121892,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94187028,"statfs":{"total":96636764160,"available":96447516672,"internally_reserved":0,"allocated":124817408,"data_stored":138632531,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":177904,"kb_used_data":116392,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94193936,"statfs":{"total":96636764160,"available":96454590464,"internally_reserved":0,"allocated":119185408,"data_stored":134922007,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:24:57.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:24:57.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:57.187+0000 7f542b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:57.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:57.191+0000 7f542b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:57.370 INFO:tasks.rados.rados.0.smithi064.stdout:3910: done (1 left) 2024-03-04T06:24:57.370 INFO:tasks.rados.rados.0.smithi064.stdout:3911: read oid 42 snap -1 2024-03-04T06:24:57.370 INFO:tasks.rados.rados.0.smithi064.stdout:3911: expect deleted 2024-03-04T06:24:57.371 INFO:tasks.rados.rados.0.smithi064.stdout:3912: write (excl) oid 6 current snap is 515 2024-03-04T06:24:57.371 INFO:tasks.rados.rados.0.smithi064.stdout:3912: seq_num 1757 ranges {729899=690036,1998627=471271,3166316=363583} 2024-03-04T06:24:57.380 INFO:tasks.rados.rados.0.smithi064.stdout:3912: writing smithi06431743-6 from 729899 to 1419935 tid 1 2024-03-04T06:24:57.388 INFO:tasks.rados.rados.0.smithi064.stdout:3912: writing smithi06431743-6 from 1998627 to 2469898 tid 2 2024-03-04T06:24:57.395 INFO:tasks.rados.rados.0.smithi064.stdout:3912: writing smithi06431743-6 from 3166316 to 3529899 tid 3 2024-03-04T06:24:57.395 INFO:tasks.rados.rados.0.smithi064.stdout:3913: snap_remove snap 514 2024-03-04T06:24:58.374 INFO:tasks.rados.rados.0.smithi064.stdout:3913: done (3 left) 2024-03-04T06:24:58.374 INFO:tasks.rados.rados.0.smithi064.stdout:3914: read oid 36 snap 487 2024-03-04T06:24:58.374 INFO:tasks.rados.rados.0.smithi064.stdout:3914: expect deleted 2024-03-04T06:24:58.374 INFO:tasks.rados.rados.0.smithi064.stdout:3915: read oid 26 snap -1 2024-03-04T06:24:58.375 INFO:tasks.rados.rados.0.smithi064.stdout:3915: expect deleted 2024-03-04T06:24:58.375 INFO:tasks.rados.rados.0.smithi064.stdout:3916: snap_remove snap 513 2024-03-04T06:24:59.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:24:59.385 INFO:tasks.rados.rados.0.smithi064.stdout:3916: done (5 left) 2024-03-04T06:24:59.385 INFO:tasks.rados.rados.0.smithi064.stdout:3917: snap_remove snap 478 2024-03-04T06:24:59.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:59.390+0000 7f1ce466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:24:59.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:24:59.395+0000 7f1ce466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:00.389 INFO:tasks.rados.rados.0.smithi064.stdout:3917: done (5 left) 2024-03-04T06:25:00.389 INFO:tasks.rados.rados.0.smithi064.stdout:3918: write oid 31 current snap is 515 2024-03-04T06:25:00.389 INFO:tasks.rados.rados.0.smithi064.stdout:3918: seq_num 1758 ranges {746706=765285,2057109=731282,3193739=352967} 2024-03-04T06:25:00.398 INFO:tasks.rados.rados.0.smithi064.stdout:3918: writing smithi06431743-31 from 746706 to 1511991 tid 1 2024-03-04T06:25:00.404 INFO:tasks.rados.rados.0.smithi064.stdout:3918: writing smithi06431743-31 from 2057109 to 2788391 tid 2 2024-03-04T06:25:00.409 INFO:tasks.rados.rados.0.smithi064.stdout:3918: writing smithi06431743-31 from 3193739 to 3546706 tid 3 2024-03-04T06:25:00.409 INFO:tasks.rados.rados.0.smithi064.stdout:3919: snap_create 2024-03-04T06:25:00.409 INFO:tasks.rados.rados.0.smithi064.stdout:3909: expect (ObjNum 1749 snap 509 seq_num 1749) 2024-03-04T06:25:00.553 INFO:tasks.rados.rados.0.smithi064.stdout:3912: finishing write tid 1 to smithi06431743-6 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3912: finishing write tid 2 to smithi06431743-6 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3912: finishing write tid 3 to smithi06431743-6 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3912: finishing write tid 4 to smithi06431743-6 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3912: finishing write tid 5 to smithi06431743-6 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3912: oid 6 updating version 0 to 1003 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3912: oid 6 version 1003 is already newer than 1001 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3912: oid 6 version 1003 is already newer than 1002 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3912: oid 6 updating version 1003 to 1004 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 6 v 1004 (ObjNum 1757 snap 515 seq_num 1757) dirty exists 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3912: left oid 6 (ObjNum 1757 snap 515 seq_num 1757) 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3918: finishing write tid 1 to smithi06431743-31 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3918: finishing write tid 2 to smithi06431743-31 2024-03-04T06:25:00.554 INFO:tasks.rados.rados.0.smithi064.stdout:3918: finishing write tid 3 to smithi06431743-31 2024-03-04T06:25:00.561 INFO:tasks.rados.rados.0.smithi064.stdout:3918: finishing write tid 4 to smithi06431743-31 2024-03-04T06:25:00.562 INFO:tasks.rados.rados.0.smithi064.stdout:3918: finishing write tid 5 to smithi06431743-31 2024-03-04T06:25:00.562 INFO:tasks.rados.rados.0.smithi064.stdout:3918: oid 31 updating version 0 to 438 2024-03-04T06:25:00.562 INFO:tasks.rados.rados.0.smithi064.stdout:3918: oid 31 updating version 438 to 440 2024-03-04T06:25:00.562 INFO:tasks.rados.rados.0.smithi064.stdout:3918: oid 31 updating version 440 to 441 2024-03-04T06:25:00.562 INFO:tasks.rados.rados.0.smithi064.stdout:3918: oid 31 version 441 is already newer than 440 2024-03-04T06:25:00.562 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 31 v 441 (ObjNum 1758 snap 515 seq_num 1758) dirty exists 2024-03-04T06:25:00.562 INFO:tasks.rados.rados.0.smithi064.stdout:3918: left oid 31 (ObjNum 1758 snap 515 seq_num 1758) 2024-03-04T06:25:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3909: done (6 left) 2024-03-04T06:25:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3911: done (5 left) 2024-03-04T06:25:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3912: done (4 left) 2024-03-04T06:25:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3914: done (3 left) 2024-03-04T06:25:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3915: done (2 left) 2024-03-04T06:25:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3918: done (1 left) 2024-03-04T06:25:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3919: done (0 left) 2024-03-04T06:25:01.401 INFO:tasks.rados.rados.0.smithi064.stdout:3920: snap_create 2024-03-04T06:25:01.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:25:01.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:01.582+0000 7f1cd915d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:01.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:01.586+0000 7f1cd915d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:01.949 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:25:01.949 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:25:01.950 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:02.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:02.054+0000 7fb97462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:02.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:02.060+0000 7fb97462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:02.252 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:02.253 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:02.264 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1539,"stamp":"2024-03-04T06:25:01.396615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95227407,"num_objects":1286,"num_object_clones":1066,"num_object_copies":3858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":186,"num_read":6967,"num_read_kb":5474263,"num_write":5932,"num_write_kb":1582734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17971,"ondisk_log_size":17971,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":787984,"kb_used_data":529392,"kb_used_omap":0,"kb_used_meta":258560,"kb_avail":376699376,"statfs":{"total":386547056640,"available":385740161024,"internally_reserved":0,"allocated":542097408,"data_stored":607830269,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":264765440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3496285,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":1247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027124"},"pg_stats":[{"pgid":"3.f","version":"1026'984","reported_seq":3473,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-03-04T06:24:57.395931+0000","last_change":"2024-03-04T06:24:57.435675+0000","last_active":"2024-03-04T06:24:57.395931+0000","last_peered":"2024-03-04T06:24:57.395931+0000","last_clean":"2024-03-04T06:24:57.395931+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:24:57.395931+0000","last_undegraded":"2024-03-04T06:24:57.395931+0000","last_fullsized":"2024-03-04T06:24:57.395931+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":984,"log_dups_size":0,"ondisk_log_size":984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":375,"num_read_kb":295545,"num_write":337,"num_write_kb":88619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"1026'827","reported_seq":3482,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-04T06:25:00.397733+0000","last_change":"2024-03-04T06:24:58.402660+0000","last_active":"2024-03-04T06:25:00.397733+0000","last_peered":"2024-03-04T06:25:00.397733+0000","last_clean":"2024-03-04T06:25:00.397733+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:25:00.397733+0000","last_undegraded":"2024-03-04T06:25:00.397733+0000","last_fullsized":"2024-03-04T06:25:00.397733+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6521974,"num_objects":65,"num_object_clones":47,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":261,"num_read_kb":210901,"num_write":267,"num_write_kb":70489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"1015'1362","reported_seq":3690,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.409753+0000","last_change":"2024-03-04T06:24:56.416210+0000","last_active":"2024-03-04T06:24:56.409753+0000","last_peered":"2024-03-04T06:24:56.409753+0000","last_clean":"2024-03-04T06:24:56.409753+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:24:56.409753+0000","last_undegraded":"2024-03-04T06:24:56.409753+0000","last_fullsized":"2024-03-04T06:24:56.409753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1362,"log_dups_size":0,"ondisk_log_size":1362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10169918,"num_objects":100,"num_object_clones":85,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":605,"num_read_kb":483934,"num_write":460,"num_write_kb":125339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"1026'1126","reported_seq":3457,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-03-04T06:24:57.383459+0000","last_change":"2024-03-04T06:24:57.446539+0000","last_active":"2024-03-04T06:24:57.383459+0000","last_peered":"2024-03-04T06:24:57.383459+0000","last_clean":"2024-03-04T06:24:57.383459+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:24:57.383459+0000","last_undegraded":"2024-03-04T06:24:57.383459+0000","last_fullsized":"2024-03-04T06:24:57.383459+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1126,"log_dups_size":0,"ondisk_log_size":1126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5194221,"num_objects":71,"num_object_clones":61,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":341,"num_read_kb":257705,"num_write":335,"num_write_kb":92846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"1015'437","reported_seq":3234,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.402767+0000","last_change":"2024-03-04T06:24:55.396963+0000","last_active":"2024-03-04T06:24:56.402767+0000","last_peered":"2024-03-04T06:24:56.402767+0000","last_clean":"2024-03-04T06:24:56.402767+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:24:56.402767+0000","last_undegraded":"2024-03-04T06:24:56.402767+0000","last_fullsized":"2024-03-04T06:24:56.402767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":123,"num_read_kb":93148,"num_write":126,"num_write_kb":34613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"1026'1092","reported_seq":3538,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-03-04T06:24:57.398806+0000","last_change":"2024-03-04T06:24:57.453560+0000","last_active":"2024-03-04T06:24:57.398806+0000","last_peered":"2024-03-04T06:24:57.398806+0000","last_clean":"2024-03-04T06:24:57.398806+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:24:57.398806+0000","last_undegraded":"2024-03-04T06:24:57.398806+0000","last_fullsized":"2024-03-04T06:24:57.398806+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1092,"log_dups_size":0,"ondisk_log_size":1092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092917,"num_objects":78,"num_object_clones":66,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":451,"num_read_kb":362212,"num_write":370,"num_write_kb":106221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":3002,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-03-04T06:24:57.395192+0000","last_change":"2024-03-04T06:24:57.407009+0000","last_active":"2024-03-04T06:24:57.395192+0000","last_peered":"2024-03-04T06:24:57.395192+0000","last_clean":"2024-03-04T06:24:57.395192+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:24:57.395192+0000","last_undegraded":"2024-03-04T06:24:57.395192+0000","last_fullsized":"2024-03-04T06:24:57.395192+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"1025'719","reported_seq":3323,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.404978+0000","last_change":"2024-03-04T06:24:56.432365+0000","last_active":"2024-03-04T06:24:56.404978+0000","last_peered":"2024-03-04T06:24:56.404978+0000","last_clean":"2024-03-04T06:24:56.404978+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:24:56.404978+0000","last_undegraded":"2024-03-04T06:24:56.404978+0000","last_fullsized":"2024-03-04T06:24:56.404978+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":719,"log_dups_size":0,"ondisk_log_size":719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227373,"num_objects":46,"num_object_clones":37,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":290,"num_read_kb":255610,"num_write":203,"num_write_kb":55194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2718,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.399195+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:24:56.399195+0000","last_peered":"2024-03-04T06:24:56.399195+0000","last_clean":"2024-03-04T06:24:56.399195+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:24:56.399195+0000","last_undegraded":"2024-03-04T06:24:56.399195+0000","last_fullsized":"2024-03-04T06:24:56.399195+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2529,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.425766+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:24:56.425766+0000","last_peered":"2024-03-04T06:24:56.425766+0000","last_clean":"2024-03-04T06:24:56.425766+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:24:56.425766+0000","last_undegraded":"2024-03-04T06:24:56.425766+0000","last_fullsized":"2024-03-04T06:24:56.425766+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"1025'1040","reported_seq":3523,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.419945+0000","last_change":"2024-03-04T06:24:56.441759+0000","last_active":"2024-03-04T06:24:56.419945+0000","last_peered":"2024-03-04T06:24:56.419945+0000","last_clean":"2024-03-04T06:24:56.419945+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:24:56.419945+0000","last_undegraded":"2024-03-04T06:24:56.419945+0000","last_fullsized":"2024-03-04T06:24:56.419945+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8253524,"num_objects":75,"num_object_clones":61,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":455,"num_read_kb":370096,"num_write":340,"num_write_kb":91466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2520,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-03-04T06:24:57.398179+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:24:57.398179+0000","last_peered":"2024-03-04T06:24:57.398179+0000","last_clean":"2024-03-04T06:24:57.398179+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:24:57.398179+0000","last_undegraded":"2024-03-04T06:24:57.398179+0000","last_fullsized":"2024-03-04T06:24:57.398179+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"1018'736","reported_seq":3427,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.398318+0000","last_change":"2024-03-04T06:24:55.386409+0000","last_active":"2024-03-04T06:24:56.398318+0000","last_peered":"2024-03-04T06:24:56.398318+0000","last_clean":"2024-03-04T06:24:56.398318+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:24:56.398318+0000","last_undegraded":"2024-03-04T06:24:56.398318+0000","last_fullsized":"2024-03-04T06:24:56.398318+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135413,"num_objects":54,"num_object_clones":45,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":333,"num_read_kb":275602,"num_write":233,"num_write_kb":66901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2529,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.408796+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:24:56.408796+0000","last_peered":"2024-03-04T06:24:56.408796+0000","last_clean":"2024-03-04T06:24:56.408796+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:24:56.408796+0000","last_undegraded":"2024-03-04T06:24:56.408796+0000","last_fullsized":"2024-03-04T06:24:56.408796+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"1018'2015","reported_seq":4073,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.399978+0000","last_change":"2024-03-04T06:24:55.394686+0000","last_active":"2024-03-04T06:24:56.399978+0000","last_peered":"2024-03-04T06:24:56.399978+0000","last_clean":"2024-03-04T06:24:56.399978+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:24:56.399978+0000","last_undegraded":"2024-03-04T06:24:56.399978+0000","last_fullsized":"2024-03-04T06:24:56.399978+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2015,"log_dups_size":0,"ondisk_log_size":2015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14388871,"num_objects":149,"num_object_clones":122,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":848,"num_read_kb":678782,"num_write":693,"num_write_kb":186619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2520,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-03-04T06:24:57.389000+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:24:57.389000+0000","last_peered":"2024-03-04T06:24:57.389000+0000","last_clean":"2024-03-04T06:24:57.389000+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:24:57.389000+0000","last_undegraded":"2024-03-04T06:24:57.389000+0000","last_fullsized":"2024-03-04T06:24:57.389000+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"1026'2385","reported_seq":4121,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-04T06:24:58.402776+0000","last_change":"2024-03-04T06:24:58.402776+0000","last_active":"2024-03-04T06:24:58.402776+0000","last_peered":"2024-03-04T06:24:58.402776+0000","last_clean":"2024-03-04T06:24:58.402776+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:24:58.402776+0000","last_undegraded":"2024-03-04T06:24:58.402776+0000","last_fullsized":"2024-03-04T06:24:58.402776+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2385,"log_dups_size":0,"ondisk_log_size":2385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12018721,"num_objects":164,"num_object_clones":140,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":20,"num_read":754,"num_read_kb":555488,"num_write":750,"num_write_kb":203150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"1021'995","reported_seq":3517,"reported_epoch":1025,"state":"active+clean+snaptrim","last_fresh":"2024-03-04T06:24:56.426404+0000","last_change":"2024-03-04T06:24:56.426404+0000","last_active":"2024-03-04T06:24:56.426404+0000","last_peered":"2024-03-04T06:24:56.426404+0000","last_clean":"2024-03-04T06:24:56.426404+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:24:56.426404+0000","last_undegraded":"2024-03-04T06:24:56.426404+0000","last_fullsized":"2024-03-04T06:24:56.426404+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":995,"log_dups_size":0,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445864,"num_objects":71,"num_object_clones":58,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":456,"num_read_kb":381496,"num_write":333,"num_write_kb":90436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2639,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-04T06:24:58.400391+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:24:58.400391+0000","last_peered":"2024-03-04T06:24:58.400391+0000","last_clean":"2024-03-04T06:24:58.400391+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:24:58.400391+0000","last_undegraded":"2024-03-04T06:24:58.400391+0000","last_fullsized":"2024-03-04T06:24:58.400391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"1026'1179","reported_seq":3641,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-04T06:24:58.398219+0000","last_change":"2024-03-04T06:24:58.413226+0000","last_active":"2024-03-04T06:24:58.398219+0000","last_peered":"2024-03-04T06:24:58.398219+0000","last_clean":"2024-03-04T06:24:58.398219+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:24:58.398219+0000","last_undegraded":"2024-03-04T06:24:58.398219+0000","last_fullsized":"2024-03-04T06:24:58.398219+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1179,"log_dups_size":0,"ondisk_log_size":1179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924847,"num_objects":88,"num_object_clones":71,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":412,"num_read_kb":309847,"num_write":378,"num_write_kb":101479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2585,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.401576+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:24:56.401576+0000","last_peered":"2024-03-04T06:24:56.401576+0000","last_clean":"2024-03-04T06:24:56.401576+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:24:56.401576+0000","last_undegraded":"2024-03-04T06:24:56.401576+0000","last_fullsized":"2024-03-04T06:24:56.401576+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"1024'1538","reported_seq":3738,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.396147+0000","last_change":"2024-03-04T06:24:55.401455+0000","last_active":"2024-03-04T06:24:56.396147+0000","last_peered":"2024-03-04T06:24:56.396147+0000","last_clean":"2024-03-04T06:24:56.396147+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:24:56.396147+0000","last_undegraded":"2024-03-04T06:24:56.396147+0000","last_fullsized":"2024-03-04T06:24:56.396147+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1538,"log_dups_size":0,"ondisk_log_size":1538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9514049,"num_objects":105,"num_object_clones":89,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":532,"num_read_kb":421164,"num_write":488,"num_write_kb":135907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2641,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-04T06:25:00.395728+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:00.395728+0000","last_peered":"2024-03-04T06:25:00.395728+0000","last_clean":"2024-03-04T06:25:00.395728+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:00.395728+0000","last_undegraded":"2024-03-04T06:25:00.395728+0000","last_fullsized":"2024-03-04T06:25:00.395728+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"1026'1439","reported_seq":3842,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-04T06:24:58.402943+0000","last_change":"2024-03-04T06:24:58.402943+0000","last_active":"2024-03-04T06:24:58.402943+0000","last_peered":"2024-03-04T06:24:58.402943+0000","last_clean":"2024-03-04T06:24:58.402943+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:24:58.402943+0000","last_undegraded":"2024-03-04T06:24:58.402943+0000","last_fullsized":"2024-03-04T06:24:58.402943+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1439,"log_dups_size":0,"ondisk_log_size":1439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9749347,"num_objects":108,"num_object_clones":93,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":628,"num_read_kb":522642,"num_write":482,"num_write_kb":131561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2529,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-04T06:24:56.417636+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:24:56.417636+0000","last_peered":"2024-03-04T06:24:56.417636+0000","last_clean":"2024-03-04T06:24:56.417636+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:24:56.417636+0000","last_undegraded":"2024-03-04T06:24:56.417636+0000","last_fullsized":"2024-03-04T06:24:56.417636+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94637039,"num_objects":1284,"num_object_clones":1066,"num_object_copies":3852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":186,"num_read":6864,"num_read_kb":5474172,"num_write":5795,"num_write_kb":1580840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17874,"ondisk_log_size":17874,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706694,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":236832,"kb_used_data":165592,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94135008,"statfs":{"total":96636764160,"available":96394248192,"internally_reserved":0,"allocated":169566208,"data_stored":190519260,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706694,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":185620,"kb_used_data":122700,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94186220,"statfs":{"total":96636764160,"available":96446689280,"internally_reserved":0,"allocated":125644800,"data_stored":140415816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":187604,"kb_used_data":124684,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94184236,"statfs":{"total":96636764160,"available":96444657664,"internally_reserved":0,"allocated":127676416,"data_stored":141961795,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706698,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":177928,"kb_used_data":116416,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94193912,"statfs":{"total":96636764160,"available":96454565888,"internally_reserved":0,"allocated":119209984,"data_stored":134933398,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:02.399 INFO:tasks.rados.rados.0.smithi064.stdout:3920: done (0 left) 2024-03-04T06:25:02.399 INFO:tasks.rados.rados.0.smithi064.stdout:3921: snap_create 2024-03-04T06:25:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:3921: done (0 left) 2024-03-04T06:25:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:3922: read oid 44 snap -1 2024-03-04T06:25:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:3922: expect deleted 2024-03-04T06:25:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:3923: read oid 15 snap -1 2024-03-04T06:25:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:3923: expect deleted 2024-03-04T06:25:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:3924: write (excl) oid 47 current snap is 518 2024-03-04T06:25:03.357 INFO:tasks.rados.rados.0.smithi064.stdout:3924: seq_num 1759 ranges {763513=440534,1715591=474940,2821162=423353} 2024-03-04T06:25:03.363 INFO:tasks.rados.rados.0.smithi064.stdout:3924: writing smithi06431743-47 from 763513 to 1204047 tid 1 2024-03-04T06:25:03.369 INFO:tasks.rados.rados.0.smithi064.stdout:3924: writing smithi06431743-47 from 1715591 to 2190531 tid 2 2024-03-04T06:25:03.375 INFO:tasks.rados.rados.0.smithi064.stdout:3924: writing smithi06431743-47 from 2821162 to 3244515 tid 3 2024-03-04T06:25:03.375 INFO:tasks.rados.rados.0.smithi064.stdout:3925: write (excl) oid 3 current snap is 518 2024-03-04T06:25:03.375 INFO:tasks.rados.rados.0.smithi064.stdout:3925: seq_num 1760 ranges {780320=515783,2057720=734951,3532232=48088} 2024-03-04T06:25:03.382 INFO:tasks.rados.rados.0.smithi064.stdout:3925: writing smithi06431743-3 from 780320 to 1296103 tid 1 2024-03-04T06:25:03.393 INFO:tasks.rados.rados.0.smithi064.stdout:3925: writing smithi06431743-3 from 2057720 to 2792671 tid 2 2024-03-04T06:25:03.398 INFO:tasks.rados.rados.0.smithi064.stdout:3925: writing smithi06431743-3 from 3532232 to 3580320 tid 3 2024-03-04T06:25:03.398 INFO:tasks.rados.rados.0.smithi064.stdout:3926: write (excl) oid 20 current snap is 518 2024-03-04T06:25:03.398 INFO:tasks.rados.rados.0.smithi064.stdout:3926: seq_num 1761 ranges {797127=591032,2116202=478609,3159655=437472} 2024-03-04T06:25:03.407 INFO:tasks.rados.rados.0.smithi064.stdout:3926: writing smithi06431743-20 from 797127 to 1388159 tid 1 2024-03-04T06:25:03.415 INFO:tasks.rados.rados.0.smithi064.stdout:3926: writing smithi06431743-20 from 2116202 to 2594811 tid 2 2024-03-04T06:25:03.422 INFO:tasks.rados.rados.0.smithi064.stdout:3926: writing smithi06431743-20 from 3159655 to 3597127 tid 3 2024-03-04T06:25:03.422 INFO:tasks.rados.rados.0.smithi064.stdout:3927: read oid 19 snap 480 2024-03-04T06:25:03.422 INFO:tasks.rados.rados.0.smithi064.stdout:3927: expect (ObjNum 1637 snap 466 seq_num 1637) 2024-03-04T06:25:03.423 INFO:tasks.rados.rados.0.smithi064.stdout:3928: read oid 25 snap -1 2024-03-04T06:25:03.423 INFO:tasks.rados.rados.0.smithi064.stdout:3928: expect (ObjNum 1707 snap 494 seq_num 1707) 2024-03-04T06:25:03.423 INFO:tasks.rados.rados.0.smithi064.stdout:3929: write (excl) oid 11 current snap is 518 2024-03-04T06:25:03.423 INFO:tasks.rados.rados.0.smithi064.stdout:3929: seq_num 1762 ranges {413934=666281,1774684=738620,3187078=426856} 2024-03-04T06:25:03.429 INFO:tasks.rados.rados.0.smithi064.stdout:3929: writing smithi06431743-11 from 413934 to 1080215 tid 1 2024-03-04T06:25:03.438 INFO:tasks.rados.rados.0.smithi064.stdout:3929: writing smithi06431743-11 from 1774684 to 2513304 tid 2 2024-03-04T06:25:03.443 INFO:tasks.rados.rados.0.smithi064.stdout:3929: writing smithi06431743-11 from 3187078 to 3613934 tid 3 2024-03-04T06:25:03.443 INFO:tasks.rados.rados.0.smithi064.stdout:3930: delete oid 4 current snap is 518 2024-03-04T06:25:03.443 INFO:tasks.rados.rados.0.smithi064.stdout:3924: finishing write tid 1 to smithi06431743-47 2024-03-04T06:25:03.453 INFO:tasks.rados.rados.0.smithi064.stdout:3928: expect (ObjNum 1707 snap 494 seq_num 1707) 2024-03-04T06:25:03.648 INFO:tasks.rados.rados.0.smithi064.stdout:3927: expect (ObjNum 1637 snap 466 seq_num 1637) 2024-03-04T06:25:03.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:25:03.762 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:03.761+0000 7fe6b8b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:03.765 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:03.765+0000 7fe6b8b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:03.924 INFO:tasks.rados.rados.0.smithi064.stdout:3925: finishing write tid 1 to smithi06431743-3 2024-03-04T06:25:03.924 INFO:tasks.rados.rados.0.smithi064.stdout:3924: finishing write tid 2 to smithi06431743-47 2024-03-04T06:25:03.924 INFO:tasks.rados.rados.0.smithi064.stdout:3924: finishing write tid 3 to smithi06431743-47 2024-03-04T06:25:03.924 INFO:tasks.rados.rados.0.smithi064.stdout:3924: finishing write tid 4 to smithi06431743-47 2024-03-04T06:25:03.924 INFO:tasks.rados.rados.0.smithi064.stdout:3926: finishing write tid 1 to smithi06431743-20 2024-03-04T06:25:03.924 INFO:tasks.rados.rados.0.smithi064.stdout:3924: finishing write tid 5 to smithi06431743-47 2024-03-04T06:25:03.924 INFO:tasks.rados.rados.0.smithi064.stdout:3924: oid 47 updating version 0 to 1134 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3924: oid 47 version 1134 is already newer than 1131 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3924: oid 47 version 1134 is already newer than 1132 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3924: oid 47 version 1134 is already newer than 1133 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 47 v 1134 (ObjNum 1759 snap 518 seq_num 1759) dirty exists 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3924: left oid 47 (ObjNum 1759 snap 518 seq_num 1759) 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3929: finishing write tid 1 to smithi06431743-11 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3922: done (8 left) 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3923: done (7 left) 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3924: done (6 left) 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3927: done (5 left) 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3928: done (4 left) 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3930: done (3 left) 2024-03-04T06:25:03.925 INFO:tasks.rados.rados.0.smithi064.stdout:3931: delete oid 26 current snap is 518 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3925: finishing write tid 2 to smithi06431743-3 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3929: finishing write tid 2 to smithi06431743-11 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3926: finishing write tid 2 to smithi06431743-20 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3925: finishing write tid 3 to smithi06431743-3 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3926: finishing write tid 3 to smithi06431743-20 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3929: finishing write tid 3 to smithi06431743-11 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3925: finishing write tid 4 to smithi06431743-3 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3925: finishing write tid 5 to smithi06431743-3 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3925: oid 3 updating version 0 to 1049 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3925: oid 3 version 1049 is already newer than 1048 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3925: oid 3 updating version 1049 to 1050 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:3925: oid 3 updating version 1050 to 1051 2024-03-04T06:25:03.926 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 1051 (ObjNum 1760 snap 518 seq_num 1760) dirty exists 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3925: left oid 3 (ObjNum 1760 snap 518 seq_num 1760) 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3926: finishing write tid 4 to smithi06431743-20 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3926: finishing write tid 5 to smithi06431743-20 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3926: oid 20 updating version 0 to 1545 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3926: oid 20 updating version 1545 to 1546 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3926: oid 20 updating version 1546 to 1548 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3926: oid 20 version 1548 is already newer than 1548 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 20 v 1548 (ObjNum 1761 snap 518 seq_num 1761) dirty exists 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3926: left oid 20 (ObjNum 1761 snap 518 seq_num 1761) 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3929: finishing write tid 4 to smithi06431743-11 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3929: finishing write tid 5 to smithi06431743-11 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3929: oid 11 updating version 0 to 1371 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3929: oid 11 version 1371 is already newer than 1369 2024-03-04T06:25:03.927 INFO:tasks.rados.rados.0.smithi064.stdout:3929: oid 11 version 1371 is already newer than 1368 2024-03-04T06:25:03.928 INFO:tasks.rados.rados.0.smithi064.stdout:3929: oid 11 version 1371 is already newer than 1370 2024-03-04T06:25:03.928 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 11 v 1371 (ObjNum 1762 snap 518 seq_num 1762) dirty exists 2024-03-04T06:25:03.928 INFO:tasks.rados.rados.0.smithi064.stdout:3929: left oid 11 (ObjNum 1762 snap 518 seq_num 1762) 2024-03-04T06:25:03.928 INFO:tasks.rados.rados.0.smithi064.stdout:3925: done (3 left) 2024-03-04T06:25:03.928 INFO:tasks.rados.rados.0.smithi064.stdout:3926: done (2 left) 2024-03-04T06:25:03.928 INFO:tasks.rados.rados.0.smithi064.stdout:3929: done (1 left) 2024-03-04T06:25:03.928 INFO:tasks.rados.rados.0.smithi064.stdout:3931: done (0 left) 2024-03-04T06:25:03.929 INFO:tasks.rados.rados.0.smithi064.stdout:3932: snap_create 2024-03-04T06:25:04.408 INFO:tasks.rados.rados.0.smithi064.stdout:3932: done (0 left) 2024-03-04T06:25:04.408 INFO:tasks.rados.rados.0.smithi064.stdout:3933: read oid 16 snap -1 2024-03-04T06:25:04.408 INFO:tasks.rados.rados.0.smithi064.stdout:3933: expect (ObjNum 1721 snap 500 seq_num 1721) 2024-03-04T06:25:04.409 INFO:tasks.rados.rados.0.smithi064.stdout:3934: delete oid 21 current snap is 519 2024-03-04T06:25:04.442 INFO:tasks.rados.rados.0.smithi064.stdout:3934: done (1 left) 2024-03-04T06:25:04.442 INFO:tasks.rados.rados.0.smithi064.stdout:3935: write oid 50 current snap is 519 2024-03-04T06:25:04.443 INFO:tasks.rados.rados.0.smithi064.stdout:3935: seq_num 1766 ranges {481162=683630,1608612=629605,2780417=418610} 2024-03-04T06:25:04.448 INFO:tasks.rados.rados.0.smithi064.stdout:3935: writing smithi06431743-50 from 481162 to 1164792 tid 1 2024-03-04T06:25:04.454 INFO:tasks.rados.rados.0.smithi064.stdout:3935: writing smithi06431743-50 from 1608612 to 2238217 tid 2 2024-03-04T06:25:04.460 INFO:tasks.rados.rados.0.smithi064.stdout:3935: writing smithi06431743-50 from 2780417 to 3199027 tid 3 2024-03-04T06:25:04.461 INFO:tasks.rados.rados.0.smithi064.stdout:3936: read oid 39 snap 480 2024-03-04T06:25:04.461 INFO:tasks.rados.rados.0.smithi064.stdout:3936: expect (ObjNum 1654 snap 472 seq_num 1654) 2024-03-04T06:25:04.461 INFO:tasks.rados.rados.0.smithi064.stdout:3937: write oid 49 current snap is 519 2024-03-04T06:25:04.461 INFO:tasks.rados.rados.0.smithi064.stdout:3937: seq_num 1767 ranges {497969=758879,1667094=489616,2924193=629882} 2024-03-04T06:25:04.467 INFO:tasks.rados.rados.0.smithi064.stdout:3937: writing smithi06431743-49 from 497969 to 1256848 tid 1 2024-03-04T06:25:04.472 INFO:tasks.rados.rados.0.smithi064.stdout:3937: writing smithi06431743-49 from 1667094 to 2156710 tid 2 2024-03-04T06:25:04.480 INFO:tasks.rados.rados.0.smithi064.stdout:3937: writing smithi06431743-49 from 2924193 to 3554075 tid 3 2024-03-04T06:25:04.480 INFO:tasks.rados.rados.0.smithi064.stdout:3938: write oid 28 current snap is 519 2024-03-04T06:25:04.480 INFO:tasks.rados.rados.0.smithi064.stdout:3938: seq_num 1768 ranges {514776=434128,1609223=633274,2718910=557507} 2024-03-04T06:25:04.485 INFO:tasks.rados.rados.0.smithi064.stdout:3938: writing smithi06431743-28 from 514776 to 948904 tid 1 2024-03-04T06:25:04.493 INFO:tasks.rados.rados.0.smithi064.stdout:3938: writing smithi06431743-28 from 1609223 to 2242497 tid 2 2024-03-04T06:25:04.498 INFO:tasks.rados.rados.0.smithi064.stdout:3938: writing smithi06431743-28 from 2718910 to 3276417 tid 3 2024-03-04T06:25:04.498 INFO:tasks.rados.rados.0.smithi064.stdout:3939: read oid 34 snap -1 2024-03-04T06:25:04.498 INFO:tasks.rados.rados.0.smithi064.stdout:3939: expect (ObjNum 1564 snap 443 seq_num 1564) 2024-03-04T06:25:04.498 INFO:tasks.rados.rados.0.smithi064.stdout:3940: read oid 35 snap -1 2024-03-04T06:25:04.498 INFO:tasks.rados.rados.0.smithi064.stdout:3940: expect (ObjNum 1722 snap 500 seq_num 1722) 2024-03-04T06:25:04.498 INFO:tasks.rados.rados.0.smithi064.stdout:3941: snap_create 2024-03-04T06:25:04.498 INFO:tasks.rados.rados.0.smithi064.stdout:3933: expect (ObjNum 1721 snap 500 seq_num 1721) 2024-03-04T06:25:04.651 INFO:tasks.rados.rados.0.smithi064.stdout:3935: finishing write tid 1 to smithi06431743-50 2024-03-04T06:25:04.651 INFO:tasks.rados.rados.0.smithi064.stdout:3936: expect (ObjNum 1654 snap 472 seq_num 1654) 2024-03-04T06:25:04.851 INFO:tasks.rados.rados.0.smithi064.stdout:3937: finishing write tid 1 to smithi06431743-49 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:3935: finishing write tid 2 to smithi06431743-50 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:3938: finishing write tid 1 to smithi06431743-28 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:3935: finishing write tid 3 to smithi06431743-50 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:3935: finishing write tid 4 to smithi06431743-50 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:3935: finishing write tid 5 to smithi06431743-50 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:3935: oid 50 updating version 0 to 1447 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:3935: oid 50 version 1447 is already newer than 1447 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:3935: oid 50 updating version 1447 to 1448 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:3935: oid 50 version 1448 is already newer than 1445 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 50 v 1448 (ObjNum 1766 snap 519 seq_num 1766) dirty exists 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:3935: left oid 50 (ObjNum 1766 snap 519 seq_num 1766) 2024-03-04T06:25:04.852 INFO:tasks.rados.rados.0.smithi064.stdout:3937: finishing write tid 2 to smithi06431743-49 2024-03-04T06:25:04.853 INFO:tasks.rados.rados.0.smithi064.stdout:3938: finishing write tid 2 to smithi06431743-28 2024-03-04T06:25:04.853 INFO:tasks.rados.rados.0.smithi064.stdout:3940: expect (ObjNum 1722 snap 500 seq_num 1722) 2024-03-04T06:25:04.960 INFO:tasks.rados.rados.0.smithi064.stdout:3938: finishing write tid 3 to smithi06431743-28 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3937: finishing write tid 3 to smithi06431743-49 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3937: finishing write tid 4 to smithi06431743-49 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3938: finishing write tid 4 to smithi06431743-28 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3937: finishing write tid 5 to smithi06431743-49 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3937: oid 49 updating version 0 to 725 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3937: oid 49 updating version 725 to 726 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3937: oid 49 updating version 726 to 728 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3937: oid 49 version 728 is already newer than 727 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 49 v 728 (ObjNum 1767 snap 519 seq_num 1767) dirty exists 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3937: left oid 49 (ObjNum 1767 snap 519 seq_num 1767) 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3938: finishing write tid 5 to smithi06431743-28 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3938: oid 28 updating version 0 to 2023 2024-03-04T06:25:04.961 INFO:tasks.rados.rados.0.smithi064.stdout:3938: oid 28 updating version 2023 to 2025 2024-03-04T06:25:04.962 INFO:tasks.rados.rados.0.smithi064.stdout:3938: oid 28 updating version 2025 to 2026 2024-03-04T06:25:04.962 INFO:tasks.rados.rados.0.smithi064.stdout:3938: oid 28 version 2026 is already newer than 2024 2024-03-04T06:25:04.962 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 2026 (ObjNum 1768 snap 519 seq_num 1768) dirty exists 2024-03-04T06:25:04.962 INFO:tasks.rados.rados.0.smithi064.stdout:3938: left oid 28 (ObjNum 1768 snap 519 seq_num 1768) 2024-03-04T06:25:04.962 INFO:tasks.rados.rados.0.smithi064.stdout:3939: expect (ObjNum 1564 snap 443 seq_num 1564) 2024-03-04T06:25:05.414 INFO:tasks.rados.rados.0.smithi064.stdout:3933: done (7 left) 2024-03-04T06:25:05.415 INFO:tasks.rados.rados.0.smithi064.stdout:3935: done (6 left) 2024-03-04T06:25:05.415 INFO:tasks.rados.rados.0.smithi064.stdout:3936: done (5 left) 2024-03-04T06:25:05.415 INFO:tasks.rados.rados.0.smithi064.stdout:3937: done (4 left) 2024-03-04T06:25:05.415 INFO:tasks.rados.rados.0.smithi064.stdout:3938: done (3 left) 2024-03-04T06:25:05.415 INFO:tasks.rados.rados.0.smithi064.stdout:3939: done (2 left) 2024-03-04T06:25:05.415 INFO:tasks.rados.rados.0.smithi064.stdout:3940: done (1 left) 2024-03-04T06:25:05.415 INFO:tasks.rados.rados.0.smithi064.stdout:3941: done (0 left) 2024-03-04T06:25:05.415 INFO:tasks.rados.rados.0.smithi064.stdout:3942: snap_remove snap 505 2024-03-04T06:25:05.837 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:25:05.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:05.952+0000 7f53d8f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:05.957 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:05.957+0000 7f53d8f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:06.417 INFO:tasks.rados.rados.0.smithi064.stdout:3942: done (0 left) 2024-03-04T06:25:06.418 INFO:tasks.rados.rados.0.smithi064.stdout:3943: snap_create 2024-03-04T06:25:07.266 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:25:07.267 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:25:07.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:07.388 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:07.384+0000 7ff28511d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:07.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:07.391+0000 7ff28511d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:07.422 INFO:tasks.rados.rados.0.smithi064.stdout:3943: done (0 left) 2024-03-04T06:25:07.422 INFO:tasks.rados.rados.0.smithi064.stdout:3944: write oid 41 current snap is 521 2024-03-04T06:25:07.422 INFO:tasks.rados.rados.0.smithi064.stdout:3944: seq_num 1769 ranges {531583=509377,1667705=493285,2862686=768779} 2024-03-04T06:25:07.427 INFO:tasks.rados.rados.0.smithi064.stdout:3944: writing smithi06431743-41 from 531583 to 1040960 tid 1 2024-03-04T06:25:07.434 INFO:tasks.rados.rados.0.smithi064.stdout:3944: writing smithi06431743-41 from 1667705 to 2160990 tid 2 2024-03-04T06:25:07.442 INFO:tasks.rados.rados.0.smithi064.stdout:3944: writing smithi06431743-41 from 2862686 to 3631465 tid 3 2024-03-04T06:25:07.442 INFO:tasks.rados.rados.0.smithi064.stdout:3945: read oid 38 snap -1 2024-03-04T06:25:07.442 INFO:tasks.rados.rados.0.smithi064.stdout:3945: expect (ObjNum 1731 snap 503 seq_num 1731) 2024-03-04T06:25:07.442 INFO:tasks.rados.rados.0.smithi064.stdout:3946: snap_create 2024-03-04T06:25:07.513 INFO:tasks.rados.rados.0.smithi064.stdout:3945: expect (ObjNum 1731 snap 503 seq_num 1731) 2024-03-04T06:25:07.639 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:07.640 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:07.652 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1548,"stamp":"2024-03-04T06:25:07.420617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":106572990,"num_objects":1294,"num_object_clones":1073,"num_object_copies":3882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":185,"num_read":7025,"num_read_kb":5530258,"num_write":5978,"num_write_kb":1596992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18088,"ondisk_log_size":18088,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":815656,"kb_used_data":573704,"kb_used_omap":0,"kb_used_meta":241920,"kb_avail":376671704,"statfs":{"total":386547056640,"available":385711824896,"internally_reserved":0,"allocated":587472896,"data_stored":662668833,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":247726080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7714842,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":49401,"num_write":36,"num_write_kb":10959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.065522"},"pg_stats":[{"pgid":"3.f","version":"1029'986","reported_seq":3490,"reported_epoch":1032,"state":"active+clean","last_fresh":"2024-03-04T06:25:03.384270+0000","last_change":"2024-03-04T06:25:02.392737+0000","last_active":"2024-03-04T06:25:03.384270+0000","last_peered":"2024-03-04T06:25:03.384270+0000","last_clean":"2024-03-04T06:25:03.384270+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:25:03.384270+0000","last_undegraded":"2024-03-04T06:25:03.384270+0000","last_fullsized":"2024-03-04T06:25:03.384270+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":986,"log_dups_size":0,"ondisk_log_size":986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":375,"num_read_kb":295545,"num_write":337,"num_write_kb":88619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"1029'831","reported_seq":3498,"reported_epoch":1033,"state":"active+clean","last_fresh":"2024-03-04T06:25:04.428764+0000","last_change":"2024-03-04T06:25:02.402818+0000","last_active":"2024-03-04T06:25:04.428764+0000","last_peered":"2024-03-04T06:25:04.428764+0000","last_clean":"2024-03-04T06:25:04.428764+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:25:04.428764+0000","last_undegraded":"2024-03-04T06:25:04.428764+0000","last_fullsized":"2024-03-04T06:25:04.428764+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":831,"log_dups_size":0,"ondisk_log_size":831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6521974,"num_objects":65,"num_object_clones":47,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":261,"num_read_kb":210901,"num_write":267,"num_write_kb":70489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"1032'1371","reported_seq":3728,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.929196+0000","last_change":"2024-03-04T06:25:01.454353+0000","last_active":"2024-03-04T06:25:05.929196+0000","last_peered":"2024-03-04T06:25:05.929196+0000","last_clean":"2024-03-04T06:25:05.929196+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:25:05.929196+0000","last_undegraded":"2024-03-04T06:25:05.929196+0000","last_fullsized":"2024-03-04T06:25:05.929196+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11514478,"num_objects":101,"num_object_clones":86,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":619,"num_read_kb":501815,"num_write":465,"num_write_kb":127129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"1032'1134","reported_seq":3479,"reported_epoch":1032,"state":"active+clean","last_fresh":"2024-03-04T06:25:03.504249+0000","last_change":"2024-03-04T06:25:02.384065+0000","last_active":"2024-03-04T06:25:03.504249+0000","last_peered":"2024-03-04T06:25:03.504249+0000","last_clean":"2024-03-04T06:25:03.504249+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:25:03.504249+0000","last_undegraded":"2024-03-04T06:25:03.504249+0000","last_fullsized":"2024-03-04T06:25:03.504249+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1134,"log_dups_size":0,"ondisk_log_size":1134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8757734,"num_objects":71,"num_object_clones":61,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":343,"num_read_kb":257706,"num_write":340,"num_write_kb":94155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"1029'441","reported_seq":3266,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.914794+0000","last_change":"2024-03-04T06:25:00.461507+0000","last_active":"2024-03-04T06:25:05.914794+0000","last_peered":"2024-03-04T06:25:05.914794+0000","last_clean":"2024-03-04T06:25:05.914794+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:25:05.914794+0000","last_undegraded":"2024-03-04T06:25:05.914794+0000","last_fullsized":"2024-03-04T06:25:05.914794+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3546706,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":124,"num_read_kb":93149,"num_write":131,"num_write_kb":36421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"1031'1098","reported_seq":3555,"reported_epoch":1032,"state":"active+clean","last_fresh":"2024-03-04T06:25:03.385261+0000","last_change":"2024-03-04T06:25:02.391992+0000","last_active":"2024-03-04T06:25:03.385261+0000","last_peered":"2024-03-04T06:25:03.385261+0000","last_clean":"2024-03-04T06:25:03.385261+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:25:03.385261+0000","last_undegraded":"2024-03-04T06:25:03.385261+0000","last_fullsized":"2024-03-04T06:25:03.385261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1098,"log_dups_size":0,"ondisk_log_size":1098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092917,"num_objects":78,"num_object_clones":66,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":451,"num_read_kb":362212,"num_write":370,"num_write_kb":106221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":3019,"reported_epoch":1032,"state":"active+clean","last_fresh":"2024-03-04T06:25:03.383392+0000","last_change":"2024-03-04T06:25:00.539039+0000","last_active":"2024-03-04T06:25:03.383392+0000","last_peered":"2024-03-04T06:25:03.383392+0000","last_clean":"2024-03-04T06:25:03.383392+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:25:03.383392+0000","last_undegraded":"2024-03-04T06:25:03.383392+0000","last_fullsized":"2024-03-04T06:25:03.383392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"1033'728","reported_seq":3355,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.926653+0000","last_change":"2024-03-04T06:25:01.448742+0000","last_active":"2024-03-04T06:25:05.926653+0000","last_peered":"2024-03-04T06:25:05.926653+0000","last_clean":"2024-03-04T06:25:05.926653+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:25:05.926653+0000","last_undegraded":"2024-03-04T06:25:05.926653+0000","last_fullsized":"2024-03-04T06:25:05.926653+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":728,"log_dups_size":0,"ondisk_log_size":728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3697969,"num_objects":47,"num_object_clones":38,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":291,"num_read_kb":255611,"num_write":208,"num_write_kb":57031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2741,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.905586+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:25:05.905586+0000","last_peered":"2024-03-04T06:25:05.905586+0000","last_clean":"2024-03-04T06:25:05.905586+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:05.905586+0000","last_undegraded":"2024-03-04T06:25:05.905586+0000","last_fullsized":"2024-03-04T06:25:05.905586+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2552,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.936344+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:25:05.936344+0000","last_peered":"2024-03-04T06:25:05.936344+0000","last_clean":"2024-03-04T06:25:05.936344+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:05.936344+0000","last_undegraded":"2024-03-04T06:25:05.936344+0000","last_fullsized":"2024-03-04T06:25:05.936344+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"1032'1051","reported_seq":3555,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.939014+0000","last_change":"2024-03-04T06:25:01.460038+0000","last_active":"2024-03-04T06:25:05.939014+0000","last_peered":"2024-03-04T06:25:05.939014+0000","last_clean":"2024-03-04T06:25:05.939014+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:25:05.939014+0000","last_undegraded":"2024-03-04T06:25:05.939014+0000","last_fullsized":"2024-03-04T06:25:05.939014+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1051,"log_dups_size":0,"ondisk_log_size":1051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9430014,"num_objects":76,"num_object_clones":62,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":457,"num_read_kb":370097,"num_write":345,"num_write_kb":92735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2534,"reported_epoch":1032,"state":"active+clean","last_fresh":"2024-03-04T06:25:03.393791+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:03.393791+0000","last_peered":"2024-03-04T06:25:03.393791+0000","last_clean":"2024-03-04T06:25:03.393791+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:03.393791+0000","last_undegraded":"2024-03-04T06:25:03.393791+0000","last_fullsized":"2024-03-04T06:25:03.393791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"1029'742","reported_seq":3454,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.907997+0000","last_change":"2024-03-04T06:25:02.431227+0000","last_active":"2024-03-04T06:25:05.907997+0000","last_peered":"2024-03-04T06:25:05.907997+0000","last_clean":"2024-03-04T06:25:05.907997+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:25:05.907997+0000","last_undegraded":"2024-03-04T06:25:05.907997+0000","last_fullsized":"2024-03-04T06:25:05.907997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6135413,"num_objects":54,"num_object_clones":45,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":333,"num_read_kb":275602,"num_write":233,"num_write_kb":66901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2552,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.931813+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:25:05.931813+0000","last_peered":"2024-03-04T06:25:05.931813+0000","last_clean":"2024-03-04T06:25:05.931813+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:25:05.931813+0000","last_undegraded":"2024-03-04T06:25:05.931813+0000","last_fullsized":"2024-03-04T06:25:05.931813+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"1033'2026","reported_seq":4109,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.919470+0000","last_change":"2024-03-04T06:25:02.434613+0000","last_active":"2024-03-04T06:25:05.919470+0000","last_peered":"2024-03-04T06:25:05.919470+0000","last_clean":"2024-03-04T06:25:05.919470+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:25:05.919470+0000","last_undegraded":"2024-03-04T06:25:05.919470+0000","last_fullsized":"2024-03-04T06:25:05.919470+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2026,"log_dups_size":0,"ondisk_log_size":2026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14893081,"num_objects":150,"num_object_clones":123,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":856,"num_read_kb":683818,"num_write":698,"num_write_kb":188207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2534,"reported_epoch":1032,"state":"active+clean","last_fresh":"2024-03-04T06:25:03.377746+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:03.377746+0000","last_peered":"2024-03-04T06:25:03.377746+0000","last_clean":"2024-03-04T06:25:03.377746+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:03.377746+0000","last_undegraded":"2024-03-04T06:25:03.377746+0000","last_fullsized":"2024-03-04T06:25:03.377746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"1032'2395","reported_seq":4142,"reported_epoch":1033,"state":"active+clean","last_fresh":"2024-03-04T06:25:04.418530+0000","last_change":"2024-03-04T06:25:02.404201+0000","last_active":"2024-03-04T06:25:04.418530+0000","last_peered":"2024-03-04T06:25:04.418530+0000","last_clean":"2024-03-04T06:25:04.418530+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:25:04.418530+0000","last_undegraded":"2024-03-04T06:25:04.418530+0000","last_fullsized":"2024-03-04T06:25:04.418530+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2395,"log_dups_size":0,"ondisk_log_size":2395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8892190,"num_objects":165,"num_object_clones":141,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":761,"num_read_kb":565781,"num_write":751,"num_write_kb":203150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"1033'1009","reported_seq":3550,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.937484+0000","last_change":"2024-03-04T06:25:01.457946+0000","last_active":"2024-03-04T06:25:05.937484+0000","last_peered":"2024-03-04T06:25:05.937484+0000","last_clean":"2024-03-04T06:25:05.937484+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:25:05.937484+0000","last_undegraded":"2024-03-04T06:25:05.937484+0000","last_fullsized":"2024-03-04T06:25:05.937484+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1009,"log_dups_size":0,"ondisk_log_size":1009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3529899,"num_objects":72,"num_object_clones":59,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":458,"num_read_kb":381497,"num_write":338,"num_write_kb":91927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2655,"reported_epoch":1033,"state":"active+clean","last_fresh":"2024-03-04T06:25:04.435935+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:25:04.435935+0000","last_peered":"2024-03-04T06:25:04.435935+0000","last_clean":"2024-03-04T06:25:04.435935+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:25:04.435935+0000","last_undegraded":"2024-03-04T06:25:04.435935+0000","last_fullsized":"2024-03-04T06:25:04.435935+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"1029'1183","reported_seq":3662,"reported_epoch":1033,"state":"active+clean","last_fresh":"2024-03-04T06:25:04.451755+0000","last_change":"2024-03-04T06:25:02.410573+0000","last_active":"2024-03-04T06:25:04.451755+0000","last_peered":"2024-03-04T06:25:04.451755+0000","last_clean":"2024-03-04T06:25:04.451755+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:25:04.451755+0000","last_undegraded":"2024-03-04T06:25:04.451755+0000","last_fullsized":"2024-03-04T06:25:04.451755+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1183,"log_dups_size":0,"ondisk_log_size":1183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924847,"num_objects":88,"num_object_clones":71,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":418,"num_read_kb":317414,"num_write":378,"num_write_kb":101479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2608,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.915973+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:25:05.915973+0000","last_peered":"2024-03-04T06:25:05.915973+0000","last_clean":"2024-03-04T06:25:05.915973+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:05.915973+0000","last_undegraded":"2024-03-04T06:25:05.915973+0000","last_fullsized":"2024-03-04T06:25:05.915973+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"1032'1548","reported_seq":3776,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.910127+0000","last_change":"2024-03-04T06:25:02.431379+0000","last_active":"2024-03-04T06:25:05.910127+0000","last_peered":"2024-03-04T06:25:05.910127+0000","last_clean":"2024-03-04T06:25:05.910127+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:25:05.910127+0000","last_undegraded":"2024-03-04T06:25:05.910127+0000","last_fullsized":"2024-03-04T06:25:05.910127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13111176,"num_objects":106,"num_object_clones":89,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":13,"num_read":546,"num_read_kb":436376,"num_write":493,"num_write_kb":137381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2655,"reported_epoch":1033,"state":"active+clean","last_fresh":"2024-03-04T06:25:04.424331+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:04.424331+0000","last_peered":"2024-03-04T06:25:04.424331+0000","last_clean":"2024-03-04T06:25:04.424331+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:04.424331+0000","last_undegraded":"2024-03-04T06:25:04.424331+0000","last_fullsized":"2024-03-04T06:25:04.424331+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"1033'1448","reported_seq":3864,"reported_epoch":1033,"state":"active+clean","last_fresh":"2024-03-04T06:25:04.546634+0000","last_change":"2024-03-04T06:25:02.403519+0000","last_active":"2024-03-04T06:25:04.546634+0000","last_peered":"2024-03-04T06:25:04.546634+0000","last_clean":"2024-03-04T06:25:04.546634+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:25:04.546634+0000","last_undegraded":"2024-03-04T06:25:04.546634+0000","last_fullsized":"2024-03-04T06:25:04.546634+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1448,"log_dups_size":0,"ondisk_log_size":1448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9934224,"num_objects":109,"num_object_clones":94,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":629,"num_read_kb":522643,"num_write":487,"num_write_kb":133253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2552,"reported_epoch":1034,"state":"active+clean","last_fresh":"2024-03-04T06:25:05.941228+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:25:05.941228+0000","last_peered":"2024-03-04T06:25:05.941228+0000","last_clean":"2024-03-04T06:25:05.941228+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:05.941228+0000","last_undegraded":"2024-03-04T06:25:05.941228+0000","last_fullsized":"2024-03-04T06:25:05.941228+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":105982622,"num_objects":1292,"num_object_clones":1073,"num_object_copies":3876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":185,"num_read":6922,"num_read_kb":5530167,"num_write":5841,"num_write_kb":1595098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17991,"ondisk_log_size":17991,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":252512,"kb_used_data":181272,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94119328,"statfs":{"total":96636764160,"available":96378191872,"internally_reserved":0,"allocated":185622528,"data_stored":209898424,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":198172,"kb_used_data":135252,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94173668,"statfs":{"total":96636764160,"available":96433836032,"internally_reserved":0,"allocated":138498048,"data_stored":156610440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":195940,"kb_used_data":133020,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94175900,"statfs":{"total":96636764160,"available":96436121600,"internally_reserved":0,"allocated":136212480,"data_stored":152366428,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706703,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":169032,"kb_used_data":124160,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94202808,"statfs":{"total":96636764160,"available":96463675392,"internally_reserved":0,"allocated":127139840,"data_stored":143793541,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:07.679 INFO:tasks.rados.rados.0.smithi064.stdout:3944: finishing write tid 1 to smithi06431743-41 2024-03-04T06:25:07.680 INFO:tasks.rados.rados.0.smithi064.stdout:3944: finishing write tid 2 to smithi06431743-41 2024-03-04T06:25:07.680 INFO:tasks.rados.rados.0.smithi064.stdout:3944: finishing write tid 3 to smithi06431743-41 2024-03-04T06:25:07.680 INFO:tasks.rados.rados.0.smithi064.stdout:3944: finishing write tid 4 to smithi06431743-41 2024-03-04T06:25:07.680 INFO:tasks.rados.rados.0.smithi064.stdout:3944: finishing write tid 5 to smithi06431743-41 2024-03-04T06:25:07.680 INFO:tasks.rados.rados.0.smithi064.stdout:3944: oid 41 updating version 0 to 1054 2024-03-04T06:25:07.680 INFO:tasks.rados.rados.0.smithi064.stdout:3944: oid 41 updating version 1054 to 1056 2024-03-04T06:25:07.680 INFO:tasks.rados.rados.0.smithi064.stdout:3944: oid 41 version 1056 is already newer than 1056 2024-03-04T06:25:07.680 INFO:tasks.rados.rados.0.smithi064.stdout:3944: oid 41 version 1056 is already newer than 1053 2024-03-04T06:25:07.680 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 41 v 1056 (ObjNum 1769 snap 521 seq_num 1769) dirty exists 2024-03-04T06:25:07.680 INFO:tasks.rados.rados.0.smithi064.stdout:3944: left oid 41 (ObjNum 1769 snap 521 seq_num 1769) 2024-03-04T06:25:08.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:25:08.148 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:08.146+0000 7f1691f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:08.151 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:08.150+0000 7f1691f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:08.428 INFO:tasks.rados.rados.0.smithi064.stdout:3944: done (2 left) 2024-03-04T06:25:08.429 INFO:tasks.rados.rados.0.smithi064.stdout:3945: done (1 left) 2024-03-04T06:25:08.429 INFO:tasks.rados.rados.0.smithi064.stdout:3946: done (0 left) 2024-03-04T06:25:08.429 INFO:tasks.rados.rados.0.smithi064.stdout:3947: write (excl) oid 27 current snap is 522 2024-03-04T06:25:08.429 INFO:tasks.rados.rados.0.smithi064.stdout:3947: seq_num 1770 ranges {548390=584626,1726187=636943,2773756=580051} 2024-03-04T06:25:08.434 INFO:tasks.rados.rados.0.smithi064.stdout:3947: writing smithi06431743-27 from 548390 to 1133016 tid 1 2024-03-04T06:25:08.441 INFO:tasks.rados.rados.0.smithi064.stdout:3947: writing smithi06431743-27 from 1726187 to 2363130 tid 2 2024-03-04T06:25:08.447 INFO:tasks.rados.rados.0.smithi064.stdout:3947: writing smithi06431743-27 from 2773756 to 3353807 tid 3 2024-03-04T06:25:08.447 INFO:tasks.rados.rados.0.smithi064.stdout:3948: read oid 38 snap 510 2024-03-04T06:25:08.447 INFO:tasks.rados.rados.0.smithi064.stdout:3948: expect (ObjNum 1731 snap 503 seq_num 1731) 2024-03-04T06:25:08.447 INFO:tasks.rados.rados.0.smithi064.stdout:3949: write (excl) oid 15 current snap is 522 2024-03-04T06:25:08.447 INFO:tasks.rados.rados.0.smithi064.stdout:3949: seq_num 1771 ranges {565197=659875,1784669=496954,2917532=791323} 2024-03-04T06:25:08.454 INFO:tasks.rados.rados.0.smithi064.stdout:3949: writing smithi06431743-15 from 565197 to 1225072 tid 1 2024-03-04T06:25:08.462 INFO:tasks.rados.rados.0.smithi064.stdout:3949: writing smithi06431743-15 from 1784669 to 2281623 tid 2 2024-03-04T06:25:08.471 INFO:tasks.rados.rados.0.smithi064.stdout:3949: writing smithi06431743-15 from 2917532 to 3708855 tid 3 2024-03-04T06:25:08.473 INFO:tasks.rados.rados.0.smithi064.stdout:3950: read oid 3 snap -1 2024-03-04T06:25:08.473 INFO:tasks.rados.rados.0.smithi064.stdout:3950: expect (ObjNum 1760 snap 518 seq_num 1760) 2024-03-04T06:25:08.474 INFO:tasks.rados.rados.0.smithi064.stdout:3951: write (excl) oid 7 current snap is 522 2024-03-04T06:25:08.474 INFO:tasks.rados.rados.0.smithi064.stdout:3951: seq_num 1772 ranges {582004=451477,1443151=640612,2544955=602595,3612817=169187} 2024-03-04T06:25:08.480 INFO:tasks.rados.rados.0.smithi064.stdout:3951: writing smithi06431743-7 from 582004 to 1033481 tid 1 2024-03-04T06:25:08.488 INFO:tasks.rados.rados.0.smithi064.stdout:3951: writing smithi06431743-7 from 1443151 to 2083763 tid 2 2024-03-04T06:25:08.496 INFO:tasks.rados.rados.0.smithi064.stdout:3951: writing smithi06431743-7 from 2544955 to 3147550 tid 3 2024-03-04T06:25:08.500 INFO:tasks.rados.rados.0.smithi064.stdout:3951: writing smithi06431743-7 from 3612817 to 3782004 tid 4 2024-03-04T06:25:08.500 INFO:tasks.rados.rados.0.smithi064.stdout:3952: read oid 12 snap -1 2024-03-04T06:25:08.500 INFO:tasks.rados.rados.0.smithi064.stdout:3952: expect (ObjNum 1674 snap 482 seq_num 1674) 2024-03-04T06:25:08.500 INFO:tasks.rados.rados.0.smithi064.stdout:3953: write (excl) oid 40 current snap is 522 2024-03-04T06:25:08.500 INFO:tasks.rados.rados.0.smithi064.stdout:3953: seq_num 1773 ranges {598811=526726,1901633=500623,2972378=530220} 2024-03-04T06:25:08.506 INFO:tasks.rados.rados.0.smithi064.stdout:3953: writing smithi06431743-40 from 598811 to 1125537 tid 1 2024-03-04T06:25:08.513 INFO:tasks.rados.rados.0.smithi064.stdout:3953: writing smithi06431743-40 from 1901633 to 2402256 tid 2 2024-03-04T06:25:08.520 INFO:tasks.rados.rados.0.smithi064.stdout:3953: writing smithi06431743-40 from 2972378 to 3502598 tid 3 2024-03-04T06:25:08.520 INFO:tasks.rados.rados.0.smithi064.stdout:3954: read oid 45 snap -1 2024-03-04T06:25:08.520 INFO:tasks.rados.rados.0.smithi064.stdout:3954: expect (ObjNum 1598 snap 453 seq_num 1598) 2024-03-04T06:25:08.520 INFO:tasks.rados.rados.0.smithi064.stdout:3955: write (excl) oid 4 current snap is 522 2024-03-04T06:25:08.520 INFO:tasks.rados.rados.0.smithi064.stdout:3955: seq_num 1774 ranges {615618=601975,1960115=644281,3399801=415817} 2024-03-04T06:25:08.528 INFO:tasks.rados.rados.0.smithi064.stdout:3955: writing smithi06431743-4 from 615618 to 1217593 tid 1 2024-03-04T06:25:08.536 INFO:tasks.rados.rados.0.smithi064.stdout:3955: writing smithi06431743-4 from 1960115 to 2604396 tid 2 2024-03-04T06:25:08.543 INFO:tasks.rados.rados.0.smithi064.stdout:3955: writing smithi06431743-4 from 3399801 to 3815618 tid 3 2024-03-04T06:25:08.544 INFO:tasks.rados.rados.0.smithi064.stdout:3956: snap_create 2024-03-04T06:25:08.544 INFO:tasks.rados.rados.0.smithi064.stdout:3947: finishing write tid 1 to smithi06431743-27 2024-03-04T06:25:08.544 INFO:tasks.rados.rados.0.smithi064.stdout:3952: expect (ObjNum 1674 snap 482 seq_num 1674) 2024-03-04T06:25:08.662 INFO:tasks.rados.rados.0.smithi064.stdout:3948: expect (ObjNum 1731 snap 503 seq_num 1731) 2024-03-04T06:25:08.817 INFO:tasks.rados.rados.0.smithi064.stdout:3949: finishing write tid 1 to smithi06431743-15 2024-03-04T06:25:08.817 INFO:tasks.rados.rados.0.smithi064.stdout:3950: expect (ObjNum 1760 snap 518 seq_num 1760) 2024-03-04T06:25:08.992 INFO:tasks.rados.rados.0.smithi064.stdout:3951: finishing write tid 1 to smithi06431743-7 2024-03-04T06:25:08.993 INFO:tasks.rados.rados.0.smithi064.stdout:3947: finishing write tid 2 to smithi06431743-27 2024-03-04T06:25:08.993 INFO:tasks.rados.rados.0.smithi064.stdout:3947: finishing write tid 3 to smithi06431743-27 2024-03-04T06:25:08.993 INFO:tasks.rados.rados.0.smithi064.stdout:3953: finishing write tid 1 to smithi06431743-40 2024-03-04T06:25:08.993 INFO:tasks.rados.rados.0.smithi064.stdout:3955: finishing write tid 1 to smithi06431743-4 2024-03-04T06:25:08.993 INFO:tasks.rados.rados.0.smithi064.stdout:3949: finishing write tid 2 to smithi06431743-15 2024-03-04T06:25:08.993 INFO:tasks.rados.rados.0.smithi064.stdout:3947: finishing write tid 4 to smithi06431743-27 2024-03-04T06:25:08.993 INFO:tasks.rados.rados.0.smithi064.stdout:3947: finishing write tid 5 to smithi06431743-27 2024-03-04T06:25:08.993 INFO:tasks.rados.rados.0.smithi064.stdout:3947: oid 27 updating version 0 to 836 2024-03-04T06:25:08.993 INFO:tasks.rados.rados.0.smithi064.stdout:3947: oid 27 version 836 is already newer than 835 2024-03-04T06:25:08.993 INFO:tasks.rados.rados.0.smithi064.stdout:3947: oid 27 version 836 is already newer than 835 2024-03-04T06:25:08.993 INFO:tasks.rados.rados.0.smithi064.stdout:3947: oid 27 version 836 is already newer than 833 2024-03-04T06:25:08.995 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 27 v 836 (ObjNum 1770 snap 522 seq_num 1770) dirty exists 2024-03-04T06:25:08.995 INFO:tasks.rados.rados.0.smithi064.stdout:3947: left oid 27 (ObjNum 1770 snap 522 seq_num 1770) 2024-03-04T06:25:08.995 INFO:tasks.rados.rados.0.smithi064.stdout:3954: expect (ObjNum 1598 snap 453 seq_num 1598) 2024-03-04T06:25:09.226 INFO:tasks.rados.rados.0.smithi064.stdout:3953: finishing write tid 2 to smithi06431743-40 2024-03-04T06:25:09.226 INFO:tasks.rados.rados.0.smithi064.stdout:3951: finishing write tid 2 to smithi06431743-7 2024-03-04T06:25:09.226 INFO:tasks.rados.rados.0.smithi064.stdout:3953: finishing write tid 3 to smithi06431743-40 2024-03-04T06:25:09.226 INFO:tasks.rados.rados.0.smithi064.stdout:3955: finishing write tid 2 to smithi06431743-4 2024-03-04T06:25:09.226 INFO:tasks.rados.rados.0.smithi064.stdout:3949: finishing write tid 3 to smithi06431743-15 2024-03-04T06:25:09.226 INFO:tasks.rados.rados.0.smithi064.stdout:3949: finishing write tid 4 to smithi06431743-15 2024-03-04T06:25:09.226 INFO:tasks.rados.rados.0.smithi064.stdout:3951: finishing write tid 3 to smithi06431743-7 2024-03-04T06:25:09.226 INFO:tasks.rados.rados.0.smithi064.stdout:3949: finishing write tid 5 to smithi06431743-15 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3949: oid 15 updating version 0 to 1015 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3949: oid 15 version 1015 is already newer than 1015 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3949: oid 15 version 1015 is already newer than 1012 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3949: oid 15 version 1015 is already newer than 1013 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 15 v 1015 (ObjNum 1771 snap 522 seq_num 1771) dirty exists 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3949: left oid 15 (ObjNum 1771 snap 522 seq_num 1771) 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3953: finishing write tid 4 to smithi06431743-40 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3953: finishing write tid 5 to smithi06431743-40 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3953: oid 40 updating version 0 to 989 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3953: oid 40 version 989 is already newer than 987 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3953: oid 40 updating version 989 to 990 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3953: oid 40 version 990 is already newer than 989 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 40 v 990 (ObjNum 1773 snap 522 seq_num 1773) dirty exists 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3953: left oid 40 (ObjNum 1773 snap 522 seq_num 1773) 2024-03-04T06:25:09.227 INFO:tasks.rados.rados.0.smithi064.stdout:3951: finishing write tid 4 to smithi06431743-7 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3955: finishing write tid 3 to smithi06431743-4 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3951: finishing write tid 5 to smithi06431743-7 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3951: finishing write tid 6 to smithi06431743-7 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3951: oid 7 updating version 0 to 748 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3951: oid 7 version 748 is already newer than 744 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3951: oid 7 version 748 is already newer than 746 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3951: oid 7 version 748 is already newer than 747 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3951: oid 7 version 748 is already newer than 745 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 7 v 748 (ObjNum 1772 snap 522 seq_num 1772) dirty exists 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3951: left oid 7 (ObjNum 1772 snap 522 seq_num 1772) 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3955: finishing write tid 4 to smithi06431743-4 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3955: finishing write tid 5 to smithi06431743-4 2024-03-04T06:25:09.228 INFO:tasks.rados.rados.0.smithi064.stdout:3955: oid 4 updating version 0 to 2401 2024-03-04T06:25:09.229 INFO:tasks.rados.rados.0.smithi064.stdout:3955: oid 4 version 2401 is already newer than 2399 2024-03-04T06:25:09.229 INFO:tasks.rados.rados.0.smithi064.stdout:3955: oid 4 version 2401 is already newer than 2398 2024-03-04T06:25:09.229 INFO:tasks.rados.rados.0.smithi064.stdout:3955: oid 4 version 2401 is already newer than 2400 2024-03-04T06:25:09.229 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 2401 (ObjNum 1774 snap 522 seq_num 1774) dirty exists 2024-03-04T06:25:09.229 INFO:tasks.rados.rados.0.smithi064.stdout:3955: left oid 4 (ObjNum 1774 snap 522 seq_num 1774) 2024-03-04T06:25:09.431 INFO:tasks.rados.rados.0.smithi064.stdout:3947: done (9 left) 2024-03-04T06:25:09.431 INFO:tasks.rados.rados.0.smithi064.stdout:3948: done (8 left) 2024-03-04T06:25:09.431 INFO:tasks.rados.rados.0.smithi064.stdout:3949: done (7 left) 2024-03-04T06:25:09.432 INFO:tasks.rados.rados.0.smithi064.stdout:3950: done (6 left) 2024-03-04T06:25:09.432 INFO:tasks.rados.rados.0.smithi064.stdout:3951: done (5 left) 2024-03-04T06:25:09.432 INFO:tasks.rados.rados.0.smithi064.stdout:3952: done (4 left) 2024-03-04T06:25:09.432 INFO:tasks.rados.rados.0.smithi064.stdout:3953: done (3 left) 2024-03-04T06:25:09.432 INFO:tasks.rados.rados.0.smithi064.stdout:3954: done (2 left) 2024-03-04T06:25:09.432 INFO:tasks.rados.rados.0.smithi064.stdout:3955: done (1 left) 2024-03-04T06:25:09.433 INFO:tasks.rados.rados.0.smithi064.stdout:3956: done (0 left) 2024-03-04T06:25:09.433 INFO:tasks.rados.rados.0.smithi064.stdout:3957: write (excl) oid 37 current snap is 523 2024-03-04T06:25:09.433 INFO:tasks.rados.rados.0.smithi064.stdout:3957: seq_num 1775 ranges {632425=677224,2018597=504292,3027224=552764} 2024-03-04T06:25:09.438 INFO:tasks.rados.rados.0.smithi064.stdout:3957: writing smithi06431743-37 from 632425 to 1309649 tid 1 2024-03-04T06:25:09.446 INFO:tasks.rados.rados.0.smithi064.stdout:3957: writing smithi06431743-37 from 2018597 to 2522889 tid 2 2024-03-04T06:25:09.453 INFO:tasks.rados.rados.0.smithi064.stdout:3957: writing smithi06431743-37 from 3027224 to 3579988 tid 3 2024-03-04T06:25:09.453 INFO:tasks.rados.rados.0.smithi064.stdout:3958: snap_remove snap 497 2024-03-04T06:25:10.223 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:25:10.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:10.331+0000 7f0f4ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:10.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:10.335+0000 7f0f4ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:10.437 INFO:tasks.rados.rados.0.smithi064.stdout:3958: done (1 left) 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:3959: delete oid 12 current snap is 523 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:3957: finishing write tid 1 to smithi06431743-37 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:3957: finishing write tid 2 to smithi06431743-37 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:3957: finishing write tid 3 to smithi06431743-37 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:3957: finishing write tid 4 to smithi06431743-37 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:3957: finishing write tid 5 to smithi06431743-37 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:3957: oid 37 updating version 0 to 840 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:3957: oid 37 version 840 is already newer than 840 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:3957: oid 37 updating version 840 to 841 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:3957: oid 37 version 841 is already newer than 838 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 37 v 841 (ObjNum 1775 snap 523 seq_num 1775) dirty exists 2024-03-04T06:25:10.438 INFO:tasks.rados.rados.0.smithi064.stdout:3957: left oid 37 (ObjNum 1775 snap 523 seq_num 1775) 2024-03-04T06:25:10.555 INFO:tasks.rados.rados.0.smithi064.stdout:3957: done (1 left) 2024-03-04T06:25:10.555 INFO:tasks.rados.rados.0.smithi064.stdout:3959: done (0 left) 2024-03-04T06:25:10.555 INFO:tasks.rados.rados.0.smithi064.stdout:3960: write oid 22 current snap is 523 2024-03-04T06:25:10.555 INFO:tasks.rados.rados.0.smithi064.stdout:3960: seq_num 1777 ranges {666039=427722,1619208=507961,2565717=691661,3660365=205674} 2024-03-04T06:25:10.562 INFO:tasks.rados.rados.0.smithi064.stdout:3960: writing smithi06431743-22 from 666039 to 1093761 tid 1 2024-03-04T06:25:10.568 INFO:tasks.rados.rados.0.smithi064.stdout:3960: writing smithi06431743-22 from 1619208 to 2127169 tid 2 2024-03-04T06:25:10.574 INFO:tasks.rados.rados.0.smithi064.stdout:3960: writing smithi06431743-22 from 2565717 to 3257378 tid 3 2024-03-04T06:25:10.578 INFO:tasks.rados.rados.0.smithi064.stdout:3960: writing smithi06431743-22 from 3660365 to 3866039 tid 4 2024-03-04T06:25:10.578 INFO:tasks.rados.rados.0.smithi064.stdout:3961: snap_remove snap 510 2024-03-04T06:25:11.440 INFO:tasks.rados.rados.0.smithi064.stdout:3961: done (1 left) 2024-03-04T06:25:11.440 INFO:tasks.rados.rados.0.smithi064.stdout:3962: write (excl) oid 3 current snap is 523 2024-03-04T06:25:11.441 INFO:tasks.rados.rados.0.smithi064.stdout:3960: finishing write tid 1 to smithi06431743-22 2024-03-04T06:25:11.441 INFO:tasks.rados.rados.0.smithi064.stdout:3962: seq_num 1778 ranges {682846=502971,1677690=651619,2993140=502933} 2024-03-04T06:25:11.447 INFO:tasks.rados.rados.0.smithi064.stdout:3962: writing smithi06431743-3 from 682846 to 1185817 tid 1 2024-03-04T06:25:11.454 INFO:tasks.rados.rados.0.smithi064.stdout:3962: writing smithi06431743-3 from 1677690 to 2329309 tid 2 2024-03-04T06:25:11.462 INFO:tasks.rados.rados.0.smithi064.stdout:3962: writing smithi06431743-3 from 2993140 to 3496073 tid 3 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3963: read oid 39 snap 500 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3963: expect (ObjNum 1687 snap 490 seq_num 1687) 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3964: read oid 29 snap 520 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3964: expect (ObjNum 1747 snap 509 seq_num 1747) 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3965: read oid 1 snap -1 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3965: expect (ObjNum 1661 snap 477 seq_num 1661) 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3966: read oid 32 snap -1 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3966: expect (ObjNum 1670 snap 482 seq_num 1670) 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3967: snap_create 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3960: finishing write tid 2 to smithi06431743-22 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3960: finishing write tid 3 to smithi06431743-22 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3960: finishing write tid 4 to smithi06431743-22 2024-03-04T06:25:11.463 INFO:tasks.rados.rados.0.smithi064.stdout:3960: finishing write tid 5 to smithi06431743-22 2024-03-04T06:25:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:3960: finishing write tid 6 to smithi06431743-22 2024-03-04T06:25:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:3960: oid 22 updating version 0 to 1460 2024-03-04T06:25:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:3960: oid 22 version 1460 is already newer than 1456 2024-03-04T06:25:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:3960: oid 22 version 1460 is already newer than 1458 2024-03-04T06:25:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:3960: oid 22 version 1460 is already newer than 1457 2024-03-04T06:25:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:3960: oid 22 version 1460 is already newer than 1459 2024-03-04T06:25:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 22 v 1460 (ObjNum 1777 snap 523 seq_num 1777) dirty exists 2024-03-04T06:25:11.464 INFO:tasks.rados.rados.0.smithi064.stdout:3960: left oid 22 (ObjNum 1777 snap 523 seq_num 1777) 2024-03-04T06:25:11.499 INFO:tasks.rados.rados.0.smithi064.stdout:3963: expect (ObjNum 1687 snap 490 seq_num 1687) 2024-03-04T06:25:11.599 INFO:tasks.rados.rados.0.smithi064.stdout:3962: finishing write tid 1 to smithi06431743-3 2024-03-04T06:25:11.599 INFO:tasks.rados.rados.0.smithi064.stdout:3964: expect (ObjNum 1747 snap 509 seq_num 1747) 2024-03-04T06:25:11.722 INFO:tasks.rados.rados.0.smithi064.stdout:3965: expect (ObjNum 1661 snap 477 seq_num 1661) 2024-03-04T06:25:11.968 INFO:tasks.rados.rados.0.smithi064.stdout:3966: expect (ObjNum 1670 snap 482 seq_num 1670) 2024-03-04T06:25:12.104 INFO:tasks.rados.rados.0.smithi064.stdout:3962: finishing write tid 2 to smithi06431743-3 2024-03-04T06:25:12.105 INFO:tasks.rados.rados.0.smithi064.stdout:3962: finishing write tid 3 to smithi06431743-3 2024-03-04T06:25:12.105 INFO:tasks.rados.rados.0.smithi064.stdout:3962: finishing write tid 4 to smithi06431743-3 2024-03-04T06:25:12.105 INFO:tasks.rados.rados.0.smithi064.stdout:3962: finishing write tid 5 to smithi06431743-3 2024-03-04T06:25:12.105 INFO:tasks.rados.rados.0.smithi064.stdout:3962: oid 3 updating version 0 to 1073 2024-03-04T06:25:12.105 INFO:tasks.rados.rados.0.smithi064.stdout:3962: oid 3 version 1073 is already newer than 1071 2024-03-04T06:25:12.105 INFO:tasks.rados.rados.0.smithi064.stdout:3962: oid 3 version 1073 is already newer than 1073 2024-03-04T06:25:12.105 INFO:tasks.rados.rados.0.smithi064.stdout:3962: oid 3 version 1073 is already newer than 1070 2024-03-04T06:25:12.105 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 3 v 1073 (ObjNum 1778 snap 523 seq_num 1778) dirty exists 2024-03-04T06:25:12.105 INFO:tasks.rados.rados.0.smithi064.stdout:3962: left oid 3 (ObjNum 1778 snap 523 seq_num 1778) 2024-03-04T06:25:12.405 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:25:12.446 INFO:tasks.rados.rados.0.smithi064.stdout:3960: done (6 left) 2024-03-04T06:25:12.447 INFO:tasks.rados.rados.0.smithi064.stdout:3962: done (5 left) 2024-03-04T06:25:12.447 INFO:tasks.rados.rados.0.smithi064.stdout:3963: done (4 left) 2024-03-04T06:25:12.447 INFO:tasks.rados.rados.0.smithi064.stdout:3964: done (3 left) 2024-03-04T06:25:12.447 INFO:tasks.rados.rados.0.smithi064.stdout:3965: done (2 left) 2024-03-04T06:25:12.447 INFO:tasks.rados.rados.0.smithi064.stdout:3966: done (1 left) 2024-03-04T06:25:12.447 INFO:tasks.rados.rados.0.smithi064.stdout:3967: done (0 left) 2024-03-04T06:25:12.447 INFO:tasks.rados.rados.0.smithi064.stdout:3968: snap_remove snap 501 2024-03-04T06:25:12.513 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:12.511+0000 7fc77f75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:12.515 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:12.515+0000 7fc77f75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:12.654 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:25:12.655 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:25:12.655 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:25:12.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:25:12.764 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:12.759+0000 7fa06339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:12.767 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:12.766+0000 7fa06339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:25:12.779 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:25:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:25:12.782 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:25:12.782 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:25:12.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:12.885+0000 7fa608aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:12.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:12.892+0000 7fa608aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:12.904 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:25:12.904 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:25:12.904 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:25:12.905 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:25:12.905 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:25:12.905 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:25:12.905 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:25:12.905 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:25:12.905 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:25:12.905 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:25:12.905 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:25:12.905 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:25:12.907 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:25:12.907 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:25:12.908 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:25:13.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:13.011+0000 7eff42a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:13.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:13.018+0000 7eff42a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:13.031 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:25:13.032 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:25:13.032 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:25:13.032 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:25:13.032 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:25:13.032 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:25:13.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:25:13.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:25:13.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:25:13.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:25:13.032 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:25:13.032 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:25:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:25:13.035 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:25:13.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:25:13.143 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:13.138+0000 7f9916c0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:13.146 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:13.146+0000 7f9916c0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:13.159 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:25:13.159 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:25:13.159 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:25:13.160 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:25:13.160 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:25:13.160 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:25:13.160 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:25:13.160 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:25:13.160 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:25:13.160 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:25:13.160 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:25:13.160 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:25:13.162 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:25:13.162 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:25:13.360 INFO:tasks.rados.rados.0.smithi064.stdout:3968: done (0 left) 2024-03-04T06:25:13.360 INFO:tasks.rados.rados.0.smithi064.stdout:3969: write oid 4 current snap is 524 2024-03-04T06:25:13.360 INFO:tasks.rados.rados.0.smithi064.stdout:3969: seq_num 1779 ranges {699653=694573,1852525=795277,3420563=479090} 2024-03-04T06:25:13.368 INFO:tasks.rados.rados.0.smithi064.stdout:3969: writing smithi06431743-4 from 699653 to 1394226 tid 1 2024-03-04T06:25:13.375 INFO:tasks.rados.rados.0.smithi064.stdout:3969: writing smithi06431743-4 from 1852525 to 2647802 tid 2 2024-03-04T06:25:13.382 INFO:tasks.rados.rados.0.smithi064.stdout:3969: writing smithi06431743-4 from 3420563 to 3899653 tid 3 2024-03-04T06:25:13.382 INFO:tasks.rados.rados.0.smithi064.stdout:3970: delete oid 29 current snap is 524 2024-03-04T06:25:13.495 INFO:tasks.rados.rados.0.smithi064.stdout:3969: finishing write tid 1 to smithi06431743-4 2024-03-04T06:25:13.511 INFO:tasks.rados.rados.0.smithi064.stdout:3969: finishing write tid 2 to smithi06431743-4 2024-03-04T06:25:13.519 INFO:tasks.rados.rados.0.smithi064.stdout:3969: finishing write tid 3 to smithi06431743-4 2024-03-04T06:25:13.535 INFO:tasks.rados.rados.0.smithi064.stdout:3969: finishing write tid 4 to smithi06431743-4 2024-03-04T06:25:13.538 INFO:tasks.rados.rados.0.smithi064.stdout:3969: finishing write tid 5 to smithi06431743-4 2024-03-04T06:25:13.538 INFO:tasks.rados.rados.0.smithi064.stdout:3969: oid 4 updating version 0 to 2415 2024-03-04T06:25:13.538 INFO:tasks.rados.rados.0.smithi064.stdout:3969: oid 4 updating version 2415 to 2417 2024-03-04T06:25:13.538 INFO:tasks.rados.rados.0.smithi064.stdout:3969: oid 4 version 2417 is already newer than 2417 2024-03-04T06:25:13.538 INFO:tasks.rados.rados.0.smithi064.stdout:3969: oid 4 updating version 2417 to 2418 2024-03-04T06:25:13.538 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 4 v 2418 (ObjNum 1779 snap 524 seq_num 1779) dirty exists 2024-03-04T06:25:13.538 INFO:tasks.rados.rados.0.smithi064.stdout:3969: left oid 4 (ObjNum 1779 snap 524 seq_num 1779) 2024-03-04T06:25:13.559 INFO:tasks.rados.rados.0.smithi064.stdout:3969: done (1 left) 2024-03-04T06:25:13.559 INFO:tasks.rados.rados.0.smithi064.stdout:3970: done (0 left) 2024-03-04T06:25:13.559 INFO:tasks.rados.rados.0.smithi064.stdout:3971: read oid 43 snap -1 2024-03-04T06:25:13.559 INFO:tasks.rados.rados.0.smithi064.stdout:3971: expect deleted 2024-03-04T06:25:13.559 INFO:tasks.rados.rados.0.smithi064.stdout:3972: snap_create 2024-03-04T06:25:14.451 INFO:tasks.rados.rados.0.smithi064.stdout:3971: done (1 left) 2024-03-04T06:25:14.452 INFO:tasks.rados.rados.0.smithi064.stdout:3972: done (0 left) 2024-03-04T06:25:14.452 INFO:tasks.rados.rados.0.smithi064.stdout:3973: read oid 3 snap -1 2024-03-04T06:25:14.452 INFO:tasks.rados.rados.0.smithi064.stdout:3973: expect (ObjNum 1778 snap 523 seq_num 1778) 2024-03-04T06:25:14.452 INFO:tasks.rados.rados.0.smithi064.stdout:3974: read oid 11 snap -1 2024-03-04T06:25:14.452 INFO:tasks.rados.rados.0.smithi064.stdout:3974: expect (ObjNum 1762 snap 518 seq_num 1762) 2024-03-04T06:25:14.452 INFO:tasks.rados.rados.0.smithi064.stdout:3975: read oid 2 snap -1 2024-03-04T06:25:14.452 INFO:tasks.rados.rados.0.smithi064.stdout:3975: expect (ObjNum 1729 snap 502 seq_num 1729) 2024-03-04T06:25:14.452 INFO:tasks.rados.rados.0.smithi064.stdout:3976: read oid 6 snap -1 2024-03-04T06:25:14.452 INFO:tasks.rados.rados.0.smithi064.stdout:3976: expect (ObjNum 1757 snap 515 seq_num 1757) 2024-03-04T06:25:14.452 INFO:tasks.rados.rados.0.smithi064.stdout:3977: delete oid 12 current snap is 525 2024-03-04T06:25:14.497 INFO:tasks.rados.rados.0.smithi064.stdout:3975: expect (ObjNum 1729 snap 502 seq_num 1729) 2024-03-04T06:25:14.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:25:14.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:14.688+0000 7f1e4395d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:14.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:14.693+0000 7f1e4395d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:14.769 INFO:tasks.rados.rados.0.smithi064.stdout:3973: expect (ObjNum 1778 snap 523 seq_num 1778) 2024-03-04T06:25:15.022 INFO:tasks.rados.rados.0.smithi064.stdout:3976: expect (ObjNum 1757 snap 515 seq_num 1757) 2024-03-04T06:25:15.201 INFO:tasks.rados.rados.0.smithi064.stdout:3974: expect (ObjNum 1762 snap 518 seq_num 1762) 2024-03-04T06:25:15.372 INFO:tasks.rados.rados.0.smithi064.stdout:3973: done (4 left) 2024-03-04T06:25:15.372 INFO:tasks.rados.rados.0.smithi064.stdout:3974: done (3 left) 2024-03-04T06:25:15.372 INFO:tasks.rados.rados.0.smithi064.stdout:3975: done (2 left) 2024-03-04T06:25:15.372 INFO:tasks.rados.rados.0.smithi064.stdout:3976: done (1 left) 2024-03-04T06:25:15.372 INFO:tasks.rados.rados.0.smithi064.stdout:3977: done (0 left) 2024-03-04T06:25:15.372 INFO:tasks.rados.rados.0.smithi064.stdout:3978: snap_create 2024-03-04T06:25:15.454 INFO:tasks.rados.rados.0.smithi064.stdout:3978: done (0 left) 2024-03-04T06:25:15.455 INFO:tasks.rados.rados.0.smithi064.stdout:3979: delete oid 35 current snap is 526 2024-03-04T06:25:15.511 INFO:tasks.rados.rados.0.smithi064.stdout:3979: done (0 left) 2024-03-04T06:25:15.511 INFO:tasks.rados.rados.0.smithi064.stdout:3980: snap_create 2024-03-04T06:25:16.455 INFO:tasks.rados.rados.0.smithi064.stdout:3980: done (0 left) 2024-03-04T06:25:16.455 INFO:tasks.rados.rados.0.smithi064.stdout:3981: delete oid 1 current snap is 527 2024-03-04T06:25:16.538 INFO:tasks.rados.rados.0.smithi064.stdout:3981: done (0 left) 2024-03-04T06:25:16.538 INFO:tasks.rados.rados.0.smithi064.stdout:3982: snap_create 2024-03-04T06:25:16.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:25:16.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:16.874+0000 7fec5751d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:16.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:16.879+0000 7fec5751d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:17.463 INFO:tasks.rados.rados.0.smithi064.stdout:3982: done (0 left) 2024-03-04T06:25:17.463 INFO:tasks.rados.rados.0.smithi064.stdout:3983: snap_remove snap 482 2024-03-04T06:25:18.164 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:25:18.164 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:25:18.164 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:18.286 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:18.281+0000 7f0c8fa64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:18.288 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:18.288+0000 7f0c8fa64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:18.463 INFO:tasks.rados.rados.0.smithi064.stdout:3983: done (0 left) 2024-03-04T06:25:18.463 INFO:tasks.rados.rados.0.smithi064.stdout:3984: write oid 28 current snap is 528 2024-03-04T06:25:18.463 INFO:tasks.rados.rados.0.smithi064.stdout:3984: seq_num 1784 ranges {783688=670818,1912229=662626,3041325=686918} 2024-03-04T06:25:18.473 INFO:tasks.rados.rados.0.smithi064.stdout:3984: writing smithi06431743-28 from 783688 to 1454506 tid 1 2024-03-04T06:25:18.481 INFO:tasks.rados.rados.0.smithi064.stdout:3984: writing smithi06431743-28 from 1912229 to 2574855 tid 2 2024-03-04T06:25:18.485 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:18.485 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:18.489 INFO:tasks.rados.rados.0.smithi064.stdout:3984: writing smithi06431743-28 from 3041325 to 3728243 tid 3 2024-03-04T06:25:18.489 INFO:tasks.rados.rados.0.smithi064.stdout:3985: write oid 45 current snap is 528 2024-03-04T06:25:18.489 INFO:tasks.rados.rados.0.smithi064.stdout:3985: seq_num 1785 ranges {400495=746067,1570711=406284} 2024-03-04T06:25:18.497 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1564,"stamp":"2024-03-04T06:25:18.462100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112757966,"num_objects":1304,"num_object_clones":1083,"num_object_copies":3912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":185,"num_read":7122,"num_read_kb":5622895,"num_write":6033,"num_write_kb":1614389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18261,"ondisk_log_size":18261,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":872668,"kb_used_data":626556,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376614692,"statfs":{"total":386547056640,"available":385653444608,"internally_reserved":0,"allocated":641593344,"data_stored":728360050,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7480402,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":3,"num_read":54,"num_read_kb":60230,"num_write":19,"num_write_kb":5337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012140"},"pg_stats":[{"pgid":"3.f","version":"1039'992","reported_seq":3524,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.467978+0000","last_change":"2024-03-04T06:25:13.412106+0000","last_active":"2024-03-04T06:25:14.467978+0000","last_peered":"2024-03-04T06:25:14.467978+0000","last_clean":"2024-03-04T06:25:14.467978+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:25:14.467978+0000","last_undegraded":"2024-03-04T06:25:14.467978+0000","last_fullsized":"2024-03-04T06:25:14.467978+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":992,"log_dups_size":0,"ondisk_log_size":992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798811,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":377,"num_read_kb":295546,"num_write":342,"num_write_kb":90141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"1042'849","reported_seq":3537,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-03-04T06:25:13.396665+0000","last_change":"2024-03-04T06:25:13.460303+0000","last_active":"2024-03-04T06:25:13.396665+0000","last_peered":"2024-03-04T06:25:13.396665+0000","last_clean":"2024-03-04T06:25:13.396665+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:25:13.396665+0000","last_undegraded":"2024-03-04T06:25:13.396665+0000","last_fullsized":"2024-03-04T06:25:13.396665+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7580815,"num_objects":67,"num_object_clones":49,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":265,"num_read_kb":210903,"num_write":277,"num_write_kb":73945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"1042'1379","reported_seq":3762,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.512438+0000","last_change":"2024-03-04T06:25:13.975738+0000","last_active":"2024-03-04T06:25:14.512438+0000","last_peered":"2024-03-04T06:25:14.512438+0000","last_clean":"2024-03-04T06:25:14.512438+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:25:14.512438+0000","last_undegraded":"2024-03-04T06:25:14.512438+0000","last_fullsized":"2024-03-04T06:25:14.512438+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1379,"log_dups_size":0,"ondisk_log_size":1379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11514478,"num_objects":101,"num_object_clones":86,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":637,"num_read_kb":520483,"num_write":465,"num_write_kb":127129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"1042'1142","reported_seq":3515,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.495282+0000","last_change":"2024-03-04T06:25:14.471023+0000","last_active":"2024-03-04T06:25:14.495282+0000","last_peered":"2024-03-04T06:25:14.495282+0000","last_clean":"2024-03-04T06:25:14.495282+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:25:14.495282+0000","last_undegraded":"2024-03-04T06:25:14.495282+0000","last_fullsized":"2024-03-04T06:25:14.495282+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1142,"log_dups_size":0,"ondisk_log_size":1142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6622816,"num_objects":72,"num_object_clones":62,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":355,"num_read_kb":271721,"num_write":341,"num_write_kb":94155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"1029'441","reported_seq":3296,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-03-04T06:25:15.481953+0000","last_change":"2024-03-04T06:25:13.919083+0000","last_active":"2024-03-04T06:25:15.481953+0000","last_peered":"2024-03-04T06:25:15.481953+0000","last_clean":"2024-03-04T06:25:15.481953+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:25:15.481953+0000","last_undegraded":"2024-03-04T06:25:15.481953+0000","last_fullsized":"2024-03-04T06:25:15.481953+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3546706,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":124,"num_read_kb":93149,"num_write":131,"num_write_kb":36421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"1042'1108","reported_seq":3589,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.471452+0000","last_change":"2024-03-04T06:25:14.474456+0000","last_active":"2024-03-04T06:25:14.471452+0000","last_peered":"2024-03-04T06:25:14.471452+0000","last_clean":"2024-03-04T06:25:14.471452+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:25:14.471452+0000","last_undegraded":"2024-03-04T06:25:14.471452+0000","last_fullsized":"2024-03-04T06:25:14.471452+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1108,"log_dups_size":0,"ondisk_log_size":1108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092917,"num_objects":78,"num_object_clones":66,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":463,"num_read_kb":375458,"num_write":370,"num_write_kb":106221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":3048,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.472190+0000","last_change":"2024-03-04T06:25:13.411932+0000","last_active":"2024-03-04T06:25:14.472190+0000","last_peered":"2024-03-04T06:25:14.472190+0000","last_clean":"2024-03-04T06:25:14.472190+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:25:14.472190+0000","last_undegraded":"2024-03-04T06:25:14.472190+0000","last_fullsized":"2024-03-04T06:25:14.472190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"1042'736","reported_seq":3380,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.462421+0000","last_change":"2024-03-04T06:25:13.976012+0000","last_active":"2024-03-04T06:25:14.462421+0000","last_peered":"2024-03-04T06:25:14.462421+0000","last_clean":"2024-03-04T06:25:14.462421+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:25:14.462421+0000","last_undegraded":"2024-03-04T06:25:14.462421+0000","last_fullsized":"2024-03-04T06:25:14.462421+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3697969,"num_objects":47,"num_object_clones":38,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":291,"num_read_kb":255611,"num_write":208,"num_write_kb":57031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2767,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-03-04T06:25:15.469874+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:25:15.469874+0000","last_peered":"2024-03-04T06:25:15.469874+0000","last_clean":"2024-03-04T06:25:15.469874+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:15.469874+0000","last_undegraded":"2024-03-04T06:25:15.469874+0000","last_fullsized":"2024-03-04T06:25:15.469874+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2573,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.474056+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:25:14.474056+0000","last_peered":"2024-03-04T06:25:14.474056+0000","last_clean":"2024-03-04T06:25:14.474056+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:14.474056+0000","last_undegraded":"2024-03-04T06:25:14.474056+0000","last_fullsized":"2024-03-04T06:25:14.474056+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"1042'1077","reported_seq":3596,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.497255+0000","last_change":"2024-03-04T06:25:13.981889+0000","last_active":"2024-03-04T06:25:14.497255+0000","last_peered":"2024-03-04T06:25:14.497255+0000","last_clean":"2024-03-04T06:25:14.497255+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:25:14.497255+0000","last_undegraded":"2024-03-04T06:25:14.497255+0000","last_fullsized":"2024-03-04T06:25:14.497255+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1077,"log_dups_size":0,"ondisk_log_size":1077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10203136,"num_objects":78,"num_object_clones":64,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":472,"num_read_kb":389167,"num_write":355,"num_write_kb":96087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2559,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.467362+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:14.467362+0000","last_peered":"2024-03-04T06:25:14.467362+0000","last_clean":"2024-03-04T06:25:14.467362+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:14.467362+0000","last_undegraded":"2024-03-04T06:25:14.467362+0000","last_fullsized":"2024-03-04T06:25:14.467362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"1042'754","reported_seq":3490,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-03-04T06:25:15.468283+0000","last_change":"2024-03-04T06:25:13.976622+0000","last_active":"2024-03-04T06:25:15.468283+0000","last_peered":"2024-03-04T06:25:15.468283+0000","last_clean":"2024-03-04T06:25:15.468283+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:25:15.468283+0000","last_undegraded":"2024-03-04T06:25:15.468283+0000","last_fullsized":"2024-03-04T06:25:15.468283+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6891728,"num_objects":55,"num_object_clones":46,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":335,"num_read_kb":275603,"num_write":239,"num_write_kb":68723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2573,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.465068+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:25:14.465068+0000","last_peered":"2024-03-04T06:25:14.465068+0000","last_clean":"2024-03-04T06:25:14.465068+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:25:14.465068+0000","last_undegraded":"2024-03-04T06:25:14.465068+0000","last_fullsized":"2024-03-04T06:25:14.465068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"1040'2030","reported_seq":4145,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-03-04T06:25:15.486096+0000","last_change":"2024-03-04T06:25:13.919649+0000","last_active":"2024-03-04T06:25:15.486096+0000","last_peered":"2024-03-04T06:25:15.486096+0000","last_clean":"2024-03-04T06:25:15.486096+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:25:15.486096+0000","last_undegraded":"2024-03-04T06:25:15.486096+0000","last_fullsized":"2024-03-04T06:25:15.486096+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2030,"log_dups_size":0,"ondisk_log_size":2030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14893081,"num_objects":150,"num_object_clones":123,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":22,"num_read":868,"num_read_kb":695791,"num_write":698,"num_write_kb":188207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2559,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.467283+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:14.467283+0000","last_peered":"2024-03-04T06:25:14.467283+0000","last_clean":"2024-03-04T06:25:14.467283+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:14.467283+0000","last_undegraded":"2024-03-04T06:25:14.467283+0000","last_fullsized":"2024-03-04T06:25:14.467283+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"1042'2420","reported_seq":4185,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-03-04T06:25:13.538006+0000","last_change":"2024-03-04T06:25:13.555241+0000","last_active":"2024-03-04T06:25:13.538006+0000","last_peered":"2024-03-04T06:25:13.538006+0000","last_clean":"2024-03-04T06:25:13.538006+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:25:13.538006+0000","last_undegraded":"2024-03-04T06:25:13.538006+0000","last_fullsized":"2024-03-04T06:25:13.538006+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2420,"log_dups_size":0,"ondisk_log_size":2420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9430014,"num_objects":167,"num_object_clones":143,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":771,"num_read_kb":571096,"num_write":762,"num_write_kb":206699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"1042'1019","reported_seq":3583,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.504496+0000","last_change":"2024-03-04T06:25:13.979223+0000","last_active":"2024-03-04T06:25:14.504496+0000","last_peered":"2024-03-04T06:25:14.504496+0000","last_clean":"2024-03-04T06:25:14.504496+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:25:14.504496+0000","last_undegraded":"2024-03-04T06:25:14.504496+0000","last_fullsized":"2024-03-04T06:25:14.504496+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1019,"log_dups_size":0,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7295096,"num_objects":72,"num_object_clones":59,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":466,"num_read_kb":391842,"num_write":343,"num_write_kb":93831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2680,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-03-04T06:25:13.407279+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:25:13.407279+0000","last_peered":"2024-03-04T06:25:13.407279+0000","last_clean":"2024-03-04T06:25:13.407279+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:25:13.407279+0000","last_undegraded":"2024-03-04T06:25:13.407279+0000","last_fullsized":"2024-03-04T06:25:13.407279+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"1042'1197","reported_seq":3691,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-03-04T06:25:13.404427+0000","last_change":"2024-03-04T06:25:13.461854+0000","last_active":"2024-03-04T06:25:13.404427+0000","last_peered":"2024-03-04T06:25:13.404427+0000","last_clean":"2024-03-04T06:25:13.404427+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:25:13.404427+0000","last_undegraded":"2024-03-04T06:25:13.404427+0000","last_fullsized":"2024-03-04T06:25:13.404427+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1197,"log_dups_size":0,"ondisk_log_size":1197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924847,"num_objects":88,"num_object_clones":71,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":418,"num_read_kb":317414,"num_write":378,"num_write_kb":101479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2634,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-03-04T06:25:15.481054+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:25:15.481054+0000","last_peered":"2024-03-04T06:25:15.481054+0000","last_clean":"2024-03-04T06:25:15.481054+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:15.481054+0000","last_undegraded":"2024-03-04T06:25:15.481054+0000","last_fullsized":"2024-03-04T06:25:15.481054+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"1042'1554","reported_seq":3807,"reported_epoch":1044,"state":"active+clean","last_fresh":"2024-03-04T06:25:15.472864+0000","last_change":"2024-03-04T06:25:13.929865+0000","last_active":"2024-03-04T06:25:15.472864+0000","last_peered":"2024-03-04T06:25:15.472864+0000","last_clean":"2024-03-04T06:25:15.472864+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:25:15.472864+0000","last_undegraded":"2024-03-04T06:25:15.472864+0000","last_fullsized":"2024-03-04T06:25:15.472864+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1554,"log_dups_size":0,"ondisk_log_size":1554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10169522,"num_objects":107,"num_object_clones":90,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":547,"num_read_kb":436376,"num_write":494,"num_write_kb":137381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2680,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-03-04T06:25:13.401075+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:13.401075+0000","last_peered":"2024-03-04T06:25:13.401075+0000","last_clean":"2024-03-04T06:25:13.401075+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:13.401075+0000","last_undegraded":"2024-03-04T06:25:13.401075+0000","last_fullsized":"2024-03-04T06:25:13.401075+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"1042'1466","reported_seq":3899,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-03-04T06:25:13.386035+0000","last_change":"2024-03-04T06:25:13.457091+0000","last_active":"2024-03-04T06:25:13.386035+0000","last_peered":"2024-03-04T06:25:13.386035+0000","last_clean":"2024-03-04T06:25:13.386035+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:25:13.386035+0000","last_undegraded":"2024-03-04T06:25:13.386035+0000","last_fullsized":"2024-03-04T06:25:13.386035+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10505662,"num_objects":110,"num_object_clones":95,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":630,"num_read_kb":522644,"num_write":493,"num_write_kb":135045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2573,"reported_epoch":1043,"state":"active+clean","last_fresh":"2024-03-04T06:25:14.475516+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:25:14.475516+0000","last_peered":"2024-03-04T06:25:14.475516+0000","last_clean":"2024-03-04T06:25:14.475516+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:14.475516+0000","last_undegraded":"2024-03-04T06:25:14.475516+0000","last_fullsized":"2024-03-04T06:25:14.475516+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":112167598,"num_objects":1302,"num_object_clones":1083,"num_object_copies":3906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":185,"num_read":7019,"num_read_kb":5622804,"num_write":5896,"num_write_kb":1612495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18164,"ondisk_log_size":18164,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706714,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":274268,"kb_used_data":198868,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94097572,"statfs":{"total":96636764160,"available":96355913728,"internally_reserved":0,"allocated":203640832,"data_stored":231785419,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706714,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":212192,"kb_used_data":149272,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94159648,"statfs":{"total":96636764160,"available":96419479552,"internally_reserved":0,"allocated":152854528,"data_stored":174757860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":206780,"kb_used_data":143860,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94165060,"statfs":{"total":96636764160,"available":96425021440,"internally_reserved":0,"allocated":147312640,"data_stored":164833237,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":179428,"kb_used_data":134556,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94192412,"statfs":{"total":96636764160,"available":96453029888,"internally_reserved":0,"allocated":137785344,"data_stored":156983534,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:18.499 INFO:tasks.rados.rados.0.smithi064.stdout:3985: writing smithi06431743-45 from 400495 to 1146562 tid 1 2024-03-04T06:25:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3985: writing smithi06431743-45 from 1570711 to 1976995 tid 2 2024-03-04T06:25:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3986: read oid 3 snap -1 2024-03-04T06:25:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3986: expect (ObjNum 1778 snap 523 seq_num 1778) 2024-03-04T06:25:18.504 INFO:tasks.rados.rados.0.smithi064.stdout:3987: snap_create 2024-03-04T06:25:18.579 INFO:tasks.rados.rados.0.smithi064.stdout:3986: expect (ObjNum 1778 snap 523 seq_num 1778) 2024-03-04T06:25:18.848 INFO:tasks.rados.rados.0.smithi064.stdout:3984: finishing write tid 1 to smithi06431743-28 2024-03-04T06:25:18.848 INFO:tasks.rados.rados.0.smithi064.stdout:3985: finishing write tid 1 to smithi06431743-45 2024-03-04T06:25:18.848 INFO:tasks.rados.rados.0.smithi064.stdout:3984: finishing write tid 2 to smithi06431743-28 2024-03-04T06:25:18.848 INFO:tasks.rados.rados.0.smithi064.stdout:3985: finishing write tid 2 to smithi06431743-45 2024-03-04T06:25:18.848 INFO:tasks.rados.rados.0.smithi064.stdout:3985: finishing write tid 3 to smithi06431743-45 2024-03-04T06:25:18.848 INFO:tasks.rados.rados.0.smithi064.stdout:3985: finishing write tid 4 to smithi06431743-45 2024-03-04T06:25:18.848 INFO:tasks.rados.rados.0.smithi064.stdout:3985: oid 45 updating version 0 to 1385 2024-03-04T06:25:18.848 INFO:tasks.rados.rados.0.smithi064.stdout:3985: oid 45 version 1385 is already newer than 1383 2024-03-04T06:25:18.848 INFO:tasks.rados.rados.0.smithi064.stdout:3985: oid 45 version 1385 is already newer than 1385 2024-03-04T06:25:18.848 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 45 v 1385 (ObjNum 1785 snap 528 seq_num 1785) dirty exists 2024-03-04T06:25:18.848 INFO:tasks.rados.rados.0.smithi064.stdout:3985: left oid 45 (ObjNum 1785 snap 528 seq_num 1785) 2024-03-04T06:25:18.849 INFO:tasks.rados.rados.0.smithi064.stdout:3984: finishing write tid 3 to smithi06431743-28 2024-03-04T06:25:18.849 INFO:tasks.rados.rados.0.smithi064.stdout:3984: finishing write tid 4 to smithi06431743-28 2024-03-04T06:25:18.849 INFO:tasks.rados.rados.0.smithi064.stdout:3984: finishing write tid 5 to smithi06431743-28 2024-03-04T06:25:18.849 INFO:tasks.rados.rados.0.smithi064.stdout:3984: oid 28 updating version 0 to 2038 2024-03-04T06:25:18.849 INFO:tasks.rados.rados.0.smithi064.stdout:3984: oid 28 version 2038 is already newer than 2037 2024-03-04T06:25:18.849 INFO:tasks.rados.rados.0.smithi064.stdout:3984: oid 28 version 2038 is already newer than 2036 2024-03-04T06:25:18.849 INFO:tasks.rados.rados.0.smithi064.stdout:3984: oid 28 updating version 2038 to 2039 2024-03-04T06:25:18.849 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 28 v 2039 (ObjNum 1784 snap 528 seq_num 1784) dirty exists 2024-03-04T06:25:18.849 INFO:tasks.rados.rados.0.smithi064.stdout:3984: left oid 28 (ObjNum 1784 snap 528 seq_num 1784) 2024-03-04T06:25:18.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:25:19.055 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:19.054+0000 7f1d62a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:19.058 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:19.058+0000 7f1d62a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:19.471 INFO:tasks.rados.rados.0.smithi064.stdout:3984: done (3 left) 2024-03-04T06:25:19.471 INFO:tasks.rados.rados.0.smithi064.stdout:3985: done (2 left) 2024-03-04T06:25:19.471 INFO:tasks.rados.rados.0.smithi064.stdout:3986: done (1 left) 2024-03-04T06:25:19.471 INFO:tasks.rados.rados.0.smithi064.stdout:3987: done (0 left) 2024-03-04T06:25:19.471 INFO:tasks.rados.rados.0.smithi064.stdout:3988: delete oid 44 current snap is 529 2024-03-04T06:25:19.497 INFO:tasks.rados.rados.0.smithi064.stdout:3988: done (0 left) 2024-03-04T06:25:19.498 INFO:tasks.rados.rados.0.smithi064.stdout:3989: read oid 34 snap -1 2024-03-04T06:25:19.498 INFO:tasks.rados.rados.0.smithi064.stdout:3989: expect (ObjNum 1564 snap 443 seq_num 1564) 2024-03-04T06:25:19.498 INFO:tasks.rados.rados.0.smithi064.stdout:3990: delete oid 41 current snap is 529 2024-03-04T06:25:19.535 INFO:tasks.rados.rados.0.smithi064.stdout:3990: done (1 left) 2024-03-04T06:25:19.536 INFO:tasks.rados.rados.0.smithi064.stdout:3991: read oid 18 snap -1 2024-03-04T06:25:19.536 INFO:tasks.rados.rados.0.smithi064.stdout:3991: expect deleted 2024-03-04T06:25:19.536 INFO:tasks.rados.rados.0.smithi064.stdout:3992: read oid 21 snap -1 2024-03-04T06:25:19.536 INFO:tasks.rados.rados.0.smithi064.stdout:3992: expect deleted 2024-03-04T06:25:19.536 INFO:tasks.rados.rados.0.smithi064.stdout:3993: snap_remove snap 515 2024-03-04T06:25:20.472 INFO:tasks.rados.rados.0.smithi064.stdout:3993: done (3 left) 2024-03-04T06:25:20.472 INFO:tasks.rados.rados.0.smithi064.stdout:3994: delete oid 4 current snap is 529 2024-03-04T06:25:20.472 INFO:tasks.rados.rados.0.smithi064.stdout:3989: expect (ObjNum 1564 snap 443 seq_num 1564) 2024-03-04T06:25:20.590 INFO:tasks.rados.rados.0.smithi064.stdout:3989: done (3 left) 2024-03-04T06:25:20.590 INFO:tasks.rados.rados.0.smithi064.stdout:3991: done (2 left) 2024-03-04T06:25:20.590 INFO:tasks.rados.rados.0.smithi064.stdout:3992: done (1 left) 2024-03-04T06:25:20.590 INFO:tasks.rados.rados.0.smithi064.stdout:3994: done (0 left) 2024-03-04T06:25:20.590 INFO:tasks.rados.rados.0.smithi064.stdout:3995: snap_remove snap 527 2024-03-04T06:25:21.127 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T06:25:21.246 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:21.244+0000 7f25f179d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:21.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:21.249+0000 7f25f179d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:21.491 INFO:tasks.rados.rados.0.smithi064.stdout:3995: done (0 left) 2024-03-04T06:25:21.492 INFO:tasks.rados.rados.0.smithi064.stdout:3996: snap_remove snap 483 2024-03-04T06:25:22.494 INFO:tasks.rados.rados.0.smithi064.stdout:3996: done (0 left) 2024-03-04T06:25:22.495 INFO:tasks.rados.rados.0.smithi064.stdout:3997: read oid 41 snap -1 2024-03-04T06:25:22.495 INFO:tasks.rados.rados.0.smithi064.stdout:3997: expect deleted 2024-03-04T06:25:22.495 INFO:tasks.rados.rados.0.smithi064.stdout:3998: write (excl) oid 14 current snap is 529 2024-03-04T06:25:22.495 INFO:tasks.rados.rados.0.smithi064.stdout:3998: seq_num 1789 ranges {467723=763416,1804639=263084} 2024-03-04T06:25:22.502 INFO:tasks.rados.rados.0.smithi064.stdout:3998: writing smithi06431743-14 from 467723 to 1231139 tid 1 2024-03-04T06:25:22.508 INFO:tasks.rados.rados.0.smithi064.stdout:3998: writing smithi06431743-14 from 1804639 to 2067723 tid 2 2024-03-04T06:25:22.508 INFO:tasks.rados.rados.0.smithi064.stdout:3999: read oid 1 snap -1 2024-03-04T06:25:22.508 INFO:tasks.rados.rados.0.smithi064.stdout:3999: expect deleted 2024-03-04T06:25:22.508 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 3 2024-03-04T06:25:22.542 INFO:tasks.rados.rados.0.smithi064.stdout:3997: done (2 left) 2024-03-04T06:25:22.542 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 2 2024-03-04T06:25:22.567 INFO:tasks.rados.rados.0.smithi064.stdout:3999: done (1 left) 2024-03-04T06:25:22.567 INFO:tasks.rados.rados.0.smithi064.stdout: waiting on 1 2024-03-04T06:25:22.606 INFO:tasks.rados.rados.0.smithi064.stdout:3998: finishing write tid 1 to smithi06431743-14 2024-03-04T06:25:22.624 INFO:tasks.rados.rados.0.smithi064.stdout:3998: finishing write tid 2 to smithi06431743-14 2024-03-04T06:25:22.633 INFO:tasks.rados.rados.0.smithi064.stdout:3998: finishing write tid 3 to smithi06431743-14 2024-03-04T06:25:22.634 INFO:tasks.rados.rados.0.smithi064.stdout:3998: finishing write tid 4 to smithi06431743-14 2024-03-04T06:25:22.635 INFO:tasks.rados.rados.0.smithi064.stdout:3998: oid 14 updating version 0 to 1570 2024-03-04T06:25:22.635 INFO:tasks.rados.rados.0.smithi064.stdout:3998: oid 14 version 1570 is already newer than 1569 2024-03-04T06:25:22.635 INFO:tasks.rados.rados.0.smithi064.stdout:3998: oid 14 version 1570 is already newer than 1568 2024-03-04T06:25:22.635 INFO:tasks.rados.rados.0.smithi064.stdout:update_object_version oid 14 v 1570 (ObjNum 1789 snap 529 seq_num 1789) dirty exists 2024-03-04T06:25:22.635 INFO:tasks.rados.rados.0.smithi064.stdout:3998: left oid 14 (ObjNum 1789 snap 529 seq_num 1789) 2024-03-04T06:25:22.635 INFO:tasks.rados.rados.0.smithi064.stdout:3998: done (0 left) 2024-03-04T06:25:22.637 INFO:tasks.rados.rados.0.smithi064.stderr:0 errors. 2024-03-04T06:25:22.637 INFO:tasks.rados.rados.0.smithi064.stderr: 2024-03-04T06:25:22.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T06:25:22.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:22.742+0000 7f225140f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:22.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:22.748+0000 7f225140f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:22.940 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:22.941 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":1051,"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","created":"2024-03-04T06:07:38.204256+0000","modified":"2024-03-04T06:25:22.488837+0000","last_up_change":"2024-03-04T06:07:45.292995+0000","last_in_change":"2024-03-04T06:07:40.328293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T06:07:46.747241+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-04T06:07:48.452826+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-03-04T06:08:00.514694+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1051","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":530,"snap_epoch":1051,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1,"score_stable":1,"optimal_score":1,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a3a055b-c92d-46db-8419-c77269e3c8fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1128228680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1128228680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1128228680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":1128228680}]},"public_addr":"172.21.15.64:6805/1128228680","cluster_addr":"172.21.15.64:6804/1128228680","heartbeat_back_addr":"172.21.15.64:6806/1128228680","heartbeat_front_addr":"172.21.15.64:6803/1128228680","state":["exists","up"]},{"osd":1,"uuid":"a0c83c7b-d570-4f81-8e8f-4de39735fcf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":7123794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":7123794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":7123794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":7123794}]},"public_addr":"172.21.15.64:6802/7123794","cluster_addr":"172.21.15.64:6801/7123794","heartbeat_back_addr":"172.21.15.64:6812/7123794","heartbeat_front_addr":"172.21.15.64:6811/7123794","state":["exists","up"]},{"osd":2,"uuid":"20499aa6-c9a3-4a33-a9ed-0a58bf7b343f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":4001264085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6803","nonce":4001264085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6807","nonce":4001264085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":4001264085}]},"public_addr":"172.21.15.66:6802/4001264085","cluster_addr":"172.21.15.66:6803/4001264085","heartbeat_back_addr":"172.21.15.66:6807/4001264085","heartbeat_front_addr":"172.21.15.66:6806/4001264085","state":["exists","up"]},{"osd":3,"uuid":"60c18871-1171-43cd-b598-6560f6021eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6805","nonce":2264908010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":2264908010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6809","nonce":2264908010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":2264908010}]},"public_addr":"172.21.15.66:6805/2264908010","cluster_addr":"172.21.15.66:6804/2264908010","heartbeat_back_addr":"172.21.15.66:6809/2264908010","heartbeat_front_addr":"172.21.15.66:6808/2264908010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":444},{"begin":447,"length":26},{"begin":474,"length":8},{"begin":483,"length":3},{"begin":487,"length":2},{"begin":492,"length":8},{"begin":501,"length":1},{"begin":503,"length":3},{"begin":507,"length":2},{"begin":512,"length":1},{"begin":515,"length":3},{"begin":529,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":485,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T06:25:22.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:23.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.057+0000 7f8932d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.063+0000 7f8932d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.256 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:23.257 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:23.268 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1571,"stamp":"2024-03-04T06:25:22.929920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":104521777,"num_objects":1308,"num_object_clones":1087,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":187,"num_read":7137,"num_read_kb":5637554,"num_write":6044,"num_write_kb":1617490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18324,"ondisk_log_size":18324,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":882188,"kb_used_data":636076,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376605172,"statfs":{"total":386547056640,"available":385643696128,"internally_reserved":0,"allocated":651341824,"data_stored":742551177,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8236189,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":15,"num_read_kb":14659,"num_write":11,"num_write_kb":3101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000799"},"pg_stats":[{"pgid":"3.f","version":"1047'994","reported_seq":3540,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.491445+0000","last_change":"2024-03-04T06:25:18.634738+0000","last_active":"2024-03-04T06:25:19.491445+0000","last_peered":"2024-03-04T06:25:19.491445+0000","last_clean":"2024-03-04T06:25:19.491445+0000","last_became_active":"2024-03-04T06:08:01.875296+0000","last_became_peered":"2024-03-04T06:08:01.875296+0000","last_unstale":"2024-03-04T06:25:19.491445+0000","last_undegraded":"2024-03-04T06:25:19.491445+0000","last_fullsized":"2024-03-04T06:25:19.491445+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":994,"log_dups_size":0,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3798811,"num_objects":82,"num_object_clones":66,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":15,"num_read":377,"num_read_kb":295546,"num_write":342,"num_write_kb":90141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"1047'853","reported_seq":3550,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.550839+0000","last_change":"2024-03-04T06:25:18.594050+0000","last_active":"2024-03-04T06:25:19.550839+0000","last_peered":"2024-03-04T06:25:19.550839+0000","last_clean":"2024-03-04T06:25:19.550839+0000","last_became_active":"2024-03-04T06:08:01.878812+0000","last_became_peered":"2024-03-04T06:08:01.878812+0000","last_unstale":"2024-03-04T06:25:19.550839+0000","last_undegraded":"2024-03-04T06:25:19.550839+0000","last_fullsized":"2024-03-04T06:25:19.550839+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":853,"log_dups_size":0,"ondisk_log_size":853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7580815,"num_objects":67,"num_object_clones":49,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":265,"num_read_kb":210903,"num_write":277,"num_write_kb":73945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"1047'1385","reported_seq":3776,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.480781+0000","last_change":"2024-03-04T06:25:18.677524+0000","last_active":"2024-03-04T06:25:19.480781+0000","last_peered":"2024-03-04T06:25:19.480781+0000","last_clean":"2024-03-04T06:25:19.480781+0000","last_became_active":"2024-03-04T06:08:01.881717+0000","last_became_peered":"2024-03-04T06:08:01.881717+0000","last_unstale":"2024-03-04T06:25:19.480781+0000","last_undegraded":"2024-03-04T06:25:19.480781+0000","last_fullsized":"2024-03-04T06:25:19.480781+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1385,"log_dups_size":0,"ondisk_log_size":1385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10657387,"num_objects":102,"num_object_clones":87,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":638,"num_read_kb":520484,"num_write":469,"num_write_kb":128255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"1047'1146","reported_seq":3532,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.481320+0000","last_change":"2024-03-04T06:25:18.636553+0000","last_active":"2024-03-04T06:25:19.481320+0000","last_peered":"2024-03-04T06:25:19.481320+0000","last_clean":"2024-03-04T06:25:19.481320+0000","last_became_active":"2024-03-04T06:08:01.868046+0000","last_became_peered":"2024-03-04T06:08:01.868046+0000","last_unstale":"2024-03-04T06:25:19.481320+0000","last_undegraded":"2024-03-04T06:25:19.481320+0000","last_fullsized":"2024-03-04T06:25:19.481320+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1146,"log_dups_size":0,"ondisk_log_size":1146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6622816,"num_objects":72,"num_object_clones":62,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":355,"num_read_kb":271721,"num_write":341,"num_write_kb":94155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"1029'441","reported_seq":3307,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.521681+0000","last_change":"2024-03-04T06:25:18.527153+0000","last_active":"2024-03-04T06:25:19.521681+0000","last_peered":"2024-03-04T06:25:19.521681+0000","last_clean":"2024-03-04T06:25:19.521681+0000","last_became_active":"2024-03-04T06:08:01.877606+0000","last_became_peered":"2024-03-04T06:08:01.877606+0000","last_unstale":"2024-03-04T06:25:19.521681+0000","last_undegraded":"2024-03-04T06:25:19.521681+0000","last_fullsized":"2024-03-04T06:25:19.521681+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3546706,"num_objects":28,"num_object_clones":25,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":124,"num_read_kb":93149,"num_write":131,"num_write_kb":36421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"1047'1114","reported_seq":3607,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.494187+0000","last_change":"2024-03-04T06:25:18.672551+0000","last_active":"2024-03-04T06:25:19.494187+0000","last_peered":"2024-03-04T06:25:19.494187+0000","last_clean":"2024-03-04T06:25:19.494187+0000","last_became_active":"2024-03-04T06:08:01.885533+0000","last_became_peered":"2024-03-04T06:08:01.885533+0000","last_unstale":"2024-03-04T06:25:19.494187+0000","last_undegraded":"2024-03-04T06:25:19.494187+0000","last_fullsized":"2024-03-04T06:25:19.494187+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1114,"log_dups_size":0,"ondisk_log_size":1114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092917,"num_objects":78,"num_object_clones":66,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":463,"num_read_kb":375458,"num_write":370,"num_write_kb":106221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":3064,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.490716+0000","last_change":"2024-03-04T06:25:18.602122+0000","last_active":"2024-03-04T06:25:19.490716+0000","last_peered":"2024-03-04T06:25:19.490716+0000","last_clean":"2024-03-04T06:25:19.490716+0000","last_became_active":"2024-03-04T06:08:01.878890+0000","last_became_peered":"2024-03-04T06:08:01.878890+0000","last_unstale":"2024-03-04T06:25:19.490716+0000","last_undegraded":"2024-03-04T06:25:19.490716+0000","last_fullsized":"2024-03-04T06:25:19.490716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"1047'738","reported_seq":3390,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.481551+0000","last_change":"2024-03-04T06:25:18.603993+0000","last_active":"2024-03-04T06:25:19.481551+0000","last_peered":"2024-03-04T06:25:19.481551+0000","last_clean":"2024-03-04T06:25:19.481551+0000","last_became_active":"2024-03-04T06:08:01.872189+0000","last_became_peered":"2024-03-04T06:08:01.872189+0000","last_unstale":"2024-03-04T06:25:19.481551+0000","last_undegraded":"2024-03-04T06:25:19.481551+0000","last_fullsized":"2024-03-04T06:25:19.481551+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":738,"log_dups_size":0,"ondisk_log_size":738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3697969,"num_objects":47,"num_object_clones":38,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":8,"num_read":291,"num_read_kb":255611,"num_write":208,"num_write_kb":57031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":2777,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.513004+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:25:19.513004+0000","last_peered":"2024-03-04T06:25:19.513004+0000","last_clean":"2024-03-04T06:25:19.513004+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:19.513004+0000","last_undegraded":"2024-03-04T06:25:19.513004+0000","last_fullsized":"2024-03-04T06:25:19.513004+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2582,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.490683+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:25:19.490683+0000","last_peered":"2024-03-04T06:25:19.490683+0000","last_clean":"2024-03-04T06:25:19.490683+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:19.490683+0000","last_undegraded":"2024-03-04T06:25:19.490683+0000","last_fullsized":"2024-03-04T06:25:19.490683+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"1047'1079","reported_seq":3610,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.488089+0000","last_change":"2024-03-04T06:25:18.628894+0000","last_active":"2024-03-04T06:25:19.488089+0000","last_peered":"2024-03-04T06:25:19.488089+0000","last_clean":"2024-03-04T06:25:19.488089+0000","last_became_active":"2024-03-04T06:08:01.880278+0000","last_became_peered":"2024-03-04T06:08:01.880278+0000","last_unstale":"2024-03-04T06:25:19.488089+0000","last_undegraded":"2024-03-04T06:25:19.488089+0000","last_fullsized":"2024-03-04T06:25:19.488089+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1079,"log_dups_size":0,"ondisk_log_size":1079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6471553,"num_objects":79,"num_object_clones":65,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":479,"num_read_kb":397957,"num_write":356,"num_write_kb":96087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2574,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.494870+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:19.494870+0000","last_peered":"2024-03-04T06:25:19.494870+0000","last_clean":"2024-03-04T06:25:19.494870+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:19.494870+0000","last_undegraded":"2024-03-04T06:25:19.494870+0000","last_fullsized":"2024-03-04T06:25:19.494870+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"1047'758","reported_seq":3501,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.513967+0000","last_change":"2024-03-04T06:25:18.626577+0000","last_active":"2024-03-04T06:25:19.513967+0000","last_peered":"2024-03-04T06:25:19.513967+0000","last_clean":"2024-03-04T06:25:19.513967+0000","last_became_active":"2024-03-04T06:08:01.874042+0000","last_became_peered":"2024-03-04T06:08:01.874042+0000","last_unstale":"2024-03-04T06:25:19.513967+0000","last_undegraded":"2024-03-04T06:25:19.513967+0000","last_fullsized":"2024-03-04T06:25:19.513967+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6891728,"num_objects":55,"num_object_clones":46,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":7,"num_read":335,"num_read_kb":275603,"num_write":239,"num_write_kb":68723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2582,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.479519+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:25:19.479519+0000","last_peered":"2024-03-04T06:25:19.479519+0000","last_clean":"2024-03-04T06:25:19.479519+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:25:19.479519+0000","last_undegraded":"2024-03-04T06:25:19.479519+0000","last_fullsized":"2024-03-04T06:25:19.479519+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"1047'2039","reported_seq":4165,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.536038+0000","last_change":"2024-03-04T06:25:18.691477+0000","last_active":"2024-03-04T06:25:19.536038+0000","last_peered":"2024-03-04T06:25:19.536038+0000","last_clean":"2024-03-04T06:25:19.536038+0000","last_became_active":"2024-03-04T06:08:01.886304+0000","last_became_peered":"2024-03-04T06:08:01.886304+0000","last_unstale":"2024-03-04T06:25:19.536038+0000","last_undegraded":"2024-03-04T06:25:19.536038+0000","last_fullsized":"2024-03-04T06:25:19.536038+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2039,"log_dups_size":0,"ondisk_log_size":2039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11245566,"num_objects":152,"num_object_clones":125,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":23,"num_read":875,"num_read_kb":701659,"num_write":704,"num_write_kb":190182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2574,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.480023+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:19.480023+0000","last_peered":"2024-03-04T06:25:19.480023+0000","last_clean":"2024-03-04T06:25:19.480023+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:19.480023+0000","last_undegraded":"2024-03-04T06:25:19.480023+0000","last_fullsized":"2024-03-04T06:25:19.480023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"1047'2428","reported_seq":4198,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.547845+0000","last_change":"2024-03-04T06:25:18.673319+0000","last_active":"2024-03-04T06:25:19.547845+0000","last_peered":"2024-03-04T06:25:19.547845+0000","last_clean":"2024-03-04T06:25:19.547845+0000","last_became_active":"2024-03-04T06:08:01.901351+0000","last_became_peered":"2024-03-04T06:08:01.901351+0000","last_unstale":"2024-03-04T06:25:19.547845+0000","last_undegraded":"2024-03-04T06:25:19.547845+0000","last_fullsized":"2024-03-04T06:25:19.547845+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":2428,"log_dups_size":0,"ondisk_log_size":2428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9430014,"num_objects":167,"num_object_clones":143,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":21,"num_read":771,"num_read_kb":571096,"num_write":762,"num_write_kb":206699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"1047'1021","reported_seq":3593,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.491303+0000","last_change":"2024-03-04T06:25:18.619748+0000","last_active":"2024-03-04T06:25:19.491303+0000","last_peered":"2024-03-04T06:25:19.491303+0000","last_clean":"2024-03-04T06:25:19.491303+0000","last_became_active":"2024-03-04T06:08:01.904196+0000","last_became_peered":"2024-03-04T06:08:01.904196+0000","last_unstale":"2024-03-04T06:25:19.491303+0000","last_undegraded":"2024-03-04T06:25:19.491303+0000","last_fullsized":"2024-03-04T06:25:19.491303+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1021,"log_dups_size":0,"ondisk_log_size":1021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7295096,"num_objects":72,"num_object_clones":59,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":11,"num_read":466,"num_read_kb":391842,"num_write":343,"num_write_kb":93831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2692,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.559023+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:25:19.559023+0000","last_peered":"2024-03-04T06:25:19.559023+0000","last_clean":"2024-03-04T06:25:19.559023+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:25:19.559023+0000","last_undegraded":"2024-03-04T06:25:19.559023+0000","last_fullsized":"2024-03-04T06:25:19.559023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"1047'1199","reported_seq":3704,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.559844+0000","last_change":"2024-03-04T06:25:18.590678+0000","last_active":"2024-03-04T06:25:19.559844+0000","last_peered":"2024-03-04T06:25:19.559844+0000","last_clean":"2024-03-04T06:25:19.559844+0000","last_became_active":"2024-03-04T06:08:01.891858+0000","last_became_peered":"2024-03-04T06:08:01.891858+0000","last_unstale":"2024-03-04T06:25:19.559844+0000","last_undegraded":"2024-03-04T06:25:19.559844+0000","last_fullsized":"2024-03-04T06:25:19.559844+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1199,"log_dups_size":0,"ondisk_log_size":1199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924847,"num_objects":88,"num_object_clones":71,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":16,"num_read":418,"num_read_kb":317414,"num_write":378,"num_write_kb":101479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2644,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.520387+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:25:19.520387+0000","last_peered":"2024-03-04T06:25:19.520387+0000","last_clean":"2024-03-04T06:25:19.520387+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:19.520387+0000","last_undegraded":"2024-03-04T06:25:19.520387+0000","last_fullsized":"2024-03-04T06:25:19.520387+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"1047'1560","reported_seq":3818,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.508201+0000","last_change":"2024-03-04T06:25:18.630165+0000","last_active":"2024-03-04T06:25:19.508201+0000","last_peered":"2024-03-04T06:25:19.508201+0000","last_clean":"2024-03-04T06:25:19.508201+0000","last_became_active":"2024-03-04T06:08:01.888958+0000","last_became_peered":"2024-03-04T06:08:01.888958+0000","last_unstale":"2024-03-04T06:25:19.508201+0000","last_undegraded":"2024-03-04T06:25:19.508201+0000","last_fullsized":"2024-03-04T06:25:19.508201+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1560,"log_dups_size":0,"ondisk_log_size":1560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10169522,"num_objects":107,"num_object_clones":90,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":14,"num_read":547,"num_read_kb":436376,"num_write":494,"num_write_kb":137381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2692,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.549927+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:19.549927+0000","last_peered":"2024-03-04T06:25:19.549927+0000","last_clean":"2024-03-04T06:25:19.549927+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:19.549927+0000","last_undegraded":"2024-03-04T06:25:19.549927+0000","last_fullsized":"2024-03-04T06:25:19.549927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"1047'1472","reported_seq":3912,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.547115+0000","last_change":"2024-03-04T06:25:18.623900+0000","last_active":"2024-03-04T06:25:19.547115+0000","last_peered":"2024-03-04T06:25:19.547115+0000","last_clean":"2024-03-04T06:25:19.547115+0000","last_became_active":"2024-03-04T06:08:01.867473+0000","last_became_peered":"2024-03-04T06:08:01.867473+0000","last_unstale":"2024-03-04T06:25:19.547115+0000","last_undegraded":"2024-03-04T06:25:19.547115+0000","last_fullsized":"2024-03-04T06:25:19.547115+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:08:00.773669+0000","last_clean_scrub_stamp":"2024-03-04T06:08:00.773669+0000","objects_scrubbed":0,"log_size":1472,"log_dups_size":0,"ondisk_log_size":1472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10505662,"num_objects":110,"num_object_clones":95,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":12,"num_read":630,"num_read_kb":522644,"num_write":493,"num_write_kb":135045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2582,"reported_epoch":1048,"state":"active+clean","last_fresh":"2024-03-04T06:25:19.485115+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:25:19.485115+0000","last_peered":"2024-03-04T06:25:19.485115+0000","last_clean":"2024-03-04T06:25:19.485115+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:19.485115+0000","last_undegraded":"2024-03-04T06:25:19.485115+0000","last_fullsized":"2024-03-04T06:25:19.485115+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":103931409,"num_objects":1306,"num_object_clones":1087,"num_object_copies":3918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":187,"num_read":7034,"num_read_kb":5637463,"num_write":5907,"num_write_kb":1615596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18227,"ondisk_log_size":18227,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706719,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":277420,"kb_used_data":202020,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94094420,"statfs":{"total":96636764160,"available":96352686080,"internally_reserved":0,"allocated":206868480,"data_stored":236505736,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706719,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":215352,"kb_used_data":152432,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94156488,"statfs":{"total":96636764160,"available":96416243712,"internally_reserved":0,"allocated":156090368,"data_stored":179481942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208816,"kb_used_data":145896,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94163024,"statfs":{"total":96636764160,"available":96422936576,"internally_reserved":0,"allocated":149397504,"data_stored":168107891,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706718,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":180600,"kb_used_data":135728,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94191240,"statfs":{"total":96636764160,"available":96451829760,"internally_reserved":0,"allocated":138985472,"data_stored":158455608,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:23.270 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-04T06:25:23.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-04T06:25:23.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T06:25:23.383 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.375+0000 7fcc64c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.386+0000 7fcc64c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.436 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.434+0000 7f69d526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.439 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.439+0000 7f69d526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.500 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-03-04T06:25:23.500 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-03-04T06:25:23.500 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T06:25:23.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-03-04T06:25:23.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.604+0000 7f04cd11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.611 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.611+0000 7f04cd11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.624 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:25:23.624 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:25:23.624 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:25:23.624 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:25:23.624 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:25:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:25:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:25:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:25:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:25:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:25:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:25:23.625 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:25:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:25:23.627 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:25:23.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-03-04T06:25:23.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.731+0000 7f4a6659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.738+0000 7f4a6659d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.750 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:25:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:25:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:25:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:25:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:25:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:25:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:25:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:25:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:25:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:25:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:25:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:25:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:25:23.754 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:25:23.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-03-04T06:25:23.862 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.857+0000 7f87ad19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.865 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.864+0000 7f87ad19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.878 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:25:23.878 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:25:23.879 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:25:23.879 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:25:23.879 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:25:23.879 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:25:23.879 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:25:23.879 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:25:23.879 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:25:23.879 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:25:23.879 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:25:23.879 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:25:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:25:23.881 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:25:23.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-03-04T06:25:23.990 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.985+0000 7fb10a79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:23.992 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:23.992+0000 7fb10a79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:24.009 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-03-04T06:25:24.010 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-03-04T06:25:24.010 INFO:teuthology.orchestra.run.smithi064.stderr:config help 2024-03-04T06:25:24.010 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-03-04T06:25:24.010 INFO:teuthology.orchestra.run.smithi064.stderr:config show 2024-03-04T06:25:24.010 INFO:teuthology.orchestra.run.smithi064.stderr:deep_scrub 2024-03-04T06:25:24.010 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_ops 2024-03-04T06:25:24.010 INFO:teuthology.orchestra.run.smithi064.stderr:dump_historic_slow_ops 2024-03-04T06:25:24.010 INFO:teuthology.orchestra.run.smithi064.stderr:dump_metrics [] 2024-03-04T06:25:24.010 INFO:teuthology.orchestra.run.smithi064.stderr:dump_ops_in_flight 2024-03-04T06:25:24.010 INFO:teuthology.orchestra.run.smithi064.stderr:dump_pgstate_history 2024-03-04T06:25:24.010 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-03-04T06:25:24.012 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T06:25:24.013 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T06:25:24.505 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_0' removed 2024-03-04T06:25:24.522 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-04T06:25:24.534 INFO:tasks.thrashosds:joining thrashosds 2024-03-04T06:25:29.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-04T06:25:29.133 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:29.131+0000 7f8e2cf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:29.136 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:29.136+0000 7f8e2cf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:29.150 INFO:teuthology.orchestra.run.smithi064.stdout:{} 2024-03-04T06:25:29.150 INFO:teuthology.orchestra.run.smithi064.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-04T06:25:29.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-04T06:25:29.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:29.265+0000 7f7ccdc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:29.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:29.273+0000 7f7ccdc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:30.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-03-04T06:25:30.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-04T06:25:30.364 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T06:25:30.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-04T06:25:30.495 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:30.488+0000 7f588a15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:30.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:30.496+0000 7f588a15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-04T06:25:31.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-04T06:25:31.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:31.491+0000 7f93ae8dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:31.500 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:31.499+0000 7f93ae8dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:32.370 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T06:25:32.386 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-04T06:25:32.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:32.500+0000 7f6a5655d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:32.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:32.508+0000 7f6a5655d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:33.359 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-04T06:25:33.378 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-04T06:25:33.493 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:33.487+0000 7f0772907640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:33.496 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:33.495+0000 7f0772907640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:34.377 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T06:25:34.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-04T06:25:34.506 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:34.500+0000 7f763799d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:34.509 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:34.508+0000 7f763799d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:35.387 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-04T06:25:35.402 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-04T06:25:35.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-03-04T06:25:35.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-03-04T06:25:35.516 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:35.510+0000 7ff1abc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:35.519 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:35.518+0000 7ff1abc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:36.396 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T06:25:36.411 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-04T06:25:36.523 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:36.517+0000 7fd7ec824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:36.526 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:36.525+0000 7fd7ec824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:37.407 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-04T06:25:37.423 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-04T06:25:37.423 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-04T06:25:37.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T06:25:37.532 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:37.528+0000 7fca89114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:37.535 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:37.534+0000 7fca89114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:37.725 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:37.725 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":1062,"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","created":"2024-03-04T06:07:38.204256+0000","modified":"2024-03-04T06:25:37.405300+0000","last_up_change":"2024-03-04T06:07:45.292995+0000","last_in_change":"2024-03-04T06:07:40.328293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T06:07:46.747241+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-04T06:07:48.452826+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a3a055b-c92d-46db-8419-c77269e3c8fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1128228680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1128228680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1128228680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":1128228680}]},"public_addr":"172.21.15.64:6805/1128228680","cluster_addr":"172.21.15.64:6804/1128228680","heartbeat_back_addr":"172.21.15.64:6806/1128228680","heartbeat_front_addr":"172.21.15.64:6803/1128228680","state":["exists","up"]},{"osd":1,"uuid":"a0c83c7b-d570-4f81-8e8f-4de39735fcf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":7123794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":7123794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":7123794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":7123794}]},"public_addr":"172.21.15.64:6802/7123794","cluster_addr":"172.21.15.64:6801/7123794","heartbeat_back_addr":"172.21.15.64:6812/7123794","heartbeat_front_addr":"172.21.15.64:6811/7123794","state":["exists","up"]},{"osd":2,"uuid":"20499aa6-c9a3-4a33-a9ed-0a58bf7b343f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":4001264085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6803","nonce":4001264085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6807","nonce":4001264085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":4001264085}]},"public_addr":"172.21.15.66:6802/4001264085","cluster_addr":"172.21.15.66:6803/4001264085","heartbeat_back_addr":"172.21.15.66:6807/4001264085","heartbeat_front_addr":"172.21.15.66:6806/4001264085","state":["exists","up"]},{"osd":3,"uuid":"60c18871-1171-43cd-b598-6560f6021eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6805","nonce":2264908010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":2264908010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6809","nonce":2264908010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":2264908010}]},"public_addr":"172.21.15.66:6805/2264908010","cluster_addr":"172.21.15.66:6804/2264908010","heartbeat_back_addr":"172.21.15.66:6809/2264908010","heartbeat_front_addr":"172.21.15.66:6808/2264908010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":444},{"begin":447,"length":26},{"begin":474,"length":8},{"begin":483,"length":3},{"begin":487,"length":2},{"begin":492,"length":8},{"begin":501,"length":1},{"begin":503,"length":3},{"begin":507,"length":2},{"begin":512,"length":1},{"begin":515,"length":3},{"begin":529,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T06:25:37.738 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-04T06:25:37.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T06:25:37.847 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:37.843+0000 7f1f0e0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:37.850 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:37.849+0000 7f1f0e0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:38.042 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":1062,"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","created":"2024-03-04T06:07:38.204256+0000","modified":"2024-03-04T06:25:37.405300+0000","last_up_change":"2024-03-04T06:07:45.292995+0000","last_in_change":"2024-03-04T06:07:40.328293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T06:07:46.747241+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-04T06:07:48.452826+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3a3a055b-c92d-46db-8419-c77269e3c8fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6805","nonce":1128228680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1128228680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1128228680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6803","nonce":1128228680}]},"public_addr":"172.21.15.64:6805/1128228680","cluster_addr":"172.21.15.64:6804/1128228680","heartbeat_back_addr":"172.21.15.64:6806/1128228680","heartbeat_front_addr":"172.21.15.64:6803/1128228680","state":["exists","up"]},{"osd":1,"uuid":"a0c83c7b-d570-4f81-8e8f-4de39735fcf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":7123794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6801","nonce":7123794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":7123794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6811","nonce":7123794}]},"public_addr":"172.21.15.64:6802/7123794","cluster_addr":"172.21.15.64:6801/7123794","heartbeat_back_addr":"172.21.15.64:6812/7123794","heartbeat_front_addr":"172.21.15.64:6811/7123794","state":["exists","up"]},{"osd":2,"uuid":"20499aa6-c9a3-4a33-a9ed-0a58bf7b343f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6802","nonce":4001264085}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6803","nonce":4001264085}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6807","nonce":4001264085}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6806","nonce":4001264085}]},"public_addr":"172.21.15.66:6802/4001264085","cluster_addr":"172.21.15.66:6803/4001264085","heartbeat_back_addr":"172.21.15.66:6807/4001264085","heartbeat_front_addr":"172.21.15.66:6806/4001264085","state":["exists","up"]},{"osd":3,"uuid":"60c18871-1171-43cd-b598-6560f6021eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6805","nonce":2264908010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6804","nonce":2264908010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6809","nonce":2264908010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.66:6808","nonce":2264908010}]},"public_addr":"172.21.15.66:6805/2264908010","cluster_addr":"172.21.15.66:6804/2264908010","heartbeat_back_addr":"172.21.15.66:6809/2264908010","heartbeat_front_addr":"172.21.15.66:6808/2264908010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":444},{"begin":447,"length":26},{"begin":474,"length":8},{"begin":483,"length":3},{"begin":487,"length":2},{"begin":492,"length":8},{"begin":501,"length":1},{"begin":503,"length":3},{"begin":507,"length":2},{"begin":512,"length":1},{"begin":515,"length":3},{"begin":529,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T06:25:38.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-04T06:25:38.056 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-04T06:25:38.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-04T06:25:38.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-04T06:25:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.172+0000 7f41b3d54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.172+0000 7f46b5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.177 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.172+0000 7fe76642d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.178 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.172+0000 7f4fb975d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.180+0000 7f4fb975d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.180+0000 7f46b5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.180+0000 7f41b3d54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.181 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.181+0000 7fe76642d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.208 INFO:teuthology.orchestra.run.smithi064.stdout:38654706736 2024-03-04T06:25:38.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-04T06:25:38.210 INFO:teuthology.orchestra.run.smithi064.stdout:38654706736 2024-03-04T06:25:38.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-04T06:25:38.316 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.312+0000 7f7f8382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.320 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.319+0000 7f7f8382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.356+0000 7f917ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:38.364+0000 7f917ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:38.541 INFO:teuthology.orchestra.run.smithi064.stdout:38654706734 2024-03-04T06:25:38.554 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706736 got 38654706734 for osd.1 2024-03-04T06:25:38.581 INFO:teuthology.orchestra.run.smithi064.stdout:38654706734 2024-03-04T06:25:38.593 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706736 got 38654706734 for osd.3 2024-03-04T06:25:39.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-04T06:25:39.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-04T06:25:39.681 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:39.674+0000 7f5b13f07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:39.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:39.684+0000 7f5b13f07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:39.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:39.712+0000 7fcbfa51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:39.720 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:39.719+0000 7fcbfa51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:39.876 INFO:teuthology.orchestra.run.smithi064.stdout:38654706736 2024-03-04T06:25:39.888 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706736 got 38654706736 for osd.1 2024-03-04T06:25:39.889 DEBUG:teuthology.parallel:result is None 2024-03-04T06:25:39.911 INFO:teuthology.orchestra.run.smithi064.stdout:38654706736 2024-03-04T06:25:39.924 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706736 got 38654706736 for osd.3 2024-03-04T06:25:39.924 DEBUG:teuthology.parallel:result is None 2024-03-04T06:25:40.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2024-03-04T06:25:40.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2024-03-04T06:25:46.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2024-03-04T06:25:46.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~16s 2024-03-04T06:25:50.472 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-04T06:25:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-04T06:25:50.479 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/thrashosds.py", line 216, in task cluster_manager.flush_all_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1762, in flush_all_pg_stats self.flush_pg_stats(range(len(self.get_osd_dump()))) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1759, in flush_pg_stats p.spawn(flush_one_osd, osd, wait_for_mon) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1737, in flush_one_osd need = int(self.raw_cluster_cmd('tell', 'osd.%d' % osd, 'flush_pg_stats')) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' 2024-03-04T06:25:50.483 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-04T06:25:50.497 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-04T06:25:50.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:50.614 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:50.610+0000 7f6d6686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:50.616 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:50.616+0000 7f6d6686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:50.805 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:50.805 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:50.817 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1596,"stamp":"2024-03-04T06:25:50.470240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628648,"kb_used_data":382536,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858712,"statfs":{"total":386547056640,"available":385903321088,"internally_reserved":0,"allocated":391716864,"data_stored":444317687,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.537170"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2811,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:37.416953+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:25:37.416953+0000","last_peered":"2024-03-04T06:25:37.416953+0000","last_clean":"2024-03-04T06:25:37.416953+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:37.416953+0000","last_undegraded":"2024-03-04T06:25:37.416953+0000","last_fullsized":"2024-03-04T06:25:37.416953+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2614,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:38.446026+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:25:38.446026+0000","last_peered":"2024-03-04T06:25:38.446026+0000","last_clean":"2024-03-04T06:25:38.446026+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:38.446026+0000","last_undegraded":"2024-03-04T06:25:38.446026+0000","last_fullsized":"2024-03-04T06:25:38.446026+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.496410+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:24.496410+0000","last_peered":"2024-03-04T06:25:24.496410+0000","last_clean":"2024-03-04T06:25:24.496410+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:24.496410+0000","last_undegraded":"2024-03-04T06:25:24.496410+0000","last_fullsized":"2024-03-04T06:25:24.496410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2614,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:38.431103+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:25:38.431103+0000","last_peered":"2024-03-04T06:25:38.431103+0000","last_clean":"2024-03-04T06:25:38.431103+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:25:38.431103+0000","last_undegraded":"2024-03-04T06:25:38.431103+0000","last_fullsized":"2024-03-04T06:25:38.431103+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.492352+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:24.492352+0000","last_peered":"2024-03-04T06:25:24.492352+0000","last_clean":"2024-03-04T06:25:24.492352+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:24.492352+0000","last_undegraded":"2024-03-04T06:25:24.492352+0000","last_fullsized":"2024-03-04T06:25:24.492352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.514316+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:25:24.514316+0000","last_peered":"2024-03-04T06:25:24.514316+0000","last_clean":"2024-03-04T06:25:24.514316+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:25:50.470254+0000","last_undegraded":"2024-03-04T06:25:24.514316+0000","last_fullsized":"2024-03-04T06:25:24.514316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2678,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:37.420869+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:25:37.420869+0000","last_peered":"2024-03-04T06:25:37.420869+0000","last_clean":"2024-03-04T06:25:37.420869+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:37.420869+0000","last_undegraded":"2024-03-04T06:25:37.420869+0000","last_fullsized":"2024-03-04T06:25:37.420869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.504080+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:24.504080+0000","last_peered":"2024-03-04T06:25:24.504080+0000","last_clean":"2024-03-04T06:25:24.504080+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:50.470256+0000","last_undegraded":"2024-03-04T06:25:24.504080+0000","last_fullsized":"2024-03-04T06:25:24.504080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2614,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:38.441046+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:25:38.441046+0000","last_peered":"2024-03-04T06:25:38.441046+0000","last_clean":"2024-03-04T06:25:38.441046+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:38.441046+0000","last_undegraded":"2024-03-04T06:25:38.441046+0000","last_fullsized":"2024-03-04T06:25:38.441046+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114956,"kb_used_data":39556,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256884,"statfs":{"total":96636764160,"available":96519049216,"internally_reserved":0,"allocated":40505344,"data_stored":44002150,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123204,"kb_used_data":60284,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248636,"statfs":{"total":96636764160,"available":96510603264,"internally_reserved":0,"allocated":61730816,"data_stored":72078415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:50.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:50.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:50.920+0000 7f01290dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:50.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:50.927+0000 7f01290dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:51.115 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:51.128 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1597,"stamp":"2024-03-04T06:25:50.935646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628648,"kb_used_data":382536,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858712,"statfs":{"total":386547056640,"available":385903321088,"internally_reserved":0,"allocated":391716864,"data_stored":444317687,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002194"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2811,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:37.416953+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:25:37.416953+0000","last_peered":"2024-03-04T06:25:37.416953+0000","last_clean":"2024-03-04T06:25:37.416953+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:37.416953+0000","last_undegraded":"2024-03-04T06:25:37.416953+0000","last_fullsized":"2024-03-04T06:25:37.416953+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2614,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:38.446026+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:25:38.446026+0000","last_peered":"2024-03-04T06:25:38.446026+0000","last_clean":"2024-03-04T06:25:38.446026+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:38.446026+0000","last_undegraded":"2024-03-04T06:25:38.446026+0000","last_fullsized":"2024-03-04T06:25:38.446026+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.496410+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:24.496410+0000","last_peered":"2024-03-04T06:25:24.496410+0000","last_clean":"2024-03-04T06:25:24.496410+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:24.496410+0000","last_undegraded":"2024-03-04T06:25:24.496410+0000","last_fullsized":"2024-03-04T06:25:24.496410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2614,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:38.431103+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:25:38.431103+0000","last_peered":"2024-03-04T06:25:38.431103+0000","last_clean":"2024-03-04T06:25:38.431103+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:25:38.431103+0000","last_undegraded":"2024-03-04T06:25:38.431103+0000","last_fullsized":"2024-03-04T06:25:38.431103+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.492352+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:24.492352+0000","last_peered":"2024-03-04T06:25:24.492352+0000","last_clean":"2024-03-04T06:25:24.492352+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:24.492352+0000","last_undegraded":"2024-03-04T06:25:24.492352+0000","last_fullsized":"2024-03-04T06:25:24.492352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.514316+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:25:24.514316+0000","last_peered":"2024-03-04T06:25:24.514316+0000","last_clean":"2024-03-04T06:25:24.514316+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:25:50.470254+0000","last_undegraded":"2024-03-04T06:25:24.514316+0000","last_fullsized":"2024-03-04T06:25:24.514316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2678,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:37.420869+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:25:37.420869+0000","last_peered":"2024-03-04T06:25:37.420869+0000","last_clean":"2024-03-04T06:25:37.420869+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:37.420869+0000","last_undegraded":"2024-03-04T06:25:37.420869+0000","last_fullsized":"2024-03-04T06:25:37.420869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.504080+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:24.504080+0000","last_peered":"2024-03-04T06:25:24.504080+0000","last_clean":"2024-03-04T06:25:24.504080+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:50.470256+0000","last_undegraded":"2024-03-04T06:25:24.504080+0000","last_fullsized":"2024-03-04T06:25:24.504080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2614,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:38.441046+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:25:38.441046+0000","last_peered":"2024-03-04T06:25:38.441046+0000","last_clean":"2024-03-04T06:25:38.441046+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:38.441046+0000","last_undegraded":"2024-03-04T06:25:38.441046+0000","last_fullsized":"2024-03-04T06:25:38.441046+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114956,"kb_used_data":39556,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256884,"statfs":{"total":96636764160,"available":96519049216,"internally_reserved":0,"allocated":40505344,"data_stored":44002150,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123204,"kb_used_data":60284,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248636,"statfs":{"total":96636764160,"available":96510603264,"internally_reserved":0,"allocated":61730816,"data_stored":72078415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:51.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:51.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:51.232+0000 7f5f0c6d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:51.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:51.238+0000 7f5f0c6d4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:51.425 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:51.426 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:51.438 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1597,"stamp":"2024-03-04T06:25:50.935646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628648,"kb_used_data":382536,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858712,"statfs":{"total":386547056640,"available":385903321088,"internally_reserved":0,"allocated":391716864,"data_stored":444317687,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002194"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2811,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:37.416953+0000","last_change":"2024-03-04T06:07:48.339135+0000","last_active":"2024-03-04T06:25:37.416953+0000","last_peered":"2024-03-04T06:25:37.416953+0000","last_clean":"2024-03-04T06:25:37.416953+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:37.416953+0000","last_undegraded":"2024-03-04T06:25:37.416953+0000","last_fullsized":"2024-03-04T06:25:37.416953+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2614,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:38.446026+0000","last_change":"2024-03-04T06:07:50.387148+0000","last_active":"2024-03-04T06:25:38.446026+0000","last_peered":"2024-03-04T06:25:38.446026+0000","last_clean":"2024-03-04T06:25:38.446026+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:38.446026+0000","last_undegraded":"2024-03-04T06:25:38.446026+0000","last_fullsized":"2024-03-04T06:25:38.446026+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.496410+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:24.496410+0000","last_peered":"2024-03-04T06:25:24.496410+0000","last_clean":"2024-03-04T06:25:24.496410+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:24.496410+0000","last_undegraded":"2024-03-04T06:25:24.496410+0000","last_fullsized":"2024-03-04T06:25:24.496410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2614,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:38.431103+0000","last_change":"2024-03-04T06:07:50.371003+0000","last_active":"2024-03-04T06:25:38.431103+0000","last_peered":"2024-03-04T06:25:38.431103+0000","last_clean":"2024-03-04T06:25:38.431103+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:25:38.431103+0000","last_undegraded":"2024-03-04T06:25:38.431103+0000","last_fullsized":"2024-03-04T06:25:38.431103+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.492352+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:24.492352+0000","last_peered":"2024-03-04T06:25:24.492352+0000","last_clean":"2024-03-04T06:25:24.492352+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:24.492352+0000","last_undegraded":"2024-03-04T06:25:24.492352+0000","last_fullsized":"2024-03-04T06:25:24.492352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.514316+0000","last_change":"2024-03-04T06:07:50.398501+0000","last_active":"2024-03-04T06:25:24.514316+0000","last_peered":"2024-03-04T06:25:24.514316+0000","last_clean":"2024-03-04T06:25:24.514316+0000","last_became_active":"2024-03-04T06:07:50.395285+0000","last_became_peered":"2024-03-04T06:07:50.395285+0000","last_unstale":"2024-03-04T06:25:50.470254+0000","last_undegraded":"2024-03-04T06:25:24.514316+0000","last_fullsized":"2024-03-04T06:25:24.514316+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2678,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:37.420869+0000","last_change":"2024-03-04T06:07:50.391349+0000","last_active":"2024-03-04T06:25:37.420869+0000","last_peered":"2024-03-04T06:25:37.420869+0000","last_clean":"2024-03-04T06:25:37.420869+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:37.420869+0000","last_undegraded":"2024-03-04T06:25:37.420869+0000","last_fullsized":"2024-03-04T06:25:37.420869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.504080+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:24.504080+0000","last_peered":"2024-03-04T06:25:24.504080+0000","last_clean":"2024-03-04T06:25:24.504080+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:50.470256+0000","last_undegraded":"2024-03-04T06:25:24.504080+0000","last_fullsized":"2024-03-04T06:25:24.504080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2614,"reported_epoch":1062,"state":"active+clean","last_fresh":"2024-03-04T06:25:38.441046+0000","last_change":"2024-03-04T06:07:50.396199+0000","last_active":"2024-03-04T06:25:38.441046+0000","last_peered":"2024-03-04T06:25:38.441046+0000","last_clean":"2024-03-04T06:25:38.441046+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:38.441046+0000","last_undegraded":"2024-03-04T06:25:38.441046+0000","last_fullsized":"2024-03-04T06:25:38.441046+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114956,"kb_used_data":39556,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256884,"statfs":{"total":96636764160,"available":96519049216,"internally_reserved":0,"allocated":40505344,"data_stored":44002150,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706744,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123204,"kb_used_data":60284,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248636,"statfs":{"total":96636764160,"available":96510603264,"internally_reserved":0,"allocated":61730816,"data_stored":72078415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:51.438 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:25:51.438 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 2702, 'reported_epoch': 1052, 'state': 'stale+active+clean', 'last_fresh': '2024-03-04T06:25:24.514316+0000', 'last_change': '2024-03-04T06:07:50.398501+0000', 'last_active': '2024-03-04T06:25:24.514316+0000', 'last_peered': '2024-03-04T06:25:24.514316+0000', 'last_clean': '2024-03-04T06:25:24.514316+0000', 'last_became_active': '2024-03-04T06:07:50.395285+0000', 'last_became_peered': '2024-03-04T06:07:50.395285+0000', 'last_unstale': '2024-03-04T06:25:50.470254+0000', 'last_undegraded': '2024-03-04T06:25:24.514316+0000', 'last_fullsized': '2024-03-04T06:25:24.514316+0000', 'mapping_epoch': 12, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3, 0], 'acting': [2, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T06:25:51.439 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:25:51.439 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 2702, 'reported_epoch': 1052, 'state': 'stale+active+clean', 'last_fresh': '2024-03-04T06:25:24.504080+0000', 'last_change': '2024-03-04T06:07:50.365236+0000', 'last_active': '2024-03-04T06:25:24.504080+0000', 'last_peered': '2024-03-04T06:25:24.504080+0000', 'last_clean': '2024-03-04T06:25:24.504080+0000', 'last_became_active': '2024-03-04T06:07:50.360536+0000', 'last_became_peered': '2024-03-04T06:07:50.360536+0000', 'last_unstale': '2024-03-04T06:25:50.470256+0000', 'last_undegraded': '2024-03-04T06:25:24.504080+0000', 'last_fullsized': '2024-03-04T06:25:24.504080+0000', 'mapping_epoch': 12, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3], 'acting': [2, 0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T06:25:51.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:25:51.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:51.553+0000 7f040355d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:51.560 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:51.559+0000 7f040355d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:51.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~22s 2024-03-04T06:25:51.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~22s 2024-03-04T06:25:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:51.816 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1093,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1064,"num_osds":4,"num_up_osds":3,"osd_up_since":1709533550,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"stale+active+clean","count":2}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643735552,"bytes_avail":385903321088,"bytes_total":386547056640},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:25:51.829 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:25:51.829 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:51.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:51.933+0000 7f905819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:51.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:51.939+0000 7f905819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:52.128 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:52.129 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:52.140 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2024-03-04T06:25:51.475682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":21,"acting":21,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628664,"kb_used_data":382552,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858696,"statfs":{"total":386547056640,"available":385903304704,"internally_reserved":0,"allocated":391733248,"data_stored":444324265,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.541720"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2812,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:37.417194+0000","last_change":"2024-03-04T06:25:50.479118+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:37.417194+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:37.417194+0000","last_undegraded":"2024-03-04T06:25:37.417194+0000","last_fullsized":"2024-03-04T06:25:37.417194+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.446268+0000","last_change":"2024-03-04T06:25:50.494728+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:38.446268+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:38.446268+0000","last_undegraded":"2024-03-04T06:25:38.446268+0000","last_fullsized":"2024-03-04T06:25:38.446268+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.496410+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:24.496410+0000","last_peered":"2024-03-04T06:25:24.496410+0000","last_clean":"2024-03-04T06:25:24.496410+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:24.496410+0000","last_undegraded":"2024-03-04T06:25:24.496410+0000","last_fullsized":"2024-03-04T06:25:24.496410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.431398+0000","last_change":"2024-03-04T06:25:50.482833+0000","last_active":"2024-03-04T06:25:50.482833+0000","last_peered":"2024-03-04T06:25:38.431398+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:25:38.431398+0000","last_undegraded":"2024-03-04T06:25:38.431398+0000","last_fullsized":"2024-03-04T06:25:38.431398+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.492352+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:24.492352+0000","last_peered":"2024-03-04T06:25:24.492352+0000","last_clean":"2024-03-04T06:25:24.492352+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:24.492352+0000","last_undegraded":"2024-03-04T06:25:24.492352+0000","last_fullsized":"2024-03-04T06:25:24.492352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:07:50.329316+0000","last_change":"2024-03-04T06:25:50.488037+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:07:49.306852+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:07:50.329316+0000","last_undegraded":"2024-03-04T06:07:50.329316+0000","last_fullsized":"2024-03-04T06:07:50.329316+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2679,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:37.421082+0000","last_change":"2024-03-04T06:25:50.491700+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:37.421082+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:37.421082+0000","last_undegraded":"2024-03-04T06:25:37.421082+0000","last_fullsized":"2024-03-04T06:25:37.421082+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.504080+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:24.504080+0000","last_peered":"2024-03-04T06:25:24.504080+0000","last_clean":"2024-03-04T06:25:24.504080+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:50.470256+0000","last_undegraded":"2024-03-04T06:25:24.504080+0000","last_fullsized":"2024-03-04T06:25:24.504080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.441197+0000","last_change":"2024-03-04T06:25:50.493405+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:38.441197+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:38.441197+0000","last_undegraded":"2024-03-04T06:25:38.441197+0000","last_fullsized":"2024-03-04T06:25:38.441197+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114964,"kb_used_data":39564,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256876,"statfs":{"total":96636764160,"available":96519041024,"internally_reserved":0,"allocated":40513536,"data_stored":44005439,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123212,"kb_used_data":60292,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248628,"statfs":{"total":96636764160,"available":96510595072,"internally_reserved":0,"allocated":61739008,"data_stored":72081704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:55.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:55.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:55.260+0000 7efe1b82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:55.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:55.266+0000 7efe1b82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:55.456 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:55.457 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:55.468 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2024-03-04T06:25:54.936490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":21,"acting":21,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628664,"kb_used_data":382552,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858696,"statfs":{"total":386547056640,"available":385903304704,"internally_reserved":0,"allocated":391733248,"data_stored":444324265,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001589"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2812,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:37.417194+0000","last_change":"2024-03-04T06:25:50.479118+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:37.417194+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:37.417194+0000","last_undegraded":"2024-03-04T06:25:37.417194+0000","last_fullsized":"2024-03-04T06:25:37.417194+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.446268+0000","last_change":"2024-03-04T06:25:50.494728+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:38.446268+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:38.446268+0000","last_undegraded":"2024-03-04T06:25:38.446268+0000","last_fullsized":"2024-03-04T06:25:38.446268+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.496410+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:24.496410+0000","last_peered":"2024-03-04T06:25:24.496410+0000","last_clean":"2024-03-04T06:25:24.496410+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:24.496410+0000","last_undegraded":"2024-03-04T06:25:24.496410+0000","last_fullsized":"2024-03-04T06:25:24.496410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.431398+0000","last_change":"2024-03-04T06:25:50.482833+0000","last_active":"2024-03-04T06:25:50.482833+0000","last_peered":"2024-03-04T06:25:38.431398+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:25:38.431398+0000","last_undegraded":"2024-03-04T06:25:38.431398+0000","last_fullsized":"2024-03-04T06:25:38.431398+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.492352+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:24.492352+0000","last_peered":"2024-03-04T06:25:24.492352+0000","last_clean":"2024-03-04T06:25:24.492352+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:24.492352+0000","last_undegraded":"2024-03-04T06:25:24.492352+0000","last_fullsized":"2024-03-04T06:25:24.492352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:07:50.329316+0000","last_change":"2024-03-04T06:25:50.488037+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:07:49.306852+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:07:50.329316+0000","last_undegraded":"2024-03-04T06:07:50.329316+0000","last_fullsized":"2024-03-04T06:07:50.329316+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2679,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:37.421082+0000","last_change":"2024-03-04T06:25:50.491700+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:37.421082+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:37.421082+0000","last_undegraded":"2024-03-04T06:25:37.421082+0000","last_fullsized":"2024-03-04T06:25:37.421082+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.504080+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:24.504080+0000","last_peered":"2024-03-04T06:25:24.504080+0000","last_clean":"2024-03-04T06:25:24.504080+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:50.470256+0000","last_undegraded":"2024-03-04T06:25:24.504080+0000","last_fullsized":"2024-03-04T06:25:24.504080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.441197+0000","last_change":"2024-03-04T06:25:50.493405+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:38.441197+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:38.441197+0000","last_undegraded":"2024-03-04T06:25:38.441197+0000","last_fullsized":"2024-03-04T06:25:38.441197+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114964,"kb_used_data":39564,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256876,"statfs":{"total":96636764160,"available":96519041024,"internally_reserved":0,"allocated":40513536,"data_stored":44005439,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123212,"kb_used_data":60292,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248628,"statfs":{"total":96636764160,"available":96510595072,"internally_reserved":0,"allocated":61739008,"data_stored":72081704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:55.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:55.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:55.572+0000 7f3d0075d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:55.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:55.578+0000 7f3d0075d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:55.768 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:55.769 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:55.781 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2024-03-04T06:25:54.936490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":21,"acting":21,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628664,"kb_used_data":382552,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858696,"statfs":{"total":386547056640,"available":385903304704,"internally_reserved":0,"allocated":391733248,"data_stored":444324265,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001589"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2812,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:37.417194+0000","last_change":"2024-03-04T06:25:50.479118+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:37.417194+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:37.417194+0000","last_undegraded":"2024-03-04T06:25:37.417194+0000","last_fullsized":"2024-03-04T06:25:37.417194+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.446268+0000","last_change":"2024-03-04T06:25:50.494728+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:38.446268+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:38.446268+0000","last_undegraded":"2024-03-04T06:25:38.446268+0000","last_fullsized":"2024-03-04T06:25:38.446268+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.496410+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:24.496410+0000","last_peered":"2024-03-04T06:25:24.496410+0000","last_clean":"2024-03-04T06:25:24.496410+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:24.496410+0000","last_undegraded":"2024-03-04T06:25:24.496410+0000","last_fullsized":"2024-03-04T06:25:24.496410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.431398+0000","last_change":"2024-03-04T06:25:50.482833+0000","last_active":"2024-03-04T06:25:50.482833+0000","last_peered":"2024-03-04T06:25:38.431398+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:25:38.431398+0000","last_undegraded":"2024-03-04T06:25:38.431398+0000","last_fullsized":"2024-03-04T06:25:38.431398+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.492352+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:24.492352+0000","last_peered":"2024-03-04T06:25:24.492352+0000","last_clean":"2024-03-04T06:25:24.492352+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:24.492352+0000","last_undegraded":"2024-03-04T06:25:24.492352+0000","last_fullsized":"2024-03-04T06:25:24.492352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:07:50.329316+0000","last_change":"2024-03-04T06:25:50.488037+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:07:49.306852+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:07:50.329316+0000","last_undegraded":"2024-03-04T06:07:50.329316+0000","last_fullsized":"2024-03-04T06:07:50.329316+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2679,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:37.421082+0000","last_change":"2024-03-04T06:25:50.491700+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:37.421082+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:37.421082+0000","last_undegraded":"2024-03-04T06:25:37.421082+0000","last_fullsized":"2024-03-04T06:25:37.421082+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.504080+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:24.504080+0000","last_peered":"2024-03-04T06:25:24.504080+0000","last_clean":"2024-03-04T06:25:24.504080+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:50.470256+0000","last_undegraded":"2024-03-04T06:25:24.504080+0000","last_fullsized":"2024-03-04T06:25:24.504080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.441197+0000","last_change":"2024-03-04T06:25:50.493405+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:38.441197+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:38.441197+0000","last_undegraded":"2024-03-04T06:25:38.441197+0000","last_fullsized":"2024-03-04T06:25:38.441197+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114964,"kb_used_data":39564,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256876,"statfs":{"total":96636764160,"available":96519041024,"internally_reserved":0,"allocated":40513536,"data_stored":44005439,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123212,"kb_used_data":60292,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248628,"statfs":{"total":96636764160,"available":96510595072,"internally_reserved":0,"allocated":61739008,"data_stored":72081704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:55.782 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:25:55.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2812, 'reported_epoch': 1063, 'state': 'peering', 'last_fresh': '2024-03-04T06:25:37.417194+0000', 'last_change': '2024-03-04T06:25:50.479118+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:37.417194+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:07:48.335407+0000', 'last_unstale': '2024-03-04T06:25:37.417194+0000', 'last_undegraded': '2024-03-04T06:25:37.417194+0000', 'last_fullsized': '2024-03-04T06:25:37.417194+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:25:55.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:25:55.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2615, 'reported_epoch': 1063, 'state': 'peering', 'last_fresh': '2024-03-04T06:25:38.446268+0000', 'last_change': '2024-03-04T06:25:50.494728+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:38.446268+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:07:50.381293+0000', 'last_unstale': '2024-03-04T06:25:38.446268+0000', 'last_undegraded': '2024-03-04T06:25:38.446268+0000', 'last_fullsized': '2024-03-04T06:25:38.446268+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:25:55.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:25:55.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2615, 'reported_epoch': 1063, 'state': 'peering', 'last_fresh': '2024-03-04T06:25:38.431398+0000', 'last_change': '2024-03-04T06:25:50.482833+0000', 'last_active': '2024-03-04T06:25:50.482833+0000', 'last_peered': '2024-03-04T06:25:38.431398+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:07:50.366678+0000', 'last_became_peered': '2024-03-04T06:07:50.366678+0000', 'last_unstale': '2024-03-04T06:25:38.431398+0000', 'last_undegraded': '2024-03-04T06:25:38.431398+0000', 'last_fullsized': '2024-03-04T06:25:38.431398+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:25:55.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:25:55.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 1063, 'state': 'peering', 'last_fresh': '2024-03-04T06:07:50.329316+0000', 'last_change': '2024-03-04T06:25:50.488037+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:07:49.306852+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T06:07:50.329316+0000', 'last_undegraded': '2024-03-04T06:07:50.329316+0000', 'last_fullsized': '2024-03-04T06:07:50.329316+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:25:55.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:25:55.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2679, 'reported_epoch': 1063, 'state': 'peering', 'last_fresh': '2024-03-04T06:25:37.421082+0000', 'last_change': '2024-03-04T06:25:50.491700+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:37.421082+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:07:50.386944+0000', 'last_unstale': '2024-03-04T06:25:37.421082+0000', 'last_undegraded': '2024-03-04T06:25:37.421082+0000', 'last_fullsized': '2024-03-04T06:25:37.421082+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:25:55.783 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:25:55.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 2702, 'reported_epoch': 1052, 'state': 'stale+active+clean', 'last_fresh': '2024-03-04T06:25:24.504080+0000', 'last_change': '2024-03-04T06:07:50.365236+0000', 'last_active': '2024-03-04T06:25:24.504080+0000', 'last_peered': '2024-03-04T06:25:24.504080+0000', 'last_clean': '2024-03-04T06:25:24.504080+0000', 'last_became_active': '2024-03-04T06:07:50.360536+0000', 'last_became_peered': '2024-03-04T06:07:50.360536+0000', 'last_unstale': '2024-03-04T06:25:50.470256+0000', 'last_undegraded': '2024-03-04T06:25:24.504080+0000', 'last_fullsized': '2024-03-04T06:25:24.504080+0000', 'mapping_epoch': 12, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3], 'acting': [2, 0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T06:25:55.783 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:25:55.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2615, 'reported_epoch': 1063, 'state': 'peering', 'last_fresh': '2024-03-04T06:25:38.441197+0000', 'last_change': '2024-03-04T06:25:50.493405+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:38.441197+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:07:50.392386+0000', 'last_unstale': '2024-03-04T06:25:38.441197+0000', 'last_undegraded': '2024-03-04T06:25:38.441197+0000', 'last_fullsized': '2024-03-04T06:25:38.441197+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:25:55.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:25:55.889 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:55.886+0000 7f9e4666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:55.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:55.891+0000 7f9e4666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:56.149 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:56.149 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive, 1 pg peering","count":2},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1097,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1064,"num_osds":4,"num_up_osds":3,"osd_up_since":1709533550,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":6},{"state_name":"active+clean","count":2},{"state_name":"stale+active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643751936,"bytes_avail":385903304704,"bytes_total":386547056640,"inactive_pgs_ratio":0.66666668653488159},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:25:56.162 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:25:56.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:56.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:56.264+0000 7f3c4391d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:56.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:56.270+0000 7f3c4391d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:56.457 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:56.458 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:56.470 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1600,"stamp":"2024-03-04T06:25:54.936490+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":21,"acting":21,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628664,"kb_used_data":382552,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858696,"statfs":{"total":386547056640,"available":385903304704,"internally_reserved":0,"allocated":391733248,"data_stored":444324265,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001589"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2812,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:37.417194+0000","last_change":"2024-03-04T06:25:50.479118+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:37.417194+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:37.417194+0000","last_undegraded":"2024-03-04T06:25:37.417194+0000","last_fullsized":"2024-03-04T06:25:37.417194+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.446268+0000","last_change":"2024-03-04T06:25:50.494728+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:38.446268+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:38.446268+0000","last_undegraded":"2024-03-04T06:25:38.446268+0000","last_fullsized":"2024-03-04T06:25:38.446268+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.496410+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:24.496410+0000","last_peered":"2024-03-04T06:25:24.496410+0000","last_clean":"2024-03-04T06:25:24.496410+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:24.496410+0000","last_undegraded":"2024-03-04T06:25:24.496410+0000","last_fullsized":"2024-03-04T06:25:24.496410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.431398+0000","last_change":"2024-03-04T06:25:50.482833+0000","last_active":"2024-03-04T06:25:50.482833+0000","last_peered":"2024-03-04T06:25:38.431398+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:07:50.366678+0000","last_became_peered":"2024-03-04T06:07:50.366678+0000","last_unstale":"2024-03-04T06:25:38.431398+0000","last_undegraded":"2024-03-04T06:25:38.431398+0000","last_fullsized":"2024-03-04T06:25:38.431398+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"active+clean","last_fresh":"2024-03-04T06:25:24.492352+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:24.492352+0000","last_peered":"2024-03-04T06:25:24.492352+0000","last_clean":"2024-03-04T06:25:24.492352+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:24.492352+0000","last_undegraded":"2024-03-04T06:25:24.492352+0000","last_fullsized":"2024-03-04T06:25:24.492352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:07:50.329316+0000","last_change":"2024-03-04T06:25:50.488037+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:07:49.306852+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:07:50.329316+0000","last_undegraded":"2024-03-04T06:07:50.329316+0000","last_fullsized":"2024-03-04T06:07:50.329316+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2679,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:37.421082+0000","last_change":"2024-03-04T06:25:50.491700+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:37.421082+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:37.421082+0000","last_undegraded":"2024-03-04T06:25:37.421082+0000","last_fullsized":"2024-03-04T06:25:37.421082+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.504080+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:24.504080+0000","last_peered":"2024-03-04T06:25:24.504080+0000","last_clean":"2024-03-04T06:25:24.504080+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:50.470256+0000","last_undegraded":"2024-03-04T06:25:24.504080+0000","last_fullsized":"2024-03-04T06:25:24.504080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2615,"reported_epoch":1063,"state":"peering","last_fresh":"2024-03-04T06:25:38.441197+0000","last_change":"2024-03-04T06:25:50.493405+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:38.441197+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:38.441197+0000","last_undegraded":"2024-03-04T06:25:38.441197+0000","last_fullsized":"2024-03-04T06:25:38.441197+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114964,"kb_used_data":39564,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256876,"statfs":{"total":96636764160,"available":96519041024,"internally_reserved":0,"allocated":40513536,"data_stored":44005439,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123212,"kb_used_data":60292,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248628,"statfs":{"total":96636764160,"available":96510595072,"internally_reserved":0,"allocated":61739008,"data_stored":72081704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:57.088 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2024-03-04T06:25:57.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~27s 2024-03-04T06:25:58.188 INFO:teuthology.orchestra.run.smithi064.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-04T06:25:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-04T06:25:59.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:59.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:59.590+0000 7f477059d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:59.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:59.596+0000 7f477059d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:59.787 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:25:59.788 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:25:59.800 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1604,"stamp":"2024-03-04T06:25:58.937261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":21,"acting":21,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628680,"kb_used_data":382568,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858680,"statfs":{"total":386547056640,"available":385903288320,"internally_reserved":0,"allocated":391749632,"data_stored":444330481,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.461578"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2813,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.479118+0000","last_change":"2024-03-04T06:25:50.479118+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:37.417194+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:50.479118+0000","last_undegraded":"2024-03-04T06:25:50.479118+0000","last_fullsized":"2024-03-04T06:25:50.479118+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2616,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.494728+0000","last_change":"2024-03-04T06:25:50.494728+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:38.446268+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:50.494728+0000","last_undegraded":"2024-03-04T06:25:50.494728+0000","last_fullsized":"2024-03-04T06:25:50.494728+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.496410+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:24.496410+0000","last_peered":"2024-03-04T06:25:24.496410+0000","last_clean":"2024-03-04T06:25:24.496410+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:57.366259+0000","last_undegraded":"2024-03-04T06:25:24.496410+0000","last_fullsized":"2024-03-04T06:25:24.496410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2621,"reported_epoch":1064,"state":"active+undersized","last_fresh":"2024-03-04T06:25:51.500659+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:51.500659+0000","last_peered":"2024-03-04T06:25:51.500659+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:51.500659+0000","last_undegraded":"2024-03-04T06:25:51.500659+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.492352+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:24.492352+0000","last_peered":"2024-03-04T06:25:24.492352+0000","last_clean":"2024-03-04T06:25:24.492352+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:57.366262+0000","last_undegraded":"2024-03-04T06:25:24.492352+0000","last_fullsized":"2024-03-04T06:25:24.492352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":4,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.488037+0000","last_change":"2024-03-04T06:25:50.488037+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:07:49.306852+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:25:50.488037+0000","last_undegraded":"2024-03-04T06:25:50.488037+0000","last_fullsized":"2024-03-04T06:25:50.488037+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2680,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.491700+0000","last_change":"2024-03-04T06:25:50.491700+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:37.421082+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:50.491700+0000","last_undegraded":"2024-03-04T06:25:50.491700+0000","last_fullsized":"2024-03-04T06:25:50.491700+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.504080+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:24.504080+0000","last_peered":"2024-03-04T06:25:24.504080+0000","last_clean":"2024-03-04T06:25:24.504080+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:50.470256+0000","last_undegraded":"2024-03-04T06:25:24.504080+0000","last_fullsized":"2024-03-04T06:25:24.504080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2616,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.493405+0000","last_change":"2024-03-04T06:25:50.493405+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:38.441197+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:50.493405+0000","last_undegraded":"2024-03-04T06:25:50.493405+0000","last_fullsized":"2024-03-04T06:25:50.493405+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114972,"kb_used_data":39572,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256868,"statfs":{"total":96636764160,"available":96519032832,"internally_reserved":0,"allocated":40521728,"data_stored":44008547,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123220,"kb_used_data":60300,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248620,"statfs":{"total":96636764160,"available":96510586880,"internally_reserved":0,"allocated":61747200,"data_stored":72084812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:25:59.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:25:59.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:59.904+0000 7f207b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:25:59.911 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:25:59.911+0000 7f207b15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:00.098 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:00.099 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:00.111 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1604,"stamp":"2024-03-04T06:25:58.937261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":21,"acting":21,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628680,"kb_used_data":382568,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858680,"statfs":{"total":386547056640,"available":385903288320,"internally_reserved":0,"allocated":391749632,"data_stored":444330481,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.461578"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2813,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.479118+0000","last_change":"2024-03-04T06:25:50.479118+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:37.417194+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:50.479118+0000","last_undegraded":"2024-03-04T06:25:50.479118+0000","last_fullsized":"2024-03-04T06:25:50.479118+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2616,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.494728+0000","last_change":"2024-03-04T06:25:50.494728+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:38.446268+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:50.494728+0000","last_undegraded":"2024-03-04T06:25:50.494728+0000","last_fullsized":"2024-03-04T06:25:50.494728+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.496410+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:24.496410+0000","last_peered":"2024-03-04T06:25:24.496410+0000","last_clean":"2024-03-04T06:25:24.496410+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:57.366259+0000","last_undegraded":"2024-03-04T06:25:24.496410+0000","last_fullsized":"2024-03-04T06:25:24.496410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2621,"reported_epoch":1064,"state":"active+undersized","last_fresh":"2024-03-04T06:25:51.500659+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:51.500659+0000","last_peered":"2024-03-04T06:25:51.500659+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:51.500659+0000","last_undegraded":"2024-03-04T06:25:51.500659+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.492352+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:24.492352+0000","last_peered":"2024-03-04T06:25:24.492352+0000","last_clean":"2024-03-04T06:25:24.492352+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:57.366262+0000","last_undegraded":"2024-03-04T06:25:24.492352+0000","last_fullsized":"2024-03-04T06:25:24.492352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":4,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.488037+0000","last_change":"2024-03-04T06:25:50.488037+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:07:49.306852+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:25:50.488037+0000","last_undegraded":"2024-03-04T06:25:50.488037+0000","last_fullsized":"2024-03-04T06:25:50.488037+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2680,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.491700+0000","last_change":"2024-03-04T06:25:50.491700+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:37.421082+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:50.491700+0000","last_undegraded":"2024-03-04T06:25:50.491700+0000","last_fullsized":"2024-03-04T06:25:50.491700+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.504080+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:24.504080+0000","last_peered":"2024-03-04T06:25:24.504080+0000","last_clean":"2024-03-04T06:25:24.504080+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:50.470256+0000","last_undegraded":"2024-03-04T06:25:24.504080+0000","last_fullsized":"2024-03-04T06:25:24.504080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2616,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.493405+0000","last_change":"2024-03-04T06:25:50.493405+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:38.441197+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:50.493405+0000","last_undegraded":"2024-03-04T06:25:50.493405+0000","last_fullsized":"2024-03-04T06:25:50.493405+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114972,"kb_used_data":39572,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256868,"statfs":{"total":96636764160,"available":96519032832,"internally_reserved":0,"allocated":40521728,"data_stored":44008547,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123220,"kb_used_data":60300,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248620,"statfs":{"total":96636764160,"available":96510586880,"internally_reserved":0,"allocated":61747200,"data_stored":72084812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:00.111 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:00.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2813, 'reported_epoch': 1064, 'state': 'peering', 'last_fresh': '2024-03-04T06:25:50.479118+0000', 'last_change': '2024-03-04T06:25:50.479118+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:37.417194+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:07:48.335407+0000', 'last_unstale': '2024-03-04T06:25:50.479118+0000', 'last_undegraded': '2024-03-04T06:25:50.479118+0000', 'last_fullsized': '2024-03-04T06:25:50.479118+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:00.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:00.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2616, 'reported_epoch': 1064, 'state': 'peering', 'last_fresh': '2024-03-04T06:25:50.494728+0000', 'last_change': '2024-03-04T06:25:50.494728+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:38.446268+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:07:50.381293+0000', 'last_unstale': '2024-03-04T06:25:50.494728+0000', 'last_undegraded': '2024-03-04T06:25:50.494728+0000', 'last_fullsized': '2024-03-04T06:25:50.494728+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:00.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:00.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 2582, 'reported_epoch': 1052, 'state': 'stale+active+clean', 'last_fresh': '2024-03-04T06:25:24.496410+0000', 'last_change': '2024-03-04T06:07:50.364545+0000', 'last_active': '2024-03-04T06:25:24.496410+0000', 'last_peered': '2024-03-04T06:25:24.496410+0000', 'last_clean': '2024-03-04T06:25:24.496410+0000', 'last_became_active': '2024-03-04T06:07:50.360433+0000', 'last_became_peered': '2024-03-04T06:07:50.360433+0000', 'last_unstale': '2024-03-04T06:25:57.366259+0000', 'last_undegraded': '2024-03-04T06:25:24.496410+0000', 'last_fullsized': '2024-03-04T06:25:24.496410+0000', 'mapping_epoch': 12, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1, 2], 'acting': [0, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T06:26:00.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:00.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2621, 'reported_epoch': 1064, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:51.500659+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:51.500659+0000', 'last_peered': '2024-03-04T06:25:51.500659+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:51.500659+0000', 'last_undegraded': '2024-03-04T06:25:51.500659+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:00.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:00.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 2582, 'reported_epoch': 1052, 'state': 'stale+active+clean', 'last_fresh': '2024-03-04T06:25:24.492352+0000', 'last_change': '2024-03-04T06:07:50.384457+0000', 'last_active': '2024-03-04T06:25:24.492352+0000', 'last_peered': '2024-03-04T06:25:24.492352+0000', 'last_clean': '2024-03-04T06:25:24.492352+0000', 'last_became_active': '2024-03-04T06:07:50.379292+0000', 'last_became_peered': '2024-03-04T06:07:50.379292+0000', 'last_unstale': '2024-03-04T06:25:57.366262+0000', 'last_undegraded': '2024-03-04T06:25:24.492352+0000', 'last_fullsized': '2024-03-04T06:25:24.492352+0000', 'mapping_epoch': 12, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2, 1], 'acting': [0, 2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T06:26:00.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:00.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 4, 'reported_epoch': 1064, 'state': 'peering', 'last_fresh': '2024-03-04T06:25:50.488037+0000', 'last_change': '2024-03-04T06:25:50.488037+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:07:49.306852+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T06:25:50.488037+0000', 'last_undegraded': '2024-03-04T06:25:50.488037+0000', 'last_fullsized': '2024-03-04T06:25:50.488037+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:00.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:00.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2680, 'reported_epoch': 1064, 'state': 'peering', 'last_fresh': '2024-03-04T06:25:50.491700+0000', 'last_change': '2024-03-04T06:25:50.491700+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:37.421082+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:07:50.386944+0000', 'last_unstale': '2024-03-04T06:25:50.491700+0000', 'last_undegraded': '2024-03-04T06:25:50.491700+0000', 'last_fullsized': '2024-03-04T06:25:50.491700+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:00.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:00.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 2702, 'reported_epoch': 1052, 'state': 'stale+active+clean', 'last_fresh': '2024-03-04T06:25:24.504080+0000', 'last_change': '2024-03-04T06:07:50.365236+0000', 'last_active': '2024-03-04T06:25:24.504080+0000', 'last_peered': '2024-03-04T06:25:24.504080+0000', 'last_clean': '2024-03-04T06:25:24.504080+0000', 'last_became_active': '2024-03-04T06:07:50.360536+0000', 'last_became_peered': '2024-03-04T06:07:50.360536+0000', 'last_unstale': '2024-03-04T06:25:50.470256+0000', 'last_undegraded': '2024-03-04T06:25:24.504080+0000', 'last_fullsized': '2024-03-04T06:25:24.504080+0000', 'mapping_epoch': 12, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3], 'acting': [2, 0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T06:26:00.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:00.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2616, 'reported_epoch': 1064, 'state': 'peering', 'last_fresh': '2024-03-04T06:25:50.493405+0000', 'last_change': '2024-03-04T06:25:50.493405+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:38.441197+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:07:50.392386+0000', 'last_unstale': '2024-03-04T06:25:50.493405+0000', 'last_undegraded': '2024-03-04T06:25:50.493405+0000', 'last_fullsized': '2024-03-04T06:25:50.493405+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:00.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:00.216+0000 7fd36c75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:00.221+0000 7fd36c75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:00.479 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:00.479 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive, 1 pg peering","count":2},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1102,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":5},{"state_name":"stale+active+clean","count":3},{"state_name":"active+undersized","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643768320,"bytes_avail":385903288320,"bytes_total":386547056640,"inactive_pgs_ratio":0.55555558204650879},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:00.492 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:00.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:00.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:00.595+0000 7f2424064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:00.601 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:00.601+0000 7f2424064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:00.789 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:00.801 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1604,"stamp":"2024-03-04T06:25:58.937261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":21,"acting":21,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628680,"kb_used_data":382568,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858680,"statfs":{"total":386547056640,"available":385903288320,"internally_reserved":0,"allocated":391749632,"data_stored":444330481,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.461578"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2813,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.479118+0000","last_change":"2024-03-04T06:25:50.479118+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:37.417194+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:07:48.335407+0000","last_unstale":"2024-03-04T06:25:50.479118+0000","last_undegraded":"2024-03-04T06:25:50.479118+0000","last_fullsized":"2024-03-04T06:25:50.479118+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2616,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.494728+0000","last_change":"2024-03-04T06:25:50.494728+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:38.446268+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:07:50.381293+0000","last_unstale":"2024-03-04T06:25:50.494728+0000","last_undegraded":"2024-03-04T06:25:50.494728+0000","last_fullsized":"2024-03-04T06:25:50.494728+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.496410+0000","last_change":"2024-03-04T06:07:50.364545+0000","last_active":"2024-03-04T06:25:24.496410+0000","last_peered":"2024-03-04T06:25:24.496410+0000","last_clean":"2024-03-04T06:25:24.496410+0000","last_became_active":"2024-03-04T06:07:50.360433+0000","last_became_peered":"2024-03-04T06:07:50.360433+0000","last_unstale":"2024-03-04T06:25:57.366259+0000","last_undegraded":"2024-03-04T06:25:24.496410+0000","last_fullsized":"2024-03-04T06:25:24.496410+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2621,"reported_epoch":1064,"state":"active+undersized","last_fresh":"2024-03-04T06:25:51.500659+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:51.500659+0000","last_peered":"2024-03-04T06:25:51.500659+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:51.500659+0000","last_undegraded":"2024-03-04T06:25:51.500659+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2582,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.492352+0000","last_change":"2024-03-04T06:07:50.384457+0000","last_active":"2024-03-04T06:25:24.492352+0000","last_peered":"2024-03-04T06:25:24.492352+0000","last_clean":"2024-03-04T06:25:24.492352+0000","last_became_active":"2024-03-04T06:07:50.379292+0000","last_became_peered":"2024-03-04T06:07:50.379292+0000","last_unstale":"2024-03-04T06:25:57.366262+0000","last_undegraded":"2024-03-04T06:25:24.492352+0000","last_fullsized":"2024-03-04T06:25:24.492352+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":4,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.488037+0000","last_change":"2024-03-04T06:25:50.488037+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:07:49.306852+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T06:25:50.488037+0000","last_undegraded":"2024-03-04T06:25:50.488037+0000","last_fullsized":"2024-03-04T06:25:50.488037+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2680,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.491700+0000","last_change":"2024-03-04T06:25:50.491700+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:37.421082+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:07:50.386944+0000","last_unstale":"2024-03-04T06:25:50.491700+0000","last_undegraded":"2024-03-04T06:25:50.491700+0000","last_fullsized":"2024-03-04T06:25:50.491700+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2702,"reported_epoch":1052,"state":"stale+active+clean","last_fresh":"2024-03-04T06:25:24.504080+0000","last_change":"2024-03-04T06:07:50.365236+0000","last_active":"2024-03-04T06:25:24.504080+0000","last_peered":"2024-03-04T06:25:24.504080+0000","last_clean":"2024-03-04T06:25:24.504080+0000","last_became_active":"2024-03-04T06:07:50.360536+0000","last_became_peered":"2024-03-04T06:07:50.360536+0000","last_unstale":"2024-03-04T06:25:50.470256+0000","last_undegraded":"2024-03-04T06:25:24.504080+0000","last_fullsized":"2024-03-04T06:25:24.504080+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2616,"reported_epoch":1064,"state":"peering","last_fresh":"2024-03-04T06:25:50.493405+0000","last_change":"2024-03-04T06:25:50.493405+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:38.441197+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:07:50.392386+0000","last_unstale":"2024-03-04T06:25:50.493405+0000","last_undegraded":"2024-03-04T06:25:50.493405+0000","last_fullsized":"2024-03-04T06:25:50.493405+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114972,"kb_used_data":39572,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256868,"statfs":{"total":96636764160,"available":96519032832,"internally_reserved":0,"allocated":40521728,"data_stored":44008547,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123220,"kb_used_data":60300,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248620,"statfs":{"total":96636764160,"available":96510586880,"internally_reserved":0,"allocated":61747200,"data_stored":72084812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:02.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2024-03-04T06:26:02.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~32s 2024-03-04T06:26:03.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:03.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:03.920+0000 7f70b84dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:03.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:03.927+0000 7f70b84dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:04.116 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:04.116 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:04.127 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1606,"stamp":"2024-03-04T06:26:02.937977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001487"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:04.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:04.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:04.235+0000 7f693dd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:04.241 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:04.241+0000 7f693dd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:04.428 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:04.428 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:04.441 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1606,"stamp":"2024-03-04T06:26:02.937977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001487"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:04.441 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:04.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:04.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:04.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:04.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:04.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:04.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:04.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:04.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:04.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:04.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:04.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:04.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:04.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:04.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:04.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:04.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:04.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:04.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:04.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:04.547+0000 7f60c9f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:04.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:04.552+0000 7f60c9f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:04.811 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:04.811 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 4 pgs inactive","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1106,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:04.824 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:04.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:04.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:04.927+0000 7f3e39add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:04.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:04.933+0000 7f3e39add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:05.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:05.124 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:05.136 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1607,"stamp":"2024-03-04T06:26:04.938318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001457"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:07.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~38s 2024-03-04T06:26:07.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~38s 2024-03-04T06:26:08.138 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:08.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:08.255+0000 7f1c506dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:08.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:08.262+0000 7f1c506dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:08.453 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:08.454 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:08.466 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1608,"stamp":"2024-03-04T06:26:06.938729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.572481"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:08.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:08.574 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:08.570+0000 7f1b34d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:08.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:08.576+0000 7f1b34d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:08.764 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:08.764 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:08.776 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1608,"stamp":"2024-03-04T06:26:06.938729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.572481"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:08.777 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:08.777 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:08.777 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:08.777 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:08.777 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:08.777 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:08.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:08.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:08.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:08.882+0000 7f6a6b02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:08.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:08.887+0000 7f6a6b02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:09.147 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:09.147 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 4 pgs inactive","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1110,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:09.160 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:09.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:09.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:09.264+0000 7f615462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:09.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:09.270+0000 7f615462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:09.461 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:09.461 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:09.473 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1609,"stamp":"2024-03-04T06:26:08.939209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.574445"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:12.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:12.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:12.590+0000 7ff628c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:12.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:12.597+0000 7ff628c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:12.790 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:12.790 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:12.802 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1610,"stamp":"2024-03-04T06:26:10.939537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002275"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:12.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:12.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:12.905+0000 7f8d1b66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:12.913 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:12.911+0000 7f8d1b66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:13.099 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:13.099 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:13.111 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1611,"stamp":"2024-03-04T06:26:12.939946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002339"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:13.112 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:13.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:13.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:13.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:13.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:13.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:13.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:13.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:13.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:13.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:13.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:13.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:13.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:13.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:13.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:13.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:13.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:13.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:13.113 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:13.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:13.216+0000 7f75b639d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:13.222 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:13.221+0000 7f75b639d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:13.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~43s 2024-03-04T06:26:13.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~43s 2024-03-04T06:26:13.479 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:13.479 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 4 pgs inactive","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1115,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:13.492 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:13.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:13.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:13.594+0000 7fdf11514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:13.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:13.600+0000 7fdf11514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:13.787 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:13.788 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:13.800 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1611,"stamp":"2024-03-04T06:26:12.939946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002339"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:16.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:16.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:16.919+0000 7f11d995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:16.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:16.926+0000 7f11d995d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:17.112 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:17.113 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:17.124 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1613,"stamp":"2024-03-04T06:26:16.940872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002553"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:17.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:17.233 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:17.229+0000 7f3ef819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:17.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:17.235+0000 7f3ef819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:17.422 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:17.423 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:17.435 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1613,"stamp":"2024-03-04T06:26:16.940872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002553"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:17.436 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:17.436 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:17.436 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:17.436 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:17.436 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:17.436 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:17.437 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:17.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:17.544 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:17.541+0000 7fc95b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:17.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:17.547+0000 7fc95b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:17.804 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:17.804 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 4 pgs inactive","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1119,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:17.817 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:17.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:17.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:17.920+0000 7f8c33a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:17.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:17.926+0000 7f8c33a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:18.117 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:18.117 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:18.129 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1613,"stamp":"2024-03-04T06:26:16.940872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002553"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:18.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~49s 2024-03-04T06:26:18.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~49s 2024-03-04T06:26:21.130 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:21.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:21.247+0000 7f01846dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:21.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:21.254+0000 7f01846dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:21.444 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:21.456 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1615,"stamp":"2024-03-04T06:26:20.941759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002549"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:21.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:21.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:21.561+0000 7fa19ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:21.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:21.567+0000 7fa19ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:21.754 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:21.754 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:21.766 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1615,"stamp":"2024-03-04T06:26:20.941759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002549"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:21.767 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:21.767 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:21.767 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:21.767 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:21.768 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:21.768 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:21.768 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:21.768 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:21.768 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:21.768 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:21.768 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:21.768 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:21.768 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:21.768 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:21.769 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:21.769 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:21.769 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:21.769 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:21.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:21.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:21.872+0000 7f277a22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:21.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:21.877+0000 7f277a22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:22.135 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 4 pgs inactive","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1123,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:22.147 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:22.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:22.255 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:22.251+0000 7f9beb19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:22.258 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:22.257+0000 7f9beb19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:22.449 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:22.450 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:22.462 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1615,"stamp":"2024-03-04T06:26:20.941759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002549"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706774,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:24.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~54s 2024-03-04T06:26:24.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~54s 2024-03-04T06:26:25.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:25.584 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:25.581+0000 7f18ba424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:25.587 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:25.587+0000 7f18ba424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:25.775 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:25.788 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1617,"stamp":"2024-03-04T06:26:24.942561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002614"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:25.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:25.895 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:25.891+0000 7f6a4806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:25.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:25.897+0000 7f6a4806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:26.086 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:26.087 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:26.099 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1617,"stamp":"2024-03-04T06:26:24.942561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002614"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:26.100 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:26.100 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:26.100 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:26.100 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:26.100 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:26.100 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:26.100 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:26.100 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:26.101 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:26.101 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:26.101 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:26.101 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:26.101 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:26.101 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:26.101 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:26.101 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:26.101 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:26.101 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:26.102 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:26.207 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:26.204+0000 7f2238f07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:26.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:26.209+0000 7f2238f07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:26.468 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 4 pgs inactive","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1128,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:26.483 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:26.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:26.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:26.586+0000 7f8481f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:26.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:26.592+0000 7f8481f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:26.784 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:26.785 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:26.797 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1617,"stamp":"2024-03-04T06:26:24.942561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002614"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:29.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~59s 2024-03-04T06:26:29.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~59s 2024-03-04T06:26:29.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:29.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:29.917+0000 7f454079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:29.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:29.923+0000 7f454079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:30.116 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:30.116 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:30.129 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1619,"stamp":"2024-03-04T06:26:28.943449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002577"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706784,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706784,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:30.130 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:30.237 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:30.233+0000 7f916db9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:30.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:30.239+0000 7f916db9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:30.428 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:30.429 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:30.441 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1619,"stamp":"2024-03-04T06:26:28.943449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002577"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706784,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706784,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:30.442 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:30.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:30.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:30.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:30.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:30.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:30.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:30.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:30.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:30.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:30.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:30.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:30.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:30.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:30.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:30.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:30.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:30.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:30.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:30.554 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:30.551+0000 7f3cdd510640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:30.557 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:30.556+0000 7f3cdd510640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:30.814 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 4 pgs inactive","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1132,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:30.826 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:30.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:30.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:30.929+0000 7f78c375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:30.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:30.935+0000 7f78c375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:31.123 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:31.124 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:31.136 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1620,"stamp":"2024-03-04T06:26:30.943927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002594"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706784,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706784,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:34.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:34.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:34.255+0000 7f2237b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:34.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:34.261+0000 7f2237b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:34.454 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:34.455 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:34.466 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1621,"stamp":"2024-03-04T06:26:32.944330+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002570"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:34.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:34.571+0000 7fcc2d707640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:34.577+0000 7fcc2d707640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:34.767 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:34.768 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:34.780 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1621,"stamp":"2024-03-04T06:26:32.944330+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002570"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:34.781 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:34.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:34.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:34.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:34.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:34.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:34.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:34.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:34.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:34.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:34.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:34.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:34.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:34.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:34.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:34.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:34.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:34.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:34.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:34.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:34.886+0000 7f0aaded4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:34.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:34.891+0000 7f0aaded4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:34.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~65s 2024-03-04T06:26:34.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~65s 2024-03-04T06:26:35.148 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:35.148 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 4 pgs inactive","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1136,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:35.161 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:35.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:35.268 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:35.264+0000 7f75af62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:35.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:35.271+0000 7f75af62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:35.462 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:35.474 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1622,"stamp":"2024-03-04T06:26:34.944817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002664"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706789,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:38.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:38.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:38.593+0000 7f7d92f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:38.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:38.600+0000 7f7d92f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:38.790 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:38.790 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:38.802 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1623,"stamp":"2024-03-04T06:26:36.945323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002761"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:38.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:38.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:38.906+0000 7f7d4e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:38.915 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:38.912+0000 7f7d4e46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:39.100 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:39.100 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:39.112 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1624,"stamp":"2024-03-04T06:26:38.945778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002791"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:39.113 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:39.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:39.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:39.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:39.114 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:39.114 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:39.114 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:39.114 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:39.114 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:39.114 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:39.114 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:39.114 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:39.114 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:39.115 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:39.115 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:39.115 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:39.115 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:39.115 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:39.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:39.221 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:39.218+0000 7fb636e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:39.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:39.224+0000 7fb636e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:39.480 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:39.480 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 4 pgs inactive","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1141,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:39.492 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:39.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:39.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:39.596+0000 7f092199d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:39.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:39.602+0000 7f092199d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:39.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:39.791 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:39.803 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1624,"stamp":"2024-03-04T06:26:38.945778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002791"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:40.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~70s 2024-03-04T06:26:40.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~70s 2024-03-04T06:26:42.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:42.925 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:42.921+0000 7fb8b482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:42.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:42.927+0000 7fb8b482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:43.115 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:43.116 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:43.128 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2024-03-04T06:26:42.946615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002688"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706799,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706799,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:43.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:43.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:43.232+0000 7fd01935d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:43.238 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:43.238+0000 7fd01935d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:43.430 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:43.441 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2024-03-04T06:26:42.946615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002688"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706799,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706799,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:43.441 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:43.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:43.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:43.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:43.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:43.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:43.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:43.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:43.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:43.442 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:43.442 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:43.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:43.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:43.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:43.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:43.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:43.443 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:43.443 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:43.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:43.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:43.547+0000 7f82d815d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:43.552 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:43.552+0000 7f82d815d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:43.808 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:43.811 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 4 pgs inactive","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1145,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:43.820 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:43.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:43.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:43.925+0000 7fa8d679d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:43.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:43.931+0000 7fa8d679d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:44.121 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:44.121 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:44.133 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2024-03-04T06:26:42.946615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002688"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706799,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706799,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:45.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~76s 2024-03-04T06:26:45.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~76s 2024-03-04T06:26:47.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:47.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:47.253+0000 7f473ce64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:47.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:47.259+0000 7f473ce64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:47.448 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:47.448 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:47.460 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1628,"stamp":"2024-03-04T06:26:46.947503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002685"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:47.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:47.569 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:47.565+0000 7ff2ba51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:47.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:47.571+0000 7ff2ba51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:47.762 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:47.774 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1628,"stamp":"2024-03-04T06:26:46.947503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002685"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:47.775 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:47.775 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:47.775 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:47.775 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:47.775 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:47.775 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:47.776 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:47.776 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:47.776 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:47.776 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:47.776 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:47.776 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:47.776 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:47.776 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:47.776 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:47.776 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:47.777 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:47.777 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:47.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:47.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:47.879+0000 7f8ea835d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:47.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:47.884+0000 7f8ea835d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:48.145 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:48.145 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 4 pgs inactive","count":4},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1149,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:48.157 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:48.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:48.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:48.261+0000 7f7b90d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:48.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:48.267+0000 7f7b90d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:48.456 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:48.468 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1628,"stamp":"2024-03-04T06:26:46.947503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002685"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:51.133 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~81s 2024-03-04T06:26:51.133 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~81s 2024-03-04T06:26:51.470 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:51.590 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:51.587+0000 7f03c6824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:51.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:51.594+0000 7f03c6824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:51.785 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:51.785 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:51.797 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1630,"stamp":"2024-03-04T06:26:50.948517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002739"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:51.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:51.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:51.901+0000 7effa215d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:51.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:51.907+0000 7effa215d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:52.095 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:52.095 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:52.107 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1630,"stamp":"2024-03-04T06:26:50.948517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002739"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:52.108 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:52.108 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:52.108 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:52.108 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:52.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:52.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:52.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:52.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:52.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:52.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:52.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:52.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:52.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:52.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:52.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:52.110 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:52.110 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:52.110 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:52.110 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:52.217 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:52.214+0000 7f88f791d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:52.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:52.219+0000 7f88f791d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:52.484 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:52.484 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1154,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:52.498 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:52.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:52.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:52.601+0000 7fde16f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:52.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:52.608+0000 7fde15d3b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:52.797 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:52.797 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:52.809 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1630,"stamp":"2024-03-04T06:26:50.948517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002739"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:55.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:55.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:55.928+0000 7f6b4e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:55.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:55.934+0000 7f6b4e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:56.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:56.125 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:56.136 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1632,"stamp":"2024-03-04T06:26:54.949441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002825"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:56.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:56.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:56.240+0000 7fd11a46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:56.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:56.246+0000 7fd11a46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:56.435 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:56.436 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:56.448 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1632,"stamp":"2024-03-04T06:26:54.949441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002825"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706809,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:26:56.449 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:26:56.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:56.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:26:56.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:56.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:26:56.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:56.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:26:56.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:56.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:26:56.450 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:56.450 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:26:56.450 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:56.450 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:26:56.450 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:56.450 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:26:56.450 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:26:56.450 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:26:56.450 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:26:56.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:26:56.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~86s 2024-03-04T06:26:56.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~86s 2024-03-04T06:26:56.556 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:56.554+0000 7f182cd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:56.559 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:56.559+0000 7f182cd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:56.819 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:56.819 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1158,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:26:56.832 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:26:56.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:26:56.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:56.936+0000 7f4925f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:56.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:26:56.942+0000 7f4925f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:26:57.131 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:26:57.132 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:26:57.144 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1633,"stamp":"2024-03-04T06:26:56.949842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:00.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:00.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:00.263+0000 7fc9bc82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:00.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:00.270+0000 7fc9bc82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:00.459 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:00.460 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:00.471 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1634,"stamp":"2024-03-04T06:26:58.950331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002828"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:00.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:00.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:00.575+0000 7f889306d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:00.582 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:00.582+0000 7f889306d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:00.768 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:00.780 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1634,"stamp":"2024-03-04T06:26:58.950331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002828"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:00.780 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:00.780 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:00.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:00.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:00.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:00.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:00.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:00.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:00.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:00.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:00.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:00.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:00.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:00.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:00.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:00.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:00.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:00.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:00.782 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:00.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:00.884+0000 7fa4a922d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:00.890 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:00.889+0000 7fa4a922d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:01.147 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:01.148 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1162,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:01.160 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:01.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:01.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:01.263+0000 7fed60b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:01.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:01.269+0000 7fed60b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:01.461 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:01.461 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:01.473 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2024-03-04T06:27:00.950706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002714"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706814,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:01.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~92s 2024-03-04T06:27:01.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~92s 2024-03-04T06:27:04.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:04.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:04.591+0000 7f43eb35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:04.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:04.598+0000 7f43eb35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:04.786 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:04.786 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:04.798 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1636,"stamp":"2024-03-04T06:27:02.951120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002602"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:04.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:04.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:04.903+0000 7f23306dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:04.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:04.909+0000 7f23306dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:05.102 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:05.102 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:05.115 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2024-03-04T06:27:04.951624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002642"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:05.116 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:05.116 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:05.116 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:05.116 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:05.116 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:05.116 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:05.116 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:05.116 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:05.117 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:05.117 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:05.117 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:05.117 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:05.117 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:05.117 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:05.117 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:05.117 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:05.117 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:05.117 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:05.118 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:05.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:05.220+0000 7f87c222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:05.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:05.225+0000 7f87c222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:05.486 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:05.486 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1167,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:05.499 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:05.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:05.605 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:05.601+0000 7f6c6a107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:05.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:05.608+0000 7f6c6a107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:05.794 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:05.795 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:05.807 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2024-03-04T06:27:04.951624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002642"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706819,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:07.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~97s 2024-03-04T06:27:07.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~97s 2024-03-04T06:27:08.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:08.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:08.925+0000 7f02e979d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:08.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:08.931+0000 7f02e979d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:09.123 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:09.123 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:09.137 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2024-03-04T06:27:08.952658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002815"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:09.138 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:09.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:09.240+0000 7efc32c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:09.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:09.247+0000 7efc32c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:09.437 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:09.437 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:09.449 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2024-03-04T06:27:08.952658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002815"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:09.450 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:09.450 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:09.450 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:09.450 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:09.450 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:09.451 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:09.451 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:09.451 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:09.451 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:09.451 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:09.451 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:09.451 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:09.451 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:09.451 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:09.451 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:09.452 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:09.452 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:09.452 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:09.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:09.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:09.556+0000 7f4e9ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:09.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:09.561+0000 7f4e9ff9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:09.819 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:09.819 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1171,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:09.832 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:09.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:09.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:09.935+0000 7f8e5c11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:09.942 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:09.941+0000 7f8e5c11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:10.127 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:10.127 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:10.139 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2024-03-04T06:27:08.952658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002815"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706824,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:12.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~103s 2024-03-04T06:27:12.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~103s 2024-03-04T06:27:13.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:13.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:13.259+0000 7f0ee951d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:13.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:13.265+0000 7f0ee951d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:13.453 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:13.454 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:13.466 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-03-04T06:27:12.953600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002893"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706829,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706829,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:13.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:13.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:13.570+0000 7efdc8b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:13.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:13.576+0000 7efdc8b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:13.766 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:13.766 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:13.779 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-03-04T06:27:12.953600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002893"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706829,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706829,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:13.780 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:13.780 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:13.780 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:13.780 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:13.780 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:13.780 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:13.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:13.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:13.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:13.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:13.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:13.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:13.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:13.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:13.781 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:13.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:13.782 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:13.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:13.782 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:13.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:13.886+0000 7f510c79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:13.891 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:13.891+0000 7f510c79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:14.151 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:14.151 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1175,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:14.164 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:14.164 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:14.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:14.267+0000 7fb51931d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:14.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:14.273+0000 7fb51931d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:14.461 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:14.461 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:14.473 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2024-03-04T06:27:12.953600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002893"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706829,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706829,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:17.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:17.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:17.593+0000 7f689e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:17.600 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:17.600+0000 7f689e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:17.790 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:17.790 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:17.802 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2024-03-04T06:27:16.954669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003044"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:17.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:17.909 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:17.906+0000 7f180cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:17.912 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:17.912+0000 7f180cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:18.101 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:18.102 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:18.114 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2024-03-04T06:27:16.954669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003044"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:18.115 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:18.115 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:18.115 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:18.115 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:18.115 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:18.115 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:18.116 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:18.116 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:18.116 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:18.116 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:18.116 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:18.116 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:18.116 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:18.116 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:18.116 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:18.116 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:18.117 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:18.117 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:18.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:18.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~108s 2024-03-04T06:27:18.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~108s 2024-03-04T06:27:18.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:18.220+0000 7f0d20424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:18.226 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:18.226+0000 7f0d20424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:18.490 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:18.490 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1180,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:18.502 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:18.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:18.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:18.605+0000 7fc1ef79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:18.612 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:18.612+0000 7fc1ef79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:18.801 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:18.812 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2024-03-04T06:27:16.954669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003044"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:21.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:21.936 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:21.932+0000 7f7117ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:21.939 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:21.939+0000 7f7117ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:22.129 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:22.141 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1645,"stamp":"2024-03-04T06:27:20.955844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003185"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:22.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:22.249 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:22.245+0000 7f8f8e024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:22.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:22.251+0000 7f8f8e024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:22.440 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:22.441 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:22.453 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1645,"stamp":"2024-03-04T06:27:20.955844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003185"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706834,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:22.453 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:22.453 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:22.454 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:22.454 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:22.454 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:22.454 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:22.454 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:22.454 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:22.454 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:22.454 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:22.455 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:22.455 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:22.455 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:22.455 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:22.455 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:22.455 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:22.455 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:22.455 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:22.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:22.561 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:22.558+0000 7f92f2c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:22.564 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:22.563+0000 7f92f2c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:22.824 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:22.824 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1184,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:22.837 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:22.837 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:22.943 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:22.940+0000 7f4198b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:22.946 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:22.946+0000 7f4198b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:23.131 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:23.132 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:23.144 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1646,"stamp":"2024-03-04T06:27:22.956389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003241"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706839,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706839,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:23.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~113s 2024-03-04T06:27:23.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~113s 2024-03-04T06:27:26.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:26.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:26.262+0000 7faf596dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:26.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:26.269+0000 7faf596dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:26.458 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:26.469 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1647,"stamp":"2024-03-04T06:27:24.956977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003377"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706839,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706839,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:26.470 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:26.576 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:26.573+0000 7f2d93a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:26.579 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:26.579+0000 7f2d93a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:26.765 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:26.765 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:26.777 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1647,"stamp":"2024-03-04T06:27:24.956977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003377"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706839,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706839,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:26.778 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:26.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:26.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:26.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:26.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:26.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:26.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:26.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:26.779 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:26.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:26.779 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:26.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:26.779 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:26.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:26.779 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:26.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:26.779 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:26.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:26.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:26.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:26.883+0000 7fc30542d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:26.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:26.888+0000 7fc30542d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:27.149 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:27.149 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1188,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:27.162 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:27.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:27.269 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:27.265+0000 7fc00e15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:27.272 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:27.271+0000 7fc00e15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:27.459 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:27.459 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:27.471 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1648,"stamp":"2024-03-04T06:27:26.957360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:28.963 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~119s 2024-03-04T06:27:28.963 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~119s 2024-03-04T06:27:30.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:30.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:30.590+0000 7f67e8d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:30.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:30.596+0000 7f67e8d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:30.783 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:30.784 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:30.796 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1649,"stamp":"2024-03-04T06:27:28.957863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003194"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:30.796 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:30.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:30.899+0000 7f4015a06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:30.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:30.906+0000 7f4015a06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:31.091 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:31.091 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:31.103 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2024-03-04T06:27:30.958435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003230"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:31.104 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:31.104 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:31.105 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:31.106 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:31.106 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:31.106 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:31.106 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:31.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:31.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:31.210+0000 7f88a935d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:31.216 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:31.215+0000 7f88a935d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:31.474 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:31.474 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1193,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:31.487 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:31.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:31.593 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:31.590+0000 7f1cec217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:31.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:31.596+0000 7f1cec217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:31.787 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:31.787 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:31.799 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2024-03-04T06:27:30.958435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003230"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:34.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~124s 2024-03-04T06:27:34.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~124s 2024-03-04T06:27:34.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:34.921 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:34.917+0000 7fdb6e0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:34.924 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:34.924+0000 7fdb6e0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:35.110 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:35.111 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:35.122 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2024-03-04T06:27:34.959432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003043"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706849,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706849,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:35.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:35.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:35.228+0000 7fbb0be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:35.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:35.234+0000 7fbb0be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:35.422 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:35.434 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2024-03-04T06:27:34.959432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003043"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706849,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706849,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:35.435 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:35.435 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:35.435 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:35.435 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:35.435 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:35.435 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:35.435 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:35.435 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:35.436 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:35.436 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:35.436 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:35.436 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:35.436 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:35.436 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:35.436 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:35.436 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:35.436 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:35.437 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:35.437 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:35.543 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:35.540+0000 7f284cd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:35.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:35.545+0000 7f284cd9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:35.803 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:35.803 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1197,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:35.816 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:35.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:35.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:35.919+0000 7f50cc95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:35.926 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:35.926+0000 7f50cc95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:36.115 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:36.127 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2024-03-04T06:27:34.959432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003043"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706849,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706849,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:39.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:39.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:39.246+0000 7f242191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:39.254 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:39.253+0000 7f242191d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:39.445 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:39.446 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:39.458 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2024-03-04T06:27:38.960691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003331"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:39.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:39.565 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:39.562+0000 7f7bf8b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:39.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:39.568+0000 7f7bf8b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:39.755 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:39.755 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:39.767 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2024-03-04T06:27:38.960691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003331"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:39.768 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:39.768 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:39.768 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:39.768 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:39.769 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:39.769 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:39.769 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:39.769 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:39.769 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:39.769 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:39.769 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:39.769 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:39.769 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:39.769 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:39.769 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:39.770 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:39.770 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:39.770 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:39.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:39.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~130s 2024-03-04T06:27:39.771 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~130s 2024-03-04T06:27:39.876 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:39.874+0000 7fc1f031d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:39.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:39.879+0000 7fc1f031d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:40.136 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:40.136 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1201,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:40.149 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:40.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:40.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:40.252+0000 7fba13b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:40.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:40.258+0000 7fba13b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:40.451 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:40.461 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2024-03-04T06:27:38.960691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003331"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:43.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:43.586 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:43.582+0000 7f04d7464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:43.589 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:43.589+0000 7f04d7464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:43.782 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:43.782 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:43.794 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1656,"stamp":"2024-03-04T06:27:42.961682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003247"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:43.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:43.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:43.898+0000 7f239326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:43.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:43.905+0000 7f239326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:44.094 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:44.094 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:44.106 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1656,"stamp":"2024-03-04T06:27:42.961682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003247"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:44.107 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:44.107 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:44.107 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:44.107 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:44.108 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:44.108 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:44.108 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:44.108 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:44.108 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:44.108 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:44.108 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:44.108 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:44.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:44.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:44.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:44.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:44.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:44.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:44.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:44.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:44.212+0000 7f84dcb54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:44.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:44.218+0000 7f84dcb54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:44.479 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:44.480 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1206,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:44.493 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:44.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:44.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:44.595+0000 7f0f4182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:44.602 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:44.602+0000 7f0f4182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:44.793 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:44.793 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:44.805 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1656,"stamp":"2024-03-04T06:27:42.961682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003247"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:45.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~135s 2024-03-04T06:27:45.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~135s 2024-03-04T06:27:47.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:47.928 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:47.924+0000 7ffaff46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:47.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:47.931+0000 7ffaff46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:48.122 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:48.123 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:48.135 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1658,"stamp":"2024-03-04T06:27:46.962684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003251"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:48.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:48.242 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:48.238+0000 7f0c6a79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:48.245 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:48.245+0000 7f0c6a79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:48.434 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:48.446 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1658,"stamp":"2024-03-04T06:27:46.962684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003251"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:48.447 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:48.447 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:48.448 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:48.448 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:48.448 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:48.448 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:48.448 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:48.448 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:48.448 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:48.448 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:48.448 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:48.448 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:48.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:48.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:48.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:48.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:48.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:48.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:48.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:48.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:48.552+0000 7fc574c0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:48.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:48.558+0000 7fc574c0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:48.818 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:48.819 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1210,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:48.831 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:48.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:48.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:48.934+0000 7faac4c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:48.941 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:48.941+0000 7faac4c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:49.127 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:49.127 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:49.139 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1659,"stamp":"2024-03-04T06:27:48.963239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003328"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:50.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~141s 2024-03-04T06:27:50.580 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~141s 2024-03-04T06:27:52.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:52.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:52.259+0000 7f040f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:52.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:52.266+0000 7f040f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:52.455 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:52.455 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:52.468 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1660,"stamp":"2024-03-04T06:27:50.963772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003080"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:52.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:52.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:52.571+0000 7feb9e79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:52.578 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:52.578+0000 7feb9e79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:52.765 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:52.765 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:52.777 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1660,"stamp":"2024-03-04T06:27:50.963772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003080"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:52.778 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:52.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:52.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:52.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:52.779 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:52.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:52.779 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:52.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:52.779 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:52.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:52.779 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:52.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:52.779 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:52.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:52.780 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:52.780 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:52.780 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:52.780 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:52.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:52.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:52.882+0000 7f513639d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:52.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:52.888+0000 7f513639d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:53.144 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:53.144 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1214,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:53.157 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:53.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:53.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:53.259+0000 7f74f159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:53.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:53.265+0000 7f74f159d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:53.456 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:53.457 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:53.469 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1661,"stamp":"2024-03-04T06:27:52.964425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003179"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:55.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~146s 2024-03-04T06:27:55.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~146s 2024-03-04T06:27:56.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:56.592 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:56.588+0000 7f6492b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:56.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:56.595+0000 7f6492b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:56.783 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:56.783 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:56.795 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1662,"stamp":"2024-03-04T06:27:54.965005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003322"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:56.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:56.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:56.898+0000 7f0119cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:56.905+0000 7f0119cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:57.091 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:57.091 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:57.103 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2024-03-04T06:27:56.965556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:27:57.104 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:27:57.104 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:57.104 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:27:57.104 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:57.104 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:27:57.104 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:57.104 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:27:57.104 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:57.105 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:27:57.105 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:57.105 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:27:57.105 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:57.105 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:27:57.105 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:57.105 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:27:57.105 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:27:57.105 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:27:57.105 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:27:57.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:27:57.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:57.209+0000 7f63a491d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:57.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:57.214+0000 7f63a491d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:57.475 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:57.475 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1219,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:27:57.489 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:27:57.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:27:57.596 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:57.592+0000 7f06e4a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:57.599 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:27:57.599+0000 7f06e4a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:27:57.791 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:27:57.792 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:27:57.804 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2024-03-04T06:27:56.965556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:00.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:00.927 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:00.923+0000 7f7226107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:00.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:00.929+0000 7f7226107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:01.117 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:01.118 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:01.130 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2024-03-04T06:28:00.966551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003312"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:01.131 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:01.236 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:01.233+0000 7f83c766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:01.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:01.239+0000 7f83c766d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:01.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~151s 2024-03-04T06:28:01.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~151s 2024-03-04T06:28:01.427 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:01.427 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:01.439 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2024-03-04T06:28:00.966551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003312"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:01.440 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:01.440 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:01.440 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:01.440 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:01.440 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:01.440 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:01.440 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:01.441 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:01.441 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:01.441 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:01.441 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:01.441 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:01.441 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:01.441 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:01.441 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:01.441 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:01.441 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:01.441 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:01.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:01.547 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:01.544+0000 7f25c2624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:01.550 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:01.550+0000 7f25c2624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:01.810 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1223,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:01.823 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:01.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:01.929 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:01.926+0000 7fc372307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:01.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:01.932+0000 7fc372307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:02.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:02.124 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:02.136 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2024-03-04T06:28:00.966551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003312"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706874,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:05.138 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:05.259 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:05.255+0000 7fb8d8e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:05.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:05.262+0000 7fb8d8e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:05.450 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:05.461 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2024-03-04T06:28:04.967466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003041"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:05.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:05.568 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:05.565+0000 7f93b479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:05.571 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:05.571+0000 7f93b479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:05.758 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:05.759 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:05.770 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2024-03-04T06:28:04.967466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003041"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:05.771 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:05.771 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:05.771 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:05.771 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:05.772 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:05.772 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:05.772 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:05.772 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:05.772 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:05.772 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:05.772 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:05.772 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:05.772 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:05.772 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:05.773 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:05.773 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:05.773 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:05.773 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:05.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:05.879 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:05.876+0000 7f0d71f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:05.882 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:05.882+0000 7f0d71f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:06.144 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:06.144 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1227,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:06.157 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:06.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:06.264 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:06.260+0000 7faedd19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:06.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:06.266+0000 7faedd19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:06.460 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:06.460 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:06.472 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2024-03-04T06:28:04.967466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003041"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:06.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~157s 2024-03-04T06:28:06.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~157s 2024-03-04T06:28:09.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:09.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:09.590+0000 7ff5c1813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:09.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:09.597+0000 7ff5c1813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:09.786 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:09.787 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:09.798 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1669,"stamp":"2024-03-04T06:28:08.968756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003200"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:09.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:09.903+0000 7fd472d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:09.910 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:09.909+0000 7fd472d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:10.097 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:10.098 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:10.110 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1669,"stamp":"2024-03-04T06:28:08.968756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003200"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:10.111 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:10.111 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:10.111 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:10.111 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:10.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:10.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:10.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:10.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:10.112 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:10.112 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:10.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:10.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:10.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:10.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:10.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:10.113 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:10.113 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:10.114 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:10.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:10.220 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:10.218+0000 7f644cf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:10.223 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:10.223+0000 7f644cf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:10.484 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1232,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:10.497 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:10.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:10.603 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:10.599+0000 7f6031e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:10.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:10.605+0000 7f6031e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:10.797 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:10.809 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1669,"stamp":"2024-03-04T06:28:08.968756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003200"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:12.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~162s 2024-03-04T06:28:12.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~162s 2024-03-04T06:28:13.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:13.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:13.928+0000 7efe98e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:13.935 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:13.935+0000 7efe98e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:14.125 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:14.125 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:14.137 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1671,"stamp":"2024-03-04T06:28:12.970073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003521"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:14.138 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:14.244 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:14.241+0000 7f06890dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:14.247 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:14.247+0000 7f06890dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:14.434 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:14.434 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:14.447 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1671,"stamp":"2024-03-04T06:28:12.970073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003521"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:14.448 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:14.448 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:14.448 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:14.448 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:14.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:14.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:14.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:14.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:14.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:14.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:14.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:14.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:14.449 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:14.449 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:14.450 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:14.450 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:14.450 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:14.450 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:14.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:14.555 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:14.552+0000 7fe74275d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:14.558 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:14.558+0000 7fe74275d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:14.817 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:14.818 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1236,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:14.830 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:14.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:14.937 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:14.933+0000 7fa15146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:14.940 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:14.940+0000 7fa15146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:15.127 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:15.128 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:15.140 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1672,"stamp":"2024-03-04T06:28:14.970674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003682"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:17.601 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~168s 2024-03-04T06:28:17.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~168s 2024-03-04T06:28:18.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:18.262 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:18.258+0000 7f19d686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:18.265 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:18.265+0000 7f19d686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:18.453 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:18.453 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:18.465 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1673,"stamp":"2024-03-04T06:28:16.971154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:18.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:18.573 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:18.569+0000 7fd80039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:18.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:18.575+0000 7fd80039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:18.762 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:18.763 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:18.774 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1673,"stamp":"2024-03-04T06:28:16.971154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:18.775 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:18.775 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:18.775 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:18.775 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:18.775 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:18.775 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:18.776 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:18.776 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:18.776 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:18.776 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:18.776 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:18.776 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:18.776 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:18.776 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:18.776 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:18.776 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:18.777 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:18.777 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:18.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:18.883 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:18.880+0000 7f3c1a2dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:18.886 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:18.885+0000 7f3c1a2dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:19.145 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:19.145 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1240,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:19.160 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:19.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:19.267 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:19.263+0000 7fce2839d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:19.270 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:19.269+0000 7fce2839d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:19.458 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:19.459 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:19.470 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1674,"stamp":"2024-03-04T06:28:18.971896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003796"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:22.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:22.594 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:22.590+0000 7fe69d2dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:22.597 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:22.597+0000 7fe69d2dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:22.785 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:22.785 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:22.797 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1675,"stamp":"2024-03-04T06:28:20.972575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003818"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:22.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:22.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:22.900+0000 7fd02c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:22.907+0000 7fd02c59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:23.006 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~173s 2024-03-04T06:28:23.006 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~173s 2024-03-04T06:28:23.094 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:23.095 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:23.106 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1676,"stamp":"2024-03-04T06:28:22.973198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003749"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:23.107 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:23.107 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:23.107 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:23.107 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:23.107 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:23.108 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:23.108 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:23.108 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:23.108 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:23.108 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:23.108 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:23.108 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:23.108 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:23.108 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:23.108 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:23.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:23.109 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:23.109 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:23.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:23.215 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:23.212+0000 7fa56ad9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:23.218 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:23.218+0000 7fa56ad9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:23.475 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:23.475 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1245,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:23.488 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:23.488 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:23.595 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:23.591+0000 7fa5fd35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:23.598 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:23.597+0000 7fa5fd35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:23.784 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:23.784 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:23.797 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1676,"stamp":"2024-03-04T06:28:22.973198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003749"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:26.799 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:26.919 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:26.915+0000 7f0da6d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:26.923 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:26.922+0000 7f0da6d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:27.112 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:27.112 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:27.124 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1678,"stamp":"2024-03-04T06:28:26.974324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003650"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:27.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:27.232 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:27.228+0000 7f285ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:27.235 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:27.234+0000 7f285ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:27.424 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:27.425 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:27.437 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1678,"stamp":"2024-03-04T06:28:26.974324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003650"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:27.438 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:27.438 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:27.438 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:27.438 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:27.438 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:27.438 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:27.438 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:27.439 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:27.439 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:27.439 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:27.439 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:27.439 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:27.439 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:27.439 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:27.439 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:27.439 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:27.439 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:27.440 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:27.440 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:27.546 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:27.543+0000 7f9bca6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:27.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:27.548+0000 7f9bca6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:27.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:27.810 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1249,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:27.823 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:27.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:27.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:27.926+0000 7f25c279d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:27.933 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:27.932+0000 7f25c279d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:28.122 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:28.123 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:28.134 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1678,"stamp":"2024-03-04T06:28:26.974324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003650"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:28.409 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~178s 2024-03-04T06:28:28.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~178s 2024-03-04T06:28:31.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:31.257 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:31.253+0000 7f6885d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:31.260 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:31.260+0000 7f6885d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:31.452 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:31.452 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:31.465 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1680,"stamp":"2024-03-04T06:28:30.975522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003626"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:31.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:31.572 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:31.569+0000 7f69b7154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:31.575 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:31.575+0000 7f69b7154640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:31.763 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:31.764 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:31.776 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1680,"stamp":"2024-03-04T06:28:30.975522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003626"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:31.776 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:31.777 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:31.777 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:31.777 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:31.777 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:31.777 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:31.777 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:31.777 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:31.777 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:31.777 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:31.777 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:31.777 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:31.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:31.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:31.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:31.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:31.778 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:31.778 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:31.778 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:31.885 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:31.882+0000 7f6fddb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:31.888 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:31.887+0000 7f6fddb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:32.144 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:32.144 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1253,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:32.157 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:32.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:32.263 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:32.260+0000 7f2a6079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:32.266 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:32.266+0000 7f2a6079d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:32.458 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:32.615 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:32.615 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1680,"stamp":"2024-03-04T06:28:30.975522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003626"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:33.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~184s 2024-03-04T06:28:33.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~184s 2024-03-04T06:28:35.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:35.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:35.735+0000 7f292699d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:35.742 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:35.741+0000 7f292699d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:35.934 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:35.934 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:35.947 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1682,"stamp":"2024-03-04T06:28:34.976600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003402"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:35.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:36.054 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:36.050+0000 7fd764464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:36.057 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:36.057+0000 7fd764464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:36.246 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:36.258 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1682,"stamp":"2024-03-04T06:28:34.976600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003402"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:36.260 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:36.260 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:36.260 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:36.260 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:36.261 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:36.261 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:36.261 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:36.261 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:36.261 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:36.262 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:36.262 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:36.262 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:36.262 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:36.262 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:36.263 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:36.263 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:36.263 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:36.263 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:36.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:36.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:36.366+0000 7f74d0b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:36.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:36.372+0000 7f74d0b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:36.630 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1258,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:36.643 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:36.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:36.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:36.745+0000 7f31a7d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:36.752 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:36.752+0000 7f31a7d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:36.941 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:36.941 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:36.953 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1682,"stamp":"2024-03-04T06:28:34.976600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003402"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:39.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~189s 2024-03-04T06:28:39.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~189s 2024-03-04T06:28:39.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:40.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:40.071+0000 7f0327a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:40.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:40.078+0000 7f0327a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:40.266 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:40.266 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:40.278 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1684,"stamp":"2024-03-04T06:28:38.977510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003185"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:40.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:40.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:40.382+0000 7fe7f4f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:40.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:40.388+0000 7fe7f4f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:40.577 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:40.578 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:40.589 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1684,"stamp":"2024-03-04T06:28:38.977510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003185"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:40.590 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:40.590 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:40.590 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:40.590 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:40.590 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:40.590 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:40.590 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:40.590 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:40.590 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:40.590 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:40.590 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:40.591 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:40.591 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:40.591 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:40.591 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:40.591 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:40.591 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:40.591 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:40.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:40.696 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:40.694+0000 7f70c255d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:40.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:40.699+0000 7f70c255d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:40.958 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:40.958 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1262,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:40.971 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:40.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:41.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:41.074+0000 7f53d4c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:41.081 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:41.080+0000 7f53d4c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:41.270 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:41.270 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:41.282 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1685,"stamp":"2024-03-04T06:28:40.978070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003234"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:44.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:44.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:44.400+0000 7f75e586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:44.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:44.407+0000 7f75e586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:44.595 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:44.606 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1686,"stamp":"2024-03-04T06:28:42.978751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003228"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:44.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:44.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~195s 2024-03-04T06:28:44.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~195s 2024-03-04T06:28:44.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:44.710+0000 7fb82031d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:44.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:44.717+0000 7fb82031d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:44.905 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:44.905 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:44.916 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1686,"stamp":"2024-03-04T06:28:42.978751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003228"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:44.917 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:44.917 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:44.917 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:44.917 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:44.917 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:44.917 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:44.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:44.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:45.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:45.021+0000 7f46f342d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:45.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:45.027+0000 7f46f342d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:45.288 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:45.288 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1267,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:45.301 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:45.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:45.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:45.404+0000 7fc35999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:45.411 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:45.410+0000 7fc35999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:45.597 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:45.597 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:45.609 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1687,"stamp":"2024-03-04T06:28:44.979222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003113"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:48.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:48.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:48.728+0000 7f86b1a2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:48.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:48.734+0000 7f86b1a2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:48.926 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:48.926 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:48.938 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1688,"stamp":"2024-03-04T06:28:46.979652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003051"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:48.939 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:49.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:49.042+0000 7fd0b8f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:49.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:49.048+0000 7fd0b8f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:49.235 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:49.235 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:49.247 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1689,"stamp":"2024-03-04T06:28:48.980172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003126"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:49.247 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:49.248 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:49.248 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:49.248 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:49.248 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:49.248 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:49.248 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:49.248 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:49.248 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:49.248 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:49.249 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:49.249 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:49.249 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:49.249 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:49.249 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:49.249 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:49.249 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:49.249 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:49.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:49.355 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:49.352+0000 7f02a439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:49.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:49.357+0000 7f02a439d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:49.620 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1271,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:49.633 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:49.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:49.740 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:49.736+0000 7fd8f18ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:49.743 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:49.743+0000 7fd8f18ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:49.929 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:49.930 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:49.942 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1689,"stamp":"2024-03-04T06:28:48.980172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003126"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706924,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:50.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~200s 2024-03-04T06:28:50.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~200s 2024-03-04T06:28:52.943 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:53.068 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:53.064+0000 7f3e5f22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:53.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:53.071+0000 7f3e5f22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:53.258 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:53.259 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:53.271 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1691,"stamp":"2024-03-04T06:28:52.981299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003228"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:53.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:53.378 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:53.374+0000 7fbe2062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:53.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:53.381+0000 7fbe2062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:53.570 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:53.570 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:53.581 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1691,"stamp":"2024-03-04T06:28:52.981299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003228"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:53.582 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:53.582 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:53.582 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:53.583 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:53.583 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:53.583 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:53.583 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:53.583 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:53.583 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:53.583 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:53.583 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:53.583 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:53.583 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:53.583 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:53.584 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:53.584 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:53.584 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:53.584 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:53.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:53.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:53.688+0000 7fb23366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:53.693 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:53.693+0000 7fb23366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:53.951 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:53.951 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1275,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:53.964 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:53.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:54.071 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:54.068+0000 7f9b5011d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:54.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:54.074+0000 7f9b5011d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:54.265 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:54.266 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:54.278 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1691,"stamp":"2024-03-04T06:28:52.981299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003228"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:55.430 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~205s 2024-03-04T06:28:55.430 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~205s 2024-03-04T06:28:57.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:57.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:57.397+0000 7fa97551d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:57.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:57.403+0000 7fa97551d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:57.594 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:57.594 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:57.606 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1693,"stamp":"2024-03-04T06:28:56.982397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003174"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:57.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:57.714 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:57.710+0000 7f102c95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:57.717 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:57.716+0000 7f102c95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:57.905 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:57.917 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1693,"stamp":"2024-03-04T06:28:56.982397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003174"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:28:57.918 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:28:57.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:57.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:28:57.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:57.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:28:57.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:57.919 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:28:57.919 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:57.919 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:28:57.919 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:57.919 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:28:57.919 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:57.919 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:28:57.919 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:57.919 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:28:57.919 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:28:57.920 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:28:57.920 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:28:57.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:28:58.026 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:58.023+0000 7f1ec655d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:58.029 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:58.029+0000 7f1ec655d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:58.293 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:58.293 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1280,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:28:58.306 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:28:58.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:28:58.412 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:58.409+0000 7fb6c9b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:58.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:28:58.415+0000 7fb6c9b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:28:58.604 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:28:58.604 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:28:58.616 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1693,"stamp":"2024-03-04T06:28:56.982397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003174"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:00.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~211s 2024-03-04T06:29:00.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~211s 2024-03-04T06:29:01.618 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:01.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:01.734+0000 7fdb4646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:01.741 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:01.741+0000 7fdb4646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:01.928 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:01.940 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1695,"stamp":"2024-03-04T06:29:00.983512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003339"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:01.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:02.047 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:02.044+0000 7f2a83e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:02.050 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:02.050+0000 7f2a83e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:02.238 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:02.250 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1695,"stamp":"2024-03-04T06:29:00.983512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003339"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:02.250 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:02.251 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:02.251 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:02.251 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:02.251 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:02.251 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:02.251 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:02.251 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:02.251 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:02.251 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:02.252 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:02.252 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:02.252 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:02.252 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:02.252 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:02.252 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:02.252 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:02.252 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:02.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:02.358 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:02.355+0000 7f872039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:02.361 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:02.361+0000 7f872039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:02.627 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1284,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:02.639 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:02.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:02.746 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:02.742+0000 7ff4fb46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:02.749 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:02.749+0000 7ff4fb46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:02.936 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:02.937 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:02.948 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1695,"stamp":"2024-03-04T06:29:00.983512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003339"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:05.950 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:06.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:06.068+0000 7f4345307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:06.076 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:06.075+0000 7f4345307640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:06.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~216s 2024-03-04T06:29:06.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~216s 2024-03-04T06:29:06.264 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:06.264 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:06.274 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1697,"stamp":"2024-03-04T06:29:04.984430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003130"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:06.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:06.381 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:06.378+0000 7f708e11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:06.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:06.384+0000 7f708e11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:06.573 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:06.574 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:06.586 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1697,"stamp":"2024-03-04T06:29:04.984430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003130"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:06.587 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:06.587 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:06.587 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:06.587 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:06.587 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:06.587 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:06.587 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:06.587 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:06.587 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:06.588 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:06.588 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:06.588 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:06.588 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:06.588 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:06.588 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:06.588 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:06.588 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:06.588 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:06.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:06.694 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:06.691+0000 7f3decb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:06.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:06.696+0000 7f3decb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:06.953 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1288,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:06.966 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:06.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:07.072 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:07.069+0000 7f117d914640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:07.075 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:07.075+0000 7f117d914640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:07.263 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:07.263 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:07.275 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1698,"stamp":"2024-03-04T06:29:06.984991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003084"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:10.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:10.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:10.394+0000 7fbf92664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:10.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:10.400+0000 7fbf92664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:10.594 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:10.595 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:10.606 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1699,"stamp":"2024-03-04T06:29:08.985649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003251"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:10.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:10.713 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:10.710+0000 7f294df9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:10.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:10.716+0000 7f294df9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:10.904 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:10.904 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:10.916 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1699,"stamp":"2024-03-04T06:29:08.985649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003251"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:10.917 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:10.917 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:10.917 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:10.917 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:10.917 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:10.917 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:10.917 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:10.917 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:10.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:10.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:10.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:10.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:10.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:10.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:10.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:10.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:10.918 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:10.918 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:10.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:11.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:11.022+0000 7f0f94f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:11.027 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:11.027+0000 7f0f94f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:11.284 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:11.284 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1293,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:11.296 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:11.296 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:11.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:11.399+0000 7f058075d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:11.406 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:11.405+0000 7f058075d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:11.593 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:11.605 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1700,"stamp":"2024-03-04T06:29:10.986205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003176"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:11.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~222s 2024-03-04T06:29:11.642 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~222s 2024-03-04T06:29:14.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:14.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:14.724+0000 7f027122d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:14.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:14.731+0000 7f027122d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:14.918 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:14.930 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1701,"stamp":"2024-03-04T06:29:12.986689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003176"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:14.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:15.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:15.033+0000 7ff5f339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:15.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:15.039+0000 7ff5f339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:15.226 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:15.226 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:15.238 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1702,"stamp":"2024-03-04T06:29:14.987165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003214"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:15.239 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:15.239 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:15.239 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:15.239 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:15.240 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:15.240 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:15.240 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:15.240 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:15.240 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:15.240 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:15.240 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:15.240 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:15.240 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:15.240 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:15.241 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:15.241 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:15.241 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:15.241 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:15.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:15.347 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:15.344+0000 7f3adc0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:15.349+0000 7f3adc0dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:15.611 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:15.611 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1297,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:15.624 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:15.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:15.731 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:15.727+0000 7f5dcb59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:15.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:15.733+0000 7f5dcb59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:15.924 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:15.935 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1702,"stamp":"2024-03-04T06:29:14.987165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003214"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:17.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~227s 2024-03-04T06:29:17.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~227s 2024-03-04T06:29:18.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:19.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:19.055+0000 7f03fd514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:19.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:19.062+0000 7f03fd514640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:19.257 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:19.257 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:19.269 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1704,"stamp":"2024-03-04T06:29:18.988204+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003212"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:19.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:19.377 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:19.373+0000 7f16a0add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:19.380 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:19.380+0000 7f16a0add640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:19.572 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:19.584 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1704,"stamp":"2024-03-04T06:29:18.988204+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003212"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:19.585 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:19.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:19.585 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:19.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:19.585 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:19.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:19.585 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:19.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:19.585 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:19.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:19.586 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:19.586 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:19.586 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:19.586 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:19.586 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:19.586 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:19.586 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:19.586 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:19.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:19.692 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:19.689+0000 7fcaaf31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:19.695 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:19.695+0000 7fcaaf31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:19.950 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:19.951 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1301,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:19.963 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:19.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:20.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:20.066+0000 7f89a9b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:20.073 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:20.073+0000 7f89a9b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:20.260 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:20.272 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1704,"stamp":"2024-03-04T06:29:18.988204+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003212"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706954,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:22.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~232s 2024-03-04T06:29:22.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~232s 2024-03-04T06:29:23.273 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:23.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:23.390+0000 7fd5a826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:23.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:23.399+0000 7fd5a826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:23.586 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:23.587 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:23.598 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1706,"stamp":"2024-03-04T06:29:22.989259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003053"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:23.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:23.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:23.702+0000 7fd18f35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:23.709 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:23.708+0000 7fd18f35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:23.901 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:23.901 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:23.910 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1706,"stamp":"2024-03-04T06:29:22.989259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003053"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:23.911 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:23.911 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:23.911 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:23.911 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:23.911 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:23.911 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:23.911 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:23.912 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:23.912 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:23.912 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:23.912 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:23.912 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:23.912 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:23.912 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:23.912 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:23.912 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:23.912 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:23.913 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:23.913 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:24.018 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:24.016+0000 7f085795d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:24.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:24.021+0000 7f085795d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:24.279 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:24.279 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1306,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:24.291 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:24.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:24.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:24.396+0000 7fc02fd07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:24.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:24.403+0000 7fc02fd07640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:24.591 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:24.591 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:24.603 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1706,"stamp":"2024-03-04T06:29:22.989259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003053"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:27.604 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:27.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:27.720+0000 7f631d354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:27.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:27.726+0000 7f631d354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:27.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~238s 2024-03-04T06:29:27.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~238s 2024-03-04T06:29:27.914 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:27.915 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:27.926 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1708,"stamp":"2024-03-04T06:29:26.990111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002945"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:27.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:28.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:28.030+0000 7f5f9279d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:28.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:28.035+0000 7f5f9279d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:28.228 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:28.240 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1708,"stamp":"2024-03-04T06:29:26.990111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002945"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:28.241 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:28.241 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:28.241 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:28.241 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:28.242 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:28.242 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:28.242 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:28.242 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:28.242 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:28.242 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:28.242 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:28.242 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:28.242 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:28.242 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:28.243 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:28.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:28.243 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:28.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:28.243 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:28.348 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:28.345+0000 7f26bf06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:28.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:28.349+0000 7f26bf06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:28.608 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:28.608 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1310,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:28.621 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:28.621 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:28.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:28.724+0000 7fa7ac8dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:28.730 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:28.730+0000 7fa7ac8dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:28.919 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:28.920 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:28.932 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1708,"stamp":"2024-03-04T06:29:26.990111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002945"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:31.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:32.056 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:32.052+0000 7ff5f479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:32.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:32.059+0000 7ff5f479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:32.248 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:32.260 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1710,"stamp":"2024-03-04T06:29:30.991173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002969"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:32.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:32.367 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:32.364+0000 7f4518b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:32.370 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:32.370+0000 7f4518b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:32.559 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:32.559 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:32.572 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1710,"stamp":"2024-03-04T06:29:30.991173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002969"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:32.572 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:32.573 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:32.573 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:32.573 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:32.573 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:32.573 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:32.573 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:32.573 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:32.573 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:32.573 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:32.573 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:32.573 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:32.574 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:32.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:32.574 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:32.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:32.574 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:32.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:32.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:32.679 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:32.677+0000 7f3078f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:32.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:32.682+0000 7f3078f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:32.940 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:32.940 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1314,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:32.953 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:32.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:33.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:33.056+0000 7f43666dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:33.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:33.062+0000 7f43666dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:33.252 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:33.252 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:33.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~243s 2024-03-04T06:29:33.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~243s 2024-03-04T06:29:33.264 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1711,"stamp":"2024-03-04T06:29:32.991632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002945"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:36.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:36.386 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:36.382+0000 7f40980dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:36.389 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:36.389+0000 7f40980dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:36.580 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:36.591 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1712,"stamp":"2024-03-04T06:29:34.992059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002800"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:36.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:36.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:36.695+0000 7fcac826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:36.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:36.701+0000 7fcac826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:36.891 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:36.903 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1712,"stamp":"2024-03-04T06:29:34.992059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002800"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:36.903 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:36.903 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:36.904 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:36.904 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:36.904 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:36.904 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:36.904 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:36.904 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:36.904 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:36.904 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:36.904 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:36.904 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:36.905 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:36.905 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:36.905 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:36.905 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:36.905 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:36.905 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:36.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:37.012 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:37.009+0000 7f9c5859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:37.015 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:37.014+0000 7f9c5859d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:37.272 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:37.272 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1319,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:37.285 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:37.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:37.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:37.388+0000 7f1f8671d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:37.394 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:37.394+0000 7f1f8671d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:37.583 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:37.583 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:37.595 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1713,"stamp":"2024-03-04T06:29:36.992507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002845"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:38.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~249s 2024-03-04T06:29:38.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~249s 2024-03-04T06:29:40.597 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:40.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:40.714+0000 7f7c2706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:40.721 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:40.720+0000 7f7c2706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:40.911 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:40.911 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:40.924 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1714,"stamp":"2024-03-04T06:29:38.993203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003092"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:40.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:41.030 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:41.027+0000 7f28a495d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:41.033 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:41.033+0000 7f28a495d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:41.222 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:41.222 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:41.233 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1715,"stamp":"2024-03-04T06:29:40.993627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002973"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:41.234 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:41.234 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:41.234 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:41.235 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:41.235 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:41.235 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:41.235 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:41.235 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:41.235 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:41.235 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:41.235 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:41.235 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:41.235 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:41.236 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:41.236 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:41.236 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:41.236 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:41.236 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:41.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:41.342 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:41.339+0000 7fa0b8f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:41.344 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:41.344+0000 7fa0b8f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:41.604 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:41.604 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1323,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:41.617 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:41.618 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:41.724 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:41.720+0000 7fd536314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:41.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:41.726+0000 7fd536314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:41.914 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:41.914 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:41.926 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1715,"stamp":"2024-03-04T06:29:40.993627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002973"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:44.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~254s 2024-03-04T06:29:44.071 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~254s 2024-03-04T06:29:44.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:45.048 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:45.045+0000 7fefc1224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:45.051 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:45.051+0000 7fefc1224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:45.237 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:45.238 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:45.250 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1717,"stamp":"2024-03-04T06:29:44.994600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002967"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:45.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:45.357 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:45.353+0000 7f667bb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:45.360 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:45.360+0000 7f667bb1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:45.550 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:45.550 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:45.562 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1717,"stamp":"2024-03-04T06:29:44.994600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002967"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:45.563 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:45.563 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:45.563 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:45.563 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:45.563 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:45.563 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:45.563 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:45.563 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:45.564 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:45.564 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:45.564 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:45.564 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:45.564 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:45.564 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:45.564 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:45.564 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:45.564 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:45.564 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:45.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:45.670 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:45.667+0000 7f647175d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:45.673 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:45.672+0000 7f647175d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:45.938 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:45.939 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1327,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:45.951 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:45.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:46.059 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:46.055+0000 7fa55e35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:46.062 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:46.061+0000 7fa55e35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:46.250 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:46.251 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:46.262 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1717,"stamp":"2024-03-04T06:29:44.994600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002967"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:49.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:49.384 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:49.381+0000 7fe82f354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:49.387 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:49.387+0000 7fe82f354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:49.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~259s 2024-03-04T06:29:49.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~259s 2024-03-04T06:29:49.577 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:49.578 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:49.589 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1719,"stamp":"2024-03-04T06:29:48.995733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003225"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:49.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:49.697 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:49.693+0000 7fb95d107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:49.699 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:49.699+0000 7fb95d107640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:49.891 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:49.892 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:49.903 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1719,"stamp":"2024-03-04T06:29:48.995733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003225"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:49.904 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:49.904 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:49.904 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:49.904 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:49.904 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:49.904 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:49.904 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:49.904 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:49.905 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:49.905 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:49.905 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:49.905 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:49.905 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:49.905 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:49.905 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:49.905 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:49.907 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:49.907 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:49.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:50.014 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:50.011+0000 7f198739d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:50.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:50.016+0000 7f198739d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:50.275 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:50.275 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1332,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:50.288 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:50.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:50.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:50.391+0000 7f83fed14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:50.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:50.398+0000 7f83fed14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:50.588 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:50.589 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:50.601 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1719,"stamp":"2024-03-04T06:29:48.995733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003225"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:53.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:53.723 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:53.720+0000 7f966019d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:53.726+0000 7f966019d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:53.915 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:53.916 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:53.927 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1721,"stamp":"2024-03-04T06:29:52.996599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002972"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:53.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:54.031+0000 7f11e619d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:54.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:54.037+0000 7f11e619d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:54.226 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:54.226 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:54.239 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1721,"stamp":"2024-03-04T06:29:52.996599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002972"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:54.239 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:54.239 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:54.240 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:54.240 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:54.240 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:54.240 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:54.240 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:54.240 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:54.240 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:54.240 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:54.240 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:54.240 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:54.241 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:54.241 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:54.241 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:54.241 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:54.241 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:54.241 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:54.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:54.346 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:54.343+0000 7fd847a2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:54.350 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:54.349+0000 7fd847a2e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:54.616 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:54.616 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1336,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:54.628 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:54.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:54.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:54.731+0000 7f1550c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:54.738 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:54.737+0000 7f1550c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:54.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~265s 2024-03-04T06:29:54.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~265s 2024-03-04T06:29:54.925 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:54.926 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:54.937 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1721,"stamp":"2024-03-04T06:29:52.996599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002972"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:57.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:58.060 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:58.056+0000 7f723122d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:58.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:58.062+0000 7f723122d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:58.250 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:58.251 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:58.262 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1723,"stamp":"2024-03-04T06:29:56.997562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002961"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:58.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:58.369 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:58.366+0000 7f5f30f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:58.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:58.372+0000 7f5f30f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:58.560 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:58.561 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:58.572 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1723,"stamp":"2024-03-04T06:29:56.997562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002961"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:29:58.573 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:29:58.573 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:58.573 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:29:58.573 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:29:58.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:29:58.575 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:29:58.575 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:29:58.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:29:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:58.677+0000 7fce7715d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:58.683 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:58.682+0000 7fce7715d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:58.943 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:58.943 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1340,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:29:58.955 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:29:58.956 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:29:59.063 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:59.059+0000 7fef2ad5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:59.066 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:29:59.065+0000 7fef2ad5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:29:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:29:59.255 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:29:59.267 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1724,"stamp":"2024-03-04T06:29:58.998018+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002953"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:00.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~270s 2024-03-04T06:30:00.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~270s 2024-03-04T06:30:02.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:02.391 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:02.387+0000 7fb8a66dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:02.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:02.394+0000 7fb8a66dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:02.583 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:02.595 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1725,"stamp":"2024-03-04T06:30:00.998628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:02.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:02.702 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:02.698+0000 7f8b8011d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:02.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:02.705+0000 7f8b8011d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:02.895 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:02.896 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:02.908 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1725,"stamp":"2024-03-04T06:30:00.998628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:02.909 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:30:02.909 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:02.909 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:30:02.909 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:02.909 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:02.910 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:30:02.911 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:02.911 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:30:03.016 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:03.014+0000 7f06c3e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:03.019 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:03.019+0000 7f06c3e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:03.283 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1345,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:30:03.295 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:30:03.295 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:03.398+0000 7fe90680f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:03.404 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:03.404+0000 7fe90680f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:03.591 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:03.592 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:03.604 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1726,"stamp":"2024-03-04T06:30:02.999283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003137"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:05.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~276s 2024-03-04T06:30:05.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~276s 2024-03-04T06:30:06.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:06.726 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:06.722+0000 7f23d379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:06.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:06.729+0000 7f23d379d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:06.918 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:06.929 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1727,"stamp":"2024-03-04T06:30:04.999962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003362"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:06.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:07.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:07.033+0000 7f1d1e6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:07.039 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:07.039+0000 7f1d1e6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:07.228 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:07.229 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:07.241 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1728,"stamp":"2024-03-04T06:30:07.000424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003326"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:07.242 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:30:07.242 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:07.242 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:30:07.242 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:07.242 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:30:07.242 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:07.242 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:30:07.242 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:07.243 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:30:07.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:07.243 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:30:07.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:07.243 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:30:07.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:07.243 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:30:07.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:07.243 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:30:07.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:07.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:30:07.349 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:07.346+0000 7f02ff22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:07.352 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:07.351+0000 7f02ff22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:07.615 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:07.615 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1349,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:30:07.627 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:30:07.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:07.734 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:07.731+0000 7f7d4c62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:07.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:07.737+0000 7f7d4c62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:07.926 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:07.927 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:07.939 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1728,"stamp":"2024-03-04T06:30:07.000424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003326"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:10.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:11.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:11.058+0000 7f0b7bf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:11.065 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:11.064+0000 7f0b7bf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:11.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~281s 2024-03-04T06:30:11.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~281s 2024-03-04T06:30:11.257 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:11.257 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:11.269 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1730,"stamp":"2024-03-04T06:30:11.001342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003323"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:11.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:11.376 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:11.373+0000 7f93f031d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:11.379 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:11.379+0000 7f93f031d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:11.567 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:11.568 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:11.580 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1730,"stamp":"2024-03-04T06:30:11.001342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003323"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:11.580 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:30:11.580 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:30:11.581 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:11.582 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:30:11.582 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:11.582 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:30:11.582 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:11.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:30:11.687 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:11.684+0000 7fa8228dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:11.690 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:11.690+0000 7fa8228dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:11.955 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1353,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:30:11.968 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:30:11.968 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:12.074 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:12.070+0000 7f41d5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:12.077 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:12.076+0000 7f41d5b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:12.265 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:12.266 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:12.278 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1730,"stamp":"2024-03-04T06:30:11.001342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003323"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:15.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:15.400 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:15.395+0000 7f8093a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:15.403 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:15.402+0000 7f8093a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:15.592 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:15.592 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:15.604 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1732,"stamp":"2024-03-04T06:30:15.002449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:15.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:15.712 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:15.707+0000 7fad2d35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:15.715 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:15.713+0000 7fad2d35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:15.901 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:15.913 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1732,"stamp":"2024-03-04T06:30:15.002449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:15.914 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:30:15.914 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:15.914 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:30:15.914 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:15.914 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:30:15.914 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:15.914 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:30:15.914 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:15.914 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:30:15.915 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:15.915 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:30:15.915 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:15.915 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:30:15.915 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:15.915 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:30:15.915 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:15.915 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:30:15.915 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:15.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:30:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:16.018+0000 7f718fa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:16.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:16.023+0000 7f718fa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:16.279 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:16.279 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1358,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:30:16.292 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:30:16.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:16.398 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:16.395+0000 7f973886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:16.401 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:16.400+0000 7f973886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:16.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~286s 2024-03-04T06:30:16.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~286s 2024-03-04T06:30:16.593 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:16.594 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:16.605 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1732,"stamp":"2024-03-04T06:30:15.002449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:19.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:19.729 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:19.725+0000 7f1e29cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:19.732 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:19.731+0000 7f1e29cdd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:19.920 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:19.920 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:19.932 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1734,"stamp":"2024-03-04T06:30:19.003575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003150"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:19.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:20.040 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:20.036+0000 7fa251d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:20.042 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:20.042+0000 7fa251d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:20.230 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:20.242 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1734,"stamp":"2024-03-04T06:30:19.003575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003150"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:20.243 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:30:20.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:20.243 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:30:20.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:20.243 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:30:20.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:20.243 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:30:20.243 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:20.244 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:30:20.244 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:20.244 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:30:20.244 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:20.244 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:30:20.244 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:20.244 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:30:20.244 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:20.244 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:30:20.244 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:20.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:30:20.351 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:20.348+0000 7f552259d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:20.354 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:20.353+0000 7f552259d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:20.618 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:20.618 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1362,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:30:20.630 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:30:20.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:20.737 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:20.733+0000 7f29d679d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:20.739 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:20.739+0000 7f29d679d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:20.926 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:20.927 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:20.939 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1734,"stamp":"2024-03-04T06:30:19.003575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003150"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:21.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~292s 2024-03-04T06:30:21.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~292s 2024-03-04T06:30:23.940 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:24.061 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:24.057+0000 7f3adc354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:24.064 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:24.064+0000 7f3adc354640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:24.252 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:24.253 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:24.264 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1736,"stamp":"2024-03-04T06:30:23.004404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003062"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:24.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:24.372 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:24.368+0000 7fa488d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:24.374 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:24.374+0000 7fa488d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:24.561 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:24.562 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:24.573 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1736,"stamp":"2024-03-04T06:30:23.004404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003062"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:24.574 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:30:24.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:24.574 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:30:24.574 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:24.575 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:30:24.575 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:24.575 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:30:24.575 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:24.575 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:30:24.575 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:24.575 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:30:24.575 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:24.575 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:30:24.575 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:24.576 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:30:24.576 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:24.576 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:30:24.576 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:24.576 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:30:24.682 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:24.679+0000 7fd0900dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:24.685 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:24.684+0000 7fd0900dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:24.947 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:24.948 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1366,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:30:24.960 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:30:24.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:25.067 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:25.063+0000 7f61b3b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:25.070 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:25.069+0000 7f61b3b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:25.258 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:25.258 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:25.270 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1737,"stamp":"2024-03-04T06:30:25.004902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003073"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:27.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~297s 2024-03-04T06:30:27.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~297s 2024-03-04T06:30:28.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:28.392 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:28.388+0000 7fa6ab6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:28.395 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:28.395+0000 7fa6ab6dd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:28.585 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:28.585 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:28.597 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1738,"stamp":"2024-03-04T06:30:27.005398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:28.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:28.704 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:28.700+0000 7fb53711d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:28.707 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:28.707+0000 7fb53711d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:28.899 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:28.912 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1738,"stamp":"2024-03-04T06:30:27.005398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002949"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:28.912 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean 2024-03-04T06:30:28.912 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 2819, 'reported_epoch': 1066, 'state': 'undersized+degraded+peered', 'last_fresh': '2024-03-04T06:25:58.377010+0000', 'last_change': '2024-03-04T06:25:58.376565+0000', 'last_active': '2024-03-04T06:25:50.479118+0000', 'last_peered': '2024-03-04T06:25:58.377010+0000', 'last_clean': '2024-03-04T06:25:37.417194+0000', 'last_became_active': '2024-03-04T06:07:48.335407+0000', 'last_became_peered': '2024-03-04T06:25:58.376565+0000', 'last_unstale': '2024-03-04T06:25:58.377010+0000', 'last_undegraded': '2024-03-04T06:25:58.372745+0000', 'last_fullsized': '2024-03-04T06:25:58.372152+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:47.297968+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:28.913 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean 2024-03-04T06:30:28.913 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395477+0000', 'last_change': '2024-03-04T06:25:58.395107+0000', 'last_active': '2024-03-04T06:25:50.494728+0000', 'last_peered': '2024-03-04T06:25:58.395477+0000', 'last_clean': '2024-03-04T06:25:38.446268+0000', 'last_became_active': '2024-03-04T06:07:50.381293+0000', 'last_became_peered': '2024-03-04T06:25:58.395107+0000', 'last_unstale': '2024-03-04T06:25:58.395477+0000', 'last_undegraded': '2024-03-04T06:25:58.395477+0000', 'last_fullsized': '2024-03-04T06:25:58.386346+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:28.913 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-03-04T06:30:28.913 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.381019+0000', 'last_change': '2024-03-04T06:25:58.380590+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.381019+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.380590+0000', 'last_unstale': '2024-03-04T06:25:58.381019+0000', 'last_undegraded': '2024-03-04T06:25:58.381019+0000', 'last_fullsized': '2024-03-04T06:25:58.376316+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:28.913 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-03-04T06:30:28.913 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 2627, 'reported_epoch': 1066, 'state': 'active+undersized', 'last_fresh': '2024-03-04T06:25:58.374588+0000', 'last_change': '2024-03-04T06:25:51.500071+0000', 'last_active': '2024-03-04T06:25:58.374588+0000', 'last_peered': '2024-03-04T06:25:58.374588+0000', 'last_clean': '2024-03-04T06:25:38.431398+0000', 'last_became_active': '2024-03-04T06:25:51.500071+0000', 'last_became_peered': '2024-03-04T06:25:51.500071+0000', 'last_unstale': '2024-03-04T06:25:58.374588+0000', 'last_undegraded': '2024-03-04T06:25:58.374588+0000', 'last_fullsized': '2024-03-04T06:25:51.484841+0000', 'mapping_epoch': 1063, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:28.913 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-03-04T06:30:28.913 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393828+0000', 'last_change': '2024-03-04T06:25:58.393464+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.393828+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.393464+0000', 'last_unstale': '2024-03-04T06:25:58.393828+0000', 'last_undegraded': '2024-03-04T06:25:58.393828+0000', 'last_fullsized': '2024-03-04T06:25:58.384366+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:28.913 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-03-04T06:30:28.913 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 10, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.384419+0000', 'last_change': '2024-03-04T06:25:58.384014+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.384419+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.384014+0000', 'last_unstale': '2024-03-04T06:25:58.384419+0000', 'last_undegraded': '2024-03-04T06:25:58.384419+0000', 'last_fullsized': '2024-03-04T06:25:58.379783+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:28.914 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-03-04T06:30:28.914 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 2686, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.395513+0000', 'last_change': '2024-03-04T06:25:58.395110+0000', 'last_active': '2024-03-04T06:25:50.491700+0000', 'last_peered': '2024-03-04T06:25:58.395513+0000', 'last_clean': '2024-03-04T06:25:37.421082+0000', 'last_became_active': '2024-03-04T06:07:50.386944+0000', 'last_became_peered': '2024-03-04T06:25:58.395110+0000', 'last_unstale': '2024-03-04T06:25:58.395513+0000', 'last_undegraded': '2024-03-04T06:25:58.395513+0000', 'last_fullsized': '2024-03-04T06:25:58.384275+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:28.914 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-03-04T06:30:28.914 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.397459+0000', 'last_change': '2024-03-04T06:25:58.397089+0000', 'last_active': '2024-03-04T06:07:49.306852+0000', 'last_peered': '2024-03-04T06:25:58.397459+0000', 'last_clean': '2024-03-04T06:07:49.306852+0000', 'last_became_active': '0.000000', 'last_became_peered': '2024-03-04T06:25:58.397089+0000', 'last_unstale': '2024-03-04T06:25:58.397459+0000', 'last_undegraded': '2024-03-04T06:25:58.397459+0000', 'last_fullsized': '2024-03-04T06:25:58.381325+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T06:30:28.914 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-03-04T06:30:28.914 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 2622, 'reported_epoch': 1066, 'state': 'undersized+peered', 'last_fresh': '2024-03-04T06:25:58.393077+0000', 'last_change': '2024-03-04T06:25:58.392702+0000', 'last_active': '2024-03-04T06:25:50.493405+0000', 'last_peered': '2024-03-04T06:25:58.393077+0000', 'last_clean': '2024-03-04T06:25:38.441197+0000', 'last_became_active': '2024-03-04T06:07:50.392386+0000', 'last_became_peered': '2024-03-04T06:25:58.392702+0000', 'last_unstale': '2024-03-04T06:25:58.393077+0000', 'last_undegraded': '2024-03-04T06:25:58.393077+0000', 'last_fullsized': '2024-03-04T06:25:58.384909+0000', 'mapping_epoch': 1065, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'last_clean_scrub_stamp': '2024-03-04T06:07:49.306852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T06:30:28.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T06:30:29.020 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:29.017+0000 7fb541617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:29.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:29.022+0000 7fb541617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:29.280 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"cb66367d-3032-46d0-abf8-73af48df9eb9","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 8 pgs inactive","count":8},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 9 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":1371,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":1},"osdmap":{"epoch":1066,"num_osds":4,"num_up_osds":2,"osd_up_since":1709533557,"num_in_osds":4,"osd_in_since":1709532460,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"undersized+peered","count":7},{"state_name":"active+undersized","count":1},{"state_name":"undersized+degraded+peered","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":643801088,"bytes_avail":385903255552,"bytes_total":386547056640,"inactive_pgs_ratio":0.8888888955116272,"degraded_objects":4,"degraded_total":6,"degraded_ratio":0.66666666666666652},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-04T06:23:44.908625+0000","services":{}},"progress_events":{}} 2024-03-04T06:30:29.293 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-04T06:30:29.293 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:29.399 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:29.395+0000 7f06b775d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:29.402 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:29.402+0000 7f06b775d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:29.592 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:29.604 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1739,"stamp":"2024-03-04T06:30:29.005823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002803"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:32.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:32.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~303s 2024-03-04T06:30:32.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~303s 2024-03-04T06:30:32.707 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-04T06:30:32.725 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:32.721+0000 7fdbbbd14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:32.728 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:32.728+0000 7fdbbbd14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:32.915 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T06:30:32.915 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-03-04T06:30:32.927 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":1740,"stamp":"2024-03-04T06:30:31.006260+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":10,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":628712,"kb_used_data":382600,"kb_used_omap":0,"kb_used_meta":246080,"kb_avail":376858648,"statfs":{"total":386547056640,"available":385903255552,"internally_reserved":0,"allocated":391782400,"data_stored":444343233,"data_compressed":89883,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":251985920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002685"},"pg_stats":[{"pgid":"1.0","version":"11'97","reported_seq":2819,"reported_epoch":1066,"state":"undersized+degraded+peered","last_fresh":"2024-03-04T06:25:58.377010+0000","last_change":"2024-03-04T06:25:58.376565+0000","last_active":"2024-03-04T06:25:50.479118+0000","last_peered":"2024-03-04T06:25:58.377010+0000","last_clean":"2024-03-04T06:25:37.417194+0000","last_became_active":"2024-03-04T06:07:48.335407+0000","last_became_peered":"2024-03-04T06:25:58.376565+0000","last_unstale":"2024-03-04T06:25:58.377010+0000","last_undegraded":"2024-03-04T06:25:58.372745+0000","last_fullsized":"2024-03-04T06:25:58.372152+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:47.297968+0000","last_clean_scrub_stamp":"2024-03-04T06:07:47.297968+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395477+0000","last_change":"2024-03-04T06:25:58.395107+0000","last_active":"2024-03-04T06:25:50.494728+0000","last_peered":"2024-03-04T06:25:58.395477+0000","last_clean":"2024-03-04T06:25:38.446268+0000","last_became_active":"2024-03-04T06:07:50.381293+0000","last_became_peered":"2024-03-04T06:25:58.395107+0000","last_unstale":"2024-03-04T06:25:58.395477+0000","last_undegraded":"2024-03-04T06:25:58.395477+0000","last_fullsized":"2024-03-04T06:25:58.386346+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.381019+0000","last_change":"2024-03-04T06:25:58.380590+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.381019+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.380590+0000","last_unstale":"2024-03-04T06:25:58.381019+0000","last_undegraded":"2024-03-04T06:25:58.381019+0000","last_fullsized":"2024-03-04T06:25:58.376316+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2627,"reported_epoch":1066,"state":"active+undersized","last_fresh":"2024-03-04T06:25:58.374588+0000","last_change":"2024-03-04T06:25:51.500071+0000","last_active":"2024-03-04T06:25:58.374588+0000","last_peered":"2024-03-04T06:25:58.374588+0000","last_clean":"2024-03-04T06:25:38.431398+0000","last_became_active":"2024-03-04T06:25:51.500071+0000","last_became_peered":"2024-03-04T06:25:51.500071+0000","last_unstale":"2024-03-04T06:25:58.374588+0000","last_undegraded":"2024-03-04T06:25:58.374588+0000","last_fullsized":"2024-03-04T06:25:51.484841+0000","mapping_epoch":1063,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393828+0000","last_change":"2024-03-04T06:25:58.393464+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.393828+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.393464+0000","last_unstale":"2024-03-04T06:25:58.393828+0000","last_undegraded":"2024-03-04T06:25:58.393828+0000","last_fullsized":"2024-03-04T06:25:58.384366+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.384419+0000","last_change":"2024-03-04T06:25:58.384014+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.384419+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.384014+0000","last_unstale":"2024-03-04T06:25:58.384419+0000","last_undegraded":"2024-03-04T06:25:58.384419+0000","last_fullsized":"2024-03-04T06:25:58.379783+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2686,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.395513+0000","last_change":"2024-03-04T06:25:58.395110+0000","last_active":"2024-03-04T06:25:50.491700+0000","last_peered":"2024-03-04T06:25:58.395513+0000","last_clean":"2024-03-04T06:25:37.421082+0000","last_became_active":"2024-03-04T06:07:50.386944+0000","last_became_peered":"2024-03-04T06:25:58.395110+0000","last_unstale":"2024-03-04T06:25:58.395513+0000","last_undegraded":"2024-03-04T06:25:58.395513+0000","last_fullsized":"2024-03-04T06:25:58.384275+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.397459+0000","last_change":"2024-03-04T06:25:58.397089+0000","last_active":"2024-03-04T06:07:49.306852+0000","last_peered":"2024-03-04T06:25:58.397459+0000","last_clean":"2024-03-04T06:07:49.306852+0000","last_became_active":"0.000000","last_became_peered":"2024-03-04T06:25:58.397089+0000","last_unstale":"2024-03-04T06:25:58.397459+0000","last_undegraded":"2024-03-04T06:25:58.397459+0000","last_fullsized":"2024-03-04T06:25:58.381325+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2622,"reported_epoch":1066,"state":"undersized+peered","last_fresh":"2024-03-04T06:25:58.393077+0000","last_change":"2024-03-04T06:25:58.392702+0000","last_active":"2024-03-04T06:25:50.493405+0000","last_peered":"2024-03-04T06:25:58.393077+0000","last_clean":"2024-03-04T06:25:38.441197+0000","last_became_active":"2024-03-04T06:07:50.392386+0000","last_became_peered":"2024-03-04T06:25:58.392702+0000","last_unstale":"2024-03-04T06:25:58.393077+0000","last_undegraded":"2024-03-04T06:25:58.393077+0000","last_fullsized":"2024-03-04T06:25:58.384909+0000","mapping_epoch":1065,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T06:07:49.306852+0000","last_clean_scrub_stamp":"2024-03-04T06:07:49.306852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654707024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114988,"kb_used_data":39588,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94256852,"statfs":{"total":96636764160,"available":96519016448,"internally_reserved":0,"allocated":40538112,"data_stored":44014923,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654707024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":123236,"kb_used_data":60316,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94248604,"statfs":{"total":96636764160,"available":96510570496,"internally_reserved":0,"allocated":61763584,"data_stored":72091188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":208848,"kb_used_data":145928,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94162992,"statfs":{"total":96636764160,"available":96422903808,"internally_reserved":0,"allocated":149430272,"data_stored":168122827,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706723,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":181640,"kb_used_data":136768,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94190200,"statfs":{"total":96636764160,"available":96450764800,"internally_reserved":0,"allocated":140050432,"data_stored":160114295,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-04T06:30:32.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T06:30:33.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:33.031+0000 7f38c806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:33.037 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-04T06:30:33.037+0000 7f38c806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-04T06:30:33.109 INFO:tasks.ceph.osd.1:Sent signal 15 2024-03-04T06:30:33.110 INFO:tasks.ceph.osd.3:Sent signal 15 2024-03-04T06:30:33.110 INFO:tasks.ceph.mon.a:Sent signal 15 2024-03-04T06:30:33.111 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-03-04T06:30:33.111 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-03-04T06:30:33.111+0000 7fa5af7fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 29287) UID: 0 2024-03-04T06:30:33.111 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-03-04T06:30:33.111+0000 7fa5af7fc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-04T06:30:33.111 INFO:tasks.ceph.mgr.x.smithi066.stderr:2024-03-04T06:30:33.111+0000 7f2bc5df7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 29211) UID: 0 2024-03-04T06:32:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-04T06:32:32.964 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/thrashosds.py", line 216, in task cluster_manager.flush_all_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1762, in flush_all_pg_stats self.flush_pg_stats(range(len(self.get_osd_dump()))) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1759, in flush_pg_stats p.spawn(flush_one_osd, osd, wait_for_mon) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1737, in flush_one_osd need = int(self.raw_cluster_cmd('tell', 'osd.%d' % osd, 'flush_pg_stats')) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2778, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2718, in is_clean self.dump_pgs_not_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2739, in dump_pgs_not_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-04T06:32:32.966 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-04T06:32:32.966 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-04T06:32:32.966 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-04T06:32:32.966 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T06:32:32.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T06:32:32.966 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/thrashosds.py", line 216, in task cluster_manager.flush_all_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1762, in flush_all_pg_stats self.flush_pg_stats(range(len(self.get_osd_dump()))) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1759, in flush_pg_stats p.spawn(flush_one_osd, osd, wait_for_mon) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1737, in flush_one_osd need = int(self.raw_cluster_cmd('tell', 'osd.%d' % osd, 'flush_pg_stats')) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2778, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2718, in is_clean self.dump_pgs_not_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2739, in dump_pgs_not_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log' 2024-03-04T06:32:32.967 INFO:tasks.ceph.osd.0:Stopped 2024-03-04T06:32:32.967 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-04T06:32:32.967 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T06:32:33.068 INFO:tasks.ceph.osd.1:Stopped 2024-03-04T06:32:33.068 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-04T06:32:33.068 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T06:32:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T06:32:33.068 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/thrashosds.py", line 216, in task cluster_manager.flush_all_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1762, in flush_all_pg_stats self.flush_pg_stats(range(len(self.get_osd_dump()))) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1759, in flush_pg_stats p.spawn(flush_one_osd, osd, wait_for_mon) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1737, in flush_one_osd need = int(self.raw_cluster_cmd('tell', 'osd.%d' % osd, 'flush_pg_stats')) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2778, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2718, in is_clean self.dump_pgs_not_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2739, in dump_pgs_not_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi066 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2024-03-04T06:32:33.069 INFO:tasks.ceph.osd.2:Stopped 2024-03-04T06:32:33.069 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-04T06:32:33.069 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T06:32:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T06:32:33.069 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/thrashosds.py", line 216, in task cluster_manager.flush_all_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1762, in flush_all_pg_stats self.flush_pg_stats(range(len(self.get_osd_dump()))) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1759, in flush_pg_stats p.spawn(flush_one_osd, osd, wait_for_mon) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1737, in flush_one_osd need = int(self.raw_cluster_cmd('tell', 'osd.%d' % osd, 'flush_pg_stats')) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2778, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2718, in is_clean self.dump_pgs_not_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2739, in dump_pgs_not_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi066 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log' 2024-03-04T06:32:33.070 INFO:tasks.ceph.osd.3:Stopped 2024-03-04T06:32:33.070 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-04T06:32:33.070 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-04T06:32:33.070 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T06:32:33.070 INFO:tasks.ceph.mgr.x:Stopped 2024-03-04T06:32:33.070 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-04T06:32:33.070 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-04T06:32:33.070 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T06:32:33.071 INFO:tasks.ceph.mon.a:Stopped 2024-03-04T06:32:33.071 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-04T06:32:33.071 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-03-04T06:32:33.108 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-04T06:32:33.164 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-04T06:32:33.238 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-04T06:32:33.238 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-04T06:32:33.303 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-04T06:32:33.304 DEBUG:teuthology.orchestra.run.smithi066:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-04T06:32:33.375 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-04T06:32:33.375 DEBUG:teuthology.orchestra.run.smithi066:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-04T06:32:33.451 INFO:tasks.ceph:Archiving mon data... 2024-03-04T06:32:33.452 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638/data/mon.a.tgz 2024-03-04T06:32:33.452 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-04T06:32:33.467 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.6Yrb9tGtmG 2024-03-04T06:32:33.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.6Yrb9tGtmG 2024-03-04T06:32:33.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.6Yrb9tGtmG 2024-03-04T06:32:33.694 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.6Yrb9tGtmG is 313KB 2024-03-04T06:32:33.790 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.6Yrb9tGtmG 2024-03-04T06:32:33.804 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-04T06:32:33.804 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-04T06:32:33.846 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-04T06:32:33.924 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-04T06:32:33.939 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-04T06:32:33.939 INFO:tasks.ceph:Archiving crash dumps... 2024-03-04T06:32:33.941 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638/remote/smithi064/crash 2024-03-04T06:32:33.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-04T06:32:33.990 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/lib/ceph/crash to /home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638/remote/smithi066/crash 2024-03-04T06:32:33.991 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-04T06:32:34.034 INFO:tasks.ceph:Compressing logs... 2024-03-04T06:32:34.034 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T06:32:34.036 DEBUG:teuthology.orchestra.run.smithi066:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T06:32:34.119 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29371.log 2024-03-04T06:32:34.119 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29291.log 2024-03-04T06:32:34.119 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.29371.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-04T06:32:34.119 INFO:teuthology.orchestra.run.smithi066.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29371.log.gz 2024-03-04T06:32:34.119 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.29291.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-04T06:32:34.119 INFO:teuthology.orchestra.run.smithi066.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29291.log.gz 2024-03-04T06:32:34.119 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-04T06:32:34.119 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29411.log 2024-03-04T06:32:34.119 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29331.log 2024-03-04T06:32:34.119 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.29411.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29411.log.gz 2024-03-04T06:32:34.120 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.29331.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29331.log.gz 2024-03-04T06:32:34.302 INFO:teuthology.orchestra.run.smithi066.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-04T06:32:34.568 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75984.log 2024-03-04T06:32:34.568 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36484.log 2024-03-04T06:32:34.568 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75257.log 2024-03-04T06:32:34.568 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.75984.log.gz 2024-03-04T06:32:34.568 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42366.log 2024-03-04T06:32:34.568 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36484.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36484.log.gz 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47512.log 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75257.log.gz 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40571.log 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42366.log.gz 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38533.log 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47512.log.gz 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50527.log 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40571.log.gz 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53860.log 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38533.log.gz 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73635.log 2024-03-04T06:32:34.569 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50527.log.gz 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70990.log 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.53860.log.gz 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58426.log 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73635.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.73635.log.gz 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72466.log 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.70990.log.gz 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33425.log 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.58426.log.gz 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41196.log 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72466.log: 49.6%/var/log/ceph/ceph-client.admin.33425.log: -- replaced with /var/log/ceph/ceph-client.admin.72466.log.gz 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87879.log 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33425.log.gz 2024-03-04T06:32:34.570 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33781.log 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41196.log.gz 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73285.log 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.87879.log.gz 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85821.log 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33781.log.gz 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66289.log 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.73285.log.gz 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56736.log 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85821.log.gz 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39963.log 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.66289.log.gz 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.56736.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.66030.log 2024-03-04T06:32:34.571 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.56736.log.gz 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47999.log 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39963.log.gz 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48358.log 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66030.log.gz 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39634.log 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47999.log.gz 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81158.log 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48358.log.gz 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35476.log/var/log/ceph/ceph-client.admin.39634.log: 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39634.log.gz 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81158.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81158.log.gz 2024-03-04T06:32:34.572 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80570.log 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35476.log.gz 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67963.log 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39425.log 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80570.log.gz 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82250.log 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67963.log.gz 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80528.log 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39425.log.gz 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69529.log 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82250.log.gz 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82796.log 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80528.log.gz 2024-03-04T06:32:34.573 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73459.log 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.69529.log.gz 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70276.log 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82796.log.gz 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36993.log 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73459.log.gz 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68332.log 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.70276.log.gz 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37300.log 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36993.log.gz 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56776.log 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.68332.log.gz 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82712.log 2024-03-04T06:32:34.574 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37300.log.gz 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64685.log 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56776.log.gz 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76739.log 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82712.log.gz 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35338.log 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.64685.log.gz 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41681.log 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76739.log.gz 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37420.log 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35338.log.gz 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76641.log 2024-03-04T06:32:34.575 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41681.log.gz 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74002.log 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37420.log.gz 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52924.log 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76641.log.gz 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53295.log 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.74002.log.gz 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35189.log 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52924.log.gz 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50736.log 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.53295.log.gz 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81704.log 2024-03-04T06:32:34.576 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35189.log.gz 2024-03-04T06:32:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83258.log 2024-03-04T06:32:34.577 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50736.log.gz 2024-03-04T06:32:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60957.log 2024-03-04T06:32:34.577 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81704.log.gz 2024-03-04T06:32:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86325.log 2024-03-04T06:32:34.577 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83258.log.gz 2024-03-04T06:32:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60252.log 2024-03-04T06:32:34.577 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.60957.log.gz 2024-03-04T06:32:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30107.log 2024-03-04T06:32:34.577 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86325.log.gz 2024-03-04T06:32:34.577 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48715.log 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.60252.log.gz 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38224.log 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30107.log.gz 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40531.log 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48715.log.gz 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56469.log 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38224.log.gz 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31692.log 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40531.log.gz 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49591.log 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56469.log.gz 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45770.log 2024-03-04T06:32:34.578 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31692.log.gz 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53113.log 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49591.log.gz 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52221.log 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45770.log.gz 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49462.log 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.53113.log.gz 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45532.log 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52221.log.gz 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38800.log 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49462.log.gz 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67331.log 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45532.log.gz 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87669.log 2024-03-04T06:32:34.579 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38800.log.gz 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72549.log 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67331.log.gz 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49724.log 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87669.log.gz 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63695.log 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.72549.log.gz 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66478.log 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49724.log.gz 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61146.log 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63695.log.gz 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55557.log 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66478.log.gz 2024-03-04T06:32:34.580 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88047.log 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.61146.log.gz 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89099.log 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55557.log.gz 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36434.log 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88047.log.gz 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42693.log 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.89099.log.gz 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75575.log 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36434.log.gz 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84854.log 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42693.log.gz 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55508.log 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75575.log.gz 2024-03-04T06:32:34.581 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74964.log 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84854.log.gz 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77228.log 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.55508.log.gz 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73824.log 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.74964.log.gz 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72326.log 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.77228.log.gz 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50785.log 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.73824.log.gz 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33901.log 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.72326.log.gz 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78138.log 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50785.log.gz 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56378.log 2024-03-04T06:32:34.582 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33901.log.gz 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44905.log 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.78138.log.gz 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46756.log 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.56378.log.gz 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74914.log 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44905.log.gz 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29924.log 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46756.log.gz 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45672.log 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.74914.log.gz 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29924.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29924.log.gzgzip 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65748.log 2024-03-04T06:32:34.583 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82208.log 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45672.log.gz 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57980.log 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.65748.log.gz 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70368.log 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82208.log.gz 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57980.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57980.log.gz 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72277.log 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85232.log 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70368.log.gz 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35946.log 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.72277.log.gz 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42068.log 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85232.log.gz 2024-03-04T06:32:34.584 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80276.log 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35946.log.gz 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72641.log 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42068.log.gz 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85610.log 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80276.log.gz 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66925.log 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.72641.log.gz 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38493.log 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85610.log.gz 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71339.log 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66925.log.gz 2024-03-04T06:32:34.585 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44303.log 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38493.log.gz 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82460.log 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71339.log.gz 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62941.log 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44303.log.gz 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71479.log 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82460.log.gz 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77801.log 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62941.log.gz 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49893.log 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71479.log.gz 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58837.log 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.77801.log.gz 2024-03-04T06:32:34.586 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52568.log 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49893.log.gz 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32967.log 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58837.log.gz 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74411.log 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52568.log.gz 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53413.log 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32967.log.gz 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48966.log 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74411.log.gz 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58020.log 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.53413.log.gz 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74600.log 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48966.log.gz 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34426.log 2024-03-04T06:32:34.587 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.58020.log.gz 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77148.log 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74600.log.gz 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39236.log 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34426.log.gz 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67156.log 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.77148.log.gz 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50647.log 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39236.log.gz 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56189.log 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67156.log.gz 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80822.log 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50647.log.gz 2024-03-04T06:32:34.588 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40340.log 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.56189.log.gz 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43408.log 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80822.log.gz 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89267.log 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40340.log.gz 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52826.log 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43408.log.gz 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32172.log 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.89267.log.gz 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83384.log 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52826.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52826.log.gz 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47861.log 2024-03-04T06:32:34.589 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32172.log.gz 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38453.log 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83384.log.gz 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56567.log 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47861.log.gz 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88679.log 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38453.log.gz 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46487.log 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56567.log.gz 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74369.log 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.88679.log.gz 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77550.log 2024-03-04T06:32:34.590 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46487.log.gz 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81830.log 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.74369.log.gz 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41423.log 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.77550.log.gz 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35986.log 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81830.log.gz 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46079.log 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41423.log.gz 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88847.log 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35986.log.gz 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79394.log 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46079.log.gz 2024-03-04T06:32:34.591 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46309.log 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88847.log.gz 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87837.log 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.79394.log.gz 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69975.log 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46309.log.gz 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55428.log 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87837.log.gz 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60490.log 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69975.log.gz 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34911.log 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55428.log.gz 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57762.log 2024-03-04T06:32:34.592 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.60490.log.gz 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67883.log 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34911.log.gz 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50122.log 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.57762.log.gz 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88341.log 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.67883.log.gz 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73873.log 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50122.log.gz 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79100.log 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.88341.log.gz 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74831.log 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73873.log.gz 2024-03-04T06:32:34.593 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52359.log 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79100.log.gz 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67702.log 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74831.log.gz 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55768.log 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52359.log.gz 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34773.log 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67702.log.gz 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55881.log 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.55768.log.gz 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36624.log 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34773.log.gz 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41543.log 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.55881.log.gz 2024-03-04T06:32:34.594 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30425.log 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36624.log.gz 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37561.log 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41543.log.gz 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32041.log 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30425.log.gz 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57713.log 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37561.log.gz 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68521.log 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32041.log.gz 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29882.log 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57713.log.gz 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40858.log 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.68521.log.gz 2024-03-04T06:32:34.595 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53909.log 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29882.log.gz 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70782.log 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40858.log.gz 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31953.log 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.53909.log.gz 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50429.log 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70782.log.gz 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43801.log 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31953.log.gz 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82292.log 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50429.log.gz 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74698.log 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43801.log.gz 2024-03-04T06:32:34.596 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40103.log 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82292.log.gz 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37917.log 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74698.log.gz 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57622.log 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40103.log.gz 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47022.log 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37917.log.gz 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32927.log 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57622.log.gz 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30613.log 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.47022.log.gz 2024-03-04T06:32:34.597 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46267.log 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32927.log.gz 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66072.log 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30613.log.gz 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34297.log 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46267.log.gz 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76452.log 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66072.log.gz 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33613.log 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34297.log.gz 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75655.log 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.76452.log.gz 2024-03-04T06:32:34.598 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78596.log 2024-03-04T06:32:34.599 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33613.log.gz 2024-03-04T06:32:34.599 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84896.log 2024-03-04T06:32:34.599 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75655.log.gz 2024-03-04T06:32:34.599 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86283.log 2024-03-04T06:32:34.599 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.78596.log.gz 2024-03-04T06:32:34.599 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50865.log 2024-03-04T06:32:34.599 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84896.log.gz 2024-03-04T06:32:34.599 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33554.log 2024-03-04T06:32:34.599 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86283.log.gz 2024-03-04T06:32:34.599 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85568.log 2024-03-04T06:32:34.599 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50865.log.gz 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57573.log 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33554.log.gz 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48119.log 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85568.log.gz 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79436.log 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57573.log.gz 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71948.log 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48119.log.gz 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43043.log 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79436.log.gz 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80948.log 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.71948.log.gz 2024-03-04T06:32:34.600 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70459.log 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43043.log.gz 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73194.log 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80948.log.gz 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70144.log 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70459.log.gz 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41988.log 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73194.log.gz 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63964.log 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70144.log.gz 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85400.log 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41988.log.gz 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64552.log 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63964.log.gz 2024-03-04T06:32:34.601 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85695.log 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85400.log.gz 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62455.log 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.64552.log.gz 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68759.log 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85695.log.gz 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86535.log 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62455.log.gz 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36308.log 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.68759.log.gz 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31471.log 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86535.log.gz 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76830.log 2024-03-04T06:32:34.602 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36308.log.gz 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81578.log 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31471.log.gz 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43275.log 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76830.log.gz 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88721.log 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81578.log.gz 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41156.log 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43275.log.gz 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68192.log 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88721.log.gz 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30008.log 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41156.log.gz 2024-03-04T06:32:34.603 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57182.log 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.68192.log.gz 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55922.log 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30008.log.gz 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82418.log 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57182.log.gz 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75326.log 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55922.log.gz 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35651.log 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82418.log.gz 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79898.log 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75326.log.gz 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57007.log 2024-03-04T06:32:34.604 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35651.log.gz 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78890.log 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.79898.log.gz 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59056.log 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.57007.log.gz 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36084.log 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.78890.log.gz 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75128.log 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.59056.log.gz 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89141.log 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36084.log.gz 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66836.log 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75128.log.gz 2024-03-04T06:32:34.605 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84560.log 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.89141.log.gz 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65043.log 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66836.log.gz 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42108.log 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.84560.log.gz 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69164.log 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.65043.log.gz 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75844.log 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42108.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42108.log.gz 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52439.log 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69164.log.gz 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40052.log 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75844.log.gz 2024-03-04T06:32:34.606 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86997.log 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52439.log.gz 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29776.log 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40052.log.gz 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46120.log 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86997.log.gz 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58180.log 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29776.log.gz 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57671.log 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.46120.log.gz 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49813.log 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.58180.log.gz 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53602.log 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57671.log.gz 2024-03-04T06:32:34.607 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59007.log 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49813.log.gz 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77508.log 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.53602.log.gz 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85652.log 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59007.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.59007.log.gz 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41939.log 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.77508.log.gz 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46218.log 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85652.log.gz 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46616.log 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41939.log.gz 2024-03-04T06:32:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80318.log 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46218.log.gz 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66974.log 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46616.log.gz 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81536.log 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80318.log.gz 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81788.log 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.66974.log.gz 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53562.log 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81536.log.gz 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58699.log 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81788.log.gz 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77368.log 2024-03-04T06:32:34.609 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.53562.log.gz 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87291.log 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.58699.log.gz 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56656.log 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.77368.log.gz 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47204.log 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87291.log.gz 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84434.log 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.56656.log.gz 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31587.log 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47204.log.gz 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69906.log 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84434.log.gz 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47770.log 2024-03-04T06:32:34.610 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31587.log.gz 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55299.log 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.69906.log.gz 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82880.log 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47770.log.gz 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75615.log 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.55299.log.gz 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69256.log 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82880.log.gz 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47632.log 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.75615.log.gz 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62133.log 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69256.log.gz 2024-03-04T06:32:34.611 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81032.log 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47632.log.gz 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61692.log 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62133.log.gz 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78722.log 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81032.log.gz 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74131.log 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.61692.log.gz 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82754.log 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.78722.log.gz 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88173.log 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.74131.log.gz 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37659.log 2024-03-04T06:32:34.612 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82754.log.gz 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75208.log 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88173.log.gz 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77417.log 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37659.log.gz 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64363.log 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.75208.log.gz 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54941.log 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.77417.log.gz 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89015.log 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.64363.log.gz 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59715.log 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.54941.log.gz 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43681.log 2024-03-04T06:32:34.613 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.89015.log.gz 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82586.log 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.59715.log.gz 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72415.log 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43681.log.gz 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32132.log 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82586.log.gz 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57493.log 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.72415.log.gz 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76372.log 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32132.log.gz 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57940.log 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57493.log.gz 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70409.log 2024-03-04T06:32:34.614 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76372.log.gz 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45721.log 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57940.log.gz 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52617.log 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70409.log.gz 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33334.log 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45721.log.gz 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36904.log 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52617.log.gz 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36764.log 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33334.log.gz 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63308.log 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36904.log.gz 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54714.log 2024-03-04T06:32:34.615 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36764.log.gz 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38880.log 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63308.log.gz 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66205.log 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.54714.log.gz 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67923.log 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38880.log.gz 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37470.log 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.66205.log.gz 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45861.log 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.67923.log.gz 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42902.log 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37470.log.gz 2024-03-04T06:32:34.616 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37788.log 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45861.log.gz 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54368.log 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42902.log.gz 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30108.log 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37788.log.gz 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75753.log 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.54368.log.gz 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61467.log 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30108.log.gz 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83342.log 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75753.log.gz 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58468.log 2024-03-04T06:32:34.617 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.61467.log.gz 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64811.log 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83342.log.gz 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76292.log 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.58468.log.gz 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49273.log 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.64811.log.gz 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44736.log 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.76292.log.gz 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49273.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28233.log 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49273.log.gz 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64321.log 2024-03-04T06:32:34.618 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44736.log.gz 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.tmp-client.admin.28233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61517.log 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28233.log.gz 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38573.log 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.64321.log.gz 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63139.log 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.61517.log.gz 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32361.log 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38573.log.gz 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49773.log 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63139.log.gz 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71577.log 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32361.log.gz 2024-03-04T06:32:34.619 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38413.log 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49773.log.gz 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39058.log 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71577.log.gz 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64727.log 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38413.log.gz 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89519.log 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39058.log.gz 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52052.log 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.64727.log.gz 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40278.log 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.89519.log.gz 2024-03-04T06:32:34.620 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77108.log 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52052.log.gz 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83552.log 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40278.log.gz 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53700.log 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.77108.log.gz 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82838.log 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.83552.log.gz 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71388.log 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.53700.log.gz 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43534.log 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82838.log.gz 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83720.log 2024-03-04T06:32:34.621 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71388.log.gz 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40145.log 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43534.log.gz 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79562.log 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83720.log.gz 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51175.log 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40145.log.gz 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80696.log 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.79562.log.gz 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81452.log 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51175.log.gz 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44394.log 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80696.log.gz 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77277.log 2024-03-04T06:32:34.622 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81452.log.gz 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42773.log 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44394.log.gz 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30276.log 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77277.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.77277.log.gz 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68899.log 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42773.log.gz 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76159.log 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30276.log.gz 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79016.log 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.68899.log.gz 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57453.log 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76159.log.gz 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59553.log 2024-03-04T06:32:34.623 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.79016.log.gz 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32619.log 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.57453.log.gz 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35906.log 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.59553.log.gz 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44836.log 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32619.log.gz 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43334.log 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35906.log.gz 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59804.log 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44836.log.gz 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86157.log 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43334.log.gz 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64951.log 2024-03-04T06:32:34.624 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59804.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.59804.log.gz 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49322.log 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.86157.log.gz 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61006.log 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.64951.log.gz 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48168.log 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49322.log.gz 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43450.log 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.61006.log.gz 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66113.log 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48168.log.gz 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43450.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43450.log.gz 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37610.log 2024-03-04T06:32:34.625 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55650.log 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66113.log.gz 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84350.log 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37610.log.gz 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45085.log 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55650.log.gz 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66756.log 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84350.log.gz 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62413.log 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45085.log.gz 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52875.log 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66756.log.gz 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41325.log 2024-03-04T06:32:34.626 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62413.log.gz 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75942.log 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.52875.log.gz 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78532.log 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41325.log.gz 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35378.log 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75942.log.gz 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76117.log 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.78532.log.gz 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48797.log 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35378.log.gz 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55021.log 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.76117.log.gz 2024-03-04T06:32:34.627 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71081.log 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48797.log.gz 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72046.log 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55021.log.gz 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87459.log 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71081.log.gz 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39565.log 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.72046.log.gz 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55159.log 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.87459.log.gz 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34079.log 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39565.log.gz 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68850.log 2024-03-04T06:32:34.628 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.55159.log.gz 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30950.log 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34079.log.gz 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76069.log 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.68850.log.gz 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33047.log 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30950.log.gz 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65848.log 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.76069.log.gz 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77844.log 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33047.log.gz 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35298.log 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.65848.log.gz 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82334.log 2024-03-04T06:32:34.629 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.77844.log.gz 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75446.log 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35298.log.gz 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81914.log 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82334.log.gz 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84308.log 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75446.log.gz 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69668.log 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81914.log.gz 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68779.log 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84308.log.gz 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54416.log 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69668.log.gz 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67023.log 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.68779.log.gz 2024-03-04T06:32:34.630 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52666.log 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.54416.log.gz 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64594.log 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67023.log.gz 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40012.log 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52666.log.gz 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84056.log 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.64594.log.gz 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57533.log 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40012.log.gz 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70892.log 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84056.log.gz 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76592.log 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57533.log.gz 2024-03-04T06:32:34.631 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84140.log 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70892.log.gz 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57860.log 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.76592.log.gz 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45441.log 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84140.log.gz 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53522.log 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57860.log.gz 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54287.log 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45441.log.gz 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45341.log 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.53522.log.gz 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36217.log 2024-03-04T06:32:34.632 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.54287.log.gz 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86661.log 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45341.log.gz 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36217.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36217.log.gz 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65490.log 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71857.log 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86661.log.gz 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45134.log 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.65490.log.gz 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70235.log 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.71857.log.gz 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62546.log 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45134.log.gz 2024-03-04T06:32:34.633 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70235.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.70235.log.gz 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84014.log 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.62546.log.gz 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66576.log 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42388.log 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84014.log.gz 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64182.log 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66576.log.gz 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47344.log 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42388.log.gz 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33861.log 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.64182.log.gz 2024-03-04T06:32:34.634 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44518.log 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47344.log.gz 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50945.log 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33861.log.gz 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40938.log 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44518.log.gz 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67464.log 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50945.log.gz 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59854.log 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40938.log.gz 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52746.log 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67464.log.gz 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-04T06:32:34.635 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84224.log 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.59854.log.gz 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77928.log 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52746.log.gz 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76921.log 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84224.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84224.log.gz 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45392.log 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.77928.log.gz 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42733.log 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76921.log.gz 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50171.log 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45392.log.gz 2024-03-04T06:32:34.636 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85484.log 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42733.log.gz 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64013.log 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50171.log.gz 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87123.log 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85484.log.gz 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85190.log 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.64013.log.gz 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88553.log 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87123.log.gz 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67513.log 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85190.log.gz 2024-03-04T06:32:34.637 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38622.log 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.88553.log.gz 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61894.log 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67513.log.gz 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84098.log 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38622.log.gz 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63468.log 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.61894.log.gz 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53820.log 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84098.log.gz 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31470.log 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.63468.log.gz 2024-03-04T06:32:34.638 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36391.log 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.53820.log.gz 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36855.log 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31470.log.gz 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59196.log 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36391.log.gz 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79058.log 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36855.log.gz 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40898.log 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.59196.log.gz 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50567.log 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79058.log.gz 2024-03-04T06:32:34.639 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34346.log 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40898.log.gz 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33683.log 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50567.log.gz 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86913.log 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34346.log.gz 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61244.log 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33683.log.gz 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66338.log 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86913.log.gz 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87753.log 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.61244.log.gz 2024-03-04T06:32:34.640 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66338.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.66338.log.gz 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35826.log 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72745.log 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.87753.log.gz 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62012.log 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35826.log.gz 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54098.log 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.72745.log.gz 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86241.log 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.62012.log.gz 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69708.log 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.54098.log.gz 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30447.log 2024-03-04T06:32:34.641 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86241.log.gz 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83678.log 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69708.log.gz 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61559.log 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30447.log.gz 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67843.log 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83678.log.gz 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37519.log 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.61559.log.gz 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34684.log 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.67843.log.gz 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45910.log 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37519.log.gz 2024-03-04T06:32:34.642 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67198.log 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34684.log.gz 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49364.log 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45910.log.gz 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73243.log 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67198.log.gz 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81242.log 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49364.log.gz 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56329.log 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73243.log.gz 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35169.log 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81242.log.gz 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87417.log 2024-03-04T06:32:34.643 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56329.log.gz 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43233.log 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35169.log.gz 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81998.log 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.87417.log.gz 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30554.log 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43233.log.gz 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34386.log 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81998.log.gz 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62322.log 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30554.log.gz 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83132.log 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34386.log.gz 2024-03-04T06:32:34.644 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80066.log 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62322.log.gz 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60797.log 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83132.log.gz 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74042.log 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.80066.log.gz 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57266.log 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.60797.log.gz 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84686.log 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74042.log.gz 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83006.log 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57266.log.gz 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49224.log 2024-03-04T06:32:34.645 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84686.log.gz 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83300.log 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83006.log.gz 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82922.log 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49224.log.gz 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63428.log 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83300.log.gz 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71528.log 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82922.log.gz 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66618.log 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63428.log.gz 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52479.log 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71528.log.gz 2024-03-04T06:32:34.646 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31906.log 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66618.log.gz 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63021.log 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52479.log.gz 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48582.log 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31906.log.gz 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46930.log 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.63021.log.gz 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47155.log 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48582.log.gz 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66865.log 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46930.log.gz 2024-03-04T06:32:34.647 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81494.log 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47155.log.gz 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50687.log 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66865.log.gz 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40809.log 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81494.log.gz 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48877.log 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50687.log.gz 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62630.log 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40809.log.gz 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62362.log 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48877.log.gz 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78638.log 2024-03-04T06:32:34.648 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.62630.log.gz 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46714.log 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.62362.log.gz 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38269.log 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.78638.log.gz 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46039.log 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46714.log.gz 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49015.log 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38269.log.gz 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37340.log 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46039.log.gz 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63517.log 2024-03-04T06:32:34.649 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49015.log.gz 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81116.log 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37340.log.gz 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42028.log 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63517.log.gz 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61384.log 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.81116.log.gz 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38175.log 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42028.log.gz 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70634.log 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.61384.log.gz 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86451.log 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38175.log.gz 2024-03-04T06:32:34.650 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67555.log 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.70634.log.gz 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79226.log 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.86451.log.gz 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39694.log 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.67555.log.gz 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89603.log 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.79226.log.gz 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68052.log 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39694.log.gz 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48757.log 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.89603.log.gz 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41583.log 2024-03-04T06:32:34.651 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.68052.log.gz 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56056.log 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48757.log.gz 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81284.log 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41583.log.gz 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35071.log 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56056.log.gz 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47672.log 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81284.log.gz 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53651.log 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35071.log.gz 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47064.log 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47672.log.gz 2024-03-04T06:32:34.652 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86367.log 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.53651.log.gz 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81074.log 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47064.log.gz 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46978.log 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86367.log.gz 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83048.log 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81074.log.gz 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42246.log 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46978.log.gz 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87375.log 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83048.log.gz 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52310.log 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42246.log.gz 2024-03-04T06:32:34.653 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78012.log 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87375.log.gz 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82502.log 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52310.log.gz 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73962.log 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.78012.log.gz 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71437.log 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82502.log.gz 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58293.log 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.73962.log.gz 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57413.log 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71437.log.gz 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88763.log 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58293.log.gz 2024-03-04T06:32:34.654 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68570.log 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57413.log.gz 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46169.log 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.88763.log.gz 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83216.log 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.68570.log.gz 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33145.log 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46169.log.gz 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48217.log 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83216.log.gz 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47324.log 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33145.log.gz 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60392.log 2024-03-04T06:32:34.655 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48217.log.gz 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34704.log 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47324.log.gz 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84476.log 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.60392.log.gz 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55748.log 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34704.log.gz 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76879.log 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84476.log.gz 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66667.log 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55748.log.gz 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48673.log 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.76879.log.gz 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65980.log 2024-03-04T06:32:34.656 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66667.log.gz 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56874.log 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48673.log.gz 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70783.log 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.65980.log.gz 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46889.log 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56874.log.gz 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87333.log 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.70783.log.gz 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50031.log 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46889.log.gz 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30782.log 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87333.log.gz 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71766.log 2024-03-04T06:32:34.657 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50031.log.gz 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29524.log 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30782.log.gz 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45183.log 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71766.log.gz 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40491.log 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29524.log.gz 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72186.log 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45183.log.gz 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63915.log 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40491.log.gz 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38006.log 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.72186.log.gz 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32321.log 2024-03-04T06:32:34.658 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63915.log.gz 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86871.log 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38006.log.gz 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72914.log 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32321.log.gz 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79940.log 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86871.log.gz 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73593.log 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.72914.log.gz 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58384.log 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.79940.log.gz 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51043.log 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.73593.log.gz 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30910.log 2024-03-04T06:32:34.659 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58384.log.gz 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80864.log 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51043.log.gz 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52261.log 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30910.log.gz 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54545.log 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80864.log.gz 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43093.log 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.52261.log.gz 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63873.log 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.54545.log.gz 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60301.log 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43093.log.gz 2024-03-04T06:32:34.660 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79814.log 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63873.log.gz 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68101.log 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.60301.log.gz 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66427.log 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.79814.log.gz 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53344.log 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.68101.log.gz 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86199.log 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.66427.log.gz 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41721.log 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.53344.log.gz 2024-03-04T06:32:34.661 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81746.log 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86199.log.gz 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40451.log 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41721.log.gz 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44945.log 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81746.log.gz 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58748.log 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40451.log.gz 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47552.log 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44945.log.gz 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88931.log 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58748.log.gz 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85526.log 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47552.log.gz 2024-03-04T06:32:34.662 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75048.log 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88931.log.gz 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78323.log 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85526.log.gz 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61286.log 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.75048.log.gz 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77466.log 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.78323.log.gz 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73096.log 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.61286.log.gz 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80612.log 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.77466.log.gz 2024-03-04T06:32:34.663 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73096.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73096.log.gz 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35518.log 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80612.log.gz 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84518.log 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35518.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35518.log.gz 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73376.log 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59101.log 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.84518.log.gz 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73376.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73376.log.gz 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45263.log 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47721.log 2024-03-04T06:32:34.664 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.59101.log.gz 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45263.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45263.log.gz 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.75006.log 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79352.log 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47721.log.gz 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75006.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.75006.log.gz 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84266.log 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87501.log 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79352.log.gz 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73145.log 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84266.log.gz 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81410.log 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.87501.log.gz 2024-03-04T06:32:34.665 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73145.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73145.log.gz 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34119.log 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80654.log 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81410.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81410.log.gz 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34119.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34119.log.gz 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74649.log 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75495.log 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80654.log: /var/log/ceph/ceph-client.admin.74649.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80654.log.gz 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68381.log 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74649.log.gz 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67653.log 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.75495.log.gz 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62670.log 2024-03-04T06:32:34.666 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.68381.log.gz 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80738.log 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.67653.log.gz 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79184.log 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62670.log.gz 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62763.log 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80738.log.gz 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89561.log 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79184.log.gz 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64769.log 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62763.log.gz 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54049.log 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.89561.log.gz 2024-03-04T06:32:34.667 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57900.log 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.64769.log.gz 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67239.log 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.54049.log.gz 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57900.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57900.log.gz 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51674.log 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80234.log 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.67239.log.gz 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51674.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51674.log.gz 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73326.log 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79688.log 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.80234.log.gz 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73326.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.73326.log.gz 2024-03-04T06:32:34.668 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38046.log 2024-03-04T06:32:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62224.log 2024-03-04T06:32:34.669 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.79688.log.gz 2024-03-04T06:32:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87207.log 2024-03-04T06:32:34.669 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38046.log.gz 2024-03-04T06:32:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62224.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62224.log.gz 2024-03-04T06:32:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66527.log 2024-03-04T06:32:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48540.log 2024-03-04T06:32:34.669 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87207.log.gz 2024-03-04T06:32:34.669 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32272.log 2024-03-04T06:32:34.669 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66527.log.gz 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38840.log 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48540.log.gz 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53740.log 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32272.log.gz 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81200.log 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38840.log.gz 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85148.log 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.53740.log.gz 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81200.log: gzip 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81200.log.gz -5 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.65939.log 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52101.log 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85148.log.gz 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48622.log 2024-03-04T06:32:34.670 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.65939.log.gz 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52101.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.52101.log.gz 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49551.log 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73509.log 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48622.log.gz 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45819.log 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49551.log.gz 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41632.log 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.73509.log.gz 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45819.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45819.log.gz 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34039.log 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79772.log 2024-03-04T06:32:34.671 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41632.log.gz 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34140.log 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34039.log.gz 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52003.log 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79772.log.gz 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39138.log 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34140.log.gz 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74082.log 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52003.log.gz 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39138.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.41018.log 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39138.log.gz 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73922.log 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.74082.log.gz 2024-03-04T06:32:34.672 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36257.log 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41018.log.gz 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34555.log 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73922.log.gz 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43142.log 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36257.log.gz 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34555.log: gzip -5 --verbose -- 49.6% /var/log/ceph/ceph-client.admin.71668.log 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34555.log.gz 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50905.log 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43142.log.gz 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65261.log 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71668.log.gz 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59943.log 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50905.log.gz 2024-03-04T06:32:34.673 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65212.log 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.65261.log.gz 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41761.log 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.59943.log.gz 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85022.log 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.65212.log.gz 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41761.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41761.log.gz -5 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62718.log 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69298.log 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85022.log.gz 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69030.log 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62718.log.gz 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76250.log 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69298.log.gz 2024-03-04T06:32:34.674 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47959.log 2024-03-04T06:32:34.675 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.69030.log.gz 2024-03-04T06:32:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77760.log 2024-03-04T06:32:34.675 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.76250.log.gz 2024-03-04T06:32:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63119.log 2024-03-04T06:32:34.675 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47959.log.gz 2024-03-04T06:32:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78180.log 2024-03-04T06:32:34.675 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.77760.log.gz 2024-03-04T06:32:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63119.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63119.log.gz 2024-03-04T06:32:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86577.log 2024-03-04T06:32:34.675 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41236.log 2024-03-04T06:32:34.675 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.78180.log.gz 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37679.log 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86577.log.gz 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37211.log 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41236.log.gz 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37679.log: 50.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.37679.log.gz 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39018.log 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54634.log 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37211.log.gz 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63070.log 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39018.log.gz 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63735.log 2024-03-04T06:32:34.676 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.54634.log.gz 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63070.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63070.log.gz 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31742.log 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47106.log 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63735.log: 50.0%/var/log/ceph/ceph-client.admin.31742.log: -- replaced with /var/log/ceph/ceph-client.admin.63735.log.gz 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36526.log 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31742.log.gz 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47810.log 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47106.log.gz 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82082.log 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36526.log.gz 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81326.log 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47810.log.gz 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76501.log 2024-03-04T06:32:34.677 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82082.log.gz 2024-03-04T06:32:34.678 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78222.log 2024-03-04T06:32:34.678 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81326.log.gz 2024-03-04T06:32:34.678 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80990.log 2024-03-04T06:32:34.678 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.76501.log.gz 2024-03-04T06:32:34.678 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70055.log 2024-03-04T06:32:34.678 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.78222.log.gz 2024-03-04T06:32:34.678 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80990.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80990.log.gz 2024-03-04T06:32:34.678 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44478.log 2024-03-04T06:32:34.678 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85274.log 2024-03-04T06:32:34.678 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70055.log.gz 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34506.log 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44478.log.gz 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44776.log 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85274.log.gz 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84980.log 49.6% 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34506.log.gz 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56118.log 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44776.log.gz 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84980.log: 50.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.84980.log.gz 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44558.log 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60203.log 2024-03-04T06:32:34.679 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.56118.log.gz 2024-03-04T06:32:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44558.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44558.log.gz -5 2024-03-04T06:32:34.680 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45043.log 2024-03-04T06:32:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51723.log 2024-03-04T06:32:34.680 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.60203.log.gz 2024-03-04T06:32:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89309.log 2024-03-04T06:32:34.680 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45043.log.gz 2024-03-04T06:32:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88089.log 2024-03-04T06:32:34.680 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51723.log.gz 2024-03-04T06:32:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89309.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.89309.log.gz 2024-03-04T06:32:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67373.log 2024-03-04T06:32:34.680 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70193.log 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88089.log.gz 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70015.log 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67373.log.gz 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59595.log 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.70193.log.gz 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70015.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.70015.log.gz 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54803.log 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60063.log 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.59595.log.gz 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82544.log 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.54803.log.gz 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86829.log 2024-03-04T06:32:34.681 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.60063.log.gz 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82544.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.82544.log.gz 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47463.log 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54007.log 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86829.log.gz 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47463.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47463.log.gz 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87921.log 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72508.log 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.54007.log.gz 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62901.log 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87921.log.gz 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63824.log 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.72508.log.gz 2024-03-04T06:32:34.682 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62901.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.62901.log.gz 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43583.log 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51583.log 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63824.log.gz 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61600.log 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43583.log.gz 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89477.log 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51583.log.gz 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61600.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.61600.log.gz 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30722.log 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37380.log 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.89477.log.gz 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36175.log 2024-03-04T06:32:34.683 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30722.log.gz 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87627.log 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37380.log.gz 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36175.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36175.log.gz 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49631.log 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44211.log 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.87627.log.gz 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88889.log 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49631.log.gz 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38126.log 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44211.log.gz 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88889.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.88889.log.gz 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76412.log 2024-03-04T06:32:34.684 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57364.log 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38126.log.gz 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76412.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76412.log.gz 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50380.log 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32481.log 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57364.log.gz 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82628.log 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50380.log.gz 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30951.log 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32481.log.gz 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82628.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82628.log.gz 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54900.log 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.30951.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.59327.log 2024-03-04T06:32:34.685 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30951.log.gz 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51534.log 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.54900.log.gz 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50220.log 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.59327.log.gz 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51534.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.51534.log.gz 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53958.log 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76332.log 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50220.log.gz 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43981.log 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.53958.log.gz 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56616.log 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76332.log.gz 2024-03-04T06:32:34.686 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43981.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43981.log.gz 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55608.log 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87165.log 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.56616.log.gz 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55608.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55608.log.gz 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89057.log 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62812.log 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87165.log.gz 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89057.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.89057.log.gz 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39098.log 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44598.log 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62812.log: /var/log/ceph/ceph-client.admin.39098.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.62812.log.gz 2024-03-04T06:32:34.687 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55468.log 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39098.log.gz 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53015.log 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44598.log.gz 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57811.log 50.0% 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55468.log.gz 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51134.log 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.53015.log.gz 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57315.log 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57811.log.gz 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77720.log 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.51134.log.gz 2024-03-04T06:32:34.688 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87795.log 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57315.log.gz 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87543.log 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.77720.log.gz 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87795.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87795.log.gz 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60023.log 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70850.log 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87543.log.gz 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72137.log 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.60023.log.gz 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69214.log 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70850.log.gz 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72137.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.72137.log.gz 2024-03-04T06:32:34.689 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58510.log 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66387.log 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69214.log.gz 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58510.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58510.log.gz -5 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50080.log 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32659.log 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.66387.log.gz 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50080.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50080.log.gzgzip 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65897.log 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40185.log 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32659.log.gz 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65897.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.65897.log.gz 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43761.log 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64461.log 2024-03-04T06:32:34.690 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40185.log.gz 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64635.log 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43761.log.gz 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71906.log 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.64461.log.gz 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79604.log 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.64635.log.gz 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83468.log 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71906.log.gz 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58559.log 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.79604.log.gz 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48470.log 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83468.log.gz 2024-03-04T06:32:34.691 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58559.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.58559.log.gzgzip 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.83930.log 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74220.log 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48470.log.gz 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83930.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83930.log.gz 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66247.log 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78054.log 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.74220.log.gz 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66247.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.66247.log.gz 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75168.log 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31100.log 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.78054.log.gz 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75168.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75168.log.gz 2024-03-04T06:32:34.692 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44079.log 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66163.log 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31100.log.gz 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44079.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44079.log.gz 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47244.log 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49413.log 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66163.log.gz 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53253.log 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47244.log.gz 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61097.log 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49413.log.gz 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49104.log 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.53253.log.gz 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74460.log 2024-03-04T06:32:34.693 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.61097.log.gz 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49104.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49104.log.gz 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50478.log 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76781.log 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.74460.log.gz 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50478.log: gzip 49.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.50478.log.gz 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.73773.log 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51625.log 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76781.log.gz 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73773.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.73773.log.gz 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60570.log 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77639.log 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51625.log.gz 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60570.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.60570.log.gz 2024-03-04T06:32:34.694 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59461.log 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78448.log 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77639.log: /var/log/ceph/ceph-client.admin.59461.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.77639.log.gz 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56014.log 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.59461.log.gz 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83972.log 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.78448.log.gz 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56014.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56014.log.gz 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81368.log 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31861.log 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83972.log.gz 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61426.log 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81368.log.gz 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33732.log 2024-03-04T06:32:34.695 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31861.log.gz 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72375.log 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.61426.log.gz 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32679.log 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33732.log.gz 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89183.log 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.72375.log.gz 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.32679.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33614.log 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32679.log.gz 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68661.log 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.89183.log.gz 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35031.log 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33614.log.gz 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68661.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.68661.log.gz 2024-03-04T06:32:34.696 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.74509.log 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62084.log 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35031.log.gz 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59983.log 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74509.log.gz 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68150.log 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62084.log.gz 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59983.log: gzip 49.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.59983.log.gz 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32401.log 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75793.log 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.68150.log.gz 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32401.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32401.log.gz 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89225.log 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40711.log 2024-03-04T06:32:34.697 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75793.log.gz 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89225.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.89225.log.gz -5 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47910.log 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80192.log 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40711.log.gz 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47910.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47910.log.gz 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77680.log 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33821.log 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80192.log.gz 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77680.log: gzip -5 50.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.77680.log.gz 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr: /var/log/ceph/ceph-client.admin.64272.log 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.31743.log 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33821.log.gz 2024-03-04T06:32:34.698 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49982.log 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.64272.log.gz 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81956.log 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.0.31743.log: 50.4% -- replaced with /var/log/ceph/ceph-client.0.31743.log.gz 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60441.log 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49982.log.gz 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56427.log 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81956.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81956.log.gz 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73418.log 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.60441.log.gz 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60877.log 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.56427.log.gz 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40236.log 2024-03-04T06:32:34.699 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73418.log.gz 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49671.log 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.60877.log.gz 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76690.log 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40236.log.gz 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49671.log: gzip -5 --verbose -- 49.6% /var/log/ceph/ceph-client.admin.33285.log -- replaced with /var/log/ceph/ceph-client.admin.49671.log.gz 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51309.log 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76690.log.gz 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87585.log 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33285.log.gz 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48449.log 2024-03-04T06:32:34.700 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51309.log.gz 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87585.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.87585.log.gz 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65708.log 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58608.log 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48449.log.gz 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65708.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.65708.log.gz 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34991.log 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55699.log 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58608.log.gz 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44170.log 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34991.log.gz 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30277.log 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55699.log.gz 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44170.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44170.log.gz 2024-03-04T06:32:34.701 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54981.log 2024-03-04T06:32:34.702 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86745.log 2024-03-04T06:32:34.702 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30277.log.gz 2024-03-04T06:32:34.702 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54981.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.57048.log -- replaced with /var/log/ceph/ceph-client.admin.54981.log.gz 2024-03-04T06:32:34.702 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-04T06:32:34.702 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61734.log 2024-03-04T06:32:34.702 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86745.log.gz 2024-03-04T06:32:34.702 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57048.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.57048.log.gz 2024-03-04T06:32:34.702 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.77019.log 2024-03-04T06:32:34.702 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77188.log 2024-03-04T06:32:34.702 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.61734.log.gz 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88595.log 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.77019.log.gz 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56098.log 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.77188.log.gz 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51954.log 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.88595.log.gz 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31184.log 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56098.log.gz 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55972.log 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51954.log.gz 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31268.log 2024-03-04T06:32:34.703 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31184.log.gz 2024-03-04T06:32:34.704 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78512.log 2024-03-04T06:32:34.704 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55972.log.gz 2024-03-04T06:32:34.704 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33185.log 2024-03-04T06:32:34.704 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31268.log.gz 2024-03-04T06:32:34.704 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84602.log 2024-03-04T06:32:34.704 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.78512.log.gz 2024-03-04T06:32:34.704 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72956.log 2024-03-04T06:32:34.704 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33185.log.gz 2024-03-04T06:32:34.704 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37966.log 2024-03-04T06:32:34.704 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84602.log.gz 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51912.log 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.72956.log.gz 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42455.log 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37966.log.gz 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75088.log 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51912.log.gz 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88637.log 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42455.log.gz 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65659.log 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75088.log.gz 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78764.log 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88637.log.gz 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30783.log 2024-03-04T06:32:34.705 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.65659.log.gz 2024-03-04T06:32:34.706 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43721.log 2024-03-04T06:32:34.706 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.78764.log.gz 2024-03-04T06:32:34.706 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85989.log 2024-03-04T06:32:34.706 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30783.log.gz 2024-03-04T06:32:34.706 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85947.log 2024-03-04T06:32:34.706 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43721.log.gz 2024-03-04T06:32:34.706 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74320.log 2024-03-04T06:32:34.706 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85989.log.gz 2024-03-04T06:32:34.706 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67106.log 2024-03-04T06:32:34.706 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85947.log.gz 2024-03-04T06:32:34.706 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45581.log 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74320.log.gz 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38671.log 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67106.log.gz 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30218.log 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45581.log.gz 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77886.log 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38671.log.gz 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51814.log 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30218.log.gz 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46527.log 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.77886.log.gz 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31999.log 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51814.log.gz 2024-03-04T06:32:34.707 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50260.log 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46527.log.gz 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53064.log 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31999.log.gz 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45223.log 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.50260.log.gz 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87711.log 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.53064.log.gz 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65570.log 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45223.log.gz 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31651.log 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87711.log.gz 2024-03-04T06:32:34.708 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62861.log 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.65570.log.gz 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37073.log 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31651.log.gz 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80024.log 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.62861.log.gz 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32441.log 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37073.log.gz 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80150.log 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.80024.log.gz 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68710.log 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32441.log.gz 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36811.log 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80150.log.gz 2024-03-04T06:32:34.709 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.68710.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30890.log 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.68710.log.gz 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36811.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.70585.log 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36811.log.gz 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70326.log 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30890.log.gz 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61650.log 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70585.log.gz 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79856.log 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70326.log.gz 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84812.log 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.61650.log.gz 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59245.log 2024-03-04T06:32:34.710 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.79856.log.gz 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45999.log 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84812.log.gz 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40669.log 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.59245.log.gz 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41503.log 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45999.log.gz 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85779.log 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40669.log.gz 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41503.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41503.log.gz -5 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.72235.log 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30278.log 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85779.log.gz 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60530.log 2024-03-04T06:32:34.711 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.72235.log.gz 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67415.log 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30278.log.gz 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51863.log 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.60530.log.gz 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68290.log 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.67415.log.gz 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52399.log 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51863.log.gz 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33941.log 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.68290.log.gz 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68430.log 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52399.log.gz 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39516.log 2024-03-04T06:32:34.712 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33941.log.gz 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60748.log 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.68430.log.gz 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79730.log 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39516.log.gz 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69438.log 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.60748.log.gz 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44128.log 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79730.log.gz 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46407.log 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69438.log.gz 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48837.log 2024-03-04T06:32:34.713 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44128.log.gz 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35558.log 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46407.log.gz 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83888.log 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48837.log.gz 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63566.log 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35558.log.gz 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83846.log 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83888.log.gz 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67745.log 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.63566.log.gz 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32739.log 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83846.log.gz 2024-03-04T06:32:34.714 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82124.log 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67745.log.gz 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43183.log 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32739.log.gz 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39376.log 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82124.log.gz 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37748.log 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43183.log.gz 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88257.log 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39376.log.gz 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51442.log 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37748.log.gz 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80486.log 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.88257.log.gz 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78324.log 2024-03-04T06:32:34.715 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51442.log.gz 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72865.log 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80486.log.gz 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35609.log 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.78324.log.gz 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34951.log 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.72865.log.gz 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49144.log 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35609.log.gz 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35866.log 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34951.log.gz 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72725.log 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49144.log.gz 2024-03-04T06:32:34.716 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77326.log 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35866.log.gz 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60650.log 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.72725.log.gz 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48308.log 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.77326.log.gz 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57091.log 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.60650.log.gz 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77570.log 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48308.log.gz 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41463.log 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57091.log.gz 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46797.log 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.77570.log.gz 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65301.log 2024-03-04T06:32:34.717 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41463.log.gz 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79646.log 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46797.log.gz 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82040.log 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.65301.log.gz 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39774.log 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79646.log.gz 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78932.log 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82040.log.gz 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59903.log 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39774.log.gz 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78321.log 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.78932.log.gz 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31586.log 2024-03-04T06:32:34.718 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.59903.log.gz 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62273.log 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.78321.log.gz 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31586.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31586.log.gz 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75893.log 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37113.log 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62273.log.gz 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80360.log 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.75893.log.gz 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69837.log 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37113.log.gz 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72095.log 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80360.log.gz 2024-03-04T06:32:34.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86619.log 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.69837.log.gz 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80108.log 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.72095.log.gz 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35742.log 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86619.log.gz 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35784.log 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.80108.log.gz 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83594.log 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35742.log.gz 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52519.log 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35784.log.gz 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42148.log 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83594.log.gz 2024-03-04T06:32:34.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62496.log 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52519.log.gz 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59287.log 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42148.log.gz 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61055.log 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.62496.log.gz 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38691.log 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.59287.log.gz 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33376.log 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.61055.log.gz 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82376.log 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38691.log.gz 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82964.log 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.33376.log.gz 2024-03-04T06:32:34.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85316.log 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82376.log.gz 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52786.log 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82964.log.gz 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57224.log 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85316.log.gz 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80906.log 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52786.log.gz 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59147.log 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.57224.log.gz 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33007.log 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80906.log.gz 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88131.log 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.59147.log.gz 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44261.log 2024-03-04T06:32:34.722 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33007.log.gz 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33236.log 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88131.log.gz 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42595.log 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44261.log.gz 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33236.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33236.log.gz 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69886.log 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42595.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.42595.log.gz 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64501.log 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76199.log 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.69886.log.gz 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74271.log 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.64501.log.gz 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74180.log 2024-03-04T06:32:34.723 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76199.log.gz 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49064.log 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.74271.log.gz 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51772.log 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74180.log.gz 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54496.log 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49064.log.gz 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56238.log 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51772.log.gz 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55343.log 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.54496.log.gz 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59635.log 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56238.log.gz 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37252.log 2024-03-04T06:32:34.724 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55343.log.gz 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50607.log 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.59635.log.gz 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79520.log 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37252.log.gz 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62588.log 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50607.log.gz 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60917.log 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79520.log.gz 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40760.log 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.62588.log.gz 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65401.log 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.60917.log.gz 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54147.log 2024-03-04T06:32:34.725 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40760.log.gz 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45630.log 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.65401.log.gz 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64902.log 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.54147.log.gz 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69081.log 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45630.log.gz 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71619.log 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.64902.log.gz 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54456.log 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69081.log.gz 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88427.log 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71619.log.gz 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74781.log 2024-03-04T06:32:34.726 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.54456.log.gz 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88469.log 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.88427.log.gz 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64114.log 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74781.log.gz 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67065.log 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.88469.log.gz 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55110.log 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.64114.log.gz 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31817.log 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67065.log.gz 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38760.log 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.55110.log.gz 2024-03-04T06:32:34.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32838.log 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31817.log.gz 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64093.log 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38760.log.gz 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85863.log 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32838.log.gz 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52706.log 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.64093.log.gz 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70543.log 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.85863.log.gz 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88805.log 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52706.log.gz 2024-03-04T06:32:34.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52966.log 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70543.log.gz 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47414.log 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88805.log.gz 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54307.log 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52966.log.gz 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56287.log 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47414.log.gz 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54754.log 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.54307.log.gz 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59511.log 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.56287.log.gz 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30614.log 2024-03-04T06:32:34.729 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.54754.log.gz 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67604.log 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.59511.log.gz 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44816.log 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30614.log.gz 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65799.log 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67604.log.gz 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63348.log 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44816.log.gz 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43000.log 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.65799.log.gz 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31469.log 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63348.log.gz 2024-03-04T06:32:34.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50825.log 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43000.log.gz 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81620.log 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31469.log.gz 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69389.log 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.50825.log.gz 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75306.log 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81620.log.gz 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71230.log 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69389.log.gz 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52141.log 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.75306.log.gz 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50994.log 2024-03-04T06:32:34.731 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71230.log.gz 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44638.log 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52141.log.gz 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58797.log 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50994.log.gz 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87249.log 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44638.log.gz 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86115.log 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.58797.log.gz 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42553.log 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.87249.log.gz 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62981.log 2024-03-04T06:32:34.732 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86115.log.gz 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67287.log 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42553.log.gz 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34862.log 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.62981.log.gz 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31226.log 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.67287.log.gz 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86703.log 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34862.log.gz 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47284.log 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31226.log.gz 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64053.log 2024-03-04T06:32:34.733 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86703.log.gz 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41067.log 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47284.log.gz 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83762.log 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.64053.log.gz 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35692.log 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41067.log.gz 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75704.log 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83762.log.gz 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44436.log 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35692.log.gz 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44994.log 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.75704.log.gz 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48917.log 2024-03-04T06:32:34.734 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44436.log.gz 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32579.log 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44994.log.gz 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76025.log 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48917.log.gz 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83174.log 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32579.log.gz 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33990.log 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76025.log.gz 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79268.log 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83174.log.gz 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31310.log 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33990.log.gz 2024-03-04T06:32:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50300.log 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79268.log.gz 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31310.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31310.log.gz 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45283.log 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50300.log.gz 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40978.log 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48079.log 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45283.log.gz 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30109.log 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40978.log.gz 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82670.log 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48079.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48079.log.gz 2024-03-04T06:32:34.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43939.log 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30109.log.gz 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29832.log 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82670.log.gz 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64412.log 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43939.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43939.log.gz 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54594.log 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29832.log.gz 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86409.log 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.64412.log.gz 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60161.log 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54594.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.54594.log.gz 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86955.log 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.86409.log.gz 2024-03-04T06:32:34.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71170.log 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.60161.log.gz 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76550.log 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86955.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86955.log.gz 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81662.log 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71170.log.gz 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69579.log 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76550.log.gz 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59755.log 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81662.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81662.log.gz 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52181.log 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.69579.log.gz 2024-03-04T06:32:34.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36133.log 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.59755.log.gz 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78680.log 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52181.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52181.log.gz 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63588.log 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36133.log.gz 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46358.log 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.78680.log.gz 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38978.log 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63588.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63588.log.gz 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39187.log 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46358.log.gz 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46447.log 2024-03-04T06:32:34.739 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38978.log.gz 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76970.log 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39187.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39187.log.gz 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53155.log 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46447.log.gz 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.76970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70723.log 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.76970.log.gz 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68003.log 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.53155.log.gz 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70723.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70723.log.gzgzip 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.84392.log 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61195.log 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68003.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.68003.log.gz 2024-03-04T06:32:34.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56965.log 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84392.log.gz 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58657.log 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.61195.log.gz 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73005.log 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.56965.log.gz 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87081.log 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58657.log.gz 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83636.log 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73005.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73005.log.gz 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49184.log 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87081.log.gz 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69481.log 2024-03-04T06:32:34.741 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83636.log.gz 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74873.log 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49184.log.gz 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49933.log 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.69481.log.gz 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64851.log 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74873.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74873.log.gz 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46847.log 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49933.log.gz 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35120.log 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.64851.log.gz 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53204.log 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46847.log.gz 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79310.log 2024-03-04T06:32:34.742 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35120.log.gz 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36575.log 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.53204.log.gz 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50340.log 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79310.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79310.log.gz 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70095.log 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36575.log.gz 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78096.log 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50340.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.50340.log.gz 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70095.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.33096.log 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70095.log.gz 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42853.log 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78096.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.78096.log.gz 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75535.log 2024-03-04T06:32:34.743 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33096.log.gz 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42853.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42853.log.gz 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85905.log 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84182.log 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75535.log.gz 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71717.log 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85905.log.gz 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51400.log 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84182.log.gz 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71717.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71717.log.gz 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32788.log 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51400.log.gz 2024-03-04T06:32:34.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29552.log 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77970.log 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32788.log.gz 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86073.log 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29552.log.gz 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71291.log 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.77970.log.gz 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63388.log 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.86073.log.gz 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29289.log 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4%/var/log/ceph/ceph-client.admin.63388.log: -- replaced with /var/log/ceph/ceph-client.admin.71291.log.gz 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88511.log 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.63388.log.gz 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29289.log: gzip -5 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29289.log.gz 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.70501.log 2024-03-04T06:32:34.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88511.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88511.log.gz 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61814.log 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45959.log 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70501.log.gz 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60610.log 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.61814.log.gz 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85737.log 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.45959.log.gz 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60699.log 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.60610.log.gz 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84644.log 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85737.log.gz 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-04T06:32:34.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41890.log 2024-03-04T06:32:34.747 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.60699.log.gz 2024-03-04T06:32:34.747 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75397.log 2024-03-04T06:32:34.747 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84644.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84644.log.gz 2024-03-04T06:32:34.747 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65530.log 2024-03-04T06:32:34.747 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41890.log.gz 2024-03-04T06:32:34.747 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.75397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66716.log 2024-03-04T06:32:34.747 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.75397.log.gz 2024-03-04T06:32:34.747 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35252.log 2024-03-04T06:32:34.747 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.65530.log.gz 2024-03-04T06:32:34.747 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38086.log 2024-03-04T06:32:34.747 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.66716.log.gz 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61854.log 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35252.log.gz 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88384.log 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38086.log.gz 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65003.log 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.61854.log.gz 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34813.log 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88384.log.gz 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86031.log 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.65003.log.gz 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43337.log 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34813.log.gz 2024-03-04T06:32:34.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51485.log 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86031.log.gz 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31372.log 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43337.log.gz 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71808.log 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51485.log.gz 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68479.log 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31372.log.gz 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40620.log 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.71808.log.gz 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72816.log 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.68479.log.gz 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65123.log 2024-03-04T06:32:34.749 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40620.log.gz 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78848.log 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.72816.log.gz 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30426.log 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.65123.log.gz 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68990.log 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.78848.log.gz 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87039.log 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30426.log.gz 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80402.log 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.68990.log.gz 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43841.log 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87039.log.gz 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34466.log 2024-03-04T06:32:34.750 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80402.log.gz 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65352.log 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43841.log.gz 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78806.log 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34466.log.gz 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34604.log 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.65352.log.gz 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43492.log 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.78806.log.gz 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41801.log 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34604.log.gz 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40318.log 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43492.log.gz 2024-03-04T06:32:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70941.log 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41801.log.gz 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58251.log 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40318.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40318.log.gz 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71997.log 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.70941.log.gz 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46567.log 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58251.log.gz 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89393.log 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.71997.log.gz 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61992.log 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46567.log.gz 2024-03-04T06:32:34.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37868.log 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.89393.log.gz 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39872.log 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.61992.log.gz 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37868.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37868.log.gz 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54238.log 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68948.log 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39872.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39872.log.gz 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55839.log 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.54238.log.gz 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54189.log 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.68948.log.gz 2024-03-04T06:32:34.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38929.log 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.55839.log.gz 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83510.log 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.54189.log.gz 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43632.log 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38929.log.gz 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83510.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83510.log.gz 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36715.log 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38315.log 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43632.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43632.log.gz 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49853.log 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36715.log.gz 2024-03-04T06:32:34.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77068.log 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.38315.log.gz 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63775.log 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49853.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49853.log.gz 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.77068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51225.log 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.77068.log.gz 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83804.log 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.63775.log.gz 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.51225.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49511.log 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51225.log.gz 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83804.log: gzip 50.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.83804.log.gz -- 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr: /var/log/ceph/ceph-client.admin.68241.log 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58333.log 2024-03-04T06:32:34.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49511.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49511.log.gz 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88973.log 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.68241.log.gz 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87963.log 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58333.log.gz 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42813.log 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.88973.log.gz 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.87963.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87963.log.gz 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79982.log 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66796.log 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42813.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42813.log.gz 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89435.log 2024-03-04T06:32:34.756 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79982.log.gz 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.66796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65450.log 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.66796.log.gz 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36666.log 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89435.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.89435.log.gz 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86787.log 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.65450.log.gz 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78974.log 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36666.log.gz 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56696.log 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86787.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86787.log.gz 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55250.log 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.78974.log.gz 2024-03-04T06:32:34.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82166.log 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.56696.log.gz 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58877.log 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55250.log.gz 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.82166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38364.log 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82166.log.gz 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86493.log 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.58877.log.gz 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42644.log 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38364.log.gz 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.86493.log: gzip 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86493.log.gz -5 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.69748.log 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63259.log 2024-03-04T06:32:34.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42644.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42644.log.gz 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39823.log 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.69748.log.gz 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42504.log 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.63259.log.gz 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41841.log 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39823.log: /var/log/ceph/ceph-client.admin.42504.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39823.log.gz 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64222.log 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42504.log.gz 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39586.log 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41841.log.gz 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69628.log 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.64222.log.gz 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73733.log 2024-03-04T06:32:34.759 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39586.log.gz 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48407.log 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69628.log.gz 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65163.log 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73733.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73733.log.gz 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48257.log 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48407.log.gz 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44687.log 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.65163.log.gz 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36953.log 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48257.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48257.log.gz 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68619.log 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44687.log.gz 2024-03-04T06:32:34.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59675.log 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.36953.log.gz 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53393.log 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.68619.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.68619.log.gz 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44344.log 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.59675.log.gz 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88299.log 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.53393.log.gz 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51267.log 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44344.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44344.log.gz 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65610.log 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.88299.log.gz 2024-03-04T06:32:34.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40411.log 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51267.log.gz 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61335.log 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65610.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.65610.log.gz 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80780.log 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40411.log.gz 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69123.log 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.61335.log.gz 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45490.log 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80780.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80780.log.gz 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34644.log 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.69123.log.gz 2024-03-04T06:32:34.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31058.log 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45490.log.gz 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41276.log 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34644.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34644.log.gz 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.31058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42286.log 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31058.log.gz 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58917.log 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41276.log.gz 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79142.log 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42286.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42286.log.gz 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51092.log 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58917.log.gz 2024-03-04T06:32:34.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69340.log 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.79142.log.gz 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81872.log 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51092.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.51092.log.gz 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85442.log 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69340.log.gz 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.81872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41116.log 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81872.log.gz 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37162.log 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85442.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85442.log.gz 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31142.log 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41116.log.gz 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30742.log 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.37162.log.gz 2024-03-04T06:32:34.764 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.31142.log: -- /var/log/ceph/ceph-client.admin.32223.log 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31142.log.gz 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30742.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30742.log.gz 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39327.log 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63647.log 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32223.log.gz 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39467.log 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39327.log.gz 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58118.log 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.63647.log.gz 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5/var/log/ceph/ceph-client.admin.39467.log: --verbose -- /var/log/ceph/ceph-client.admin.60837.log 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39467.log.gz 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58118.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58118.log.gz 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29966.log 2024-03-04T06:32:34.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30386.log 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.60837.log.gz 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32530.log 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29966.log.gz 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47592.log 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30386.log.gz 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42951.log 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32530.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32530.log.gz 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62182.log 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47592.log.gz 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56518.log 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42951.log.gz 2024-03-04T06:32:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72683.log 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.62182.log.gz 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71130.log 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56518.log.gz 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84938.log 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.72683.log.gz 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69788.log 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71130.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71130.log.gz 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58160.log 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84938.log.gz 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.69788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39914.log 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.69788.log.gz 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88005.log 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58160.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58160.log.gz 2024-03-04T06:32:34.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53780.log 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39914.log.gz 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59420.log 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.88005.log.gz 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84770.log 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53780.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.53780.log.gz 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89351.log 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.59420.log.gz 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84770.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84770.log.gz 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74551.log 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61943.log 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.89351.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.89351.log.gz 2024-03-04T06:32:34.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60350.log 2024-03-04T06:32:34.769 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.74551.log.gz 2024-03-04T06:32:34.769 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54674.log 2024-03-04T06:32:34.769 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61943.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.61943.log.gz 2024-03-04T06:32:34.769 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85064.log 2024-03-04T06:32:34.769 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.60350.log.gz 2024-03-04T06:32:34.769 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32090.log 2024-03-04T06:32:34.769 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.54674.log.gz 2024-03-04T06:32:34.769 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53482.log 2024-03-04T06:32:34.769 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85064.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.85064.log.gz 2024-03-04T06:32:34.769 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83090.log 2024-03-04T06:32:34.769 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32090.log.gz 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48039.log 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.53482.log.gz 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55388.log 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83090.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83090.log.gz 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88215.log 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48039.log.gz 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30612.log 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55388.log.gz 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35427.log 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.88215.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.88215.log.gz 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36350.log 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30612.log.gz 2024-03-04T06:32:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74740.log 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35427.log.gz 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34201.log 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36350.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36350.log.gz 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.74740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33514.log 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.74740.log.gz 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37033.log 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34201.log.gz 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61774.log 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33514.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33514.log.gz 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36035.log 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.37033.log.gz 2024-03-04T06:32:34.771 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63210.log 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.61774.log.gz 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73684.log 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36035.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36035.log.gz 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79478.log 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.63210.log.gz 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84728.log 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73684.log.gz 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85358.log 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.79478.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.79478.log.gz 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.84728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55061.log 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84728.log.gz 2024-03-04T06:32:34.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46665.log 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85358.log.gz 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58957.log 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55061.log.gz 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42326.log 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46665.log.gz 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73551.log 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58957.log.gz 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71032.log 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42326.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42326.log.gz 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54852.log 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.73551.log.gz 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65083.log 2024-03-04T06:32:34.773 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.71032.log.gz 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33474.log 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54852.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.54852.log.gz 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.65083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80444.log 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.65083.log.gz 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30110.log 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33474.log.gz 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67794.log 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.80444.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80444.log.gz 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.30110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32887.log 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30110.log.gz 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.67794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43890.log 2024-03-04T06:32:34.774 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.67794.log.gz 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56825.log 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.32887.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32887.log.gz 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71210.log 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43890.log.gz 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39285.log 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56825.log.gz 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56915.log 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.71210.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.71210.log.gz 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78428.log 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39285.log.gz 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73054.log 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.56915.log.gz 2024-03-04T06:32:34.775 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55201.log 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.78428.log.gz 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.73054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70683.log 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.73054.log.gz 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42197.log 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.55201.log.gz 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83426.log 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.70683.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.70683.log.gz 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57140.log 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42197.log.gz 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.83426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34248.log 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83426.log.gz 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51330.log 2024-03-04T06:32:34.776 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57140.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.57140.log.gz 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.34248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44030.log 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.34248.log.gz 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41374.log 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51330.log.gz 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37828.log 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44030.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44030.log.gz 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59378.log 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41374.log.gz 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60112.log 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37828.log.gz 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72599.log 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59378.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.59378.log.gz 2024-03-04T06:32:34.777 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39734.log 2024-03-04T06:32:34.778 INFO:teuthology.orchestra.run.smithi064.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.60112.log.gz 2024-03-04T06:32:34.778 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.72599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85106.log 2024-03-04T06:32:34.778 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.72599.log.gz 2024-03-04T06:32:34.778 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29368.log 2024-03-04T06:32:34.778 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39734.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39734.log.gz 2024-03-04T06:32:34.778 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.85106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78322.log 2024-03-04T06:32:34.778 INFO:teuthology.orchestra.run.smithi064.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85106.log.gz 2024-03-04T06:32:34.778 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.29368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58069.log 2024-03-04T06:32:34.778 INFO:teuthology.orchestra.run.smithi064.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29368.log.gz 2024-03-04T06:32:34.778 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.78322.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.78322.log.gz 2024-03-04T06:32:34.778 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58069.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.58069.log.gz 2024-03-04T06:32:35.112 INFO:teuthology.orchestra.run.smithi064.stderr: 89.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-04T06:32:39.385 INFO:teuthology.orchestra.run.smithi064.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-04T06:32:39.675 INFO:teuthology.orchestra.run.smithi066.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-04T06:32:39.795 INFO:teuthology.orchestra.run.smithi064.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-04T06:32:39.797 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-04T06:32:39.797 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m5.750s 2024-03-04T06:32:39.797 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m11.734s 2024-03-04T06:32:39.797 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.865s 2024-03-04T06:32:40.957 INFO:teuthology.orchestra.run.smithi066.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-04T06:32:40.959 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-04T06:32:40.959 INFO:teuthology.orchestra.run.smithi066.stderr:real 0m6.865s 2024-03-04T06:32:40.959 INFO:teuthology.orchestra.run.smithi066.stderr:user 0m12.197s 2024-03-04T06:32:40.959 INFO:teuthology.orchestra.run.smithi066.stderr:sys 0m0.412s 2024-03-04T06:32:40.960 INFO:tasks.ceph:Archiving logs... 2024-03-04T06:32:40.960 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638/remote/smithi064/log 2024-03-04T06:32:40.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T06:32:43.865 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/log/ceph to /home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638/remote/smithi066/log 2024-03-04T06:32:43.866 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T06:32:45.867 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/thrashosds.py", line 216, in task cluster_manager.flush_all_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1762, in flush_all_pg_stats self.flush_pg_stats(range(len(self.get_osd_dump()))) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1759, in flush_pg_stats p.spawn(flush_one_osd, osd, wait_for_mon) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1737, in flush_one_osd need = int(self.raw_cluster_cmd('tell', 'osd.%d' % osd, 'flush_pg_stats')) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2778, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2718, in is_clean self.dump_pgs_not_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2739, in dump_pgs_not_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-04T06:32:45.868 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-04T06:32:45.877 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/thrashosds.py", line 216, in task cluster_manager.flush_all_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1762, in flush_all_pg_stats self.flush_pg_stats(range(len(self.get_osd_dump()))) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1759, in flush_pg_stats p.spawn(flush_one_osd, osd, wait_for_mon) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1737, in flush_one_osd need = int(self.raw_cluster_cmd('tell', 'osd.%d' % osd, 'flush_pg_stats')) File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_633e69c9dc5e6e1fdcdd29ad1365e50bee5ffccd/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' 2024-03-04T06:32:45.878 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-04T06:32:45.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T06:32:45.881 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T06:32:45.972 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-04T06:32:45.982 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-04T06:32:45.983 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T06:32:45.985 DEBUG:teuthology.orchestra.run.smithi066:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T06:32:45.999 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-04T06:32:46.000 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-03-04T06:32:46.003 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T06:32:46.003 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-04T06:32:46.003 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 63 +1386us[+1355us] +/- 99ms 2024-03-04T06:32:46.003 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 7 377 62 -952us[ -983us] +/- 54ms 2024-03-04T06:32:46.003 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 +177us[ +146us] +/- 79ms 2024-03-04T06:32:46.004 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T06:32:46.005 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T06:32:46.005 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-03-04T06:32:46.005 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 +1378us[+1378us] +/- 99ms 2024-03-04T06:32:46.005 INFO:teuthology.orchestra.run.smithi066.stdout:^* hv02.front.sepia.ceph.com 3 7 377 63 -902us[ -917us] +/- 54ms 2024-03-04T06:32:46.005 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +248us[ +248us] +/- 79ms 2024-03-04T06:32:46.005 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T06:32:46.006 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-04T06:32:46.015 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-04T06:32:46.016 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-04T06:32:46.027 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T06:32:46.144 DEBUG:teuthology.orchestra.run.smithi066:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T06:32:46.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T06:32:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T06:32:46.251 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T06:32:46.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T06:32:46.285 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-04T06:32:46.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-04T06:32:46.336 INFO:teuthology.task.internal:Duration was 2072.072740 seconds 2024-03-04T06:32:46.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-04T06:32:46.348 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-04T06:32:46.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T06:32:46.350 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T06:32:46.392 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T06:32:46.402 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T06:32:46.755 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-04T06:32:46.755 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-03-04T06:32:46.755 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T06:32:46.815 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi066.front.sepia.ceph.com 2024-03-04T06:32:46.815 DEBUG:teuthology.orchestra.run.smithi066:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T06:32:46.845 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-04T06:32:46.845 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T06:32:46.858 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T06:32:46.918 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-04T06:32:46.919 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T06:32:46.927 DEBUG:teuthology.orchestra.run.smithi066:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T06:32:47.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-04T06:32:47.083 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-04T06:32:47.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T06:32:47.109 DEBUG:teuthology.orchestra.run.smithi066:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T06:32:47.140 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-04T06:32:47.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T06:32:47.155 DEBUG:teuthology.orchestra.run.smithi066:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T06:32:47.178 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-03-04T06:32:47.209 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = core 2024-03-04T06:32:47.235 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T06:32:47.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T06:32:47.260 DEBUG:teuthology.orchestra.run.smithi066:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T06:32:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T06:32:47.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-04T06:32:47.302 INFO:teuthology.task.internal:Transferring archived files... 2024-03-04T06:32:47.303 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638/remote/smithi064 2024-03-04T06:32:47.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T06:32:47.340 DEBUG:teuthology.misc:Transferring archived files from smithi066:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638/remote/smithi066 2024-03-04T06:32:47.340 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T06:32:47.379 INFO:teuthology.task.internal:Removing archive directory... 2024-03-04T06:32:47.379 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T06:32:47.381 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T06:32:47.428 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-04T06:32:47.441 INFO:teuthology.task.internal:Not uploading archives. 2024-03-04T06:32:47.442 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-04T06:32:47.456 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-04T06:32:47.457 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T06:32:47.459 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T06:32:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 06:32 /home/ubuntu/cephtest 2024-03-04T06:32:47.482 INFO:teuthology.orchestra.run.smithi066.stdout: 265301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 06:32 /home/ubuntu/cephtest 2024-03-04T06:32:47.483 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-04T06:32:47.496 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-04T06:32:47.515 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi064 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' 2024-03-04T06:32:47.516 INFO:teuthology.nuke:Checking targets against current locks 2024-03-04T06:32:47.540 DEBUG:teuthology.nuke:shortname: smithi064 2024-03-04T06:32:47.541 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T06:32:47.563 DEBUG:teuthology.nuke:shortname: smithi066 2024-03-04T06:32:47.563 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T06:32:47.566 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 05:47:51.790222', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlsRiv6BZzoFANvoD9juHdrbxi3MXvw3zBu/1yDTuv1+gmkPLYJCr7XgsZ+YtJJj4DCcQADwtCwRXr5hwaIUMQ='} 2024-03-04T06:32:47.585 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-03-04_05:47:36-crimson-rados:thrash-wip-nitzan-crimson-shard-pg-mapping-get-or-create-distro-default-smithi/7580638', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 05:47:51.788217', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMFZcv76CLd2CM3w5dPYjbX7QYeAj/jEaNR1nJSu76qsrrYM794w1HuRY73Z3wK0f7ezvi7iPqh582AZT/KuZOk='} 2024-03-04T06:32:47.617 INFO:teuthology.orchestra.console.smithi064:Power off 2024-03-04T06:32:47.618 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T06:32:47.636 INFO:teuthology.orchestra.console.smithi066:Power off 2024-03-04T06:32:47.636 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T06:32:47.645 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-03-04T06:32:47.645 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T06:32:47.658 DEBUG:teuthology.orchestra.console.smithi066:power off output: Chassis Power Control: Down/Off 2024-03-04T06:32:47.659 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T06:32:47.667 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-04T06:32:47.682 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-04T06:32:51.667 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T06:32:51.682 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T06:32:51.777 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-04T06:32:51.790 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-04T06:32:55.783 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T06:32:55.793 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T06:32:55.896 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-03-04T06:32:55.903 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-04T06:32:55.997 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-03-04T06:32:59.908 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T06:33:00.021 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is off 2024-03-04T06:33:00.122 INFO:teuthology.orchestra.console.smithi066:Power off completed 2024-03-04T06:33:00.226 INFO:teuthology.run:Summary data: description: crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} duration: 2072.072740316391 failure_reason: 'Command failed on smithi064 with status 6: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats''' flavor: crimson owner: scheduled_nmordech@teuthology status: fail success: false 2024-03-04T06:33:00.226 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T06:33:00.298 INFO:teuthology.run:FAIL